Skip to main content
Springer Nature Link
Log in

An Exploratory Study of Conventional Machine Learning and Large Language Models for Sentiment Analysis

  • Conference paper
  • First Online:
HCI International 2024 – Late Breaking Papers (HCII 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15382))

Included in the following conference series:

  • 542 Accesses

  • 2 Citations

Abstract

Sentiment analysis is the use of natural language processing to identify affective states and determine people’s opinions in various analytical applications such as customer reviews and social media analyses. Large language models (LLMs) such as GPT-4o demonstrate impressive performance in text generation tasks. Despite numerous studies in the extant literature, few have compared the performance of conventional machine learning models with LLMs for sentiment analysis. This study aims to fill this gap by conducting an evaluation of these models using a balanced dataset of 2,000 IMDb movie reviews. Our study shows that GPT-4o achieves the highest performance, while GPT-3.5 and FLAN-T5 models also show strong performance, being slightly below that of GPT-4o. Advanced LLMs outperform conventional machine learning models. Our findings highlight the advanced capabilities and user-friendliness of LLMs compared to conventional machine learning models. This research underscores the rapid evolution of LLMs for sentiment analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
€34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (Germany)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 53.49
Price includes VAT (Germany)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 70.61
Price includes VAT (Germany)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

') var buybox = document.querySelector("[data-id=id_"+ timestamp +"]").parentNode var buyingOptions = buybox.querySelectorAll(".buying-option") ;[].slice.call(buyingOptions).forEach(initCollapsibles) var buyboxMaxSingleColumnWidth = 480 function initCollapsibles(subscription, index) { var toggle = subscription.querySelector(".buying-option-price") subscription.classList.remove("expanded") var form = subscription.querySelector(".buying-option-form") var priceInfo = subscription.querySelector(".price-info") var buyingOption = toggle.parentElement if (toggle && form && priceInfo) { toggle.setAttribute("role", "button") toggle.setAttribute("tabindex", "0") toggle.addEventListener("click", function (event) { var expandedBuyingOptions = buybox.querySelectorAll(".buying-option.expanded") var buyboxWidth = buybox.offsetWidth ;[].slice.call(expandedBuyingOptions).forEach(function(option) { if (buyboxWidth buyboxMaxSingleColumnWidth) { toggle.click() } else { if (index === 0) { toggle.click() } else { toggle.setAttribute("aria-expanded", "false") form.hidden = "hidden" priceInfo.hidden = "hidden" } } }) } initialStateOpen() if (window.buyboxInitialised) return window.buyboxInitialised = true initKeyControls() })()

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://paperswithcode.com/sota/sentiment-analysis-on-imdb (accessed on 2024/05/19).

  2. 2.

    Https://huggingface.co/docs/datasets/en/index (accessed on 2024/05/19).

  3. 3.

    Https://platform.openai.com/settings/organization/limits (accessed on 2024/05/19).

  4. 4.

    https://github.com/M-Taghizadeh/flan-t5-base-imdb-text-classification (accessed on 2024/05/19).

  5. 5.

    https://huggingface.co/google/flan-t5-base (accessed on 2024/05/19).

  6. 6.

    https://platform.openai.com/docs/api-reference/ (accessed on 2024/05/19).

References

  • Acheampong, F.A., Nunoo-Mensah, H., Chen, W.: Transformer models for text-based emotion detection: a review of BERT-based approaches. Artif. Intell. Rev. 54(8), 5789–5829 (2021)

    Article  MATH  Google Scholar 

  • Adeborna, E., Siau, K.: An approach to sentiment analysis – the case of airline quality rating. In: Proceedings of the Pacific Asia Conference on Information Systems. Chengdu, China (2014)

    Google Scholar 

  • Baccianella, S., Esuli, A., Sebastiani, F.: SentiWordNet 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In: Proceedings of the International Conference on Language Resources and Evaluation, pp. 2200–2204 (2010, May)

    Google Scholar 

  • Birjali, M., Kasri, M., Beni-Hssane, A.: A comprehensive survey on sentiment analysis: approaches, challenges and trends. Knowl.-Based Syst. 226, 107134 (2021)

    Article  MATH  Google Scholar 

  • Brown, T., et al.: Language models are few-shot learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901 (2020)

    Google Scholar 

  • Cambria, E., Speer, R., Havasi, C., Hussain, A.: SenticNet: a publicly available semantic resource for opinion mining. In: Commonsense Knowledge: Papers from the 2010 AAAI Fall Symposium. Fall Symposium Series Technical Reports, FS-10-02. Arlington, VA (2010)

    Google Scholar 

  • Chung, H.W., et al.: Scaling instruction-finetuned language models. J. Mach. Learn. Res. 25(70), 1–53 (2024)

    MATH  Google Scholar 

  • Domingos, P.: A few useful things to know about machine learning. Commun. ACM 55(10), 78–87 (2012)

    Article  MATH  Google Scholar 

  • Fu, Z., et al.: Decoder-only or encoder-decoder? Interpreting language model as a regularized encoder-decoder. arXiv preprint arXiv:2304.04052 (2023)

  • Joulin, A., Grave, E., Bojanowski, P., Mikolov, T.: Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759 (2016)

  • Kadhim, A.I., Cheah, Y.N., Ahamed, N.H.: Text document preprocessing and dimension reduction techniques for text document clustering. In: 2014 4th International Conference on Artificial Intelligence with Applications in Engineering and Technology, pp. 69–73. IEEE (2014)

    Google Scholar 

  • Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2019)

  • Li, W., Qi, F., Tang, M., Yu, Z.: Bidirectional LSTM with self-attention mechanism and multi-channel features for sentiment classification. Neurocomputing 387, 63–77 (2020)

    Article  MATH  Google Scholar 

  • Nah, F.F.H., Cai, J., Zheng, R., Pang, N.: An activity system-based perspective of generative AI: challenges and research directions. AIS Trans. Hum.-Comput. Interact. 15(3), 247–267 (2023)

    Article  Google Scholar 

  • Nah, F.F.H., Zheng, R., Cai, J., Siau, K., Chen, L.: Generative AI and ChatGPT: applications, challenges, and AI-human collaboration. J. Inform. Technol. Case Appl. Res. 25(3), 277–304 (2023)

    Google Scholar 

  • OpenAI. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/. Accessed 19 May 2024

  • Pan, S.L., Nishant, R., Tuunanen, T., Nah, F.F.H.: Literature review in the generative AI era-how to make a compelling contribution. J. Strat. Inf. Syst. 32(3), 1–4 (2023)

    Article  Google Scholar 

  • Ravindran, S.K., Nah, F.F.H.: Prescriptive analytics: a game changer for business. Cutter Bus. Technol. J. 30(10/11), 11–17 (2017)

    MATH  Google Scholar 

  • Vaswani, A., et al.: Attention is all you need. Adv. Neural Inform. Proces. Syst. 30 (2017)

    Google Scholar 

  • Wang, Z., Xie, Q., Feng, Y., Ding, Z., Yang, Z., Xia, R.: Is ChatGPT a good sentiment analyzer? A preliminary study. arXiv preprint arXiv:2304.04339 (2023)

  • Wankhade, M., Rao, A.C.S., Kulkarni, C.: A survey on sentiment analysis methods, applications, and challenges. Artif. Intell. Rev. 55(7), 5731–5780 (2022)

    Article  MATH  Google Scholar 

  • Yadav, A., Vishwakarma, D.K.: Sentiment analysis using deep learning architectures: a review. Artif. Intell. Rev. 53(6), 4335–4385 (2020)

    Article  MATH  Google Scholar 

  • Yuan, B., Siau, K.: A research stream on sentiment analysis. In: Proceedings of the Americas Conference on Information Systems. Boston, MA (2017)

    Google Scholar 

  • Yuan, B., Siau, K.: Lexicons in sentiment analytics. In: Proceedings of the Twelve Annual Midwest Association for Information Systems Conference. Springfield, IL (2017)

    Google Scholar 

  • Zhang, W., Deng, Y., Liu, B., Pan, S.J., Bing, L.: Sentiment analysis in the era of large language models: a reality check. arXiv preprint arXiv:2305.15005 (2023)

  • Zhao, W., Siau, K.: Machine learning approaches to sentiment analytics. In: Proceedings of the Twelve Annual Midwest Association for Information Systems Conference. Springfield, IL (2017)

    Google Scholar 

  • Zhong, Q., Ding, L., Liu, J., Du, B., Tao, D.: Can ChatGPT understand too? A comparative study on ChatGPT and fine-tuned BERT. arXiv preprint arXiv:2302.10198 (2023)

Download references

Author information

Authors and Affiliations

  1. University of Oklahoma, Norman, OK, 73019, USA

    Cui Zou

  2. City University of Hong Kong, Kowloon Tong, Kowloon, Hong Kong SAR, China

    Jingyuan Cai & Fiona Fui-Hoon Nah

  3. The University of Tulsa, Tulsa, OK, 74104, USA

    Langtao Chen

Authors
  1. Cui Zou
    View author publications

    Search author on:PubMed Google Scholar

  2. Jingyuan Cai
    View author publications

    Search author on:PubMed Google Scholar

  3. Langtao Chen
    View author publications

    Search author on:PubMed Google Scholar

  4. Fiona Fui-Hoon Nah
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Cui Zou .

Editor information

Editors and Affiliations

  1. Siemens Corporation, Princeton, NJ, USA

    Helmut Degen

  2. and Technology - Hellas, Foundation for Research, Heraklion, Crete, Greece

    Stavroula Ntoa

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zou, C., Cai, J., Chen, L., Nah, F.FH. (2024). An Exploratory Study of Conventional Machine Learning and Large Language Models for Sentiment Analysis. In: Degen, H., Ntoa, S. (eds) HCI International 2024 – Late Breaking Papers. HCII 2024. Lecture Notes in Computer Science, vol 15382. Springer, Cham. https://doi.org/10.1007/978-3-031-76827-9_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-76827-9_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-76826-2

  • Online ISBN: 978-3-031-76827-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics