The Use of Machine Learning Classifiers to Evaluate the Sentiment Categories of ChatGPT-Related Tweets
DOI:
https://doi.org/10.64922/jteas.v1i1.19Keywords:
Sentiment Analysis, Natural Language Processing, Machine Learning, ChatGPT, TweetsAbstract
This study evaluates the performance of conventional machine learning classifiers for sentiment analysis of ChatGPT-related tweets. While deep learning approaches have demonstrated advanced capabilities, their substantial computational requirements present practical limitations. Using a dataset of 219,294 tweets from November 2022 to April 2023, we assess four classifiers (Logistic Regression, Support Vector Machines, Random Forest, and Naive Bayes) with two feature extraction techniques (Bag of Words and TF-IDF) across balanced and imbalanced datasets. Results indicate that Linear SVM with TF-IDF vectorization achieves the highest accuracy (84.02%) and macro F1-score (80.81%) without balancing techniques. Random Forest with Bag of Words and balancing techniques shows competitive performance (80.08% macro F1-score), while Naive Bayes consistently underperforms across configurations. n-gram analysis reveals predominantly positive public sentiment toward ChatGPT, focusing on capabilities and utility, while negative sentiments center on performance limitations and broader AI implications. This research provides empirical insights into sentiment analysis methodologies for emerging AI technologies and demonstrates the continued effectiveness of conventional machine learning approaches in resource-constrained environments.
References
Aziz, R. H. H., & Dimililer, N. (2020). Twitter sentiment analysis using an ensemble weighted majority vote classifier. In 2020 International Conference on Advanced Science and Engineering (ICOASE). https://doi.org/10.1109/icoase51841.2020.9436590
Bordoloi, M., & Biswas, S. K. (2023). Sentiment analysis: A survey on design framework, applications and future scopes. Artificial Intelligence Review, 56(11), 12505–12560. https://doi.org/10.1007/s10462-023-10442-2
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates. https://doi.org/10.4324/9780203771587
Dey, A., Jenamani, M., & Thakkar, J. J. (2018). Senti-N-Gram: An N-gram lexicon for sentiment analysis. Expert Systems with Applications, 103, 92–105. https://doi.org/10.1016/j.eswa.2018.03.004
Gehrmann, S., Dernoncourt, F., Li, Y., Carlson, E. T., Wu, J. T., Welt, J., John, F. J., Moseley, E. T., Grant, D. W., Tyler, P. D., & Celi, L. A. (2017). Comparing rule-based and deep learning models for patient phenotyping. arXiv. https://doi.org/10.48550/arxiv.1703.08705
Gifari, M. K., Lhaksmana, K. M., & Dwifebri, P. M. (2021). Sentiment analysis on movie review using ensemble stacking model. In 2021 International Conference Advancement in Data Science, E-learning and Information Systems (ICADEIS). https://doi.org/10.1109/icadeis52521.2021.9702088
Guo, B., Zhang, C., Wang, Z., Guo, J., & Yu, Z. (2023). How close is ChatGPT to human experts? Comparison corpus, evaluation, and detection (arXiv:2301.07597). arXiv. https://doi.org/10.48550/arXiv.2301.07597
Hasan, A., Moin, S., Karim, A., & Shamshirband, S. (2018). Machine learning-based sentiment analysis for Twitter accounts. Mathematical and Computational Applications, 23(1), 11. https://doi.org/10.3390/mca23010011
Kowsari, K., Jafari Meimandi, K., Heidarysafa, M., Mendu, S., Barnes, L., & Brown, D. (2019). Text classification algorithms: A survey. Information, 10(4), 150. https://doi.org/10.3390/info10040150
Leiter, C., Zhang, R., Chen, Y., Belouadi, J., Larionov, D., Fresen, V., & Eger, S. (2023). ChatGPT: A meta-analysis after 2.5 months (arXiv:2302.13795). arXiv. https://doi.org/10.48550/arXiv.2302.13795
Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies, 5(1), 1–167. https://doi.org/10.2200/S00416ED1V01Y201204HLT016
Mercha, E. M., & Benbrahim, H. (2023). Machine learning and deep learning for sentiment analysis across languages: A survey. Neurocomputing, 531, 195–216. https://doi.org/10.1016/j.neucom.2023.02.015
Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002) (pp. 79–86). https://doi.org/10.3115/1118693.1118704
Parveen, H., Hada, B., Lalwani, P., & Gupta, M. (2023). Sentiment analysis on social media using machine learning techniques: A comprehensive review. Multimedia Tools and Applications, 82(14), 21591–21630. https://doi.org/10.1007/s11042-022-14160-9
Sallam, M., Salim, N. A., Barakat, M., & Al-Tammemi, A. B. (2023). ChatGPT applications in medical, dental, pharmacy, and public health education: A descriptive study highlighting the advantages and limitations. Narra J, 3(1), e103. https://doi.org/10.52225/narra.v3i1.103
Sravya, K., Sowmya, G., Yamini, P., Anusha, P., & Krishna, P. S. (2021). Sentiment analysis on Twitter. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3920078
Tan, K. L., Lee, C. P., & Lim, K. M. (2023). A survey of sentiment analysis: Approaches, datasets, and future research. Applied Sciences, 13(7), 4550. https://doi.org/10.3390/app13074550
Tan, K. L., Lee, C. P., Lim, K. M., & Anbananthen, K. S. M. (2022). Sentiment analysis with ensemble hybrid deep learning model. IEEE Access, 10, 103694–103704. https://doi.org/10.1109/access.2022.3210182
Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731–5780. https://doi.org/10.1007/s10462-022-10144-1
Zainuddin, N., & Selamat, A. (2015). Sentiment analysis using support vector machine. In 2014 International Conference on Computer, Communications, and Control Technology (I4CT) (pp. 333–337). https://doi.org/10.1109/i4ct.2014.6914200
Zhang, X., Zhao, J., & LeCun, Y. (2015). Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems (Vol. 28). https://doi.org/10.48550/arXiv.1509.01626
Zhang, Y., & Wallace, B. C. (2015). A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv. https://doi.org/10.48550/arxiv.1510.03820
