Uncovering Discourse Bias in Comment Classification

Dr. Inbal Yahav of the Social Intelligence Lab at Bar-Ilan University's Graduate School of Business presented groundbreaking joint work with David Schwartz and Onn Shehory, at the 2015 INFORMS annual meeting in Philadelphia PA. Her discussion highlighted the previously unaddressed problem of Discourse Bias in Comment Classification using tf-idf,

From the talk abstract:"Text mining and natural language processing have gained great momentum in recent years, with user-generated content becoming widely available. One key use is comment classification, with much attention being given to sentiment analysis and opinion mining. An essential step in the process of comment classification is text pre-processing; a step in which each linguistic term is assigned with a weight that commonly increases with its appearance in the studied text, yet is offset by the frequency of the term in the domain of interest. A common practice is to use the well-known tf-idf formula to compute these weights. We reveal the bias introduced by between-participants' discourse to the study of comments in social media, and propose a correction. We find that content extracted from between-participants' discourse is often highly correlated, resulting in dependency structures between observations in the study. Ignoring this bias can manifest in a non-robust analysis at best, and can lead to an entirely wrong conclusion at worst. We propose a statistical correction to tf-idf that accounts for this bias. We illustrate the effects of both the bias and correction with real data from Facebook."