Google Scholar Page here
William Lowe and Kenneth Benoit. 2012. “Qualitative Validation of Quantitative Text Scaling“. Paper prepared for presentation at the 70th Annual Conference of the Midwest Political Science Association. Palmer House Hotel, Chicago, 12-15 April 2012. Version: 10 April 2012.
Statistical methods for scaling latent traits from political texts have received widespread attention in political science, typically for measuring the left-right policy positions of political actors. Validation and interpretation of these estimates typically involves a combination of a priori identification of dimensions present in texts examined, external comparison to independent data, as well as basic reasonability standards to establish face validity. In this paper, we apply a new benchmark to validating scaling estimates: qualitative human readings of the texts. Our validation compares human interpreted differences to statistical point estimates, as well as human perceptions of differences to statistically derived confidence intervals. For testing we draw on texts from a budget debate taking place in Ireland in late 2009, implementing a historically unprecedented level of austerity measures, represented by 14 speeches made in the Irish Dáil by key spokespersons from all of the major parties. We compare the human positioning of the texts to those of the “unsupervised” unidimensional Poisson scaling model of Slapin and Proksch (2008). We also compare human perceptions of difference to statistical conclusions reached by different approaches to computing statistical confidence intervals from the text scaling model, including non-parametric bootstrapping of the texts. Our results confirm the basic validity of the statistical estimates, and suggest that the most appropriate form of measuring error is non-parametric bootstrapping of the textual data rather than using confidence intervals that depend on unrealistic parametric assumptions of the model.
Kenneth Benoit, Drew Conway, Michael Laver, and Slava Mikhaylov. 2012. “Crowd-sourced data coding for the social sciences: massive non-expert human coding of political texts.“ Paper prepared for presentation at the 70th Annual Conference of the Midwest Political Science Association. Palmer House Hotel, Chicago, 12-15 April 2012.
A large part of empirical social science relies heavily on data that are not observed in the field, but are generated by researchers sitting at their desks. Clearly, third party users of such coded data must satisfy themselves in relation to both reliability and validity. This paper discusses some of these matters for a widely used type of coded data, derived from content analysis of political texts. Comparing multiple “expert” and crowd-sourced codings of the same texts, as well as with independent estimates of the same latent quantities, we assess the extent to which we can estimate these quantities reliably using the cheap and scalable method of crowd sourcing. Our results show that, contrary to naive preconceptions and reflecting concerns that are often swept under the carpet, a set of expert coders is also a crowd. We find that deploying a crowd of non-expert coders on the same texts raises issues relating to coding quality that need careful consideration. If these issues can be resolved by careful specification and design, crowdsourcing offers the prospect of cheap, scalable and replicable text coding. While these results concern text coding, we see no reason why they do not extend to other forms of expert coded data in the social sciences.
William Lowe and Kenneth Benoit. 2011. “Estimating Uncertainty in Quantitative Text Analysis“. Paper prepared for the 2011 Midwest Political Science Association. Version: 30 March 2011.
Several methods have now become popular in political science for scaling latent traits— usually left-right policy positions—from political texts. Following a great deal of de- velopment, application, and replication, we now have a fairly good understanding of the estimates produced by scaling models such as “Wordscores”, “Wordfish”, and other variants (i.e. Monroe and Maeda’s two-dimensional estimates). Less well understood, however, are the appropriate methods for estimating uncertainty around these esti- mates, which are based on untested assumptions about the stochastic processes that generate text. In this paper we address this gap in our understanding on three fronts. First, we lay out the model assumptions of scaling models and how to generate un- certainty estimates that would be appropriate if all assumptions are correct. Second, we examine a set of real texts to see where and to what extent these assumptions fail. Finally, we introduce a sequence of bootstrap methods to deal with assumption failure and demonstrate their application using a series of simulated and real political texts.