Google Scholar Page here
Kenneth Benoit and Paul Nulty. April 8, 2012. “Classification Methods for Scaling Latent Political Traits.” Paper prepared for presentation at the Annual Meeting of the Midwest Political Science Association, April 11–14, 2013, Chicago. Version 2.
Quantitative methods for scaling latent political traits have much in common with supervised machine learning methods commonly applied to tasks such as email spam detection and product recommender systems. Despite commonalities, however, the research goals and philosophical underpinnings are quite different: machine learning is usually concerned with predicting a knowable or known class, most often with a practical application in mind. Estimating political traits through text, by contrast, involves measuring latent quantities that are inherently unobservable through direct means, and where human “verification” is unreliable, prohibitively costly, or otherwise unavailable. In this paper we show that not only can the Naive Bayes classifier, one of the most widely used machine learning classification methods, can be successfully adapted to measuring latent traits, and also that it is equivalent in general form to \cite{lbg:2003}’s “Wordscores” algorithm for measuring policy positions. We revisit several prominent applications of Wordscores reformulated as Naive Bayes, demonstrating the equivalence but also revealing areas where the original Wordscores algorithm can be substantially improved using standard techniques from machine learning. From this we issue some concrete recommendations for future applications of supervised machine learning to scale latent political traits.
Daniel Schwarz, Denise Traber, and Kenneth Benoit. April 5, 2012. “Estimating the Policy Preferences of Legislators in Parliamentary Systems: Comparing Speeches to Votes.“ Paper prepared for presentation at the Annual Meeting of the Midwest Political Science Association, April 11–14, 2013, Chicago.
Well-established methods exist for measuring party positions, but reliable means for esti- mating intra-party preferences remain underdeveloped. Most efforts focus on estimating the ideal points of individual legislators based on inductive scaling of roll call votes. Yet in most parliaments, roll call data suffer from two problems: selection bias due to unrecorded votes, and strong party discipline which tends to make votes strategic rather than sincere in- dications of preference. In contrast, legislative speeches are relatively unconstrained, since party leaders are less likely to punish MPs for speaking sincerely as long as they vote with the party line. This conventional wisdom remains essentially untested, despite the grow- ing application of statistical analysis of textual data to measure policy preferences. Our paper addresses this lacuna by exploiting a rich feature of the Swiss legislature: On most bills, legislators both vote and speak many times. Using this data, we compare text-based scaling of ideal points to vote-based scaling from a crucial piece of energy legislation. Our findings confirm that roll call votes underestimate intra-party differences, and vindicate the use of text scaling to measure legislator ideal points. Using regression models we further explain the difference between roll-call and text scalings with energy policy preferences at constituency level.
Michael Laver and Kenneth Benoit. March 7, 2012. “The Basic Arithmetic of Legislative Decisions.” Paper prepared for the Conference in Honor of Norman Schofield, Washington University in St Louis, 26-27 April 2013.
Despite the huge number of possible seat distributions following a general election in a multi- party parliamentary democracy, there are far fewer equivalence classes of seat distribution sharing important strategic features. We define an exclusive and exhaustive partition of the universe of theoretically-possible n-party systems into five basic types, the understanding of which facilitates more fruitful modeling of legislative politics, including government formation. A common type of legislative party system has a “strongly-dominant” party in the privileged position of being able to play off the other parties against each other. Another is a “top-three” party system in which the three largest parties are perfect substitutes for each other in the set of winning coalitions, but no other party is ever pivotal. Having defined a partition of legislative party systems and elaborated logical implications of this partition, we classify a large set of postwar European legislatures. We show empirically that many of these are close to critical boundary conditions, so that the stochastic processes involved in any legislative election could easily flip the resulting legislature from one type to another. This is of more than hypothetical interest, since we also show that important political outcomes differ systematically between the basic party system types – outcomes that include the duration of government formation negotiations, the type of coalition cabinet that forms, and the stability of the resulting government.
Kenneth Benoit, Drew Conway, Michael Laver, and Slava Mikhaylov. 2012. “Crowd-sourced data coding for the social sciences: massive non-expert human coding of political texts.” Paper prepared for presentation at the 3rd annual “New Directions in Analyzing Text as Data conference:, Harvard University, October 5-6, 2012.
A large part of empirical social science relies heavily on data that are not observed in the field, but are generated by researchers sitting at their desks, raising obvious issues of both reliability and validity. This paper addresses these issues for a widely used type of coded data, derived from the content analysis of political text. Comparing estimates derived from multiple “expert” and crowd-sourced codings of the same texts, as well as other independent estimates of the same latent quantities, we investigate whether we can analyze political text in a reliable and valid way using the cheap and scalable method of crowd sourcing. Our results show that, contrary to naive preconceptions and reflecting concerns often swept under the carpet, a set of expert coders is also a crowd. We find that deploying a crowd of non-expert coders on the same texts, with careful specification and design to address issues of coder quality, offers the prospect of cheap, scalable and replicable human text coding. Even as computational text analysis becomes more effective, human coding will always be needed, both to validate and interpret computational results and to calibrate supervised methods. While our specific findings here concern text coding, they have implications for all expert coded data in the social sciences.
Older papers:
William Lowe and Kenneth Benoit. 2011. “Estimating Uncertainty in Quantitative Text Analysis“. Paper prepared for the 2011 Midwest Political Science Association. Version: 30 March 2011.
Several methods have now become popular in political science for scaling latent traits— usually left-right policy positions—from political texts. Following a great deal of de- velopment, application, and replication, we now have a fairly good understanding of the estimates produced by scaling models such as “Wordscores”, “Wordfish”, and other variants (i.e. Monroe and Maeda’s two-dimensional estimates). Less well understood, however, are the appropriate methods for estimating uncertainty around these esti- mates, which are based on untested assumptions about the stochastic processes that generate text. In this paper we address this gap in our understanding on three fronts. First, we lay out the model assumptions of scaling models and how to generate un- certainty estimates that would be appropriate if all assumptions are correct. Second, we examine a set of real texts to see where and to what extent these assumptions fail. Finally, we introduce a sequence of bootstrap methods to deal with assumption failure and demonstrate their application using a series of simulated and real political texts.
