Current Research

Google Scholar Page here

Thomas Däubler and Kenneth Benoit. June 18, 2013. “The Empirical Determinants of Manifesto Content.” Paper prepared for presentation at the 3rd Annual General Conference of the European Political Science Association, 20-22 June 2013, Barcelona.

Party manifestos form the largest source of textual data for estimating party policy posi- tions, typically based on methods that assume that longer manifestos with more units of text provide more confident estimates. Despite using them extensively for nearly three decades, however, we know little to nothing about what explains why either the overall length of manifestos or their scope of issue coverage varies so highly across parties, elections, and contexts. Here, we critically test the notion that political context affects overall length and manifesto content. We use multi-level modeling to predict manifesto length and issue scope in a large number of coded party manifestos covering the post-war period. Our findings indicate that manifesto length and the scope of issue coverage can be largely explained by a combination of political variables related to party size, policy orientation, as well as election-specific factors related to political competition and the timing of elections.

Alexander Herzog and Kenneth Benoit. June 18, 2013. “The Most Unkindest Cuts: Government Cohesion and Economic Crisis.” Paper prepared for presentation at the 3rd Annual General Conference of the European Political Science Association, 20-22 June 2013, Barcelona.

Economic crisis and the resulting need for austerity budgets has divided many governing parties in Europe, despite the strict party discipline exercised over the legislative votes to approve these harsh budgets. Our analysis attempts to measure divisions in governing coalitions by applying automated text analysis methods to scale the positions that MPs express in budget debates. Our test case is Ireland, a country that has experienced both periods of rapid economic growth as well as one deep financial and economic crisis. Our analysis includes all annual budget debates during the time period from 1983 to 2013. We demonstrate that government cohesion as expressed through legislative speeches has significantly decreased as the economic crisis deepened, the result of government backbenchers expressing speaking against the painful austerity budgets introduced by their own governments. While ministers are bounded by the doctrine of collective cabinet responsibility and hence always vote for the finance min- isters’ budget proposal, we find that party backbenchers’ position-taking is systematically related to the economic vulnerability of their constituencies and to the safety of their electoral margins.

Kenneth Benoit and Paul Nulty. April 8, 2013. “Classification Methods for Scaling Latent Political Traits.” Paper prepared for presentation at the Annual Meeting of the Midwest Political Science Association, April 11–14, 2013, Chicago.

Quantitative methods for scaling latent political traits have much in common with supervised machine learning methods commonly applied to tasks such as email spam detection and product recommender systems. Despite commonalities, however, the research goals and philosophical underpinnings are quite different: machine learning is usually concerned with predicting a knowable or known class, most often with a practical application in mind. Estimating political traits through text, by contrast, involves measuring latent quantities that are inherently unobservable through direct means, and where human “verification” is unreliable, prohibitively costly, or otherwise unavailable. In this paper we show that not only can the Naive Bayes classifier, one of the most widely used machine learning classification methods, can be successfully adapted to measuring latent traits, and also that it is equivalent in general form to \cite{lbg:2003}’s “Wordscores” algorithm for measuring policy positions. We revisit several prominent applications of Wordscores reformulated as Naive Bayes, demonstrating the equivalence but also revealing areas where the original Wordscores algorithm can be substantially improved using standard techniques from machine learning. From this we issue some concrete recommendations for future applications of supervised machine learning to scale latent political traits.

Daniel Schwarz, Denise Traber, and Kenneth Benoit. April 5, 2013. “Estimating the Policy Preferences of Legislators in Parliamentary Systems: Comparing Speeches to Votes.“ Paper prepared for presentation at the Annual Meeting of the Midwest Political Science Association, April 11–14, 2013, Chicago.

Well-established methods exist for measuring party positions, but reliable means for esti- mating intra-party preferences remain underdeveloped. Most efforts focus on estimating the ideal points of individual legislators based on inductive scaling of roll call votes. Yet in most parliaments, roll call data suffer from two problems: selection bias due to unrecorded votes, and strong party discipline which tends to make votes strategic rather than sincere in- dications of preference. In contrast, legislative speeches are relatively unconstrained, since party leaders are less likely to punish MPs for speaking sincerely as long as they vote with the party line. This conventional wisdom remains essentially untested, despite the grow- ing application of statistical analysis of textual data to measure policy preferences. Our paper addresses this lacuna by exploiting a rich feature of the Swiss legislature: On most bills, legislators both vote and speak many times. Using this data, we compare text-based scaling of ideal points to vote-based scaling from a crucial piece of energy legislation. Our findings confirm that roll call votes underestimate intra-party differences, and vindicate the use of text scaling to measure legislator ideal points. Using regression models we further explain the difference between roll-call and text scalings with energy policy preferences at constituency level.

Kenneth Benoit, Drew Conway, Michael Laver, and Slava Mikhaylov. 2012. “Crowd-sourced data coding for the social sciences: massive non-expert human coding of political texts.” Paper prepared for presentation at the 3rd annual “New Directions in Analyzing Text as Data conference:, Harvard University, October 5-6, 2012.

 A large part of empirical social science relies heavily on data that are not observed in the field, but are generated by researchers sitting at their desks, raising obvious issues of both reliability and validity. This paper addresses these issues for a widely used type of coded data, derived from the content analysis of political text. Comparing estimates derived from multiple “expert” and crowd-sourced codings of the same texts, as well as other independent estimates of the same latent quantities, we investigate whether we can analyze political text in a reliable and valid way using the cheap and scalable method of crowd sourcing. Our results show that, contrary to naive preconceptions and reflecting concerns often swept under the carpet, a set of expert coders is also a crowd. We find that deploying a crowd of non-expert coders on the same texts, with careful specification and design to address issues of coder quality, offers the prospect of cheap, scalable and replicable human text coding. Even as computational text analysis becomes more effective, human coding will always be needed, both to validate and interpret computational results and to calibrate supervised methods. While our specific findings here concern text coding, they have implications for all expert coded data in the social sciences.

Older papers:

William Lowe and Kenneth Benoit. 2011. “Estimating Uncertainty in Quantitative Text Analysis“. Paper prepared for the 2011 Midwest Political Science Association. Version: 30 March 2011.

Several methods have now become popular in political science for scaling latent traits— usually left-right policy positions—from political texts. Following a great deal of de- velopment, application, and replication, we now have a fairly good understanding of the estimates produced by scaling models such as “Wordscores”, “Wordfish”, and other variants (i.e. Monroe and Maeda’s two-dimensional estimates). Less well understood, however, are the appropriate methods for estimating uncertainty around these esti- mates, which are based on untested assumptions about the stochastic processes that generate text. In this paper we address this gap in our understanding on three fronts. First, we lay out the model assumptions of scaling models and how to generate un- certainty estimates that would be appropriate if all assumptions are correct. Second, we examine a set of real texts to see where and to what extent these assumptions fail. Finally, we introduce a sequence of bootstrap methods to deal with assumption failure and demonstrate their application using a series of simulated and real political texts.