Research paper guidelines

Some Questions for Any Research Paper (by William Phelan)

  1. What is the variation being explained?
    1. Conceptually, what is the variation in the dependent variable (Y)?
    2. How is the concept measured in practice?
    3. Factually, what is the variation in the dependent variable? Is there any variation in the dependent variable?
    4. Is that variation in these cases just a small part of a possible wider variation?
    5. Is it clear what the factual support underlying the placing of different cases in different outcomes is?
    6. Do you have a clear idea what the variation actually looks like across the cases? Is it robust against claims that the difference between cases is trivial or unimportant?
    7. Are you conceiving of and/or measuring the dependent variable in the same way as other scholars, or differently? If differently, is the use of a different concept or measurement justified?
  2. What are the alternative explanations used in this paper?
    1. What is the “key causal variable” in this paper?
    2. How many alternative explanations are considered in this paper?
    3. Are there obvious alternative explanations unconsidered in this paper? Are other explanations in the previous scholarship included?
    4. Are any of the alternative explanations “strawman” alternatives? How would they be better or more fairly stated or considered?
    5. Can the argument of this paper be written in the form of a “rough equation”?
  3. How do we know that X causes Y?
    1. Does X covary with Y, holding all other effects on Y constant?
    2. Do you have a causal mechanism? Would that causal mechanism apply to other cases in politics? To similar circumstances outside politics?
    3. Is there the possibility of endogeneity i.e. that Y also causes variation in X?
  4. Falsification and Hypothesis Testing?
    1. Does the paper deal explicitly with falsification?
    2. What is empirical evidence would lead to the rejection of the argument about the influence of the รข”key causal variable”?
    3. Have all the “observable implications” of the various alternative explanations been tested? There are often more observable implications than the author has included.
  5. Case Studies and Case Selection:
    1. Does the author state explicitly why these cases have been chosen?
    2. Have cases been chosen to rule out the influence of possible alternative explanations? (One possibility here is a natural experiment?)
    3. How are do the cases considered here relate to a wider universe of possible cases? A random sample? The whole universe? Or neither?
    4. If cases are not randomly selected or the whole universe, then how does the nature of the cases selected affect the research agenda?
    5. Do the cases have extreme outcomes (i.e. deviant dependent variables), or extreme inputs (i.e. unusual independent variables)?
    6. Does the case study belong to one of the case study types discussed in Gerring 2007 Chapter 5? (Also e.g. Gerring 2006 on the single outcome study)
  6. Operationalisation of Concepts:
    1. How does the paper move from concept to measurements?
    2. If provided only with those measurements, would they convincingly indicate the concept?
    3. Are necessary conditions in concepts reflected in necessary conditions in measurements?
    4. Does the answer change if you use alternative measures of this concept? with all frequently used measurements of that concept?
  7. Theory in Political Science
    1. Does the paper make an explicit claim the approach it takes to incentives for actor behaviour?
    2. Does the paper make a new claim about incentives for actor behaviour?
    3. Does the paper properly understand the incentives drawn from economic literature about rational actors?
  8. What methods have been used to assure unbiased use of sources, including narrative sources?
    1. Where archival research has been done, the author responsible for everything in the archive as a whole, including actively looking for material unhelpful to their argument. (If less is done than that, the scope of research undertaken should be stated.)
    2. Are selection issues related to narrative or secondary sources addressed?
    3. Would it be straightforward to another researcher to check and replicate these findings? Is the research “public”? (KKV p8)
      1. Are the footnotes accurate?
      2. Are the databases/ archives/ interviewees available to other researchers?
      3. Is the statistical programming (etc) used clear, public, and replicable?

EXTRAS

  1. Is the paper overly credulous? Overly cautious on “controversial” topics? Contain unexamined normative assumptions?
  2. Why does the paper have that cute but vague or confusing title? What would the straightforward, scientific title be?
  3. Does the paper admit its limitations?
    1. Theoretical limitations
    2. Empirical findings which do not easily fit with the argument? (Yes, there are some.)
  4. Can we accept the paper’s particular findings for this particular case without accepting the overall conclusions for other cases?
  5. What is new in this paper?
    1. New in relation to international milk quotas (eg.)?
    2. New in relation to international trade politics?
    3. New in relation to international relations scholarship as a whole?
    4. New in relation to social science as a whole?