Surviving Statistical Spitting Matches

What is a good legislative staffer to do?

• Legislation by anecdote
• The human tendency to find patterns
• News versus reality
  Typical New York jury verdicts
  Extreme New York jury verdicts

Crucial systematic information can be provided by statistics

Lies, damn lies, and statistics
Statistical spitting matches
Descriptive vs. causal information

The spitballs

Biased sample spitball
Biased researcher spitball
Biased question spitball
Misspecification spitball
Spurious relationship spitball
Wrong question spitball

Consuming statistics: Things to think about to avoid being consumed by statistics or snowed by statistical analysts

Statistics provide evidence they seldom give definitive answers
Statistics can and do give the wrong answer

• Type I Error
• seeing something that isn't really there
• "rejecting a null hypothesis that is in fact true"
• Type II Error
• failing to see something that is actually there
• "failing to reject a null hypothesis that is in fact false"
• Type II-1/2 error
• assuming that if you don't see something, the reverse is true
• "fallacy of affirming the consequence"
• "no statistically discernible effect" versus "no effect"
D&D: Design and Data

• If the design is wrong, or the data are wrong, no one can answer the question
• Type III Error: Asking the wrong question

Sorting it out: Asking the right questions

Has the analyst measured the phenomenon or the perception of the phenomenon?

• Perceptions vs. reality
  Perceived bias in court by group
Does the analyst know what he is talking about?
So what: Statistical vs. policy significance?

• How big is big enough?
  Bias in court
• How hard does a researcher need to look to find a difference?
• How close is close enough?
• Can you tell me what the difference means in a way that I can explain it to Joe Voter?
Where did the data come from? Were the data designed to get a specific answer?

• How were the questions asked?
• Who designed the data collection?
• Asking questions to get specific answers vs. asking questions to get specific information
• Can the other side tell you what is wrong with the data?
Are the data adequate to the task?

• The problem of specialized populations
  Bolts (not needles) in the haystack
  Where is race?
• Statistical power
What did you find that did not support your argument?

• Tell me the probability of your being wrong?
• What is the chance that what you have seen is just a product of random sampling or random processes?
• The Two Hat Random Process Model
• Give me a "confidence interval" or an "interval estimate"
What happened when you looked at this for various subgroups?

• Individual comparisons vs. patterns
• Gender differences in evaluations of court experience
Compared to what? Compared to when? Comparison base

• Short vs. long time frames
  New Jersey disciplinary cases
• Gee Whiz
• Adding the baseline
• Taking the long view
• One year doth not a trend make
What is the right comparison?

• What is the question I am trying to answer?
• Do these statistics speak to that question or another queation?
  Means vs. medians in jury awards
Are these broad assertions or specific findings?

• How much of this is results vs. interpretations?
  Routine approval of adult guardianships
• Are courts just rubber stamps?
• Is the guardianship program being used as intended?
How has the analyst's professional orientation affected his or her interpretations?

• Lawyers: something is wrong because this is supposed to be adversarial
• Social scientist: not invoked until not an issue
What has been lumped together here?

• Aggregate trends vs. specific trends
• Analysis: breaking things apart
• Civil case load growth in federal courts
  Government action
• Representation in social security appeals
  Type of cases
• Changes in the base over time
  Political science grads and IR Grads
Spurious effects: What else might account for the pattern?

• What else has changed?
  Skyrocketing jury verdicts in California
  Did tort reform really work?
• Is there a ceiling effect here?
  Growth in abortions
The specification argument: What is left out?

• Bias in sentencing
• Going to jail and for how long
• Race of the victim effects
• Does it matter?

Getting help

• Statistical knowledge vs. substantive knowledge
• The neutral expert
• Is there such a thing?
• Why is the analyst on the side s/he is on?
• Why is the analyst saying what s/he is saying?
• Did s/he set out to get a particular answer because of predispositions?
• Was s/he brought in because his/her research had led him/her in a particular direction?