AL-90 · Selected Studies — Sport Management

Umpire Accuracy Lab

MLB umpires called 34,294 incorrect ball-strike calls during the 2018 season — an average of 14 per game, 1.6 per inning. Worse: the error rate rises significantly with two-strike counts, where umpires are biased toward calling false strikes. This lab explores those findings (inspired by the BU Questrom research program on MLB umpire accuracy) in two modes: a measurement lesson on error rates and bias detection, and a structured debate on automated ball-strike technology.

📊 Data: Inspired by: BU Questrom umpire accuracy research program (Mark T. Williams + graduate students); 11-season dataset analysis, ~4 million pitches. See also: bu.edu/articles/2019/mlb-umpires-strike-zone-accuracy/

Section A — Scale of the Problem

Incorrect calls (2018)

34,294

Ball-strike calls, full season

Per Game

~14

Incorrect calls per game

Per Inning

~1.6

Incorrect calls per inning

Games ended wrong (2018)

Final call of game was incorrect

11-season dataset

~4M

Pitches analyzed (inspired by BU)

Games ended wrong (11 seasons)

Final call was incorrect

Umpire accuracy rate (est.)

~85%

Borderline calls only (est.)

ABS accuracy rate (est.)

>99%

Hawk-Eye / TrackMan systems

Note on these figures: The 34,294 and associated per-game/per-inning figures are inspired by and consistent with findings from the BU Questrom research program on umpire accuracy. Exact values may vary by study parameters and season analyzed. See bu.edu/articles/2019/mlb-umpires-strike-zone-accuracy/ for published context.

Section B — Two-Strike Count Bias

The Pattern: With a 2-strike count (0-2, 1-2, 2-2, 3-2), umpires are psychologically predisposed to "protect the hitter" — they become less likely to call strike 3 on borderline pitches. This results in elevated false-ball rates on pitches that are actually strikes when the batter is one strike away from striking out.

This is consistent with a broad body of cognitive psychology research on loss aversion under pressure: calling strike 3 on a questionable pitch feels like a decisive, consequential act. The umpire hesitates. The borderline pitch becomes a ball. The pitcher who deserved a strikeout loses it.

Illustrative Error Rates by Count Type

Illustrative rates based on direction and magnitude of BU research findings, not exact study percentages.

No-strike counts (lower error rate)

0-0

~4%

1-0

~4%

2-0

~4.5%

3-0

~4.4%

One-strike counts (moderate error rate)

0-1

~5.5%

1-1

~5.8%

2-1

~5.7%

3-1

~5.5%

Two-strike counts (elevated error rate — two-strike bias)

0-2

~8%

1-2

~8.5%

2-2

~9%

3-2

~9.5%

Section C — Performance Evaluation Framework for Umpires

MLB umpires are one of the most measurable professional workers in the world. Every decision they make is logged, timestamped, and cross-referenceable against a ground-truth measurement system. Most workers — surgeons, teachers, lawyers — face nothing comparable.

ACCURACY RATE

Correct calls ÷ Total calls. Overall accuracy across all umpires typically 85–90% (borderline pitches only). Clear balls and obvious strikes are nearly always called correctly. The ~10–15% error rate applies specifically to pitches near the strike zone boundary.

BIAS COEFFICIENT

Asymmetry between missed balls (calling a ball a strike) and missed strikes (calling a strike a ball). Two-strike count analysis reveals a systematic bias toward missed strikes — umpires are more likely to err toward keeping batters alive. This is not random noise; it is directional, predictable error.

COUNT-SPECIFIC ACCURACY

Performance degrades under specific high-pressure conditions — a critical performance evaluation insight for any worker. A surgeon who performs 95% of surgeries well but degrades under time pressure or specific patient profiles has a very different risk profile than one who performs uniformly. Umpire accuracy falling in 2-strike counts is a conditional performance problem.

IMPLICATIONS FOR PERFORMANCE MANAGEMENT

If 99% of an umpire's calls are correct on obvious pitches but they struggle on borderline 2-strike counts, a performance improvement intervention should target specifically that condition — not overall retraining. The data enables targeted interventions impossible without granular performance logging.

Discussion Questions

If you were an MLB team, how would you use umpire accuracy data? (e.g., game preparation, at-bat strategy, pitcher selection for high-stakes counts)
Is an accuracy rate of ~85% acceptable for a professional whose errors directly affect game outcomes and potentially team playoff qualification?
What does "two-strike count bias" suggest about cognitive load and decision-making under pressure? What other professions show similar conditional performance degradation?
If umpire error is not random but systematic, who bears the cost? Are pitchers harmed more than batters, or vice versa?
Should individual umpire accuracy data be publicly disclosed? What are the arguments for and against transparency?

Automated Ball-Strike (ABS) Technology — Structured Debate

ABS uses camera-tracking systems (Hawk-Eye, TrackMan) to call balls and strikes automatically. It is currently used in Triple-A and independent leagues. Select a position to view a full argument set.

Arguments FOR ABS (Robot Umpires)

1Accuracy: Technology is simply more accurate than humans▶

ABS tracking systems (Hawk-Eye, TrackMan) call balls and strikes with greater than 99.5% accuracy compared to approximately 85–90% for human umpires on borderline pitches. When the purpose of the rule is to distinguish balls from strikes, a system that does so 99.5% of the time is objectively superior to one that does so 87% of the time. Improvement in call accuracy is not an aesthetic preference; it is a direct measure of how well the officiating system fulfills its stated purpose.

Evidence Requirements

Strengthens: Peer-reviewed studies confirming >99% ABS accuracy across diverse pitch types; Triple-A implementation data showing reduced controversy; player satisfaction surveys post-ABS implementation.

Weakens: Evidence that ABS misses specific pitch types (cutters on corner, knuckleballs); systematic calibration errors; player complaints about ABS accuracy in practice.

2Consistency: Human accuracy degrades; ABS is perfectly consistent▶

Human umpires are affected by fatigue, heat, crowd noise, game pressure, and count-specific cognitive biases. Research shows accuracy declines later in games, in high-pressure counts, and potentially in high-temperature conditions. An ABS system calls the same pitch the same way in the 1st inning and the 9th inning, in 45°F and 95°F, with the bases empty and the World Series on the line. Consistency is itself a form of fairness — every team and player is evaluated on the same standard.

Evidence Requirements

Strengthens: Studies documenting umpire accuracy decline by inning or game; temperature/fatigue correlation data; multi-game performance variance across umpires.

Weakens: Evidence that ABS systems also degrade (calibration drift, sensor wear); data showing human umpire consistency within acceptable tolerance ranges.

3Bias elimination: No two-strike bias, no favoritism for star pitchers▶

Research has documented that elite pitchers (Jacob deGrom, Max Scherzer) consistently receive more favorable borderline calls than average pitchers — a reputation effect. Two-strike count bias systematically disadvantages pitchers on borderline third strikes. ABS eliminates both: every pitcher gets the same zone, every count is treated identically, every batter faces the same standard regardless of career accomplishments. In a sport where a single game can determine playoff seeding, systematic bias has material consequences for outcomes.

Evidence Requirements

Strengthens: Studies documenting reputation effects on call rates by pitcher ERA tier; statistical demonstration that two-strike call rates differ from zero-strike rates on matched pitches.

Weakens: Evidence that observed "favoritism" is actually explained by elite pitchers locating pitches in the zone more precisely; studies showing no statistically significant reputation effect.

4Player trust: Umpire inconsistency distorts pitcher and batter behavior▶

Pitchers who know umpire X has a high outside corner will pitch differently than against umpire Y who squeezes the zone. Batters adjust their swing decisions based on which umpire is working. This means a significant portion of player strategic decision-making is spent adapting to umpire idiosyncrasies rather than responding to actual pitch quality. ABS would eliminate this noise, focusing the competition entirely on pitcher-batter interaction. The pitcher who throws the best pitch wins, not the pitcher who best games the umpire's tendencies.

Evidence Requirements

Strengthens: Analysis of catcher framing value (catchers paid premium for presenting pitches well implies zone inconsistency exists); evidence that pitcher pitch-mix changes by umpire assignment.

Weakens: Evidence that adaptation to umpire tendencies is a legitimate skill and source of competitive differentiation; argument that "gaming the umpire" is itself a real-world applicable skill.

5Precedent: ABS is proven at Triple-A and independent leagues▶

ABS has been in use in Triple-A (AAA) baseball since 2023, and in independent leagues for several years prior. Data from those implementations shows the technology operates reliably in live game conditions, players adapt without significant disruption, and game pace is maintained. The transition from "future technology" to "proven in the minor leagues" eliminates the empirical uncertainty argument against ABS. The question is no longer whether ABS can work — it demonstrably does — but whether MLB is willing to make the institutional commitment to implement it.

Evidence Requirements

Strengthens: Triple-A implementation data on controversy rate, player complaint rate, technical failure incidents; player and manager surveys on experience with ABS.

Weakens: Documentation of specific ABS failures in minor league games; evidence that minor league conditions (fewer games, lower stakes) don't transfer to 162-game MLB playoff pressure.

Arguments AGAINST ABS

1Zone definition: No two hitters have the same dimensions — a rigid ABS zone is unfair▶

The official strike zone is defined as the 17-inch width of home plate by the space between "the top of the knees and the midpoint of the torso" — a player-specific definition. Aaron Judge stands 6'7". José Altuve stands 5'6". An ABS system either applies identical absolute coordinates to all batters (clearly wrong) or must compute individualized zones for each batter (creating new implementation complexity and potential disputes). Human umpires naturally adjust their zone to the batter's stance; an ABS system must be explicitly programmed to do so, and those programming choices become their own source of controversy.

Evidence Requirements

Strengthens: Data showing that ABS systems in Triple-A apply different zones by player height; documentation of specific player complaints about ABS zone calibration; examples of pitches that ABS called differently from the rulebook intent.

Weakens: Evidence that current ABS implementations successfully individualize zones; data showing umpires themselves do not consistently adjust for batter height.

2The batter's eye: Working corners is part of baseball's texture▶

Part of baseball's strategic depth is the cat-and-mouse between pitchers working the margins of the strike zone and batters (and catchers) learning which umpires have generous strike zones, which squeeze the outside corner, which favor fastballs over breaking balls. This umpire-specific strategic meta-game is a legitimate part of competition. A pitcher who studies an umpire's tendencies and exploits them is exhibiting a skill that translates broadly to reading and adapting to decision-makers under pressure — a valuable cognitive skill ABS would make irrelevant. What's lost is not merely "tradition" but a genuine competitive dimension.

Evidence Requirements

Strengthens: Evidence that catcher framing (worth $15M+ in contract value) requires umpire inconsistency to create value; analysis showing that pitcher performance varies significantly by umpire assignment in ways consistent with strategic exploitation.

Weakens: Evidence that "reading the umpire" is actually primarily about exploiting systematic error (bias) rather than skill-based zone management; argument that zone inconsistency hurts batters disproportionately.

3Technical failure risk: A playoff-game ABS failure creates a legitimacy crisis▶

Camera calibration degrades over time. Sensors can malfunction. Software can have edge-case failures. A human umpire who makes an error can be reprimanded, reassigned, or overruled by replay; an ABS system failure during a playoff game with no human override creates a different and potentially larger legitimacy crisis. The question is not whether ABS is usually reliable, but whether its failure modes are more or less disruptive than human error. A systematic technology failure — affecting all calls for an inning or game — has no human analog. The risk profile is different, not necessarily lower.

Evidence Requirements

Strengthens: Documentation of ABS system failures in Triple-A games; analysis of how MLB protocols would handle mid-game ABS failure; evidence that current backup protocols are insufficient.

Weakens: Evidence that ABS systems have redundant backups (multiple cameras, independent tracking systems); data showing zero systematic failures in multi-season Triple-A implementation.

4Craft loss: Pitch-framing becomes worthless — destroying a legitimate skill▶

Pitch-framing — the ability of a catcher to receive a pitch in a manner that influences the umpire toward calling it a strike — is a quantifiable skill worth tens of millions of dollars in market value. José Trevino, Yasmani Grandal, and Jonah Heim have all received significant contracts partly because of their elite framing metrics. ABS renders this skill entirely worthless. Beyond the immediate financial consequences for players who built careers on framing, it eliminates a non-obvious craft skill that adds strategic richness to the catcher-pitcher-umpire interaction. Whether framing skill is "exploiting bias" or "genuine craft" is itself a legitimate debate — but eliminating it removes a dimension of the game.

Evidence Requirements

Strengthens: Contract data showing framing value priced into catcher salaries (statcast framing runs above average); evidence that elite framers consistently gain 20-30 extra strikes per year.

Weakens: Argument that framing is entirely parasitic on umpire error (a skill at exploiting a defect, not contributing to fair competition); evidence that elimination of framing value would be offset by catchers developing other skills.

5Tradition: Baseball's relationship with human error is part of its mythology▶

Baseball's cultural history is inseparable from memorable umpiring moments — some controversial, some beloved. Armando Galarraga's near-perfect game destroyed by Jim Joyce's incorrect safe call in 2010 generated more sustained human interest than most actual perfect games. The pine tar game, Don Denkinger's 1985 call, Jeffrey Maier's home run — these moments became cultural reference points precisely because they involved human imperfection in high-stakes situations. A fully automated game removes the possibility of these kinds of stories. Whether this is a loss depends on what you believe baseball is for: perfect officiating of rules, or human drama in a structured competitive setting. That is a values question, not a statistical one.

Evidence Requirements

Strengthens: Survey data on fan preference for human umpires vs. ABS; audience engagement metrics showing controversial calls increase post-game viewership; historical analysis of how officiating controversy has generated long-term fan interest.

Weakens: Fan surveys showing strong preference for ABS accuracy over human drama; evidence that incorrect calls generate negative rather than positive fan engagement; argument that tradition is a poor basis for institutional design in a professional sport.

Error Analysis Framework — For Any Performance Evaluation System

The umpire accuracy question is a special case of a general problem: how do you evaluate whether a decision-making system (human or automated) is performing adequately? Five questions structure any rigorous analysis.

What is the reference standard?

An "error" is only definable if there is a ground truth to compare against. For umpires: does the pitch cross the geometrically defined strike zone? Hawk-Eye and TrackMan provide that ground truth. Without a reference standard, you cannot measure accuracy — you can only measure consistency. Critical implication: Many real-world performance evaluations lack a genuine reference standard (what is the "correct" legal ruling? the "correct" diagnosis for a complex case?). Umpiring is unusually tractable because the ground truth is physical and measurable.

What is the capture rate?

Are all decisions logged? For umpires: yes — every pitch is tracked by MLB's Statcast system. This is a nearly unique situation in professional performance evaluation. Most workers' decisions are not logged at a 99%+ capture rate. Teachers' moment-to-moment instructional decisions are not captured. Surgeons' judgment calls in the operating room are not captured. The high capture rate is what makes umpire performance evaluation possible at this level of precision. Key question: What is the capture rate for the decisions you care about evaluating?

Are errors random or systematic?

Random noise is unavoidable and theoretically unbiased — it doesn't favor any party over time. Systematic error (bias) is directional, persistent, and correctable. The two-strike count bias in umpire calling is systematic: it favors batters over pitchers in high-leverage counts, predictably and consistently. A random error rate of 10% is fundamentally different from a systematic bias rate of 10%. Why this matters: Systematic errors can be corrected through training, incentives, or technology. Random errors can only be reduced, not eliminated. Knowing which you're dealing with determines your intervention strategy.

What are the consequence asymmetries?

Is a false strike (calling a ball a strike) worse than a false ball (calling a strike a ball)? In different contexts: with a batter ahead 3-0, a missed strike matters more than a missed ball. In a 0-2 count, a missed third strike costs a pitcher an out. ABS debates partly rest on this: if errors were symmetric, the efficiency argument for ABS would be weaker. The two-strike bias suggests umpire errors are asymmetric — they disproportionately help batters at the margin that matters most. General principle: All error analysis should quantify consequence asymmetries, not just error rates.

What is the comparison benchmark?

How good could this process be? The technology limit for ABS is known: ~99.5% accuracy. The human limit is less certain, but consistent evidence suggests 90–92% on borderline pitches is near the ceiling for a human umpire working at game speed. Without a benchmark, "85% accuracy" sounds like a number without context. With ABS as a comparison, it becomes clear that ~10 percentage points of accuracy are available through technology adoption — the question is whether that gain is worth the costs of transition. Ask always: What is the best this system could theoretically be? Are we near that limit, or far from it?

Apply This Framework to Your Industry

The five-question framework applies to any human decision-making system subject to error analysis. Consider these applications:

Medical Diagnosis

Reference standard: Biopsy, lab result, confirmed outcome. Capture rate: Varies — some decisions are logged, many are not. Systematic errors: Diagnosis bias by patient demographics (documented in multiple studies). Consequence asymmetry: False negative (missing a disease) typically worse than false positive. Benchmark: Computer-aided diagnosis (CAD) for mammography reads, AI for radiology — what is their measured accuracy vs. human radiologists?

Legal Decisions (Sentencing)

Reference standard: Hard to define — what is the "correct" sentence? Capture rate: Sentencing decisions are logged. Systematic errors: Documented disparities by defendant race, time of day (Israeli parole board hunger study). Consequence asymmetry: False imprisonment vs. wrongful release — deeply asymmetric consequences. Benchmark: Actuarial risk tools vs. judicial discretion — ongoing empirical debate.

Financial Audit

Reference standard: Actual financial statement accuracy (knowable after the fact, sometimes). Capture rate: Auditor decisions are documented in workpapers. Systematic errors: Auditors who develop long relationships with client management may develop confirmation bias. Consequence asymmetry: Missing a fraud (false negative) vs. flagging a non-issue (false positive) — different regulatory and reputational consequences. Benchmark: AI-assisted audit tools are now being evaluated against human auditor accuracy in large-sample studies.