Referee Bias Analysis

Overview

A personal statistical analysis using the Crowdstorming dataset (Silberzahn et al., 2018) to examine whether referees issue red cards at different rates depending on players’ skin tone.

The dataset is a well-known one in the methodology world: 29 independent analyst teams were given the same data and the same question, and came back with effect sizes ranging from 0.89 to 2.93 (odds ratio units), with roughly two-thirds finding a significant effect and the rest finding none. That variability is most of what makes it an interesting dataset to sit down with personally, rather than just a red-card-counting exercise.

The main goal was to refresh R skills after working mainly in Python for the last few years, and to get some hands-on practice with statistical modelling techniques I don’t reach for often, while poking at a real methodological question rather than a toy problem.

Data

Player–referee dyads (~146,000 of them) from the top divisions of England, Germany, France, and Spain, 2012–13 season. Each row records how many games a given player and referee crossed paths for, and what happened across those games: goals, yellow cards, yellow-red cards, straight red cards.

Skin tone was rated by two independent coders on a 5-point scale from player photos, available for 1585 of 2053 players. The two raters agreed closely (r = 0.92), and I averaged their scores into a single continuous measure.

A few data quality quirks turned up along the way, which felt worth documenting rather than quietly smoothing over:

  • The dataset’s README states 1586 players had usable photos; the data itself shows 1585. Never resolved, noted and moved on.
  • Missingness in position data turned out to be statistically related to skin tone (p < 10⁻¹⁵), but the actual size of the difference was tiny (about 0.03 on a 0–1 scale). This is a nice live example of statistical significance and practical importance pulling apart when the sample size is large.

Approach

Built three increasingly complex models, checking at each stage whether the skin tone effect held up as complexity, and realism, increased:

  1. Player-level negative binomial. Simplest possible starting point: aggregate to one row per player, model total red cards against skin tone, offset by total games played. Negative binomial rather than Poisson, since the data showed clear overdispersion (variance noticeably exceeding the mean).
  2. Crossed random-effects model. The naive model above ignores that both players and referees appear repeatedly in the data, and referees plausibly differ in card-happiness independent of who they’re refereeing. Moved to dyad-level data with a GLMM (lme4::glmer), using crossed random intercepts for player and referee, not nested, since any referee sees many players and vice versa.
  3. Adding position. Added a four-category position grouping (Goalkeeper / Defender / Midfielder / Attacker) as a fixed effect, since red cards are overwhelmingly foul-related and defenders are structurally more exposed to that risk than, say, forwards.

Results so far

The skin tone effect turned out to be remarkably stable across all three specifications, somewhere between a 37% and 44% higher red-card rate for the darkest-rated players compared to the lightest, and statistically significant throughout (p ≈ 0.001–0.002 in each model). All three estimates sit comfortably inside the original study’s 0.89–2.93 range.

What surprised me more than the headline number was the position result: I’d expected defenders to clearly outpace goalkeepers in red card rate, given how much more they’re involved in last-ditch tackling, but the difference wasn’t statistically significant, only attackers (lower than goalkeepers, as expected) reached conventional significance.

Limitations worth flagging

Several dataset features turned out to be less useful than they first look, which felt worth being upfront about:

  • League, club, and age (via birthday) are all recorded as of dataset creation, not as of each game. Since dyads span multiple seasons, a player’s league or club may well have changed between games within the same dyad and these fields can’t be reliably tied to the specific games being analysed, so I’ve left them out rather than use them misleadingly.
  • Relatedly, there’s no way to ask whether say younger players get more red cards for rash play, since age can’t be reliably reconstructed at the time of each game.

Tools

R, glm.nb (MASS), glmer (lme4), tidyverse for wrangling.

Status

Write-up in progress. Next step: extending the analysis to yellow cards as a parallel outcome, to see whether any skin tone effect shows up for minor infractions too, or looks specific to red cards.

A fuller, more formal write-up (with full model tables, diagnostics, and proper citations!) is in progress in LaTeX and a PDF will be linked here once it’s in a reasonable state.