The main goal of this project is to create and deploy an app that shows interactive results from a statistical model that detects players who are cheating on lichess. It is currently a work-in-progress and can be viewed on GitHub.

Introduction: the state of the art for cheat detection in online chess games are typically models that look at individual moves of a player's games, and examine features such as a player’s average centipawn loss, time usage on certain moves, and performance in critical positions. However, it is not feasible to run these models on every player’s games so it is valuable to have a simpler statistical model that flags players who are likely cheating in the first place.

Background on rating systems: One useful but simple feature that is easily extracted from millions of games are the players’ ratings and the result of the games. Based on the players’ ratings, we can calculate the expected result and compare it to the actual result, and obtain a performance difference (i.e. against a similarly rated opponent, your expected performance = 0.5, but you win the game so your actual performance = 1.0, giving you a performance difference = 0.5).

A player who decides to start cheating will play above their expected strength, and we can set a threshold to detect such players if they play too far above their expected strength for some minimum period of time. However, it’s worth noting that new players with a starting rating of 1500, or players returning from a period of inactivity (with a stale rating) are more likely to play significantly above or below their rating.

In the exploratory plot below, we can see that generally speaking, established players play very close to their expected strength, but new players display a lot more variability in performance

Background on Model Development: The idea for the model is straightforward: we set a threshold on improvement and too much improvement is flagged as suspicious. To determine these thresholds, we would ideally look at the proportion of confirmed cheaters above the improvement threshold as we move it downwards (e.g. a really high threshold might flag 1/1 cheater, whereas a lower threshold might flag 10/50 cheaters)

However, lichess only discloses whether an account is open, closed, or had a terms of service violation, so we have imperfect labels. Therefore, we estimate the number of cheaters as X = 0.00 * N_open + 0.75 * N_closed + 1.00 * N_violation, where N_open is the number of open accounts, N_closed is the number of closed accounts, and N_violation is the number of accounts that have violated the terms of service (so there are N accounts = N_open + N_closed + N_violation). The choice of 0.75 for closed accounts is arbitrary, and is based on personal experience, but this is an important hyperparameter that changes how the selective the model is in flagging cheaters.

So which metric do we optimize the thresholds around? Accuracy is possible, but will suffer from heavily imbalanced data in the tails. Consider a situation where at the very end of the tail, there’s a non-cheater has an average performance difference of 0.4, and a cheater has an average performance difference of 0.5, with no remaining players above them. A threshold of 0.41 will flag only the cheater, and give an accuracy of 1/1 = 100%, but when the threshold is lowered to 0.39, the accuracy becomes 1 / 2 = 50% and more importantly, regardless of the mix of players and cheaters below this threshold, we will never again reach 100% accuracy, causing us to set the threshold too high.

Therefore, a metric that rewards more data is sensible, so we construct 95% confidence intervals around the accuracy, using Wilson’s confidence interval, and pick the maximum lower bound across possible thresholds. Wilson’s confidence interval will have a notably wide interval when values of n are small, so this counterbalances high accuracy rates at high thresholds with very few players.

The below dashboard shows how the thresholds are selected for each time control, rating range, and activity level:

Model Outputs: So what types of players will this model flag? Since we’re only looking at rating change behavior, we are inevitably going to flag some false positives, but this is expected – this pipeline is intended to be more of a broad sweep for atypical rating behavior than definitive cheat detection for individual players. Looking at all of the players above each threshold, we see a common pattern of rapid improvement, and a decent number of accounts that were closed or had violations. I believe most of the false positives are instances of relatively new accounts rapidly gaining rating (and without looking at individual games, it doesn’t seem trivial to separate cheating from legitimately gaining rating).

However, one area for improvement might be to analyze rating gain behavior among established players, and look for a sudden gain in rating after a plateau, which would indicate an [apparent] gain in strength not explainable by typical improvement.