Introduction

There was a heated discussion recently, and I decided to quantify the rating deflation over the recent 1.5 years.

After accounting for natural improvement, I found roughly 150-190 rating points of deflation. The following is the rating gap between AI period and pre-AI period.

Background

• We need to estimate the recent rating dynamics and compare it with pre-AI dynamics.
• Participants have natural improvement when they train, especially lower rated participants. We need to account for that.
• We need to somehow restrict the data to likely legitimate users.

Methods

I estimated rating dynamics in two periods:

1. January 2025 — June 2026. I call it the AI period.
2. January 2023 — June 2024. I call it the pre-AI period.

For each period I formed 4 cohorts, based on the contestant rating at the start of the period:

1. rating $$$\ge 2400$$$
2. rating $$$[2100, 2400)$$$
3. rating $$$[1900, 2100)$$$
4. rating $$$[1600, 1900)$$$

I used several simple filtering criteria to increase the probability of legitimacy of sampled contestants. For example, I excluded several regions from sampling.

For each cohort I used all users left after filtering.

For each user, I computed delta as rating at the end of the period minus rating at the start of the period. For each cohort, I computed mean delta and median delta. The gap is AI period delta minus pre-AI period delta. The mean-gap CI is a Welch two-sample 95% confidence interval. The median-gap CI is a 95% bootstrap interval.

Results

The main tables compare raw rating change in the AI period with raw rating change in the pre-AI period.

So the pre-AI period had clear natural improvement in the lower three cohorts. In the AI period this improvement disappeared, and the high-rated cohorts lost rating on average.