TLDR;

I built an enterprise grade machine learning model to predict a user rating 6 months in the future. The model has mean absolute error of 65.15 (that is, $$$ E \left [ \left | \hat{x} - x \right | \right ] = 65.15 $$$). You can use it here:

https://fbrunodr.com/predict-codeforces-rating

Motivation:

Check this thread: https://codeforces.me/blog/entry/143626?#comment-1282206

Before going forward with this post I have to admit a pretty important thing: I did not do what I promised, as I did not build a foundational model on top of codeforces data. Reasons:

1. Takes to much time to train on my personal laptop (or money to rent gpus, which I am not willing to expend for a toy project).

2. I still almost went down the path of finetuning some feature extractor model (such as this one), but then I remembered I had to deploy this somewhere. My website runs in a small dedicated server (I don't do serveless to avoid unexpected bills), so running a medium language model there was not a viable option. I also did not feel like renting gpus for that (again because of money).

So I did not strictly build a state of the art machine learning rating predictor model... But I did the next best thing which is: feature engineering + tree decision model. I describe in detail how I did this in the next sections and how you could train an actual foundational model for this task at the end (if you are actually willing to waste time or money on this).

Data collection

This is actually the most important section, as you need lots of data to train a machine learning model. I heavily used the codeforces API for that (even getting IP banned a couple times). Anyway, here is what I did:

1. Used https://codeforces.me/api/user.ratedList?activeOnly=false&includeRetired=false to get non-retired users (people who have logged at least once in the past month).

2. Selected 1.5 k top users + 50 k uniformly random users from the previous step.

3. Collected submissions data + rating data + blogs data from each selected user and saved all the data.

Just to have an idea, the total data I collected from codeforces adds up to 26.06 GB!! (and I didn't even use all the users).

Label preparation

As I said in the previous section I used a decision tree machine learning model to predict rating. A decision tree model works like this:

$$$ \text{model}: \text{tabular data} \rightarrow \text{prediction} $$$

So we need to get labeled data in tabular format to train the model. For that I did the following:

For each user on each 1st day of each month from Jan 2021 to Nov 2024 we extract features from this user as well as their rating 6 months in the future (30 * 24 * 60 * 60 seconds in the future to be precise).

The extracted features are the tabular data and the rating 6 months in the future is the label. By the way, what are those features? Well, you can set them to any data you want (be aware of data leakage, we don't want the model seeing in the future). For this specific model the chosen features can be seen here. Here are some of the most important features to give you an idea:

• Number of problems done during a contest in the last 3 months

• Time since account creation

• Number of ACs on problems much above current user rating in the last 3 months

• User's region

• Delta rating in the past 12 months

Model training

After collecting and preparing all the data it is time to train the model. First I separated the users into training and validation groups, using a 80:20 split. The idea is that we only train on the training users and only evaluate in the validation users. The patterns the model learn in the training group should apply in the validation group, even if the model has never "seen" those users before (which is going to happen in production). We also separate traininig and validation chronologically: we only trained with data from [2021-01-01, 2023-11-01] and only validated on data from [2023-01-01, 2024-11-01]. We do this because we also want the model to be robust to time variation (as the data used on production is going to be at least 6 months more recent than the data used to train, so if the model overfits to a time window it's not going to be so useful in production). Notice I had to cap the validation to 2024-11-01, because 6 months in the future from that is 2025-05-01, which is close to current date (maybe I could have gone a single month more, as I collected this data in June anyway, but whatever, doesn't make that big of a difference), and the labels are always 6 months in the future.

Anyway, after doing all this work (yeah bro, data collection and processing is 90 % of the job) we are finally ready to actually training the model: simply feed (feature, label) pairs to a decision tree model and let it do its work. I used catboost by the way, for three reasons:

• Basically as good as XGBoost and Lightgbm, which are other sota decision tree models.

• It has support for categorical features out of the box (so I don't have to one-hot encode things, which is annoying).

• 😸s (yes, I meant cats. This weight a lot when deciding for 😸boost vs lightgbm).

How to improve your rating (according to data science)

Training a foundational model to get SOTA performance