akshat.jaipuriar's blog

ML Based Rating Change Predictor and Problem Recommender
By akshat.jaipuriar, 18 months ago, In English

Hello Codeforces,

I wanted to share this little project I made using ML to predict your ratings after the next 5 contests and recommend a few problems based on the predicted rating to help you grow based on your Top 3 weakest topics.

The project uses a Random Forest Regressor to learn patterns of different users in different rating ranges using factors like

  1. Acceptance Ratio

  2. Rating Trend

  3. Tag Diversity

  4. TagWise Success

Application Link: cf-rating-predictor
GitHub Repo: CFpredictor

This is my first time implementing something ML-based so please forgive me if I did something wrong ;)

Thanks if you have any suggestions do let me know :)

  • Vote: I like it
  • +41
  • Vote: I do not like it

Comments Comments (22)
| Write comment?
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

How accurate was it on your test set?

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Amazing!

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Thought it was funny that 952D - I'm Feeling Lucky! got recommended.

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Out of the 10 tasks it gave me, I've solved 3 of them previously. It should check if I've solved it before.

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

It shouldn't recommend problems which have *special problem tag.

It recommended for me these problems: 530A, 130E, 130C.

→ Reply
»
18 months ago, hide # |
 
Vote: I like it +25 Vote: I do not like it

That's a nice thing you did! I'm happy you decided to try something like that, and thanks for open-sourcing your project. I did some AI projects at a student camp in Serbia, so I would just like to add some constructive criticism.

First, it would be nice if you explained your methodology somewhere. I looked at your repository, and luckily, your code was quite easy to follow.

I would like to discuss the methodology of you giving problem predictions. You find the user's weak topics (where a topic is a CF tag) and randomly select 10 problems from those topics. And weak topics are those with a low accepted/submissions ratio. I see three problems with such an approach.

  1. Does a low accepted/ submissions ratio mean a weak topic? It means a topic for which you used a lot of submissions to solve a problem on average. Which usually corresponds to topics for which you struggle with implementation or have a lot of edge cases. But that doesn't mean weak topic, does it? Like, it can be a math problem where the solution is 10 lines of code, but deriving the formula is the hard part. I feel like a far better metric is time spent on a problem with a certain topic. Though it is quite difficult to get from the codeforces API (only with contest problems, but unless someone did a ton of contests, that's not much data)
  2. It does not account for improvement. Like, I may have made a ton of WAs 5 years ago, or did a lot more greedy problems back then when I wasn't as good. And nowadays, I make almost no WAs or barely meet the topic at my level. So weak topic is determined by what I did 5 years ago, which I do not think makes much sense, as one can improve a lot in that time.
  3. Using current codeforces tags for topics isn't that great. I feel like a better system to categorize problems is needed. However, I am unaware of the existence of such a system, and generally, this issue is not as big as the other 2.

Now, your methodology for rating prediction is kinda unclear. I am interested in how you collected the data you have. How good is your model at predicting? What is the error? What are the errors of some simple solutions, such as predicting the current rating, or the average/ median of the past 3 ratings?

Though I will note that rating prediction is always just for fun, so it doesn't matter how correct it is. I just think that such data and methodology explanation would make it better.

Again, I do think it is good that you made such a project. Do not take this comment as discouragement from making similar projects in the future. I just wanted to provide some constructive criticism.

→ Reply
  • »
    »
    18 months ago, hide # ^ |
     
    Vote: I like it +12 Vote: I do not like it

    Thanks for your comment I would answer all your queries.
    1. Yes actually the main idea of the project was to build a rating predictor so I didnt put much effort on the recommender and obviously as you mentioned getting the accepted / submissions was the best metric to identify weak topics. I too thought of time spent per problem but as you said we cant get that information from the API.
    2. Actually If you carefully see the Code I fetch only the last 1000 submissions per user so that I dont get too much old data for a user to reflect on their latest trends.
    3.I would definitely research on this one and see what i can do about this too

    As for the Rating Predictor I sample data of around 1500 users of almost all the rating ranges from 900 to 4000 (obviously there is less data available for rating ranges such as yours and above so the model can be a bit off for these ranges).
    In train.py you can see I have trained the model on 80/20 part with the testing results as followed:
    Evaluation on test set:
    MAE : 33.06
    RMSE: 65.88
    Avg % Error: 1.70%
    Again this data isnt uniform but for the whole set it may vary for different ranges.

    Thanks again for your comment and I wont take this as a discouragement ofc it was just a fun project.
    Hope It cleared your queries Thanks :)

    → Reply
    • »
      »
      »
      18 months ago, hide # ^ |
       
      Vote: I like it 0 Vote: I do not like it

      Thank you for your answers! I am wondering how exactly you picked the 1500 users to sample? From what I understood, you did not just randomly choose people from CF users, as that would result in a lot of grays and just a few reds. But how did you ensure that all ratings are represented?

      → Reply
      • »
        »
        »
        »
        18 months ago, hide # ^ |
         
        Vote: I like it +12 Vote: I do not like it

        At first I took a little big ranges of delta around 400 but this resulted in a lot of biasness towards the upper limit of the rating range so I just checked the underrepresented rating ranges by narrowing down the delta to fetch users of all rating ranges.

        The exact implementation of the API calling and all is in the scraper.py script in the repo where I manually changed the ratings ranges to ensure fair representation.

        These are the analysis of the training data-
        Rating Stats:
        Min Rating: 948
        Max Rating: 3849
        Average Rating: 1893.72 

        Rating Distribution:
0-400          0
400-800        0
800-1000      99
1000-1100     50
1100-1200    100
1200-1300     50
1300-1400    100
1400-1500    100
1500-1600     50
1600-1700     50
1700-1800     99
1800-1900    100
1900-2000     50
2000-2100     50
2100-2200    100
2200-2300     50
2300-2400    100
2400-2600    100
2600-2800     92
2800-3000     45
3000-3400     31
3400-4000     14

        So this was my way any suggestions are welcome. Thanks!

        → Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

lol ur going to lose 28 points on the next 5 contests. actually sounds like an ai-curse/-bless

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

idea is cool however I tested it on some accounts and the prediction has always been negative delta. a bit too pessimistic i believe

→ Reply
»
18 months ago, hide # |
 
Vote: I like it +5 Vote: I do not like it

ok now I'm a bit frightened :(

-157

→ Reply
  • »
    »
    18 months ago, hide # ^ |
     
    Vote: I like it +23 Vote: I do not like it

    Hey! the model may not be this accurate for accounts such as yours which have exponential growth because it predicts ratings based on multiple factors with number of contests, questions solved, tag diversity etc. so just relax hoping you become red after the next 5 contests :)

    → Reply
    • »
      »
      »
      18 months ago, hide # ^ |
       
      Vote: I like it 0 Vote: I do not like it

      Thank you for your assurance but I don't really think red is possible for me in the mean time :(

      → Reply
      • »
        »
        »
        »
        18 months ago, hide # ^ |
         
        Vote: I like it +7 Vote: I do not like it

        I don't think so seeing your graph you would be red in the next 2-3 contests only trust yourself :)

        → Reply
        • »
          »
          »
          »
          »
          18 months ago, hide # ^ |
           
          Vote: I like it 0 Vote: I do not like it

          You might've forgotten additional rating delta ToT

          According to Carrot my performance is about 2200 — 2300 but hardly higher

          → Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

For tourist, it says:

Predicted Rating after 5 contests: 3589

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

What happened? For my handle, the machine just says:

Error: 'contestId'

But when I tried other handles, it works, so it may not be the problem of my computer, but why?

→ Reply
»
18 months ago, hide # |
 
Vote: I like it 0 Vote: I do not like it

Has anyone tried tourist yet? Because it pointed out that he's gonna get -260 in the next 5 contests

→ Reply