Hi beautiful people of Codeforces! Hope you'd all be doing fine and keeping safe.
I'd like to start by mentioning that it's been a while since I graduated from university ever since which I haven't completely been able to indulge myself in competitive coding in the same way I did back in my university. But I sure haven't forgotten those days. I had some time off from work recently and decided to spend my time giving back to the community from where I have learned so much.
In the thought of so, I've built this tool which I like to call Harwest (a play on the word Harvest since the package name wasn't available anymore :'D) It's one of those tools which I would have loved to have during the time when I used to practice rigorously on this platform.
Harwest allows you to manage all the submissions that you've made on Codeforces, or on other OJs (WIP) into one single Git repository which you can then push to your personal Github / BitBucket / Gitlab or any other SCM platform of your choice.
Harwest completely automates this entire process with minimal effort from your side of answering just a few basic questions, and even goes to the limit of pushing the Git repository to the remote (Github/BitBucket..) for you!
Here's a sample repository built solely using Harwest: https://github.com/nileshsah/harwest-sample
The project directory structure is as follows:
```
nellex@HQ:~/accepted$ tree
.
├── codeforces
│ ├── <contest-id>
│ │ └── <problem-level>
│ └── 592
│ └── A.cpp
├── README.md
├── submissions.json
└── .git
```
Harwest creates a local git repository for you, scrapes all your submissions directly from Codeforces, maintains them in a nicely organized way, and commits them to the repository with the actual date of your submission!
Now what the last statement enables us to do is to maintain a complete contribution graph on GitHub (or others) accurate to the time when you actually made the submission and not when you made the commit. Yes! It means complete tracking of your submissions on a day by day basis. Keep track of the number of days you've been lazy for, the number of problems you solved on a given day, and maintain a complete streak of daily practice! Heck, let this streak be your new motivation to practice continuously to become a better version of yourself every single day
I've detailed the entire process of configuring Harwest on your local computer with an example in https://github.com/nileshsah/harwest-tool#installation Let me know if something needs more clarification.
It's built using Python so you would require PIP with Python version 3.5 or above to install the package which you might have to set up in case you haven't used it before. It's a great package manager which I'd recommend learning a bit about nonetheless since it'll come in handy in the future.
Along with PIP, you'd also need git on your computer which would come pre-installed if you're using Linux or MacOS though might would have to work your way up in case you're using windows. I'd also strongly encourage you to learn about Git since it's something that is heavily used in the software industry and early exposure to it would sure make your life a lot easier in the future. There are numerous documents on the internet which can help you get started on Git, one of them is https://www.freecodecamp.org/news/an-introduction-to-git-for-absolute-beginners-86fa1d32ff71/ Though For operating Harwest, you'd require minimal to no experience with Git.
The first time Harwest runs, it'll scrape all your submissions starting from the first page till the last which might be a bit time-consuming if you had made a lot of submissions. If you don't want to parse the results till the very end, you can stop processing by hitting <Ctrl> + C. The next runs of Harwest will no more try to scrape the results from the old pages and instead will push your new changes directly. You can then make use of the harwest codeforces --start-page X feature to restart scraping from the page where you left off.
I haven't got a chance to extensively test Harwest much and hence the term [Beta]. It seems to be working fine on MacOS and Ubuntu but I guess the auto push feature is causing a bit of an issue on Windows especially if you're using GitHub for Windows integration Do keep me posted if you face any other issues and I'd be happy to fix it at once.
As for a workaround for the Windows user, if you want to leverage the automated push feature then specify the remote URL for your repository when asked as https://username:[email protected]/username/repository.git to bypass the login step during the push. Alternatively, you can always leave it empty and do a push from the generated repository by yourself.
Nonetheless, I had a fun time building this tool and I really hope it would be helpful for you. Contributions to the project are more than welcome!
TL;DR
Automate the management of your submissions on GitHub with https://github.com/nileshsah/harwest-tool to get something like https://github.com/nileshsah/harwest-sample
Update <2020-12-27>: Support for AtCoder is now available!
Harwest now supports AtCoder! Massive thanks to s59_60r for leading the effort of finding the ideal way to integrate with the platform. Not to forget the crucial contributions from I_love_Hoang_Yen, YouKn0wWho, and others for fixing some major performance issues and making Harwest more stable. Much appreciated.
The sample submissions repository https://github.com/nileshsah/harwest-sample has been updated to reflect what new changes this update brings. In order to switch to this release, please upgrade Harwest by running:
pip install --upgrade harwest
Ensure that you're on version harwest-0.3.1, The new release is backward compatible and will not break any of your existing harvested submissions. After switching to the new version, you can then harvest submissions from AtCoder by running
harwest atcoder
In addition to the new integration with AtCoder, we have introduced --full-scan capability on platforms to avoid the pain of dealing with re-starting harwest from the page where it stopped processing due to an error. It can be triggered as:
harwest codeforces --full-scan
OR
harwest atcoder --full-scan
Please do let me know if you face any difficulties in the process, would be happy to help at once.
Happy Harwesting! :)
Auto comment: topic has been updated by nellex (previous revision, new revision, compare).
I installed Harwest but when I try to run the command.
It says harwest: command not found.
I’m using Ubuntu.
Hey sam571128! Thanks for trying. I'm guessing it might be so because the directory under which pip installed the package may not be present in your PATH. Could you try running the command
pip3 uninstall harwest>
/home/nellex/.local/bin/harwestThe path
/home/nellex/.local/binis something you'd like to register under $PATH asexport PATH=/home/nellex/.local/bin:$PATHor you can execute the command directly as/home/nellex/.local/bin/harwestLet me know if that works.
Sorry for replying so late. But yea, it worked.
This is such a good idea! unfortunately I'm getting an error when running harwest codeforces (and putting in my handle):
Thanks a ton AkshajK I've updated the code to skip SSL verification. Can you try installing the new version by running
pip3 install harwest==0.2.9Let me know if the problem persists.Would like to extend my heartiest thanks to YouKn0wWho for helping me out in fixing some of the major encoding issues in Harwest on the windows platform. In case you were a windows user and have already tried Harwest then you might notice ^M characters in the generated submission files. If so, then I'd recommend deleting the locally created repository and starting over with
harwest --initafter updating to the latest version by runningpip3 install harwest==0.2.9Apologies for the inconvenience.As for YouKn0wWho, the amount of hard work you've put in this year itself is simply remarkable! https://github.com/ShahjalalShohag/ProblemSolving
I have installed it and pushed my submissions to github. Now if I make more submissions, how do I push them again so that it only takes the submissions which are not pushed yet?
Hey anshu_2761! Thanks for giving Harwest a try! Harwest keeps track of your past submissions and fetches only the ones which are new. Whenever you'd like to sync the submissions in your GitHub repository with that on Codeforces, just run
harwest codeforcesand it'll do the job for you. Will also push automatically if you have set it up with auto-push else you can manuallycdinto the local submissions directory and issue agit push origin masteryourself.I believe this was your question right? Also, I'd recommend running a
pip3 install harwest==0.2.9again since I pushed in some new set of fixes which would be good to pull.Nice tool!
However, the process was stopping a lot before finishing all pages, and I had to use the commnad
harwest codeforces --start-page xmultiple times, then I wrote this python snippet to do the job.Thanks Mohammad_Yasser! Nice hack I'd say! I would like to apologize that I didn't stress-test the tool enough to check its breaking limits. I_love_Hoang_Yen just helped me out in introducing a performance update (PR#3) which should address your issue a bit. Updating to the latest version of Harwest should help. Could you try executing the commands:
If it doesn't fetch the latest version of
0.2.10then maybe you can do a force install usingpip install harwest==0.2.10Let me know if that helps and thanks for trying Harwest out :)
Thank you!
It worked fine. The update is a significant improvement.
I_love_Hoang_Yen you sure are winning hearts here :D
Thanks =)
It didn't work out for me. Like it does until page 9 and then when I check my repository, it has done only until some submission of page 8. I start again but it stops at page 9 again. If I start at page 10 instead, it does until some pages(14 or 15) and then again same problem occurs and then again after that. Can u help me out here? :( I would appreciate it
Sorry to hear that nmnsharma007 and apologies for getting back to you late. I'm taking a look at what could be the possible issue behind it. Meanwhile, maybe you can use the hack which Mohammad_Yasser proposed of running the script: https://ideone.com/01Yc9C
^ It should just be a one-time overhead when you're setting up Harwest for the first time. But I'm on it to identify what could be going wrong for you. Hope that works.
No problem at all. By the way,where should I put the script?Thanks for the help :) UPD: It worked. Thanks
It is just me or this tool doesn't work with gym submissions?
Hey timreizin! Thanks for trying Harwest!
You'd be correct that the first iteration of this tool does not support gym submissions or any other submission which is not public :/ The approach was taken to avoid having an extra overhead of requesting and maintaining the user's codeforces password. Though it sure is something that we're looking at including in the next iterations of the tool. In fact, it has already been reported here: https://github.com/nileshsah/harwest-tool/issues/7
Will prioritize it in case more people would like to see this feature. Hope that's alright.
Thanks for fast reply. Thank you for this tool.
Thanks for this amazing tool!
Please add support for other platforms as well :)
Glad you liked it kgoel! :) We're already working on an integration with AtCoder (all credits to s59_60r).
Which other platforms would you like to see on Harwest?
How about Codechef?
That's great! Thank you!
Other platforms like Codechef, Leetcode :)
https://www.stopstalk.com/dashboard something similar to yours.
Thanks for bringing this up daddy_puff! StopStalk is undoubtedly a wonderful tool to track your activity across different OJs. Huge respect for its creator.
Harwest though serves a bit of a different space. It enriches the belongingness of your submissions by allowing you to host them on a platform of your choice rather than being present on a remote system.
Harwest fosters the idea of taking immense pride in putting up your solutions on a platform like GitHub since it reflects on the amount of hard work you had put in to succeed early on in your career. As the years go by, you'll realize that all these other tools will fade away but it will only be your Git profile that sticks with you for a while. And I promise you, every time you look back, these contributions will bring back nothing but the good memories of the not so long past: https://github.com/ngthanhtrung23/CompetitiveProgramming
Thanks for the amazing tool please also consider including codechef, atcoder
Bug(maybe): After scanning all pages it showing I sovled 319 but my actual solves are 371
Thanks sainad! We'll be launching the support for AtCoder in a matter of few days and will then proceed to add integration for CodeChef. Contributions are always welcomed!
The discrepancy that you mentioned might arise from the fact that some of your solutions might not be publicly available, especially gym submissions which Harwest at the moment does not scan. In order to support these private submissions, we'll have to enable login flow on Harwest which we might consider in the future if we have more bandwidth. Hope it is alright for now.
Auto comment: topic has been updated by nellex (previous revision, new revision, compare).
Support for AtCoder has been added! Please refer to the update section of the blog for further details.
GitHub
It seems that I can never get all my submissions (I should have around 600). I've tried
harwest codeforces -fmultiple times but all of them yield different results, yet none producing all my submissions. Looking at the commmit graph, it seems that harwest ignores a certain period of time, which has happened in all my previous attempts (the actual time is different though). Bug?Edit: When I run something similar to this, I manage to get all my submissions.
Hi skittles1412! Thanks for bringing this up. It sure is quite strange as to why you had to face this problem. I'm mostly guessing that it might be so because some of the requests to fetch the submissions are failing from the server-side. Ideally running a full-scan should have picked up these missed solutions. Hmm, I'll try to reproduce the issue at my end and check if I can locate it. Meanwhile, thanks for sharing a temporary fix.
I believe the atcoder harwest for java submissions does not work
Hi bleh0.5! Could you let me know what issue are you facing? Harwest should support Java submissions from AtCoder
Error message:
I just assumed it had to do with being unable to do java because I recently switched languages
Thanks for sharing the issue bleh0.5. The necessary file extension has been updated as part of the issue https://github.com/nileshsah/harwest-tool/issues/14
Please consider upgrading your Harwest version to 0.3.2 to solve the issue at your end. You can do so by running
pip3 install --upgrade harwestPlease ensure it says — Successfully installed harwest-0.3.2Do let us know if it still doesn't work for you.
Ok it works, thanks for the amazing tool!
nellex thank you so much for this awesome tool. Loved it.
Traceback (most recent call last): File "c:\users\shubham\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main return run_code(code, main_globals, None, File "c:\users\shubham\appdata\local\programs\python\python38\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\Users\SHUBHAM\AppData\Local\Programs\Python\Python38\Scripts\harwest.exe__main.py", line 7, in File "c:\users\shubham\appdata\local\programs\python\python38\lib\site-packages\harwest\harwest.py", line 110, in main print("Hey there!", "\U0001F44B", File "c:\users\shubham\appdata\local\programs\python\python38\lib\encodings\cp1252.py", line 19, in encode return codecs.charmap_encode(input,self.errors,encoding_table)[0] UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f44b' in position 0: character maps to
/ __ / /_/ / / | |/ |/ / ( ) /_ /_/ /_/__,_/_/ |__/|__/___/____/__/
==========================================
This is a rather interesting case shubham_409, can you run the following commands on your terminal before running Harwest and let me know if it fixes the issue?
It worked like a charm using powershell but not by git bash. Thanks for making it. It would be great if it could support other ojs also.
Glad that you liked it shubham_409. We're looking at on-boarding CodeChef next. Will let you know as soon as it's available :)
eagerly waiting for codechef!
Can you please tell what's the issue?
Hey raja_n, thanks for reporting the issue. It's something that we're aware of hence the paragraph in blog about the workaround:
It was reported on https://github.com/nileshsah/harwest-tool/issues/10 and it turns out that it might get fixed if you update your git version. Can you possibly try that and let me know if that fixes the issue for you as well? Thanks!
I have updated git version but now it's showing below error:
Hey raja_n, seems like there are already some files in your CP-Practice repository: https://github.com/Rajan-226/CP-Practice I'd recommend deleting this repository on GitHub and creating a new empty one without any README/Licence files. It should work then.
Another alternative is to
cdto your local directory ofCP Practiceand issue agit pullcommand to fetch the remote changes onto your local.Thanks for replying fast. Yes, it works.
Are you planning to add any other platform like codechef, leetcode?
Adding other platforms would be great!
I was expecting that it will become green in github at the time when I submitted a correct solution in codeforces. Am I wrong somewhere ?
C:\python\Scripts>harwest Traceback (most recent call last): File "c:\python\lib\site-packages\git__init__.py", line 83, in refresh() File "c:\python\lib\site-packages\git__init__.py", line 73, in refresh if not Git.refresh(path=path): File "c:\python\lib\site-packages\git\cmd.py", line 278, in refresh raise ImportError(err) ImportError: Bad git executable. The git executable must be specified in one of the following ways: — be included in your $PATH — be set via $GIT_PYTHON_GIT_EXECUTABLE — explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the $GIT_PYTHON_REFRESH environment variable. Use one of the following values: — quiet|q|silence|s|none|n|0: for no warning or exception — warn|w|warning|1: for a printed warning — error|e|raise|r|2: for a raised exception
Example: export GIT_PYTHON_REFRESH=quiet
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "c:\python\lib\runpy.py", line 193, in run_module_as_main return run_code(code, main_globals, None, File "c:\python\lib\runpy.py", line 86, in run_code exec(code, run_globals) File "C:\python\Scripts\harwest.exe__main.py", line 4, in File "c:\python\lib\site-packages\harwest\harwest.py", line 7, in from harwest.lib.codeforces.workflow import CodeforcesWorkflow File "c:\python\lib\site-packages\harwest\lib\codeforces\workflow.py", line 1, in from harwest.lib.abstractworkflow import AbstractWorkflow File "c:\python\lib\site-packages\harwest\lib\abstractworkflow.py", line 5, in from harwest.lib.utils.repository import Repository File "c:\python\lib\site-packages\harwest\lib\utils\repository.py", line 5, in from git import Repo, GitCommandError File "c:\python\lib\site-packages\git__init_.py", line 85, in raise ImportError('Failed to initialize: {0}'.format(exc)) ImportError: Failed to initialize: Bad git executable. The git executable must be specified in one of the following ways: — be included in your $PATH — be set via $GIT_PYTHON_GIT_EXECUTABLE — explicitly set via git.refresh()
All git commands will error until this is rectified.
This initial warning can be silenced or aggravated in the future by setting the $GIT_PYTHON_REFRESH environment variable. Use one of the following values: — quiet|q|silence|s|none|n|0: for no warning or exception — warn|w|warning|1: for a printed warning — error|e|raise|r|2: for a raised exception
Example: export GIT_PYTHON_REFRESH=quiet
Harwest is not working.What should i do??
Read the error message
orz
You're awesome great sir. This is absolutely wonderful.
Thanks a bunch! :)
Hi nellex
Currently, the recent GitHub projects use main as the master branch instead of the master. So when I use the feature to automatically push code to GitHub using harwest, it shows an error as my project doesn't have master as my branch.
cmdline: git push origin master stderr: 'error: src refspec master does not match any'This is the error you get
My suggestion is to take a branch as an argument during the init process in which you want to push the code and push code to that branch automatically.
I too am having the same error, but idk how I can -
"take a branch as an argument during the init process in which you want to push the code and push code to that branch automatically."
I found this error massage anyone can help me?
Error Massage:
Traceback (most recent call last): File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 194, in _run_module_as_main return run_code(code, main_globals, None, File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 87, in run_code exec(code, run_globals) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\Scripts\harwest.exe__main.py", line 7, in File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\harwest\harwest.py", line 115, in main args.func(args) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\harwest\harwest.py", line 70, in codeforces process_platform(args, "Codeforces", CodeforcesWorkflow) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\harwest\harwest.py", line 90, in process_platform workflow(configs).run(start_page_index=args.start_page, full_scan=full_scan) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\harwest\lib\abstractworkflow.py", line 105, in run self.repository.push() File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\harwest\lib\utils\repository.py", line 52, in push self.git.push(*args) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\git\cmd.py", line 542, in return lambda *args, **kwargs: self._call_process(name, *args, **kwargs) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\git\cmd.py", line 1005, in _call_process return self.execute(call, **exec_kwargs) File "C:\Users\USER\AppData\Local\Programs\Python\Python38-32\lib\site-packages\git\cmd.py", line 822, in execute raise GitCommandError(command, status, stderr_value, stdout_value) git.exc.GitCommandError: Cmd('git') failed due to: exit code(1) cmdline: git push origin master stderr: 'To https://github.com/yousuf7902/Codeforces_Solved-Auto-Generate.git ! [rejected] master -> master (fetch first) error: failed to push some refs to 'https://github.com/yousuf7902/Codeforces_Solved-Auto-Generate.git' hint: Updates were rejected because the remote contains work that you do hint: not have locally. This is usually caused by another repository pushing hint: to the same ref. You may want to first integrate the remote changes hint: (e.g., 'git pull ...') before pushing again. hint: See the 'Note about fast-forwards' in 'git push --help' for details.'
nellex I am getting this error. I had scrolled through the blog comments and found that there was a suggestion to update harwest which I have done.