MikeMirzayanov's blog

Tutorials crowdsourcing
By MikeMirzayanov, 15 years ago, translation, In English
In case you are not familiar with the term, this the way of filling the site with content by the joint effort of the community. You might read a Wikipedia entry.
Basically, the idea to apply crowdsourcing to contests tutorials seems quite natural, but… it works badly. As you see, it is not the first time there has been no one willing to help the community. The question is: why and what is the way-out?

Let’s discuss it together. What should be changed/improved in order to make contests tutorials appear at Codeforces?
  • Vote: I like it
  • +6
  • Vote: I do not like it

Comments Comments (14)
| Write comment?
15 years ago, # |
  Vote: I like it 0 Vote: I do not like it
Looking at the chart at http://en.wikipedia.org/wiki/File:Crowdsourcing_process2.jpg, I would agree that step 6 is integral to success. I, however, am not aware of the current policy, and couldn't rapidly locate the blog post about it. Whatever the current payoff is (if anything at all), it looks like it isn't enough. Note that an award needn't be something physical or financial. Extra contribution points, registration tickets for competitions once they start filling to capacity, badges in profiles, additional handle decoration -- any of these could turn out to be the magic formula.
→ Reply
15 years ago, # |
  Vote: I like it 0 Vote: I do not like it
It seems there isn't an easy way to edit a tutorial, there is only a way to write a whole one.  For example, if I want to write the tutorial, what do I have to do?  For now, it seems first I have to solve all the problems in the contest, then I have to write out solution to each one, explain them and submit it to MikeMirzayanov.  However, that is a lot of effort.  What if I am only interested in writing the solution for one problem?  Sometimes I want to write a tutorial for only one or two problems, but not all 5.  What if I can write something , but not necessarily right away (like in a few weeks)?   There is no Wiki-style template set up where people can edit things right away.  At the same time, if there is we will have a problem with quality control.  This can be solved by some sort of evaluation system, but it doesn't exist at the moment.  I really like CodeForces style contest, but it needs more infrastructure for some things.  For one, may be we can have a more elaborate searchable Forum, then Wiki Pages for problem tutorials (but each user's contribution ought to be separated), and then may be even a way for problem setters to submit problems, if that ever becomes a need.  In general, there should be better ways to locate information.

All of the above is easy to say, but I know for a fact it means work for the administrators to implement things.  At the moment, CodeForces is pretty good and functional, so may be improvements can be made slowly for the future.


→ Reply
15 years ago, # |
  Vote: I like it +5 Vote: I do not like it

Personal thoughts

There are some minor details I guess they could make members hesitate to contribute:
  1. MikeMirzayanov: "The round tutorial is waiting for a volunteer. It is desirable that it will be one of the leaders of today's competition. The tutorial should be in Russian and in English."
  2. Intuitively, a good tutorial = a complete one = you should solve all problems and write all analysis for each of them = take much time
 In long-term vision, I think it's hard and inconsistent to manage discrete blog contents:
  1. If I was looking for all tutorials of Codeforces beta round #9, which tags should I use to search?: [tutorial], [analysis], [solution], [CF 9], [round 9], [round #9],... or some tags in Russian?
  2. If admin added tutorial links to contest page, would he add all links? or which standard he would use to select?
  3. How if I'm interested in all discussions about a particular problem?

Let's clarify our goal for a tutorial

  1. Do we need an overview of the entire match (like Topcoder)?
  2. For each problem, we prefer as many alternate solutions as possible?
  3. Each solution should be tied with a sample source code?
  4. Written both in Russian and English = one person write both English and Russian versions (possibly use translators) or it could be 2 different writers?
  5. How to evaluate the quality of tutorials?
  6. Some awards for contributions?

Personal opinions

Just few opinions aim at above questions:
  1. In fact, Mike often writes a short summary blog after each contest. With a bit more details (like naming the winners), it could become an overview.
  2. After someone gets AC a problem, one more button [Contribute a hint] will appear with some sincere messages like "Congrats! you've passed all tests! We'd love to know your idea for this problem." In short, we're trying to collect all correct hints for each problem. And one can contribute hints for any problems he/she likes.
  3. In the problem page, one more column [Submitted hints] where people can find all hints for a particular problem.
  4. Community will evaluate each hint by voting +/- points. For any yellow / red members or problem-setters, once they click [I like it] <=> +2, instead of +1. The hint with highest plus points will be at the top place. It can reduce the workload of admins, and treat all contributors in a fair way :)
  5. Awards for contributions: IMHO, if one is willing to share his/her ideas, being awarded or not, simply doesn't matter at all. However, in order to encourage people, we can create a simple rank-list for contributors, like if his/her hints can get high positions = add contribution points and set he/she as a candidate for Friendliest coder of the month (a badge in profile?)
As yiuyuho said, it takes time to implement additional features. But once our goal is already clear, improvements can be made step by step.

→ Reply
  • 15 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it
    I think your suggestions are very constructive.
    → Reply
  • 15 years ago, # ^ |
      Vote: I like it +1 Vote: I do not like it
    Yes.  The idea of having to submit hints should work quite well.  Here is another system came to my mind.

    1.  For each contest (after actual contest), there is a page linked from the contest page to a tutorial / summary page.

    2. There is a short match summary - I assume Mike or some other admins can write a little bit about it, then other statistical information can be automated.

    3.  For each problem, there are tutorial(s), written by different authors / volunteers.  On the page itself only one is shown: the one with the most +, breaking tie by contest rating of author.  But, there is an option to see other tutorials (for the same problem, by a different author, sorted by +, breaking tie by rating of author).  A reader can then rate the tutorials with +/- (similar to #4 above).

    4. An author can write at most one tutorial (post) for each problem/language combination.  He/She can, however, always edit and improve his current version, and previous version are also available to readers (like TopCoder Forum).  This is to prevent information overflow or having to look through chains of posts for information.

    5. Tutorials for a different language (Russian, Chinese, etc.) can also be posted, in which case readers seeking help in a different language can get it.

    6.  Readers can post comments for each tutorial, suggest improvement, but only the original author can make changes to it.

    OK, let's see some use cases.

    1. Code Forces Round 8 occurs.

    2. I almost got problem D, but not yet, so I spent some more time playing with it.  Finally I solved it a few hours after contest and decided I want to write about I find out.

    3. I write a tutorial about it.  At that time other people might have written tutorials for this problem, in which case I may decide not to spend extra effort.  Otherwise, what I wrote will be the first to appear for everyone else. (I can also choose to write short hints for different problems, and they will appear as tutorials written by me).

    4.  Someone reads my tutorial, are unclear about certain explanation, found some typos.  He post a comment on this tutorial, and I got a notification.  

    5.  I decided to make changes and improve my writeup for the problem.

    6. Another reader reads my tutorial, and decides to write/translate it in his own language (Russian, Chinese, etc.)  Now, there are tutorials in multiple languages.  Since all posts and edits are time-stamped, the original write can always claim authorship (and the other becomes a translation).

    7.  Anyone can write for any problem at any time after contest, things will increase and improve, and everything for one problem is located at one place.

    It doesn't seem to be too much work on the infrastructure - just need a specific page for tutorials for each contest, and each piece of tutorial can be considered as similar to a post, with addition restriction that each author/problem/language combination only exists once, and with a different +/- system. 

    As for reward, the number of +'s received on tutorials and number of first place tutorials can be a statistic on the author's page (may be even with ranking, not sure if too much ranking is a good thing).
    → Reply
    • 15 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it
      Having things split up has another fundamental advantage: Newcomers may want to contribute, but find it hard to understand/solve the harder problems.  More seasoned coder may want to contribution, but do not want to waste time on the easier/boring tasks.  So, with the split, newcomers will write about the easier tasks, while more seasoned coders can focus on the more difficult problems.


      → Reply
    • 15 years ago, # ^ |
        Vote: I like it 0 Vote: I do not like it
      This system is great but somewhat time-consuming to the admins. But we can get over it by the following unmatured idea.
      The slogan of Codeforces is "The only programming constests Web 2.0 platform". How exciting! Considering people here are all programmers, may I suggest crowdsourcing the implementation work to the users too if need be?  Don't forget what the spirit of Web 2.0 is: contribute, share and enjoy. 
      I deem the best way is going step by step. After all, Rome was not build in one day, the community would not be either.
      So guys, action! This is our own community, isn't it?
      → Reply
    • 15 years ago, # ^ |
        Vote: I like it +3 Vote: I do not like it
      I like this very much! It combines "small hints" and full-blown tutorials, and it's consistent with CF slogan "Web 2.0 contests". The only problem I see now is the following: someone writes a non-trivial idea for a problem, and it gains rating quickly; after that someone decides to write a full tutorial, but it's lost because few people will read it after they figured out the solution. I think the reward should be determined by the admin; perhaps, there need to be a 'crowd choice' reward for a comment that received the maximum rating and 'admin choice' reward for the best tutorial. The links to both will appear on the problem's page.
      → Reply
15 years ago, # |
  Vote: I like it 0 Vote: I do not like it
It seems that not enough people are willing to simply share their ideas. As we know, TopCoder tried to implement a similar system, and it wasn't successful although they had a wiki set up, each user could contribute to each individual problem, there was only one language to write it in, etc. The idea about the hints, however, seems like it could work.
→ Reply
  • 15 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it
    Oh, maybe, if womeone has trouble with English but wants to write a problem's solution, h (she) made it in russian first, then someone else just translate it into English, getting exra reputation (internal currency).
    → Reply
15 years ago, # |
  Vote: I like it 0 Vote: I do not like it
After I saw the invitation to write tutorial after round 7, I thought "maybe I can do it". Then, what made me not doing anything with that:
- Tutorial must be also in Russian. I don't speak Russian, so I'm not the best one to do it.
- I must write solutions to all problems. That means I must have done all of them. I don't know how to do (or at least prove) solution to last task, so again I'm not the right person here. That also means a lot of work since 5 completely described tasks is much.
- Finally I supposed that you will have rather too many volunteers than too less. Now I see the assumption was definitely wrong.

Generally I can declare myself to do some stuff and help to develop the site. I'm not the one with a bunch of time but I'd like to do something.
→ Reply
  • 15 years ago, # ^ |
      Vote: I like it 0 Vote: I do not like it
    Insightful. 
    Simply put, there should be an easy way to contribute.
    → Reply
15 years ago, # |
  Vote: I like it 0 Vote: I do not like it
I think each problemsetter sould write a sample solution in some common language and, of course, the tutorial.
→ Reply
15 years ago, # |
  Vote: I like it +5 Vote: I do not like it
I agree with yiuyuho and freepascal145 that it should be easier to contribute a smaller chunk of work. I don't know why the match summary and individual problem analyses must be written as part of a single tutorial; they make sense on their own. For example, the CodeJam site presents the match analysis in chunks too, and that seems to work well.

Similarly, the requirement to write both Russian and English texts limits the potential contributors to those that are fluent in both languages, even though I imagine there are Russians that find writing in English too hard, but would be willing to translate a text into Russian, or vice versa. At any rate, it seems to me that having a Russian/English analysis only is better than having none at all, but the current rules prevent it.

So the summarize the above: I think you should consider reducing the amount of work to be done by a single person at a time, in order to reduce the threshold of contributing.

As an aside, I've edited a couple of problem analyses at TopCoder, and two things which sometimes put me off there, were the awkwardness of the wiki software (it's difficult and time consuming to properly format formulas and code in your analysis) and the fact that you don't know whether somebody else is writing a problem analysis too (possibly obsoleting your contribution). That last possibility is especially discouraging, because getting your contribution published is the only reward you get for your work.

Finally, I find the best way to solve a task with a group of people in any situation is to break it down into smaller subtasks that can be completed individually, and then assign each subtask to a person so that someone is specifically  responsible for its completion. That may not be directly applicable to the current situation, but the closer you can get, the better.
→ Reply