P.s. I don't hate Polygon. Polygon is great system, that has no alternatives and makes problem setting much easier. I personally use Polygon, but I think with this specification it will be better.

To me, it was not very clear why this was not done before. I can only see the lack of a single documented format and lots of forks and interpretations of the existing one as an utter failure of the competitive programming community; and the lack of action towards putting right what once went wrong is astounding. I use a lot of software (though mostly FOSS), and I can't name one other popular format that is both undocumented (in any way, even unofficially) and used by many vendors.

People use Polygon: Codeforces uses it, almost everyone in the Russian CP world uses it, convertors are written for every other judge, and there's no alternative with better features. If it looks like an industry standard, used like an industry standard, and regarded as a industry standard, then there is no reason for it not to be an industry standard.

Kilometers of crutches and legacy code already exist, there's little we can do about it. We can, however, work towards decreasing the rate at which the pile grows. Introducing run-twice programs was a major change, introducing graders was a major change, and I have lots of ideas on my mind that would be considered major changes too. And when I say "major change", I mean rewiring half the architecture. I eyed all of this and encapsulated it till any other idea I could think about wouldn't require any changes. I'm advising people to spend some time now to save yourself time later.

It would be cool if guys familiar with how problems are prepared for the listed or other foreign platforms came here. After all, maybe they don’t use the polygon at all during preparation, but they have their own, perhaps more successful format.

A friend told me CodeChef has a rather dumb Polygon analogue, little more than input and output files and a checker. Can't say much about the other platforms, but given that I haven't seen any non-standard problems on AtCoder, I don't think it's much different elsewhere. CMS, according to the examples on GitHub, seems to only support a few hard-coded types of problems, nothing Polygon doesn't. I'm interested in how Leetcode works, though: it seems to use graders that work for multiple languages, and while I dreamt of that too, I deemed it too non-trivial for this iteration.

I read the document itself, IMHO, there is a global conceptual flaw: I think it is impossible to leave the writing of the strategy to problemsetters, because in the general case, as far as I understand, the strategy is almost an arbitrary program, moreover, it also has increased rights during the execution (because it needs to be able to run other programs). What if the strategy catches TL? ML? Trying to break the system?..

Of course it is possible to make it work. All you need is run the strategy in a sandbox like an interactive problem, and pass commands via standard input and output streams. I think I put it like this somewhere but can't find it now. Anyway, I don't see any security problems: a strategy merely tells the judge what to run and expects to be able to read and modify files generated during the submission judgement, but nothing else. And DoS isn't a problem either: even an interpreted language doesn't need more than a few seconds of CPU time and tens of megabytes to trigger evaluation of thousands of tests, or even await and parse the results, and await asyncio.sleep(1000000) can be fixed by disallowing the strategy to run idly for more than a few milliseconds when no other programs are run.

In codechef you have to prepare tasks on the polygon. And when the coordinator (or someone like that) accepts your work, you transfer tests to codechef judge. And in judge, you can only add a test, add a check, add the correct output for the test. And add a few more another settings. At least, that was about six months ago

Firstly, thanks for a reply!

At least, I didn't see any word about how judging reports should look like (both for jury and for participant): like, which files are exactly taken into account and should be saved for future usage?

I'm the proponent for data preservation: if you can generate stuff and have space to save it, do it. So, yes, if you have a run-twice problem, save all the generated files, they may come in useful when you rejudge a deterministic submission after the competition and get a different result and have no idea what went wrong. For the participant, standard rules apply: show them the verdict and metrics during the contest and everything else afterwards. I think this is mentioned in 6. Valuation.

And what about content of pipes? I don't think Linux pipe is enough tool to record all content in the pipe, which, in the turn, may be more useful for examining the strategy than something like first interactor stderr.

I left that implementation-defined as I'm not sure it's always feasible. I think that if you make split the FIFO into two pipes and have an intermediate process that reads from one end and writes to the other end while logging the data, or simply inject the logger into the libc's read and write functions on the interactors's end, it would work.

Provided idea of Python strategy code (with unfortunately no formal spec to examine)

I'm not quite sure what you'd expect from a formal spec. Perhaps the most complete spec would be a sample implementation; I'm working on releasing that now.

isn't clear about when execution of interactive problems start: there is just no await of anything

1. Pipelines says this:

...Invocation is performed even if the future is not awaited.

So when you invoke a program, it starts whenever the judge decides, which is, if you care about efficiency, either right away, or after the previous test finishes, or perhaps when a new invoker is available, or something.

Of course, you could just prefix the invocation with await, but that would make the tests run consecutively independent of the judge's parallelization features; or you could spawn a coroutine per test and use asyncio.gather, but that would just be more code that means the same thing.

It seems, we should use smth like asyncio.gather with some undefined behaviour on edge cases: e.g. events like “one process sends data, and another one quit” may lead to different results depending on which process is interactor, and which is solution.

Yeah, some clarifications are certainly in order. Let's say that CF overrides everything else, and otherwise all failure verdicts of a user process override the rest. So if the user program just exits successfully, and the interactor receives nothing on the pipe, it returns PE or WA and it's alright. If the user program crashes and the interactor receives nothing, it's still PE or WA, but RE overrides WA. And if the interactor crashes and then the user programs fails, you get CF, which is what you want.

Also, as it was already told, some judges (e.g. Yandex Contest) have features, which are not covered by the spec.

That's why I'm posting this here and not going ahead with implementing yet another format immediately--to hear more about these features. So far I didn't see anyone present any examples.

I'm almost sure nobody will wait some committee to approve the changes to standard before implementing needed feature. And in some time we'll get again to what we have now — several judges have done the same thing different ways. While it's possible to standardize such features later, it may be impossible to convert all “old-style” problems to a new format.

I have attempted to encapsulate everything related to programming here. You can compile programs however you want, you can run programs, in any order, with any arguments, in any structure--what more do you need? The only fixed detail left is the existence of tests and that the user submits a file. If you ever get tired of tests, just add a single empty test and do everything in the strategy--at least it'll work in other judges then. If you ever need the user to submit a directory or a text string, simulate that with an Archive and a plain-text file. What more do you need?

By the way, this spec is too restrictive. Just a few examples: you write, that you restrict file-only I/O just because you personally think it's good.

I use judges to teach programming, and I would be very disappoined if one day all judges break my lab on working with files because some competitive programming enthusiast invented a spec which disallows that.

Don't put words in my mouth. Where did you find this?

Or you have some strict naming conventions for file/class naming, which seems absolutely non-user-friendly and excessive.

Mhm, alright, that was certainly unnecessary, I legit forgot about that part. I'll rewrite that chunk to allow arbitrary names.

By the way, how file would be named, if user types it into the field right in the judge interface?

I don't know, it's your judge, choose the name yourself? Any name works for me if the compiler likes it.

OK, let's move further. Ejudge already have (far more simplified) “strategies”, as described here (Russian only).

Unfortunately you seemingly inserted the URL as an image instead of a link so I can't read it.

Another issue with the whole idea is that while you change current problem.xml file so much that Polygon (as the main source of problems for ALL judges) still has to be rewritten to support all possible problems,

Polygon will have to be rewritten anyway if it's to support any format except what it does now, so that's a weird argument. And even then this seems wrong: Polygon generates packages but can't parse them, and Polygon-generated archives are compatible. So modifications are only requires if features are to be added.

you leave some strange ideas without any objective but for “compatibility”: XML as file format For me, XML is dealbreaker, and the most I'll be able to provide is yet another converter from Polygon archive to our internal package format.

I mean, I'm not a fan of XML either, and it's hard to find anyone who thinks it's any good, but are you proposing to replace it with a modern alternative so that whoever implements the format has to support both XML and the other alternative?

and some ad-hoc solutions (like putting “run-twice” in tags,

That's not my idea; ask ejudge, or perhaps ROI organizers--that's what I found in their run-twice packages.

or having both testsets and groups in the same time).

They're different though. Perhaps it'd be better to have a single testset and add offline option to groups, but is there much difference?

So, while the general idea seems very nice to me (honestly, I tried to did something like yours idea, not as a public standard, but for one judge), now it seems far enough from ready,

...which nevertheless doesn't mean I can't try,

and even after that, I don't believe judges will have a motivation to implement it.

Maybe legacy systems won't rewrite what already works, but that doesn't mean that they won't consult it if they decide on new features, or that modern judges won't appear. I prefer to once build a system that works and use it till the end of life, and I can't be the only one.

P.S. I wouldn't be so sure to advise everyone to run untrusted (for judge) code at ring 2 :)

I mean, no major OS supports that without kernel patches, and those who can write a patch know it's not secure to just patch the LDT. Between you and me, though, do you see any problem with this if segment limits are set to disallow access to kernel memory?