This is not restrictive, because anything can be passed via the test comment.

Hmm, OK, that idea looks good to me. The most important here is to preserve possibility to implement something like group dependencies, but, if we can use test.verdict and modify some global state outside of with test:, it should work.

they seem to think even this is too radical

I think, there are two different directions of changes, which are orthogonal. The first one is functionality (like strategies), and implementation of this one may take months (and possibly may require rewriting everything from the scratch). The second one is syntax sugar (like XML versus anything-modern, deprecation of outdated features, etc) and it's relatively easy to implement (and it's unlikely that some “core” code needs to be modified at all).

Anyway, my point was not about “this is all useless”, but just about “it doesn't seem production-ready for me” (and, in fact, we found some issues, which is good!). I still think the overall idea is great.

If you don't want the problemsetter to list what to save, then where do you store this information?

Now it's hardcoded, as the strategies do.

Supposedly it's not stored in the problem package, but then why can't you store a boolean map

In fact, I still worry that showing all the data (for run-twice, for example) will overload the interface, but OK, some hardcoding will remain here.

Would that work for you?

‌Maybe. In fact, just now I have remembered how it implemented in Ejudge (but I didn't check): problem contains only the tests, and scoring options may be redefined at contest level — so we can use the same problem in different contests under different rules. Maybe such spec should just be moved out of problem description.

Do you need to?

I would be glad if I had such opportunity. Now I think it's possible to make strategy which may (possibly unintentionally) DoS the judge. But I can't make an example from my head.

So, basically, when the first program starts, there's nothing attached to the write end of the pipe and so it's supposed to terminate immediately, perhaps with RE. This might not be the best resolution, but perhaps a warning should be emitted if a pipe is read from but never written to.

I wrote my comment not to receive some ad-hoc solution (which, of course, exist), but to tell that imho it should be stated in spec.

If “f is not being written to” is the sufficient condition to run the first user immediately, then in default interactive strategy user will be run before interactor and in the worst case it will cause RE (because at the moment user starts, nothing writes to its input pipe) and in the best user will consume a bit wall clock limit, which is also not perfect.

And could please you elaborate on "re-testing everything"? I think there's no reason to re-run pretests.

Here I don't understand how it will work with the strategies. Do we have to save some “intermediate state” when online tests end, and restart from the same state (by state I mean current global namespace of Python program with strategy) during offline ones? Or we should run the strategy two times separately for such sets?

Perhaps you misinterpreted the comment.

Yes, exactly, now I realized. Sorry for that.

Do you propose creating a new format, completely unrelated to the existing ones?

Yes. If Polygon will not support such format, you still wouldn't be able to use new features. If it will, the format doesn't matter, they can simply change it.

But also, XML's your dealbreaker, really? I mean, yes, it's bad, but there are so many more important details than what characters you use as field separators

Yes. In fact, I have to maintain a converter from whatever Polygon produce to our internal format, but I definitely will not support problem export in XML (because it requires writing more than one line of code). And it may lead to much manual work for whoever who wants to move problems from one judge to another.

Should we, though?

In fact, I think the answer is obvious if there will be no XML (so, such problems should anyway be converted).

Oh, really, I missed some points from the spec.

I'm the proponent for data preservation: if you can generate stuff and have space to save it, do it.

And it's generally not true (I think that's the reason Codeforces will never show the whole participant output for tests). But, for instance, I'd prefer to save the whole checker output (I know cases when it's more than 1 KB and it's important), and not participant's one. It seems hard for me now to determine “more important” files just from given strategies.

And I don't want to give problemsetters rights to choose that by themselves, because my drives are not infinite, and so I would need to implement resource quotas and etc.

For the participant, standard rules apply: show them the verdict and metrics during the contest and everything else afterwards.

I want it to be more customized. Like show some verdicts during the contest (like how it made on ROI) or show participant a checker output (when it comes to competitive programming, we may give some solution metrics here).

I'm not quite sure what you'd expect from a formal spec.

Something like “Judge should run strategy in environment with a module which provides the following functions:

async def user(stdin: Optional[FileLike], stdout: Optional[FileLike]) -> None

” (here, of course, signature is random, but it doesn't matter for the example)

...Invocation is performed even if the future is not awaited.

OK, I still don't understand: is strategy a some custom Pythonic-like DSL, or it's just a script in Python? Because in the latter case, I don't know how to validate strategies at all (because the thing we're facing is a halting problem).

Lack of awaits makes unclear the moment when execution should be started. So, we may do some deadlocks:

f, g = File(), File()
p = Pipe()
user(stdin=p, stdout=f)
checker(test.input, f, test.answer)
user(stdin=f, stdout=p)

I don't know who will write such, but I also don't know how to run it when we have Turing-complete language. This concrete example can be caught, but we can make it a way more complex.

So far I didn't see anyone present any examples.

E.g. in Yandex Contest, the run can have several scores. It is used at Yandex Cup, when we have several scoreboards at once. And, I'll reply here out of order, because that's connected things:

They're different though. Perhaps it'd be better to have a single testset and add offline option to groups, but is there much difference?

Yes, but still offline is bad :) The only reason for judges have offline groups really “offline” is a lack of computational resources to evaluate all submissions just when they arrived.

However, if we have them, there is no reason to postpone judging process at such tests. So, now we have several scores (visible to participant and visible to jury) instead of re-testing everything.

What more do you need?

Oh, I understand now what seemed me strange at first — the defaults. There is a default for run-twice, but no default for plain text string in answer :) But OK, that's weird issue.

Don't put words in my mouth. Where did you find this?

Right in the part about files:

Note that in this example, we allow the user to read from both stdin and input.txt, and write to boht stdout and output.txt (though not at once unless buffering is disabled). The author believes this is more correct than only supporting file I/O.

Unfortunately you seemingly inserted the URL as an image instead of a link so I can't read it.

Fixed now.

so that whoever implements the format has to support both XML and the other alternative

That whoever just don't need to support XML :)

That's not my idea; ask ejudge, or perhaps ROI organizers--that's what I found in their run-twice packages.

I think it's invention of Polygon owners :) (and I still think it's not late to drop it)

I mean, no major OS supports that without kernel patches, and those who can write a patch know it's not secure to just patch the LDT. Between you and me, though, do you see any problem with this if segment limits are set to disallow access to kernel memory?

Can't think of any problem now, but I wouldn't be so sure about the part “those who can write a patch know” :)

Wait though. I mentioned fixed names in very few places, and only when graders are used. Graders have to access the user code somehow, and doing that without fixed module names is difficult.

I see some spec changed today, but I read it yesterday :) Then it's OK if naming is used only for graders.

Before edits, it was like not only for graders, e.g.:

The user submission MUST have file name solution.cl.

While I think that the spec is mostly finished and I'm publishing it here for better visibility, I'd be glad to listen to your input and apply changes where necessary!

So, if I understand correctly, you consider this strategy as (almost) “production-ready”. But, in fact, I think it lacks some important points (which, as I suspect, would, in turn, require to rewrite at least the whole DSL part).

I agree with geranazavr555 about impossibility of implementing strategies in suggested way.

At least, I didn't see any word about how judging reports should look like (both for jury and for participant): like, which files are exactly taken into account and should be saved for future usage? E.g. we have twice-run problem. Do jury really need all 8 “result” files as an output? And what about content of pipes? I don't think Linux pipe is enough tool to record all content in the pipe, which, in the turn, may be more useful for examining the strategy than something like first interactor stderr.

Provided idea of Python strategy code (with unfortunately no formal spec to examine) isn't clear about when execution of interactive problems start: there is just no await of anything. It seems, we should use smth like asyncio.gather with some undefined behaviour on edge cases: e.g. events like “one process sends data, and another one quit” may lead to different results depending on which process is interactor, and which is solution.

Also, as it was already told, some judges (e.g. Yandex Contest) have features, which are not covered by the spec. I'm almost sure nobody will wait some committee to approve the changes to standard before implementing needed feature. And in some time we'll get again to what we have now — several judges have done the same thing different ways. While it's possible to standardize such features later, it may be impossible to convert all “old-style” problems to a new format.

By the way, this spec is too restrictive. Just a few examples: you write, that you restrict file-only I/O just because you personally think it's good. It may be good for regular Codeforces round, but not for educational contests — I use judges to teach programming, and I would be very disappoined if one day all judges break my lab on working with files because some competitive programming enthusiast invented a spec which disallows that. Or you have some strict naming conventions for file/class naming, which seems absolutely non-user-friendly and excessive. By the way, how file would be named, if user types it into the field right in the judge interface?

OK, let's move further. Ejudge already have (far more simplified) “strategies”, as described here (Russian only). While I don't think this design is great, it requires “proxy strategy” to be implemented, and there is no obvious way except to save such proxy binary in each problem package (as testlib.h now in some judges).

Another issue with the whole idea is that while you change current problem.xml file so much that Polygon (as the main source of problems for ALL judges) still has to be rewritten to support all possible problems, you leave some strange ideas without any objective but for “compatibility”: XML as file format and some ad-hoc solutions (like putting “run-twice” in tags, or having both testsets and groups in the same time). For me, XML is dealbreaker, and the most I'll be able to provide is yet another converter from Polygon archive to our internal package format.

So, while the general idea seems very nice to me (honestly, I tried to did something like yours idea, not as a public standard, but for one judge), now it seems far enough from ready, and even after that, I don't believe judges will have a motivation to implement it.

P.S. I wouldn't be so sure to advise everyone to run untrusted (for judge) code at ring 2 :)

"All teams advanced to Round 3 and TOP-50 participants of online-mirror of Round 3 will get t-shirts."

(Translation from Russian)

I think that you don't need to print N (see example, line 4) to stdout (maybe you need to print it to log file).

UPD: Fixed.