November 2018 - DOMjudge-devel

Automatic testing of jury solutions
by Maarten Sijm 29 Nov '18

29 Nov '18

Hi, I am planning to set up automatic testing of jury solutions using continuous integration. >From the admin manual (section 4.7), I've found that testing jury solutions can be done by uploading the problems to the system, but I would like to circumvent this manual task. I already have a naive shell script set up that does some basic diffing, which works for simple problems. However, I thought there might be already some tool that automatically verifies solutions. I've found that Kattis has a tool to verify problems: https://github.com/Kattis/problemtools However, I can imagine that this `verifyproblem` tool uses different settings than what would be used on a DomJudge instance. Could I get some advice on how I could best set this up? Kind regards, M.P. Sijm

2 1

Strange Behavior
by Marc Furon 10 Nov '18

10 Nov '18

I am getting ready for our contest tomorrow. Thanks to Keith, I have Apache and php-fpm configured. I tried installing the domjudge.conf file for php-fpm but it looks like my system is using the default www.conf configuration. However, as I test submitting here at the site, it's not working. I log in as a user, get the initial screen (as part of the "demo" contest), click Submit, select my source code, problem, and language. I'm asked to confirm if I really want to submit. I then go right back to the same screen. There is no entry on the screen for the submission. Nothing happens on the judgehost (which did register, and is waiting for work). Where can I check to see what is going on? I'm also trying to set up IRC but I've not used it before. Marc

1 0

Re: Apache/DOMjudge Tuning
by Keith Johnson 10 Nov '18

10 Nov '18

Probably I've broken the mailing list thread, but I wanted to reply to you before its too late. > In the interest of avoiding the issue we had last year we want to > ensure we have plenty of web service resources available for DOMjudge > this weekend. I have installed DOMjudge 6.0.2 and it seems to be > working. > > When I look at the current processes running (with the default > configuration) I see five httpd processes and five php-fpm processes. > From what I can tell, it looks like I should use php-fpm tuning > parameters to greatly increase the number of php-fpm processes > running. (We have 16 GB of RAM, so I see no problem with doing so.) > > Does the DOMjudge team have any recommendations on which parameters > to change to support 100 teams? Am I looking in the right place to > ensure that we don't run out of web service resources again? 100 teams should be pretty easy to support on hardware with those specifications. For PHP-FPM, there are a couple of relevant parameters to tune based on your system. The first is pm = static. This disables scaling of fpm workers so there are always a fixed number of workers available. To control how many fixed workers, that's the pm.max_children variable. It looks like the domjudge-fpm.conf file that ships with domjudge seems to set these, and gives some guidance for how to set it, but I think the comment might be a little bit out of date since the move to symfony which requires a bit more memory. Running domjudge on master a few weeks back, I seem to recall 100-150mb per fpm worker on average. So the value you choose for max_children should be based on this. E.g. if you want to use 10gb of your server memory then you'd want to set it to around 70(which is probably plenty for 100 teams). You should also make sure that pm.max_requests is set to some reasonably large non-zero value. This just protects you from unknown memory leaks by restarting fpm worker processes after some number of requests(I use 5000). I've never used apache in front of php-fpm, but you may want to check the MaxRequestWorkers/KeepAliveTimeout/KeepAlive setting. If you are on a local network/not using ssl probably you can just disable keepalive to ensure you don't run into the errors encountered last year. The best thing you can do though is to do some load testing to see what happens under load. You can probably accomplish this with just apachebench, but make sure you point it at a real page(i.e. something that isn't just a 302 redirect or a static file or something like that). A good page is probably the login page; do a curl -I on it first to make sure it returns a 200 OK + some actual content. For 100 teams you're probably only looking at like 10 requests/second tops(team pages refresh every 30 seconds, which is like 3 reqs/sec. Add in some spectators and people who open multiple tabs and that gets you to 10ish). That said your server should be able to do way more requests/sec than that. An apachebench command that would simulate using keepalive + 150 concurrent users would be something like this: ab -k -c 150 -n 15000 example.com/domjudge/login In the results you'll want to pay attention to a few things: The 99th/all percentiles at the bottom, and the max time for each. You'd want to see something like 95% of requests less than a couple hundred ms, and 100% of requests under 1-2 seconds or so. I think there is also a count for errors(if it happens to blow up), but you want that to be zero. Play with the values for the -c argument to test more/less concurrent "users" so you can get a good idea for what your server can handle(I would keep increasing this until you discover at what point it starts to fail). You might also want to watch memory usage of the php-fpm workers and tune the pm.max_children values as needed. An aside, I don’t run apache(opting for nginx instead), but I try to configure it to set caching headers for css/png/jpg/etc, as this will reduce the load on the server. Make sure you’re all set with customizations as it’ll be difficult to css/js after the fact. You can skip this step if you’d like, 100 teams is not that many and a properly configured server should be able to easily serve that many requests. The judge hosts check in periodically, so make sure you account for them in your load testing. More judge hosts = more requests making it to the domjudge server. Sometimes substantially so, e.g. in the case of a problem with ~100 test cases and a tiny runtime you’ll probably see multiple requests per second just to judge that. So if you have lots of judge hosts that can negatively impact the performance of your web server. I hope that rambling is helpful in some way and provides you some useful tuning options. -Keith

2 4

ICPC Import Error
by Marc Furon 09 Nov '18

09 Nov '18

I just tried to import the team information from icpc.global. I generated a web token, went to the import screen, and entered the Southern-California-2018 contest and token. When I went to import the teams I received this error: error: Error decoding JSON data '<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"><meta name="theme-color" content="#000000"><link rel="manifest" href="/manifest.json"><link rel="shortcut icon" href="/favicon.ico"><link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700" rel="stylesheet"><link href="https://fonts.googleapis.com/css?family=Playfair+Display:400,700" rel="stylesheet"><title>The ICPC International Collegiate Programming Contest</title><script src="https://www.google.com/recaptcha/api.js" async defer="defer"></script><link href="/static/css/main.f25ca146.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script type="text/javascript" src="/static/js/main.ee9a0359.js"></script></body></html>': Syntax error I do not know if this is a DOMjudge problem, or an icpc.global problem. I'll fall back to importing from files. Marc

3 7

Apache/DOMjudge Tuning
by Marc Furon 08 Nov '18

08 Nov '18

In the interest of avoiding the issue we had last year we want to ensure we have plenty of web service resources available for DOMjudge this weekend. I have installed DOMjudge 6.0.2 and it seems to be working. When I look at the current processes running (with the default configuration) I see five httpd processes and five php-fpm processes. From what I can tell, it looks like I should use php-fpm tuning parameters to greatly increase the number of php-fpm processes running. (We have 16 GB of RAM, so I see no problem with doing so.) Does the DOMjudge team have any recommendations on which parameters to change to support 100 teams? Am I looking in the right place to ensure that we don't run out of web service resources again? Marc

1 0

Login Screen?
by Marc Furon 05 Nov '18

05 Nov '18

I am setting up 6.0.2 in preparation for Southern California's contest next weekend. We run on Fedora, in this case Fedora 28. I installed DOMjudge to /opt/domjudge, configured MariaDB, and copied the Apache configuration file to /etc/httpd/conf.d. I can connect to localhost/domjudge and I see the default scoreboard. However, when I click "Login", I get a blank screen! As far as I can tell everything is there. Does anyone have any ideas about what might be going on? Marc

2 4

judgehost API changes to do performance improvements
by Tobias Werth 01 Nov '18

01 Nov '18

Hi, Since I have noticed that we are doing very poorly for problems with many test cases, I have looked into what causes how much of the slowdown (compared to verifyproblem it's 50x for the insane WF problem). The least invasive improvement we can do is in making some changes to the judgehost API. I would like to do those changes before we make the next release aka before NWERC so that we don't (have to) change the API in the next release again. Currently the API requests for one judging look like: API request POST judgehosts/next-judging/bar-0 *Judging submission s92 (endpoint default) (t1/p9/cpp), id j257...* API request GET config data=name=script_timelimit API request GET config data=name=script_memory_limit API request GET config data=name=script_filesize_limit API request GET config data=name=process_limit API request GET config data=name=output_storage_limit *^-- above requests could be grouped together already today, but that doesn't bring a big win (I'll do it anyway)* Working directory: /home/sitowert/domjudge/output/judgings/bar-0/endpoint-default/c2-s92-j257 API request GET contests/2/submissions/92/source-code API request PUT judgehosts/update-judging/bar-0/257 data=compile_success=1&output_compile=a25pZ2h0REsuY2M6IEluIGZ1bmN0aW9uICdpbnQgbWFpbigpJzoKa25pZ2h0REsuY2M... executing chroot script: 'chroot-startstop.sh start' API request GET config data=name=timelimit_overshoot *^--- could also be grouped to the config requests above* API request GET testcases/next-to-judge/257 Running testcase 1... API request POST judgehosts/add-judging-run/bar-0/257 data=testcaseid=94&runresult=correct&runtime=0.001&output_run=MTIK&output_error=&output_system=Q29ycmVjdC... Testcase 1 done, result: correct API request GET testcases/next-to-judge/257 Running testcase 2... API request POST judgehosts/add-judging-run/bar-0/257 data=testcaseid=95&runresult=correct&runtime=0.001&output_run=Ngo%3D&output_error=&output_system=Q29ycmVj... Testcase 2 done, result: correct API request GET testcases/next-to-judge/257 Running testcase 3... API request GET testcases/95/file/input API request GET testcases/95/file/output *^--- this is only done for unknown/changed testdata* *-----------------------------------------------------------------------------------------------------------* In the future, I want to have it rather like this flow: API request POST judgehosts/next-judging/bar-0 *^--- this will additionally return all the md5sums for all inputs/outputs of the problem* *Judging submission s92 (endpoint default) (t1/p9/cpp), id j257...* API request GET config data=name=script_timelimit,script_memory_limit,script_filesize_limit...timelimit_overshoot Working directory: /home/sitowert/domjudge/output/judgings/bar-0/endpoint-default/c2-s92-j257 API request GET contests/2/submissions/92/source-code API request PUT judgehosts/update-judging/bar-0/257 data=compile_success=1&output_compile=a25pZ2h0REsuY2M6IEluIGZ1bmN0aW9uICdpbnQgbWFpbigpJzoKa25pZ2h0REsuY2M... executing chroot script: 'chroot-startstop.sh start' API request GET testcases/95/file/input API request GET testcases/95/file/output *^--- request all unknown/changed testdata* Running testcase 1... Testcase 1 done, result: correct Running testcase 2... Testcase 2 done, result: correct Running testcase 3... Testcase 3 done, result: wrong-answer API request POST judgehosts/add-judging-run/bar-0/257 data=TBD *^--- this is actually the result of three testcases* API request GET testcases/next-to-judge/257 *^--- this is the check whether we should continue judging* *-----------------------------------------------------------------------------------------------------------* The main idea is to not do the next-to-judge calls (as long as possible) and to group the add-judging-run calls. The next-to-judge calls are not necessary as long as all previous test cases are correct. For the most commonly used judging model (fail on first error) this is true for at least N-1 of N test cases. Now we still want to ping back from time to time to a) signal progress, and b) not to post too much data back in the database at once. So I could imagine reasonable default values is to post back at least every 10s and if we accumulated more than twice the output limit but that should be configurable. For the actual grouping/batching I can see two options: either allow the one POST add-judging-run call to allow multiple results to be posted or follow something like this: https://developers.facebook.com/docs/graph-api/making-multiple-requests to do batched requests across the API. For a problem with N test cases and M unknown test cases, this approach will bring us down from 9 + N*2 + M*2 API calls to 4 + M*2 + X/10 + 2 API calls (<-- in the best case of a correct problem of course with X seconds of total judging time) Any objections or thoughts? Tobi

2 2