- DOMjudge-devel - domjudge.org

Re: Apache/DOMjudge Tuning
by Keith Johnson 10 Nov '18

10 Nov '18

Probably I've broken the mailing list thread, but I wanted to reply to you before its too late. > In the interest of avoiding the issue we had last year we want to > ensure we have plenty of web service resources available for DOMjudge > this weekend. I have installed DOMjudge 6.0.2 and it seems to be > working. > > When I look at the current processes running (with the default > configuration) I see five httpd processes and five php-fpm processes. > From what I can tell, it looks like I should use php-fpm tuning > parameters to greatly increase the number of php-fpm processes > running. (We have 16 GB of RAM, so I see no problem with doing so.) > > Does the DOMjudge team have any recommendations on which parameters > to change to support 100 teams? Am I looking in the right place to > ensure that we don't run out of web service resources again? 100 teams should be pretty easy to support on hardware with those specifications. For PHP-FPM, there are a couple of relevant parameters to tune based on your system. The first is pm = static. This disables scaling of fpm workers so there are always a fixed number of workers available. To control how many fixed workers, that's the pm.max_children variable. It looks like the domjudge-fpm.conf file that ships with domjudge seems to set these, and gives some guidance for how to set it, but I think the comment might be a little bit out of date since the move to symfony which requires a bit more memory. Running domjudge on master a few weeks back, I seem to recall 100-150mb per fpm worker on average. So the value you choose for max_children should be based on this. E.g. if you want to use 10gb of your server memory then you'd want to set it to around 70(which is probably plenty for 100 teams). You should also make sure that pm.max_requests is set to some reasonably large non-zero value. This just protects you from unknown memory leaks by restarting fpm worker processes after some number of requests(I use 5000). I've never used apache in front of php-fpm, but you may want to check the MaxRequestWorkers/KeepAliveTimeout/KeepAlive setting. If you are on a local network/not using ssl probably you can just disable keepalive to ensure you don't run into the errors encountered last year. The best thing you can do though is to do some load testing to see what happens under load. You can probably accomplish this with just apachebench, but make sure you point it at a real page(i.e. something that isn't just a 302 redirect or a static file or something like that). A good page is probably the login page; do a curl -I on it first to make sure it returns a 200 OK + some actual content. For 100 teams you're probably only looking at like 10 requests/second tops(team pages refresh every 30 seconds, which is like 3 reqs/sec. Add in some spectators and people who open multiple tabs and that gets you to 10ish). That said your server should be able to do way more requests/sec than that. An apachebench command that would simulate using keepalive + 150 concurrent users would be something like this: ab -k -c 150 -n 15000 example.com/domjudge/login In the results you'll want to pay attention to a few things: The 99th/all percentiles at the bottom, and the max time for each. You'd want to see something like 95% of requests less than a couple hundred ms, and 100% of requests under 1-2 seconds or so. I think there is also a count for errors(if it happens to blow up), but you want that to be zero. Play with the values for the -c argument to test more/less concurrent "users" so you can get a good idea for what your server can handle(I would keep increasing this until you discover at what point it starts to fail). You might also want to watch memory usage of the php-fpm workers and tune the pm.max_children values as needed. An aside, I don’t run apache(opting for nginx instead), but I try to configure it to set caching headers for css/png/jpg/etc, as this will reduce the load on the server. Make sure you’re all set with customizations as it’ll be difficult to css/js after the fact. You can skip this step if you’d like, 100 teams is not that many and a properly configured server should be able to easily serve that many requests. The judge hosts check in periodically, so make sure you account for them in your load testing. More judge hosts = more requests making it to the domjudge server. Sometimes substantially so, e.g. in the case of a problem with ~100 test cases and a tiny runtime you’ll probably see multiple requests per second just to judge that. So if you have lots of judge hosts that can negatively impact the performance of your web server. I hope that rambling is helpful in some way and provides you some useful tuning options. -Keith

2 4

ICPC Import Error
by Marc Furon 09 Nov '18

09 Nov '18

I just tried to import the team information from icpc.global. I generated a web token, went to the import screen, and entered the Southern-California-2018 contest and token. When I went to import the teams I received this error: error: Error decoding JSON data '<!doctype html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1,shrink-to-fit=no"><meta name="theme-color" content="#000000"><link rel="manifest" href="/manifest.json"><link rel="shortcut icon" href="/favicon.ico"><link href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700" rel="stylesheet"><link href="https://fonts.googleapis.com/css?family=Playfair+Display:400,700" rel="stylesheet"><title>The ICPC International Collegiate Programming Contest</title><script src="https://www.google.com/recaptcha/api.js" async defer="defer"></script><link href="/static/css/main.f25ca146.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div><script type="text/javascript" src="/static/js/main.ee9a0359.js"></script></body></html>': Syntax error I do not know if this is a DOMjudge problem, or an icpc.global problem. I'll fall back to importing from files. Marc

3 7

Apache/DOMjudge Tuning
by Marc Furon 08 Nov '18

08 Nov '18

In the interest of avoiding the issue we had last year we want to ensure we have plenty of web service resources available for DOMjudge this weekend. I have installed DOMjudge 6.0.2 and it seems to be working. When I look at the current processes running (with the default configuration) I see five httpd processes and five php-fpm processes. From what I can tell, it looks like I should use php-fpm tuning parameters to greatly increase the number of php-fpm processes running. (We have 16 GB of RAM, so I see no problem with doing so.) Does the DOMjudge team have any recommendations on which parameters to change to support 100 teams? Am I looking in the right place to ensure that we don't run out of web service resources again? Marc

1 0

Login Screen?
by Marc Furon 05 Nov '18

05 Nov '18

I am setting up 6.0.2 in preparation for Southern California's contest next weekend. We run on Fedora, in this case Fedora 28. I installed DOMjudge to /opt/domjudge, configured MariaDB, and copied the Apache configuration file to /etc/httpd/conf.d. I can connect to localhost/domjudge and I see the default scoreboard. However, when I click "Login", I get a blank screen! As far as I can tell everything is there. Does anyone have any ideas about what might be going on? Marc

2 4

judgehost API changes to do performance improvements
by Tobias Werth 01 Nov '18

01 Nov '18

Hi, Since I have noticed that we are doing very poorly for problems with many test cases, I have looked into what causes how much of the slowdown (compared to verifyproblem it's 50x for the insane WF problem). The least invasive improvement we can do is in making some changes to the judgehost API. I would like to do those changes before we make the next release aka before NWERC so that we don't (have to) change the API in the next release again. Currently the API requests for one judging look like: API request POST judgehosts/next-judging/bar-0 *Judging submission s92 (endpoint default) (t1/p9/cpp), id j257...* API request GET config data=name=script_timelimit API request GET config data=name=script_memory_limit API request GET config data=name=script_filesize_limit API request GET config data=name=process_limit API request GET config data=name=output_storage_limit *^-- above requests could be grouped together already today, but that doesn't bring a big win (I'll do it anyway)* Working directory: /home/sitowert/domjudge/output/judgings/bar-0/endpoint-default/c2-s92-j257 API request GET contests/2/submissions/92/source-code API request PUT judgehosts/update-judging/bar-0/257 data=compile_success=1&output_compile=a25pZ2h0REsuY2M6IEluIGZ1bmN0aW9uICdpbnQgbWFpbigpJzoKa25pZ2h0REsuY2M... executing chroot script: 'chroot-startstop.sh start' API request GET config data=name=timelimit_overshoot *^--- could also be grouped to the config requests above* API request GET testcases/next-to-judge/257 Running testcase 1... API request POST judgehosts/add-judging-run/bar-0/257 data=testcaseid=94&runresult=correct&runtime=0.001&output_run=MTIK&output_error=&output_system=Q29ycmVjdC... Testcase 1 done, result: correct API request GET testcases/next-to-judge/257 Running testcase 2... API request POST judgehosts/add-judging-run/bar-0/257 data=testcaseid=95&runresult=correct&runtime=0.001&output_run=Ngo%3D&output_error=&output_system=Q29ycmVj... Testcase 2 done, result: correct API request GET testcases/next-to-judge/257 Running testcase 3... API request GET testcases/95/file/input API request GET testcases/95/file/output *^--- this is only done for unknown/changed testdata* *-----------------------------------------------------------------------------------------------------------* In the future, I want to have it rather like this flow: API request POST judgehosts/next-judging/bar-0 *^--- this will additionally return all the md5sums for all inputs/outputs of the problem* *Judging submission s92 (endpoint default) (t1/p9/cpp), id j257...* API request GET config data=name=script_timelimit,script_memory_limit,script_filesize_limit...timelimit_overshoot Working directory: /home/sitowert/domjudge/output/judgings/bar-0/endpoint-default/c2-s92-j257 API request GET contests/2/submissions/92/source-code API request PUT judgehosts/update-judging/bar-0/257 data=compile_success=1&output_compile=a25pZ2h0REsuY2M6IEluIGZ1bmN0aW9uICdpbnQgbWFpbigpJzoKa25pZ2h0REsuY2M... executing chroot script: 'chroot-startstop.sh start' API request GET testcases/95/file/input API request GET testcases/95/file/output *^--- request all unknown/changed testdata* Running testcase 1... Testcase 1 done, result: correct Running testcase 2... Testcase 2 done, result: correct Running testcase 3... Testcase 3 done, result: wrong-answer API request POST judgehosts/add-judging-run/bar-0/257 data=TBD *^--- this is actually the result of three testcases* API request GET testcases/next-to-judge/257 *^--- this is the check whether we should continue judging* *-----------------------------------------------------------------------------------------------------------* The main idea is to not do the next-to-judge calls (as long as possible) and to group the add-judging-run calls. The next-to-judge calls are not necessary as long as all previous test cases are correct. For the most commonly used judging model (fail on first error) this is true for at least N-1 of N test cases. Now we still want to ping back from time to time to a) signal progress, and b) not to post too much data back in the database at once. So I could imagine reasonable default values is to post back at least every 10s and if we accumulated more than twice the output limit but that should be configurable. For the actual grouping/batching I can see two options: either allow the one POST add-judging-run call to allow multiple results to be posted or follow something like this: https://developers.facebook.com/docs/graph-api/making-multiple-requests to do batched requests across the API. For a problem with N test cases and M unknown test cases, this approach will bring us down from 9 + N*2 + M*2 API calls to 4 + M*2 + X/10 + 2 API calls (<-- in the best case of a correct problem of course with X seconds of total judging time) Any objections or thoughts? Tobi

2 2

Domjudge
by Marcel Bezdrighin 12 Oct '18

12 Oct '18

After installing the new 6.0 domjudge the same way as I did 5.3.3, I keep getting an 404 error when trying to access the main page: The requested URL /home/marcel/domjudge/domserver/webapp/web/app.php was not found on this server. 5.3.3 is working fine if I follow the same steps. The directory exists and the file “app.php” is also present. Moreover if I try to access “app.php” directly then I’m redirected to /public/ but then I get exactly the same error. It also seems the static files are working fine because I can access them easily. This seems to be a problem with setting up apache and how Symfony works, but I’m not very good at this stuff so I couldn’t solve it.

2 1

Time Factor Not Getting Accounted when calculating maxruntime
by Abhishek Kumar 30 Aug '18

30 Aug '18

Hi Team, I was running Scala on Domjudge version 5.1 but the compiler gave timelimit execption.So i increased the timefactor in language entry but still it gave timelimit exception When i further looked into it, it was always passing maxruntime as 1 in every case since the below sql query always returned null *File Path*(domserver/www/api/index.php) *Sql Query*: SELECT s.submitid, s.cid, s.teamid, s.probid, s.langid, s.rejudgingid, CEILING(time_factor*timelimit) AS maxruntime, p.memlimit, p.outputlimit, special_run AS run, special_compare AS compare, special_compare_args AS compare_args, compile_script FROM submission s LEFT JOIN problem p USING (probid) LEFT JOIN language l USING (langid) WHERE submitid = 5326 So i need to know how can i change this .Since i want my time factor get accounted when calculating maxruntime Eagerly waiting for your reply Thanks Abhishek

2 1

Problem in judgedeamon script
by Guillaume MILAN 03 Aug '18

03 Aug '18

Hello, I am trying to run on a fresh installed domjudge (maintainer conf) server. After launching the ./bin/judgedeamon I got the following crash when the first submission is made for a contest. [Aug 03 15:25:02.335] judgedaemon[16141]: Running compile.sh caused a runguard error/crash: 'runguard error - 50029 - creating cgroup' And after the participants can't submit any more solutions. Did you already face this problem? I have seen the issue:* https://github.com/DOMjudge/domjudge-packaging/issues/11 <https://github.com/DOMjudge/domjudge-packaging/issues/11>* But I didn't understood how they manage to fix it. Is there a way to reinstall all the cgroup from scratch? Guillaume

2 1

Additional Flags for runguard
by Sarthak Garg 28 Jul '18

28 Jul '18

What all changes do i have to make if I wish to add an additional flag while executing runguard. i want to add a new flag for network access: net_flag ./run testdata.in program.out sudo -n /opt/domjudge/judgehost/bin/runguard -r /opt/data/30000309/codeTest_2202/30000309271547/622/c1-s2-j301-27-50/testcase187783/.. --nproc=640 --no-core --streamsize=40000 --user=domjudge-run --walltime=1:1.1 --cputime=1:1.1 --filesize=40000 --stderr=program.err --outmeta=program.meta -- /testcase187783/execdir/program

2 2

DOMjudge hackathon(s)
by Nicky Gerritsen 28 Jul '18

28 Jul '18

Hi all, As the programming contest season is rapidly approaching, we would like to make sure we have a stable version of DOMjudge which we can release. Our main goal for the next release (which will be 6.0) currently is to incorporate all changes we made during last season, especially during and on the way to the world finals. There is a separate icpc-live branch we desperately want to get rid of, but it still requires some work, for example to create config flags to put some features “behind”. Furthermore it would probably be nice to continue with the new looks. Because our previous hackathons resulted in a lot of completed work in a small amount of time, we would like to organise another one (or two) during the summer. If you want to join (everyone is welcome, even if you never coded on DOMjudge before), please fill in this Doodle poll: https://doodle.com/poll/rwuuasewi7k4zrwe <https://doodle.com/poll/rwuuasewi7k4zrwe> As most of the core developers are in the Europe/Amsterdam timezone, we will probably plan a whole day (somewhere between 10:00 and 20:00 CEST), but you are always welcome to join for shorter periods. Some of us might meet up in real life, but of course you are welcome to join “online”. If you have any more questions feel free to contact us! On behalf of the DOMjudge team, Best, Nicky

1 2