Adding contest data in bulk
===========================

DOMjudge offers three ways to add or update contest data in bulk: using API
endpoints, using the CLI (which wraps the API) and using the jury interface.
In general, we follow the `CCS specification`_ for all file formats involved.

For using the API, the following examples require you to set up admin credentials
in your `.netrc`_ file. You need to install `httpie`_ and replace the
``<API_URL>`` in the examples below with the API URL of your local DOMjudge
installation.

To use the CLI, you need to replace ``<WEBAPP_DIR>`` with the path to
the ``webapp`` directory of the DOMserver.

.. _external-ids:

External ID's
-------------

Most entities in DOMjudge have an `eternal ID`. External ID's are used to link
entities in DOMjudge to entities in an external system, e.g. the ICPC CMS. The
API uses the external ID's to expose entities to other systems. When you create
an entity in DOMjudge, specifying the external ID is optional; DOMjudge will
use an automatically generated ID if you don't provide one. However, if this ID
is not unique, you will get a message telling you that you need to provide your
own external ID.

When importing entities in bulk as described below, the external ID will be
populated with the ID as specified in the files you import.

Importing team categories
-------------------------

There are two formats to import team categories: a JSON format and a legacy TSV format.

Using JSON
^^^^^^^^^^

Prepare a file called ``groups.json`` which contains the team categories.
It should be a JSON array with objects, each object should contain the following
fields:

- ``id``: the category ID to use. Must be unique
- ``icpc_id`` (optional): an ID from an external system, e.g. from the ICPC CMS, may be empty
- ``name``: the name of the team category as shown on the scoreboard
- ``hidden`` (defaults to ``false``): if ``true``, teams in this category will
  not be shown on the scoreboard
- ``sortorder`` (defaults to ``0``): the sort order of the team category to use
  on the scoreboard. Categories with the same sortorder will be grouped together.

The ``id`` field will be the ID used for the group.

Example ``groups.json``::

  [{
    "id": "13337",
    "icpc_id": "123",
    "name": "Companies",
    "hidden": true
  }, {
    "id": "47",
    "name": "Participants"
  }, {
    "id": "23",
    "name": "Spectators"
  }]

To import the file using the jury interface, go to `Import / export`, select
`groups` under `Import JSON / YAML`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/groups" json@groups.json

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f json=groups.json users/groups

Using the legacy TSV format
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Prepare a file called ``groups.tsv`` which contains the team categories.
The first line should contain ``File_Version 1`` (tab-separated).
Each of the following lines must contain the following elements separated by tabs:

- the category ID. Must be unique
- the name of the team category as shown on the scoreboard

The ``id`` field will be the ID used for the group.

Example ``groups.tsv``::

   File_Version   1
   13337	Companies
   47	Participants
   23	Spectators

To import the file using the jury interface, go to `Import / export`, select
`groups` under `Tab-separated import`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/groups" tsv@groups.tsv

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f tsv=groups.tsv users/groups


Importing team affiliations
---------------------------

.. note::

    The team TSV import automatically imports team affiliations as well.

Prepare a file called ``organizations.json`` which contains the affiliations.
It should be a JSON array with objects, each object should contain the following
fields:

- ``id``: the affiliation ID. Must be unique
- ``icpc_id`` (optional): an ID from an external system, e.g. from the ICPC CMS, may be empty
- ``name``: the affiliation short name as used in the jury interface and certain
  exports
- ``formal_name``: the affiliation name as used on the scoreboard
- ``country``: the country code in form of ISO 3166-1 alpha-3

The ``id`` field will be the ID used for the affiliation.

Example ``organizations.json``::

  [{
    "id": "INST-42",
    "icpc_id": "42",
    "name": "LU",
    "formal_name": "Lund University",
    "country": "SWE"
  }, {
    "id": "INST-43",
    "icpc_id": "43",
    "name": "FAU",
    "formal_name": "Friedrich-Alexander-University Erlangen-Nuremberg",
    "country": "DEU"
  }]

To import the file using the jury interface, go to `Import / export`, select
`organizations` under `Import JSON / YAML`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/organizations" json@organizations.json

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f json=organizations.json users/organizations

Importing teams
---------------

There are two formats to import teams: a JSON format and a legacy TSV format.

Using JSON
^^^^^^^^^^

Prepare a file called ``teams.json`` which contains the teams.
It should be a JSON array with objects, each object should contain the following
fields:

- ``id``: the team ID. Must be unique
- ``icpc_id`` (optional): an ID from an external system, e.g. from the ICPC CMS, may be empty
- ``label``: (optional): the label, for example the seat number
- ``group_ids``: an array with one element: the category ID this team belongs to
- ``name``: the team name as used in the web interface
- ``members`` (optional): Members of the team as one long string
- ``display_name`` (optional): the team display name. If provided, will display
  this instead of the team name in certain places, like the scoreboard
- ``organization_id``: the ID of the team affiliation this team belongs to
- ``location.description`` (optional): the location of the team

The ``id`` field will be the ID used for the team and the ``group_ids`` and ``organization_id``
fields are the values as provided during the import of the other files listed above.

Example ``teams.json``::

  [{
    "id": "1",
    "icpc_id": "447047",
    "label": "1",
    "group_ids": ["24"],
    "name": "¡i¡i¡",
    "organization_id": "INST-42",
    "location": {"description": "AUD 10"}
  }, {
    "id": "2",
    "icpc_id": "447837",
    "label": "2",
    "group_ids": ["25"],
    "name": "Pleading not FAUlty",
    "organization_id": "INST-43"
  }]

To import the file using the jury interface, go to `Import / export`, select
`teams` under `Import JSON / YAML`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/teams" json@teams.json

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f json=teams.json users/teams


Using the legacy TSV format
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Prepare a file called ``teams2.tsv`` which contains the teams.
The first line should contain ``File_Version	2`` (tab-separated).
Each of the following lines must contain the following elements separated by tabs:

- the team ID. Must be unique
- an ID from an external system, e.g. from the ICPC CMS, may be empty
- the category ID this team belongs to
- the team name as used in the web interface
- the institution name as used on the scoreboard
- the institution short name as used in the jury interface and certain exports
- a country code in form of ISO 3166-1 alpha-3
- an external institution ID, e.g. from the ICPC CMS, may be empty

The team `id` field will be the ID used for the team and the category ID field is the value
as provided during the import of the other files listed above.

Example ``teams2.tsv``::

   File_Version   2
   1	447047	24	¡i¡i¡	Lund University	LU	SWE	INST-42
   2	447837	25	Pleading not FAUlty	Friedrich-Alexander-University Erlangen-Nuremberg	FAU	DEU	INST-43


To import the file using the jury interface, go to `Import / export`, select
`teams` under `Tab-separated import`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/teams" tsv@teams2.tsv

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f tsv=teams2.tsv users/teams

Importing accounts
------------------

There are two formats to import accounts: a YAML format and a legacy TSV format.

Using YAML
^^^^^^^^^^

Prepare a file called ``accounts.yaml`` which contains the accounts.
It should be a YAML array with objects, each object should contain the following
fields:

- ``id``: the account ID. Must be unique
- ``username``: the account username. Must be unique
- ``password``: the password to use for the account
- ``type``: the user type, one of ``team``, ``judge``, ``admin`` or ``balloon``, ``jury`` will be interpret as ``judge``
- ``team_id``: (optional) the ID of the team this account belongs to
- ``name``: (optional) the full name of the account
- ``ip`` (optional): IP address to link to this account

The ``id`` field will be the ID used for the user and the ``team_id`` field is the value as provided during
the team import.

Example ``accounts.yaml``::

   - id: team001
     username: team001
     password: P3xm33imve
     type: team
     name: team001
     ip: 10.10.2.1

   - id: team002
     username: team002
     password: qd4WHeJXbd
     type: team
     name: team002
     ip: 10.10.2.2

   - id: john
     username: john
     password: Uf4PYRA7mJ
     type: judge
     name: John Doe

.. note::

    You can also use a JSON file instead of YAML. Make sure to name it
    ``accounts.json`` in that case.

To import the file using the jury interface, go to `Import / export`, select
`accounts` under `Import JSON / YAML`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/accounts" yaml@accounts.yaml

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f yaml=accounts.yaml users/accounts


Using the legacy TSV format
^^^^^^^^^^^^^^^^^^^^^^^^^^^

Prepare a file called ``accounts.tsv`` which contains the team credentials.
The first line should contain ``accounts  1`` (tab-separated).
Each of the following lines must contain the following elements separated by tabs:

- the user type, one of ``team``, ``judge``, ``admin`` or ``balloon``, ``jury`` will be interpret as ``judge``
- the full name of the user
- the username
- the password
- (optional) the IP address to the user

Example ``accounts.tsv``::

   accounts	1
   team	team001	team001	P3xm33imve	10.10.2.1
   team	team002	team002	qd4WHeJXbd	10.10.2.2
   judge	John Doe	john	Uf4PYRA7mJ

To import the file using the jury interface, go to `Import / export`, select
`accounts` under `Tab-separated import`, select your file and click `Import`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/users/accounts" tsv@accounts.tsv

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f tsv=accounts.tsv users/accounts

Importing contest metadata
--------------------------

Prepare a file called ``contest.yaml`` which contains the contest information.

Example ``contest.yaml``::

   id:                         practice
   name:                       DOMjudge open practice session
   start_time:                 2020-04-30T10:00:00+01:00
   duration:                   2:00:00
   scoreboard_freeze_duration: 0:30:00
   penalty_time:               20

.. note::

    You can also use a JSON file instead of YAML. Make sure to name it
    ``contest.json`` in that case.

To import the file using the jury interface, go to `Import / export`, then
`Contest -> Import JSON / YAML`, select your file under `File`
and click `Import`.

To import the file using the API run the following commands::

    http --check-status -b -f POST "<API_URL>/contests" yaml@contest.yaml

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f yaml=contest.yaml contests

This call returns the new contest ID, which you need to import problems.

Importing problem metadata
--------------------------

Prepare a file called ``problems.yaml`` which contains the problemset information.

Example ``problems.yaml``::

  - id:     hello
    label:  A
    name:   Hello World
    color:  Orange
    rgb:    '#FF7109'

  - id:     boolfind
    label:  B
    name:   Boolfind
    color:  Forest Green
    rgb:    '#008100'

.. note::

    You can also use a JSON file instead of YAML. Make sure to name it
    ``problems.json`` in that case.

    The minimum required fields are `id` and `label`.

To import the file using the jury interface, go to `Import / export`, then
`Problems -> Import JSON / YAML`, select your file under `File`
and click `Import`.

To import the file using the API run the following commands::

    http --check-status -b -f POST "<API_URL>/contests/<CID>/problems/add-data" data@problems.yaml

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -f data=problems.yaml contests/<CID>/problems/add-data

Replace ``<CID>`` with the contest ID that was returned when importing the
contest metadata.

Importing problems
------------------

Prepare your problems in the :doc:`ICPC problem format <problem-format>` and
create a ZIP file for each problem.

To import the file using the jury interface, go to `Problems`, select the contest
you want to import the problems into, select your file under `Problem archive(s)`
and click `Upload`.

To import the file using the API run the following command::

    http --check-status -b -f POST "<API_URL>/contests/<CID>/problems" zip@problem.zip problem="<PROBID>"

To import the file using the CLI run the following command::

    <WEBAPP_DIR>/bin/console api:call -m POST -d problem=<PROBID> -f zip=problem.zip contest/<CID>/problems

Replace ``<CID>`` with the contest ID that the previous command returns and
``<PROBID>`` with the problem ID (you can get that from the web interface or
the API).

Putting all API imports together
--------------------------------

If you prepare your contest configuration as we described in the previous
subsections, you can also use the script that we provide in
`misc-tools/import-contest`.

Call it from your contest folder like this::

    misc-tools/import-contest <API_URL>

to use the API, or simply::

    misc-tools/import-contest

to use the CLI. In this case you must run it from the DOMserver.

Importing from ICPC CMS API
---------------------------

DOMjudge also offers a direct import/refresh of teams from the ICPC CMS API from
within the DOMjudge web interface. You need a  a team category named 'Participants'
where they will be placed and a ICPC Web Services Token.

To create a Web Services Token, log into the ICPC CMS and click "Export > Web
Services Tokens". Make sure you add the scopes "Export", "Standings Upload",
and "MyICPC".  Under the `Import / Export` menu, enter the token as specified.
Use the contest abbreviation and year as Contest ID (see the URL in the ICPC
CMS).

Based on the 'ICPC ID', teams and their affiliations will be added if they do not
exist or updated when they do. Teams will be set to 'enabled' if their ICPC CMS
status is 'ACCEPTED', of disabled otherwise. Affiliations are not updated or
deleted even when all teams cancel.

Importing DOMjudge configuration
--------------------------------

DOMjudge exposes its configuration at the `<API_URL>/config` endpoint in JSON
form and accepts a `PUT` request to load/update configuration.

You can retrieve the current configuration using the API via::

    http --check-status --pretty=format "<API_URL>/config" > config.json

or via the CLI using::

    <WEBAPP_DIR>/bin/console api:call config > config.json

For your convenience, we added a script to update configuration from a file
called `config.json` in your current directory::

    misc-tools/configure-domjudge <API_URL>

to use the API, or simply::

    misc-tools/configure-domjudge

to use the CLI. In this case you must run it from the DOMserver.

.. _CCS specification: https://ccs-specs.icpc.io/2022-07/ccs_system_requirements#appendix-file-formats
.. _.netrc: https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html
.. _httpie: https://httpie.org/
