You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+9Lines changed: 9 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,16 @@ It sets up a dummy classification task, evaluated with accuracy metric on a publ
6
6
## Structure of the bundle
7
7
8
8
-`ingestion_program/`: contains the ingestion program that will be run on participant's submissions. It is responsible for loading the code from the submission, passing the training data to train the model, and generating predictions on the test datasets.
9
+
It contains:
10
+
*`metadata.yaml`: A file describing how to run the ingestion program for `codabench`. For a single script ingestion program in `ingestion.py`, no need to edit it.
11
+
*`ingestion.py`: A script to run the ingestion. The role of this script is to load the submission code and produce predictions that can be evaluated with the `scoring_program`.
12
+
In our example, the submission define a `train_model` function that is called with the training data, and returns a model from which we can call the `predict` method.
13
+
The predictions are then stored as a csv file, to be loaded with the `scoring_program`.
9
14
-`scoring_program/`: contains the scoring program that will be run to evaluate the predictions generated by the ingestion program. It loads the predictions and the ground truth labels, computes the evaluation metric (accuracy in this case), and outputs the score.
15
+
It contains:
16
+
*`metadata.yaml`: A file describing how to run the scoring program for `codabench`. For a single script ingestion program in `scoring.py`, no need to edit it.
17
+
*`scoring.py`: A script to run the scoring. This script loads the prediction dumped from the ingestion program and produce a single json file containing the scores associated with the submission.
18
+
In our example, we compute `accuracy` on two test sets (public and private) as well as runtime.
10
19
-`solution/`: contains a sample solution submission that participants can use as a reference. Here, this is a simple Random Forest classifier.
11
20
-`*_phase/`: contains the data for a given phase, including input data and reference labels. Running `setup_data.py` will generate dummy data for a development phase.
12
21
-`competition.yaml`: configuration file for the codabench competition, specifying phases, tasks, and evaluation metrics.
0 commit comments