Skip to content

Commit 77e050f

Browse files
Add EvaluationTracker to docs and fix its docstring (#464)
* Fix definition of public in docstring * Fix push_to_tensorboard param name in docstring * Fix docstring style * Add EvaluationTracker to docs * Fix docstring style * Move docstring to class header * Add attributes to docstring * Fix style * Fix style * Fix style * Fix style * Fix style * Fix style * Fix internal links in docstring
1 parent d442b27 commit 77e050f

File tree

2 files changed

+31
-25
lines changed

2 files changed

+31
-25
lines changed

docs/source/package_reference/logging.mdx

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1-
# Loggers
1+
# Logging
2+
3+
## EvaluationTracker
4+
[[autodoc]] logging.evaluation_tracker.EvaluationTracker
25

36
## GeneralConfigLogger
47
[[autodoc]] logging.info_loggers.GeneralConfigLogger

src/lighteval/logging/evaluation_tracker.py

Lines changed: 27 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -82,16 +82,35 @@ def default(self, o):
8282

8383

8484
class EvaluationTracker:
85-
"""
86-
Keeps track of the overall evaluation process and relevant informations.
85+
"""Keeps track of the overall evaluation process and relevant information.
8786
88-
The [`EvaluationTracker`] contains specific loggers for experiments details
89-
([`DetailsLogger`]), metrics ([`MetricsLogger`]), task versions
90-
([`VersionsLogger`]) as well as for the general configurations of both the
91-
specific task ([`TaskConfigLogger`]) and overall evaluation run
92-
([`GeneralConfigLogger`]). It compiles the data from these loggers and
87+
The [`~logging.evaluation_tracker.EvaluationTracker`] contains specific loggers for experiments details
88+
([`~logging.evaluation_tracker.DetailsLogger`]), metrics ([`~logging.evaluation_tracker.MetricsLogger`]), task versions
89+
([`~logging.evaluation_tracker.VersionsLogger`]) as well as for the general configurations of both the
90+
specific task ([`~logging.evaluation_tracker.TaskConfigLogger`]) and overall evaluation run
91+
([`~logging.evaluation_tracker.GeneralConfigLogger`]). It compiles the data from these loggers and
9392
writes it to files, which can be published to the Hugging Face hub if
9493
requested.
94+
95+
Args:
96+
output_dir (`str`): Local folder path where you want results to be saved.
97+
save_details (`bool`, defaults to True): If True, details are saved to the `output_dir`.
98+
push_to_hub (`bool`, defaults to False): If True, details are pushed to the hub.
99+
Results are pushed to `{hub_results_org}/details__{sanitized model_name}` for the model `model_name`, a public dataset,
100+
if `public` is True else `{hub_results_org}/details__{sanitized model_name}_private`, a private dataset.
101+
push_to_tensorboard (`bool`, defaults to False): If True, will create and push the results for a tensorboard folder on the hub.
102+
hub_results_org (`str`, *optional*): The organisation to push the results to.
103+
See more details about the datasets organisation in [`EvaluationTracker.save`].
104+
tensorboard_metric_prefix (`str`, defaults to "eval"): Prefix for the metrics in the tensorboard logs.
105+
public (`bool`, defaults to False): If True, results and details are pushed to public orgs.
106+
nanotron_run_info ([`~nanotron.config.GeneralArgs`], *optional*): Reference to information about Nanotron models runs.
107+
108+
**Attributes**:
109+
- **details_logger** ([`~logging.info_loggers.DetailsLogger`]) -- Logger for experiment details.
110+
- **metrics_logger** ([`~logging.info_loggers.MetricsLogger`]) -- Logger for experiment metrics.
111+
- **versions_logger** ([`~logging.info_loggers.VersionsLogger`]) -- Logger for task versions.
112+
- **general_config_logger** ([`~logging.info_loggers.GeneralConfigLogger`]) -- Logger for general configuration.
113+
- **task_config_logger** ([`~logging.info_loggers.TaskConfigLogger`]) -- Logger for task configuration.
95114
"""
96115

97116
def __init__(
@@ -105,23 +124,7 @@ def __init__(
105124
public: bool = False,
106125
nanotron_run_info: "GeneralArgs" = None,
107126
) -> None:
108-
"""
109-
Creates all the necessary loggers for evaluation tracking.
110-
111-
Args:
112-
output_dir (str): Local folder path where you want results to be saved
113-
save_details (bool): If True, details are saved to the output_dir
114-
push_to_hub (bool): If True, details are pushed to the hub.
115-
Results are pushed to `{hub_results_org}/details__{sanitized model_name}` for the model `model_name`, a public dataset,
116-
if `public` is True else `{hub_results_org}/details__{sanitized model_name}_private`, a private dataset.
117-
push_results_to_tensorboard (bool): If True, will create and push the results for a tensorboard folder on the hub
118-
hub_results_org (str): The organisation to push the results to. See
119-
more details about the datasets organisation in
120-
[`EvaluationTracker.save`]
121-
tensorboard_metric_prefix (str): Prefix for the metrics in the tensorboard logs
122-
public (bool): If True, results and details are pushed in private orgs
123-
nanotron_run_info (GeneralArgs): Reference to informations about Nanotron models runs
124-
"""
127+
"""Creates all the necessary loggers for evaluation tracking."""
125128
self.details_logger = DetailsLogger()
126129
self.metrics_logger = MetricsLogger()
127130
self.versions_logger = VersionsLogger()

0 commit comments

Comments
 (0)