[EVAL]

[barneylogo]

Evaluation short description

I've run nanotron script for training model with my own fineweb dataset
so I only got this result, model checkpoints

I'd like to evaluate this using lighteval
Please help this problem.
I've tried with this command

accelerate launch --multi_gpu --num_processes=8 -m \
    lighteval accelerate \
    --model_args "pretrained=./checkpoints/10" \
    --tasks "lighteval|truthfulqa:mc|0|0" \
    --override_batch_size 1 \
    --output_dir="./evals/"

but how to use checkpoints for evaluating.
Python version: 3.10.12

[clefourrier]

Hi!
If you're using a nanotron model, you should not launch lighteval with accelerate, but you should instead use nanotron, with lighteval nanotron (You can do lighteval nanotron --help to get the arguments to use).

[clefourrier]

Careful, most of the nanotron configuration should be provided with configuration files, you can look for examples here)

[barneylogo]

hello @clefourrier
Thank you for your answer.
I've tried this command lighteval nanotron --checkpoint-config-path './checkpoint/10/config.yaml' --lighteval-override './examples/nanotron/lighteval_config_overrride_template.yaml'
But getting this error

  File "/root/ben/lighteval/.venv/bin/lighteval", line 5, in <module>
    from lighteval.__main__ import cli_evaluate
  File "/root/ben/lighteval/src/lighteval/__main__.py", line 31, in <module>
    from lighteval.tasks.registry import Registry, taskinfo_selector
  File "/root/ben/lighteval/src/lighteval/tasks/registry.py", line 34, in <module>
    import lighteval.tasks.default_tasks as default_tasks
  File "/root/ben/lighteval/src/lighteval/tasks/default_tasks.py", line 22, in <module>
    import lighteval.tasks.tasks_prompt_formatting as prompt
  File "/root/ben/lighteval/src/lighteval/tasks/tasks_prompt_formatting.py", line 32, in <module>
    from lighteval.logging.hierarchical_logger import hlog_warn
  File "/root/ben/lighteval/src/lighteval/logging/hierarchical_logger.py", line 33, in <module>
    from nanotron.logging import get_logger
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/logging.py", line 38, in <module>
    from nanotron.config.config import LoggingArgs
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/__init__.py", line 2, in <module>
    from nanotron.config.config import *
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/config.py", line 13, in <module>
    from nanotron.config.lighteval_config import LightEvalConfig
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/lighteval_config.py", line 5, in <module>
    from nanotron.config.parallelism_config import ParallelismArgs
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/parallelism_config.py", line 4, in <module>
    from nanotron.config.utils_config import (
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/config/utils_config.py", line 8, in <module>
    from nanotron.parallel.pipeline_parallel.engine import (
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/parallel/pipeline_parallel/__init__.py", line 1, in <module>
    from nanotron.parallel.pipeline_parallel.engine import PipelineEngine
  File "/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/parallel/pipeline_parallel/engine.py", line 8, in <module>
    from nanotron.logging import log_rank
ImportError: cannot import name 'log_rank' from partially initialized module 'nanotron.logging' (most likely due to a circular import) (/root/ben/lighteval/.venv/lib/python3.10/site-packages/nanotron/logging.py)

[barneylogo]

I've tried to install by pip install -e .
but after that I am getting this error

Traceback (most recent call last):
  File "/root/ben/lighteval/.venv/bin/lighteval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/root/ben/lighteval/src/lighteval/__main__.py", line 61, in cli_evaluate
    from lighteval.main_nanotron import main as main_nanotron
  File "/root/ben/lighteval/src/lighteval/main_nanotron.py", line 35, in <module>
    from lighteval.models.nanotron_model import NanotronLightevalModel
  File "/root/ben/lighteval/src/lighteval/models/nanotron_model.py", line 32, in <module>
    from nanotron import distributed as dist
ModuleNotFoundError: No module named 'nanotron'

If I install nanotron by pip install '.[nanotron]', got above error again ImportError: cannot import name 'log_rank' from partially initialized module 'nanotron.logging'

[clefourrier]

Interesting, thanks for the report, we'll investigate this!

[barneylogo]

Thank you
Is there maybe some python version problem?

[clefourrier]

No, I think we might have added a small regression on the nanotron models. @zzhhjj has been working on adding unit tests for nanotron models in this PR #160, but since we don't have them yet we might be breaking things from time to time (we tend to focus on the accelerate models when testing, since we thought nanotron was only used internally, and internally people can fix the codebase easily).

[barneylogo]

Undertand, Thank you
So, which solution is good for me to evaluate nanotron model checkpoints?

[NathanHB]

Hi ! It seems to be an issue with the nanotron library. To replicate the issue, simply open a python interpreter and run from nanotron.logging import get_logger. We invite you to move this issue to the nanotron repo, closing it here, but feel free to open another one if you meet any issues with the lib :)

[barneylogo]

hello @NathanHB
thank you for your message
if so, can I solve this problem using old lighteval version?

[barneylogo]

I am in huggingface discord community.
if available, can we solve problem in discord?

[clefourrier]

Hi @barney49 , no, you would need to use an old nanotron version, but I'm not sure which.
As @NathanHB indicated, you'll need to open an issue on the nanotron repository, as we use github at HF to track problems and make sure no information is lost.

[barneylogo]

understand
I've tried using old nanotron version, maybe 2.0. There is 0.1, 0.2, 0.4
but it is also not working
I'd like to know about old lighteval version