Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix taskcluster train scripts #895

Merged
merged 1 commit into from
Oct 22, 2024

Fix taskcluster train scripts

2d0fb8c
Select commit
Loading
Failed to load commit list.
Merged

Fix taskcluster train scripts #895

Fix taskcluster train scripts
2d0fb8c
Select commit
Loading
Failed to load commit list.
firefoxci-taskcluster / extract-best-ru-en-1/2 succeeded Oct 22, 2024 in 41m 6s

FirefoxCI (pull_request)

extract best ru-en 1/2

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2024-10-22T18:41:08.944Z
Resolved: 2024-10-22T18:41:15.235Z
Task Execution Time: 6 seconds, 291 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 0

Artifacts

- public/build/file.1.nbest.out
- public/logs/live_backing.log
- public/logs/live.log


[taskcluster 2024-10-22 18:41:09.038Z] Task ID: DChbFsRORzuhyc25-y6BCg
[taskcluster 2024-10-22 18:41:09.038Z] Worker ID: 6868397010072522708
[taskcluster 2024-10-22 18:41:09.038Z] Worker Group: us-central1-b
[taskcluster 2024-10-22 18:41:09.038Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-10-22 18:41:09.038Z] Worker Pool: translations-1/b-linux-large-gcp-300gb
[taskcluster 2024-10-22 18:41:09.038Z] Worker Version: 38.0.5
[taskcluster 2024-10-22 18:41:09.038Z] Public IP: 34.132.120.51
[taskcluster 2024-10-22 18:41:09.038Z] Hostname: translations-1-b-linux-large-gcp-300gb-vorxoxbdt8kgbdpvihotmw
[taskcluster 2024-10-22 18:41:09.038Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-WhHlxCxnQYW05fB1pTTteQ" -> /builds/worker/checkouts

[taskcluster 2024-10-22 18:41:12.581Z] Image 'public/image.tar.zst' from task 'WhHlxCxnQYW05fB1pTTteQ' loaded.  Using image ID sha256:502a53cc11fc8a59c0ca778225c5a4e38adf93683b8d4000d345029404ff6cf9.
[taskcluster 2024-10-22 18:41:12.629Z] === Task Starting ===
[setup 2024-10-22T18:41:12.966Z] run-task started in /builds/worker
[setup 2024-10-22T18:41:12.966Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c zstd -d --rm $MOZ_FETCHES_DIR/*.zst && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && python3 $VCS_PATH/pipeline/translate/bestbleu.py -i "$MOZ_FETCHES_DIR/file.1.nbest" -r "$MOZ_FETCHES_DIR/file.1.ref" -o $TASK_WORKDIR/artifacts/file.1.nbest.out -m bleu
[setup 2024-10-22T18:41:12.966Z] Python version: 3.10.12
[cache 2024-10-22T18:41:12.967Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-10-22T18:41:12.967Z] volume /builds/worker/checkouts is a cache
[setup 2024-10-22T18:41:12.967Z] running as worker:worker
[vcs 2024-10-22T18:41:12.967Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-10-22T18:41:12.969Z] executing ['git', 'fetch', '--tags', '--force', 'https://github.com/mozilla/firefox-translations-training', 'fix_tc_train_args']
[vcs 2024-10-22T18:41:13.172Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-10-22T18:41:13.172Z]  * branch            fix_tc_train_args -> FETCH_HEAD
[vcs 2024-10-22T18:41:13.179Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/mozilla/firefox-translations-training', 'fix_tc_train_args']
[vcs 2024-10-22T18:41:13.381Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-10-22T18:41:13.381Z]  * branch            fix_tc_train_args -> FETCH_HEAD
[vcs 2024-10-22T18:41:13.388Z] executing ['git', 'checkout', '-f', '-B', 'fix_tc_train_args', '2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c']
[vcs 2024-10-22T18:41:13.394Z] Reset branch 'fix_tc_train_args'
[vcs 2024-10-22T18:41:13.395Z] executing ['git', 'submodule', 'init']
[vcs 2024-10-22T18:41:13.415Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-10-22T18:41:13.452Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-10-22T18:41:13.463Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-10-22T18:41:13.475Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-10-22T18:41:13.487Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-10-22T18:41:13.504Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-10-22T18:41:13.516Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-10-22T18:41:13.534Z] Submodule path 'inference/3rd_party/browsermt-marian-dev': checked out '2781d735d4a10dca876d61be587afdab2726293c'
[vcs 2024-10-22T18:41:13.545Z] Submodule path 'inference/3rd_party/ssplit-cpp': checked out 'a311f9865ade34db1e8e080e6cc146f55dafb067'
[vcs 2024-10-22T18:41:13.546Z] cleaning git checkout...
[vcs 2024-10-22T18:41:13.546Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-10-22T18:41:13.549Z] removing []
[vcs 2024-10-22T18:41:13.549Z] successfully cleaned git checkout!
[vcs 2024-10-22T18:41:13.551Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c' title='Built from firefox-translations-training commit 2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c'>2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c</a>
[setup 2024-10-22T18:41:13.551Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-10-22T18:41:13.551Z] fetching artifacts
[fetches 2024-10-22T18:41:13.551Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OKehpaxNSAyoHgMCmUseRw/artifacts/public/build/file.1.ref.zst to /builds/worker/fetches/file.1.ref.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SayY_CicRTC_dVJpKZVgdA/artifacts/public/build/file.1.nbest.zst to /builds/worker/fetches/file.1.nbest.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OKehpaxNSAyoHgMCmUseRw/artifacts/public/build/file.1.ref.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SayY_CicRTC_dVJpKZVgdA/artifacts/public/build/file.1.nbest.zst

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SayY_CicRTC_dVJpKZVgdA/artifacts/public/build/file.1.nbest.zst resolved to 47810 bytes with sha256 9001af39b1b5cb02832544f2492bda0ecc33f6aa80cd1c099df844c7e1c10462 in 0.148s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/SayY_CicRTC_dVJpKZVgdA/artifacts/public/build/file.1.nbest.zst
Extracting /builds/worker/fetches/file.1.nbest.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OKehpaxNSAyoHgMCmUseRw/artifacts/public/build/file.1.ref.zst resolved to 18566 bytes with sha256 b4e3f7022763e20f3924121dd409edcdb76ac43e9acbdc88c31b371e07b547e2 in 0.197s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OKehpaxNSAyoHgMCmUseRw/artifacts/public/build/file.1.ref.zst
Extracting /builds/worker/fetches/file.1.ref.zst to /builds/worker/fetches
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 0.20002050900001223, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-10-22T18:41:13.838Z] finished fetching artifacts
[task 2024-10-22T18:41:13.838Z] executing ['bash', '-c', 'zstd -d --rm $MOZ_FETCHES_DIR/*.zst && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && python3 $VCS_PATH/pipeline/translate/bestbleu.py -i "$MOZ_FETCHES_DIR/file.1.nbest" -r "$MOZ_FETCHES_DIR/file.1.ref" -o $TASK_WORKDIR/artifacts/file.1.nbest.out -m bleu']
[task 2024-10-22T18:41:13.841Z] 
[task 2024-10-22T18:41:13.842Z] Decompress:  1/ 2 files. Current: ...ile.1.nbest.zst : 0 MB...    
[task 2024-10-22T18:41:13.843Z]                                                                                
[task 2024-10-22T18:41:13.843Z] 
[task 2024-10-22T18:41:13.843Z]                                                                                
[task 2024-10-22T18:41:13.843Z] 2 files decompressed : 1012839 bytes total 
[fetches 2024-10-22T18:41:14.256Z] removing /builds/worker/fetches
[fetches 2024-10-22T18:41:14.257Z] finished
[taskcluster 2024-10-22 18:41:14.486Z] === Task Finished ===
[taskcluster 2024-10-22 18:41:14.674Z] Successful task run with exit code: 0 completed in 5.636 seconds