[taskcluster 2024-10-22 19:37:23.962Z] Task ID: ZokblYKnT5WhPVxoy7GCuw
[taskcluster 2024-10-22 19:37:23.962Z] Worker ID: 1555522593769538130
[taskcluster 2024-10-22 19:37:23.962Z] Worker Group: us-west1-a
[taskcluster 2024-10-22 19:37:23.962Z] Worker Node Type: projects/887720501152/machineTypes/n2-highmem-32
[taskcluster 2024-10-22 19:37:23.962Z] Worker Pool: translations-1/b-linux-large-gcp
[taskcluster 2024-10-22 19:37:23.962Z] Worker Version: 38.0.5
[taskcluster 2024-10-22 19:37:23.962Z] Public IP: 35.227.190.252
[taskcluster 2024-10-22 19:37:23.962Z] Hostname: translations-1-b-linux-large-gcp-bl9xbixpqkewjwijljjgua
[taskcluster 2024-10-22 19:37:23.962Z] using cache "translations-level-1-checkouts-v3-7afeb851dd97df8f3607-WhHlxCxnQYW05fB1pTTteQ" -> /builds/worker/checkouts

[taskcluster 2024-10-22 19:37:24.420Z] Image 'public/image.tar.zst' from task 'WhHlxCxnQYW05fB1pTTteQ' loaded.  Using image ID sha256:502a53cc11fc8a59c0ca778225c5a4e38adf93683b8d4000d345029404ff6cf9.
[taskcluster 2024-10-22 19:37:24.432Z] === Task Starting ===
[setup 2024-10-22T19:37:24.683Z] run-task started in /builds/worker
[setup 2024-10-22T19:37:24.683Z] Invoked by command: --firefox_translations_training-checkout=/builds/worker/checkouts/vcs/ -- bash -c export BMT_MARIAN=$MOZ_FETCHES_DIR && export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MOZ_FETCHES_DIR/cuda-toolkit/lib64" && zstd --rm -d $MOZ_FETCHES_DIR/*.zst && $VCS_PATH/pipeline/quantize/export.sh $MOZ_FETCHES_DIR $MOZ_FETCHES_DIR/lex.s2t.pruned $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts
[setup 2024-10-22T19:37:24.683Z] Python version: 3.10.12
[cache 2024-10-22T19:37:24.684Z] cache /builds/worker/checkouts exists; requirements: gid=1000 uid=1000 version=1
[volume 2024-10-22T19:37:24.684Z] volume /builds/worker/checkouts is a cache
[setup 2024-10-22T19:37:24.684Z] running as worker:worker
[vcs 2024-10-22T19:37:24.685Z] executing ['git', 'config', '--global', '--add', 'safe.directory', '/builds/worker/checkouts/vcs']
[vcs 2024-10-22T19:37:24.686Z] executing ['git', 'fetch', '--tags', '--force', 'https://github.com/mozilla/firefox-translations-training', 'fix_tc_train_args']
[vcs 2024-10-22T19:37:24.933Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-10-22T19:37:24.934Z]  * branch            fix_tc_train_args -> FETCH_HEAD
[vcs 2024-10-22T19:37:24.941Z] executing ['git', 'fetch', '--no-tags', 'https://github.com/mozilla/firefox-translations-training', 'fix_tc_train_args']
[vcs 2024-10-22T19:37:25.269Z] From https://github.com/mozilla/firefox-translations-training
[vcs 2024-10-22T19:37:25.269Z]  * branch            fix_tc_train_args -> FETCH_HEAD
[vcs 2024-10-22T19:37:25.276Z] executing ['git', 'checkout', '-f', '-B', 'fix_tc_train_args', '2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c']
[vcs 2024-10-22T19:37:25.281Z] Reset branch 'fix_tc_train_args'
[vcs 2024-10-22T19:37:25.282Z] executing ['git', 'submodule', 'init']
[vcs 2024-10-22T19:37:25.300Z] executing ['git', 'submodule', 'update', '--force']
[vcs 2024-10-22T19:37:25.385Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-10-22T19:37:25.395Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-10-22T19:37:25.405Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-10-22T19:37:25.430Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-10-22T19:37:25.546Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-10-22T19:37:25.564Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-10-22T19:37:25.630Z] Submodule path 'inference/3rd_party/browsermt-marian-dev': checked out '2781d735d4a10dca876d61be587afdab2726293c'
[vcs 2024-10-22T19:37:25.644Z] Submodule path 'inference/3rd_party/ssplit-cpp': checked out 'a311f9865ade34db1e8e080e6cc146f55dafb067'
[vcs 2024-10-22T19:37:25.644Z] cleaning git checkout...
[vcs 2024-10-22T19:37:25.644Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-10-22T19:37:25.647Z] removing []
[vcs 2024-10-22T19:37:25.647Z] successfully cleaned git checkout!
[vcs 2024-10-22T19:37:25.648Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c' title='Built from firefox-translations-training commit 2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c'>2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c</a>
[setup 2024-10-22T19:37:25.649Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-10-22T19:37:25.649Z] fetching artifacts
[fetches 2024-10-22T19:37:25.649Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
attempt 1/5Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LT4F01raQkKWVNlk-YAL0A/artifacts/public/build/model.intgemm.alphas.bin to /builds/worker/fetches/model.intgemm.alphas.bin

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DFpFNF65Ro2hC7gkqwgJcw/artifacts/public/build/lex.s2t.pruned.zst to /builds/worker/fetches/lex.s2t.pruned.zstattempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cb12ATGYSJaD1A0qz5oD6g/artifacts/public/build/marian.tar.zst to /builds/worker/fetches/marian.tar.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cb12ATGYSJaD1A0qz5oD6g/artifacts/public/build/marian.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DFpFNF65Ro2hC7gkqwgJcw/artifacts/public/build/lex.s2t.pruned.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LT4F01raQkKWVNlk-YAL0A/artifacts/public/build/model.intgemm.alphas.bin
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DL1419gWTqWep_47yjP9hg/artifacts/public/build/vocab.spm to /builds/worker/fetches/vocab.spm
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DL1419gWTqWep_47yjP9hg/artifacts/public/build/vocab.spm
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QnEgC95GRbOxLw0YNH2esQ/artifacts/public/build/cuda-toolkit.tar.zst to /builds/worker/fetches/cuda-toolkit.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QnEgC95GRbOxLw0YNH2esQ/artifacts/public/build/cuda-toolkit.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DFpFNF65Ro2hC7gkqwgJcw/artifacts/public/build/lex.s2t.pruned.zst resolved to 7601 bytes with sha256 8de6f9bfb1216c5db65742fddaef13a37006d8fac88c35ef4c411d8055207583 in 0.044s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DFpFNF65Ro2hC7gkqwgJcw/artifacts/public/build/lex.s2t.pruned.zst
Extracting /builds/worker/fetches/lex.s2t.pruned.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DL1419gWTqWep_47yjP9hg/artifacts/public/build/vocab.spm resolved to 255318 bytes with sha256 baddcca4f6c928751cc9eba95633a925bc8f631c5cb578d5003a4becd86c14a4 in 0.068s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DL1419gWTqWep_47yjP9hg/artifacts/public/build/vocab.spm
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LT4F01raQkKWVNlk-YAL0A/artifacts/public/build/model.intgemm.alphas.bin resolved to 9081145 bytes with sha256 4b5c9d70b03131633f92d8572e427f181dd61c21e167ed1e1183a2bb5777b36f in 0.229s
Extracting /builds/worker/fetches/model.intgemm.alphas.bin to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cb12ATGYSJaD1A0qz5oD6g/artifacts/public/build/marian.tar.zst resolved to 166219622 bytes with sha256 bda12b3a9a211caea7bb4a4d4c6c7d66bb77727bd9f1fc2e048891388d244ebe in 3.567s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/cb12ATGYSJaD1A0qz5oD6g/artifacts/public/build/marian.tar.zst
Extracting /builds/worker/fetches/marian.tar.zst to /builds/worker/fetches
/builds/worker/fetches/marian.tar.zst extracted in 1.223s
Removing /builds/worker/fetches/marian.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QnEgC95GRbOxLw0YNH2esQ/artifacts/public/build/cuda-toolkit.tar.zst resolved to 3294844747 bytes with sha256 e10c969eb045db360f74507890d78e7547a9c038a017302a4d40630e7f2b6d74 in 57.513s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QnEgC95GRbOxLw0YNH2esQ/artifacts/public/build/cuda-toolkit.tar.zst
Extracting /builds/worker/fetches/cuda-toolkit.tar.zst to /builds/worker/fetches
/builds/worker/fetches/cuda-toolkit.tar.zst extracted in 13.847s
Removing /builds/worker/fetches/cuda-toolkit.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 71.63664864700002, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-10-22T19:38:37.363Z] finished fetching artifacts
[task 2024-10-22T19:38:37.363Z] executing ['bash', '-c', 'export BMT_MARIAN=$MOZ_FETCHES_DIR && export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$MOZ_FETCHES_DIR/cuda-toolkit/lib64" && zstd --rm -d $MOZ_FETCHES_DIR/*.zst && $VCS_PATH/pipeline/quantize/export.sh $MOZ_FETCHES_DIR $MOZ_FETCHES_DIR/lex.s2t.pruned $MOZ_FETCHES_DIR/vocab.spm $TASK_WORKDIR/artifacts']
[task 2024-10-22T19:38:37.365Z] 
[task 2024-10-22T19:38:37.365Z] s/lex.s2t.pruned.zst : 0 MB...     
[task 2024-10-22T19:38:37.365Z]                                                                                
[task 2024-10-22T19:38:37.365Z] /builds/worker/fetches/lex.s2t.pruned.zst: 23260 bytes 
[task 2024-10-22T19:38:37.367Z] + set -euo pipefail
[task 2024-10-22T19:38:37.367Z] + echo '###### Exporting a quantized model'
[task 2024-10-22T19:38:37.367Z] ###### Exporting a quantized model
[task 2024-10-22T19:38:37.367Z] + test -v SRC
[task 2024-10-22T19:38:37.367Z] + test -v TRG
[task 2024-10-22T19:38:37.367Z] + test -v BMT_MARIAN
[task 2024-10-22T19:38:37.367Z] + model_dir=/builds/worker/fetches
[task 2024-10-22T19:38:37.367Z] + shortlist=/builds/worker/fetches/lex.s2t.pruned
[task 2024-10-22T19:38:37.367Z] + vocab=/builds/worker/fetches/vocab.spm
[task 2024-10-22T19:38:37.367Z] + output_dir=/builds/worker/artifacts
[task 2024-10-22T19:38:37.367Z] + mkdir -p /builds/worker/artifacts
[task 2024-10-22T19:38:37.368Z] + model=/builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2024-10-22T19:38:37.368Z] + cp /builds/worker/fetches/model.intgemm.alphas.bin /builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2024-10-22T19:38:37.375Z] + pigz /builds/worker/artifacts/model.ruen.intgemm.alphas.bin
[task 2024-10-22T19:38:37.406Z] + shortlist_bin=/builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2024-10-22T19:38:37.406Z] + /builds/worker/fetches/marian-conv --shortlist /builds/worker/fetches/lex.s2t.pruned 50 50 0 --dump /builds/worker/artifacts/lex.50.50.ruen.s2t.bin --vocabs /builds/worker/fetches/vocab.spm /builds/worker/fetches/vocab.spm
[task 2024-10-22T19:38:37.434Z] [2024-10-22 19:38:37] [data] Loading SentencePiece vocabulary from file /builds/worker/fetches/vocab.spm
[task 2024-10-22T19:38:37.436Z] [2024-10-22 19:38:37] [data] Loading SentencePiece vocabulary from file /builds/worker/fetches/vocab.spm
[task 2024-10-22T19:38:37.438Z] [2024-10-22 19:38:37] [data] Importing text lexical shortlist as /builds/worker/fetches/lex.s2t.pruned 50 50 0
[task 2024-10-22T19:38:37.439Z] [2024-10-22 19:38:37] [data] Saving binary shortlist dump to /builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2024-10-22T19:38:37.439Z] [2024-10-22 19:38:37] Finished
[task 2024-10-22T19:38:37.445Z] + pigz /builds/worker/artifacts/lex.50.50.ruen.s2t.bin
[task 2024-10-22T19:38:37.447Z] + vocab_out=/builds/worker/artifacts/vocab.ruen.spm
[task 2024-10-22T19:38:37.447Z] + cp /builds/worker/fetches/vocab.spm /builds/worker/artifacts/vocab.ruen.spm
[task 2024-10-22T19:38:37.449Z] + pigz /builds/worker/artifacts/vocab.ruen.spm
[task 2024-10-22T19:38:37.460Z] + echo '### Export is completed. Results: /builds/worker/artifacts'
[task 2024-10-22T19:38:37.460Z] ### Export is completed. Results: /builds/worker/artifacts
[task 2024-10-22T19:38:37.460Z] + echo '###### Done: Exporting a quantized model'
[task 2024-10-22T19:38:37.460Z] ###### Done: Exporting a quantized model
[fetches 2024-10-22T19:38:37.461Z] removing /builds/worker/fetches
[fetches 2024-10-22T19:38:38.463Z] finished
[taskcluster 2024-10-22 19:38:38.703Z] === Task Finished ===
[taskcluster 2024-10-22 19:38:39.359Z] Successful task run with exit code: 0 completed in 75.398 seconds