Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix taskcluster train scripts #895

Merged
merged 1 commit into from
Oct 22, 2024

Fix taskcluster train scripts

2d0fb8c
Select commit
Loading
Failed to load commit list.
Merged

Fix taskcluster train scripts #895

Fix taskcluster train scripts
2d0fb8c
Select commit
Loading
Failed to load commit list.
firefoxci-taskcluster / alignments-student-ru-en succeeded Oct 22, 2024 in 1m 8s

FirefoxCI (pull_request)

alignments for student training corpus ru-en

Details

View task in Taskcluster | View logs in Taskcluster | View task group in Taskcluster

Task Status

Started: 2024-10-22T18:55:04.397Z
Resolved: 2024-10-22T18:56:12.499Z
Task Execution Time: 1 minute, 8 seconds, 102 milliseconds
Task Status: completed
Reason Resolved: completed
RunId: 1

Artifacts

- public/build/corpus.aln.zst
- public/logs/live_backing.log
- public/logs/live.log


[taskcluster 2024-10-22T18:55:04.419Z] Worker Type (translations-1/b-linux-large-gcp-1tb-32-256-d2g) settings:
[taskcluster 2024-10-22T18:55:04.419Z]   {
[taskcluster 2024-10-22T18:55:04.419Z]     "config": {
[taskcluster 2024-10-22T18:55:04.419Z]       "deploymentId": ""
[taskcluster 2024-10-22T18:55:04.419Z]     },
[taskcluster 2024-10-22T18:55:04.419Z]     "generic-worker": {
[taskcluster 2024-10-22T18:55:04.419Z]       "engine": "multiuser",
[taskcluster 2024-10-22T18:55:04.419Z]       "go-arch": "amd64",
[taskcluster 2024-10-22T18:55:04.419Z]       "go-os": "linux",
[taskcluster 2024-10-22T18:55:04.419Z]       "go-version": "go1.22.2",
[taskcluster 2024-10-22T18:55:04.419Z]       "release": "https://github.com/taskcluster/taskcluster/releases/tag/v64.3.0",
[taskcluster 2024-10-22T18:55:04.419Z]       "revision": "b66b6614b9b79077f1bfc78f97afb8c25c0c4b30",
[taskcluster 2024-10-22T18:55:04.419Z]       "source": "https://github.com/taskcluster/taskcluster/commits/b66b6614b9b79077f1bfc78f97afb8c25c0c4b30",
[taskcluster 2024-10-22T18:55:04.419Z]       "version": "64.3.0"
[taskcluster 2024-10-22T18:55:04.419Z]     },
[taskcluster 2024-10-22T18:55:04.419Z]     "image": "projects/taskcluster-imaging/global/images/gw-fxci-gcp-l1-gui-googlecompute-2024-08-22t22-48-09z",
[taskcluster 2024-10-22T18:55:04.419Z]     "instance-id": "7209816334232341101",
[taskcluster 2024-10-22T18:55:04.419Z]     "instance-type": "projects/887720501152/machineTypes/n2-highmem-32",
[taskcluster 2024-10-22T18:55:04.419Z]     "local-ipv4": "10.138.0.81",
[taskcluster 2024-10-22T18:55:04.419Z]     "project-id": "fxci-production-level1-workers",

...(180 lines hidden)...

[vcs 2024-10-22T18:55:24.154Z] Cloning into '/builds/worker/checkouts/vcs/3rd_party/preprocess'...
[vcs 2024-10-22T18:55:25.244Z] Cloning into '/builds/worker/checkouts/vcs/inference/3rd_party/browsermt-marian-dev'...
[vcs 2024-10-22T18:55:26.765Z] Cloning into '/builds/worker/checkouts/vcs/inference/3rd_party/ssplit-cpp'...
[vcs 2024-10-22T18:55:27.281Z] Submodule path '3rd_party/browsermt-marian-dev': checked out '11c6ae7c46be21ef96ed10c60f28022fa968939f'
[vcs 2024-10-22T18:55:27.295Z] Submodule path '3rd_party/extract-lex': checked out '42fa605b53f32eaf6c6e0b5677255c21c91b3d49'
[vcs 2024-10-22T18:55:27.309Z] Submodule path '3rd_party/fast_align': checked out 'cab1e9aac8d3bb02ff5ae58218d8d225a039fa11'
[vcs 2024-10-22T18:55:27.336Z] Submodule path '3rd_party/kenlm': checked out 'bbf4fc511266c5d4515047055d7bdec659a6e158'
[vcs 2024-10-22T18:55:27.442Z] Submodule path '3rd_party/marian-dev': checked out 'e8a1a2530fb84cbff7383302ebca393e5875c441'
[vcs 2024-10-22T18:55:27.466Z] Submodule path '3rd_party/preprocess': checked out '64307314b4d5a9a0bd529b5c1036b0710d995eec'
[vcs 2024-10-22T18:55:27.535Z] Submodule path 'inference/3rd_party/browsermt-marian-dev': checked out '2781d735d4a10dca876d61be587afdab2726293c'
[vcs 2024-10-22T18:55:27.552Z] Submodule path 'inference/3rd_party/ssplit-cpp': checked out 'a311f9865ade34db1e8e080e6cc146f55dafb067'
[vcs 2024-10-22T18:55:27.552Z] cleaning git checkout...
[vcs 2024-10-22T18:55:27.552Z] executing ['git', 'clean', '-nxdff']
[vcs 2024-10-22T18:55:27.555Z] removing []
[vcs 2024-10-22T18:55:27.555Z] successfully cleaned git checkout!
[vcs 2024-10-22T18:55:27.557Z] TinderboxPrint:<a href='https://github.com/mozilla/firefox-translations-training/commit/2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c' title='Built from firefox-translations-training commit 2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c'>2d0fb8cf64a76c1014ef9eeac29bbe483d5f3b6c</a>
[setup 2024-10-22T18:55:27.557Z] MOZ_FETCHES_DIR is /builds/worker/fetches
[fetches 2024-10-22T18:55:27.557Z] fetching artifacts
[fetches 2024-10-22T18:55:27.557Z] executing ['/usr/bin/python3', '-u', '/usr/local/bin/fetch-content', 'task-artifacts']
attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KLqKD5jeRhqciLg3OwUmxQ/artifacts/public/build/corpus.priors to /builds/worker/fetches/corpus.priors
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KLqKD5jeRhqciLg3OwUmxQ/artifacts/public/build/corpus.priorsattempt 1/5

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.en.zst to /builds/worker/fetches/corpus.en.zstattempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.ru.zst to /builds/worker/fetches/corpus.ru.zst

attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOOnKkkKSwe1Qwikcu3ghg/artifacts/public/build/extract_lex.tar.zst to /builds/worker/fetches/extract_lex.tar.zstDownloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.ru.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.en.zst

Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOOnKkkKSwe1Qwikcu3ghg/artifacts/public/build/extract_lex.tar.zstattempt 1/5attempt 1/5
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VwyvswswQmecIxHim7KG7w/artifacts/public/build/marian.tar.zst to /builds/worker/fetches/marian.tar.zst


Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EBhuRvmXRg-JQpAJoHsgsA/artifacts/public/build/fast-align.tar.zst to /builds/worker/fetches/fast-align.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EBhuRvmXRg-JQpAJoHsgsA/artifacts/public/build/fast-align.tar.zst
Downloading https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VwyvswswQmecIxHim7KG7w/artifacts/public/build/marian.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EBhuRvmXRg-JQpAJoHsgsA/artifacts/public/build/fast-align.tar.zst resolved to 722752 bytes with sha256 f3b22724daab04267d54ff010187e6014c149ed32da9a2f5d3be9800d3bbdbac in 0.054s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/EBhuRvmXRg-JQpAJoHsgsA/artifacts/public/build/fast-align.tar.zst
Extracting /builds/worker/fetches/fast-align.tar.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KLqKD5jeRhqciLg3OwUmxQ/artifacts/public/build/corpus.priors resolved to 257875 bytes with sha256 892d1b8baaea9c1987cb1fef5a7e0e0a995f4db1b445beef9ffc23d772ae5495 in 0.058s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/KLqKD5jeRhqciLg3OwUmxQ/artifacts/public/build/corpus.priors
Extracting /builds/worker/fetches/corpus.priors to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOOnKkkKSwe1Qwikcu3ghg/artifacts/public/build/extract_lex.tar.zst resolved to 620577 bytes with sha256 4ac8d20c1f44c9c6de6482502b9fcf83d6114d3664c34b40690d92711d6a48b9 in 0.058s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/JOOnKkkKSwe1Qwikcu3ghg/artifacts/public/build/extract_lex.tar.zst
Extracting /builds/worker/fetches/extract_lex.tar.zst to /builds/worker/fetches
/builds/worker/fetches/fast-align.tar.zst extracted in 0.009s
Removing /builds/worker/fetches/fast-align.tar.zst
/builds/worker/fetches/extract_lex.tar.zst extracted in 0.007s
Removing /builds/worker/fetches/extract_lex.tar.zst
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.ru.zst resolved to 83739 bytes with sha256 b397329e91940d1b6e7c0a4674e2f945ec53995240446d6dcb3e4b186b3ee56e in 0.085s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.ru.zst
Extracting /builds/worker/fetches/corpus.ru.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.en.zst resolved to 7635 bytes with sha256 1a3af6ee2b6325961b0f22b55f12ca7824c54812cc6004b1ad574272d880ac7a in 0.107s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/V6AHwFI2SFGzHdERkomBNA/artifacts/public/build/corpus.en.zst
Extracting /builds/worker/fetches/corpus.en.zst to /builds/worker/fetches
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VwyvswswQmecIxHim7KG7w/artifacts/public/build/marian.tar.zst resolved to 684170205 bytes with sha256 4c7d64fbb4d7a1edae9a834c53b4de9ac716972ffb460b9d8439fa68a85f5d68 in 14.550s
Verified size of https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VwyvswswQmecIxHim7KG7w/artifacts/public/build/marian.tar.zst
Extracting /builds/worker/fetches/marian.tar.zst to /builds/worker/fetches
/builds/worker/fetches/marian.tar.zst extracted in 5.309s
Removing /builds/worker/fetches/marian.tar.zst
PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"name": "fetch_content", "value": 19.939762560000005, "lowerIsBetter": true, "shouldAlert": false, "subtests": []}]}
[fetches 2024-10-22T18:55:47.574Z] finished fetching artifacts
[task 2024-10-22T18:55:47.574Z] executing ['bash', '-c', 'pip3 install --upgrade pip setuptools && pip3 install -r $VCS_PATH/pipeline/alignments/requirements/alignments.txt && export BIN=$MOZ_FETCHES_DIR && export PATH=$PATH:$MOZ_FETCHES_DIR && export PYTHONPATH=$PYTHONPATH:$VCS_PATH && python3 $VCS_PATH/pipeline/alignments/align.py --corpus_src=$MOZ_FETCHES_DIR/corpus.ru.zst --corpus_trg=$MOZ_FETCHES_DIR/corpus.en.zst --output_path=$TASK_WORKDIR/artifacts/corpus.aln.zst --tokenization=moses --priors_input_path=$MOZ_FETCHES_DIR/corpus.priors']
[task 2024-10-22T18:55:47.868Z] WARNING: The directory '/builds/worker/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
[task 2024-10-22T18:55:47.869Z] Defaulting to user installation because normal site-packages is not writeable
[task 2024-10-22T18:55:47.892Z] Requirement already satisfied: pip in /usr/lib/python3/dist-packages (22.0.2)
[task 2024-10-22T18:55:48.037Z] Collecting pip
[task 2024-10-22T18:55:48.112Z]   Downloading pip-24.2-py3-none-any.whl (1.8 MB)
[task 2024-10-22T18:55:48.273Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 11.4 MB/s eta 0:00:00
[task 2024-10-22T18:55:48.282Z] Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (59.6.0)
[task 2024-10-22T18:55:48.535Z] Collecting setuptools
[task 2024-10-22T18:55:48.546Z]   Downloading setuptools-75.2.0-py3-none-any.whl (1.2 MB)
[task 2024-10-22T18:55:48.604Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 22.3 MB/s eta 0:00:00
[task 2024-10-22T18:55:48.739Z] Installing collected packages: setuptools, pip
[task 2024-10-22T18:55:51.934Z] Successfully installed pip-24.2 setuptools-75.2.0
[task 2024-10-22T18:55:52.538Z] WARNING: The directory '/builds/worker/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
[task 2024-10-22T18:55:52.539Z] Defaulting to user installation because normal site-packages is not writeable
[task 2024-10-22T18:55:52.678Z] Collecting certifi==2024.7.4 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 7))
[task 2024-10-22T18:55:52.704Z]   Downloading certifi-2024.7.4-py3-none-any.whl (162 kB)
[task 2024-10-22T18:55:52.813Z] Collecting charset-normalizer==3.3.2 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 11))
[task 2024-10-22T18:55:52.824Z]   Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)
[task 2024-10-22T18:55:53.050Z] Collecting cython==3.0.9 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 103))
[task 2024-10-22T18:55:53.060Z]   Downloading Cython-3.0.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
[task 2024-10-22T18:55:53.089Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.6/3.6 MB 128.4 MB/s eta 0:00:00
[task 2024-10-22T18:55:53.178Z] Collecting eflomal==1.0.0b1 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 163))
[task 2024-10-22T18:55:53.187Z]   Downloading eflomal-1.0.0b1.tar.gz (102 kB)
[task 2024-10-22T18:55:53.200Z]   Installing build dependencies: started
[task 2024-10-22T18:55:58.904Z]   Installing build dependencies: finished with status 'done'
[task 2024-10-22T18:55:58.905Z]   Getting requirements to build wheel: started
[task 2024-10-22T18:56:00.083Z]   Getting requirements to build wheel: finished with status 'done'
[task 2024-10-22T18:56:00.084Z]   Preparing metadata (pyproject.toml): started
[task 2024-10-22T18:56:00.582Z]   Preparing metadata (pyproject.toml): finished with status 'done'
[task 2024-10-22T18:56:00.601Z] Collecting idna==3.8 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 166))
[task 2024-10-22T18:56:00.609Z]   Downloading idna-3.8-py3-none-any.whl (66 kB)
[task 2024-10-22T18:56:00.765Z] Collecting numpy==1.26.4 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 170))
[task 2024-10-22T18:56:00.773Z]   Downloading numpy-1.26.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
[task 2024-10-22T18:56:00.822Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 397.2 MB/s eta 0:00:00
[task 2024-10-22T18:56:00.895Z] Collecting opus-fast-mosestokenizer==0.0.8.5 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 208))
[task 2024-10-22T18:56:00.904Z]   Downloading opus_fast_mosestokenizer-0.0.8.5-cp310-cp310-manylinux1_x86_64.whl (837 kB)
[task 2024-10-22T18:56:00.908Z]      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 837.2/837.2 kB 404.9 MB/s eta 0:00:00
[task 2024-10-22T18:56:00.944Z] Collecting requests==2.31.0 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 223))
[task 2024-10-22T18:56:00.952Z]   Downloading requests-2.31.0-py3-none-any.whl (62 kB)
[task 2024-10-22T18:56:01.029Z] Collecting tqdm==4.66.4 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 227))
[task 2024-10-22T18:56:01.038Z]   Downloading tqdm-4.66.4-py3-none-any.whl (78 kB)
[task 2024-10-22T18:56:01.081Z] Collecting urllib3==2.2.2 (from -r /builds/worker/checkouts/vcs/pipeline/alignments/requirements/alignments.txt (line 231))
[task 2024-10-22T18:56:01.099Z]   Downloading urllib3-2.2.2-py3-none-any.whl (121 kB)
[task 2024-10-22T18:56:01.127Z] WARNING: The candidate selected for download or install is a yanked version: 'eflomal' candidate (version 1.0.0b1 at https://files.pythonhosted.org/packages/1d/cc/2cdf9c90a6bd8fd7323b47bf5a60fe738bfe8465844a2136b0543f5094e8/eflomal-1.0.0b1.tar.gz (from https://pypi.org/simple/eflomal/))
[task 2024-10-22T18:56:01.127Z] Reason for being yanked: <none given>
[task 2024-10-22T18:56:01.128Z] Building wheels for collected packages: eflomal
[task 2024-10-22T18:56:01.129Z]   Building wheel for eflomal (pyproject.toml): started
[task 2024-10-22T18:56:05.413Z]   Building wheel for eflomal (pyproject.toml): finished with status 'done'
[task 2024-10-22T18:56:05.415Z]   Created wheel for eflomal: filename=eflomal-1.0.0b1-cp310-cp310-linux_x86_64.whl size=319766 sha256=26dba63d27004937c0caed44741b266bab15a7095c3c2ac6e4f96ba3177b354e
[task 2024-10-22T18:56:05.415Z]   Stored in directory: /tmp/pip-ephem-wheel-cache-prsszt3s/wheels/b4/0c/55/4a2c6f00793a0d2fdaef4376b8cb3d9c6ae8df16366d6e31d1
[task 2024-10-22T18:56:05.417Z] Successfully built eflomal
[task 2024-10-22T18:56:05.458Z] Installing collected packages: urllib3, tqdm, opus-fast-mosestokenizer, numpy, idna, cython, charset-normalizer, certifi, requests, eflomal
[task 2024-10-22T18:56:07.965Z] Successfully installed certifi-2024.7.4 charset-normalizer-3.3.2 cython-3.0.9 eflomal-1.0.0b1 idna-3.8 numpy-1.26.4 opus-fast-mosestokenizer-0.0.8.5 requests-2.31.0 tqdm-4.66.4 urllib3-2.2.2
[task 2024-10-22T18:56:08.406Z] [alignments] Running with arguments: ['/builds/worker/checkouts/vcs/pipeline/alignments/align.py', '--corpus_src=/builds/worker/fetches/corpus.ru.zst', '--corpus_trg=/builds/worker/fetches/corpus.en.zst', '--output_path=/builds/worker/artifacts/corpus.aln.zst', '--tokenization=moses', '--priors_input_path=/builds/worker/fetches/corpus.priors']
[task 2024-10-22T18:56:08.409Z] [alignments] Starting generating alignments.
[task 2024-10-22T18:56:08.409Z] [alignments] Decompressing file /builds/worker/fetches/corpus.ru.zst
[task 2024-10-22T18:56:08.410Z] 
[task 2024-10-22T18:56:08.411Z] etches/corpus.ru.zst : 0 MB...     
[task 2024-10-22T18:56:08.411Z]                                                                                
[task 2024-10-22T18:56:08.411Z] /builds/worker/fetches/corpus.ru.zst: 314990 bytes 
[task 2024-10-22T18:56:08.412Z] [alignments] Decompressing file /builds/worker/fetches/corpus.en.zst
[task 2024-10-22T18:56:08.413Z] 
[task 2024-10-22T18:56:08.414Z] etches/corpus.en.zst : 0 MB...     
[task 2024-10-22T18:56:08.414Z]                                                                                
[task 2024-10-22T18:56:08.414Z] /builds/worker/fetches/corpus.en.zst: 348320 bytes 
[task 2024-10-22T18:56:08.414Z] [tokenizer] Tokenizing /builds/worker/fetches/corpus.ru with Moses tokenizer
[task 2024-10-22T18:56:08.443Z] 
[task 2024-10-22T18:56:08.622Z] 0it [00:00, ?it/s]
[task 2024-10-22T18:56:08.622Z] 1609it [00:00, 8982.40it/s]
[task 2024-10-22T18:56:08.622Z] [tokenizer] Tokenizing /builds/worker/fetches/corpus.en with Moses tokenizer
[task 2024-10-22T18:56:08.646Z] 
[task 2024-10-22T18:56:08.818Z] 0it [00:00, ?it/s]
[task 2024-10-22T18:56:08.818Z] 1609it [00:00, 9352.90it/s]
[task 2024-10-22T18:56:08.893Z] [alignments] Splitting corpus into parts
[task 2024-10-22T18:56:08.911Z] [alignments] Processing part aa
[task 2024-10-22T18:56:08.911Z] [alignments] Using provided priors: /builds/worker/fetches/corpus.priors
[task 2024-10-22T18:56:08.911Z] [alignments] Calculating alignments...
[task 2024-10-22T18:56:08.924Z] [eflomal] Prepared 1609 sentences for alignment
[task 2024-10-22T18:56:08.924Z] [eflomal] Reading lexical priors...
[task 2024-10-22T18:56:08.940Z] [eflomal] 66 (of 4714) pairs of lexical priors used
[task 2024-10-22T18:56:08.944Z] /builds/worker/.local/lib/python3.10/site-packages/eflomal/bin/eflomal -m 3 -s /tmp/tmphrau3g7j -t /tmp/tmperv5_7tg -n 3 -N 0.2 -1 31 -2 31 -3 125 -f /builds/worker/artifacts/tmp_aln/aln.fwd.aa -r /builds/worker/artifacts/tmp_aln/aln.rev.aa -p /tmp/tmp3p_w644n
[task 2024-10-22T18:56:08.948Z] Read texts (1609 sentences): 0.003 s
[task 2024-10-22T18:56:08.948Z] Vocabulary sizes are 6898 (source), 214 (target)
[task 2024-10-22T18:56:08.952Z] Created alignment structures: 0.003 s
[task 2024-10-22T18:56:08.954Z] Randomized alignment: 0.002 s
[task 2024-10-22T18:56:08.954Z] Aligning with model 1 (31 iterations)
[task 2024-10-22T18:56:08.956Z] Created alignment structures: 0.007 s
[task 2024-10-22T18:56:08.957Z] Randomized alignment: 0.002 s
[task 2024-10-22T18:56:08.957Z] Aligning with model 1 (31 iterations)
[task 2024-10-22T18:56:09.043Z] Done: 0.086 s
[task 2024-10-22T18:56:09.043Z] Aligning with model 2 (31 iterations)
[task 2024-10-22T18:56:09.045Z] Done: 0.091 s
[task 2024-10-22T18:56:09.045Z] Aligning with model 2 (31 iterations)
[task 2024-10-22T18:56:09.152Z] Done: 0.109 s
[task 2024-10-22T18:56:09.152Z] Aligning with model 3 (125 iterations)
[task 2024-10-22T18:56:09.161Z] Done: 0.116 s
[task 2024-10-22T18:56:09.161Z] Aligning with model 3 (125 iterations)
[task 2024-10-22T18:56:09.667Z] Done: 0.506 s
[task 2024-10-22T18:56:09.681Z] Final argmax iteration: 0.014 s
[task 2024-10-22T18:56:09.681Z] Writing alignments to /builds/worker/artifacts/tmp_aln/aln.rev.aa for 1609 sentencess
[task 2024-10-22T18:56:10.024Z] Done: 0.871 s
[task 2024-10-22T18:56:10.041Z] Final argmax iteration: 0.017 s
[task 2024-10-22T18:56:10.041Z] Writing alignments to /builds/worker/artifacts/tmp_aln/aln.fwd.aa for 1609 sentencess
[task 2024-10-22T18:56:10.043Z] [alignments] Merging alignments: ['/builds/worker/artifacts/tmp_aln/aln.fwd.aa']
[task 2024-10-22T18:56:10.045Z] [alignments] Merging alignments: ['/builds/worker/artifacts/tmp_aln/aln.rev.aa']
[task 2024-10-22T18:56:10.046Z] [alignments] Symmetrizing alignments...
[task 2024-10-22T18:56:10.059Z] [alignments] Remapping alignments to whitespace tokenization
[task 2024-10-22T18:56:10.095Z] 
[task 2024-10-22T18:56:10.132Z] 0it [00:00, ?it/s]
[task 2024-10-22T18:56:10.132Z] 1609it [00:00, 43657.88it/s]
[task 2024-10-22T18:56:10.139Z] [alignments] Compressing final alignments
[task 2024-10-22T18:56:10.141Z] 
[task 2024-10-22T18:56:10.141Z]                                                                                
[task 2024-10-22T18:56:10.141Z] Read :  0 /  0 MB ==> 31%
[task 2024-10-22T18:56:10.141Z]                                                                                
[task 2024-10-22T18:56:10.141Z] /builds/worker/artifacts/tmp_aln/aln.remapped : 31.36%   ( 19320 =>   6058 bytes, /builds/worker/artifacts/tmp_aln/aln.remapped.zst) 
[task 2024-10-22T18:56:10.142Z] [alignments] Finished generating alignments.
[fetches 2024-10-22T18:56:10.167Z] removing /builds/worker/fetches
[fetches 2024-10-22T18:56:10.426Z] finished
+ exit_code=0
+ podman cp taskcontainer:/builds/worker/artifacts artifact0
+ podman rm taskcontainer
92ece85abe074c8254ae6626abb115ed958bfac1affb98e283a8ed6b85410bbf
+ exit 0
[taskcluster 2024-10-22T18:56:12.018Z]    Exit Code: 0
[taskcluster 2024-10-22T18:56:12.018Z]    User Time: 8.552944s
[taskcluster 2024-10-22T18:56:12.018Z]  Kernel Time: 3.787036s
[taskcluster 2024-10-22T18:56:12.018Z]    Wall Time: 1m1.964788779s
[taskcluster 2024-10-22T18:56:12.018Z]       Result: SUCCEEDED
[taskcluster 2024-10-22T18:56:12.019Z] === Task Finished ===
[taskcluster 2024-10-22T18:56:12.019Z] Task Duration: 1m1.965128477s
[taskcluster 2024-10-22T18:56:12.049Z] Uploading artifact public/build/corpus.aln.zst from file /task_172962320959689/artifact0/corpus.aln.zst with content encoding "identity", mime type "application/zstd" and expiry 2025-01-20T18:00:07.983Z
[taskcluster 2024-10-22T18:56:12.221Z] [mounts] Preserving cache: Moving "/task_172962320959689/cache0" to "caches/Sqf4BixPQre3mPCFGsS65A"
[taskcluster 2024-10-22T18:56:12.269Z] [mounts] Denying task_172962320959689 access to 'caches/Sqf4BixPQre3mPCFGsS65A'
[taskcluster 2024-10-22T18:56:12.302Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-01-20T18:00:07.983Z