Optimize alignments (#703)

* Add fast Moses tokenizer * Tokenize corpus and remap alignments * Use moses tokenizer in taskcluster * Add tests for index mapping * Add packaged to build fast moses tokenizer * Fix an issue with LD_LIBRARY_PATH for fast moses tokenizer * Rename tokenization function * Rename chunking parameter * Relock poetry * Rerun linter
mozilla · Jun 27, 2024 · 1ba10cf · 1ba10cf
1 parent 43967d0
commit 1ba10cf
Show file tree

Hide file tree

Showing 13 changed files with 819 additions and 482 deletions.
diff --git a/docker/Dockerfile b/docker/Dockerfile
@@ -27,6 +27,10 @@ RUN apt-get update -qq \
                           libopenblas-dev \
                           openssl \
                           libssl-dev  \
+                          pkg-config \
+                          libre2-dev \
+                          libglib2.0-dev \
+                          python3-pybind11  \
          && apt-get clean
 
 RUN mkdir /builds/worker/tools && \