Modernize font creation #624

gagern · 2017-01-11T20:50:40Z

Automate the build process, instead of writing a complicated manual
Update sources used by the docker image away from google code
Exclude MathJax-dev checkout from docker image, to allow image reuse
Try to keep the individual image layers small by combining RUN commands and removing temporary files used only during each individual step

* Automate the build process, instead of writing a complicated manual * Update sources used by the docker image away from google code * Exclude MathJax-dev checkout from docker image, to allow image reuse * Try to keep the individual image layers small by combining RUN commands and removing temporary files used only during each individual step

kevinbarabash · 2017-01-11T23:21:09Z

@gagern cool. Is there a way to supply a local path for MathJax-dev? If not, pushing to a remote branch is fine since people will have to do that anyways to create a PR.

gagern · 2017-01-12T06:44:53Z

I was wondering about the best approach for local repos, too. Theoretically it should be possible to run a git server locally, and have the docker connect to that. But seeing all the trouble the screenshotter had to go to in order to identify an ip address to be used for this, I'd rather avoid that. Perhaps we should drop the git checkout altogether, and instead download the tarball, or offer a way to copy a local tarball into the container instead. Then we can have the outer script create such a tarball in case the argument is a local directory, or use it as is if it is a file, or download it if it is a url.

Also use hash of Dockerfile to compute image tag, in order to ensure creation of a new image if the Dockerfile changes.

kevinbarabash · 2017-01-12T15:06:02Z

Another option might be to install whatever dependencies MathJax-dev needs to run the make scripts correctly during development.

gagern · 2017-01-12T15:30:37Z

I'll leave that to you. Personally I think having both the docker approach and a non-dockerized version would be nice. So I'd say merge this and if one of us finds the time we can write a Makefile to build all dependencies and use that.

kevinbarabash · 2017-01-15T23:28:05Z

dockers/MathJaxFonts/Dockerfile

+RUN wget "https://archive.apache.org/dist/xmlgraphics/batik/batik-1.7.zip" \
+ && unzip batik-*.zip batik-*/batik-ttf2svg.jar \
+ && mv batik-*/batik-ttf2svg.jar /usr/share/java/ \
+ && rm -r batik-*


I'm curious about the benefit of combining multiple commands vs. running them separate?

Each RUN command creates a separate layer of docker file system, while commands combined into one RUN will just contribute one layer that contains their combined effect. In this case, the resulting layer should only contain /usr/share/java/batik-ttf2svg.jar, as everything else is created and removed in the same RUN. This keeps the number of layers down and the total size of the image small, in order to conserve resources.

kevinbarabash · 2017-01-15T23:28:57Z

dockers/MathJaxFonts/Dockerfile

+ && sed -i "1s/^/#include <cstddef>/" ttf2eot-*/OpenTypeUtilities.h \
+ && make -C ttf2eot-*/ \
+ && mv ttf2eot-*/ttf2eot /usr/bin/ \
+ && rm -r ttf2eot*


The rm commands are necessary are they?

Not necessary for the image to perform as intended, no. The goal is again to clean up all the build-time resources once the desired command has been installed, to keep the size of the layer down.

kevinbarabash · 2017-01-15T23:29:21Z

dockers/MathJaxFonts/Dockerfile

-RUN cp MathJax-dev/default.cfg MathJax-dev/custom.cfg
-RUN make -C MathJax-dev custom.cfg.pl
-
+RUN wget "https://github.com/google/woff2/archive/d9a74803fa884559879e3205cfe6f257a2d85519.tar.gz" -O woff2.tar.gz \


kevinbarabash · 2017-01-15T23:34:58Z

dockers/MathJaxFonts/buildFonts.sh

+                        popd
+                    elif [[ -f "$1" ]]; then
+                        FILE="$1"
+                    elif [[ "$1" = http?(s)://* ]]; then


I tried running ./buildFonts.sh /local/path/to/MathJax-dev and got the following error:

./buildFonts.sh: line 97: syntax error in conditional expression: unexpected token `('

Can you please try whether a shopt -s extglob at the beginning of the script fixes this?

I gave that a try, but that gives the same output. :(

Well, we can make that just cover http for now, or use [[ "$1" = http://* ]] || [[ "$1" == https://* ]] if we can't get this glob syntax to work as intended.

Sorry... I pasted it in the wrong .sh file. It's running now. I'll post back once it completes.

It worked! All of the font files were modified. Some of those changes might be due to version changes in the deps and/or running in the docker vs. OS/X itself.

I'll check whether running the tool on my system will modify them again or not.

I was running it against HEAD on master of khan/MathJax-dev which has a number commits since the last time the font files were generated.

I forget whether I ran the screenshot tests after all the font files were changed. I want to get this merged so I'm going to run the script again and then run the screenshot tests and if everything passes I'll merge this.

kevinbarabash · 2017-01-15T23:36:52Z

dockers/MathJaxFonts/buildFonts.sh

+tar cf /fonts.tar ${filetypes[*]/#/fonts/OTF/TeX/}"
+
+echo "Creating and starting docker container from image $IMAGE"
+CONTAINER=$(docker create "$IMAGE" /bin/sh -c "${CMDS}")


So the docker image created by the DockerFile is a base image that you then modify to run the commands to actually create the fonts, is that correct?

Not exactly. Docker uses an image as a template to create containers. So in our case the image contains all the tools we need to create a font, while the container then runs the named commands to actually perform the font creation. You can use the same image to build several fonts, instantiating one container each time you do so. The commands are executed once inside the container constructed here, but the file system that results from running these commands is not preserved, contrary to how images work.

A "container" seems like an "instance" from the description you gave. This is probably one of the reasons why I've had trouble grokking docker.

A container is an instance, yes. Container is to image like process is to executable file.

kevinbarabash · 2017-01-15T23:41:12Z

Just looking at all the deps, it's going to be a pain to create a cross platform script to install all the deps. I think I installed all the deps manually the last time I edited a font and it was a pain. This will be a huge improvement. Thanks for making it work with local directories.

kevinbarabash · 2017-08-20T05:37:09Z

I added shopt -s extglob at the start of buildFonts.sh as suggested in the comments above.

To test this I checked out Khan/MathJax-dev@677fd6a and ran

./dockers/MathJaxFonts/buildFonts.sh /absolute/path/toMathJax-dev

This resulted in all font files being updated. To verify that it was only the timestamps being updated I made sure I had a locally copy of https://github.com/Khan/KaTeX/blob/master/.gitattributes and followed its instructions modifying ~/.gitconfig appropriately.

Then I ran git diff static/fonts/*.ttf and git diff static/fonts/*.woff and saw that only the timestamps had been modified in the TTF and WOFF files. The output looked like:

diff --git a/static/fonts/KaTeX_Size4-Regular.woff b/static/fonts/KaTeX_Size4-Regular.woff
index 7826c6c..eeab3da 100644
--- a/static/fonts/KaTeX_Size4-Regular.woff
+++ b/static/fonts/KaTeX_Size4-Regular.woff
@@ -63,12 +63,12 @@
     <!-- Most of this table will be recalculated by the compiler -->
     <tableVersion value="1.0"/>
     <fontRevision value="1.0"/>
-    <checkSumAdjustment value="0xee9e05b4"/>
+    <checkSumAdjustment value="0xd67232c6"/>
     <magicNumber value="0x5f0f3cf5"/>
     <flags value="00000000 00011111"/>
     <unitsPerEm value="1000"/>
-    <created value="Mon Jun 29 22:52:19 2015"/>
-    <modified value="Mon Jun 29 22:52:19 2015"/>
+    <created value="Sun Aug 20 05:11:16 2017"/>
+    <modified value="Sun Aug 20 05:11:16 2017"/>
     <xMin value="-1903"/>
     <yMin value="-1250"/>
     <xMax value="1902"/>
@@ -2596,7 +2596,7 @@ http://scripts.sil.org/OFL
     <version value="1"/>
     <FFTimeStamp value="Tue Jul 31 14:57:19 2012"/>
     <sourceCreated value="Mon Oct 12 19:21:51 2009"/>
-    <sourceModified value="Mon Jun 29 22:52:00 2015"/>
+    <sourceModified value="Sun Aug 20 05:11:01 2017"/>
   </FFTM>
 
 </ttFont>

kevinbarabash

@gagern thanks for modernizing our font creating. I really like the ability to point this script at different folders. In the past I made changes to https://github.com/Khan/MathJax-dev from inside the docker instance which worked, but was cumbersome. This is much easier. Sorry for the delay in getting this reviewed.

gagern mentioned this pull request Jan 11, 2017

Implement \verb #614

Merged

Allow creating fonts from local directory

3998468

Also use hash of Dockerfile to compute image tag, in order to ensure creation of a new image if the Dockerfile changes.

kevinbarabash added the GH Review: review-needed label Jan 12, 2017

kevinbarabash reviewed Jan 15, 2017

View reviewed changes

edemaine mentioned this pull request May 10, 2017

Thickening fonts #695

Open

kevinbarabash added 2 commits August 20, 2017 01:14

Merge branch 'master' into fonts

420edf6

make glob syntax work on MacOS

1a257e4

kevinbarabash added GH review: accepted and removed GH Review: review-needed labels Aug 20, 2017

kevinbarabash self-requested a review August 20, 2017 05:37

kevinbarabash approved these changes Aug 20, 2017

View reviewed changes

kevinbarabash merged commit a9708a2 into KaTeX:master Aug 20, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modernize font creation #624

Modernize font creation #624

gagern commented Jan 11, 2017

kevinbarabash commented Jan 11, 2017

gagern commented Jan 12, 2017

kevinbarabash commented Jan 12, 2017

gagern commented Jan 12, 2017

kevinbarabash Jan 15, 2017

gagern Jan 16, 2017

kevinbarabash Jan 15, 2017

gagern Jan 16, 2017

kevinbarabash Jan 15, 2017

kevinbarabash Jan 15, 2017

gagern Jan 16, 2017

kevinbarabash Jan 17, 2017

gagern Jan 17, 2017

kevinbarabash Jan 17, 2017

kevinbarabash Jan 17, 2017

gagern Jan 17, 2017

kevinbarabash Jan 17, 2017

kevinbarabash Jul 9, 2017

kevinbarabash Jan 15, 2017

gagern Jan 16, 2017

kevinbarabash Jan 17, 2017

gagern Jan 17, 2017

kevinbarabash commented Jan 15, 2017

kevinbarabash commented Aug 20, 2017

kevinbarabash left a comment •

edited

Loading

Modernize font creation #624

Modernize font creation #624

Conversation

gagern commented Jan 11, 2017

kevinbarabash commented Jan 11, 2017

gagern commented Jan 12, 2017

kevinbarabash commented Jan 12, 2017

gagern commented Jan 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kevinbarabash commented Jan 15, 2017

kevinbarabash commented Aug 20, 2017

kevinbarabash left a comment • edited Loading

Choose a reason for hiding this comment

kevinbarabash left a comment •

edited

Loading