Make process improvements

[gouenji-shuuya]

• Move to bash as that was implicitly expected (ref. #vidyut on Discord)
• Some refactoring.
• Sub-make is correctly called when using make create_all_data.
• Use -jnproc in make.
• Ignore venv in git.

[akprasad]

Thank you for this! Forgive the late look.

[akprasad]

Thank you very much!

[akprasad]

My understanding is that -j controls the number of make recipes that run in parallel. If so, what is the benefit of using -j here?

[gouenji-shuuya]

Cargo can respect the jobserver settings. Refer rust-lang/rust#42682.

Even if the workflow is currently serial, there is a possibility of decoupling steps in future for faster builds.

For instance, create_kosha executes successfully before create_sandhi_rules, so there is no dependency (though the latter is quick). I think the cloning can also be parallelized, by use of recipes like this:

mkdir build

get_corpus_data:
	@if [[ -e "data/raw/dcs" ]]; then 				\
		echo "Training data already exists -- skipping fetch.";	\
	else 								\
		echo "Training data does not exist -- fetching.";	\
		mkdir -p "data/raw/dcs";				\
		git clone https://github.com/OliverHellwig/sanskrit.git	\
			--depth=1 build/dcs-data;			\
		mv build/dcs-data/dcs/data/conllu data/raw/dcs/conllu;	\
	fi


get_linguistic_data:
	@if [[ -e "data/raw/lex" ]]; then 				\
		echo "Lexical data already exists -- skipping fetch.";	\
	else 								\
		echo "Lexical data does not exist -- fetching.";	\
		mkdir -p "data/raw/lex";				\
		git clone https://github.com/sanskrit/data.git		\
			--depth=1 build/data-git;			\
		python3 build/data-git/bin/make_data.py 		\
			--make_prefixed_verbals;			\
		mv build/data-git/all-data/* data/raw/lex;		\
	fi

So the create_all_data.sh file really is a bottleneck. There are several make commands there and they execute one by one. Maybe a bit of parallelization will help, like in test/train?

[akprasad]

Sorry for not getting to this earlier!

Given the repo's evolution, there are some conflicts now, so I'll close the PR for the sake of housekeeping. If you're still interested, please resolve the conflicts and reopen the PR.