Document iree.build package

[ScottTodd]

These PRs have started implementing a new iree.build package:

• [python] Add an iree.build package with API/tooling for program building. #18630
• [iree.build] Implement iree-compile action. #18993
• [iree.build] Add turbine_generate iree.build rule. iree-turbine#249

Documentation tasks:

• Mention the new package in release notes (e.g. Release tracker - 2024/09 #18432)
• Add a section to https://iree-python-api.readthedocs.io/en/latest/index.html (source: https://github.com/iree-org/iree/tree/main/docs/api_docs/python)
• Add a sample (Colab notebook?), perhaps based on https://github.com/iree-org/iree/tree/main/compiler/bindings/python/test/build_api

Other related tasks:

• Use the new API in https://github.com/iree-org/iree-test-suites and other downstream projects

We could wait until more of the package architecture and usage solidifies, but trying it out early would help inform the design.

[stellaraccident]

Thanks.

It really needs a handful more patches and a first use or two for me to stand behind it. Soon. Need it for some very near term priorities.

[ScottTodd]

I started trying out the iree.build package over in https://github.com/iree-org/iree-test-suites on this branch today: https://github.com/ScottTodd/iree-test-suites/tree/onnx-ops-build, as an alternate way to implement the onnx operator test conversion script. The package feels a bit heavy for that usage, but I'm learning as I go and that will help with documentation. The turbine_generate rule from iree-org/iree-turbine#249 may help with writing PyTorch tests too.

We also have downstream usage in shortfin here: https://github.com/nod-ai/shark-ai/blob/main/shortfin/python/shortfin_apps/sd/components/builders.py.

[stellaraccident]

Thanks
This desperately needs to be finished. Just ran out of time while juggling priorities :/

[stellaraccident]

I think that there are likely some API wrappers that would make single shot cases work well, in addition to the more complicated pipeline cases

[ScottTodd]

For https://github.com/iree-org/iree-test-suites/blob/main/onnx_ops/import_onnx_tests.py, which I started migrating to use iree.build yesterday, the usage will be like:

def import_action(folder_with_upstream_onnx_test_case):
  # .onnx -> .mlir
  import_onnx(folder / "model.onnx")
  # input_0.pb, input_1.pb, ..., input_n.pb --> input_0.bin, ...
  # output_0.pb, ... -> output_0.bin, ...
  # run_module_io_flags.txt
  import_test_inputs_and_outputs_from_pb_to_bin(folder)

for test_case in test_cases: # around 1000 of these
  # Enqueue a build task for the test case. Each of these is independent.
  import_action(test_case)

So at least one custom build action to convert the .pb upstream test files to .bin IREE test files, then ~1000 groups of actions that can run in parallel. Right now I'm using with Pool(args.jobs) as pool to run the imports in parallel. The 1 folder with N files in it --> 1 folder with N+1 files in it relationship is an interesting case for the dependency modeling, compared to other flows that go 1 file --> 1 file --> 1 file.

Since the build graph goes wide instead of deep, I'm not sure if using iree.build makes a whole lot of sense, but uniformity and more code samples across our projects will help users too.

---

I also found a few spots where the code could be reworked to better support Windows, like

iree/compiler/bindings/python/iree/build/executor.py

Lines 323 to 334 in 1aada43

	def allocate_file(
	self, path: str, namespace: FileNamespace = FileNamespace.GEN
	) -> BuildFile:
	"""Allocates a file in the build tree with local path |path|.
	If |path| is absoluate (starts with '/'), then it is used as-is. Otherwise,
	it is joined with the path of this context.
	"""
	if not path.startswith("/"):
	path = join_namespace(self.path, path)
	build_file = BuildFile(executor=self.executor, path=path, namespace=namespace)
	return build_file

-        if not path.startswith("/"):
+        if not Path(path).is_absolute():

I'll send a PR once I'm a bit further along.

[ScottTodd]

Published initial API docs: https://iree-python-api.readthedocs.io/en/latest/compiler/build.html