Skip to content

Latest commit

 

History

History
173 lines (134 loc) · 7.25 KB

lang_python.adoc

File metadata and controls

173 lines (134 loc) · 7.25 KB
Raw

Kaitai Struct: Python notes

Table of Contents

Overview

Supported Python versions

Kaitai Struct supports Python 3.4 and later as well as Python 2.7. It is strongly recommended to use Python 3.8 or newer, because Python 3.7 and 2.7 have reached end-of-life and no longer receive official support from the Python developers.

Kaitai Struct compiler invocation

To generate Python modules, Kaitai Struct compiler should be invoked with -t python option:

kaitai-struct-compiler -t python [--python-package .] my_spec.ksy

This normally generates my_spec.py parser module.

--python-package is an optional CLI arg allows you to select a parent package name used by one generated file to import symbols from another one.

It is recommended to just follow the algorithm:

  • Determine if you need --python-package:

    • If you don’t intend to embed the KSC-generated files and just want to play a bit in a REPL shell (IPython/Jupyter) in the same dir, omit this arg.

    • If you intend to embed the compilation result into an own package, use --python-package ..

  • Put all the KSC-generated code into an own subpackage, in example kaitai. If your build process regenerates the files on every build, you may want (and probably should, if you don’t care about users being unable to build your package without KSC) .gitignore that dir in order not to junk VCS history.

If you don’t want to follow the algorithm blindly, just read about the actions involved in the Net and read the discussion in the PR. Don’t send us issues about the recommendations/generated code contradicting PEP 8 - we don’t stick to it. If you want PEP 8 compliant code, you can apply external tools like autopep8 and black to generated code.

Installing runtime library

Code generated by Kaitai Struct compiler requires additional runtime library (kaitaistruct at PyPI, sources at GitHub).

Runtime library API version must match compiler version that you’re using (i.e. if you use ksc v0.9, then you must use compatible v0.9 runtime).

If you are using snapshot KSC, you should also use snapshot runtime.

Using release version

With stable version of Kaitai Struct compiler, one can use the following methods:

  • On Debian / Ubuntu, one can install it from apt repositories:

    • For Python 3.x:

      sudo apt-get install python3-kaitaistruct

      Reference: package info for Debian, package info for Ubuntu

    • For Python 2.x:

      sudo apt-get install python-kaitaistruct

      Reference: package info for Debian, package info for Ubuntu

  • Add the following dependency specifier into the package configuration of the packaging tool you use:

    kaitaistruct

    and then run python3 -m pip install --upgrade -e ./ in the dir with your project in order to install your package and all its dependencies in an editable way. If you don’t want to package your software, but only install its dependencies, add kaitaistruct into requirements.txt.

  • Otherwise, you can just install the package manually using pip (Python package manager):

    python3 -m pip install --upgrade kaitaistruct

Using latest (snapshot) version

If you’re using latest (unreleased aka "bleeding edge" aka "unstable" aka "snapshot" aka "nightly") build of Kaitai Struct compiler, that will require latest runtime libraries as well:

  • You can a make pip to use a snapshot version of the runtime by adding a PEP 508 dependency specifier into the config of your favorite build tool:

    kaitaistruct @ git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git

  • Otherwise, you can just install the package manually using pip (Python package manager):

    python3 -m pip install --upgrade --pre git+https://github.com/kaitai-io/kaitai_struct_python_runtime.git

Installing dependencies for legacy Pythons manually

Enums implementation in Python requires enum support as described in PEP-0435. This support is available out of the box in standard libraries since Python v3.4, or since v2.4 using enum34 polyfill.

  • One usually doesn’t need to install this lib manually, pip will install it automatically if it is needed.

  • One can also install it manually:

    • One can use just python -m pip install enum34. Don’t do it on later Pythons or the apps using just enum will be broken!

    • On Debian / Ubuntu, this library can be installed with sudo apt-get install -y python-enum34.

Automation of compilation process

Setuptools is the de-facto standard way of building Python packages. This library allows extension with plugins. https://gitlab.com/KOLANICH has developed an extension to setuptools (GitHub mirror is also present), allowing you to automate compilation of *.ksy files as the part of Python module build process. The library provides a mostly declarative syntax to facilitate installation: you add a section [tool.kaitai] to your pyproject.toml where you enumerate the files that you want compiled, output directory, compilation flags, source code. Post-compilation transformations are currently supported only via legacy syntax via setup.py. The links to examples can be found in the project docs.

Here are the docs in JSON Schema format.

Please remember that this lib is very unstable and very likely to be improved in a way that changes its API (but due to the API simplicity it’d be easy to fix your code), but I guess it’s much better to use this than to reinvent the wheel. Contributions are welcome!

Please note that this tool is not available on PyPI and should be installed manually, since currently build_system of pyproject.toml doesn’t support PEP 508 specifiers.