Timepix analysis & calibration

This repository contains code related to the data analysis of Timepix based gaseous detectors.

It contains code to calibrate a Timepix ASIC and perform event shape analysis of data to differentiate between background events (mainly cosmic muons) and signal events (X-rays).

The software in this repository is at the heart of my PhD thesis,

https://phd.vindaar.de

All data required to reproduce the results of my thesis, including results reconstructed with this code, can be found here:

https://zenodo.org/uploads/10521887

CAST

Many parts of this repository are specifically related to an InGrid based X-ray detector in use at the CERN Axion Solar Telescope: http://cast.web.cern.ch/CAST/

Project structure

This repository contains a big project combining several tools used to analyze data based on Timepix detectors as well as the CAST experiment.

NOTE: If you are mainly interested in using the reconstruction and analysis utilities for TOS data, the Analysis folder is what you’re looking for. See the Installation section for more information.

UPDATE <2024-09-11 Wed 16:10>: For a more updated overview of the repository structure, see the overview here: https://phd.vindaar.de/html/software.html#sec:appendix:timepix_analysis

• Analysis: Is the ingrid module, which contains the major programs of this repository raw_data_manipulation and reconstruction and to a lesser extent (depending on your use case) likelihood.
  • raw_data_manipulation: Reads folders of raw TOS data and outputs to a HDF5 file. Supported TOS data types:
    • old ~2015 era Virtex V6 TOS
    • current Virtex V6 TOS
    • current SRS TOS
  • reconstruction: Takes the output of the above program and performs reconstruction of clusters within the data, i.e. calculate geometric properties.
  • likelihood: Performs an event shape likelihood based analysis on the reconstructed data comparing with reference X-ray datasets.

  The other files in the folder are imported by these programs. An exception is skeleton program analysis, which will eventually become a wrapper of the other programs so that a nicer interface can be provided. A combination of a https://github.com/yglukhov/nimx based GUI with a readline based command line interface will be developed.

• InGridDatabase: A Nim program which provides, writes to and reads from the InGrid database. If the a folder describing the used detector is given to it (containing fsr, threshold, thresholdMeans, ToT calibration and / or SCurves and an additional file containing the chip name and additional information) it can be added to that database, which is simply a HDF5 file. The analysis progam makes use of this database to read calibration relevant data from it. TODO: link to explanation of required folder structure and add files / folders for current chips part of database.
• LogReader: A Nim tool to read and process CAST slow control and tracking log files. From these environmental sensors can be read if needed for data analysis puposes of CAST data as well as information about when solar trackings took place. If a HDF5 file is given the tracking information is added to the appropriate runs.
• NimUtil: The helpers nimble module. It contains general procedures used in the rest of the code, which are unrelated to CAST or Timepix detectors.
• Plotting: A Nim tool to create plots of Timepix calibration data. Reads from the InGrid database and plots ToT calibration (+ fits) and SCurves.
• README.org: this file. :)
• resources: Contains data, which is needed for analysis purposes, e.g. information about run numbers for data taking periods, the 2014/15 background rates etc. TODO: maybe add folders for known chips for InGrid database in here or at least an example directory.
• Tools: Directory for other smaller tools, for which a separate directory in the root of the repository does not make sense (either used too infrequently or are very specific and small tools).
• SolarEclipticToEarth: A simple Python tool part of solar chameleon analysis, which calculates the projection of the solar ecliptic onto Earth (chameleon flux potentially varies greatly depending on solar latitude). TODO: should be moved to Tools.
• Tests: Some very simple “test cases”, which typically just test new features separately from the rest of the analysis programs.
• VerticalShiftProblem: A simple Python tool to plot CAST log data to debug a problem with the belt, which slipped and caused misalignment. That problem has since been fixed. TODO: should be moved to Tools.
• CDL-RootToHdf5: A Python tool to (currently only) convert X-ray calibration data from the CAST detector lab from ROOT trees to HDF5 files. This could be easily extended to be a ROOT to HDF5 converter. TODO: this should be moved to Tools.
• endTimeExtractor: A Nim tool to extract the following information from a TOS run:
  • start of the Run
  • end of the Run
  • total run time

  and output it as an Org date string. TODO: should be moved to Tools.

• InGrid-Python: An (outdated) Python module containing additional functions used in the Nim analysis (fit of Fe55 spectrum and polya gas gain fit done using https://github.com/yglukhov/nimpy) and the Python plotting tool (see below).
• Figs: Plots, which are created from the analysis and have been used in a talk etc.

Installation

The project has only a few dependencies, which are all mostly easy to install. The Nim compiler is only a dependency to compile the Nim programs. But if you just wish to run the built binaries, the Nim compiler is not a dependency! E.g. compiling the raw_data_manipulation and reconstruction on an x86-64 linux system creates an (almost) dependency free binary.

The following shared libraries are linked at runtime:

• libhdf5
• libnlopt
• libmpfit
• libpcre

Their installation procedures are explained below.

For instructions to install the dependencies, see sec. #sec:deps. Note that for NLopt and MPFIT the build tool (see sec. #sec:install:build)

Nim

Nim is obviously required to compile the Nim projects of this repository. There are two approaches to install the Nim compiler. Using choosenim or cloning the Nim repository.

Clone the Nim repository and build the compiler locally

Go to some folder where you wish to store the Nim compiler, e.g. ~/src or create a folder if does not exist:

cd ~/
mkdir src

Please replace this directory by your choice in the rest of this section.

Then clone the git repository from GitHub (assuming git is installed):

git clone https://github.com/nim-lang/nim

enter the folder:

cd nim

and if you’re on a Unix system run:

sh build_all.sh

to build the compiler and additional tools like nimble (Nim’s package manager), nimsuggest (allows smart auto complete for Nim procs), etc.

Now add the following to your PATH variable in your shell’s configuration file, e.g. ~/.bashrc:

# add location of Nim's binaries to PATH
export PATH=$PATH:$HOME/src/nim/bin

and finally reload the shell via

source ~/.bashrc

or the appropriate shell config (or start a new shell).

With this approach updating the Nim compiler is trivial. First update your local git repository by pulling from the devel branch:

cd ~/src/nim
git pull origin devel

and finally use Nim’s build tool koch to update the Nim compiler:

./koch boot -d:release

Choosenim

An alternative to the above mentioned method is to use choosenim. Type the following into your terminal:

curl https://nim-lang.org/choosenim/init.sh -sSf | sh

Then follow the instructions and extend the PATH variable in your shell’s configuration file, e.g. ~/.bashrc. Finally reload that file via:

source ~/.bashrc

or simply start a new shell.

Install the TimepixAnalysis framework

Once the dependencies are installed, we can prepare the framework.

Preparing the TimepixAnalysis repository

We start by cloning the TimepixAnalysis repository somewhere, e.g.:

cd ~/src
git clone https://github.com/Vindaar/TimepixAnalysis

External dependency overview

On a fresh build of Debian, installing the following packages should have you covered in terms of dependencies:

git \
build-essential \
locate \
cmake \
libhdf5-dev \
libnlopt0 \
libnlopt-dev \
libcairo2-dev \
liblapack-dev \
libpcre3-dev \
libblosc1 \
libblosc-dev \
libgtk-3-dev \
libwebkit2gtk-4.0

locate may also be called mlocate. After installing it, make sure to run sudo updatedb to update the locate database.

Using the build tool to build (most) binaries

As of <2024-09-11 Wed 18:48> there is now a build tool to automate the compilation of (most; all relevant for the majority of users) binaries. In addition, the nimble (i.e. other Nim packages) dependencies are now fixed using a lock file, so that precisely the versions that are fixed are pulled and used. The latter should hopefully remove the occurrence for spurious compilation failures due to random version mismatches.

NOTE: <2024-09-12 Thu 12:19> As of right now, before running the nimble setup command below, you will need to manually install weave using nimble. There is a current issue causing the setup step to fail otherwise. So run:

nimble install weave

First we need to setup the Nimble dependencies:

cd Analysis
nimble setup

The command pulls all dependencies written in Analysis/nimble.lock. Afterwards, any compilation within the Analysis directory will only use those packages.

Next, we compile the build tool found in the root of the repository:

nim c buildTpa

./buildTpa -h

Usage:
  main [optional-params]
Options:
  -h, --help                         print this cligen-erated help

  --help-syntax                      advanced: prepend,plurals,..

  -l=, --locateTool= string "locate" Program to use to detect installed shared
                                     libraries on the system.

  -a, --allowClone   bool   true     If true will automatically clone a git
                                     repository and build shared library
                                     dependencies.

  -c=, --clonePath=  string "~/src"  Base path in which cloned directories will
                                     be installed.

  --args=            string ""       An additional command line argument string
                                     passed to all programs being compiled.

A few things to note:

• it tries to use locate to determine if the NLopt (libnlopt.so) and MPFIT (libmpfit.so) shared libraries can be found by ld.so
• if not as long as allowClone is true it will pull the code for these libraries and build them manually. In that case you still need to make sure the shared libraries can be found by ld.so on your system. By default (changed via --clonePath:/foo/bar) the repositories will be cloned into ~/src. See sec #sec:deps for more information.

All you need to do to build the binaries then is:

./buildTpa

It builds:

• parse_raw_tpx3
• raw_data_manipulation
• reconstruction
• runAnalysisChain
• fake_event_generator
• plotBackgroundRate
• plotBackgroundClusters
• plotData

Symbolic links to the location of the binaries are found in the ./bin directory of this repository. I recommend to add the path to that directory to your .zshrc / .bashrc (or whatever else your setup looks like).

Assuming TPA is located in ~/src/TimepixAnalysis that might look like:

Zsh

path+=$HOME/src/TimepixAnalysis/bin

Bash

export PATH=$PATH:$HOME/src/TimepixAnalysis/bin

Troubleshooting

If you run into problems trying to run one of the programs, it might be an easy fix.

An error such as

could not import: H5P_LST_FILE_CREATE_g

means that you compiled against a different HDF5 libary version than the one you have installed and is being tried to link at run time. Solution: compile the program with the -d:H5_LEGACY option, e.g.:

nim c -d:release --threads:on -d:H5_LEGACY raw_data_manipulation.nim

Another common problem is an error such as:

Error: cannot open file: docopt

This indicates that the module named docopt (only an example) could not be imported. Most likely a simple

nimble install docopt

would suffice. A call to nimble install with a package name will try to install a package from the path declared in the packages.json from here: https://github.com/nim-lang/packages/blob/master/packages.json

If you know that you need the #head of such a package, you can install it via

nimble install "docopt@#head"

Note: depending on your shell the " may not be needed. Note 2: instead of a simple package name, you may also hand nimble a full path to a git or mercurial repository. This is necessary in some cases, e.g. for the seqmath module, because we depend on a fork:

nimble install "https://github.com/vindaar/seqmath#head"

List of nimble dependencies

For a list of Nimble dependencies, see Analysis/ingrid.nimble.

Dependencies

HDF5

The major dependency of the Nim projects is HDF5. On a reasonably modern Linux distribution the libhdf5 should be part of the package repositories. The supported HDF5 versions are:

• 1.8: as a legacy mode, compile the Nim projects with -d:H5_LEGACY
• 1.10: the current HDF5 version and the default
• versions newer than 1.10 might require the -d:H5_FUTURE compilation flag.

If the HDF5 library is not available on your OS, you may download the binaries or the source code from the HDF group.

Ubuntu

On Ubuntu systems the following packages install all you need:

sudo apt-get install libhdf5-103 libhdf5-dev

In addition hdf5-tools might come in handy.

Void Linux

On Void you need:

sudo xbps-install -S hdf5 hdf5-devel

HDF View

HDF View is a very useful tool to look at HDF5 files with a graphical user interface. For HEP users: it is very similar to ROOT’s TBrowser.

Although many package repositories contain a version of HDF View, it is typically relatively old. The current version is version 3.0.0, which has some nice features, so it may be a good idea to install it manually.

NLopt

The NLopt library is a nonlinear optimization library, which is used in this project to fit the rotation angle of clusters and perform fits of the gas gain. The Nim wrapper is found at https://github.com/vindaar/nimnlopt. To build the C library follow the following instructions, (taken from here):

git clone https://github.com/stevengj/nlopt # clone the repository
cd nlopt
mkdir build
cd build
cmake ..
make
sudo make install

This introduces cmake as a dependency. Note that this installs the libnlopt.so system wide. If you do not wish to do that, you need to set your LD_PRELOAD_PATH accordingly!

Afterwards installation of the Nim nlopt module is sufficient (done automatically later).

Ubuntu

On Ubuntu systems the following packages install all you need:

sudo apt-get install libnlopt0 libnlopt-dev

Void Linux

sudo xbps-install -S nlopt nlopt-devel

MPfit

MPfit is a non-linear least squares fitting library. It is required as a dependency, since it’s used to perform different fits in the analysis. The Nim wrapper is located at https://github.com/vindaar/nim-mpfit. Compilation of this shared object is easiest by cloning the git repository of the Nim wrapper:

cd ~/src
git clone https://github.com/vindaar/nim-mpfit
cd nim-mpfit

And then build the library from the c_src directory as follows:

cd c_src
gcc -c -Wall -Werror -fpic mpfit.c mpfit.h
gcc -shared -o libmpfit.so mpfit.o

which should create the libmpfit.so. Now install that library system wide (again to avoid having to deal with LD_PRELOAD_PATH manually). Depending on your system, a suitable choice may be /usr/local/lib/:

sudo cp libmpfit.so /usr/local/lib

Finally, you may install the Nim wrapper via

nimble install

or tell nimble to point to the directory of the respitory here via:

nimble develop

The latter makes updating the package much easier, since updating the git repository is enough.

PCRE

Perl Compatible Regular Expressions (PCRE) is a library for regular expression matching. On almost any unix system, this library is already available. For some distributions (possibly some CentOS or Scientific Linux) it may not be.

This currently means you’ll have to build this library by yourself.

Different RE implementations

The default RE library in Nim is a wrapper around PCRE, due to PCRE’s very high performance. However, the performance critical parts do not depend on PCRE anymore. In principle we could thus replace the re module with https://github.com/nitely/nim-regex, a purely Nim based regex engine. PRs welcome! :)

Blosc

Blosc is a compression library used to compress the binary data in the HDF5 files. By default however Zlib compression is used, so this is typically not needed. If one wishes to read Timepix3 based HDF5 files, blosc support is mandatory (in Analysis/ingrid/parse_raw_tpx3.nim and after that in Analysis/ingrid/raw_data_manipulation.nim).

Ubuntu

On Ubuntu systems the following packages install all you need:

sudo apt-get install libblosc1 libblosc-dev

Void Linux

sudo xbps-install -S c-blosc c-blosc-devel

Usage

NOTE: <2024-09-11 Wed 16:16> This is also a bit outdated. Again, see the instructions from: https://phd.vindaar.de/html/software.html#sec:appendix:timepix_analysis

In general the usage of the analysis programs is straight forward and explained in the docstring, which can be echoed by calling a program with the -h or --help option:

raw_data_manipulation -h

would print:

Usage:
  main [REQUIRED,optional-params]
Version: 12e1820 built on: 2024-09-11 at 14:52:49
Options:
  -h, --help                               print this cligen-erated help

  --help-syntax                            advanced: prepend,plurals,..

  -p=, --path=        string      REQUIRED set path

  -r=, --runType=     RunTypeKind REQUIRED Select run type (Calib | Back | Xray)
                                           The following are parsed case insensetive:
                                             Calib = {"calib", "calibration", "c"}
                                             Back = {"back", "background", "b"}
                                             Xray = {"xray", "xrayfinger", "x"}

  -o=, --out=         string      ""       Filename of output file. If none
                                           given will be set to run_file.h5.

  -n, --nofadc        bool        false    Do not read FADC files.

  -i, --ignoreRunList bool        false    If set ignores the run list 2014/15
                                           to indicate using any rfOldTos run

  -c=, --config=      string      ""       Path to the configuration file to use. Default is config.toml
                                           in directory of this source file.

  --overwrite         bool        false    If set will overwrite runs already existing in the
                                           file. By default runs found in the file will be skipped.
                                           HOWEVER: overwriting is assumed, if you only hand a
                                           run folder!

  -t, --tpx3          bool        false    Convert data from a Timepix3 H5 file
                                           to TPA format instead of a Tpx1 run
                                           directory

  -k, --keepExtracted bool        false    If a .tar.gz archive is given for a run folder and this flag is true
                                           we won't remove the extracted archive afterwards.

  -e=, --extractTo=   string      "/tmp/"  If a .tar.gz archive is given extract
                                           the data to this directory.

similar docstrings are available for all programs.

In order to analyze a raw TOS run, we’d perform the following steps. The command line arguments are examples. Those required will be explained, for the others see the doc strings.

Raw data manipulation

Assuming we have a TOS run folder located in ~/data/Run_168_180702-15-24/:

raw_data_manipulation -p ~/data/Run_168_180702-15-24/ --runType=calibration --out=run_168.h5

where we give the runType (either calibration, background or X-ray finger run), which is useful to store in the resulting HDF5 file. For calibration runs several additional reconstruction steps are also done automatically during the reconstruction phase. We also store the data in a file called run168.h5. The default filename is run_file.h5. The HDF5 file now contains two groups (runs and reconstruction). runs stores the raw data. reconstruction is still mainly empty, some datasets are linked from the =runs group.

Alternatively you may also hand a directory, which contains several run folders. So if you had several runs located in ~/data, simply handing that would work. The program would work on all runs in data after another. Each run is stored in its own group in the resulting HDF5 file.

Reconstruction

Afterwards we go on to the reconstruction phase. Here the raw data is read back from the HDF5 file and clusters within events are separated and geometric properties calculated. This is done by:

reconstruction -i run_168.h5 --out reco_168.h5

After the reconstruction is done and depending on whether the run type is calibration or background / X-ray finger run, you can continue to calculate futher properties, e.g. the energy of all clusters.

The next step is to apply the ToT calibration to calculate the charge of all clusters via:

reconstruction -i reco_168.h5 --only_charge

Note: this requires an entry for your chip in the ingrid database. See below for more information.

Once the charges are calibrated, you may calculate the gas gain of the run via:

reconstruction -i run_168.h5 --only_gas_gain

A purely pixel based energy calculation is available via --only_energy=<energy_per_electron_in_eV> and a gas gain based one via --only_energy_from_e. However, the latter requires calibration runs that need to be analyzed before hand.

Likelihood

The likelihood analysis is the final step done in order to filter out events, which are not X-ray like, based on a likelihood cut or MLP classifier. The likelihood program however, needs two different input files. This is not yet as streamlined as it should be, which is why it’s not explained here in detail. Take a look at the docstring of the program or ask me (@Vindaar).

TODO: make the CDL data part of the repository somehow?

Adding a chip to the InGrid database

If you wish to perform charge calibration and from that energy calibration, you need to add your chip to the ingrid database.

See the explanation in InGridDatabase/README.org for details on how to do this. Existing data files are found in resources/ChipCalibrations/.

Plotting

There are a large number of different tools available to visualize the data created by the programs in this repository.

License

The code in this repository is published under the MIT license.