Pipeline Features

Introduction

The pipeline offers some convenient features which will make your life easier.

Ctrl+C for Graceful Exits

If you launched a pipeline and need to stop for whatever reason, you can simply hit CTRL+C once. The pipeline will continue executing its current cycle and then proceeds with the finish procedure. This means that usually everything is cleaned up gracefully.

Here is an example where the pipeline while processing a ROOT file is suddenly interrupted. The resulting file is still readable and properly closed:

> analyse.py JTE_muon-CC_3-100GeV_401.root -o test.h5
Pipeline and module initialisation took 0.008s (CPU 0.000s).
loading root....  /afs/.in2p3.fr/system/amd64_sl7/usr/local/root/v5.34.23/
loading aalib...  /pbs/throng/km3net/src/Jpp/v9.0.8454/externals/aanet//libaa.so
--------------------------[ Blob     250 ]---------------------------
--------------------------[ Blob     500 ]---------------------------
--------------------------[ Blob     750 ]---------------------------
^C
==========================================
Got CTRL+C, waiting for current cycle...
Press CTRL+C again if you're in hurry!
==========================================
================================[ . ]================================
++ km3pipe.io.hdf5.HDF5Sink: HDF5 file written to: test.h5
============================================================
904 cycles drained in 21.279237s (CPU 20.320000s). Memory peak: 150.32 MB
  wall  mean: 0.023490s  medi: 0.020623s  min: 0.016564s  max: 1.540565s  std: 0.050769s
  CPU   mean: 0.022434s  medi: 0.020000s  min: 0.010000s  max: 0.990000s  std: 0.032663s
EventFile io / wall time = 1.72786 / 20.5438 (8.41063 % spent on io.)

Time / Memory / CPU Usage

Pass the Pipeline(timeit=True) argument to show detailed time and CPU

usage for every module in the pipeline, and overall memory:

Pipeline and module initialisation took 0.015s (CPU 0.013s).
--------------------------[ Blob     100 ]---------------------------
--------------------------[ Blob     200 ]---------------------------
--------------------------[ Blob     300 ]---------------------------
--------------------------[ Blob     400 ]---------------------------
--------------------------[ Blob     500 ]---------------------------
================================[ . ]================================
============================================================
500 cycles drained in 6.592123s (CPU 6.513101s). Memory peak: 312.91 MB
  wall  mean: 0.013066s  medi: 0.012130s  min: 0.004782s  max: 0.114435s  std: 0.007302s
  CPU   mean: 0.012913s  medi: 0.012014s  min: 0.004772s  max: 0.110865s  std: 0.007137s
HDF5Pump - process: 2.510s (CPU 2.456s) - finish: 0.003s (CPU 0.003s)
  wall  mean: 0.005021s  medi: 0.004509s  min: 0.003390s  max: 0.038976s  std: 0.002127s
  CPU   mean: 0.004912s  medi: 0.004434s  min: 0.003382s  max: 0.035576s  std: 0.001951s
StatusBar - process: 0.001s (CPU 0.001s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000205s  medi: 0.000223s  min: 0.000128s  max: 0.000280s  std: 0.000065s
  CPU   mean: 0.000206s  medi: 0.000223s  min: 0.000129s  max: 0.000282s  std: 0.000066s
filter_muons - process: 0.059s (CPU 0.059s) - finish: 0.000s (CPU 0.000s)
  wall  mean: 0.000119s  medi: 0.000104s  min: 0.000091s  max: 0.000326s  std: 0.000038s
  CPU   mean: 0.000119s  medi: 0.000104s  min: 0.000091s  max: 0.000327s  std: 0.000037s
DOMHits - process: 3.839s (CPU 3.821s) - finish: 0.038s (CPU 0.038s)
  wall  mean: 0.007678s  medi: 0.007034s  min: 0.000944s  max: 0.075129s  std: 0.005830s
  CPU   mean: 0.007642s  medi: 0.007021s  min: 0.000945s  max: 0.074965s  std: 0.005801s

Services

Modules can provide services with can be used in the whole pipeline, without the need to attach anything to the blob.

This part of the documentation is WIP, but the feature is there ;)

Automatic HDF5 Writing

The Table class has the attribute h5loc which can also be set upon initialisation. It is the target location in an HDF5 file if a km3pipe.io.hdf5.HDF5Sink is attached to the pipeline. Use this datastructure if you want to dump out your data into an HDF5 file.