Output Run Stats

[beauremus]

We should provide stats on each run for troubleshooting purposes. These don't need to be versioned so we included them in the .gitignore. I think we should output the stats of each run to a file named by its date-time. We should also have a cleanup function that only allows a week of stats to persist.

Things to time:

• DPM data acquisition
• HDF file output

Things to log:

• DAQ errors
• Python errors
• Debug log
• Empty DAQ for any device

[beauremus]

Edited the OP to include "Things to log."

[beauremus]

Edited the OP to include "Empty DAQ" in "Things to log."

[beauremus]

Let's focus on stats gathering first and we can worry about maintenance and cleanup later.

[beauremus]

Python errors and debug log issues are solved because cronjobs output to mail /var/log/mail/gmpsai-prod.

[beauremus]

Concerning "Empty DAQ for any device":
I think the only way to determine that a device has no data is to compare the list of devices in the device request input file with the keys in the .h5 output file.
A key is only added to the h5 file in the case where data is returned. If the key isn't present in the h5 file then no data was received for that device.

There's another step after this to determine why the key isn't in the h5 file. If a status comes through for a device, it will not be in the list of keys. The status should be evaluated instead of assuming no data.

If there is no key in the h5 file and there is no related status, then no data is available for that request. Otherwise, the error is reported and should be evaluated or understood.

[sakshi-fermi]

Estimated time for logging:

Best case: 1 day

Best guess: 2-3 days

Worst case: 5 days

[sakshi-fermi]

Python errors and debug log issues are solved because cronjobs output to mail /var/log/mail/gmpsai-prod.

These were not logged in /var/log/mail/gmpsai-prod after Feb 12 because of an unknown issue but will be logged in a separate log file

[sakshi-fermi]

Things to log
This is addressed by e7939d2 2ff31c0 eb660df .