[Bug]: KeyError: 'DXT_MPIIO'

[TheAssembler1]

What happened?

Installation:

module load python

conda create -n py310-dxt python=3.10
source activate py310-dxt
conda install arrow-cpp=10.0.1 pyarrow=10.0.1

git clone https://github.com/hpc-io/dxt-explorer
cd dxt-explorer

pip install .

Version:

(dxt) nlewi26@login09:~$ dxt-explorer --version
dxt-explorer 2.0 (Aug 17, 2022)

Running:

dxt-explorer nlewi26_python3.11_id2344413-2344413_1-22-25076-16292832508521119378_1.darshan

Error:

2025-01-24 07:38:12,766 dxt - INFO - generating dataframes
Traceback (most recent call last):
  File "/global/homes/n/nlewi26/.conda/envs/dxt/bin/dxt-explorer", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 1343, in main
    EXPLORE.run()
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 101, in run
    self.generate_plot(filename, report)
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 657, in generate_plot
    self.subset_dataset(file, file_ids, report)
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 380, in subset_dataset
    report.records["DXT_MPIIO"],
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'DXT_MPIIO'

What did you expect to happen?

Expected it to recognize key.

Version

0.2 (Default)

Relevant log output

(dxt) nlewi26@login09:~/jobs/mlio$ dxt-explorer nlewi26_python3.11_id2344413-2344413_1-22-25076-16292832508521119378_1.darshan > output
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_142_of_168.npz (ID 15574852553362390192)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_18_of_42.npz (ID 9313854936428881693)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_005_of_168.npz (ID 2189178886938123643)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_077_of_168.npz (ID 5679382181147560338)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_140_of_168.npz (ID 18346293632301621705)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_092_of_168.npz (ID 17638754425548557416)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_127_of_168.npz (ID 5324047599973260730)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_083_of_168.npz (ID 9598375349459669094)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_104_of_168.npz (ID 8194892759598150122)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_023_of_168.npz (ID 2250722648392585743)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_152_of_168.npz (ID 12695194526511581456)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_016_of_168.npz (ID 5105428889479267458)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_075_of_168.npz (ID 10870841199232815301)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_01_of_42.npz (ID 1751494221474529590)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_17_of_42.npz (ID 7132093997108583507)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_31_of_72.npz (ID 1417483849489518107)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_17_of_72.npz (ID 9199053659832439687)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_116_of_168.npz (ID 1573832761910585846)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_163_of_168.npz (ID 4934377727074971069)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_080_of_168.npz (ID 1081379618384078616)
2025-01-24 08:02:23,942 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_164_of_168.npz (ID 10502828401307771267)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_009_of_168.npz (ID 13958696976739407023)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_018_of_168.npz (ID 11142676250589350714)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_130_of_168.npz (ID 2457611078014676002)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_027_of_168.npz (ID 825486632821881787)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_160_of_168.npz (ID 7886163937726334682)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_057_of_168.npz (ID 2265701679991060044)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_122_of_168.npz (ID 15084238488501430807)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_108_of_168.npz (ID 10062864943083370261)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_148_of_168.npz (ID 11782705082841030311)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_033_of_168.npz (ID 9746720335508337051)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_081_of_168.npz (ID 1031545744343797485)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_019_of_168.npz (ID 8191384591736090664)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_109_of_168.npz (ID 1734546357535807647)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_046_of_168.npz (ID 7366877565120237232)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_039_of_168.npz (ID 11905061797026345726)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_162_of_168.npz (ID 300535487380944748)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_07_of_42.npz (ID 16402083141181190590)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_05_of_42.npz (ID 5661360736971243007)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_01_of_72.npz (ID 13917006509081550471)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_15_of_72.npz (ID 16141264426513392903)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_25_of_42.npz (ID 6240022355593715642)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_068_of_168.npz (ID 5878456695213660309)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_041_of_168.npz (ID 3523944294060966215)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_114_of_168.npz (ID 1998679836206173208)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_036_of_168.npz (ID 16629101979473080569)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_135_of_168.npz (ID 13674712397491331689)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_030_of_168.npz (ID 15824035774976776244)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_063_of_168.npz (ID 3879901484067705659)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_118_of_168.npz (ID 7749890785557655364)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_143_of_168.npz (ID 18155820407590648445)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_029_of_168.npz (ID 7169520744376893915)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_121_of_168.npz (ID 14330262246536157174)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_032_of_168.npz (ID 6211505398370145307)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_117_of_168.npz (ID 4314460600101039875)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_090_of_168.npz (ID 2744131650068549584)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-5-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate18275097308174043336 (ID 13952600143735504744)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-5-3-0.pb-1_temp/part-00000-of-00001.index.tempstate17989651261520020282 (ID 7970178175640125692)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-5-3-0.pb-1_temp/part-00000-of-00001.index (ID 4593753461580856654)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-5-3-0.pb-1.index (ID 13960996781385398539)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpc4a5cc248fba4586a20662d48dd5690c (ID 11972267278299906684)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-5-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate4386260022081645098 (ID 8379623953071860092)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-5-3-0.pb-1_temp/part-00000-of-00001.index.tempstate15588186576919986940 (ID 3857396849316468672)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-5-3-0.pb-1_temp/part-00000-of-00001.index (ID 17157882465384277846)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-5-3-0.pb-1.index (ID 1694310897271588910)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmp35555257fae24693938e0a5affd0707d (ID 961970208847296070)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_07_of_72.npz (ID 6544042366117785445)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_25_of_72.npz (ID 583547147719935374)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_30_of_42.npz (ID 4664145830360262289)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_28_of_42.npz (ID 12934913979291837835)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_08_of_72.npz (ID 70009983845443800)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_026_of_168.npz (ID 11741794337634479977)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_053_of_168.npz (ID 3373911201886674707)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_115_of_168.npz (ID 1845394950428077732)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_136_of_168.npz (ID 2137338959065924256)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_125_of_168.npz (ID 12557452665856739746)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-6-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate695148531810779501 (ID 7386684703970610115)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-6-3-0.pb-1_temp/part-00000-of-00001.index.tempstate14970881529202176799 (ID 14490587582709198136)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-6-3-0.pb-1_temp/part-00000-of-00001.index (ID 13004025637773838014)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-6-3-0.pb-1.index (ID 3069382847151080064)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpaaf5356dbd4d443db3e371868c9296e8 (ID 15470490321788718153)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-6-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate15659803227714406350 (ID 3963569627215739065)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-6-3-0.pb-1_temp/part-00000-of-00001.index.tempstate5146767937130381763 (ID 12944795677145676804)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-6-3-0.pb-1_temp/part-00000-of-00001.index (ID 3868108672868649703)
2025-01-24 08:02:23,943 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-6-3-0.pb-1.index (ID 9803967334514208188)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpb5ed2c4d2c624affbce1a81cb03b0fbb (ID 8253916407419383908)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_045_of_168.npz (ID 9215504301268862678)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_025_of_168.npz (ID 18274133331339501782)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_086_of_168.npz (ID 1487608012189054999)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_008_of_168.npz (ID 1480277517249070050)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_102_of_168.npz (ID 2035722703406273664)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_020_of_168.npz (ID 14101873496433413429)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_110_of_168.npz (ID 478279752111579855)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_001_of_168.npz (ID 16437673053885344371)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_037_of_168.npz (ID 13004951676286527785)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-7-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate3954315449499718644 (ID 4894931735799667442)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-7-3-0.pb-1_temp/part-00000-of-00001.index.tempstate16950832135726746126 (ID 265931037316297774)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-7-3-0.pb-1_temp/part-00000-of-00001.index (ID 2396319515964295055)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-7-3-0.pb-1.index (ID 17362613944676527725)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpecae3608076a4b15bd4dd0b8eab62f3f (ID 5985553609328636214)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-7-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate6011178214364679308 (ID 10850334205703780501)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-7-3-0.pb-1_temp/part-00000-of-00001.index.tempstate4165309825407273316 (ID 7015988350263579399)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-7-3-0.pb-1_temp/part-00000-of-00001.index (ID 13754904194559536612)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-7-3-0.pb-1.index (ID 15656326355124906980)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmp2fd0bdaef74c4af4b50742d5e16253ec (ID 3694396620732670971)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_34_of_72.npz (ID 16215909926470738582)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_22_of_72.npz (ID 3375019958247544790)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_26_of_42.npz (ID 15922115582586446354)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_22_of_42.npz (ID 13282923788924780116)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_050_of_168.npz (ID 2663905290598662244)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_107_of_168.npz (ID 13153085356213193411)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_161_of_168.npz (ID 16443237781289312333)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_003_of_168.npz (ID 2535332265524187196)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_071_of_168.npz (ID 782839937057823870)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_066_of_168.npz (ID 8102582876657956855)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-8-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate2275403944564427799 (ID 1359937624023757621)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-8-3-0.pb-1_temp/part-00000-of-00001.index.tempstate8687333926292304831 (ID 4374208308987542848)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-8-3-0.pb-1_temp/part-00000-of-00001.index (ID 496916808511670813)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-8-3-0.pb-1.index (ID 4922219345566133327)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpafffdfebebde4cc98954d000d287178a (ID 1869437389104659632)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-8-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate3346995586177374328 (ID 7079389050085986264)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-8-3-0.pb-1_temp/part-00000-of-00001.index.tempstate16901986586822281801 (ID 3939589896062784659)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-8-3-0.pb-1_temp/part-00000-of-00001.index (ID 18098590845517573049)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-8-3-0.pb-1.index (ID 14703426406056183285)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmp11be2e18a3c24e3fbd92ff5f3d7a4e39 (ID 5931594475347172800)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_126_of_168.npz (ID 17741002537652898912)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_085_of_168.npz (ID 14626081049586525296)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_156_of_168.npz (ID 6726042310150991387)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_064_of_168.npz (ID 18200311668397943135)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-9-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate11986340207032560696 (ID 2447593830925852071)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-9-3-0.pb-1_temp/part-00000-of-00001.index.tempstate5068057219027269605 (ID 1149473608776801305)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-9-3-0.pb-1_temp/part-00000-of-00001.index (ID 5636394964664206275)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-9-3-0.pb-1.index (ID 12003341831661514315)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpbbdefb8d44674819b74e2cba1a8fe78f (ID 7659023938267045158)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-9-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate1152122174405923161 (ID 15678207178568026172)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-9-3-0.pb-1_temp/part-00000-of-00001.index.tempstate18161210402870589469 (ID 6492977440246466866)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-9-3-0.pb-1_temp/part-00000-of-00001.index (ID 13556266816527555421)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-9-3-0.pb-1.index (ID 12575668201923922864)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmp5bb2eba8ed104826be2d229f4c4ebac5 (ID 16261357793342242838)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_23_of_42.npz (ID 2834093337066332619)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_32_of_42.npz (ID 14145406162626171057)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_14_of_72.npz (ID 13590040273393658608)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_33_of_42.npz (ID 6840832958652065135)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_091_of_168.npz (ID 4920903614293162491)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_155_of_168.npz (ID 2036766670729014849)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_078_of_168.npz (ID 6041613811839789004)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_096_of_168.npz (ID 3443925943449617138)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_052_of_168.npz (ID 14181732211953509109)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_007_of_168.npz (ID 17345949034570391729)
2025-01-24 08:02:23,944 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_082_of_168.npz (ID 13335465004650872553)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-10-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate8608033125853370449 (ID 4106482679752192837)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-10-3-0.pb-1_temp/part-00000-of-00001.index.tempstate6367564417859194737 (ID 2327995850299518802)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-10-3-0.pb-1_temp/part-00000-of-00001.index (ID 7331473072171600366)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/model-10-3-0.pb-1.index (ID 16025507239858198041)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmpe5b50a46cdf54ce38d1ff99fea9fc076 (ID 5020399325983749013)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-10-3-0.pb-1_temp/part-00000-of-00001.data-00000-of-00001.tempstate992030440243115558 (ID 7943289202219781857)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-10-3-0.pb-1_temp/part-00000-of-00001.index.tempstate7761733893440764962 (ID 8042594656352481676)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-10-3-0.pb-1_temp/part-00000-of-00001.index (ID 15358356175360707740)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/layer-0-10-3-0.pb-1.index (ID 10729737047685915756)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/checkpoint/checkpoint.tmp8c759d63452c4ba3b988f188310ee4a9 (ID 5039614535477590231)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_35_of_42.npz (ID 12013565336452846773)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_133_of_168.npz (ID 11175171219115211549)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_24_of_72.npz (ID 5022918171717763917)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_134_of_168.npz (ID 317872763921659613)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_065_of_168.npz (ID 14415973002305746204)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_11_of_42.npz (ID 16440298366223875149)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_138_of_168.npz (ID 11602289801572560875)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_124_of_168.npz (ID 5595446833308353330)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_16_of_72.npz (ID 4042028686697415896)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_106_of_168.npz (ID 14186333133702180209)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_074_of_168.npz (ID 16557632103376284382)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_120_of_168.npz (ID 7295371958937457707)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_16_of_42.npz (ID 13941153099930292459)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_034_of_168.npz (ID 13200764554714248929)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_141_of_168.npz (ID 16031986189544895836)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_29_of_42.npz (ID 10971801428958807656)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_145_of_168.npz (ID 1078182441207275814)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_095_of_168.npz (ID 14290238608426007079)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_21_of_42.npz (ID 17320455753051932827)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_055_of_168.npz (ID 815107053099007299)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_002_of_168.npz (ID 6478277746026484805)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_09_of_72.npz (ID 15152668483221909928)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_093_of_168.npz (ID 5649273132252589181)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_02_of_72.npz (ID 3718577526839088702)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_03_of_72.npz (ID 17182356294522752588)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_149_of_168.npz (ID 13898830913579415323)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_097_of_168.npz (ID 12033315632789796938)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_33_of_72.npz (ID 10130495586768610455)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_056_of_168.npz (ID 7878986786420196373)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_23_of_72.npz (ID 10322891676837697868)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_038_of_168.npz (ID 14787054723624308098)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_119_of_168.npz (ID 9542862955565043423)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_14_of_42.npz (ID 8337630467905588976)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_26_of_72.npz (ID 8533959295207748059)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_167_of_168.npz (ID 1430023396491777313)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_006_of_168.npz (ID 17480384163991031622)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_062_of_168.npz (ID 11970959115521994235)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_05_of_72.npz (ID 11511064453431392388)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_03_of_42.npz (ID 14622973325324070797)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_150_of_168.npz (ID 15730208075932718269)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_12_of_42.npz (ID 9059764155374499269)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_010_of_168.npz (ID 3717625893954900236)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_098_of_168.npz (ID 1058219762306855973)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_166_of_168.npz (ID 14900917712704569590)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_013_of_168.npz (ID 14246936147435625510)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_00_of_72.npz (ID 13701502904365531031)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_028_of_168.npz (ID 5776813063689605091)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_014_of_168.npz (ID 7745105679122588245)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_13_of_42.npz (ID 12259375343592486724)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_139_of_168.npz (ID 16043716473361279688)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_18_of_72.npz (ID 10191317120615563747)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_06_of_42.npz (ID 899066112483806989)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_132_of_168.npz (ID 5659939155108650190)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_12_of_72.npz (ID 13713840662977108795)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_024_of_168.npz (ID 9015210993596446433)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_004_of_168.npz (ID 1415943913622123911)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_147_of_168.npz (ID 9700241923580704678)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_31_of_42.npz (ID 15325845077638368795)
2025-01-24 08:02:23,945 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_042_of_168.npz (ID 15882524910085810440)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_00_of_42.npz (ID 17881636706614959654)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_113_of_168.npz (ID 5091285341052886809)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_11_of_72.npz (ID 8709129892869241166)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_111_of_168.npz (ID 13881536581347016028)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_044_of_168.npz (ID 12303121162250577228)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_20_of_72.npz (ID 8190248716692401068)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_131_of_168.npz (ID 3294341227693763084)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_19_of_72.npz (ID 8369566793216575643)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_094_of_168.npz (ID 18325761976548084807)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_158_of_168.npz (ID 7018305810714782134)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_129_of_168.npz (ID 8945674764100946827)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_021_of_168.npz (ID 4491583446692989049)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_21_of_72.npz (ID 3874304018412883868)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_087_of_168.npz (ID 9818756743703984907)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_20_of_42.npz (ID 15182686940603433786)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_28_of_72.npz (ID 17701533318549470113)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_076_of_168.npz (ID 11028900106555147692)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_06_of_72.npz (ID 17960694135437324842)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_043_of_168.npz (ID 11290527134487197754)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_011_of_168.npz (ID 3347418897696470915)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/valid/img_35_of_72.npz (ID 7186719355082929357)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_099_of_168.npz (ID 16407592449578043772)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_067_of_168.npz (ID 7428004580422387415)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_123_of_168.npz (ID 7695389252855143323)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_084_of_168.npz (ID 15821765300222325847)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_031_of_168.npz (ID 15356014570771236278)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_105_of_168.npz (ID 4218269539342902930)
2025-01-24 08:02:23,946 dxt - INFO - FILE: /pscratch/sd/n/nlewi26/dlio_data/unet3d/data/train/img_100_of_168.npz (ID 18158224344040187816)
2025-01-24 08:02:23,946 dxt - INFO - generating dataframes
Traceback (most recent call last):
  File "/global/homes/n/nlewi26/.conda/envs/dxt/bin/dxt-explorer", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 1343, in main
    EXPLORE.run()
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 101, in run
    self.generate_plot(filename, report)
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 657, in generate_plot
    self.subset_dataset(file, file_ids, report)
  File "/global/homes/n/nlewi26/.conda/envs/dxt/lib/python3.11/site-packages/explorer/dxt.py", line 380, in subset_dataset
    report.records["DXT_MPIIO"],
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'DXT_MPIIO'