Code

Python Dataset Reader#2414

Code

Merged

fiedorowicz1 merged 25 commits intoLBANN:developLBANN/lbann:developfrom fiedorowicz1:distconv-python-data-readerfiedorowicz1/lbann:distconv-python-data-readerCopy head branch name to clipboardApr 4, 2024

+1,306-10

Commits

Commits on Apr 4, 2024

• Add skeleton for new python data reader

  fiedorowicz1

  committed
  746650a

  Copy full SHA for 746650a

• Implement basic functionality

  fiedorowicz1

  committed
  64a76e6

  Copy full SHA for 64a76e6

• Fix initialization for distconv

  fiedorowicz1

  committed
  7eb92a6

  Copy full SHA for 7eb92a6

• Add support for labels

  fiedorowicz1

  committed
  9a7a150

  Copy full SHA for 9a7a150

• Add python library supporting classes

  fiedorowicz1

  committed
  8fc35e7

  Copy full SHA for 8fc35e7

• clang format

  fiedorowicz1

  committed
  e6465c5

  Copy full SHA for e6465c5

• Raise exception if rank/io parts not set

  fiedorowicz1

  committed
  c20d4b4

  Copy full SHA for c20d4b4

• Rename to python dataset

  fiedorowicz1

  committed
  88c52bd

  Copy full SHA for 88c52bd

• Add optional module dir argument to add to path

  fiedorowicz1

  committed
  6027f19

  Copy full SHA for 6027f19

• Add unit tests

  fiedorowicz1

  committed
  05d1395

  Copy full SHA for 05d1395

• Simplify naming

  fiedorowicz1

  committed
  faee18a

  Copy full SHA for faee18a

• Add cosmoflow example and reader helper

  fiedorowicz1

  committed
  4702069

  Copy full SHA for 4702069

• Update release notes

  fiedorowicz1

  committed
  9e4cba8

  Copy full SHA for 9e4cba8

• Save dataset pickle in work dir

  fiedorowicz1

  committed
  44eda6d

  Copy full SHA for 44eda6d

• Overhaul new data reader to support prefetching multiple samples/batches

  fiedorowicz1

  committed
  a2f9a15

  Copy full SHA for a2f9a15

• Fix worker index calculation

  fiedorowicz1

  committed
  ac2d403

  Copy full SHA for ac2d403

• clang-format

  fiedorowicz1

  committed
  5cf06ab

  Copy full SHA for 5cf06ab

• Clarify proto comments

  fiedorowicz1

  committed
  753a52f

  Copy full SHA for 753a52f

• Throw error if file fails to open

  fiedorowicz1

  committed
  944b90f

  Copy full SHA for 944b90f

• Add docstrings and type hints

  fiedorowicz1

  committed
  de0e5fa

  Copy full SHA for de0e5fa

• Update CosmoFlow example and enable parallel IO

  fiedorowicz1

  committed
  0af89f8

  Copy full SHA for 0af89f8

• Add basic sample size checking, remove label reconstruction, general clean up

  fiedorowicz1

  committed
  fa668c9

  Copy full SHA for fa668c9

• Switch to multiprocessing pool

  fiedorowicz1

  committed
  6c33696

  Copy full SHA for 6c33696

• Implement response shuffling for distconv

  fiedorowicz1

  committed
  63288ac

  Copy full SHA for 63288ac

• fix typo

  Show description for 75aac4e
  

  fiedorowicz1

  and

  tbennun

  committed
  75aac4e

  Copy full SHA for 75aac4e