Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possibility to save training visualization images #1016

Open
1 of 4 tasks
auesro opened this issue Oct 28, 2022 · 5 comments
Open
1 of 4 tasks

Possibility to save training visualization images #1016

auesro opened this issue Oct 28, 2022 · 5 comments
Labels
enhancement New feature or request good first issue This issue is relatively self-contained.

Comments

@auesro
Copy link

auesro commented Oct 28, 2022

Bug description

There is apparently no working way to save the images shown during the training of models. I have tried manually editing the training_config.json fields save_visualizations and delete_viz_images to True and False, respectively, before training, but after training is completed both appear as True....and of course the viz folder is gone.

Expected behaviour

Both training and validation images used during training are saved to defined folder.

Actual behaviour

Images are displayed during training but not saved.

Your personal set up

Environment packages 
# packages in environment at /home/auesro/mambaforge/envs/Sleap:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   0.15.0                   pypi_0    pypi
aom                       3.3.0                h27087fc_1    conda-forge
astunparse                1.6.3                    pypi_0    pypi
attrs                     21.2.0                   pypi_0    pypi
backports-zoneinfo        0.2.1                    pypi_0    pypi
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.9.24            ha878542_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                4.2.4                    pypi_0    pypi
cattrs                    1.1.1                    pypi_0    pypi
certifi                   2021.10.8                pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
clang                     5.0                      pypi_0    pypi
colorama                  0.4.5                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
cuda-nvcc                 11.3.58              h2467b9f_0    nvidia
cudatoolkit               11.3.1               ha36c431_9    nvidia
cudnn                     8.2.1.32             h86fa8c9_0    conda-forge
cycler                    0.11.0                   pypi_0    pypi
efficientnet              1.0.0                    pypi_0    pypi
ffmpeg                    5.0.1                h594f047_1    conda-forge
flatbuffers               1.12                     pypi_0    pypi
fonttools                 4.37.1                   pypi_0    pypi
freetype                  2.10.4               h0708190_1    conda-forge
gast                      0.4.0                    pypi_0    pypi
geos                      3.9.1                h9c3ff4c_2    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               1.35.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.44.0                   pypi_0    pypi
h5py                      3.1.0           nompi_py37h1e651dc_100    conda-forge
hdf5                      1.10.6          nompi_h3c11f04_101    conda-forge
hdmf                      3.4.2                    pypi_0    pypi
icu                       70.1                 h27087fc_0    conda-forge
idna                      3.3                      pypi_0    pypi
image-classifiers         1.0.0                    pypi_0    pypi
imageio                   2.15.0                   pypi_0    pypi
imgaug                    0.4.0                    pypi_0    pypi
imgstore                  0.2.9                    pypi_0    pypi
importlib-metadata        4.11.1                   pypi_0    pypi
importlib-resources       5.9.0                    pypi_0    pypi
jbig                      2.1               h7f98852_2003    conda-forge
joblib                    1.1.0                    pypi_0    pypi
jpeg                      9e                   h166bdaf_1    conda-forge
jsmin                     3.0.1                    pypi_0    pypi
jsonpickle                1.2                      pypi_0    pypi
jsonschema                4.16.0                   pypi_0    pypi
keras                     2.6.0                    pypi_0    pypi
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.4.4                    pypi_0    pypi
lame                      3.100             h7f98852_1001    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0                8_openblas    conda-forge
libcblas                  3.9.0                8_openblas    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libdrm                    2.4.109              h7f98852_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            7.5.0               h14aa051_20    conda-forge
libgfortran4              7.5.0               h14aa051_20    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0                8_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.12          pthreads_hb3c22a3_1    conda-forge
libpciaccess              0.16                 h516909a_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_16    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libva                     2.14.0               h7f98852_0    conda-forge
libvpx                    1.11.0               h9c3ff4c_3    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.13               h22db469_0    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
markdown                  3.3.6                    pypi_0    pypi
matplotlib                3.5.3                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
ndx-pose                  0.1.1                    pypi_0    pypi
nettle                    3.6                  he412f7d_0    conda-forge
networkx                  2.6.3                    pypi_0    pypi
numpy                     1.19.5           py37h3e96413_3    conda-forge
oauthlib                  3.2.0                    pypi_0    pypi
olefile                   0.46               pyh9f0ad1d_1    conda-forge
opencv-python             4.5.5.62                 pypi_0    pypi
opencv-python-headless    4.5.5.62                 pypi_0    pypi
openh264                  2.1.1                h780b84a_0    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   3.0.5                h166bdaf_2    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.3.5            py37he8f5f7f_0    conda-forge
pillow                    8.4.0            py37h0f21c89_0    conda-forge
pip                       22.3               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10                   pypi_0    pypi
protobuf                  3.19.4                   pypi_0    pypi
psutil                    5.9.2                    pypi_0    pypi
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pygments                  2.13.0                   pypi_0    pypi
pykalman                  0.9.5                    pypi_0    pypi
pynwb                     2.1.0                    pypi_0    pypi
pyparsing                 3.0.6                    pypi_0    pypi
pyrsistent                0.18.1                   pypi_0    pypi
pyside2                   5.14.1                   pypi_0    pypi
python                    3.7.12          hf930737_100_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-rapidjson          1.8                      pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pytz-deprecation-shim     0.1.0.post0              pypi_0    pypi
pywavelets                1.3.0                    pypi_0    pypi
pyzmq                     23.2.1                   pypi_0    pypi
qimage2ndarray            1.9.0                    pypi_0    pypi
qtpy                      2.2.1              pyhd8ed1ab_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rich                      10.16.1                  pypi_0    pypi
ruamel-yaml               0.17.21                  pypi_0    pypi
ruamel-yaml-clib          0.2.6                    pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scikit-learn              1.0.2                    pypi_0    pypi
scikit-video              1.1.11                   pypi_0    pypi
scipy                     1.5.3            py37h8911b10_0    conda-forge
seaborn                   0.12.0                   pypi_0    pypi
segmentation-models       1.0.1                    pypi_0    pypi
setuptools                59.8.0           py37h89c1867_1    conda-forge
setuptools-scm            6.4.2                    pypi_0    pypi
shapely                   1.7.1            py37h48c49eb_5    conda-forge
shiboken2                 5.14.1                   pypi_0    pypi
six                       1.15.0             pyh9f0ad1d_0    conda-forge
sleap                     1.2.8                    pypi_0    pypi
sqlite                    3.38.2               h4ff8645_0    conda-forge
svt-av1                   0.9.1                h27087fc_0    conda-forge
tensorboard               2.6.0                    pypi_0    pypi
tensorboard-data-server   0.6.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tensorflow                2.6.3                    pypi_0    pypi
tensorflow-estimator      2.6.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
threadpoolctl             3.1.0                    pypi_0    pypi
tifffile                  2021.11.2                pypi_0    pypi
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1                    pypi_0    pypi
typing-extensions         3.10.0.2                 pypi_0    pypi
tzdata                    2022.2                   pypi_0    pypi
tzlocal                   4.2                      pypi_0    pypi
urllib3                   1.26.8                   pypi_0    pypi
werkzeug                  2.0.3                    pypi_0    pypi
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
wrapt                     1.12.1                   pypi_0    pypi
x264                      1!161.3030           h7f98852_1    conda-forge
x265                      3.5                  h924138e_3    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libx11               1.7.2                h7f98852_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h7f98852_1    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-xextproto            7.3.0             h7f98852_1002    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zipp                      3.7.0                    pypi_0    pypi
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge
Logs
@auesro auesro added the bug Something isn't working label Oct 28, 2022
@roomrys roomrys added enhancement New feature or request and removed bug Something isn't working labels Oct 28, 2022
@talmo talmo added bug Something isn't working and removed enhancement New feature or request labels Nov 7, 2022
@talmo
Copy link
Collaborator

talmo commented Nov 7, 2022

Hi @auesro,

Is this when training through the GUI or CLI? If from the GUI, Do you select the training_config.json or initial_config.json from the dropdown?

Talmo

@auesro
Copy link
Author

auesro commented Nov 14, 2022

Hi @talmo,
This was training through the GUI. I select the training_config.json.

A

@roomrys
Copy link
Collaborator

roomrys commented Nov 17, 2022 •
edited
Loading

Analysis

After taking a look at the code it seems that we use the save_visualizations not to save visualizations post-training, but to save visualizations during training so that the user can view these images during training.

I've come to this conclusion based on:

  1. save_visualizations is populated by the GUI option to "Visualize Predictions During Training"
  2. the visualization directory is always deleted but only if save_visualizations == True - a little counter-intuitive to the variable name if images were meant to be saved post-training

    sleap/sleap/nn/training.py

    Lines 931 to 937 in 5093f69

    # Run post-training actions.
    if self.config.outputs.save_outputs:
    if (
    self.config.outputs.save_visualizations
    and self.config.outputs.delete_viz_images
    ):
    self.cleanup()

Proposed Solution

We can either change the meaning of save_visualizations to mean "Save the visualizations (even after training)" or add a new checkbox named "Delete Visualizations" and populate delete_viz_images using this new checkbox.

Follow-up Bug (separate PR: #1052)

The real bug here is that, when using the GUI, if you select a config file to load, SLEAP only populates the parameters listed in the GUI AND uses the parameters listed in the GUI (plus defaults from a template config file). SLEAP will not use the "loaded" config file to set the unseen (non-GUI) parameters.

@roomrys roomrys added enhancement New feature or request good first issue This issue is relatively self-contained. and removed bug Something isn't working labels Nov 17, 2022
@auesro
Copy link
Author

auesro commented Nov 18, 2022

It makes sense, @roomrys , but then what about the other variable (which we dont have access to through the GUI) delete_viz_images?If set to False it should leave the images in place, right? I havent gone through the code to actually look at what id does though.

Thanks!

@roomrys
Copy link
Collaborator

roomrys commented Nov 18, 2022

Hi @auesro

Yes, that is the problem I want to address more broadly in #1052. For this PR here, I think it would be sufficient to add a checkbox exposing delete_viz_images in the GUI.

Thanks,
Liezl

This was referenced Jun 21, 2024
Closed
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request good first issue This issue is relatively self-contained.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@talmo @auesro @roomrys