Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Imviz parser for Roman-like ASDF files #2351

Merged
merged 7 commits into from
Aug 9, 2023

Conversation

bmorris3
Copy link
Contributor

@bmorris3 bmorris3 commented Aug 8, 2023

Description

There are lots of "Roman-like" ASDF files that do not pass validation by roman_datamodels, but currently on main, attempting to parse a file that cannot be fully parsed by roman_datamodels raises an error.

A minimal "Roman-like" image in an ASDF file could be constructed with:

import asdf
import numpy as np

tree = {
    'roman': {
        'data': np.arange(16).reshape((4, 4)),
    },
}

af = asdf.AsdfFile(tree=tree)

If the above ASDF file is passed to roman_datamodels, an error is raised, even though it's pretty clear that we could parse the data node as an image:

>>> import roman_datamodels.datamodels as rdd
>>> rdd.open(af)
TypeError: Unknown datamodel type: <class 'dict'>

This PR allows the parser to use asdf to look for the usual Roman tree structure in files ending in ".asdf", even when roman_datamodels can't parse the file. This allows non-standard Roman-like data products to be loaded.

I say "Roman-like" because this parser is still looking for a top-level node called "roman", which is standard in Roman data products. If that node isn't there, this parser will raise an error. If we need in the future, we can revise this parser logic to accept the preferred top-level node as a kwarg.

Change log entry

  • Is a change log needed? If yes, is it added to CHANGES.rst? If you want to avoid merge conflicts,
    list the proposed change log here for review and add to CHANGES.rst before merge. If no, maintainer
    should add a no-changelog-entry-needed label.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

  • Are two approvals required? Branch protection rule does not check for the second approval. If a second approval is not necessary, please apply the trivial label.
  • Do the proposed changes actually accomplish desired goals? Also manually run the affected example notebooks, if necessary.
  • Do the proposed changes follow the STScI Style Guides?
  • Are tests added/updated as required? If so, do they follow the STScI Style Guides?
  • Are docs added/updated as required? If so, do they follow the STScI Style Guides?
  • Did the CI pass? If not, are the failures related?
  • Is a milestone set? Set this to bugfix milestone if this is a bug fix and needs to be released ASAP; otherwise, set this to the next major release milestone.
  • After merge, any internal documentations need updating (e.g., JIRA, Innerspace)? 🐱

@bmorris3 bmorris3 added bug Something isn't working imviz labels Aug 8, 2023
@bmorris3 bmorris3 added this to the 3.6.2 milestone Aug 8, 2023
@bmorris3 bmorris3 marked this pull request as ready for review August 8, 2023 19:50
@codecov
Copy link

codecov bot commented Aug 8, 2023

Codecov Report

Patch coverage: 72.91% and project coverage change: +0.08% 🎉

Comparison is base (2bb043a) 90.66% compared to head (33cd9b0) 90.74%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2351      +/-   ##
==========================================
+ Coverage   90.66%   90.74%   +0.08%     
==========================================
  Files         152      153       +1     
  Lines       17434    17470      +36     
==========================================
+ Hits        15807    15854      +47     
+ Misses       1627     1616      -11     
Files Changed Coverage Δ
jdaviz/configs/imviz/tests/test_parser_roman.py 12.50% <0.00%> (+1.38%) ⬆️
jdaviz/configs/imviz/plugins/parsers.py 87.36% <66.66%> (-1.12%) ⬇️
jdaviz/configs/imviz/tests/test_parser.py 99.45% <66.66%> (+0.26%) ⬆️
jdaviz/configs/imviz/tests/test_parser_asdf.py 100.00% <100.00%> (ø)

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@pllim pllim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try-except isn't pretty but good enough for Roman.

jdaviz/configs/imviz/plugins/parsers.py Outdated Show resolved Hide resolved
@pllim pllim added the Extra CI Run cron jobs in PR label Aug 9, 2023
@pllim
Copy link
Contributor

pllim commented Aug 9, 2023

Since this touches Roman, I also enabled the Roman cron jobs here. Let's see... 🤞

@pllim
Copy link
Contributor

pllim commented Aug 9, 2023

CHANGES.rst Outdated Show resolved Hide resolved
@bmorris3
Copy link
Contributor Author

bmorris3 commented Aug 9, 2023

@pllim that's likely from the latest release of roman_datamodels. Bumping the pin and trying again.

@bmorris3
Copy link
Contributor Author

bmorris3 commented Aug 9, 2023

Bumping the roman_datamodels pin didn't fix the issue because the package data ASDF file also needed to be updated for the latest roman_datamodels pin. This was also the case in #2302.

To avoid this problem occurring indefinitely into the future, I've made a pytest fixture to generate this test data for tests where it's needed in c44067e.

@bmorris3 bmorris3 mentioned this pull request Aug 9, 2023
9 tasks
@pllim pllim added the 💤 backport-v3.6.x on-merge: backport to v3.6.x label Aug 9, 2023
@@ -290,6 +292,12 @@ def mos_image():
return CCDData(data, wcs=wcs, unit='Jy', meta=header)


@pytest.fixture
@pytest.mark.skipif(not HAS_ROMAN_DATAMODELS, reason="roman_datamodels is not installed")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Putting a skipif in a fixture is too confusing. If you want a test to skip, it is more explicit to put skipif on the test function.

If create_wfi_image_model needs Roman datamodels, then maybe have this return None if that package isn't installed with a if-else?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternately, you can move all the stuff that needs Roman data models into a special test module and use pytest.importorskip.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of 8a12ef8, we do both. There's a roman_datamodels-specific test in jdaviz/configs/imviz/tests/test_parser_roman.py that uses importorskip, and now the fixture returns None unless roman_datamodels is available.

@pllim
Copy link
Contributor

pllim commented Aug 9, 2023

Diff LGTM now but I want to see the Roman cron job status before approving. Thanks!

@pllim pllim added the trivial Only needs one approval instead of two label Aug 9, 2023
@pllim pllim merged commit fa5d292 into spacetelescope:main Aug 9, 2023
14 of 18 checks passed
meeseeksmachine pushed a commit to meeseeksmachine/jdaviz that referenced this pull request Aug 9, 2023
pllim added a commit that referenced this pull request Aug 9, 2023
…1-on-v3.6.x

Backport PR #2351 on branch v3.6.x (Improve Imviz parser for Roman-like ASDF files)
bmorris3 added a commit to bmorris3/jdaviz that referenced this pull request Aug 11, 2023
* Improve Roman-like ASDF support

* Update jdaviz/configs/imviz/plugins/parsers.py

Co-authored-by: P. L. Lim <[email protected]>

* Parser fix after Pey Lian's revision

* Update CHANGES.rst

Co-authored-by: P. L. Lim <[email protected]>

* bump rdm pin

* removing static test Roman datamodel, generating it in a pytest fixture instead

* removing skipif on the romandep fixture

---------

Co-authored-by: P. L. Lim <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Extra CI Run cron jobs in PR imviz trivial Only needs one approval instead of two 💤 backport-v3.6.x on-merge: backport to v3.6.x
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants