Ingest COGs directly into GEE

[jterry64]

Pull request checklist

Please check if your PR fulfills the following requirements:

• Make sure you are requesting to pull a topic/feature/bugfix branch (right side). Don't request your master!
• Make sure you are making a pull request against the develop branch (left side). Also you should start your branch off our develop.
• Check the commit's or even all commits' message styles matches our requested structure.
• Check your code additions will fail neither code linting checks nor unit test.

Pull request type

Please check the type of change your PR introduces:

• Bugfix
• Feature
• Code style update (formatting, renaming)
• Refactoring (no functional changes, no api changes)
• Build related changes
• Documentation content changes
• Other (please describe):

What is the current behavior?

Currently, we upload data to GEE as COG-backed assets. Some important, large datasets like integrated alerts hit a limit of 10 MB for metadata size, causing it to fail.

Issue Number: GTC-2953

What is the new behavior?

Just directly ingest image to GEE using GEE API. This is "fire-and-forget"; this kicks off the ingestion job on GEE, but doesn't check if it succeeds, since it can take multiple hours. For integrated alerts, which is updated daily, this is fine.

Does this introduce a breaking change?

• Yes
• No

Other information

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.23%. Comparing base (14f51c5) to head (458125e).

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop     #587   +/-   ##
========================================
  Coverage    81.23%   81.23%           
========================================
  Files          130      130           
  Lines         5857     5857           
========================================
  Hits          4758     4758           
  Misses        1099     1099           

Flag	Coverage Δ
unittests	81.23% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[danscales]

Except for the missing call at line 118, this looks good. I didn't try to verify that the GEE api calls are correct, but they seem reasonable.

[danscales]

You're missing the call to ingest_in_gee here! I think you mean:

asset_id = ingest_in_gee(dataset, implementation, gcs_path)

[jterry64]

Fixed

[jterry64]

@danscales fixed your comments, and tested the script, and it now works. The only issue I couldn't figure out is how to set the ACL for the new asset to allow public reading. For the COG backed asset, the asset is created immediately, and then you can set the ACL.

For the direct ingestion, the asset doesn't actually exist until it's finished ingesting, so you can't modify the ACL. Because we just fire and forget, we don't wait around until we can modify it... not sure what to do here, but unfortunately don't think I can figure it out today. If someone could pick this up and figure out a way to make sure it's publicly accessible after creation, then we could merge this PR.

[jterry64]

If don't have access to the GEE project, talk to @tanderegg

[jterry64]

@danscales alternatively, once this item is completed for DIST alerts, we can just split the integrated alerts COG into a mosaic where each COG is under the header limit, and then upload those as an image collection: https://gfw.atlassian.net/browse/GTC-2980

You can check the header size with this script: https://github.com/rouault/cog_validator/blob/master/validate_cloud_optimized_geotiff.py