Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Radiometric terrain-correction algorithm is slow and memory hungry #7

Closed
2 tasks done
alexamici opened this issue Mar 21, 2022 · 2 comments
Closed
2 tasks done

Radiometric terrain-correction algorithm is slow and memory hungry #7

alexamici opened this issue Mar 21, 2022 · 2 comments
Assignees
@aurghs
Labels
performance

Comments

@alexamici
Copy link
Member

alexamici commented Mar 21, 2022 •
edited
Loading

At the minimum:

  • review what step can be simplified
  • review in what step we may reduce memory pressure by moving computations to dask out-of-core
@aurghs aurghs self-assigned this Mar 22, 2022
@alexamici
Copy link
Member Author

alexamici commented Jun 21, 2022 •
edited
Loading

The work behind #29 ha made the GTC generation fairly efficient and parallel. Main limitation is the need for approx 12-16Gb of RAM per process (2-3 threads per process). The tricky step is the .interp, all the rest fits the Dask model quite easily.

RTC generation has improved a lot and now uses https://flox.readthedocs.io/ with method="map-reduce"for the heavy .groupby operation.

@alexamici
Copy link
Member Author

With the merge of #32 full image RTC products are possible on a 32Gb Macbook.

Both memory and CPU performance is now much better.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aurghs aurghs

Labels
performance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alexamici @aurghs