Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support precalc running in lambda #329

Open
avmey opened this issue Aug 15, 2024 · 1 comment
Open

Support precalc running in lambda #329

avmey opened this issue Aug 15, 2024 · 1 comment
Milestone

Comments

@avmey
Copy link
Contributor

avmey commented Aug 15, 2024

Precalc CLI should offer a ‘cloud’ option (specific deployed precalc lambda) that sends precalc input parameters and returns precalc metrics. Potential need to cache here, opportunity to use GeoprocessingHandler? But should be deployed with every project.

Has to support clipping to geographies.
Will need to scope this more.

@twelch
Copy link
Contributor

twelch commented Nov 11, 2024

An improved method for precalc, once there is a switch to storing geoprocessing function results in S3.

Currently:

  • precalc command is used to calculate overall metrics for combinations of geography and datasource, on the developers local computer.
  • precalc command calculates multiple metrics, whether they get used or not (area, sum, count)
  • whether to precalc a geography or datasource is configured in geographies.json and datasources.json. There isn't a clear separation of concerns.
  • precalc metrics are saved to precalc.json and is imported directly by report client. This has potential to bloat bundle size.

Solution:

  • once geoprocessing function results are stored in S3 bucket and there is potential to store multiple results in S3.
  • overall metrics can be calculated directly by the geoprocessing function and stored in an overall metric results JSON file.
  • typical sketch metrics can be stored in their own results file.
  • when geoprocessing function runs, it can look for overall results, and only calculate them if not present.

Benefits:

  • keeps calculation accessible to report developer to adapt to their needs, and calculate only what they need.

To figure out:

  • how and when to clear overall metrics, when datasources are updated. Perhaps s3 last updated timestamp can be incorporated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

2 participants