Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add fallback in case remote codelists are inaccessible #48

Open
jpmckinney opened this issue Nov 11, 2021 · 1 comment
Open

Add fallback in case remote codelists are inaccessible #48

jpmckinney opened this issue Nov 11, 2021 · 1 comment
Milestone

Comments

@jpmckinney
Copy link
Member

jpmckinney commented Nov 11, 2021

5427afc fixed several bugs, by caching codelists in-memory for a day.

However, if a remote URL is inaccessible, then an error is raised.

We could implement a fallback, where the codelists are also written to disk. Whenever the in-memory cache expires, the code attempts to retrieve the codelist. If it succeeds, it compares checksums and updates the file on disk if changed. If it fails, then it logs a warning and reads from disk.

When reading/writing to disk, we can use file locking (if the file is shared between workers).

In terms of where to write the file, https://docs.docker.com/storage/tmpfs/ might be appropriate. Alternately, we can write the files to the host (via a mount), so that the on-disk cache survives any worker restarts, and so that Salt can install initial copies of the codelists, to bootstrap this fallback method.

@jpmckinney
Copy link
Member Author

Noting that this does come up in practice.

@jpmckinney jpmckinney added this to the Cleanup milestone Jan 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant