feat(cache): re-work job results cache

[uael]

Important

Reworked job results caching with in-memory storage, SHA-256 hashing, and improved S3 handling.

• Caching Logic:
  • Introduced in-memory caching using lazy_static in common.rs.
  • Updated cached_result_path() to use SHA-256 hashing.
  • Modified get_cached_resource_value_if_valid() to check in-memory cache before database.
  • Updated save_in_cache() to store results in-memory and database.
• S3 Handling:
  • Added S3 ETag validation in get_cached_resource_value_if_valid().
  • Updated arg_value_hash_additions() to handle S3 resources.
• Refactoring:
  • Removed DefaultHasher usage, replaced with sha2::Sha256.
  • Simplified handle_queued_job() and process_completed_job() logic in worker.rs.
• Miscellaneous:
  • Added new SQL query file for resource insertion.
  • Minor logging and error handling improvements.

^{This description was created by for 35c923c. It will automatically update as commits are pushed.}

[cloudflare-workers-and-pages]

Deploying windmill with    Cloudflare Pages

Latest commit:	35c923c
Status:	✅  Deploy successful!
Preview URL:	https://318bfe4d.windmill.pages.dev
Branch Preview URL:	https://uael-job-results-cache.windmill.pages.dev

View logs

[ellipsis-dev]

❌ Changes requested. Reviewed everything up to 35c923c in 2 minutes and 16 seconds

More details

• Looked at 653 lines of code in 8 files
• Skipped 0 files when reviewing.
• Skipped posting 2 drafted comments based on config settings.

1. backend/windmill-worker/src/common.rs:752

• Draft comment:
  The cache expiration handling in get_cached_resource_value_if_valid could be improved. Currently, the cache entry is removed if expired, but it should also ensure that the cache is updated with a fresh value if available to avoid unnecessary cache misses.
• Reason this comment was not posted:
  Comment looked like it was already resolved.

2. backend/windmill-worker/src/common.rs:625

• Draft comment:
  In cached_result_path, ensure that a unique cache key is generated even when both job.script_hash and job.script_path are None to avoid potential cache collisions.
• Reason this comment was not posted:
  Comment did not seem useful.

Workflow ID: wflow_ijfvDA1YgXBogktR

---

Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.