Add push-cache-effect by zmrocze · Pull Request #165 · hercules-ci/hercules-ci-effects

zmrocze · 2024-01-11T16:46:40Z

Motivation

We want to push some paths to either the cachix or attic cache, additionaly to the cache configured for herculesCI. We discussed it recently on mlabs slack. If you guys would be interested here's a module doing that to be used like below:

populate-cache-effect = {
      enable = true;
      caches = {
        mlabs-attic = {
          type = "attic";
          secretName = "cardano-nix-attic-token";
          branches = ["master"];
          packages = [test-drv];
        };
        mlabs-cachix = {
          type = "cachix";
          secretName = "karol-cachix-token";
          branches = ["master" "push-to-attic"];
          packages = [test-drv];
        };
      };
    };

it's a flake-parts module producing effects under herculesCI.onPush.default.outputs.effects.populate-cache-effect option.

Questions

It works. But I don't understand the structure of this repository, so I'd like to ask first how to incorporate that into the codebase:

where/how to export this
if it's fine the way it is with default attic-client-pkg cachix-pkg

Maintainer checklist

Documentation (function reference docs, setup guide, option reference docs)
Has tests (VM test, free account, and/or test instructions)
Is the corresponding module up to date?

zmrocze · 2024-01-11T16:51:32Z

Hi @roberth,

it works as witnessed on this demo repository.

I've moved the pr again, sorry. That is the previous, there was a discussion, I tried to act on all points and believe they're fixed. Go ahead and review if you can please!

brainrake · 2024-01-30T23:43:31Z

+            packages = lib.mkOption {
+              type = with lib.types; listOf package;
+              description = "List of packages to push to the cache.";
+              example = "[ pkgs.hello ]";


As a default, we could use flakeToOutputs from hercules-ci-agent to match the behavior of the default build
https://github.com/hercules-ci/hercules-ci-agent/blob/0e9e67bb7d41ea2bbb13982887ff3b9c511a2aba/hercules-ci-agent/data/default-herculesCI-for-flake.nix#L48
or combine herculesCI.onPush.*.outputs to match the hercules ci build phase
https://docs.hercules-ci.com/hercules-ci-agent/evaluation#attribute-herculesCI.onPush-outputs

Interesting! Didn't know flakeToOutputs

This would involve writing a attribute tree traversal to match the agent behavior, with the right recuresIntoAttrs behavior, rejecting things with _type, etc.
That could be done, but I wouldn't encode into this module the assumption that every output should be pushed.

Note also that this effect can not replicate the whole agent behavior, such as pushing all the right build dependencies.
This means that it's not a replacement for the cache that's used by the agents transparently.
More significantly, it also means that this effect is not 100% suitable as a build cache or "development cache", so I doubt that e.g. the checks or the devShell dependencies should be pushed by it. A better use case for this effect is as a "release cache" - for only the end products, such as packages, or perhaps also paths for deployment, such as nixosConfigurations or the inputs of effects (if flake consumers use the same effect functions).

Resolved: we leave it unspecified and the responsibility of the caller since replicating default builder behavior is infeasible.

brainrake · 2024-01-31T01:29:38Z

@@ -0,0 +1,160 @@
+{ lib, withSystem, config, ... }:
+let pkgs-x86_64-linux = withSystem "x86_64-linux" ({ pkgs, ... }: pkgs);


We should support other systems.

I would see what happens if we try to push non x86_64-linux derivations using an x86_64-linux effect before.

Those will be scheduled during the build phase of the job, and substituted into the agent's store before the effect is launched.

Tested: pushing derivations built on other systems to attic works.

brainrake · 2024-01-31T01:34:38Z

+  imports = [ ../../flake-module.nix ];
+
+  options = {
+    push-cache-effect = {


We can have a cachix-push-effect and attic-push-effect to mirror cachix-deploy-effect instead of the type option. That could resolve passing the type in an unobtrusive way.

brainrake · 2024-01-31T01:40:08Z

+            secretsMap = { token-file = "${cacheOptions.secretName}"; };
+            userSetupScript = ''
+              attic login \
+                server-name \


This makes for a very confusing error message. We should use server at least.

Resolved: using default.

brainrake · 2024-01-31T01:46:10Z

+        '';
+        type = lib.types.attrsOf (lib.types.submodule ({ name, ... }: {
+          options = {
+            name = lib.mkOption {


The default should be be cachix|attic-push-effect or cachix|attic-push-${attrname}-effect to mirror cachix-deploy-effect.

brainrake · 2024-01-31T01:54:54Z

+
+                ```
+                  "data": {
+                    "name": "my-cache-name",


Should name and endpoint be in the secret? I think the attribute name can be used as name, and the endpoint doesn't need to be secret, it can be an option. But it's convenient to manage together with the token.

I think both are acceptable solutions.

brainrake · 2024-05-02T19:18:58Z

needs documentation

brainrake · 2024-05-02T19:20:58Z

i would separate the two effects since the abstraction is too complicated, there are significant differences, and there is still repetition.

brainrake · 2024-05-02T19:21:02Z

I'd also add mkCachixPushEffect and mkAtticPushEffect functions.

zmrocze · 2024-05-06T16:21:56Z

@brainrake what if instead type would take a list, i.e.: types = ["attic" "cachix"]?
How it is currently I agree that the effect should be split into two effects.

zmrocze requested a review from roberth as a code owner January 11, 2024 16:46

zmrocze mentioned this pull request Jan 11, 2024

Add populate-cache-effect #163

Closed

3 tasks

brainrake suggested changes Jan 30, 2024

View reviewed changes

This was referenced Jan 30, 2024

Push packages to attic mlabs-haskell/cardano.nix#12

Merged

Hercules CI's effect to push to attic mlabs-haskell/spo-anywhere#10

Merged

brainrake reviewed Jan 31, 2024

View reviewed changes

brainrake mentioned this pull request Mar 14, 2024

Add populate-cache-effect mlabs-haskell/hercules-ci-effects#1

Closed

3 tasks

brainrake force-pushed the push-cache-effect branch from 832a4a3 to f5ed263 Compare March 14, 2024 06:08

zmrocze mentioned this pull request May 5, 2024

Cardano-db-sync module mlabs-haskell/cardano.nix#59

Merged

zmrocze and others added 12 commits December 9, 2025 14:56

Add populate-cache-effect

79e0adf

Remove attic flake input & Rename to push & small fixes

4b445ec

Fix branches option

d4a0598

Rename

4a9bb42

Export push-cache effect

6c8d409

Move export

576339f

Format

e0c3ed0

Fix

13e452c

Fix path

52c05e2

Rename and unformat

1b5e244

change attic server name to "default"

1ab08b4

fix push-cache-effect when type=cachix branches=null

1361c6d

aciceri force-pushed the push-cache-effect branch from 5ad8f96 to 1361c6d Compare December 9, 2025 13:57

		@@ -0,0 +1,160 @@
		{ lib, withSystem, config, ... }:
		let pkgs-x86_64-linux = withSystem "x86_64-linux" ({ pkgs, ... }: pkgs);

Conversation

zmrocze commented Jan 11, 2024

Motivation

Questions

Maintainer checklist

Uh oh!

zmrocze commented Jan 11, 2024

Uh oh!

brainrake Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brainrake Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brainrake Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brainrake Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brainrake Jan 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

brainrake commented May 2, 2024

Uh oh!

brainrake commented May 2, 2024

Uh oh!

brainrake commented May 2, 2024

Uh oh!

zmrocze commented May 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brainrake Jan 30, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading

brainrake Jan 31, 2024 •

edited

Loading