Add support for conda lock file

[munishchouhan]

depends upon seqeralabs/libseqera#25
the above PR needs 'git revert d088604' before merging

This PR will add the following

1. upload lockfile to bucket
2. download lockfile from bucket
3. link in build page

[pditommaso]

Good start. To tell the truth, still not sure we should go ahead with this approach or just store the lock file in the surreal db like we are doing for the conda env, even tho it there's the possibility to have the same problem as #559

[munishchouhan]

Good start. To tell the truth, still not sure we should go ahead with this approach or just store the lock file in the surreal db like we are doing for the conda env, even tho it there's the possibility to have the same problem as #559

#559 will be solved in surrealdb version 2.0.0

[pditommaso]

I'm bit confused by this comment in the issue

SURREAL_HTTP_MAX_ML_BODY_SIZE (defaults to 4 GiB)
SURREAL_HTTP_MAX_SQL_BODY_SIZE (defaults to 1 MiB)
SURREAL_HTTP_MAX_RPC_BODY_SIZE (defaults to 4 MiB)
SURREAL_HTTP_MAX_KEY_BODY_SIZE (defaults to 16 KiB)
SURREAL_HTTP_MAX_SIGNUP_BODY_SIZE (defaults to 1 KiB)
SURREAL_HTTP_MAX_SIGNIN_BODY_SIZE (defaults to 1 KiB)
SURREAL_HTTP_MAX_IMPORT_BODY_SIZE (defaults to 4 GiB)

It seems suggesting the default sql body size is 1 MB, instead the error we are hitting is much smaller.

[munishchouhan]

It seems suggesting the default sql body size is 1 MB, instead the error we are hitting is much smaller.

we are using /key routes, which is constrained by 16 KiB
SURREAL_HTTP_MAX_KEY_BODY_SIZE (defaults to 16 KiB)

[munishchouhan]

if we use sql to store it then we can bypass this limit

[pditommaso]

This sounds like a plan. please give it a try

[munishchouhan]

This sounds like a plan. please give it a try

ok sure

[munishchouhan]

There is an issue accessing the Conda lock file. The lock file is present in the generated image, not in the buildkit container we are running in Wave.
so either we need to pull the image in another pod and get the file or we need to generate conda lockfile from conda.yml file

cc @ewels @pditommaso

[munishchouhan]

in latter case of generating conda lockfile from conda file, we still need another job to achieve that

[ewels]

Better to get the file from the container - I was trying to avoid generating the lock file separately because then there's no absolute guarantee that it'll end up the same as the actual environment. If it comes from the environment itself it's certain.

[ewels]

Can we print the lock file to stdout and then capture that from the build?

[munishchouhan]

Can we print the lock file to stdout and then capture that from the build?

we can do this:

FROM {{base_image}}
COPY --chown=$MAMBA_USER:$MAMBA_USER conda.yml /tmp/conda.yml
RUN micromamba install -y -n base -f /tmp/conda.yml \
    {{base_packages}}
    && micromamba env export --explicit > environment.lock \
    && cat environment.lock
    && micromamba clean -a -y
RUN 
USER root
ENV PATH="$MAMBA_ROOT_PREFIX/bin:$PATH"

I ran it for conda package 'bwa'
i got this in the stdout

#10 12.24 # This file may be used to create an environment using:
#10 12.24 # $ conda create --name <env> --file <this file>
#10 12.24 # platform: linux-aarch64
#10 12.24 @EXPLICIT

[ewels]

Exactly - that works!

Were there a load of lines after the @EXPLICIT yeah?

We'd need to remove the line prefixes but that's all I think..

[pditommaso]

A better approach (maybe) could be: 1) creating the container "locally"; 2) copy the lock file from the built container via buildkit; 3) uploading it to the registry.

Something similar is done for singularity, here.

[munishchouhan]

A better approach (maybe) could be: 1) creating the container "locally"; 2) copy the lock file from the built container via buildkit; 3) uploading it to the registry.

Something similar is done for singularity, here.

Ok sure, I will try this one

[pditommaso]

what if the lock file contains a ' ?

[munishchouhan]

good point
I have not tested yet with surrealDB
I will use bytes datatype to save it

[munishchouhan]

changed to byte[]

[pditommaso]

Added [email protected] and shortened the lock markers. Tried a build but cannot see any lock file. @munishchouhan what am i missing?

[munishchouhan]

Added [email protected] and shortened the lock markers. Tried a build but cannot see any lock file. @munishchouhan what am i missing?

may bewave.build.conda-lock-prefix

[pditommaso]

may bewave.build.conda-lock-prefix

That was missing indeed, now I see the upload but it's not rendering in the view. checking

[pditommaso]

ok works

[munishchouhan]

@pditommaso is this integration test is enough or were you referring to some other test?

wave/src/test/groovy/io/seqera/wave/service/logs/BuildLogsServiceTest.groovy

Line 109 in fbe5f99

def 'should extract conda lockfile from s3' (){

[pditommaso]

Ideally it would nice to have a test passing trough the controller, building a conda container and check the lock file exists. S3 uploading not so important, could be mocked. More important lock file is grabbed.

[munishchouhan]

Ideally it would nice to have a test passing trough the controller, building a conda container and check the lock file exists. S3 uploading not so important, could be mocked. More important lock file is grabbed.

I gave it try and able to succeed with full flow, but when we do build in test and image is uploaded to the registry, next time when the same test runs, it will find that image and return cached and in that cache we will not have conda lockfile

I will try to mock the build and send mockedevent to do this testing

[munishchouhan]

I overcome the caching issue by providing the invalid repo. so the push image part fails and everytime we run the test, it creates a fresh build

[munishchouhan]

this e2e test tasks long time to complete, what we can do is add another module for e2e test, which runs on demand and when merging a pr to master

[pditommaso]

Ok, let's keep the e2e in a separate PR and merge this one

[munishchouhan]

Ok, let's keep the e2e in a separate PR and merge this one

yes, working on that,
Ok I will remove the e2e from this PR

[pditommaso]

this still requires the update in libseqera right?

[pditommaso]

no, already merged 👍

[munishchouhan]

Tested on local: