sysdig build dependency requires network access

[rfrohl]

Describe the bug

The sysdig dependency during build is pulled from a git repository. This makes it kinda complicated to ship falco in a distribution. I was wondering if there is a specific reason that sysdig is pulled in via git.

How to reproduce it

build falco without network access.

Expected behaviour

CMake should pull the dependency from the system first and fall back to git.

Environment
Not sure if helpful: OBS build environment, trying to package falco for openSUSE.

• Falco version:
  0.19.0
• System info:
  not applicable
• Cloud provider or hardware configuration:
  not applicable
• OS:

$ cat /etc/os-release 
NAME="openSUSE Tumbleweed"
# VERSION="20200128"
ID="opensuse-tumbleweed"
ID_LIKE="opensuse suse"
VERSION_ID="20200128"
PRETTY_NAME="openSUSE Tumbleweed"
CPE_NAME="cpe:/o:opensuse:tumbleweed:20200128"

• Kernel:
  not applicable
• Installation method:
  not applicable

Additional context

Would be willing to work on a fix, but wanted to get upstream feedback first.

[leodido]

Hello @rfrohl, every third-party dependency is taken via the network.

Building without network access is not a goal for us at the moment.

In any case, we can discuss alternatives approaches.

A possible approach could be to provide CMake variables to specify paths for all third-party dependencies, to use them rather than git repositories.

Anyway, we discuss such matters into community calls. Do you mind joining the next one (tomorrow) ?

[leodido]

/kind feature

[fntlnz]

@rfrohl We take many dependencies from the network, can you please share what is your workflow?
From what I can see it looks like you have three different stages:

stage 1. git clone Falco from a machine with internet access
stage 2. Move the Falco dir to a build machine
stage 3. Try to build Falco but it fails because of no internet

We could implement a mechanism to allow you to retrieve dependencies at stage 1 so that when you copy you can just build.

[rfrohl]

We take many dependencies from the network, can you please share what is your workflow?

Sure, sorry that it was unclear in the initial report. It is basically a normal rpm build:

• take the released version (i.e. 0.19.0.tar.gz) and commit it to the source tracker.
• write spec file, which sets up the build environment and pulls all the dependencies from the distro (like protobuf, zlib, openssl...)
• this works for most (all other? ) things beside sysdig
  • even though I have the recent version (0.26.5) present in the build environment, during the build there are always attempts to pull it in over git.

As far as I understand it at the moment this is because of the way the current cmake module for sysdig is written.

The core problem is that there is no network access during the build process. All dependencies are pulled via the build system from the versions present in the distro. Once this is done, the network is disabled and the build is started.

We could implement a mechanism to allow you to retrieve dependencies at stage 1 so that when you copy you can just build.

The only problem seems to be cmake, which pulls the sysdig dependency via git. I currently think it would be enough to change the cmake module. If this is okay for you I would try to work on this tomorrow.

One of the reasons why I opened the ticket, was to make sure that there is no dependency on unreleased features from sysdig.

[rfrohl]

Anyway, we discuss such matters into community calls. Do you mind joining the next one (tomorrow) ?

For some reason your comment did not show, when I wrote the first response. I will try to make the call.

edit: I joined the call, but to be honest I did not know what the expectation was. So I guess that did not really lead anywhere.

[afbjorklund]

I had the same issue with Buildroot (for inclusion in minikube)

kubernetes/minikube#6560

As a workaround, I downloaded the sysdig tarball and edited the URL in the cmake script.
This complicates the build process, but on the other hand it was required to make it work...

I was able to use the bundled deps for other things, everything except ncurses and libyaml.

But the downloading of the sysdig dependency failed, and thus I couldn't make the driver:

  Each download failed!

    error: downloading 'https://github.com/draios/sysdig/archive/146a431edf95829ac11bfd9c85ba3ef08789bffe.tar.gz' failed
         status_code: 1
         status_string: "Unsupported protocol"
         log:
         --- LOG BEGIN ---
         Protocol "https" not supported or disabled in libcurl

The other issue was that I had to set FALCO_VERSION to set the version (to 0.19.0)
Otherwise it was trying to use git to find it, and got the version of the build system instead...

So I would also prefer if the build didn't use git (or the network)

---

Here is my workaround, the exact syntax may of course vary between build systems.

# see cmake/modules/sysdig-repo/CMakeLists.txt
SYSDIG_VERSION = 146a431edf95829ac11bfd9c85ba3ef08789bffe
FALCO_PROBE_EXTRA_DOWNLOADS = https://github.com/draios/sysdig/archive/${SYSDIG_VERSION}.tar.gz

define FALCO_PROBE_SYSDIG_SRC
        sed -e 's|URL ".*"|URL "'$(BR2_DL_DIR)/falco-probe/$(SYSDIG_VERSION).tar.gz'"|' -i $(@D)/cmake/modules/sysdig-repo/CMakeLists.txt
endef

FALCO_PROBE_POST_EXTRACT_HOOKS += FALCO_PROBE_SYSDIG_SRC

The commented out sha256 checksum (URL_HASH) didn't seem to match, though.

# The sysdig git reference (branch name, commit hash, or tag)

# To update sysdig version for the next release, change the default below
# In case you want to test against another sysdig version just pass the variable - ie., `cmake -DSYSDIG_VERSION=dev ..`
if(NOT SYSDIG_VERSION)
  set(SYSDIG_VERSION "146a431edf95829ac11bfd9c85ba3ef08789bffe")
endif()

ExternalProject_Add(
  sysdig
  URL "https://github.com/draios/sysdig/archive/${SYSDIG_VERSION}.tar.gz"
  # URL_HASH SHA256=bd09607aa8beb863db07e695863f7dc543e2d39e7153005759d26a340ff66fa5
  CONFIGURE_COMMAND ""
  BUILD_COMMAND ""
  INSTALL_COMMAND ""
  TEST_COMMAND "")

I got a different checksum, for 0.26.5-9-g146a431 (146a431edf95829ac11bfd9c85ba3ef08789bffe.tar.gz).

sha256 6e477ac5fe9d3110b870bd4495f01541373a008c375a1934a2d1c46798b6bad6 146a431edf95829ac11bfd9c85ba3ef08789bffe.tar.gz

  File will not be verified since no URL_HASH specified

[rfrohl]

As a workaround, I downloaded the sysdig tarball and edited the URL in the cmake script.
This complicates the build process, but on the other hand it was required to make it work...

I was thinking about bundling the sysdig tarball myself, but only as a workaround. The problem with solutions like this is that dependencies won't be updated once they are embedded. Therefor I decided to open this bug instead.

Sadly I did not find the time yet to look into the cmake module. So I am not sure how much effort it would be to make the change or if there are any other problems that I missed so far. Still on my todo list though.

[fntlnz]

@rfrohl if you want I can walk you trough our cmake files, you can get up to speed fast and propose changes for your kind of use case. I have a very good understanding of how it works. LMK

[afbjorklund]

I think I actually prefer this tarball/checksum approach, if it had been git submodules it would have been more work (to avoid git)... For now, I only need to avoid the version check and change the URL.

• cmake/modules/GetFalcoVersion.cmake

• cmake/modules/sysdig-repo/CMakeLists.txt

Then it avoids trying to use git (on the wrong repo) to get the version, and avoids trying to use the network (and failing) to get the sysdig dependency... I think the same approach works for rpm.

https://build.opensuse.org/package/show/home:rfrohl/falco

I guess there is no git repo at all, in the rpm buildroot :

[ 49s] -- Falco version: 0.GIT-NOTFOUND

@rfrohl : You might want to set FALCO_VERSION too

[afbjorklund]

Workaround still holding, and I can update both of them with some simple lines:

FALCO_PROBE_VERSION = 0.21.0
FALCO_PROBE_SYSDIG_VERSION = be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94

sha256 b1c9884855d58be94a97b2e348bcdc7db995800f0405b0f4e9a7176ee2f094a7 0.21.0.tar.gz
sha256 1c69363e4c36cdaeed413c2ef557af53bfc4bf1109fbcb6d6e18dc40fe6ddec8 be1ea2d9482d0e6e2cb14a0fd7e08cbecf517f94.tar.gz

However, the URL is hard-coded so I do have to edit (dirty) the source code...

ExternalProject_Add(
  sysdig
  URL "https://github.com/draios/sysdig/archive/${SYSDIG_VERSION}.tar.gz"
  URL_HASH "${SYSDIG_CHECKSUM}"

i.e. there is no SYSDIG_URL variable to override, with the local file path.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Issues labeled "cncf", "roadmap" and "help wanted" will not be automatically closed. Please refer to a maintainer to get such label added if you think this should be kept open.

[leodido]

/help

[poiana]

@leodido:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[poiana]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

[poiana]

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

[poiana]

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue with /reopen.

Mark the issue as fresh with /remove-lifecycle rotten.

Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[afbjorklund]

@leodido : did you need help with the sysdig URL ?

if(NOT SYSDIG_URL)
  set(SYSDIG_URL "https://github.com/draios/sysdig/archive")
endif()

  sysdig
  URL "${SYSDIG_URL}/${SYSDIG_VERSION}.tar.gz"
  URL_HASH "${SYSDIG_CHECKSUM}"

Or was there some better way to use a local copy ?