Add elastic-agent-service container to packaging by blakerouse · Pull Request #5349 · elastic/elastic-agent

blakerouse · 2024-08-23T17:00:18Z

What does this PR do?

Adds a new docker variant of elastic-agent-service that includes the python connectors.

Currently this PR includes the following PR until that PR is merged - #5338

Why is it important?

Used by the Elastic Agent service.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~[ ] I have made corresponding changes to the documentation~~
~~[ ] I have made corresponding change to the default configuration files~~
~~[ ] I have added tests that prove my fix is effective or that my feature works~~
~~[ ] I have added an entry in ./changelog/fragments using the changelog tool~~ (not really user facing at this level)
~~[ ] I have added an integration test or an E2E test~~

Disruptive User Impact

None.

How to test this PR locally

$ PLATFORMS="linux/arm64" PACKAGES="docker" SNAPSHOT="true" EXTERNAL="true" DOCKER_VARIANTS="service" mage package
$ docker run -it --rm docker.elastic.co/beats-ci/elastic-agent-service:8.16.0-SNAPSHOT

seanstory

Few nits but this is super exciting. Thank you so much for your help, it would have taken us ages to figure out how to stitch into this system. 🙏

seanstory · 2024-08-23T18:57:53Z

dev-tools/packaging/packages.yml

+        source: '{{.AgentDropPath}}/archives/{{.GOOS}}-{{.AgentArchName}}.tar.gz/connectors-{{ beat_version }}{{if .Snapshot}}-SNAPSHOT{{end}}.zip'
+        mode: 0755
+      'data/{{.BeatName}}-{{ commit_short }}/components/connectors':
+        source: '{{ elastic_beats_dir }}/dev-tools/packaging/files/linux/connectors.py'


I expect this will end up being a bash script rather than python, but I don't think that really matters right now.

Yeah it doesn't matter. Just need to know what the contents should be.

seanstory · 2024-08-23T18:59:54Z

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl


-{{- if (and (contains .image_name "-complete") (contains .from "ubuntu"))  }}
+{{- if eq .Variant "service" }}
+RUN apk add --no-cache python-3.12 py3.12-pip && \


We don't support python 3.12 yet :( It's on our roadmap, but 3.11 is probably better for now. See elastic/connectors#2734 for details

I can lower the version, once we know whats in the execution script. By the time its ready you might support 3.12 :-)

I don't expect 3.12 support to make it that fast. :) We were on 3.10 for about 2 years before we just added 3.11 a few weeks ago. Urgency is not there.

seanstory · 2024-08-23T19:02:17Z

dev-tools/packaging/templates/docker/Dockerfile.elastic-agent.tmpl

-{{- if (and (contains .image_name "-complete") (contains .from "ubuntu"))  }}
+{{- if eq .Variant "service" }}
+RUN apk add --no-cache python-3.12 py3.12-pip && \
+    pip install {{ $beatHome }}/data/service_downloads/connectors-*.zip && \


Would be safer to unzip and make install, as that'll also give it a python virtual environment, and keep it resileint to any other system python deps that might end up in this image one day.

I can do that, seems a little un-needed being this is the only python used in the image. The simple pip install without requiring unzip to be installed in the image seems better to me.

it's best practice (AFAIU) to not use system python if you can help it. Also by using a make target, it allows us to hook into other setup bits. For example, our make isntall insures that bin/pip install --upgrade pip and bin/pip install --upgrade setuptools are both run, and also ensure that the right requirements/$ARCH.txt is selected, depending upon the OS that it's being run on.

blakerouse · 2024-08-26T13:38:34Z

Just note I am blocked on making any more progress on this PR, under we know what the contents of the bash script should be and I believe we are waiting for the python version of the elastic-agent-client to be included in the DRA or possibly in a pypi dist (some way of getting it).

seanstory · 2024-08-26T18:10:08Z

I am blocked on making any more progress on this PR, under we know what the contents of the bash script should be

Can we not merge it as is, and follow up later with edits to make the binary more "real"? Is there some sort of testing bar that happens for each included binary that we need to to be able to meet?

I believe we are waiting for the python version of the elastic-agent-client to be included in the DRA

@artem-shelkovnikov is currently working on that, but is needing to do some repo restructuring so that the private-repo py-elastic-agent-client lib can be installed in this docker image, but won't prevent community members from cloning and building self-managed connectors from source.

mergify · 2024-08-28T14:55:55Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-service-docker upstream/agent-service-docker
git merge upstream/main
git push upstream agent-service-docker

artem-shelkovnikov · 2024-09-04T13:55:52Z

👋 We should be unblocked on making progress on this PR.

We've recently merged elastic/connectors#2787 that has a component ready to be ran.

To run the component you'd need:

Pull the repo
Run make install-agent in the root of the repo
Use python from the virtual environment to run the component: .venv/bin/python connectors/agent/cli.py

I tested this code locally by having an agent downloaded and adding component and component spec into the components with following code (would need to be adapted to run in the docker image):

artemshelkovnikov-m2@Work-M2 components % cat connectors-agent
#!/bin/bash
FOLDER=/Users/artemshelkovnikov-m2/git_tree/connectors-py/.venv/bin

$FOLDER/elastic-agent-connectors $FOLDER/elastic-agent-connectors

artemshelkovnikov-m2@Work-M2 components % cat connectors-agent.spec.yml
version: 2
inputs:
  - name: connectors-py
    description: "Connectors Py component input"
    platforms: &platforms
      - linux/amd64
      - linux/arm64
      - darwin/amd64
      - darwin/arm64
      - windows/amd64
      - container/amd64
      - container/arm64
    outputs: &outputs
      - elasticsearch
    shippers: &shippers
      - shipper
    command: &command
      restart_monitoring_period: 5s
      maximum_restarts_per_period: 1
      timeouts:
        restart: 1s
      args: []

seanstory · 2024-09-04T13:58:48Z

Woo!! Thanks, Artem!
@blakerouse , we were also talking in a call today with @cmacknz , who suggested that we should make sure that this PR also includes some basic smoke tests that validate that Agent can start the connector binary, and do some basic communication. This will be really valuable for us since we're using a new python agent client, so anything that can catch basic problems earlier in the build/test pipeline is good. :)

blakerouse · 2024-09-06T21:18:38Z

@artem-shelkovnikov I am unable to get this to actually build, I am getting the following error calling make install-agent. I also had to change it to also install git and make in the container to even get make install-agent to run, which is why I preferred the pip install .zip method.

Let me know how to solve this issue:

28.90 Due to the length of these fields, this option is best paired with --format=json.
28.91 .venv/bin/pip install -r requirements/agent.txt
29.04 Collecting elastic-agent-client@ git+https://github.com/elastic/python-elastic-agent-client@main (from -r requirements/agent.txt (line 1))
29.04   Cloning https://github.com/elastic/python-elastic-agent-client (to revision main) to /tmp/pip-install-aqhch75y/elastic-agent-client_7cbc275ad90345f7ac781fecfb1014d3
29.04   Running command git clone --filter=blob:none --quiet https://github.com/elastic/python-elastic-agent-client /tmp/pip-install-aqhch75y/elastic-agent-client_7cbc275ad90345f7ac781fecfb1014d3
29.22   fatal: could not read Username for 'https://github.com': No such device or address
29.22   error: subprocess-exited-with-error
29.22   
29.22   × git clone --filter=blob:none --quiet https://github.com/elastic/python-elastic-agent-client /tmp/pip-install-aqhch75y/elastic-agent-client_7cbc275ad90345f7ac781fecfb1014d3 did not run successfully.
29.22   │ exit code: 128
29.22   ╰─> See above for output.
29.22   
29.22   note: This error originates from a subprocess, and is likely not a problem with pip.
29.23 error: subprocess-exited-with-error
29.23 
29.23 × git clone --filter=blob:none --quiet https://github.com/elastic/python-elastic-agent-client /tmp/pip-install-aqhch75y/elastic-agent-client_7cbc275ad90345f7ac781fecfb1014d3 did not run successfully.
29.23 │ exit code: 128
29.23 ╰─> See above for output.
29.23 
29.23 note: This error originates from a subprocess, and is likely not a problem with pip.
29.25 make: Leaving directory '/usr/share/elastic-agent/data/service/elasticsearch_connectors-8.16.0'
29.25 make: *** [Makefile:28: install-agent] Error 1
------
Dockerfile:136
--------------------
 135 |         chown elastic-agent:elastic-agent /app
 136 | >>> RUN apk add --no-cache git make python-3.11 py3.11-pip && \
 137 | >>>     unzip /usr/share/elastic-agent/data/service/connectors-*.zip -d /usr/share/elastic-agent/data/service && \
 138 | >>>     make -C /usr/share/elastic-agent/data/service/elasticsearch_connectors-* install-agent && \
 139 | >>>     chmod 0755 /usr/share/elastic-agent/data/elastic-agent-*/components/connectors
 140 |     
--------------------

blakerouse · 2024-09-06T21:20:27Z

@seanstory I can add an integration test using this container, once I have it building and working.

artem-shelkovnikov · 2024-09-09T08:20:22Z

@blakerouse the problem comes from the fact that connectors dependency is a private Elastic repository.

We had to work around this with this change: https://github.com/elastic/ci/pull/3327

Additionally, we're gonna be opening the private repo ASAP.

I don't think it would work even if we provided a zip or any other artifact, as the problem is with access to the dependency?

blakerouse · 2024-09-09T12:48:16Z

@artem-shelkovnikov Opening it to be public would make this much easier and allow developers to build this locally with our simple mage package target in the Elastic Agent repository. ETA on public access?

artem-shelkovnikov · 2024-09-09T12:56:21Z

ETA unknown - I've submitted a request to #open-source here: https://github.com/elastic/open-source/issues/443

Open source team will look at it today and best case will allow to open it up, I'll follow up here after I get a response

seanstory · 2024-09-09T16:18:08Z

We just got the approval, and https://github.com/elastic/python-elastic-agent-client now has public/open visibility 🎉
@blakerouse can you try again?

blakerouse · 2024-09-09T17:23:38Z

@seanstory @artem-shelkovnikov Nice! I did get it to build. With it built I ran the container and tried to execute the elastic-agent-connectors binary and I am getting the following error (this should be using the venv for python):

bash-5.2$ data/service/elasticsearch_connectors-8.16.0/.venv/bin/elastic-agent-connectors 
Traceback (most recent call last):
  File "/usr/share/elastic-agent/data/service/elasticsearch_connectors-8.16.0/.venv/bin/elastic-agent-connectors", line 33, in <module>
    sys.exit(load_entry_point('elasticsearch-connectors', 'console_scripts', 'elastic-agent-connectors')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/elastic-agent/data/service/elasticsearch_connectors-8.16.0/.venv/bin/elastic-agent-connectors", line 25, in importlib_load_entry_point
    return next(matches).load()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'connectors.agent'

Check its using python from venv:

bash-5.2$ head data/service/elasticsearch_connectors-8.16.0/.venv/bin/elastic-agent-connectors
#!/usr/share/elastic-agent/data/service/elasticsearch_connectors-8.16.0/.venv/bin/python

seanstory · 2024-09-09T21:08:14Z

Fixed by elastic/connectors#2811. Sorry about that. New DRAs should be available momentarily. 🤞

blakerouse · 2024-09-10T00:59:45Z

@seanstory

I don't think the DRA has that change or even the change that was merged last week, or something is wrong

The contents of the DRA only show the following

bash-5.2$ ls
VERSION                __pycache__            config.py              content_extraction.py  filtering              logger.py              protocol               services               sync_job_runner.py
__init__.py            access_control.py      connectors_cli.py      es                     kibana.py              preflight_check.py     service_cli.py         source.py              utils.py

which doesn't have the agent or cli directory at all

seanstory · 2024-09-10T18:55:09Z

For those watching, we solved the missing dir issue with a three-part combo of:

and now we're discussing further hiccups in this slack thread.

mergify · 2024-09-10T21:09:55Z

backport-v8.x has been added to help with the transition to the new branch 8.x.

mergify · 2024-09-11T11:49:52Z

backport-v8.x has been added to help with the transition to the new branch 8.x.

mergify · 2024-09-11T12:28:37Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b agent-service-docker upstream/agent-service-docker
git merge upstream/main
git push upstream agent-service-docker

elasticmachine · 2024-09-20T20:30:32Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

blakerouse · 2024-09-20T20:31:13Z

@seanstory I finally have a passing integration test for this new container. Its all working now. Just need a review and it can be merged.

testing/integration/kubernetes_agent_standalone_test.go

pkoutsovasilis · 2024-09-20T21:30:09Z

hey @blakerouse 👋 I left only a minor comment that I think you need to address. However, may I join forces with you here and push a Helm-chart based integration test? I was looking for a reason to add a custom integration/input test for the agent Helm chart and this one seems like ideal. Or maybe it can be a follow up pull request, no prob with that either I have thought about it I am gonna open a follow up PR when this one is merged 😉

blakerouse · 2024-09-21T00:15:03Z

@pkoutsovasilis Fixed

pkoutsovasilis

LGTM

pchila · 2024-09-23T12:26:46Z

magefile.go

+		// never flatten any python wheels, the packages.yml and docker should handle
+		// those specifically so that the python wheels are installed into the container
+		matches = removePythonWheels(matches, packageVersion)


Is there an alternative to adding the PythonWheel binary specs to the ExpectedBinaries just to remove them from the flattened directory? The connectors are added to ExpectedBinaries just to download the archive ?
If the handling of connectors is different from the other components' binaries we package with agent maybe we can separate the handling with a collectAdditionalDependencies target after flattening or something similar ?
My understanding is also that these dependencies are not needed to create an agent archive but get installed when we run docker build, maybe we can move the processing at that stage ?

blakerouse · 2024-09-26T11:58:51Z

buildkite test this

blakerouse · 2024-09-26T20:47:13Z

buildkite test this

elastic-sonarqube · 2024-09-30T20:08:36Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

Adds the elastic-agent-service container with the py-connectors component enabled. (cherry picked from commit 701f8b9) # Conflicts: # dev-tools/mage/manifest/manifest.go

#5634) * Add elastic-agent-service container to packaging (#5349) Adds the elastic-agent-service container with the py-connectors component enabled. (cherry picked from commit 701f8b9) # Conflicts: # dev-tools/mage/manifest/manifest.go * Update manifest.go Fix conflicts caused by mergify * Include python wheel packages when moving downloaded dependencies archives (#5647) This commit modifies movePackagesToArchive() function to include python wheel packages introduced with commit 701f8b9 . This solves an issue when packaging docker images using a DROP_PATH env var: prior to this change, the python wheel archives present in DROP_PATH were not moved to `<DROP PATH>/archive/<platform>` failing to create an image for which the Dockerfile expects to install the python wheel package * Fix docker image build when multiple platforms are specified (#5658) Force merging to unblock the unified release process --------- Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: Julien Lind <julien.lind@elastic.co> Co-authored-by: Paolo Chilà <paolo.chila@elastic.co>

blakerouse added 2 commits August 21, 2024 11:37

Add ability to select docker variant.

77dd2a5

Add elastic-agent-service container to packaging.

d30bfb2

blakerouse added Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team backport-skip labels Aug 23, 2024

blakerouse self-assigned this Aug 23, 2024

seanstory reviewed Aug 23, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into agent-service-docker

caa9dfd

Call install-agent.

f9b3a8c

Merge remote-tracking branch 'upstream/main' into agent-service-docker

995f8ee

mergify bot added the backport-v8.x label Sep 10, 2024

blakerouse removed the backport-skip label Sep 11, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 11, 2024

blakerouse requested review from andrzej-stencel and pchila September 20, 2024 20:30

blakerouse requested review from pkoutsovasilis and removed request for andrzej-stencel September 20, 2024 20:31

pkoutsovasilis reviewed Sep 20, 2024

View reviewed changes

testing/integration/kubernetes_agent_standalone_test.go Show resolved Hide resolved

pkoutsovasilis reviewed Sep 20, 2024

View reviewed changes

testing/integration/kubernetes_agent_standalone_test.go Outdated Show resolved Hide resolved

Add backin output of status command.

ec872f9

pkoutsovasilis approved these changes Sep 21, 2024

View reviewed changes

mage check

99757d3

pchila reviewed Sep 23, 2024

View reviewed changes

Merge branch 'main' into agent-service-docker

01a0517

blakerouse enabled auto-merge (squash) September 26, 2024 20:47

blakerouse added 4 commits September 27, 2024 14:32

Merge branch 'main' into agent-service-docker

93f6cf4

Merge branch 'main' into agent-service-docker

577669e

Update the size of kubernetes runner.

45ef9d4

Merge remote-tracking branch 'upstream/main' into agent-service-docker

786e95d

blakerouse merged commit 701f8b9 into elastic:main Sep 30, 2024

mergify bot mentioned this pull request Sep 30, 2024

[8.x](backport #5349) Add elastic-agent-service container to packaging #5634

Merged

2 tasks

cmacknz mentioned this pull request Oct 30, 2024

Fix elastic-agent-service Docker container entrypoint #5892

Merged

seanstory mentioned this pull request Oct 24, 2025

Add buildkite pipeline for publishing to PyPI elastic/connectors#3748

Merged

17 tasks

pchila mentioned this pull request Nov 11, 2025

Fix dir name for moving elasticsearch connectors #11106

Merged

8 tasks

Conversation

blakerouse commented Aug 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Uh oh!

seanstory left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blakerouse commented Aug 26, 2024

Uh oh!

seanstory commented Aug 26, 2024

Uh oh!

mergify bot commented Aug 28, 2024

Uh oh!

artem-shelkovnikov commented Sep 4, 2024

Uh oh!

seanstory commented Sep 4, 2024

Uh oh!

blakerouse commented Sep 6, 2024

Uh oh!

blakerouse commented Sep 6, 2024

Uh oh!

artem-shelkovnikov commented Sep 9, 2024

Uh oh!

blakerouse commented Sep 9, 2024

Uh oh!

artem-shelkovnikov commented Sep 9, 2024

Uh oh!

seanstory commented Sep 9, 2024

Uh oh!

blakerouse commented Sep 9, 2024

Uh oh!

seanstory commented Sep 9, 2024

Uh oh!

blakerouse commented Sep 10, 2024

Uh oh!

seanstory commented Sep 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented Sep 10, 2024

Uh oh!

mergify bot commented Sep 11, 2024

Uh oh!

mergify bot commented Sep 11, 2024

Uh oh!

elasticmachine commented Sep 20, 2024

Uh oh!

blakerouse commented Sep 20, 2024

Uh oh!

Uh oh!

Uh oh!

pkoutsovasilis commented Sep 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

blakerouse commented Sep 21, 2024

Uh oh!

pkoutsovasilis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

blakerouse commented Aug 23, 2024 •

edited

Loading

seanstory commented Sep 10, 2024 •

edited

Loading

pkoutsovasilis commented Sep 20, 2024 •

edited

Loading