Include python wheel packages when moving downloaded dependencies archives

[pchila]

What does this PR do?

This commit modifies movePackagesToArchive() function to include python wheel packages introduced with commit 701f8b9 .
This solves an issue when packaging docker images using a DROP_PATH env var: prior to this change, the python wheel archives present in DROP_PATH were not moved to <DROP PATH>/archive/<platform> failing to create an image for which the Dockerfile expects to install the python wheel package

Why is it important?

To have a successful docker image creation during unified build processes.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Package the docker image "service" for elastic agent using a DROP_PATH (and optionally a manifest), for example:

AGENT_DROP_PATH=./build/agentdrop MANIFEST_URL=https://snapshots.elastic.co/9.0.0-3ca48ed8/agent-package/agent-artifacts-9.0.0-SNAPSHOT.json SNAPSHOT=true PLATFORMS="linux/amd64" PACKAGES="docker" DOCKER_VARIANT="service" mage -v downloadManifest packageUsingDRA

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request does not have a backport label. Could you fix it @pchila? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

[mergify]

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[blakerouse]

Thanks for fixing this. Packaging process is complicated.

[blakerouse]

Actually, I have an open question. Does this mean that the .zip is being included into all docker images? I wanted it to only add that zip in the -service variant. It should not increase the size of all docker containers.

[pchila]

Actually, I have an open question. Does this mean that the .zip is being included into all docker images? I wanted it to only add that zip in the -service variant. It should not increase the size of all docker containers.

Just tested generating all docker image variants with this fix: only elastic-agent-service image contains the connector but it's a bit hacky.
All docker images include all flattened dependencies (extracted in the flat directory by calling flattenDependencies()): in case of the connectors, there's an explicit removal of those packages from the flat dir, so the package remains only in the archives directory where the Dockerfile picks it up.

For example, launching

AGENT_DROP_PATH=./build/agentdrop MANIFEST_URL=https://snapshots.elastic.co/9.0.0-3ca48ed8/agent-package/agent-artifacts-9.0.0-SNAPSHOT.json SNAPSHOT=true PLATFORMS="linux/amd64" PACKAGES="docker" mage -v downloadManifest packageUsingDRA

results in these packaging directories (one per docker image variant)

➜  elastic-agent git:(fix-unified-release) ✗ tree -L 1  build/package/
build/package/
├── elastic-agent
├── elastic-agent-cloud
├── elastic-agent-complete
├── elastic-agent-complete-wolfi
├── elastic-agent-service
├── elastic-agent-ubi
└── elastic-agent-wolfi

7 directories, 0 files

If we search for connectors we find them only in the elastic-agent-service variant

➜  elastic-agent git:(fix-unified-release) ✗ find  build/package/ -name 'connectors*'
build/package/elastic-agent-service/elastic-agent-linux-amd64.docker/docker-build/beat/data/service/connectors-9.0.0-SNAPSHOT.zip
build/package/elastic-agent-service/elastic-agent-linux-amd64.docker/docker-build/beat/data/elastic-agent-0ebada/components/connectors
build/package/elastic-agent-service/elastic-agent-linux-amd64.docker/docker-build/beat/data/elastic-agent-0ebada/components/connectors.spec.yml

as specified by the agent_docker_service_spec where the .zip file is taken directly from the archives directory of the respective architecture name

  # service build is based on previous cloud variant
  - &agent_docker_service_spec
    docker_variant: 'service'
    files:
      'data/service/connectors-{{ beat_version }}{{if .Snapshot}}-SNAPSHOT{{end}}.zip':
        source: '{{.AgentDropPath}}/archives/{{.GOOS}}-{{.AgentArchName}}.tar.gz/connectors-{{ beat_version }}{{if .Snapshot}}-SNAPSHOT{{end}}.zip'
        mode: 0755
      'data/{{.BeatName}}-{{ commit_short }}/components/connectors':
        source: '{{ elastic_beats_dir }}/dev-tools/packaging/files/linux/connectors.sh'
        mode: 0755
      'data/{{.BeatName}}-{{ commit_short }}/components/connectors.spec.yml':
        source: '{{ elastic_beats_dir }}/specs/connectors.spec.yml'
        mode: 0644

[pchila]

Thanks for fixing this. Packaging process is complicated.

Indeed. 😃