Add buildkite pipeline for publishing to PyPI

[lorenabalan]

Related to https://github.com/elastic/connectors-py/issues/10999

• Added new manual pipeline for publishing to PyPI: .buildkite/pypi-publish-pipeline.yml
• Added new step in main pipeline (that runs on PRs and on daily schedule) to build the packages' binary & source distributions, then twine check them; for connectors_service it will also install it and test the CLI entrypoints it exposes (e.g. connectors --help)
• Changes to pyproject.toml
  • Incorporated pytest, pyright, and ruff config into pyproject.toml and removed individual files
  • Updated license in pyproject.toml to avoid deprecation warnings (removal in 2026) - see warnings here
  • Updated packages in pyproject.toml to avoid "ambiguous packages" warning - see warning here

❓ I don't see diff.sh used anywhere.. can I delete it or am I missing sth? nevermind, I was looking for diff.sh instead of just diff

✍️ Process moving forward

Daily per PR:

• Validate that packages are built correctly
• "Publish DRA" part of pipeline.yml will contain:
  • building docker image by installing package locally by building and installing from distribution
  • build a .tar.gz and .whl of our packages

Release day:

• same process as today, a Docker image DRA is chosen as BC and then released
• additionally, trigger manual pipeline to publish to PyPI from appropriate commit; ⚠️ as agreed on 21/10/2025, this will only publish to Test PyPI.

TODO:

• Update catalog-info.yml
• Publish .tar.gz and .whl as DRA
• Ensure Docker images are built with installing code from .whl

Checklists

Pre-Review Checklist

• this PR does NOT contain credentials of any kind, such as API keys or username/passwords (double check config.yml.example)
• this PR has a meaningful title
• this PR links to all relevant github issues that it fixes or partially addresses
• if there is no GH issue, please create it. Each PR should have a link to an issue
• this PR has a thorough description
• Covered the changes with automated tests
• Tested the changes locally
• Added a label for each target release version (example: v7.13.2, v7.14.0, v8.0.0)
• For bugfixes: backport safely to all minor branches still receiving patch releases
• Considered corresponding documentation changes
• Contributed any configuration settings changes to the configuration reference
• if you added or changed Rich Configurable Fields for a Native Connector, you made a corresponding PR in Kibana

Changes Requiring Extra Attention

• Security-related changes (encryption, TLS, SSRF, etc)
• New external service dependencies added.

Related Pull Requests

Release Note

[artem-shelkovnikov]

❓ I don't see diff.sh used anywhere.. can I delete it or am I missing sth?

diff.sh is used in here: https://github.com/elastic/connectors/blob/main/.buildkite/pipeline.yml#L84

❓ 🧠 What happens if we have a bug in our package? Requesting an emergency stack release in order to release a new version on PyPI seems excessive, so does yanking every buggy package from PyPI. I'm more in favour of having different versions conventions but that will get easily confusing.

We had a {major}.{minor}.{patch}.{release} available at some point, but since we release with stack it could be hard to organise well. We might need more strict QA process for it? I'd open a discussion with our team and potentially the clients team - they can give a good hint about it.

Re: releasing
For now the process for release is "promoting" the image. So by BC1 we already need to have an image with a real package installed.

Additionally, if we feel like it's too much we can keep the current process for docker images around for now (same as we do now with make install and such) and as we polish the process we'll switch to it.

[artem-shelkovnikov]

So this adds the wheels and tarballs to the artefacts that we release, and then we'll change this in the other place that uses them, right?

[lorenabalan]

I'm not sure what "the other place" is. 😅
What this PR does is

• on push to target branch (main or version branch), build and test wheels, then publish them as DRAs (my limited understanding was that simply putting them in the right folder was enough??)
• on push to target branch, build Docker image (by building and installing from wheel; it builds again, doesn't download the one from DRA) and publish as DRA

[seanstory]

https://github.com/elastic/infra/blob/master/cd/release/release-manager/project-configs/master/connectors.gradle is where our DRA artifacts are defined to be expected. We'll need to update that.

Then, I think @artem-shelkovnikov might be referencing is Elastic Agent's consumption of our current zip artifact. It's used here (see original PR). However, I think you and I agree that this doesn't need to be changed right now.

[lorenabalan]

Raised here https://github.com/elastic/infra/pull/43022

[artem-shelkovnikov]

Let's gooooo 🚀 🌒

[seanstory]

Great work. Sorry for leaving a lot of little nits.