Skip to content

Make connectors work with python 3.10 and 3.11 #2734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
artem-shelkovnikov merged 70 commits into from
Aug 14, 2024
Merged

Make connectors work with python 3.10 and 3.11 #2734

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
f1f44c9
Make connectors work with python 3.10+
artem-shelkovnikov Aug 5, 2024
a3f0c92
Trying out python with different versions
artem-shelkovnikov Aug 5, 2024
4afbff0
Hmm maybe this?
artem-shelkovnikov Aug 5, 2024
e2dfa40
Nah forget it
artem-shelkovnikov Aug 5, 2024
31937ef
Revert correctly
artem-shelkovnikov Aug 5, 2024
7d8fbcf
Get rid of alpine + add entrypoint
artem-shelkovnikov Aug 6, 2024
3043ed1
Hmmmm
artem-shelkovnikov Aug 6, 2024
6bb2024
hmm
artem-shelkovnikov Aug 6, 2024
15e2e8b
Maybe this?
artem-shelkovnikov Aug 6, 2024
3eb646f
Well, whatever
artem-shelkovnikov Aug 6, 2024
5d8d59a
Let's see
artem-shelkovnikov Aug 9, 2024
40de2c8
Add LL
artem-shelkovnikov Aug 9, 2024
f71d125
Typo
artem-shelkovnikov Aug 9, 2024
aa6a0cc
Let's do with sudo
artem-shelkovnikov Aug 9, 2024
48fc732
Catting bashrc
artem-shelkovnikov Aug 9, 2024
c4f4c57
Whoami?
artem-shelkovnikov Aug 9, 2024
400f167
Drop LL
artem-shelkovnikov Aug 9, 2024
eaa53b9
Well let's try again
artem-shelkovnikov Aug 9, 2024
88601e3
Just for experiment
artem-shelkovnikov Aug 9, 2024
274f14f
Try this?
artem-shelkovnikov Aug 9, 2024
ffbf840
Use just python, not python3
artem-shelkovnikov Aug 9, 2024
3cd5636
Small changes here and there
artem-shelkovnikov Aug 9, 2024
0efea12
This?
artem-shelkovnikov Aug 9, 2024
973e21a
Cat profile
artem-shelkovnikov Aug 9, 2024
fae692e
Let's run stuff via shell scripts
artem-shelkovnikov Aug 9, 2024
c609d97
Hope linter/tester will work
artem-shelkovnikov Aug 9, 2024
290d58f
Sauce
artem-shelkovnikov Aug 9, 2024
525a0e1
Fix linter step
artem-shelkovnikov Aug 9, 2024
1f3b03b
Some fixes for linter/tester
artem-shelkovnikov Aug 9, 2024
cfc90fc
Fix the pipeline
artem-shelkovnikov Aug 9, 2024
9674871
Fix command for ftest
artem-shelkovnikov Aug 9, 2024
53c826f
A bit of echo
artem-shelkovnikov Aug 9, 2024
2a31de2
Source bashprofile for functional test
artem-shelkovnikov Aug 9, 2024
03fa17f
Minor fixes here and there
artem-shelkovnikov Aug 9, 2024
bba9d4b
Make it also work for nightlies
artem-shelkovnikov Aug 9, 2024
dd5d7a7
Also update labels for nightlies
artem-shelkovnikov Aug 9, 2024
ee78e91
Revert the diff plugin
artem-shelkovnikov Aug 9, 2024
7970b6e
Well that did not work
artem-shelkovnikov Aug 9, 2024
d4a9c8f
Update diff plugin
artem-shelkovnikov Aug 9, 2024
feebee5
Add version to Network Drive on nightly
artem-shelkovnikov Aug 9, 2024
205351c
Update some libs
artem-shelkovnikov Aug 9, 2024
0dfcd53
Let's try this
artem-shelkovnikov Aug 9, 2024
67da728
Let's try different plugin + one timestamp fix
artem-shelkovnikov Aug 9, 2024
6ec4728
Remove top level env?
artem-shelkovnikov Aug 9, 2024
e23a9a2
Maybe this?
artem-shelkovnikov Aug 9, 2024
0158e53
Maybe like this?
artem-shelkovnikov Aug 9, 2024
49aa01d
Try with nightly steps
artem-shelkovnikov Aug 9, 2024
7bd7095
Try with nightly steps 2
artem-shelkovnikov Aug 9, 2024
d00f0cf
Arrays?
artem-shelkovnikov Aug 9, 2024
e09ca7d
This syntax?
artem-shelkovnikov Aug 9, 2024
14e8777
One more fight
artem-shelkovnikov Aug 9, 2024
4c4dbfe
Maybe this?
artem-shelkovnikov Aug 9, 2024
85be9e4
hmm
artem-shelkovnikov Aug 9, 2024
1a1a868
This?
artem-shelkovnikov Aug 9, 2024
afc9000
Fix other
artem-shelkovnikov Aug 9, 2024
a3f8eed
Fix nightlies
artem-shelkovnikov Aug 9, 2024
ac96093
Fix oracledb ftest
artem-shelkovnikov Aug 9, 2024
c619d0b
Also fix nightly for dropbox
artem-shelkovnikov Aug 9, 2024
b911c4e
Fix oracle nightly + revert slack change
artem-shelkovnikov Aug 9, 2024
2db6f29
Autoformat
artem-shelkovnikov Aug 9, 2024
10741fa
Drop support for 3.12
artem-shelkovnikov Aug 9, 2024
10db9c7
Try perf8 with sha
artem-shelkovnikov Aug 12, 2024
05495f2
Reduce data_size to small
artem-shelkovnikov Aug 12, 2024
455d64e
Swap wolvi images to 3.12
artem-shelkovnikov Aug 12, 2024
b7ca9d5
Revert entrypoint for Dockerfile
artem-shelkovnikov Aug 12, 2024
867d90a
Move Dockerfile.ftest to -slim-bookworm
artem-shelkovnikov Aug 12, 2024
3eed917
Remove SHA for perf8
artem-shelkovnikov Aug 12, 2024
f860a51
Merge branch 'main' into artem/relax-python-requirements
artem-shelkovnikov Aug 12, 2024
b9c3693
s/3.10+/3.10 or 3.11/
artem-shelkovnikov Aug 12, 2024
439b2bc
Minor adjustments
artem-shelkovnikov Aug 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
313 changes: 235 additions & 78 deletions .buildkite/nightly_steps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,133 +2,290 @@ env:
DOCKERFILE_FTEST_PATH: "Dockerfile.ftest.wolfi"

steps:
- label: "🔨 MySQL"
command:
- ".buildkite/run_nigthly.sh mysql"
- label: "🔨 [Python {{ matrix }}] MySQL"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "mysql"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Network Drive"
command:
- ".buildkite/run_nigthly.sh network_drive"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Network Drive"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "network_drive"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Amazon S3"
command:
- ".buildkite/run_nigthly.sh s3"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Amazon S3"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "s3"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Google Cloud Storage"
command:
- ".buildkite/run_nigthly.sh google_cloud_storage"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Google Cloud Storage"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "google_cloud_storage"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Azure Blob Storage"
command:
- ".buildkite/run_nigthly.sh azure_blob_storage"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Azure Blob Storage"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "azure_blob_storage"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Postgresql"
command:
- ".buildkite/run_nigthly.sh postgresql"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Postgresql"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "postgresql"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 System Directory"
command:
- ".buildkite/run_nigthly.sh dir"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] System Directory"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "dir"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Oracle Database"
command:
- ".buildkite/run_nigthly.sh oracle"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Oracle Database"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "oracle"
DATA_SIZE: "small"
SKIP_AARCH64: "true"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Sharepoint Server"
command:
- ".buildkite/run_nigthly.sh sharepoint_server"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Sharepoint Server"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "sharepoint_server"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Sharepoint Online"
command:
- ".buildkite/run_nigthly.sh sharepoint_online"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Sharepoint Online"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "sharepoint_online"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Microsoft SQL"
command:
- ".buildkite/run_nigthly.sh mssql"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Microsoft SQL"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "mssql"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Jira"
command:
- ".buildkite/run_nigthly.sh jira"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Jira"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "jira"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Confluence"
command:
- ".buildkite/run_nigthly.sh confluence"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Confluence"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "confluence"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 ServiceNow"
command:
- ".buildkite/run_nigthly.sh servicenow"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] ServiceNow"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "servicenow"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 MongoDB"
command:
- ".buildkite/run_nigthly.sh mongodb"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] MongoDB"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "mongodb"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
# - label: "🔨 MongoDB Serverless"
# command:
# - ".buildkite/run_nigthly.sh mongodb_serverless"
matrix:
- "3.10"
- "3.11"
# - label: "🔨 [Python {{ matrix }}] MongoDB Serverless"
# command: ".buildkite/run_functional_test.sh"
# env:
# PYTHON_VERSION: "{{ matrix }}"
# CONNECTOR: "mongodb_serverless"
# DATA_SIZE: "small"
# matrix:
# - "3.10"
# - "3.11"
# artifact_paths:
# - "perf8-report-*/**/*"
- label: "🔨 GitHub"
command:
- ".buildkite/run_nigthly.sh github"
- label: "🔨 [Python {{ matrix }}] GitHub"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "github"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Google Drive"
command:
- ".buildkite/run_nigthly.sh google_drive"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Google Drive"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "google_drive"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Dropbox"
command:
- ".buildkite/run_nigthly.sh dropbox"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Dropbox"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "dropbox"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 OneDrive"
command:
- ".buildkite/run_nigthly.sh onedrive"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] OneDrive"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "onedrive"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Salesforce"
command:
- ".buildkite/run_nigthly.sh salesforce"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Salesforce"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "salesforce"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Zoom"
command:
- ".buildkite/run_nigthly.sh zoom"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Zoom"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "zoom"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Box"
command:
- ".buildkite/run_nigthly.sh box"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Box"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "box"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Notion"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "notion"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 Notion"
command:
- ".buildkite/run_nigthly.sh notion"
- label: "🔨 Redis"
command:
- ".buildkite/run_nigthly.sh redis"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] Redis"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "redis"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
- label: "🔨 GraphQL"
command:
- ".buildkite/run_nigthly.sh graphql"
matrix:
- "3.10"
- "3.11"
- label: "🔨 [Python {{ matrix }}] GraphQL"
command: ".buildkite/run_functional_test.sh"
env:
PYTHON_VERSION: "{{ matrix }}"
CONNECTOR: "graphql"
DATA_SIZE: "small"
artifact_paths:
- "perf8-report-*/**/*"
matrix:
- "3.10"
- "3.11"
Loading