Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add kedro package as a CI check #236

Merged
merged 6 commits into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions .github/workflows/all-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,39 @@ jobs:
run: make install-test-requirements
- name: Run linter for all starters
run: behave features/lint.feature

package:
strategy:
matrix:
os: [ windows-latest , ubuntu-latest ]
python-version: [ "3.8", "3.9", "3.10", "3.11", "3.12" ]
runs-on: ${{ matrix.os }}
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Set up Python ${{matrix.python-version}}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install test requirements
run: |
make install-test-requirements
- name: Add MSBuild to PATH
if: matrix.os == 'windows-latest'
uses: microsoft/setup-msbuild@v1
- name: Install Microsoft Visual C++ Redistributable
if: matrix.os == 'windows-latest'
run: |
choco install vcredist-all
- name: Setup Hadoop binary
if: matrix.os == 'windows-latest'
run: |
Invoke-WebRequest "https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/winutils.exe?raw=true" -OutFile winutils.exe
Invoke-WebRequest "https://github.com/steveloughran/winutils/blob/master/hadoop-2.7.1/bin/hadoop.dll?raw=true" -OutFile hadoop.dll
Move-Item .\hadoop.dll C:\Windows\System32
New-Item -ItemType directory -Path C:\hadoop\bin
Move-Item .\winutils.exe C:\hadoop\bin
echo "HADOOP_HOME=C:\hadoop" | Out-File -Append -Encoding ascii -FilePath $env:GITHUB_ENV
echo "PATH=$env:HADOOP_HOME\bin;$env:PATH" | Out-File -Append -Encoding ascii -FilePath $env:GITHUB_ENV
noklam marked this conversation as resolved.
Show resolved Hide resolved
- name: Package and run all starters
run: behave features/package.feature
46 changes: 46 additions & 0 deletions features/package.feature
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
Feature: Package and run all starters

Scenario: Package astro-airflow-iris project and run packaged project
Given I have prepared a config file
And I have run a non-interactive kedro new with the starter astro-airflow-iris
When I execute the kedro command "package"
Then I should get a successful exit code
When I install the project's python package
And I execute the installed project
Then I should get a successful exit code

Scenario: Package spaceflights-pandas project and run packaged project
Given I have prepared a config file
And I have run a non-interactive kedro new with the starter spaceflights-pandas
When I execute the kedro command "package"
Then I should get a successful exit code
When I install the project's python package
And I execute the installed project
Then I should get a successful exit code

Scenario: Package spaceflights-pandas-viz project and run packaged project
Given I have prepared a config file
And I have run a non-interactive kedro new with the starter spaceflights-pandas-viz
When I execute the kedro command "package"
Then I should get a successful exit code
When I install the project's python package
And I execute the installed project
Then I should get a successful exit code

Scenario: Package spaceflights-pyspark project and run packaged project
Given I have prepared a config file
And I have run a non-interactive kedro new with the starter spaceflights-pyspark
When I execute the kedro command "package"
Then I should get a successful exit code
When I install the project's python package
And I execute the installed project
Then I should get a successful exit code

Scenario: Package spaceflights-pyspark-viz project and run packaged project
Given I have prepared a config file
And I have run a non-interactive kedro new with the starter spaceflights-pyspark-viz
When I execute the kedro command "package"
Then I should get a successful exit code
When I install the project's python package
And I execute the installed project
Then I should get a successful exit code
26 changes: 26 additions & 0 deletions features/steps/run_steps.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,3 +169,29 @@ def check_status_code(context):
assert False, "Expected exit code {}" " but got {}".format(
OK_EXIT_CODE, context.result.returncode
)


@when('I execute the kedro command "{command}"')
def exec_kedro_target(context, command):
"""Execute Kedro target."""
split_command = command.split()
cmd = [context.kedro, *split_command]
context.result = subprocess.run(cmd, env=context.env, cwd=str(context.root_project_dir))


@when("I install the project's python package")
def install_project_package_via_pip(context):
"""Install a python package using pip."""
dist_dir = context.root_project_dir / "dist"
(whl_file,) = dist_dir.glob("*.whl")
subprocess.run([context.pip, "install", str(whl_file)], env=context.env)


@when("I execute the installed project")
def exec_project(context):
"""Execute installed Kedro project target."""
context.result = subprocess.run(
[context.python, "-m", context.project_name.replace("-", "_")],
cwd=context.root_project_dir,
env=context.env
)
Loading