Squashed commit of the following:

commit e1fd9d7 Merge: 370665e 2fad420 Author: Wei Chen <[email protected]> Date: Wed May 17 14:19:11 2023 -0700 Merge branch 'eng-2699-m1-refactor-wf-detail-pages-to-use-new' into eng-2699-m1-refactor-wf-detail-pages-to-use-new-2 commit 2fad420 Merge: b4485f4 676e81a Author: Wei Chen <[email protected]> Date: Wed May 17 14:06:26 2023 -0700 Merge branch 'eng-2699-m1-refactor-wf-detail-pages-to-use-new' of https://github.com/aqueducthq/aqueduct into eng-2699-m1-refactor-wf-detail-pages-to-use-new commit b4485f4 Author: Wei Chen <[email protected]> Date: Mon Apr 24 15:43:51 2023 -0700 rebase commit d3a2669 Merge: e60ac09 18011cb Author: Wei Chen <[email protected]> Date: Wed May 17 14:04:32 2023 -0700 Merge branch 'eng-2712-m1-migrate-wf-edit-trigger-endpoints-to' into eng-2943-investigate-result-content-route-tests commit 18011cb Merge: ec515cf 8cb356c Author: Wei Chen <[email protected]> Date: Wed May 17 14:02:47 2023 -0700 Merge branch 'eng-2847-m1-implement-wfiddags-endpoint-with' into eng-2712-m1-migrate-wf-edit-trigger-endpoints-to commit 8cb356c Merge: ec94379 b3946a4 Author: Wei Chen <[email protected]> Date: Wed May 17 14:02:23 2023 -0700 Merge branch 'main' of https://github.com/aqueducthq/aqueduct into eng-2847-m1-implement-wfiddags-endpoint-with commit 676e81a Author: Wei Chen <[email protected]> Date: Mon Apr 24 15:43:51 2023 -0700 rebase commit ec94379 Author: Wei Chen <[email protected]> Date: Wed May 17 13:28:38 2023 -0700 comments commit b3946a4 Author: kenxu95 <[email protected]> Date: Wed May 17 10:19:11 2023 -0700 Move any overflow rows on the details header fields into an additional column (#1322) commit e60ac09 Author: Wei Chen <[email protected]> Date: Tue May 16 20:04:49 2023 -0700 fix commit ec515cf Merge: 1d0482a f81c065 Author: Wei Chen <[email protected]> Date: Tue May 16 17:37:04 2023 -0700 Merge branch 'eng-2847-m1-implement-wfiddags-endpoint-with' into eng-2712-m1-migrate-wf-edit-trigger-endpoints-to commit f81c065 Author: Wei Chen <[email protected]> Date: Tue May 16 17:34:45 2023 -0700 lint commit 1d0482a Author: Wei Chen <[email protected]> Date: Tue May 16 17:29:44 2023 -0700 works commit 5ad5130 Author: Andre Giron <[email protected]> Date: Tue May 16 16:34:41 2023 -0700 ENG-2979: Fixes bigquery and gcs dialogs that were crashing after file uploads. (#1326) Co-authored-by: Andre Giron <[email protected]> commit 03aa76e Author: Andre Giron <[email protected]> Date: Tue May 16 16:33:35 2023 -0700 ENG-2979 S3 Dialog Validation Fixes (#1328) Co-authored-by: Andre Giron <[email protected]> commit f63c5f3 Author: kenxu95 <[email protected]> Date: Tue May 16 16:24:14 2023 -0700 Fix Snowflake save missing schema bug (#1331) commit a458ce6 Author: Wei Chen <[email protected]> Date: Tue May 16 15:27:52 2023 -0700 works commit f529c5f Author: Saurav Chhatrapati <[email protected]> Date: Tue May 16 13:36:12 2023 -0400 Fixes cloudpickle serialization for Python 3.7 (#1324) commit c335fd5 Author: Saurav Chhatrapati <[email protected]> Date: Tue May 16 12:15:48 2023 -0400 Adds error checking for whether big query dataset exists (#1319) commit d3521a4 Author: kenxu95 <[email protected]> Date: Mon May 15 20:58:16 2023 -0700 Add ability to parameterize the save operator (#1320) commit 942dec1 Author: kenxu95 <[email protected]> Date: Mon May 15 15:36:21 2023 -0700 [UI] Merge the Conda resource into the Aqueduct Server (#1311) commit ccf4e17 Author: Wei Chen <[email protected]> Date: Mon May 15 14:57:19 2023 -0700 [2/n] Adds gh actions to publish test pypi packages (#1262) commit fb67044 Author: kenxu95 <[email protected]> Date: Mon May 15 12:04:48 2023 -0700 Fix bug where duplicate fields showing in resource details dropdown (#1321) commit cfeeb2f Author: kenxu95 <[email protected]> Date: Mon May 15 11:45:31 2023 -0700 Fix generated docstrings when viewing resource methods in Jupyter (#1315) commit b544252 Author: eunice-chan <[email protected]> Date: Fri May 12 21:26:55 2023 -0500 Eng 2884 test node routes (#1299) Co-authored-by: Eunice Chan <[email protected]> Co-authored-by: Wei Chen <[email protected]> commit a8abb16 Author: Andre Giron <[email protected]> Date: Fri May 12 16:58:59 2023 -0700 Fixes broken storybook build (#1317) Co-authored-by: Andre Giron <[email protected]> commit d6ed5ca Author: kenxu95 <[email protected]> Date: Fri May 12 10:56:42 2023 -0700 Update the workflow "No Run" logo (#1312) commit c9fec70 Author: Chenggang Wu <[email protected]> Date: Fri May 12 09:30:54 2023 -0700 README typo fix (#1313) commit b686014 Author: Andre Giron <[email protected]> Date: Thu May 11 12:49:53 2023 -0700 ENG-2767 react hook form submission (#1232) Co-authored-by: Andre Giron <[email protected]> Co-authored-by: Chenggang Wu <[email protected]> Co-authored-by: kenxu95 <[email protected]> commit 86269ec Author: kenxu95 <[email protected]> Date: Thu May 11 12:09:42 2023 -0700 Add appropriate workflow count to our connected notification cards and details page (#1302) commit 7c75ad6 Author: kenxu95 <[email protected]> Date: Thu May 11 10:45:52 2023 -0700 Update the artifact storage card presentation to be consistent with the other data integrations (#1297) commit 4f9c227 Author: Andre Giron <[email protected]> Date: Wed May 10 16:55:23 2023 -0700 ENG_2960: Fix broken documentation link for Aqueduct demo resource. (#1306) Co-authored-by: Andre Giron <[email protected]> commit 6341917 Author: Andre Giron <[email protected]> Date: Wed May 10 16:28:23 2023 -0700 Release version 0.3.2 (#1307) Co-authored-by: Vikram Sreekanti <[email protected]> Co-authored-by: Andre Giron <[email protected]>
RunLLM · May 17, 2023 · 4196cf7 · 4196cf7
1 parent 80babf5
commit 4196cf7
Show file tree

Hide file tree

Showing 196 changed files with 26,312 additions and 39,787 deletions.
diff --git a/.github/actions/setup-server/action.yml b/.github/actions/setup-server/action.yml
@@ -1,5 +1,9 @@
 name: Setup Server
 description: "Sets up a server and installs the appropriate environment."
+inputs:
+  python-pkg-version:
+    description: "Specifies test Pypi package version number if not using the latest release."
+    required: false
 
 runs:
   using: composite
@@ -56,6 +60,12 @@ runs:
       shell: bash
       run: python3 scripts/install_local.py -g -s -e
 
+    - name: Update Pypi package version
+      shell: bash
+      run: |
+        if [ "${{ inputs.python-pkg-version }}" ]; then echo -e "\nversionTag: ${{ inputs.python-pkg-version }}" >> $HOME/.aqueduct/server/config/config.yml; fi
+        cat $HOME/.aqueduct/server/config/config.yml
+
     - name: Start the server again
       shell: bash
       run: (aqueduct start --disable-usage-stats --verbose > $SERVER_LOGS_FILE 2>&1 &)

diff --git a/.github/workflows/periodic-integration-tests.yml b/.github/workflows/periodic-integration-tests.yml
@@ -6,15 +6,80 @@ on:
   workflow_dispatch:
 
 jobs:
+  publish-pypi:
+    # similar to release scripts, but publish to test pypi:
+    # rm -rf dist && rm -rf build
+    # python3 -m build && twine check dist/*
+    # twine upload --repository testpypi dist/*
+    # pip3 install -i https://test.pypi.org/simple/ aqueduct-ml
+    name: Publish Test Pypi Packages
+    runs-on: [ubuntu-latest]
+    timeout-minutes: 20
+    outputs:
+      version: ${{ steps.inject_version.outputs.version }}
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: install missing dependencies
+        run: pip install build twine
+
+      - name: inject dev version
+        id: inject_version
+        run: |
+          export VERSION=0.1.dev$(date +%s)
+          rm sdk/version
+          echo $VERSION >> sdk/version
+          rm src/python/version
+          echo $VERSION >> src/python/version
+          echo version=$VERSION >> $GITHUB_OUTPUT
+
+      - name: build sdk
+        working-directory: sdk
+        run: |
+          rm -rf dist
+          rm -rf build
+          python3 -m build
+          twine check dist/*
+
+      - name: publish sdk
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+          packages-dir: sdk/dist
+          password: ${{ secrets.PYPI_API_TOKEN_SDK }}
+
+      - name: build executor
+        working-directory: src/python
+        run: |
+          rm -rf dist
+          rm -rf build
+          python3 -m build
+          twine check dist/*
+
+      - name: publish executor
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+          packages-dir: src/python/dist
+          password: ${{ secrets.PYPI_API_TOKEN_EXECUTOR }}
+
   run-k8s-tests:
     runs-on: ubuntu-latest
     timeout-minutes: 360
     name: SDK Integration Tests against K8s Compute
+    needs: publish-pypi
     steps:
       - uses: actions/checkout@v2
 
       - uses: ./.github/actions/setup-server
         timeout-minutes: 7
+        with:
+          python-pkg-version: ${{ needs.publish-pypi.outputs.version }}
 
       # TODO(ENG-2537): Use our separate GH actions credentials.
       - uses: ./.github/actions/fetch-test-config
@@ -101,12 +166,15 @@ jobs:
   run-tests-conda:
     runs-on: ubuntu-latest-4-cores
     timeout-minutes: 50
+    needs: publish-pypi
     name: All Integration Tests with Conda
     steps:
       - uses: actions/checkout@v2
 
       - uses: ./.github/actions/setup-server
         timeout-minutes: 7
+        with:
+          python-pkg-version: ${{ needs.publish-pypi.outputs.version }}
 
       - uses: conda-incubator/setup-miniconda@v2
         with:
@@ -136,7 +204,7 @@ jobs:
         run: echo "API_KEY=$(aqueduct apikey)" >> $GITHUB_ENV
 
       - name: Run the No-Concurrency Integration Tests
-        timeout-minutes: 10
+        timeout-minutes: 15
         working-directory: integration_tests/no_concurrency
         env:
           SERVER_ADDRESS: localhost:8080

diff --git a/.github/workflows/spark-periodic-tests.yml b/.github/workflows/spark-periodic-tests.yml
@@ -6,15 +6,80 @@ on:
   workflow_dispatch:
 
 jobs:
+  publish-pypi:
+    # similar to release scripts, but publish to test pypi:
+    # rm -rf dist && rm -rf build
+    # python3 -m build && twine check dist/*
+    # twine upload --repository testpypi dist/*
+    # pip3 install -i https://test.pypi.org/simple/ aqueduct-ml
+    name: Publish Test Pypi Packages
+    runs-on: [ubuntu-latest]
+    timeout-minutes: 20
+    outputs:
+      version: ${{ steps.inject_version.outputs.version }}
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.9"
+      - name: install missing dependencies
+        run: pip install build twine
+
+      - name: inject dev version
+        id: inject_version
+        run: |
+          export VERSION=0.1.dev$(date +%s)
+          rm sdk/version
+          echo $VERSION >> sdk/version
+          rm src/python/version
+          echo $VERSION >> src/python/version
+          echo version=$VERSION >> $GITHUB_OUTPUT
+
+      - name: build sdk
+        working-directory: sdk
+        run: |
+          rm -rf dist
+          rm -rf build
+          python3 -m build
+          twine check dist/*
+
+      - name: publish sdk
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+          packages-dir: sdk/dist
+          password: ${{ secrets.PYPI_API_TOKEN_SDK }}
+
+      - name: build executor
+        working-directory: src/python
+        run: |
+          rm -rf dist
+          rm -rf build
+          python3 -m build
+          twine check dist/*
+
+      - name: publish executor
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          repository-url: https://test.pypi.org/legacy/
+          packages-dir: src/python/dist
+          password: ${{ secrets.PYPI_API_TOKEN_EXECUTOR }}
+
   run-databricks-tests:
     runs-on: ubuntu-latest
     timeout-minutes: 480
+    needs: publish-pypi
     name: SDK Integration Tests against Databricks Compute
     steps:
       - uses: actions/checkout@v2
 
       - uses: ./.github/actions/setup-server
-        timeout-minutes: 5
+        timeout-minutes: 7
+        with:
+          python-pkg-version: ${{ needs.publish-pypi.outputs.version }}
 
       # TODO(ENG-2537): Use our separate GH actions credentials.
       - uses: ./.github/actions/fetch-test-config
@@ -36,7 +101,7 @@ jobs:
         if: always()
         with:
           prefix: Databricks Compute
-      
+
       # Sets it as an environmental variable.
       - name: Get the Slack ID for the current oncall
         if: always()
@@ -56,16 +121,19 @@ jobs:
           mention_users: ${{ env.ONCALL_SLACK_MEMBER_ID }}
         env:
           SLACK_WEBHOOK_URL: ${{ secrets.ACTION_MONITORING_SLACK }}
-  
+
   run-spark-tests:
     runs-on: ubuntu-latest
     timeout-minutes: 480
+    needs: publish-pypi
     name: SDK Integration Tests against Spark EMR
     steps:
       - uses: actions/checkout@v2
 
       - uses: ./.github/actions/setup-server
-        timeout-minutes: 5
+        timeout-minutes: 7
+        with:
+          python-pkg-version: ${{ needs.publish-pypi.outputs.version }}
 
       # TODO(ENG-2537): Use our separate GH actions credentials.
       - uses: ./.github/actions/fetch-test-config
@@ -106,26 +174,26 @@ jobs:
         run: |
           livy_server_url=$(aws emr describe-cluster --cluster-id ${{ env.cluster_id }} --query 'Cluster.MasterPublicDnsName' --output text):8998
           echo "livy_server_url=http://$livy_server_url" >> $GITHUB_ENV
-      
+
       - name: Update the test-credentials file with the appropriate livy server url
         working-directory: integration_tests/sdk
         run: sed -i "s#\(livy_server_url:\s*\).*#\1$(echo "${{ env.livy_server_url }}")#" test-credentials.yml
 
       - name: Run the SDK Integration Tests
         working-directory: integration_tests/sdk
         run: pytest aqueduct_tests/ -rP -vv -n 2
-      
+
       - name: Tear Down EMR cluster
         id: tear-down-emr-cluster
         if: always()
         run: |
           aws emr terminate-clusters --cluster-ids ${{ env.cluster_id }}
-      
+
       - uses: ./.github/actions/upload-artifacts
         if: always()
         with:
           prefix: Spark Compute
-      
+
       # Sets it as an environmental variable.
       - name: Get the Slack ID for the current oncall
         if: always()

diff --git a/.github/workflows/ui-build.yml b/.github/workflows/ui-build.yml
@@ -39,3 +39,7 @@ jobs:
       - name: Build OSS app
         working-directory: ./src/ui/app
         run: npm run build
+
+      - name: Build Storybook
+        working-directory: ./src/ui/common
+        run: npm run build-storybook
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,18 @@
 # Changelog
 
+## 0.3.2
+Released on May 10, 2023.
+
+### Enhancements
+* Improves error handling when the user attempts to take an action without
+    connecting to the Aqueduct server.
+* When connecting an S3 resource to Aqueduct, if the specified subdirectory
+    does not exist, it is now automatically created.
+
+### Bugfixes
+* Replaces outdated integration terminology on the UI.
+* Fixes bug where previously created resource names were not editable.
+
 ## 0.3.1
 Released on May 4, 2023.
 
@@ -39,6 +52,9 @@ Released on May 4, 2023.
 * Fixes bug where installing pre-requisites for using Aqueduct-managed
     Kubernetes clusters would fail on an M1 Mac with certain configurations.
 
+## 0.3.0
+Note: Due to a technical issue, v0.3.0 was immediately replaced by v0.3.1.
+
 ## 0.2.12
 Released on April 25, 2023.
 

diff --git a/README.md b/README.md
@@ -46,7 +46,7 @@ vicuna = aq.llm_op('vicuna_7b', engine='eks-us-east-2')
 features = vicuna(
     raw_logs,
     { 
-        prompt: 
+        "prompt": 
         "Turn this log entry into a CSV: {text}" 
     }
 )

diff --git a/integration_tests/backend/setup/flow_with_multiple_operators.py b/integration_tests/backend/setup/flow_with_multiple_operators.py
@@ -0,0 +1,43 @@
+###
+# Workflow that extracts a table, and simply passes it through
+# four operators sequentially that simply return the table.
+###
+import aqueduct
+
+
+def setup_flow_with_multiple_operators(
+    client: aqueduct.Client,
+    integration_name: str,
+    workflow_name: str = "",
+) -> str:
+    name = workflow_name if workflow_name else "Test: Multiple Operators"
+    n_runs = 1
+    integration = client.resource(name=integration_name)
+
+    @aqueduct.op
+    def op1(df):
+        return df
+
+    @aqueduct.op
+    def op2(df):
+        return df
+
+    @aqueduct.op
+    def op3(df):
+        return df
+
+    @aqueduct.op
+    def op4(df):
+        return df
+
+    reviews = integration.sql("SELECT * FROM hotel_reviews")
+    df1 = op1(reviews)
+    df2 = op2(df1)
+    df3 = op3(df2)
+    df4 = op4(df3)
+
+    flow = client.publish_flow(
+        artifacts=[df1, df2, df3, df4], name=name, description="Test description."
+    )
+
+    return flow.id(), n_runs