[Github actions]Fix integration test disk space is full (apache#6098)

### Motivation At present, there are about 28 GB of space available on the running machine of Github action, and there is less than 10GB of disk space left after packing with the `mvn install -DskipTests -Pdocker` command. therefore, the bookie is very easy to exit due to insufficient disk in the integration test. therefore, fix this problem and pack only the needed images. The current integration tests `PulsarFunctionsTest` are all in one class, which is very inconvenient to manage. For example, after the relevant tests of debezium are run, we can clean up the unused images of debezium, so adding groups to the integration test cases, so that we can control the tests more conveniently in the future. Now we have about 20G of space for testing. ``` df -h Filesystem Size Used Avail Use% Mounted on udev 3.4G 0 3.4G 0% /dev tmpfs 695M 688K 694M 1% /run /dev/sda1 84G 64G 20G 77% / tmpfs 3.4G 8.0K 3.4G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.4G 0 3.4G 0% /sys/fs/cgroup /dev/sda15 105M 3.6M 101M 4% /boot/efi /dev/sdb1 14G 41M 13G 1% /mnt ``` ``` docker images REPOSITORY TAG IMAGE ID CREATED SIZE apachepulsar/pulsar-test-latest-version 2.6.0-SNAPSHOT c599fe91d80e 2 seconds ago 2.82GB apachepulsar/pulsar-test-latest-version latest c599fe91d80e 2 seconds ago 2.82GB apachepulsar/pulsar-all latest 77d849fb13b7 3 days ago 2.76GB node 10 ea119cebc1c3 3 weeks ago 908MB node 12 6b5991bf650f 3 weeks ago 913MB buildpack-deps stretch 2adbcf3b6aff 3 weeks ago 835MB debian 9 f6c68e2ad82a 3 weeks ago 101MB debian 8 5d7d9c6338e8 3 weeks ago 129MB node 10-alpine d32bf7fc7d5f 3 weeks ago 80.5MB node 12-alpine 1cbcaddb8074 3 weeks ago 85.2MB alpine 3.10 965ea09ff2eb 3 months ago 5.55MB jekyll/builder latest 4605d94a6b36 3 months ago 564MB alpine 3.9 055936d39205 8 months ago 5.53MB alpine 3.7 6d1ef012b567 10 months ago 4.21MB alpine 3.8 dac705114996 10 months ago 4.41MB mcr.microsoft.com/azure-pipelines/node8-typescript latest 9a948d360778 15 months ago 595MB ``` ### Modifications * Add groups for integration tests * Delete no used Docker image * Pack images as needed ### Verifying this change Now the integration test can pass normally, please refer to https://github.com/AmateurEvents/pulsar/pull/5/checks?check_run_id=398472038
jiazhai · Jan 20, 2020 · 055f3bb · 055f3bb
1 parent 08337fa
commit 055f3bb
Show file tree

Hide file tree

Showing 4 changed files with 140 additions and 38 deletions.
diff --git a/.github/workflows/ci-integration-backwards-compatibility.yaml b/.github/workflows/ci-integration-backwards-compatibility.yaml
@@ -41,11 +41,20 @@ jobs:
       - name: checkout
         uses: actions/checkout@v1
 
+      - name: run install by skip tests
+        run: mvn clean install -DskipTests
+
       - name: build artifacts and docker image
-        run: mvn -B install -Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Pdocker -DskipTests
+        run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker -DskipTests
 
-      - name: clean up
+      - name: clean docker container
         run: docker system prune -f
 
+      - name: remove docker node image
+        run: docker rmi -f node:10 && docker rmi -f node:12 && docker rmi -f buildpack-deps:stretch
+
+      - name: remove docker builder and microsoft image
+        run: docker rmi -f jekyll/builder:latest && docker rmi -f mcr.microsoft.com/azure-pipelines/node8-typescript:latest
+
       - name: run integration tests
         run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-backwards-compatibility.xml -DintegrationTests -DredirectTestOutputToFile=false
diff --git a/.github/workflows/ci-integration-process.yaml b/.github/workflows/ci-integration-process.yaml
@@ -41,8 +41,26 @@ jobs:
       - name: checkout
         uses: actions/checkout@v1
 
+      - name: run install by skip tests
+        run: mvn clean install -DskipTests
+
       - name: build artifacts and docker image
-        run: mvn -B install -Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Pdocker -DskipTests
+        run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker -DskipTests
+
+      - name: clean docker container
+        run: docker system prune -f
+
+      - name: remove docker node image
+        run: docker rmi -f node:10 && docker rmi -f node:12 && docker rmi -f buildpack-deps:stretch
+
+      - name: remove docker builder and microsoft image
+        run: docker rmi -f jekyll/builder:latest && docker rmi -f mcr.microsoft.com/azure-pipelines/node8-typescript:latest
+
+      - name: run integration function
+        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-process.xml -DintegrationTests -DredirectTestOutputToFile=false -Dgroups=function
+
+      - name: run integration source
+        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-process.xml -DintegrationTests -DredirectTestOutputToFile=false -Dgroups=source
 
-      - name: run integration tests
-        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-process.xml -DintegrationTests -DredirectTestOutputToFile=false
+      - name: run integraion sink
+        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-process.xml -DintegrationTests -DredirectTestOutputToFile=false -Dgroups=sink
diff --git a/.github/workflows/ci-integration-thread.yaml b/.github/workflows/ci-integration-thread.yaml
@@ -41,8 +41,26 @@ jobs:
       - name: checkout
         uses: actions/checkout@v1
 
+      - name: run install by skip tests
+        run: mvn clean install -DskipTests
+
       - name: build artifacts and docker image
-        run: mvn -B install -Dorg.slf4j.simpleLogger.defaultLogLevel=ERROR -Pdocker -DskipTests
+        run: mvn -B -f tests/docker-images/pom.xml install -am -Pdocker -DskipTests
+
+      - name: clean docker container
+        run: docker system prune -f
+
+      - name: remove docker node image
+        run: docker rmi -f node:10 && docker rmi -f node:12 && docker rmi -f buildpack-deps:stretch
+
+      - name: remove docker builder and microsoft image
+        run: docker rmi -f jekyll/builder:latest && docker rmi -f mcr.microsoft.com/azure-pipelines/node8-typescript:latest
+
+      - name: run integration function
+        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-thread.xml -DintegrationTests -DredirectTestOutputToFile=false -Dgroups=function
+
+      - name: run integration source
+        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-thread.xml -DintegrationTests -DredirectTestOutputToFile=false -Dgroups=source
 
-      - name: run integration tests
-        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-thread.xml -DintegrationTests -DredirectTestOutputToFile=false
+      - name: run integraion sink
+        run: mvn -B -f tests/pom.xml test -DintegrationTestSuiteFile=pulsar-thread.xml -DintegrationTests -DredirectTestOutputToFile=false -Dgroups=sink