Skip to content

Commit

Permalink
Merge pull request #925 from snyk/feat/support-setup-py-docker
Browse files Browse the repository at this point in the history
feat: Support setup.py & Pipfile dependencies in the python docker images
  • Loading branch information
lili2311 authored Jan 28, 2020
2 parents 96de91d + 75f9a7d commit f02cc8a
Show file tree
Hide file tree
Showing 3 changed files with 100 additions and 26 deletions.
48 changes: 32 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,19 +60,22 @@ The package argument is optional. If no package is given, Snyk will run the comm
- **Alert** `snyk monitor` records the state of dependencies and any vulnerabilities on snyk.io so you can be alerted when new vulnerabilities or updates/patches are disclosed that affect your repositories.
- **Prevent** new vulnerable dependencies from being added to your project by running `snyk test` as part of your CI to fail tests when vulnerable Node.js or Ruby dependencies are added.

## Docker
## Snyk CLI Docker images

[See all snyk/snyk-cli images](https://hub.docker.com/r/snyk/snyk-cli)

Snyk is also provided as a set of Docker images that carry the runtime environment of each package manager. For example, the npm image will carry all of the needed setup to run `npm install` on the currently running container. Currently there are images for npm, Ruby, Maven, Gradle and SBT.

The images can perform `snyk test` by default on the specified project which is mounted to the container as a read/write volume, and `snyk monitor` if the `MONITOR` environment variable is set when running the docker container. If you want an HTML report for `test` command, make sure `--json` parameter is provided. `monitor` command appends it automatically. An HTML file called `snyk_report.html` and a CSS file called `snyk_report.css` will be generated. The image also writes a file called `snyk-res.json` for internal use and `snyk-error.log` for errors that we can look at if something goes wrong.
The images can perform `snyk test` by default on the specified project which is mounted to the container as a read/write volume, and `snyk monitor` if the `MONITOR` environment variable is set when running the docker container. If you want an HTML report for `test` command (`--json` is appended automatically). An HTML file called `snyk_report.html` and a CSS file called `snyk_report.css` will be generated. The image also writes a file called `snyk-res.json` for internal use and `snyk-error.log` for errors that we can look at if something goes wrong.

The following environment variables can be used when running the container on docker:

- `SNYK_TOKEN` - Snyk API token, obtained from [https://app.snyk.io/account](https://app.snyk.io/account).
- `USER_ID` - [OPTIONAL] Current user ID on the host machine. If not provided will take the user ID of the currently running user inside the container. This is used for CI builds such as Jenkins where we are running with a non-privileged user and want to allow the user to access the mounted project folder.
- `MONITOR` - [OPTIONAL] If set, tells the image that we want to run `snyk monitor` after running `snyk test`.
- `MONITOR` - [OPTIONAL] If set, will generate an html report via `snyk-to-html` and runs `snyk monitor` after running `snyk test`.
- `PROJECT_FOLDER` - [OPTIONAL] If set, this will cd to the directory inside the mounted project dir to run snyk inside it.
- `ENV_FLAGS` - [OPTIONAL] additional environment parameters to pass to `snyk test` when running the container.
- `TARGET_FILE` - [OPTIONAL] additional environment parameters to pass to `snyk test` & `snyk monitor` equal to `--file` option in the cli.

Docker images are tagged according to the package manager runtime they include, the package manager version and snyk version.
The general format of tags is [snyk-version]-[package-manager]-[package-manager-version] or just [package-manager]-[package-manager-version] if we want to use the latest version of snyk. Please see available tags to see the available options.
Expand All @@ -85,7 +88,9 @@ Please see the following examples on how to run Snyk inside docker:

### Node.js (npm)

We will need to mount the project root folder when running the image so that Snyk can access the code within the container. The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds. Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for npm:
The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds.

Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for npm:

```
docker run -it
Expand All @@ -98,7 +103,9 @@ docker run -it

### RubyGems

We will need to mount the project root folder when running the image so that Snyk can access the code within the container. The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds. Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for RubyGems:
The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds.

Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for RubyGems:

```
docker run -it
Expand All @@ -111,7 +118,10 @@ docker run -it

### Maven 3.5.4

We will need to mount the project root folder when running the image so that Snyk can access the code within the container and mount the local .m2 and .ivy2 folders. The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds. Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for Maven:
The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds.
You may also need to mount the local `.m2` and `.ivy2` folders.

Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for Maven:

```
docker run -it
Expand All @@ -126,10 +136,12 @@ docker run -it

### SBT 0.13.16 / SBT 1.0.4

We will need to mount the project root folder when running the image so that Snyk can access the code within the container and mount the local .m2 and .ivy2 folders. The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds. Here are examples of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for SBT:
The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds.
You may also need to mount the local `.m2` and `.ivy2` folders.

NOTE: the `dependency-tree` module is required for `snyk` to process Scala projects. Use [version 0.8.2](https://github.com/jrudolph/sbt-dependency-graph/tree/v0.8.2) for SBT 0.13.16 and [version 0.9.0](https://github.com/jrudolph/sbt-dependency-graph/tree/v0.9.0) for version SBT 1.0.4.
Here are examples of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for SBT:

*Note*: the `dependency-tree` or `sbt-dependency-graph` or `sbt-coursier` (included by default in latest sbt versions) module is required for `snyk` to process Scala projects.
```
docker run -it
-e "SNYK_TOKEN=<TOKEN>"
Expand All @@ -154,16 +166,18 @@ docker run -it

### Gradle 2.8 / Gradle 4.4 / Gradle 5.4

We will need to mount the project root folder when running the image so that Snyk can access the code within the container and mount the local .m2 and .ivy2 folders. The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds. Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for Gradle:
The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds.
You may also need to mount the local `.gradle`.

Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for Gradle:

```
docker run -it
-e "SNYK_TOKEN=<TOKEN>"
-e "USER_ID=1234"
-e "MONITOR=true"
-v "<PROJECT_DIRECTORY>:/project"
-v "/home/user/.m2:/home/node/.m2"
-v "/home/user/.ivy2:/home/node/.ivy2"
-v "/home/user/.gradle:/home/node/.gradle"
snyk/snyk-cli:gradle-2.8 test --org=my-org-name
```

Expand All @@ -173,8 +187,7 @@ docker run -it
-e "USER_ID=1234"
-e "MONITOR=true"
-v "<PROJECT_DIRECTORY>:/project"
-v "/home/user/.m2:/home/node/.m2"
-v "/home/user/.ivy2:/home/node/.ivy2"
-v "/home/user/.gradle:/home/node/.gradle"
snyk/snyk-cli:gradle-4.4 test --org=my-org-name
```

Expand All @@ -184,14 +197,17 @@ docker run -it
-e "USER_ID=1234"
-e "MONITOR=true"
-v "<PROJECT_DIRECTORY>:/project"
-v "/home/user/.m2:/home/node/.m2"
-v "/home/user/.ivy2:/home/node/.ivy2"
-v "/home/user/.gradle:/home/node/.gradle"
snyk/snyk-cli:gradle-5.4 test --org=my-org-name
```

### Docker

We will need to mount the project root folder when running the image so that Snyk can access the code within the container and Docker socket so that Snyk can access Docker daemon. The host project folder will be mounted to `/project` on the container and will be used to read the Docker file (with --file). Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for Docker:
The host project folder will be mounted to `/project` on the container and will be used to read the dependencies file and write results for CI builds.

The image being tested is expected to be available locally.

Here's an example of running `snyk test` and `snyk monitor` in the image (with the latest version of Snyk) for Docker:

```
docker run -it
Expand Down
15 changes: 7 additions & 8 deletions docker/docker-entrypoint.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,13 @@ fi

useradd -o -m -u "${USER_ID}" -d /home/node docker-user 2>/dev/null

runCmdAsDockerUser () {
runCmdAsDockerUser() {
su docker-user -m -c "$1"

return $?
}

exitWithMsg () {
exitWithMsg() {
echo "Failed to run the process ..."

if [ -f "$1" ]; then
Expand All @@ -46,7 +46,7 @@ exitWithMsg () {
## README.md for more info.
##

TEST_SETTINGS="";
TEST_SETTINGS=""
PROJECT_SUBDIR=""

if [ -n "${TARGET_FILE}" ]; then
Expand Down Expand Up @@ -77,11 +77,11 @@ if [ -n "${ENV_FLAGS}" ]; then
ADDITIONAL_ENV="-- ${ENV_FLAGS}"
fi

cd "${PROJECT_PATH}/${PROJECT_FOLDER}/${PROJECT_SUBDIR}" || \
exitWithMsg "Can't cd to ${PROJECT_PATH}/${PROJECT_FOLDER}/${PROJECT_SUBDIR}" 1
cd "${PROJECT_PATH}/${PROJECT_FOLDER}/${PROJECT_SUBDIR}" ||
exitWithMsg "Can't cd to ${PROJECT_PATH}/${PROJECT_FOLDER}/${PROJECT_SUBDIR}" 1

runCmdAsDockerUser "PATH=${PATH} snyk ${SNYK_COMMAND} ${SNYK_PARAMS} \
${ADDITIONAL_ENV} > \"${OUTPUT_FILE}\" 2>\"${ERROR_FILE}\""
${ADDITIONAL_ENV} --json > \"${OUTPUT_FILE}\" 2>\"${ERROR_FILE}\""

RC=$?

Expand All @@ -101,6 +101,7 @@ fi
runCmdAsDockerUser "touch \"${PROJECT_PATH}/${PROJECT_FOLDER}/${HTML_FILE}\""

if [ -n "$MONITOR" ]; then
echo "Monitoring & generating report ..."
runCmdAsDockerUser "PATH=$PATH snyk monitor --json ${SNYK_PARAMS} ${ADDITIONAL_ENV} > ${MONITOR_OUTPUT_FILE} 2>$ERROR_FILE"
runCmdAsDockerUser "cat ${MONITOR_OUTPUT_FILE} | jq -r \".uri\" | awk '{print \"<center><a target=\\\"_blank\\\" href=\\\"\" \$0 \"\\\">View On Snyk.io</a></center>\"}' > \"${PROJECT_PATH}/${PROJECT_FOLDER}/${HTML_FILE}\" 2>>\"${ERROR_FILE}\""
fi
Expand All @@ -114,8 +115,6 @@ sed 's/<\/head>/ <link rel=\"stylesheet\" href=\"snyk_report.css\"><\/head>/' \

runCmdAsDockerUser "cat /home/node/snyk_report.css > \
\"${PROJECT_PATH}/${PROJECT_FOLDER}/snyk_report.css\""
# fi
#

if [ $RC -ne "0" ]; then
exitWithMsg "${OUTPUT_FILE}" "$RC"
Expand Down
63 changes: 61 additions & 2 deletions docker/docker-python-entrypoint.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,65 @@
#!/bin/bash

virtualenv -p python snyk
source snyk/bin/activate
pip install -U -r "${PROJECT_PATH}/requirements.txt"

exitWithMsg() {
echo "Failed to run the process ..."

if [ -f "$1" ]; then
cat "$1"
else
echo "$1"
fi

exit "$2"
}

installRequirementsTxtDeps() {
echo "Installing dependencies from requirements file"
pip install -U -r "$1"
}

installPipfileDeps() {
pushd "${PROJECT_PATH}/"
echo "Found Pipfile"
pipenv lock
pipenv install --system
popd
}

PROJECT_SUBDIR=""
echo "Project path = ${PROJECT_PATH}"
if [ -n "${TARGET_FILE}" ]; then
if [ ! -f "${PROJECT_PATH}/${PROJECT_FOLDER}/${TARGET_FILE}" ]; then
exitWithMsg "\"${PROJECT_PATH}/${PROJECT_FOLDER}/${TARGET_FILE}\" does not exist" 1
fi

PROJECT_SUBDIR=$(dirname "${TARGET_FILE}")
MANIFEST_NAME=$(basename "${TARGET_FILE}")
TEST_SETTINGS="--file=${MANIFEST_NAME} "

echo "Target file = ${TARGET_FILE}"

case $MANIFEST_NAME in
*req*.txt)
echo "Installing dependencies from requirements file"
installRequirementsTxtDeps "${PROJECT_PATH}/$MANIFEST_NAME"
;;
*setup.py)
echo "Installing dependencies from setup.py"
pip install -U -e "${PROJECT_PATH}"
;;
*)
exitWithMsg "\"${PROJECT_PATH}/${TARGET_FILE}\" is not supported" 1
;;
esac
fi

if [ -f "${PROJECT_PATH}/requirements.txt" ]; then
echo "Found requirement.txt"
installRequirementsTxtDeps "${PROJECT_PATH}/requirements.txt"
elif [ -f "${PROJECT_PATH}/Pipfile" ]; then
installPipfileDeps
fi

bash docker-entrypoint.sh "$@"

0 comments on commit f02cc8a

Please sign in to comment.