Skip to content

Commit f498872

Browse files
justinbastressjustinbastress
authored
Implements postgres zgrab2 module (zmap#30)
* remove unnecessary indirection on net.Conn * Ignore *.pyc * fix NPE on nil handshake * refactoring -- move status to status.go; add Open() methods for ScanTarget * cherry-pick .gitignore fix * pull in TLS fix * status.go comments * trim over-generalizations * use /usr/bin/env bash instead of absolute path * remove debug tcpwrap * add integration tests for postgres * hack for cleanup.sh to work on mingw -- use //var/lib instead of /var/lib * cleanup should actually stop the process though * comments / rearrange * Bump up timeout in postgres tests; only pass user if explicitly requested to do so * add schema stubs to new.sh * Integration test fixes -- use /usr/bin/env bash; log all validation failures * add postgres schemas * fill out zcrypto.client_hello schema * handle early get of TLSLog * postgres: return SCAN_SUCCESS on success * cleanup * fix new.sh * fix typo * postgres container cleanup * build.sh docs * standardize container/image names * add not to check for success * shift mysql's connection management to ScanTarget.Open(); wrap Read/Write methods returned by ScanTarget.Open() to enforce timeouts * catch schematically-valid but non-successful scans * postgres: clean up output format; more scanning * cleanup; better error handling; get detailed protocol version error * refactor modules * clean up dangling connections * split gigantic postgres.go * remove unused * ServerParams gets its own type * refactor integration tests: run zgrab2 in its own container, which is linked to the service containers, so that we don't need to keep track of unique ports on the host any more * rename entrypoint; remove duplicate postgres tests * comments for postgres schema * Use param expansion to check for env variable [minor] This is a *very* minor change to `docker-runner/docker-run.sh` checks to see if the environment variable required to run the script has been set to a non-empty string. If not, the script exits with a non-zero status code and displays a default message: ``` ❯ docker-runner/docker-run.sh docker-runner/docker-run.sh: line 7: CONTAINER_NAME: parameter null or not set ``` This was the behavior before, but just uses a one-liner declarative bash idiom. For further reading on parameter expansion, see https://stackoverflow.com/a/307735. @justinbastress can tell me if I did something wrong and broke the intent of the script :-) * Add integration_test targets to makefile; use makefile instead of directly calling go build everywhere; run postgres schema through PEP8 linter * use make in docker-runner entrypoint * add .integration_test_setup to .gitignore * more .gitignore items * Makefile updates: Windows support; add docker-runner target; better cleanup. * docker-runner Dockerfile: start from zgrab2_runner_base image * cleanup postgres setup * make travis use make * add .gitattributes, try to prevent it from overriding lfs with crlfs in shell scripts at least * fix folder name in Makefile * update go (one of our dependencies now works only with >= 1.9) * From travis: `I don't have any idea what to do with '1.9.0'.` * explicit clean make * fix dep order * fix build.sh location * popd * use make to ensure zgrab2_runner exists * Make docker-runner an order-dependency for integration-test-cleanup; don't do a cleanup after each integration test * use explicit tag name for zgrab2_runner * Add container-clean target to Makefile, to remove cyclic dependency on docker; use .id files to track docker images; add servce-base image; use Make to build / track images * use LF in Makefiles; update .gitignore; use zgrab_service_base image in ssh container; fix line endings (?) * remove overzealous cleanup * let setup continue even if some containers are already running * zgrab depends on *.go * docker-runner depends on zgrab2 binary * clean output before running integration tests
1 parent a5d8d0b commit f498872

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+1596
-342
lines changed

‎.gitattributes

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*.sh eol=LF
2+
Makefile eol=LF

‎.gitignore

+10
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,18 @@
22
.DS_Store
33

44
# zgrab
5+
zgrab2
56
cmd/zgrab2/zgrab2
7+
cmd/zgrab2/zgrab2.exe
68
zgrab-output/
79

810
# CI dependencies
911
jp
12+
jp.exe
13+
14+
# Compiled python modules
15+
schemas/*.pyc
16+
17+
# Marker files for make
18+
.integration-test-setup
19+
docker-runner/*.id

‎.travis.yml

+4-4
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
language: go
22
go:
3-
- 1.7.4
3+
- 1.9
44
services:
55
- docker
66
before_install:
@@ -9,13 +9,13 @@ before_install:
99
# JMESPath is used to do context-specific validation of results
1010
- go get github.com/jmespath/jp && go build github.com/jmespath/jp
1111
- pip install --user zschema
12-
- ./integration_tests/setup.sh
1312
- docker ps -a
1413
install:
15-
- pushd cmd/zgrab2 && go build && popd
14+
- make clean zgrab2
1615
script:
17-
- ./integration_tests/test.sh
16+
- make integration-test
1817
after_script:
18+
# Cleanup even if the integration tests fail
1919
- ./integration_tests/cleanup.sh
2020
notifications:
2121
email:

‎Makefile

+39-4
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,44 @@
1+
ifeq ($(OS),Windows_NT)
2+
EXECUTABLE_EXTENSION := .exe
3+
else
4+
EXECUTABLE_EXTENSION :=
5+
endif
6+
7+
GO_FILES = $(shell find . -type f -name '*.go')
8+
19
all: zgrab2
210

3-
.PHONY: clean zgrab2
11+
.PHONY: all clean integration-test integration-test-clean docker-runner container-clean
12+
13+
zgrab2: $(GO_FILES)
14+
cd cmd/zgrab2 && go build && cd ../..
15+
rm -f zgrab2
16+
ln -s cmd/zgrab2/zgrab2$(EXECUTABLE_EXTENSION) zgrab2
17+
18+
docker-runner: zgrab2
19+
make -C docker-runner
20+
21+
.integration-test-setup: | docker-runner
22+
./integration_tests/setup.sh
23+
touch .integration-test-setup
24+
25+
integration-test: docker-runner .integration-test-setup
26+
rm -rf zgrab-output
27+
./integration_tests/test.sh
28+
29+
integration-test-clean:
30+
rm -f .integration-test-setup
31+
rm -rf zgrab-output
32+
./integration_tests/cleanup.sh
33+
make -C docker-runner clean
434

5-
zgrab2:
6-
cd cmd/zgrab2 && go build -o zgrab2
35+
# This is the target for re-building from source in the container
36+
container-clean:
37+
rm -f zgrab2
38+
cd cmd/zgrab2 && go build && cd ../..
39+
ln -s cmd/zgrab2/zgrab2$(EXECUTABLE_EXTENSION) zgrab2
740

841
clean:
9-
go clean
42+
cd cmd/zgrab2 && go clean
43+
rm -f .integration-test-setup
44+
rm -f zgrab2

‎docker-runner/Dockerfile

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
FROM zgrab2_runner_base:latest
2+
3+
WORKDIR /go/src/github.com/zmap
4+
5+
# Grab the currently-active version of the source
6+
ADD . zgrab2
7+
8+
# This would instead grab it from the source repo
9+
# RUN go-wrapper download github.com/zmap/zgrab2
10+
11+
WORKDIR /go/src/github.com/zmap/zgrab2
12+
13+
RUN go get -v ./...
14+
RUN go get -v -t ./...
15+
16+
# This should already be executable, but just in case...
17+
RUN chmod a+x ./docker-runner/entrypoint.sh
18+
19+
# Build on the container
20+
RUN make container-clean
21+
22+
CMD []
23+
ENTRYPOINT ["/go/src/github.com/zmap/zgrab2/docker-runner/entrypoint.sh"]

‎docker-runner/Makefile

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
ifeq ($(OS),Windows_NT)
2+
EXECUTABLE_EXTENSION := .exe
3+
else
4+
EXECUTABLE_EXTENSION :=
5+
endif
6+
7+
all: docker-runner.id
8+
9+
.PHONY: clean clean-all
10+
11+
service-base-image.id:
12+
docker build -t zgrab2_service_base:latest -f service-base.Dockerfile -q . > service-base-image.id || rm -f service-base-image.id
13+
14+
runner-base-image.id:
15+
docker build -t zgrab2_runner_base:latest -f runner-base.Dockerfile -q . > runner-base-image.id || rm -f runner-base-image.id
16+
17+
docker-runner.id: ../cmd/zgrab2/zgrab2$(EXECUTABLE_EXTENSION) runner-base-image.id service-base-image.id
18+
docker build -t zgrab2_runner:latest -f Dockerfile -q .. > docker-runner.id || rm -f docker-runner.id
19+
20+
clean:
21+
if [ -f docker-runner.id ]; then docker rmi -f $$(cat docker-runner.id) && rm -f docker-runner.id; fi
22+
23+
clean-all: clean
24+
if [ -f service-base-image.id ]; then docker rmi -f $$(cat service-base-image.id) && rm -f service-base-image.id; fi
25+
if [ -f runner-base-image.id ]; then docker rmi -f $$(cat runner-base-image.id) && rm -f runner-base-image.id; fi

‎docker-runner/docker-run.sh

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
#!/usr/bin/env bash
2+
3+
# Runs the zgrab2_runner docker image (built with docker-runner/build-runner.sh)
4+
# Links the runner image to the targetted container with the hostname alias "target",
5+
# then scans target using the arguments to the script.
6+
7+
: "${CONTAINER_NAME:?}"
8+
9+
set -e
10+
docker run --rm --link $CONTAINER_NAME:target -e ZGRAB_TARGET=target zgrab2_runner $@

‎docker-runner/entrypoint.sh

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
#!/usr/bin/env bash
2+
3+
# This is the entrypoint for the zgrab2_runner container.
4+
# Runs the zgrab2 binary, passing along any arguments, with stdin containing the single line: the ZGRAB_TARGET.
5+
6+
set -e
7+
8+
cd /go/src/github.com/zmap/zgrab2
9+
10+
if ! [ -x $ZGRAB_REBUILD ]; then
11+
if ! [ -x $ZGRAB_BRANCH ]; then
12+
git checkout $ZGRAB_BRANCH
13+
fi
14+
git pull
15+
make
16+
fi
17+
18+
set -x
19+
echo $ZGRAB_TARGET | /go/src/github.com/zmap/zgrab2/cmd/zgrab2/zgrab2 $*

‎docker-runner/runner-base.Dockerfile

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
FROM golang:1.9
2+
# Base image that already has the pre-requisites downloaded.
3+
4+
WORKDIR /go/src/github.com/zmap
5+
6+
RUN go-wrapper download github.com/zmap/zgrab2
7+
8+
WORKDIR /go/src/github.com/zmap/zgrab2
9+
10+
RUN go get -v ./...
11+
RUN go get -v -t ./...

‎docker-runner/service-base.Dockerfile

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
FROM ubuntu:16.04
2+
3+
# Base Ubuntu container with the apt cache already updated.
4+
# Many containers will be able to use this as their base.
5+
6+
RUN apt-get update

‎integration_tests/cleanup.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
# Keep cleaning up, even if something fails
44
set +e

‎integration_tests/mysql/cleanup.sh

+5-8
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,15 @@
1-
#!/bin/bash
1+
#!/usr/bin/env bash
22

33
# Keep cleaning up even if something fails
44
set +e
55

6-
# Stop all MySQL containers, but first grab the logs from them
6+
# Stop all MySQL containers.
77

88
MYSQL_VERSIONS="5.5 5.6 5.7 8.0"
99

1010
for version in $MYSQL_VERSIONS; do
11-
CONTAINER_NAME="testmysql-$version"
12-
echo "BEGIN DOCKER LOGS FROM $CONTAINER_NAME [{("
13-
docker logs --tail all $CONTAINER_NAME
14-
echo ")}] END DOCKER LOGS FROM $CONTAINER_NAME"
15-
echo "Stopping $CONTAINER_NAME..."
11+
CONTAINER_NAME="zgrab_mysql-$version"
12+
echo "mysql/cleanup: Stopping $CONTAINER_NAME..."
1613
docker stop $CONTAINER_NAME
17-
echo "...stopped."
14+
echo "mysql/cleanup: ...stopped."
1815
done

‎integration_tests/mysql/setup.sh

+31-12
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,35 @@
11
#!/bin/bash -e
22

3-
# Start all of the MySQL docker containers, and wait for them start responding on port 3306
4-
5-
echo "Launching docker containers..."
63
# NOTE: the 5.5 and 5.6 versions do not have SSL enabled
7-
CONTAINER_NAME=testmysql-5.5 MYSQL_VERSION=5.5 MYSQL_PORT=13306 ./util/launch_mysql_container.sh
8-
CONTAINER_NAME=testmysql-5.6 MYSQL_VERSION=5.6 MYSQL_PORT=23306 ./util/launch_mysql_container.sh
9-
CONTAINER_NAME=testmysql-5.7 MYSQL_VERSION=5.7 MYSQL_PORT=33306 ./util/launch_mysql_container.sh
10-
CONTAINER_NAME=testmysql-8.0 MYSQL_VERSION=8.0 MYSQL_PORT=43306 ./util/launch_mysql_container.sh
4+
versions="5.5 5.6 5.7 8.0"
5+
6+
function launch() {
7+
VERSION=$1
8+
CONTAINER_NAME="zgrab_mysql-$VERSION"
9+
if docker ps --filter "name=$CONTAINER_NAME" | grep $CONTAINER_NAME; then
10+
echo "mysql/setup: Container $CONTAINER_NAME already running -- stopping..."
11+
docker stop $CONTAINER_NAME
12+
echo "...stopped."
13+
fi
14+
docker run -itd --rm --name zgrab_mysql-$VERSION -e MYSQL_ALLOW_EMPTY_PASSWORD=true -e MYSQL_LOG_CONSOLE=true mysql:$VERSION
15+
}
16+
17+
function waitFor() {
18+
VERSION=$1
19+
CONTAINER_NAME=zgrab_mysql-$VERSION
20+
echo "mysql/setup: Waiting for mysqld process to come up on $CONTAINER_NAME..."
21+
while ! (docker exec $CONTAINER_NAME ps -Af | grep mysqld > /dev/null); do
22+
echo -n "*"
23+
sleep 1
24+
done
25+
echo "...ok."
26+
}
27+
28+
echo "mysql/setup: Launching docker containers..."
29+
for version in $versions; do
30+
launch $version
31+
done
1132

12-
echo "Waiting for MySQL to start up on all containers..."
13-
CONTAINER_NAME=testmysql-5.5 MYSQL_PORT=13306 ./util/wait_for_mysqld.sh
14-
CONTAINER_NAME=testmysql-5.6 MYSQL_PORT=23306 ./util/wait_for_mysqld.sh
15-
CONTAINER_NAME=testmysql-5.7 MYSQL_PORT=33306 ./util/wait_for_mysqld.sh
16-
CONTAINER_NAME=testmysql-8.0 MYSQL_PORT=43306 ./util/wait_for_mysqld.sh
33+
for version in $versions; do
34+
waitFor $version
35+
done

‎integration_tests/mysql/single_run.sh

-28
This file was deleted.

‎integration_tests/mysql/test.sh

+41-5
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,41 @@
1-
#!/bin/sh -e
2-
MYSQL_VERSION=5.5 MYSQL_PORT=13306 ./single_run.sh
3-
MYSQL_VERSION=5.6 MYSQL_PORT=23306 ./single_run.sh
4-
MYSQL_VERSION=5.7 MYSQL_PORT=33306 ./single_run.sh
5-
MYSQL_VERSION=8.0 MYSQL_PORT=43306 ./single_run.sh
1+
#!/usr/bin/env bash
2+
set -e
3+
4+
versions="5.5 5.6 5.7 8.0"
5+
6+
# Run the MySQL-specific integration tests:
7+
# 1. Run zgrab2 on the container
8+
# 2. Check that data.mysql.result.handshake.parsed.server_version matches $MYSQL_VERSION
9+
10+
MODULE_DIR=$(dirname $0)
11+
TEST_ROOT=$MODULE_DIR/..
12+
ZGRAB_ROOT=$MODULE_DIR/../..
13+
ZGRAB_OUTPUT=$ZGRAB_ROOT/zgrab-output
14+
15+
status=0
16+
17+
function doTest() {
18+
MYSQL_VERSION=$1
19+
CONTAINER_NAME="zgrab_mysql-$MYSQL_VERSION"
20+
OUTPUT_FILE="$ZGRAB_OUTPUT/mysql/$MYSQL_VERSION.json"
21+
echo "mysql/test: Testing MySQL Version $MYSQL_VERSION..."
22+
CONTAINER_NAME=$CONTAINER_NAME $ZGRAB_ROOT/docker-runner/docker-run.sh mysql --timeout 10 > $OUTPUT_FILE
23+
SERVER_VERSION=$($ZGRAB_ROOT/jp -u data.mysql.result.handshake.parsed.server_version < $OUTPUT_FILE)
24+
if [[ "$SERVER_VERSION" == "$MYSQL_VERSION."* ]]; then
25+
echo "Server version matches expected version: $SERVER_VERSION == $MYSQL_VERSION.*"
26+
else
27+
echo "Server version mismatch: Got $SERVER_VERSION, expected $MYSQL_VERSION.*"
28+
status=1
29+
fi
30+
echo "mysql/test: BEGIN docker+mysql logs from $CONTAINER_NAME [{("
31+
docker logs --tail all $CONTAINER_NAME
32+
echo ")}] END docker+mysql logs from $CONTAINER_NAME"
33+
}
34+
35+
mkdir -p $ZGRAB_OUTPUT/mysql
36+
37+
for version in $versions; do
38+
doTest $version
39+
done
40+
41+
exit $status

‎integration_tests/mysql/util/launch_mysql_container.sh

-14
This file was deleted.

‎integration_tests/mysql/util/wait_for_mysqld.sh

-28
This file was deleted.

0 commit comments

Comments
 (0)