Create a Go script to run Dgraph tests concurrently #6895

manishrjain · 2020-11-14T16:31:22Z

The current test.sh script runs all the packages one by one, which takes 30mins for the tests to run. Even then, we have to separate out the systests from the rest.

This PR creates an equivalent Go script which can divide up the packages in multiple goroutines. Each goroutine runs a Dgraph cluster which the tests running on that goroutine use. To avoid interactions among clusters, the script changes docker-compose to not expose the ports. Instead, use the docker generated public ports for the tests to use to interact with the cluster. It makes those available via environment variables, which get picked up by testutil package.

TODO: This PR disables a bunch of tests and packages which are failing. For now, we set concurrency to 1, due to some weird failures with tests when run with multiple clusters. Needs more work to fix up these failures.

This change is

…king to them.

It can be set with --extra_alpha_flags if needed.

In the output below: 1. Live loader runs with family.schema which includes indexes. 2. The schema says that the name predicate does not have any index 3. The query runs, expecting a term index. pipeline: [[/home/dmai/go/bin/dgraph live --schema /home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/load-uids/family.schema --files /home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/load-uids/family.json --alpha localhost:33616 --zero localhost:33567 -u groot -p password]] [Decoder]: Using assembly version of decoder Page Size: 4096 Running transaction with dgraph endpoint: localhost:33616 Login successful. Processing schema file "/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/load-uids/family.schema" Processed schema file "/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/load-uids/family.schema" Found 1 data file(s) to process Processing data file "/home/dmai/go/src/github.com/dgraph-io/dgraph/dgraph/cmd/live/load-uids/family.json" Number of TXs run : 1 Number of N-Quads processed : 43 Time spent : 67.855766ms N-Quads processed per second : 43 json:"{\"schema\":[{\"predicate\":\"age\",\"type\":\"int\"},{\"predicate\":\"aka\",\"type\":\"string\"},{\"predicate\":\"carries\",\"type\":\"string\"},{\"predicate\":\"dgraph.acl.rule\",\"type\":\"uid\",\"list\":true},{\"predicate\":\"dgraph.cors\",\"type\":\"string\",\"index\":true,\"tokenizer\":[\"exact\"],\"list\":true,\"upsert\":true},{\"predicate\":\"dgraph.drop.op\",\"type\":\"string\"},{\"predicate\":\"dgraph.graphql.p_query\",\"type\":\"string\"},{\"predicate\":\"dgraph.graphql.p_sha256hash\",\"type\":\"string\",\"index\":true,\"tokenizer\":[\"exact\"]},{\"predicate\":\"dgraph.graphql.schema\",\"type\":\"string\"},{\"predicate\":\"dgraph.graphql.schema_created_at\",\"type\":\"datetime\"},{\"predicate\":\"dgraph.graphql.schema_history\",\"type\":\"string\"},{\"predicate\":\"dgraph.graphql.xid\",\"type\":\"string\",\"index\":true,\"tokenizer\":[\"exact\"],\"upsert\":true},{\"predicate\":\"dgraph.password\",\"type\":\"password\"},{\"predicate\":\"dgraph.rule.permission\",\"type\":\"int\"},{\"predicate\":\"dgraph.rule.predicate\",\"type\":\"string\",\"index\":true,\"tokenizer\":[\"exact\"],\"upsert\":true},{\"predicate\":\"dgraph.type\",\"type\":\"string\",\"index\":true,\"tokenizer\":[\"exact\"],\"list\":true},{\"predicate\":\"dgraph.user.group\",\"type\":\"uid\",\"reverse\":true,\"list\":true},{\"predicate\":\"dgraph.xid\",\"type\":\"string\",\"index\":true,\"tokenizer\":[\"exact\"],\"upsert\":true},{\"predicate\":\"name\",\"type\":\"string\"},{\"predicate\":\"parent_to\",\"type\":\"uid\",\"list\":true},{\"predicate\":\"role\",\"type\":\"string\"},{\"predicate\":\"sibling_of\",\"type\":\"uid\",\"list\":true}],\"types\":[{\"fields\":[{\"name\":\"name\"},{\"name\":\"age\"},{\"name\":\"role\"},{\"name\":\"aka\"},{\"name\":\"carries\"},{\"name\":\"parent_to\"},{\"name\":\"sibling_of\"}],\"name\":\"FamilyMember\"},{\"fields\":[{\"name\":\"dgraph.graphql.schema\"},{\"name\":\"dgraph.graphql.xid\"}],\"name\":\"dgraph.graphql\"},{\"fields\":[{\"name\":\"dgraph.graphql.schema_history\"},{\"name\":\"dgraph.graphql.schema_created_at\"}],\"name\":\"dgraph.graphql.history\"},{\"fields\":[{\"name\":\"dgraph.graphql.p_query\"},{\"name\":\"dgraph.graphql.p_sha256hash\"}],\"name\":\"dgraph.graphql.persisted_query\"},{\"fields\":[{\"name\":\"dgraph.xid\"},{\"name\":\"dgraph.acl.rule\"}],\"name\":\"dgraph.type.Group\"},{\"fields\":[{\"name\":\"dgraph.rule.predicate\"},{\"name\":\"dgraph.rule.permission\"}],\"name\":\"dgraph.type.Rule\"},{\"fields\":[{\"name\":\"dgraph.xid\"},{\"name\":\"dgraph.password\"},{\"name\":\"dgraph.user.group\"}],\"name\":\"dgraph.type.User\"}]}" txn:<start_ts:1473 > latency:<parsing_ns:13688 processing_ns:742728 assign_timestamp_ns:655815 total_ns:1836730 > metrics:<num_uids:<key:"_total" value:0 > > load_test.go:146: Error Trace: load_test.go:146 load_test.go:212 Error: Received unexpected error: rpc error: code = Unknown desc = : rpc error: code = Unknown desc = Attribute name is not indexed with type term Test: TestLiveLoadJsonUidKeep

…pose.yml.

…cker-compose.yml.

- Use minio flags.

…yml.

… other tests.

codelingo

2 issues found. 15 rules errored during the review.

codelingo · 2020-11-17T07:09:29Z

t/main.go

+		"Only run this test")
+	count = pflag.IntP("count", "c", 0,
+		"If set, would add -count arg to go test.")
+	concurrency = pflag.IntP("concurrency", "j", 1,


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-17T07:09:29Z

t/main.go

+	return nil
+}
+
+func runTests(taskCh chan task, closer *z.Closer) error {


Returned channels or channel arguments should generally have a direction.

View Rule

codelingo

15 issues found. 12 rules errored during the review.

codelingo · 2020-11-18T03:38:26Z

testutil/docker.go

-	argv = append(argv, services...)
-	return Exec(argv...)
-}
+// DockerStart starts the specified services.


Every exported function in a program should have a doc comment. The first sentence should be a summary that starts with the name (DockerRun) being declared.
From effective go.

View Rule

codelingo · 2020-11-18T03:38:27Z

testutil/client.go

@@ -42,7 +45,10 @@ import (

 // socket addr = IP address and port number
 var (
-
+	// Instance is the instance name of the Alpha.
+	DockerPrefix  string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:27Z

tlstest/mtls_internal/backup/encryption/backup_test.go

-	backupDst      = "minio://minio1:9001/dgraph-backup?secure=false"
-	localBackupDst = "minio://localhost:9001/dgraph-backup?secure=false"
+	backupDst      string
+	localBackupDst string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:27Z

tlstest/mtls_internal/backup/encryption/backup_test.go

@@ -48,8 +48,8 @@ var (

 	mc             *minio.Client
 	bucketName     = "dgraph-backup"
-	backupDst      = "minio://minio1:9001/dgraph-backup?secure=false"
-	localBackupDst = "minio://localhost:9001/dgraph-backup?secure=false"
+	backupDst      string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:27Z

t/main.go

+			" tests to fail on any concurrency setting > 1.")
+	keepCluster = pflag.BoolP("keep", "k", false,
+		"Keep the clusters running on program end.")
+	clear = pflag.BoolP("clear", "r", false,


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:28Z

systest/backup/minio/backup_test.go

@@ -47,11 +47,18 @@ var (

 	mc             *minio.Client
 	bucketName     = "dgraph-backup"
-	backupDst      = "minio://minio1:9001/dgraph-backup?secure=false"
-	localBackupDst = "minio://localhost:9001/dgraph-backup?secure=false"
+	backupDst      string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:29Z

systest/backup/minio/backup_test.go

-	backupDst      = "minio://minio1:9001/dgraph-backup?secure=false"
-	localBackupDst = "minio://localhost:9001/dgraph-backup?secure=false"
+	backupDst      string
+	localBackupDst string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:29Z

systest/backup/encryption/backup_test.go

@@ -50,11 +50,15 @@ var (

 	mc             *minio.Client
 	bucketName     = "dgraph-backup"
-	backupDst      = "minio://minio1:9001/dgraph-backup?secure=false"
-	localBackupDst = "minio://localhost:9001/dgraph-backup?secure=false"
+	backupDst      string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:29Z

systest/backup/encryption/backup_test.go

-	backupDst      = "minio://minio1:9001/dgraph-backup?secure=false"
-	localBackupDst = "minio://localhost:9001/dgraph-backup?secure=false"
+	backupDst      string
+	localBackupDst string


Avoid global variables to improve readability and reduce complexity

View Rule

codelingo · 2020-11-18T03:38:29Z

t/main.go

+
+var _threadId int32
+
+func runTests(taskCh chan task, closer *z.Closer) error {


Returned channels or channel arguments should generally have a direction.

View Rule

manishrjain and others added 25 commits November 12, 2020 19:14

Start work on making tests run faster.

00fc22b

Working Go script for bringing up clusters, putting them down and tal…

5ff5c4e

…king to them.

Test Go can build and run acl package test

9264484

chore(compose): Don't set port offsets when not exposing host ports.

44b4cdd

chore(compose): Set multi-zero addresses for latest tag.

04cad37

test: Update docker-compose.yml without expose ports.

91f8898

test: Update tlstest/certrequireandverify/docker-compose.yml.

19ec054

test: Update tlstest/certrequest/docker-compose.yml.

4de012b

test: Update tlstest/certverifyifgiven/docker-compose.yml.

2f90848

test: Update tlstest/zero_https/no_tls/docker-compose.yml.

4cba95d

test: Update tlstest/zero_https/all_routes_tls/docker-compose.yml.

50c9587

Lots of progress

fdfed71

chore(compose): Add flags to set Alpha and Zero volumes.

d7ce11c

chore(compose): Don't set acl_access_ttl by default.

7640c2b

It can be set with --extra_alpha_flags if needed.

test: Update systest/bgindex/docker-compose.yml.

240a274

chore(compose): Support volumes of type volume.

9b65687

test: Update systest/21million/docker-compose.yml.

6730bff

test: Update systest/loader/docker-compose.yml.

d01737b

test: Update systest/license/docker-compose.yml.

124afc0

test: Update systest/ludicrous/docker-compose.yml.

07ce139

Kinda works

f68c7e7

chore(compose): Add flags to set env_file.

7aaf5a3

test: Update systest/export/docker-compose.yml.

631141f

More updates

54df63c

manishrjain requested a review from vvbalaji-dgraph as a code owner November 14, 2020 16:31

github-actions bot added area/enterprise Related to proprietary features area/integrations Related to integrations with other projects. area/live-loader Issues related to live loading. area/testing Testing related issues labels Nov 14, 2020

danielmai and others added 19 commits November 16, 2020 17:20

test: Remove container_name from tlstest/zero_https/no_tls/docker-com…

9740bd8

…pose.yml.

test: Remove container_name from tlstest/zero_https/all_routes_tls/do…

ecbec4e

…cker-compose.yml.

test: Remove container_name from systest/bgindex/docker-compose.yml.

bdec5d0

test: Remove container_name from systest/21million/docker-compose.yml.

15f42bc

test: Remove container_name from systest/loader/docker-compose.yml.

268a480

test: Remove container_name from systest/license/docker-compose.yml.

d18da43

test: Remove container_name from systest/ludicrous/docker-compose.yml.

23dd1d2

test: Remove container_name from systest/export/docker-compose.yml.

69f00dc

- Use minio flags.

test: Remove container_name from systest/1million/docker-compose.yml.

c8c26cd

test: Remove container_name from systest/group-delete/docker-compose.…

b8d9fc0

…yml.

test: Remove container_name from systest/backup/minio/docker-compose.…

8ddf108

…yml.

test: Update systest/backup/minio-large/docker-compose.yml.

95224f7

test: Update systest/backup/filesystem/docker-compose.yml.

417d38a

Able to run worker tests. But, failing.

be63e86

Worker package tests are OK.

4f61b7a

Merge branch 'danielmai/mrjn-tests-compose' into mrjn/tests

956d912

mutations_mode pkg works

ca0ebe1

Bunch of tests work now. Disabled mtls tests, which are all copies of…

a233e0a

… other tests.

Some more work

07c99d0

codelingo bot reviewed Nov 17, 2020

View reviewed changes

manishrjain added 7 commits November 17, 2020 13:30

Custom tests are passing.

6e13c2f

Add TEAMCITY option

562e9fd

Make it look nice

88723f5

Make test.sh run the t script

7a78986

Self review

b0ab871

Fix up the exit code.

8a93a6a

Bring test.sh back.

1635917

manishrjain merged commit 79d7f9c into master Nov 18, 2020

manishrjain deleted the mrjn/tests branch November 18, 2020 02:15

codelingo bot reviewed Nov 18, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create a Go script to run Dgraph tests concurrently #6895

Create a Go script to run Dgraph tests concurrently #6895

manishrjain commented Nov 14, 2020 •

edited

Loading

codelingo bot left a comment

codelingo bot Nov 17, 2020

codelingo bot Nov 17, 2020

codelingo bot left a comment

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020

codelingo bot Nov 18, 2020


		var _threadId int32

		func runTests(taskCh chan task, closer *z.Closer) error {

Create a Go script to run Dgraph tests concurrently #6895

Create a Go script to run Dgraph tests concurrently #6895

Conversation

manishrjain commented Nov 14, 2020 • edited Loading

codelingo bot left a comment

Choose a reason for hiding this comment

codelingo bot Nov 17, 2020

Choose a reason for hiding this comment

codelingo bot Nov 17, 2020

Choose a reason for hiding this comment

codelingo bot left a comment

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

codelingo bot Nov 18, 2020

Choose a reason for hiding this comment

manishrjain commented Nov 14, 2020 •

edited

Loading