cmd/gosim: add -seeds flag

- Add a -seeds flag to the metatest binary.
- Pass -seeds from cmd/gosim for both gosim test and gosim debug.
- Modify testing output to print the test's seed.
- Add a script test verifying this works.
- Update the README to showcase determinism and the seeds flag.

Author: jellevandenhooff

README.md

@@ -36,41 +36,67 @@ started with `gosim`, you need to import it in your module:
 To test code with Gosim, write a small a test in a file `simple_test.go` and then run
 it using `go test`:
 ```go
-package example_test
+package examples_test
 
 import (
+	"math/rand"
 	"testing"
 
 	"github.com/jellevandenhooff/gosim"
 )
 
 func TestGosim(t *testing.T) {
 	t.Logf("Are we in the Matrix? %v", gosim.IsSim())
+	t.Logf("Random: %d", rand.Int())
 }
 ```
 ```
-> go test -v -run TestGosim
+> go test -v -count=1 -run TestGosim
 === RUN   TestGosim
-    simple_test.go:10: Are we in the Matrix? false
---- PASS: TestGosim (0.00s simulated)
+    simple_test.go:11: Are we in the Matrix? false
+    simple_test.go:12: Random: 1225418128470362734
+--- PASS: TestGosim (0.00s)
 PASS
 ok  	example	0.216s
 ```
+The test prints that Gosim is not enabled, and each time the test runs the random number is different.
+
 To run this test with `gosim` instead, replace `go test` with
 `go run github.com/jellevandenhooff/gosim/cmd/gosim test`:
 ```
 > go run github.com/jellevandenhooff/gosim/cmd/gosim test -v -run TestGosim
-=== RUN   TestGosim
-    1 main/4     14:10:03.000 INF example/simple_test.go:11 > Are we in the Matrix? true method=t.Logf
---- PASS: TestGosim (0.00s)
+=== RUN   TestGosim (seed 1)
+    1 main/4     14:10:03.000 INF examples/simple_test.go:12 > Are we in the Matrix? true method=t.Logf
+    2 main/4     14:10:03.000 INF examples/simple_test.go:13 > Random: 811966193383742320 method=t.Logf
+--- PASS: TestGosim (0.00s simulated)
 ok  	translated/example	0.204s
 ```
 The `gosim test` command has flags similar to `go test`. The test output is
 more involved than a normal `go test`. Every log line includes the simulated
 machine and the goroutine that invoked the log to help debug tests running on
 multiple machines.
 
-If running gosim fails with errors about missing `go.sum` entries, run
+The `=== RUN` line includes the test's seed. Each time this test runs with the
+same seed it will output the same random number. To test with different seeds,
+you can pass ranges of seeds to `gosim test`:
+```
+go run github.com/jellevandenhooff/gosim/cmd/gosim test -v -seeds=1-3 -run=TestGosim .
+=== RUN   TestGosim (seed 1)
+    1 main/4     14:10:03.000 INF examples/simple_test.go:12 > Are we in the Matrix? true method=t.Logf
+    2 main/4     14:10:03.000 INF examples/simple_test.go:13 > Random: 811966193383742320 method=t.Logf
+--- PASS: TestGosim (0.00s simulated)
+=== RUN   TestGosim (seed 2)
+    1 main/4     14:10:03.000 INF examples/simple_test.go:12 > Are we in the Matrix? true method=t.Logf
+    2 main/4     14:10:03.000 INF examples/simple_test.go:13 > Random: 5374891573232646577 method=t.Logf
+--- PASS: TestGosim (0.00s simulated)
+=== RUN   TestGosim (seed 3)
+    1 main/4     14:10:03.000 INF examples/simple_test.go:12 > Are we in the Matrix? true method=t.Logf
+    2 main/4     14:10:03.000 INF examples/simple_test.go:13 > Random: 3226404213937589817 method=t.Logf
+--- PASS: TestGosim (0.00s simulated)
+ok  	translated/github.com/jellevandenhooff/gosim/examples	0.254s
+```
+
+If running `gosim` fails with errors about missing `go.sum` entries, run
 `go mod tidy` to update your `go.sum` file.
 
 ## Simulation
@@ -162,7 +188,7 @@ what happens this test is run:
 <!-- TODO: two machines and a bug? -->
 ```
 > go run github.com/jellevandenhooff/gosim/cmd/gosim test -v -run TestMachines
-=== RUN   TestMachines
+=== RUN   TestMachines (seed 1)
     1 server/5   14:10:03.000 INF example/machines_test.go:17 > starting server
     2 main/4     14:10:04.000 INF example/machines_test.go:27 > making a request
     3 server/8   14:10:04.000 INF example/machines_test.go:20 > got a request from 11.0.0.1:10000
@@ -217,7 +243,7 @@ realistic latency and timeouts without development time becoming slow.
 ## Debugging
 Gosim's simulation is deterministic, which means that running a test twice
 will result in the exact same behavior, from randomly generated numbers
-to goroutine concurrency interleavings. That is cool because it means that
+to goroutine concurrency interleavings. That is useful because it means that
 if we see an interesting log after running the program but do not fully
 understand why that log printed a certain value, we can re-run the program
 and see exactly what happened at the moment of printing.

cmd/gosim/doc.go

@@ -32,20 +32,25 @@ The -race flag translates all code with the race build tag set.
 
 The 'test' command:
 
-Usage: gosim test [-race] [-run=...] [-v] [packages]
+Usage: gosim test [-race] [-run=...] [-seeds=...] [-v] [packages]
 
 The test command translates and runs tests for the specified packages.  It first
 invokes translate, and then invokes 'go test' on the translated code, passing
 through the -run and -v flags.
 
+The -seeds flag specifies what seeds to run the tests on as a comma-separated
+list of seeds and ranges, such as -seeds=1,2,5-10. The default is -seeds=1.
+
 The 'debug' command:
 
-Usage: gosim debug [-race] [-headless] -package=[package] -test=[test] -step=[step]
+Usage: gosim debug [-race] [-headless] -package=[package] -test=[test] [-seed=...] -step=[step]
 
 The debug command translates and runs a specific test using the delve debugger.
 It first invokes translate, and then runs 'dlv test' on the specific test. The
 -step flag is the step to pause at as seen in the logs from running 'gosim test'.
 
+The -seed flag is the seed to run. The default is -seed=1.
+
 The -headless flag optionally runs delve in headless mode for use with an
 external interface like an IDE.
 

cmd/gosim/main.go

@@ -182,6 +182,7 @@ func main() {
 		logformat := testflags.String("logformat", "pretty", "gosim log formatting: raw|indented|pretty")
 		simtrace := testflags.String("simtrace", "", "set of a comma-separated traces to enable")
 		jsonlogout := testflags.String("jsonlogout", "", "path to a file to write json log to, for use with viewer")
+		seeds := testflags.String("seeds", "1", "a comma separated list of seeds and ranges to run, such as 1,2,10-100,99")
 		testflags.Parse(cmdArgs)
 
 		cfg.Race = *race
@@ -230,6 +231,9 @@ func main() {
 			}
 			args = append(args, "-jsonlogout", abs)
 		}
+		if *seeds != "" {
+			args = append(args, "-seeds", *seeds)
+		}
 
 		cmd := exec.Command(name, args...)
 		cmd.Env = append(os.Environ(), "FORCE_COLOR=1")
@@ -368,6 +372,7 @@ func main() {
 		race := debugflags.Bool("race", false, "build in -race mode")
 		pkg := debugflags.String("package", "", "package path to debug")
 		test := debugflags.String("test", "", "full test name to debug")
+		seed := debugflags.Int("seed", 1, "seed to debug")
 		step := debugflags.Int("step", 0, "step to break at")
 		headless := debugflags.Bool("headless", false, "run headless for IDE debugging")
 		debugflags.Parse(cmdArgs)
@@ -447,6 +452,7 @@ stepout`)
 			"^"+*test+"$",
 			"-test.v",
 			"-step-breakpoint="+fmt.Sprint(*step),
+			"-seeds="+fmt.Sprint(*seed),
 		)
 
 		cmd := exec.Command(name, flags...)

examples/simple_test.go

@@ -1,11 +1,13 @@
 package examples_test
 
 import (
+	"math/rand"
 	"testing"
 
 	"github.com/jellevandenhooff/gosim"
 )
 
-func TestHello(t *testing.T) {
-	t.Log(gosim.IsSim())
+func TestGosim(t *testing.T) {
+	t.Logf("Are we in the Matrix? %v", gosim.IsSim())
+	t.Logf("Random: %d", rand.Int())
 }

gosimruntime/runtime.go

@@ -90,6 +90,8 @@ func newMachine(label string) *Machine {
 }
 
 type scheduler struct {
+	seed int64
+
 	goroutines      intrusiveList[*goroutine]
 	runnable        intrusiveList[*goroutine]
 	nextGoroutineID int
@@ -124,6 +126,7 @@ func newScheduler(seed int64, logger logger, checksummer *checksummer, extraenv
 	rand := mathrand.New(mathrand.NewSource(seed))
 
 	s := &scheduler{
+		seed:            seed,
 		goroutines:      make([]*goroutine, 0, 1024),
 		runnable:        make([]*goroutine, 0, 1024),
 		nextGoroutineID: 1,
@@ -330,6 +333,10 @@ type internalRunResult struct {
 	Err       error
 }
 
+func Seed() int64 {
+	return gs.get().seed
+}
+
 func run(f func(), seed int64, enableChecksum bool, captureLog bool, logLevelOverride string, simLogger io.Writer, extraenv []string) internalRunResult {
 	if !runtimeInitialized {
 		panic("not yet initialized")

gosimruntime/testmain.go

@@ -3,11 +3,13 @@ package gosimruntime
 import (
 	"encoding/json"
 	"flag"
+	"fmt"
 	"io"
 	"log"
 	"maps"
 	"os"
 	"slices"
+	"strconv"
 	"strings"
 )
 
@@ -68,8 +70,35 @@ var supportedFlags = map[string]bool{
 	"test.v":            true,
 }
 
+type seedRange struct {
+	start, end int64
+}
+
+func parseSeeds(seedsStr string) ([]seedRange, error) {
+	var seeds []seedRange
+	for _, rangeStr := range strings.Split(seedsStr, ",") {
+		startStr, endStr, ok := strings.Cut(rangeStr, "-")
+		if !ok {
+			endStr = startStr
+		}
+
+		start, err := strconv.ParseInt(startStr, 10, 64)
+		if err != nil {
+			return nil, fmt.Errorf("bad start of seed range %q: %v", startStr, err)
+		}
+		end, err := strconv.ParseInt(endStr, 10, 64)
+		if err != nil {
+			return nil, fmt.Errorf("bad end of seed range %q: %v", endStr, err)
+		}
+
+		seeds = append(seeds, seedRange{start: start, end: end})
+	}
+	return seeds, nil
+}
+
 func TestMain(rt Runtime) {
 	simtrace := flag.String("simtrace", "", "set of comma-separated traces to enable")
+	seedsStr := flag.String("seeds", "1", "comma-separated list of seeds and ranges")
 
 	// TODO: make this flag beter; it won't work with multiple test runs?
 	jsonlogout := flag.String("jsonlogout", "", "path to a file to write json log to, for use with viewer")
@@ -96,7 +125,11 @@ func TestMain(rt Runtime) {
 			log.Fatal(err)
 		}
 
-		seed := int64(1)
+		seeds, err := parseSeeds(*seedsStr)
+		if err != nil {
+			log.Fatalf("parsing -seeds: %s", err)
+		}
+
 		enableTracer := true
 		captureLog := true
 		logLevelOverride := "INFO"
@@ -114,25 +147,30 @@ func TestMain(rt Runtime) {
 			}
 		}
 
-		for _, test := range allTestsSlice {
-			// log.Println("running", test.Name)
-			result := run(func() {
-				ok := rt.TestEntrypoint(match, skip, []Test{
-					test,
-				})
-				if !ok {
-					SetAbortError(ErrTestFailed)
+		for _, seedRange := range seeds {
+			for seed := seedRange.start; seed <= seedRange.end; seed++ {
+				// TODO: filter list of tests outside since each run call has quite some overhead
+				// for skipped tests.
+				for _, test := range allTestsSlice {
+					result := run(func() {
+						ok := rt.TestEntrypoint(match, skip, []Test{
+							test,
+						})
+						if !ok {
+							SetAbortError(ErrTestFailed)
+						}
+					}, int64(seed), enableTracer, captureLog, logLevelOverride, makeConsoleLogger(os.Stderr), []string{})
+
+					if jsonout != nil {
+						if _, err := jsonout.Write(result.LogOutput); err != nil {
+							log.Fatalf("error writing jsonout: %s", err)
+						}
+					}
+
+					if result.Failed {
+						outerOk = false
+					}
 				}
-			}, seed, enableTracer, captureLog, logLevelOverride, makeConsoleLogger(os.Stderr), []string{})
-
-			if jsonout != nil {
-				if _, err := jsonout.Write(result.LogOutput); err != nil {
-					log.Fatalf("error writing jsonout: %s", err)
-				}
-			}
-
-			if result.Failed {
-				outerOk = false
 			}
 		}
 

internal/testing/testing.go

@@ -884,7 +884,7 @@ func (t *T) Run(name string, f func(t *T)) bool {
 	// t.w = indenter{&t.common}
 
 	if t.chatty != nil {
-		t.chatty.Updatef(t.name, "=== RUN   %s\n", t.name)
+		t.chatty.Updatef(t.name, "=== RUN   %s (seed %d)\n", t.name, gosimruntime.Seed())
 	}
 	running.Store(t.name, highPrecisionTimeNow())
 

internal/tests/script/testdata/gosimseeds.txtar

@@ -0,0 +1,23 @@
+# running gosim with seeds
+exec gosim test -run TestRand -v .
+stdout 'TestRand.*seed 1'
+! stdout 'TestRand.*seed 2'
+
+exec gosim test -run TestRand -seeds 2-3,9 -v .
+! stdout 'TestRand.*seed 1$'
+stdout 'TestRand.*seed 2'
+stdout 'TestRand.*seed 3'
+! stdout 'TestRand.*seed 4'
+stdout 'TestRand.*seed 9'
+
+-- seed_test.go --
+package behavior_test
+
+import (
+	"math/rand"
+	"testing"
+)
+
+func TestRand(t *testing.T) {
+	t.Log(rand.Int())
+}