-
Notifications
You must be signed in to change notification settings - Fork 17.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
all: plan9-arm builder failing frequently with a variety of errors #49338
Comments
For most of plan9-arm failures, the cause is filesystem flakiness on the builder machines (a cluster of 3 x raspberry pi 4). I've just reconfigured the build script to stop using ramfs (which has a locking bug) for /tmp, and stop using the ad hoc directory cache (which still seems to have some flaws). This should bypass most of the failures but will make the test suite a lot slower to run (by go.17 it had already regressed to about 2 hours even with these filesystem tweaks). There are some other flakes of the "shouldn't be possible" variety, which are are rare and perplexing and impossible to replicate. They are occurring in general go runtime code, not specific to plan 9. There's a slight whiff of possible memory cache coherence trouble in some of these: It would be good to have a server-class ARM platform capable of running plan 9 as a more reliable builder. I don't know of one. |
Change https://golang.org/cl/362975 mentions this issue: |
The plan9 builders have relatively slow filesystems, and the website repo is relatively filesystem-intensive — many of its tests walk and process large trees of website content. Instead of trying to precisely tune the x/website tests to pass on the plan9 builders, let's skip this repo on them. For golang/go#49338. Change-Id: Ieaf50fb6955a6b39012a23ff72e61c1c31e4ce7c Reviewed-on: https://go-review.googlesource.com/c/build/+/362975 Trust: Bryan C. Mills <[email protected]> Run-TryBot: Bryan C. Mills <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Carlos Amedee <[email protected]> Reviewed-by: Dmitri Shuralyov <[email protected]>
Many of the failure modes seem to have cleared up around 11 Nov.; it's not clear to me what fixed them. One of the failures visible on the dashboard today is #49653; the other is the |
Change https://golang.org/cl/369018 mentions this issue: |
To prevent interleaving of output when 'go build' compiles several packages in parallel, the output mutex in the Builder struct must be locked around any calls to Builder.Print which could generate arbitrary amounts of text (ie more than is guaranteed to be written atomically to a pipe). Fixes #49987 For #49338 Change-Id: I7947df57667deeff3f03f231824298d823f8a943 Reviewed-on: https://go-review.googlesource.com/c/go/+/369018 Reviewed-by: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Trust: Russ Cox <[email protected]>
Change https://golang.org/cl/380414 mentions this issue: |
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>
A sampling from the past week or so. Undiagnosed failures in bold.
2022-01-27T05:30:27-be5769c-a991d9d/plan9-arm
That's stuck running 2022-01-27T05:30:27-bbe1937-a991d9d/plan9-arm (#50857)
2022-01-24T16:42:11-2cc1836-19d819d/plan9-arm (#50775, mitigated)
That's running a 2022-01-21T21:27:57-e7c9de2-9eba5ff/plan9-arm (#46520, not specific to |
Change https://go.dev/cl/408697 mentions this issue: |
For golang/go#52653. Updates golang/go#49338. Change-Id: Ib23be56bd79bce8b3168cb40064916a2a91dbb88 Reviewed-on: https://go-review.googlesource.com/c/build/+/408697 Auto-Submit: Bryan Mills <[email protected]> Reviewed-by: Alex Rakoczy <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]>
This test has timed out several times recently on this builder. The test is pretty slow even on Linux, and may be especially filesystem-intensive. For golang/go#49338 Change-Id: Ife2bb399b10f369f815055bb0ad44bb007f606b7 Reviewed-on: https://go-review.googlesource.com/c/text/+/380414 Trust: Bryan Mills <[email protected]> Run-TryBot: Bryan Mills <[email protected]> TryBot-Result: Gopher Robot <[email protected]> Reviewed-by: Ian Lance Taylor <[email protected]>
This comment was marked as resolved.
This comment was marked as resolved.
Found new dashboard test flakes for:
2022-12-05 22:01 plan9-arm go@185e1a7b os/exec.TestExtraFilesRace (log)
|
Found new dashboard test flakes for:
2022-12-06 20:08 plan9-arm net@1e63c2f0 go@a4a86c7b x/net/http2.TestTransportUsesGoAwayDebugError_Body (log)
|
Found new dashboard test flakes for:
2022-12-22 15:34 plan9-arm benchmarks@03c43f28 go@f721fa3b (log)
|
Found new dashboard test flakes for:
2022-12-01 20:58 plan9-arm go@cc1771e2 net/http.TestOmitHTTP2Vet (log)
2022-12-12 22:32 plan9-arm crypto@23edec0b go@357ea858 (log)
2022-12-12 23:10 plan9-arm go@b16e94d1 (log)
2022-12-13 21:00 plan9-arm benchmarks@555ee2b5 go@357ea858 (log)
2022-12-15 00:54 plan9-arm tools@5899b6a2 go@ea14d1b6 (log)2022-12-23 17:16 plan9-arm go@38cfb3be net.TestPipe (log)
2022-12-29 17:47 plan9-arm go@642fd5f7 net/http.TestServerReadTimeout (log)
|
Found new dashboard test flakes for:
2023-01-20 21:10 plan9-arm benchmarks@33cb3306 go@27b6ace2 (log)
|
Found new dashboard test flakes for:
2023-01-05 16:42 plan9-arm net@f8411da7 go@cc82867f x/net/http2/h2c.TestPropagation (log)
2023-01-19 20:43 plan9-arm go@f89badf6 (log)
2023-01-19 20:46 plan9-arm go@e08642ca (log)2023-01-20 02:42 plan9-arm tools@a7f033af go@c641d48a x/tools/gopls/internal/lsp/filecache.TestConcurrency (log)
2023-01-20 17:52 plan9-arm go@1011ccad os.TestDirFS (log)
2023-01-20 19:38 plan9-arm go@f07910bd os.TestDirFS (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestWriteAtInAppendMode (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestOpenFileLimit (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestDirFSRootDir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReaddirSmallSeek (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestDirFS (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReadDir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestReadFile (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReaddir (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReaddirnames (log)
2023-01-20 22:42 plan9-arm go@0518e33f os.TestFileReadDir (log)
2023-01-23 18:50 plan9-arm go@b90971b2 os.TestDirFS (log)
|
Found new dashboard test flakes for:
2023-01-24 19:29 plan9-arm go@083d94f6 (log)
|
Found new dashboard test flakes for:
2023-01-20 09:27 plan9-arm go@1bef0707 os.TestDirFS (log)
2023-01-25 16:38 plan9-arm go@1d3088ef os.TestFileReadDir (log)
2023-01-25 18:50 plan9-arm go@3c7884a2 (log)2023-01-26 01:06 plan9-arm go@89de906a os.TestDirFS (log)
2023-01-26 19:45 plan9-arm go@627f1286 os.TestFileReadDir (log)
2023-01-26 21:33 plan9-arm net@296f09aa go@dd44d981 x/net/http2/h2c.TestPropagation (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestWriteAtInAppendMode (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestOpenFileLimit (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReadDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReadFile (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestDirFSRootDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestDirFS (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestReaddirSmallSeek (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReadDir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReaddir (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os.TestFileReaddirnames (log)
2023-01-26 21:54 plan9-arm go@ea6d4b0e os/exec (log)
2023-01-27 03:32 plan9-arm go@4e3abee2 os.TestDirFS (log)
2023-01-30 16:32 plan9-arm go@87860873 net/http.TestServerReadTimeout (log)
2023-01-30 20:09 plan9-arm go@45bee5e7 os.TestReadFile (log)
2023-01-31 19:43 plan9-arm go@b89a840d (log)
|
Found new dashboard test flakes for:
2023-02-02 23:42 plan9-arm tools@81111180 go@a9648761 x/tools/cmd/bundle.TestBundle (log)
|
Found new dashboard test flakes for:
2025-02-26 22:41 go1.24-plan9-arm release-branch.go1.24@5d692084 cmd/cover.TestCoverWithToolExec/CoverHTML (log)
|
Found new dashboard test flakes for:
2025-03-04 13:00 gotip-plan9-arm go@32fdcd7c net/http/httputil.TestReverseProxyWebSocketHalfTCP/server_close_read (log)
2025-03-04 13:00 gotip-plan9-arm go@32fdcd7c net/http/httputil.TestReverseProxyWebSocketHalfTCP/server_close_write [ABORT] (log)
2025-03-04 13:02 gotip-plan9-arm go@fd8938c7 net/http/httputil.TestReverseProxyWebSocketHalfTCP/server_close_read (log)
2025-03-04 13:02 gotip-plan9-arm go@fd8938c7 net/http/httputil.TestReverseProxyWebSocketHalfTCP/server_close_write [ABORT] (log)
2025-03-04 16:58 gotip-plan9-arm go@ba3c57fc net/http/httputil.TestReverseProxyWebSocketHalfTCP/server_close_read (log)
2025-03-04 16:58 gotip-plan9-arm go@ba3c57fc net/http/httputil.TestReverseProxyWebSocketHalfTCP/server_close_write [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@45aade7f x/tools/gopls/doc/generate (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/deprecated (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/embeddirective (log)
|
Found new dashboard test flakes for:
2025-02-26 22:41 go1.23-plan9-arm release-branch.go1.23@e4772831 cmd/link.TestLargeReloc (log)
2025-02-26 22:41 go1.23-plan9-arm release-branch.go1.23@e4772831 context.ExampleAfterFunc_connection [ABORT] (log)
2025-03-04 18:36 go1.23-plan9-arm release-branch.go1.23@25177ecd context.ExampleAfterFunc_connection [ABORT] (log)
2025-03-04 19:14 go1.23-plan9-arm release-branch.go1.23@c01c4d41 cmd/vet.TestVet/httpresponse (log)
2025-03-04 19:14 go1.23-plan9-arm release-branch.go1.23@c01c4d41 context.ExampleAfterFunc_connection [ABORT] (log)
2025-03-04 20:31 go1.23-plan9-arm release-branch.go1.23@45aade7f context.ExampleAfterFunc_connection [ABORT] (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@45aade7f x/tools/gopls/doc/generate (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/deprecated (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/embeddirective (log)
|
Found new dashboard test flakes for:
2025-03-04 19:48 x_tools-go1.23-plan9-arm tools@d81d6fcc release-branch.go1.23@45aade7f x/tools/gopls/doc/generate (log)
2025-03-04 19:48 x_tools-go1.23-plan9-arm tools@d81d6fcc release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/deprecated (log)
2025-03-04 19:48 x_tools-go1.23-plan9-arm tools@d81d6fcc release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/embeddirective (log)
|
Found new dashboard test flakes for:
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@c01c4d41 x/tools/gopls/doc/generate (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@c01c4d41 x/tools/gopls/internal/analysis/deprecated (log)
2025-03-04 21:25 x_tools-go1.23-plan9-arm tools@8d38122b release-branch.go1.23@c01c4d41 x/tools/gopls/internal/analysis/embeddirective (log)
|
Found new dashboard test flakes for:
2025-03-05 16:17 x_tools-go1.23-plan9-arm tools@340f21a4 release-branch.go1.23@45aade7f x/tools/godoc/vfs/mapfs (log)
2025-03-05 16:17 x_tools-go1.23-plan9-arm tools@340f21a4 release-branch.go1.23@45aade7f x/tools/godoc/vfs/zipfs (log)
2025-03-05 16:17 x_tools-go1.23-plan9-arm tools@340f21a4 release-branch.go1.23@45aade7f x/tools/gopls/internal/analysis/deprecated (log)
|
Found new dashboard test flakes for:
2025-03-05 16:17 x_tools-go1.24-plan9-arm tools@07219402 release-branch.go1.24@e4119e9b x/tools/go/callgraph/rta.TestRTA/testdata/func.txtar (log)
|
Found new dashboard test flakes for:
2025-03-04 20:31 go1.24-plan9-arm release-branch.go1.24@e4119e9b runtime:cpu2 (log)
|
Found new dashboard test flakes for:
2025-03-05 19:26 gotip-plan9-arm go@061efaa8 crypto/cipher:gofips140 (log)
2025-03-05 23:02 gotip-plan9-arm go@cad4dca5 crypto/des:gofips140 (log)
|
Found new dashboard test flakes for:
2025-03-05 19:59 x_tools-go1.24-plan9-arm tools@25a90bef release-branch.go1.24@0ace2d8a x/tools/cmd/stringer.TestMultifileStringer/test_package_over_x_test_package [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-05 20:18 x_tools-go1.23-plan9-arm tools@6a5b66be release-branch.go1.23@45aade7f x/tools/gopls/internal/cmd.TestCheck (log)
|
Found new dashboard test flakes for:
2025-03-06 21:40 gotip-plan9-arm go@5af3658e crypto/aes:gofips140 (log)
|
Found new dashboard test flakes for:
2025-03-06 21:35 gotip-plan9-arm go@584e6310 go/types.TestStdTest [ABORT] (log)
2025-03-06 21:35 gotip-plan9-arm go@584e6310 internal/trace.TestMMU [ABORT] (log)
2025-03-06 21:35 gotip-plan9-arm go@584e6310 net/http.Test304Responses [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-06 21:27 gotip-plan9-arm go@f55bb135 internal/coverage/cfile.TestCoverageSnapshot (log)
|
Found new dashboard test flakes for:
2025-03-06 16:21 gotip-plan9-arm go@938b6c15 go/internal/gcimporter.TestIssue25596 [ABORT] (log)
2025-03-06 16:21 gotip-plan9-arm go@938b6c15 go/types.TestFixedbugs/issue54942.go [ABORT] (log)
2025-03-06 16:21 gotip-plan9-arm go@938b6c15 internal/godebug.TestCmdBisect [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-05 20:18 gotip-plan9-arm go@f7204d76 cmd/api.TestIssue64958 [ABORT] (log)
2025-03-06 16:06 gotip-plan9-arm go@e7f9e760 crypto/cipher:gofips140 (log)
2025-03-06 18:34 x_tools-gotip-plan9-arm tools@b08c7a26 go@f7204d76 x/tools/gopls/internal/cmd.TestCheck [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-plan9-arm tools@b08c7a26 go@f7204d76 x/tools/gopls/internal/debug.TestTemplates [ABORT] (log)
2025-03-06 18:34 x_tools-gotip-plan9-arm tools@b08c7a26 go@f7204d76 x/tools/gopls/internal/filecache (log)
|
Found new dashboard test flakes for:
2025-03-06 18:34 x_tools-go1.24-plan9-arm tools@b08c7a26 release-branch.go1.24@0ace2d8a x/tools/cmd/callgraph.TestCallgraph [ABORT] (log)
2025-03-06 18:34 x_tools-go1.24-plan9-arm tools@b08c7a26 release-branch.go1.24@0ace2d8a x/tools/cmd/deadcode.Test [ABORT] (log)
2025-03-06 18:34 x_tools-go1.24-plan9-arm tools@b08c7a26 release-branch.go1.24@0ace2d8a x/tools/cmd/signature-fuzzer/fuzz-runner.TestRunner [ABORT] (log)
2025-03-07 17:40 x_tools-go1.23-plan9-arm tools@7435a814 release-branch.go1.23@45aade7f x/tools/gopls/internal/cmd.TestPrepareRename (log)
2025-03-07 17:40 x_tools-go1.24-plan9-arm tools@7435a814 release-branch.go1.24@0ace2d8a x/tools/gopls/internal/cmd.TestCodeAction (log)
|
Found new dashboard test flakes for:
2025-03-07 19:32 gotip-plan9-arm go@e0b110b9 cmd/api.TestIssue64958 [ABORT] (log)
2025-03-07 19:32 gotip-plan9-arm go@e0b110b9 cmd/compile/internal/types2.TestStdFixed [ABORT] (log)
2025-03-07 19:32 gotip-plan9-arm go@e0b110b9 cmd/link.TestCheckLinkname/badlinkname.go [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-07 19:31 gotip-plan9-arm go@812a44eb go/internal/gcimporter.TestImportedTypes [ABORT] (log)
2025-03-07 19:31 gotip-plan9-arm go@812a44eb go/types.TestCheck/decls2 [ABORT] (log)
2025-03-07 19:31 gotip-plan9-arm go@812a44eb image/png.TestWriteRGBA/Transparent_RGBA [ABORT] (log)
|
Found new dashboard test flakes for:
2025-03-05 18:21 gotip-plan9-arm go@932a4a4b archive/tar (log)
2025-03-05 18:21 gotip-plan9-arm go@932a4a4b archive/zip (log)
2025-03-05 18:21 gotip-plan9-arm go@932a4a4b bufio (log)
|
Found new dashboard test flakes for:
2025-03-05 16:17 x_tools-gotip-plan9-arm tools@07219402 go@bc5f4a55 x/tools/gopls/internal/cmd.TestCheck (log)
|
Found new dashboard test flakes for:
2025-03-04 11:02 gotip-plan9-arm go@82791889 runtime.TestDebugLogInterleaving (log)
|
Found new dashboard test flakes for:
2025-03-03 14:37 x_tools-gotip-plan9-arm tools@0efa5e51 go@bc5f4a55 x/tools/gopls/internal/cmd.TestWorkspaceSymbol (log)
|
Found new dashboard test flakes for:
2025-03-08 17:41 x_tools-gotip-plan9-arm tools@8fa586e1 go@26ba61df x/tools/gopls/internal/cmd.TestCodeAction (log)
|
Found new dashboard test flakes for:
2025-03-07 20:37 gotip-plan9-arm go@bc5f4a55 cmd/vet.TestVet/stdversion (log)
|
Found new dashboard test flakes for:
2025-03-08 17:41 x_tools-go1.23-plan9-arm tools@8fa586e1 release-branch.go1.23@45aade7f x/tools/gopls/internal/cmd.TestDefinition [ABORT] (log)
2025-03-10 11:26 x_tools-go1.23-plan9-arm tools@5a45ac2d release-branch.go1.23@45aade7f x/tools/cmd/stringer (log)
2025-03-10 11:26 x_tools-go1.23-plan9-arm tools@5a45ac2d release-branch.go1.23@45aade7f x/tools/gopls/internal/cmd.TestDefinition (log)
2025-03-10 16:39 x_tools-go1.23-plan9-arm tools@03f197e9 release-branch.go1.23@45aade7f x/tools/gopls/internal/cmd.TestCheck (log)
|
Found new dashboard test flakes for:
2025-03-10 21:20 gotip-plan9-arm go@574a9fa6 sync:cpu10 (log)
|
Found new dashboard test flakes for:
2025-03-10 21:02 gotip-plan9-arm go@895bcf17 crypto/cipher:gofips140 (log)
|
Found new dashboard test flakes for:
2025-03-11 12:44 gotip-plan9-arm go@ae4c13af runtime:cpu4.TestDebugLogInterleaving (log)
|
The one remaining
plan9
builder is failing a significant fraction of build attempts.Many of the failures follow specific known patterns (notably #49337, #46526, and #41952). However, many do not.
@millerresearch, @0intro, @fhs: is there someone who can investigate these failures and bring the
plan9
port back up to par? With the0intro
builders missing (#49327, #49328), there is no longer anyplan9
builder consistently passing tests.greplogs --dashboard -md -l -e \\Aplan9. -E . --since=2021-10-01
2021-11-04T01:44:54-d9bb5f6/plan9-arm
2021-11-03T23:48:28-283373d/plan9-arm
2021-11-03T21:45:33-4a4e1f2-d3f5dd5/plan9-arm
2021-11-03T21:45:33-2c98350-d3f5dd5/plan9-arm
2021-11-03T19:32:33-74f99d0/plan9-arm
2021-11-03T18:37:22-cfd016d/plan9-arm
2021-11-03T15:59:11-5f0e5d2/plan9-arm
2021-11-03T09:23:34-6458b2e/plan9-arm
2021-11-03T05:00:00-519c0a2/plan9-arm
2021-11-03T00:49:50-a4b2c57/plan9-arm
2021-11-03T00:21:24-8f0ca7d/plan9-arm
2021-11-03T00:07:38-49a0063/plan9-arm
2021-11-03T00:06:52-32d2752/plan9-arm
2021-11-02T23:44:58-a0f373c/plan9-arm
2021-11-02T23:05:17-714668c-eb63cbd/plan9-arm
2021-11-02T21:18:39-60fd3ed/plan9-arm
2021-11-02T20:59:34-79024cf/plan9-arm
2021-11-02T19:56:02-c45c32b/plan9-arm
2021-11-02T19:37:42-1011e26/plan9-arm
2021-11-02T18:35:29-433ba58/plan9-arm
2021-11-02T18:35:22-0b37bd9/plan9-arm
2021-11-02T18:31:53-79c5240/plan9-arm
2021-11-02T18:24:18-f7a95d2/plan9-arm
2021-11-02T15:33:01-4d06874-da7173a/plan9-arm
2021-11-02T13:43:24-ee08195-4d7bf41/plan9-arm
2021-11-02T13:43:24-4ff95d6-4d7bf41/plan9-arm
2021-11-02T06:25:39-4ff95d6-088bb4b/plan9-arm
2021-11-02T06:21:44-3ee426a/plan9-arm
2021-11-02T06:20:21-c1ea339/plan9-arm
2021-11-02T06:19:33-4633b2d/plan9-arm
2021-11-02T05:35:11-6f327f7/plan9-arm
2021-11-02T03:55:19-4ff95d6-6f1e9a9/plan9-arm
2021-11-02T03:54:24-3c61cb3/plan9-arm
2021-11-02T03:53:56-b0472aa/plan9-arm
2021-11-02T03:52:33-9268565/plan9-arm
2021-11-01T22:55:50-02e5913/plan9-arm
2021-11-01T19:42:30-4ff95d6-5f1bcd1/plan9-arm
2021-11-01T18:53:06-b5a5b7b/plan9-arm
2021-11-01T16:31:02-03971e3-2bcf1c0/plan9-arm
2021-11-01T15:55:25-e2e910e/plan9-arm
2021-11-01T15:55:20-d2b5121/plan9-arm
2021-11-01T14:45:28-82f902a/plan9-arm
2021-11-01T13:12:37-4056934/plan9-arm
2021-11-01T13:08:16-732db40/plan9-arm
2021-11-01T02:47:30-fde4cc2/plan9-arm
2021-10-31T18:39:05-89c5270/plan9-arm
2021-10-31T18:13:09-fd09e88/plan9-arm
2021-10-31T17:52:41-3fa9ee4/plan9-arm
2021-10-30T18:30:34-6113dac/plan9-arm
2021-10-30T16:46:47-d1dceaf/plan9-arm
2021-10-30T12:32:06-351c04c-5d6d9f5/plan9-arm
2021-10-29T23:25:18-5d6d9f5/plan9-arm
2021-10-29T22:29:31-c812b97/plan9-arm
2021-10-29T22:27:26-3571ab5/plan9-arm
2021-10-29T19:35:47-5ec139f/plan9-arm
2021-10-29T18:56:29-6afdf01/plan9-arm
2021-10-29T18:35:29-9da6415/plan9-arm
2021-10-29T18:23:03-413672f/plan9-arm
2021-10-29T17:44:52-353d5b6/plan9-arm
2021-10-29T04:21:07-103d89b-f288526/plan9-arm
2021-10-29T02:16:47-2c7cdec/plan9-arm
2021-10-29T02:11:42-33c392f/plan9-arm
2021-10-29T00:04:41-d6a9af8-af05d8b/plan9-arm
2021-10-28T20:50:34-d0751b1/plan9-arm
2021-10-28T20:50:34-103d89b-d0751b1/plan9-arm
2021-10-28T20:43:39-b954024/plan9-arm
2021-10-28T20:43:02-e741e2f/plan9-arm
2021-10-28T18:48:25-e7eb6f6-f229e70/plan9-arm
2021-10-28T18:10:59-ed475fe/plan9-arm
2021-10-28T18:10:50-2bc8ed8/plan9-arm
2021-10-28T17:57:57-2666262/plan9-arm
2021-10-28T17:52:59-3124968/plan9-arm
2021-10-28T02:35:22-056dfe6/plan9-arm
2021-10-28T01:15:26-b2fe2eb/plan9-arm
2021-10-28T00:11:24-a91d0b6/plan9-arm
2021-10-27T22:17:35-79ff663/plan9-arm
2021-10-27T21:37:54-749f6e9/plan9-arm
2021-10-27T21:35:48-a3c4ac0/plan9-arm
2021-10-27T21:34:20-4fefd43/plan9-arm
2021-10-27T20:25:06-bbc0595/plan9-arm
2021-10-27T20:24:34-56dcf97/plan9-arm
2021-10-27T20:03:17-68bd512/plan9-arm
2021-10-27T19:34:49-ffd2284/plan9-arm
2021-10-27T19:18:20-514ebae/plan9-arm
2021-10-27T18:24:50-259735f/plan9-amd64-0intro
2021-10-27T18:24:50-259735f/plan9-arm
2021-10-27T16:59:43-c0ac39c/plan9-arm
2021-10-27T13:12:49-cfb5321/plan9-arm
2021-10-27T08:50:27-e1e2965-bdefb77/plan9-arm
2021-10-27T08:50:27-bdefb77/plan9-arm
2021-10-26T23:12:17-13eccaa/plan9-arm
2021-10-26T22:05:53-80be4a4/plan9-386-0intro
2021-10-26T19:04:53-9626607-1dc77a3/plan9-arm
2021-10-26T19:04:53-03971e3-1dc77a3/plan9-arm
2021-10-26T18:33:39-244f92e-76cef81/plan9-386-0intro
2021-10-26T17:10:45-d418f37-76cef81/plan9-386-0intro
2021-10-26T17:10:45-76cef81/plan9-386-0intro
2021-10-26T17:10:45-76cef81/plan9-amd64-0intro
2021-10-26T16:00:26-94870a3-1b2362b/plan9-arm
2021-10-26T14:05:47-a2b8c18/plan9-arm
2021-10-26T11:58:05-1e2820a/plan9-arm
2021-10-26T05:05:24-23fdd7f/plan9-arm
2021-10-26T01:22:47-f916b54-adfb85b/plan9-arm
2021-10-26T01:22:47-22652b4-adfb85b/plan9-arm
2021-10-26T01:07:41-41f7407/plan9-arm
2021-10-25T23:28:03-f916b54-e9eb66d/plan9-arm
2021-10-25T23:28:03-e9eb66d/plan9-arm
2021-10-25T21:03:33-60c3069/plan9-arm
2021-10-25T18:46:45-f47335e/plan9-arm
2021-10-25T17:02:16-52b10ab/plan9-arm
2021-10-25T16:07:11-fc8b4ca-c6e82e5/plan9-arm
2021-10-23T20:23:35-c96bc14-8dbf3e9/plan9-386-0intro
2021-10-23T20:23:35-8dbf3e9/plan9-arm
2021-10-23T16:18:19-d418f37-16318df/plan9-386-0intro
2021-10-23T16:18:19-c96bc14-16318df/plan9-386-0intro
2021-10-23T16:18:19-c96bc14-16318df/plan9-amd64-0intro
2021-10-23T16:18:19-affba50-16318df/plan9-386-0intro
2021-10-23T16:18:19-16318df/plan9-386-0intro
2021-10-23T16:18:19-16318df/plan9-amd64-0intro
2021-10-23T16:18:19-16318df/plan9-arm
2021-10-23T16:18:15-12e4404/plan9-arm
2021-10-23T16:18:09-c526cf8/plan9-arm
2021-10-23T16:18:04-2d205eb/plan9-arm
2021-10-23T16:17:58-3cd28ba/plan9-arm
2021-10-23T12:44:47-b0f7eb6/plan9-arm
2021-10-22T21:48:34-ae4d67c/plan9-arm
2021-10-22T21:12:07-76dd01f/plan9-arm
2021-10-22T14:05:31-ff8115d/plan9-arm
2021-10-21T17:31:42-392bb06/plan9-arm
2021-10-21T17:23:58-02a3666/plan9-arm
2021-10-21T13:05:07-8b9c0d1/plan9-arm
2021-10-21T02:35:47-9b675d0-fca46d0/plan9-arm
2021-10-20T20:39:19-d419a80/plan9-arm
2021-10-20T20:38:59-75b73d6/plan9-arm
2021-10-20T20:38:47-3ec8d4b/plan9-arm
2021-10-20T19:30:13-cbf1d01-3ff39c5/plan9-arm
2021-10-20T19:30:13-3ff39c5/plan9-arm
2021-10-20T19:29:12-4320949/plan9-arm
2021-10-20T18:48:51-cbf1d01-4e565f7/plan9-arm
2021-10-20T05:28:45-b091189/plan9-arm
2021-10-19T23:27:41-4ea6123-2be5b84/plan9-arm
2021-10-19T22:47:35-9fa8551/plan9-arm
2021-10-19T18:56:08-07e5527/plan9-arm
2021-10-19T18:43:12-d944984/plan9-arm
2021-10-19T18:43:08-1d63052/plan9-arm
2021-10-19T18:22:59-6c3cd5d/plan9-arm
2021-10-19T18:18:37-9820602/plan9-arm
2021-10-19T17:12:53-404f84d/plan9-arm
2021-10-19T17:01:42-3a07ab7/plan9-arm
2021-10-19T16:19:27-98f6e03-f92a358/plan9-arm
2021-10-19T08:44:22-1e49210/plan9-arm
2021-10-19T07:45:46-281050f-ee92daa/plan9-arm
2021-10-18T22:56:07-eba0e86/plan9-arm
2021-10-18T22:55:05-7397178/plan9-arm
2021-10-18T19:46:27-98f6e03-394a1ad/plan9-arm
2021-10-18T16:32:38-543a513/plan9-arm
2021-10-18T15:38:40-74acbaf/plan9-arm
2021-10-16T16:27:40-c96bc14-640a49b/plan9-386-0intro
2021-10-16T16:27:40-98f6e03-640a49b/plan9-arm
2021-10-16T16:27:40-640a49b/plan9-arm
2021-10-16T16:27:40-4f30a5c-640a49b/plan9-386-0intro
2021-10-16T14:50:21-c96bc14-680caf1/plan9-386-0intro
2021-10-16T14:50:21-98f6e03-680caf1/plan9-386-0intro
2021-10-16T14:50:21-4f30a5c-680caf1/plan9-386-0intro
2021-10-15T21:46:06-ed1c8db/plan9-386-0intro
2021-10-15T21:46:06-ed1c8db/plan9-amd64-0intro
2021-10-15T21:46:06-c96bc14-ed1c8db/plan9-amd64-0intro
2021-10-15T21:46:06-4f30a5c-ed1c8db/plan9-amd64-0intro
2021-10-15T21:44:07-4a7975e/plan9-386-0intro
2021-10-15T20:29:48-85cbdda/plan9-arm
2021-10-15T20:29:36-a17b2e8/plan9-386-0intro
2021-10-15T19:54:44-22951fb/plan9-arm
2021-10-15T19:03:33-98f6e03-cfe6763/plan9-arm
2021-10-15T18:07:49-98f6e03-8331f25/plan9-arm
2021-10-15T02:45:22-288d0b3-3da0ff8/plan9-arm
2021-10-15T02:02:50-ce04ca3-3da0ff8/plan9-arm
2021-10-15T02:02:50-3da0ff8/plan9-386-0intro
2021-10-15T00:16:20-ce04ca3-9e8ed86/plan9-arm
2021-10-14T23:32:09-3a269dc-9e8ed86/plan9-arm
2021-10-14T22:15:24-281050f-b59467e/plan9-arm
2021-10-14T18:44:37-a37bebc/plan9-arm
2021-10-14T18:44:29-a8c5a99/plan9-arm
2021-10-14T15:17:03-fbdf830/plan9-arm
2021-10-14T13:09:28-88e80df-24e798e/plan9-arm
2021-10-14T13:09:28-24e798e/plan9-arm
2021-10-14T12:38:31-88e80df-2feb2cc/plan9-arm
2021-10-14T07:18:59-1349c6e/plan9-arm
2021-10-14T04:18:44-acac939-9e4dc6f/plan9-arm
2021-10-14T01:51:22-276fb27/plan9-arm
2021-10-13T22:21:08-ac2ed98-b90d258/plan9-arm
2021-10-13T20:31:25-b90d258/plan9-arm
2021-10-13T18:15:48-40f82f8/plan9-amd64-0intro
2021-10-13T18:15:48-40f82f8/plan9-arm
2021-10-13T18:15:48-281050f-40f82f8/plan9-arm
2021-10-13T17:02:43-b5904f3/plan9-arm
2021-10-13T16:39:23-3986e5c/plan9-arm
2021-10-13T16:36:59-69041c7/plan9-arm
2021-10-13T15:11:16-0454d73/plan9-arm
2021-10-13T03:12:54-24389d4-4fb2e1c/plan9-arm
2021-10-12T22:02:39-732f6fa/plan9-arm
2021-10-12T21:30:45-0b930fb-3283d1a/plan9-arm
2021-10-12T21:15:50-3d051ba/plan9-arm
2021-10-12T20:53:52-ac6d706/plan9-arm
2021-10-12T20:20:41-d032b2b/plan9-arm
2021-10-12T18:30:24-281050f-6e0adde/plan9-arm
2021-10-12T12:24:09-c8768b9-4679670/plan9-arm
2021-10-12T06:55:50-c83725b-d887d3b/plan9-arm
2021-10-11T20:46:14-7023535/plan9-arm
2021-10-11T20:46:14-18fa840-7023535/plan9-arm
2021-10-11T19:20:12-18fa840-65ffee6/plan9-arm
2021-10-11T15:48:29-2478b6f/plan9-arm
2021-10-11T12:53:11-3b03d30-34f7b1f/plan9-386-0intro
2021-10-11T12:53:11-34f7b1f/plan9-arm
2021-10-11T12:42:06-59b2f51/plan9-arm
2021-10-10T15:11:26-d2d21d9/plan9-386-0intro
2021-10-10T15:11:26-d2d21d9/plan9-amd64-0intro
2021-10-10T15:11:26-d2d21d9/plan9-arm
2021-10-10T15:11:26-3b03d30-d2d21d9/plan9-386-0intro
2021-10-10T15:11:26-1af23bd-d2d21d9/plan9-386-0intro
2021-10-10T13:37:52-1af23bd-b99abf3/plan9-arm
2021-10-09T23:29:26-b99abf3/plan9-386-0intro
2021-10-09T23:29:26-b99abf3/plan9-arm
2021-10-08T22:21:41-ee04797-74abcab/plan9-arm
2021-10-08T19:41:16-c5188f2-7fcf9a1/plan9-arm
2021-10-08T19:41:16-7fcf9a1/plan9-386-0intro
2021-10-08T19:30:58-f430bda/plan9-arm
2021-10-08T19:19:09-b9e1e1b/plan9-386-0intro
2021-10-08T18:07:25-d480b5c/plan9-386-0intro
2021-10-08T17:58:41-16a3cef/plan9-arm
2021-10-08T17:32:31-d7ba1d2/plan9-386-0intro
2021-10-08T17:32:25-a7d3a0e/plan9-arm
2021-10-08T15:27:20-b3a4b2d/plan9-arm
2021-10-08T14:08:12-5b9206f/plan9-386-0intro
2021-10-08T14:08:12-5b9206f/plan9-arm
2021-10-07T22:19:58-1ceb723/plan9-arm
2021-10-07T21:39:17-6b4cf2b/plan9-arm
2021-10-07T20:37:43-d477ef3-ef2ebbe/plan9-arm
2021-10-07T19:56:19-4ab3c10/plan9-arm
2021-10-07T18:58:33-7286502/plan9-arm
2021-10-07T18:09:53-4607ebc/plan9-arm
2021-10-07T16:38:24-be571a3/plan9-arm
2021-10-07T14:37:45-ebeab63/plan9-arm
2021-10-07T14:37:36-375a1fb/plan9-arm
2021-10-07T02:30:47-2c98350-39bbf08/plan9-arm
2021-10-06T23:26:57-c8dd89e/plan9-arm
2021-10-06T23:26:57-86761ae-c8dd89e/plan9-arm
2021-10-06T21:17:08-2e107b4/plan9-arm
2021-10-06T21:06:48-197b328/plan9-arm
2021-10-06T15:53:04-4d8db00/plan9-arm
2021-10-05T23:58:29-ac60900/plan9-arm
2021-10-05T23:36:43-e82ed0c/plan9-arm
2021-10-05T22:42:11-96fface/plan9-arm
2021-10-05T20:52:58-0d6561b/plan9-arm
2021-10-05T20:35:54-990c9c6/plan9-arm
2021-10-05T19:15:03-ae83301/plan9-arm
2021-10-05T18:31:14-5d946f1/plan9-arm
2021-10-05T17:10:45-2dc2755-5140ad1/plan9-arm
2021-10-05T15:47:57-017ffcd/plan9-arm
2021-10-05T15:47:07-cfd74f7/plan9-arm
2021-10-05T01:30:56-d4b1ae0-123393a/plan9-386-0intro
2021-10-04T22:59:44-78c2529/plan9-arm
2021-10-04T22:59:13-199ec42/plan9-arm
2021-10-04T22:54:58-7c79e8e/plan9-arm
2021-10-04T22:00:53-d9952ff/plan9-386-0intro
2021-10-04T22:00:19-739328c/plan9-arm
2021-10-04T21:10:16-db89b5a-f0db7ea/plan9-arm
2021-10-04T18:15:09-ccaa907-9f8d558/plan9-arm
2021-10-04T17:17:11-cedda3a-cc5e3de/plan9-386-0intro
2021-10-04T17:17:11-ccaa907-cc5e3de/plan9-386-0intro
2021-10-04T17:17:11-cc5e3de/plan9-amd64-0intro
2021-10-04T15:50:43-9432320/plan9-386-0intro
2021-10-04T15:50:43-9432320/plan9-amd64-0intro
2021-10-02T16:05:55-0ebff1a-a7fe161/plan9-arm
2021-10-02T13:15:42-0ebff1a-7bf7bbc/plan9-arm
2021-10-01T23:51:16-c129af9/plan9-arm
2021-10-01T21:26:43-0ebff1a-0d65c27/plan9-arm
2021-10-01T19:30:53-13c407c-0d65c27/plan9-arm
2021-10-01T18:18:46-0d65c27/plan9-arm
2021-10-01T17:18:20-5279e53/plan9-arm
2021-10-01T16:45:09-58ad7eb-f19b2d5/plan9-arm
2021-10-01T16:17:08-f19b2d5/plan9-arm
2021-10-01T16:01:09-558b9da-8d494b0/plan9-arm
2021-10-01T15:59:38-e5ad363/plan9-arm
2021-10-01T14:06:36-96d3ba8/plan9-arm
The text was updated successfully, but these errors were encountered: