sled agent fails on startup looking for non-existent config-rss.toml

[davepacheco]

I believe that right now, sled agent on most real systems will fail on startup with:

[ Sep 19 15:47:38 Method "start" exited with status 0. ]
sled-agent: Failed to read config from /opt/oxide/sled-agent/pkg/config-rss.toml: No such file or directory (os error 2)

Caused by:
    No such file or directory (os error 2)
[ Sep 19 15:47:38 Stopping because all processes in service exited. ]
[ Sep 19 15:47:38 Executing stop method (:kill). ]
[ Sep 19 15:47:38 Restarting too quickly, changing state to maintenance. ]

Details coming. I think this was introduced yesterday in #9016 here:
https://github.com/oxidecomputer/omicron/pull/9016/files#diff-47887367c1e0816a1e9405c9e748d167c55abd5743fd02c55246ff0bd76e9ff9

Per @jgallagher, that then_some wants to be using then with a closure. As written, even if the file doesn't exist, it attempts to parse it and then bails out with the error.

[davepacheco]

For a little more context: config-rss.toml is shipped in some deployment scenarios in order to cause sled agent to start RSS on its own with the configured file. This happens in these two contexts:

• single-system Omicron deploy with a real sled agent and softnpu, following the how-to-run instructions (this is what we test in CI in the helios-deploy job)
• in a4x2, on sled g0 only (not g1-g3)

Everywhere else, we don't ship this file. These are the contexts where sled agent will bail out on startup right now. That includes all racklettes and racks (including dogfood, etc.).

[dancrossnyc]

Regression introduced in #9016, where I replaced an explicit if expression creating an Option<RackInitializeRequest with a call to .then_some. However, .then_some evaluates its argument eagerly, as Rust uses strict argument evaluation. The block given as the .then_some argument tries to open a file, and returns the Err variant of a Result if that fails, via the ? operator. However, that file may not exist; the intent was to only open the file if that was true, but since the argument is eagerly evaluated, it tried even if the file didn't exist, which caused this error.

I thought to use .then, which takes a closure, so is lazily evaluated, only if the file does exist (John also suggested this). But you cannot (conveniently) use ? in such a closure. So I changed it back to an explicit if expression in #9053, which should submit as soon as CI completes.

It would be helpful if we had some CI that tests the case where config-rss.toml does not exist, however.

[davepacheco]

Thanks for the quick fix!

It would be helpful if we had some CI that tests the case where config-rss.toml does not exist, however.

I think a4x2 would be our best bet for that. #7424 was close, but never landed.