Allow multiple firecracker test shards to run concurrently on a single machine

[bduffany]

Follow-up from #7429

• Add optional flock-based IP range locking to prevent concurrent test shards from trying to use the same IP range concurrently.
  • This is useful for firecracker tests, as well as running multiple firecracker executors locally.
• Don't try to clean up network namespaces in tests, because they might be in use.
  • I don't expect that too many of these will accumulate, but if it does happen, we can find a way to clean up "orphaned" namespaces, maybe by associating them with lockfiles.
• Don't try to clean up the shared executor root directory in tests either, to prevent converted ext4 image files from disappearing mid-test.
  • Added a TODO to provision these images as data deps of the test instead of converting during the test.
• Set shard_count on firecracker_test. (This won't actually buy us a ton of concurrency until we disable exclusive task scheduling on the bare pool.)

Related issues: https://github.com/buildbuddy-io/buildbuddy-internal/issues/3758

[bduffany]

I got rid of this hack because it broke networking on my local machine (because the firewall rules were only being added to the legacy tables rather than nftables). It seems like this is no longer needed for our networking tests to pass, maybe because we updated our firecracker kernel config.

[maggie-lou]

Sanity check - if path already exists (because another process already holds the lock), will os.Create return the same file descriptor?

[bduffany]

it'll return a different file descriptor (file descriptors are local to a process) but the returned descriptor will point to the same underlying file (inode), so flock should work here. (I also checked this with some debug logging)

[maggie-lou]

LGTM - but maybe check with Iain because I remember he had some thoughts/alternate ideas for other ways to implement this

[bduffany]

LGTM - but maybe check with Iain because I remember he had some thoughts/alternate ideas for other ways to implement this

Sure - for reference, I dug up the thread: https://buildbuddy-corp.slack.com/archives/C04R9V7H28N/p1697734759336109

A couple of thoughts after reading through it:

• We talked about tying namespaces to IP addresses by using a naming convention like /var/run/netns/bb-executor-192-168-XXX-XXX, but in the new world of network pooling, IP addresses and net namespaces are no longer 1-1 (because we unassign/reassign IPs when adding to/taking from the pool, but keep the net namespace around). So, I think the problem is now mostly scoped to just locking IP ranges, and net namespaces are not really as relevant.
• Towards the end of the thread, we had mostly settled on the flock approach, but we talked about possibly using a library for flocking or rolling our own. I think the current implementation of just making a flock() system call is simple enough that pulling in a dependency is not warranted, and it also doesn't seem worth spending time rolling our own. For cross-process coordination that requires more complexity than just acquiring a lock (e.g. storing data), I think sqlite is probably a better solution - e.g. podman uses sqlite to maintain container state without needing a daemon process.