x/build/builders: tracking bug for new-style SmartOS Solaris builder

[bradfitz]

This is a tracking bug for getting a new-style Solaris builder running, using the buildlet.

[4ad]

Unfortunately illumos (open source Solaris) lacks the drivers required to run on GCE. I will look into the feasibility of implementing the missing drivers myself (mostly a question of lacking enough time).

[bradfitz]

Per email conversation, feel free to do the same thing as #9495, building an API server we can hit from the coordinator to start & kill containers/VMs/zones/whatever on your side (Manta?).

[bradfitz]

I signed up for Joyent in November and forgot about it.

(Note to self: details in my email at https://mail.google.com/mail/u/0/#inbox/151260db32b1d0b2 )

[bradfitz]

I'm finally running a Solaris buildlet (the "new"-style builder) on Joyent.

A single one has been running for a few days now, doing the reverse buildlet thing, keeping a constant connection open to farmer.golang.org. You can see it in the section:

Reverse pool summary:
...
solaris-amd64-smartosbuildlet: 1/1 (1 + 0 other)

(It also shows up at http://farmer.golang.org/builders)

Now that I know one works, it'd be nice to have a dynamic number, so we can scale from 0 to dozens quickly, and have Solaris trybots and sharded tests.

Anybody want to help make this better?

I think it would involve making a custom image that runs the buildlet on boot as a service (SMF? some XML thing?), and then we'd need to do some Joyent API calls from farmer.golang.org to dynamically size the number of running images? I can do that part, if you write standalone Go code to resize the number of instances of an image. (or just stop/start them?)

/cc @4ad @bsiegert

[4ad]

What do you mean by a "custom image"?

[bradfitz]

@4ad, maybe my terminology is wrong, but I want to be able to make an API call to Joyent where I say "start a new buildlet" and it starts a container or VM and it starts the buildlet on boot.

I don't care whether it's a container or a VM, but I'd prefer it to start within seconds, so that probably means a container.

The only thing custom about it some service configuration so the buildlet starts on boot, and the buildlet's stage0 binary can be baked in to the image. The other thing we could bake in to the image is the GOROOT_BOOTSTRAP directory, so we don't have to download it & untar it for each build.

[4ad]

If we need to create a custom image, the only to create it is going through this route: https://docs.joyent.com/public-cloud/instances/infrastructure/images/creating

[4ad]

Also, since we have access to 32-core containers, is there value in doing sharded tests?

[bradfitz]

Maybe not. Or maybe it'll help more (we could use N 32-core containers). Note that the builder doesn't just do static sharding, giving 1/Nth of work to each machine. It instead has a queue of work to do, and each machine grabs work from the queue. So it effectively keeps all the machines busy, even if the work is very heterogeneously intensive.

They bill by the minute, right? Is there a 10 minute minimum or anything?

[bradfitz]

@4ad, can you point me at simple instructions for running a program on boot? I got lost in some hundred page PDF about SMF and XML. Is that the systemd unit file equivalent? Is there something simpler like the rc.local equivalent?

[4ad]

Yes, SMF is like systemd, and yes, service files are unfortunately XML. On Oracle Solaris there's a svcprop(1) thing that makes this easy, but it's not available for illumos, which is what we're using here.

There is no rc.local.

The best SMF tutorial is probably: http://www.c0t0d0s0.org/archives/4149-Solaris-Features-Service-Management-Facility.html

You can probably just adapt what I use here: https://github.com/4ad/mgk.ro/blob/master/cmd/gometad/gometad.xml

Install with:

svccfg validate gometad.xml
pfexec svccfg import gometad.xml

[4ad]

I think most instances are billed by the minute, but some are billed by the hour, we are using standard triton images, no? AFAIK those are billed by the minute with no minimum order (but we should recheck).

[bradfitz]

Thanks!

I'm using the "base64" image. My instance is called "gotriton1" so yes, Triton, even though I forget the other choices. (I'm always confused whether Triton means the Linux system call compatibility layer, or something else.)

[4ad]

I'm always confused whether Triton means the Linux system call compatibility layer, or something else.

Don't worry, I'm always confused about that as well.

[jtsylve]

I've got a WIP install script here for comment: https://go-review.googlesource.com/#/c/21791

[gopherbot]

CL https://golang.org/cl/21791 mentions this issue.

[zombiezen]

This is now running. It's a bit low capacity at the moment, but it can be reasonably scaled up now.