trafficserver: new module

[midchildan]

Creates a new module for Apache Traffic Server.

[sandydoo]

The test run doesn't seem to exit: https://github.com/cachix/devenv/actions/runs/9290213675/job/25603905574?pr=1243

[midchildan]

I couldn't reproduce the failure locally on x86_64 Linux. The CI output has multiple lines with [trafficserver ] NOTE: using command line path as RUNROOT, which suggests that Traffic Server crashed and restarted. If that is the case, I don't know why curl didn't immediately error out.

To help with debugging, I added a timeout for curl and made the console logs more verbose.

[midchildan]

Hmm, I still can't figure out the cause of the failure. I ran the test on GitHub Actions using my repository, and the Linux runner was able to complete the test successfully.

https://github.com/midchildan/devenv/actions/runs/9339577654/job/25704130159

To make things worse, the complete logs are still missing likely because it's output too late. When I run it locally, the logs appear after the shell prompt. Maybe a different setting might be able to surface the logs, I'll look into it.

[domenkozar]

Maybe something similar to #1248 (comment)

[midchildan]

Thanks, I set the hostname for ATS and also tried to get more logs. If this doesn't go well, I might have to temporarily wrap the traffic_server command with strace to get more details.

[sandydoo]

Huh, maybe it doesn't like one of the systemd restrictions on our runner: https://github.com/NixOS/nixpkgs/blob/051f920625ab5aabe37c920346e3e69d7d34400e/nixos/modules/services/continuous-integration/github-runner/service.nix#L227-L265

[Jun 10 10:04:22.158] traffic_manager NOTE: [LocalManager::pollMgmtProcessServer] Server Process terminated due to Sig 31: Bad system call

[midchildan]

I tried applying the same restrictions on NixOS 23.11 with the default kernel, but the test still completed successfully.

sudo systemd-run -t -p AmbientCapabilities= -p CapabilityBoundingSet= -p DeviceAllow= -p NoNewPrivileges=true -p PrivateDevices=true -p PrivateMounts=true -p PrivateTmp=true -p ProtectClock=true -p ProtectControlGroups=true -p ProtectHome=true -p ProtectHostname=true -p ProtectKernelLogs=true -p ProtectKernelModules=true -p ProtectKernelTunables=true -p ProtectSystem=strict -p RemoveIPC=true -p RestrictNamespaces=true -p RestrictRealtime=true -p RestrictSUIDSGID=true -p UMask=0066 -p ProtectProc=invisible -p SystemCallFilter='~@clock ~@cpu-emulation ~@module ~@obsolete ~@raw-io ~@reboot ~capset ~setdomainname ~sethostname' -p RestrictAddressFamilies='AF_INET AF_INET6 AF_UNIX AF_NETLINK' -p DynamicUser=true -p PrivateNetwork=false -p MemoryDenyWriteExecute=false -p ProcSubset=all -p LockPersonality=false -p StateDirectory=devenv -p Environment=HOME=/var/lib/devenv zsh

[midchildan]

I pushed a new commit that wraps the traffic_server command with strace to see which system call is causing the crash.

[midchildan]

It appears Traffic Server is requesting CAP_NET_ADMIN and getting itself terminated by the kernel.

In a typical setup, Traffic Server is launched as root. During startup, the server process changes to an unprivileged user and further drops unnecessary capabilities. But in doing so, it also attempts to explicitly keep the required capabilities including CAP_NET_ADMIN.

https://github.com/apache/trafficserver/blob/90fbf13db0858cef0e0a094f445d846b60a4c1ef/src/tscore/ink_cap.cc#L259

On other machines, this appears to have no effect when Traffic Server is launched as an unprivileged user. So perhaps the CI host is configured differently?

In any case, I configured Traffic Server to not attempt to change the user during launch. This should prevent the problematic code from running.

https://github.com/apache/trafficserver/blob/90fbf13db0858cef0e0a094f445d846b60a4c1ef/src/traffic_server/traffic_server.cc#L1874

[midchildan]

It appears the tests hanged before the Traffic Sever tests were able to run on the Linux runners. I pushed another commit to temporarily disable other examples so that we can see the results faster.

[sandydoo]

I brought down our CI trying to upgrade NixOS. Will re-run once I get it back up, but the fix looks promising!

[midchildan]

error: The option `services.trafficserver' does not exist. Definition values:

That's interesting, I don't see any change since last time that would cause the option to stop being defined.

[midchildan]

I tried rebasing. .github/workflows/buildtest.yml did have a conflict as a result of me temporarily disabling non-Traffic Server tests. I'm not sure if it has anything to do with the error though. The test still succeeds on my end.

[domenkozar]

Seems like the tests pass now?

[sandydoo]

Yay! Thanks for being so patient, @midchildan. If you could please drop the debug commits, and we'll merge straight away.

[midchildan]

Thank you too! I dropped e8fbe42 and ec9b9fe.