Sous Chefs Adoption

[ramereth]

This includes a variety of fixes that are needed to get the suites to pass properly.

[kitchen-porter]

	1 Warning
⚠️	This is a big Pull Request.

Generated by 🚫 Danger

[nuclearsandwich]

I started working in a side branch. @ramereth did you mention some oddness going on with plugin installation? My local kitchen runner seems to become unable to install plugins. I haven't figured out if this is a new issue uncovered by getting farther in the tests or if I'm hitting a rate limit on the jenkins update center.

I started a side branch https://github.com/sous-chefs/jenkins/tree/smoke-out-the-smoke-tests with two small changes that move us slightly forward.

[ramereth]

I started working in a side branch. @ramereth did you mention some oddness going on with plugin installation? My local kitchen runner seems to become unable to install plugins. I haven't figured out if this is a new issue uncovered by getting farther in the tests or if I'm hitting a rate limit on the jenkins update center.

What kind of issues are you running into? I believe 8d6b224 fixed one issue I had with permissions.

[nuclearsandwich]

What kind of issues are you running into?

Trying to install any plugin that isn't already installed (the breaking failure is in the checks-api plugin but if I try to install others via jenkins-cli after a kitchen login I get the same issue times out fetching the plugin from update center.

I've just triggered a fresh build to see if the same problem occurs in the same place (otherwise I'd suspect some kind of rate-limiting).

[nuclearsandwich]

Ran it again and got the same issue fresh so it isn't rate limiting. Now it's no longer Friday evening I'm realizing that it's quite possibly #762

[ramereth]

@nuclearsandwich at this point I think this PR should be finished and exclude any suites/tests that aren't currently working and try and fix them in a later PR.

[nuclearsandwich]

The remaining failing test is reliably producing a segfault in dokken (on the GitHub Action) and vagrant (for me locally).

The error we see in the cookbook log is a connection refusal when connecting to Jenkins as the process has crashed and is not listening on port 8080.

As a little refresher, Jenkins installed via the package on Ubuntu 20.04 installs an init script rather than a systemd unit. So far I can reproduce the segfault by restarting Jenkins via systemctl restart jenkins but I have not been able to cause it when running the same Jenkins command via kitchen login in the same environment although I just realized I was running it as root not as Jenkins so I'll try to improve my reproduction of the issue and figure out what high level operation is triggering the segfault. I also haven't tried to capture and investigate the core dump which is another avenue to try.

[nuclearsandwich]

I can reproduce it starting Jenkins via /usr/bin/daemon with

/usr/bin/daemon --core -f --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins -- /bin/java -Djenkins.install.runSetupWizard=false -jar /usr/share/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpListenAddress=0.0.0.0 --httpPort=8080 --ajp13Port=-1 --httpKeepAliveTimeout=300000

and then running

 java -jar /tmp/kitchen/cache/jenkins-cli.jar -s 'http://localhost:8080' -auth vagrant:vagrant who-am-i

[nuclearsandwich]

I haven't gotten any further in debugging the segfault but happened upon some information when trying to get more info out of the core dump. The java installation inside the kitchen environment is based on OpenJ9 which as far as I know is not fully tested for running Jenkins.

# java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-b08)
Eclipse OpenJ9 VM (build openj9-0.18.1, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20200122_511 (JIT enabled, AOT enabled)
OpenJ9   - 51a5857d2
OMR      - 7a1b0239a
JCL      - 8cf8a30581 based on jdk8u242-b08)

I discovered this when installing openjdk-8-dbg thinking it would get me debug symbols for my /bin/java and instead I got a /bin/java that no longer segfaults.

I've pushed b2f4768 which switches the fixture cookbook JVM installation to hotspot, which I think is the correct approach even though it affects all recipes because I'm not aware of any official stance on OpenJ9 for the Jenkins project and issues with OpenJ9 are best troubleshooted elsewhere.

[nuclearsandwich]

Me watching CI:

[nuclearsandwich]

I made the most recent commits so extra eyes on b2f4768 and 17af8d6 probably isn't a bad idea.

[xorima]

LGTM

[kitchen-porter]

Released as: 8.2.0