macos-13 job crashes without logs

[AkihiroSuda]

Description

macos-13 job crashes after (roughly) 30 minutes, without logs

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• Windows Server 2019
• Windows Server 2022

Image version and build link

Image: macos-13
Version: 20230419.1

https://github.com/lima-vm/lima/actions/runs/4799814791/jobs/8596609974?pr=1511

Is it regression?

Yes (macos-12 → macos-13)

Expected behavior

Should not crash without logs

Actual behavior

Crashes after (roughly) 30 minutes without logs

Repro steps

Run a job with macos-13 runner

[ilia-shipitsin]

interesting. I'll have a look

[ilia-shipitsin]

@AkihiroSuda , I tried to run in my fork

seems, it awaits for

time="2023-04-28T12:37:24Z" level=info msg="[hostagent] 2023/04/28 12:37:24 tcpproxy: for incoming conn 127.0.0.1:49278, error dialing \"192.168.5.15:22\": connect tcp 192.168.5.15:22: no route to host"
time="2023-04-28T12:37:34Z" level=info msg="[hostagent] Waiting for the essential requirement 1 of 3: \"ssh\""
time="2023-04-28T12:37:37Z" level=info msg="[hostagent] 2023/04/28 12:37:37 tcpproxy: for incoming conn 127.0.0.1:49279, error dialing \"192.168.5.15:22\": connect tcp 192.168.5.15:22: no route to host"
time="2023-04-28T12:37:47Z" level=info msg="[hostagent] Waiting for the essential requirement 1 of 3: \"ssh\""
time="2023-04-28T12:37:51Z" level=info msg="[hostagent] 2023/04/28 12:37:51 tcpproxy: for incoming conn 127.0.0.1:49280, error dialing \"192.168.5.15:22\": connect tcp 192.168.5.15:22: no route to host"

what's that ? how that routing is supposed to appear ? where it tries to connect to ? is it some qemu vm started on runner ?

[AkihiroSuda]

@ilia-shipitsin Thanks for testing.

• How did you get that log? Is that timeout relevant to the 30-min timeout of the GHA job?
• 192.168.5.15 is a virtual IP that is not accessible from the host. This is an equivalent of 10.0.2.15 of QEMU and Virtualbox.
  Lima uses a Unix socket to dial that virtual IP, but it looks like the virtual network stack is somehow not functional.
  I'm not sure whether that virtual network failure is relevant to the 30-min timeout of the GHA job.
• The "vz" mode uses Apple's Virtualization.framework, not QEMU.

[ilia-shipitsin]

I cloned your repo, removed all build definitions except "vz".

30 min, I guess, defined here ...

[AkihiroSuda]

AFAICS that's the timeout of the step, not the job?

[AkihiroSuda]

Removed nick-invision/retry@v2, and the job is still failing without logs
https://github.com/lima-vm/lima/actions/runs/4832217890/jobs/8610695152?pr=1511

[ilia-shipitsin]

@AkihiroSuda , I see better logs with the following invocation

    - name: Test
      run: |
        ./hack/test-example.sh examples/experimental/vz.yaml

[ilia-shipitsin]

previously I removed 30 min limit and job fails after 48 min (no logs caught)

[ilia-shipitsin]

interesting observation, when build is running, I can see logs.
when build is complete, there are no logs

[ilia-shipitsin]

@AkihiroSuda , I tried to run commands on fresh macos-13 vm

INFO[0000] Terminal is not available, proceeding without opening an editor
WARN[0000] `vmType: vz` is experimental
INFO[0000] Attempting to download the image              arch=x86_64 digest= location="https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
INFO[0002] Using cache "/Users/runner/Library/Caches/lima/download/by-url-sha256/6b15519b255a45a238b7a8154cd57da120344ea388143af2821bb790af7fc587/data"
INFO[0006] Attempting to download the nerdctl archive    arch=x86_64 digest="sha256:955f9a4853762b1258cd38c967e45b6061a181a668907059e56cc01c32f1cf21" location="https://github.com/containerd/nerdctl/releases/download/v1.3.1/nerdctl-full-1.3.1-linux-amd64.tar.gz"
INFO[0006] Using cache "/Users/runner/Library/Caches/lima/download/by-url-sha256/ced4c1bbc347f1a74f9f9f25172cdb69115d21bf84000cc529013917a24a067a/data"
INFO[0006] [hostagent] Replacing "ftp_proxy" value "http://localhost:2121" with "http://192.168.5.2:2121"
INFO[0008] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/runner/.lima/vz/serial.log")
INFO[0008] [hostagent] Setting up Rosetta share
WARN[0008] [hostagent] Unable to configure Rosetta: Rosetta is unsupported on non-ARM64 hosts

after that communication to vm was lost (no idea why).
is there a way to make limactl more verbose ?

file /Users/runner/.lima/vz/serial.log is empty

[AkihiroSuda]

is there a way to make limactl more verbose ?

limactl --debug may print more verbose logs.
Also, you may find some errors in ~/.lima/vz/ha.{stdout,stderr}.log.

But I'm not sure if that is relevant to my OP. (The problem is that I can't see the failure log)

[ilia-shipitsin]

yes, I agree that having logs sounds good.

as I discussed with colleagues, there are some edge conditions in Github Actions when logs are lost. It is bad, but we cannot fix it within "runner-images" repo. I'll ask what is proper way for tracking that issue

logs are only lost when task is completed. while it is in progress, logs are available (if that helps in investigating your issue)

[ilia-shipitsin]

@AkihiroSuda , may I ask you to open an issue at https://support.github.com/ ?

I was told by my collegue that it is proper way for customer support.
also, I've searched for similar issues there, surprisingly, no luck

actually, we had several similar issues

#7004
#3517
#736
#7188
#6378
#6350

all were closed as "it is something with resource consumption". I'm not sure that resource overconsumtion should lead us to logs "completely" lost. for me, it would be fine to keep at least those logs avaiable during run

[AkihiroSuda]

@AkihiroSuda , may I ask you to open an issue at https://support.github.com/ ?

Sure: https://support.github.com/ticket/2146507

[ilia-shipitsin]

@AkihiroSuda , I contacted colleguaes, they followed up escalated issue.
I'm closing this one.

feel free to reopen if needed

[AkihiroSuda]

they followed up escalated issue.

Yes, but the issue is still unresolved AFAICS
https://support.github.com/ticket/2146507

[ilia-shipitsin]

It is not something I have an access to ((

if you think it makes sense, we can keep this issue open

[AkihiroSuda]

if you think it makes sense, we can keep this issue open

Yes, that would be appreciated, thanks

[ilia-shipitsin]

Hello, @AkihiroSuda !

did you receive some update in https://support.github.com/ticket/2146507 ?
(I do not have permissions to check)

[AkihiroSuda]

Yes, heard that logs are only available as best-effort. So I'm fine to close this issue.