Update Windows Github Action to use static build path on local disk

[amd-justchen]

• Use static build path in order to allow ccache to hit
• Removes use of timestamped directory build path
• Uses local temporary disks on Azure VM via kubernetes container emptyDir storage mount
• AKS Node Pool now uses one of these Azure VM sizes: Standard_D96ads_v6, Standard_HX176rs, Standard_HB176rs_v4 (Old was Standard_D64s_v3)
• Output logging for free disk space
• Add taint for spot nodes
• modified gha-runner-scale-set arc-runners yaml configuration to include:

      resources:
        limits:
          ephemeral-storage: 400Gi
          memory: 300Gi
        requests:
          ephemeral-storage: 10Gi
          memory: 300Gi
      volumeMounts:
      - name: tmpbuild-dir
        mountPath: "b:"
      - name: src-ccache-dir
        mountPath: "s:"
      - name: src-ccache-dir
        mountPath: /home/runner/_work
    tolerations:
    - effect: NoSchedule
      key: kubernetes.io/os
      operator: Equal
      value: windows
    - effect: NoSchedule
      key: kubernetes.azure.com/scalesetpriority
      operator: Equal
      value: spot
    volumes:
    - name: tmpbuild-dir
      emptyDir: {}
    - name: src-ccache-dir
      emptyDir: {}

#132

[ScottTodd]

My general preference for this sort of runner-specific code is to put it in the runner configuration itself, possibly in:

• The VM configuration (e.g. yaml)
• A dockerfile
• A pre-job script (https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/running-scripts-before-or-after-a-job)

That way there can be multiple workflows sharing the same runners without needing to repeat the setup code and the workflows can stay focused on the core steps that a developer would do to reproduce the job on any machine: clone, configure, build.

You could also extract it out to a github action, maybe with a "post" step for the system health logging

• https://dev.to/abiwinanda/github-action-adding-post-steps-in-composite-actions-5ak3
• Example post handling: https://github.com/actions/checkout/blob/main/src/main.ts

[ScottTodd]

96 jobs for fetching seems like a lot... can the network actually support that? I would expect it to get pretty severely bottlenecked. We could also teach fetch_sources.py to infer from nproc or use some other heuristic to pick a good default.

[amd-justchen]

Good catch, I think my intention was to base it on threads and also was experimenting with it. I am not 100% clear on how that argument percolates to the git commands that are run, but from what I understand it passes the jobs argument to the spawned git processes. I think there shouldn't be a network bottleneck (the D96ads_v6 gets 40Gbps of max network bandwidth) assuming Github's servers are adequate.

The theory was, most of the time was spent checking out the many submodules, and applying patches to them, which seemed like a CPU intensive process, so attempt to increase the simultaneous processing of them, but I think it didn't help much as it looked like the whole process is done serially from observing task manager while it was in this step.

All in all though, I think we are trying to prevent having to do a full clone every time and persistently store the source repo files so it can just pull diffs if need be.

[ScottTodd]

Ninja should automatically run with the right parallelism here. Did it not?

[amd-justchen]

Yeah so, this is a lingering question and was trying that argument out just to test since I was observing the CPU usage metrics from the Azure dashboard for the node only hitting ~60 vcpu usage when i expected 96. It could be that the kubernetes configuration is limiting resource limits but I read it should by default allow pods to use max CPU resources. My next idea was to output CPU metrics monitoring logs from within the VM from either native windows performance counters or hwinfo32 as that may give a more accurate perspective of usage.

[amd-justchen]

Actually as of now it seems to be using nearly all 96