Set ephemeral storage to match cache

[daxmc99]

Related to #2369

Fixes https://github.com/sourcegraph/sourcegraph/issues/20869

If these values differ, the pod will be evicted for exceeding its storage allotment

Checklist

• CHANGELOG.md updated
• K8s Upgrade notes updated
• Sister deploy-sourcegraph-docker change:
• All images have a valid tag and SHA256 sum

[daxmc99]

Could @sourcegraph/search-core take a look at this to see if this resource reduction is ok?

[keegancsmith]

TIL ephemeral-storage limits! Finally! I see there is a whole bunch of stuff related to ephemeral-storage in k8s now, not just emptyDir. Would love to use GenericEphemeralVolume so we can get rid of the dir watcher daemonset stuff. I see that it is behind a feature flag though :(

[keegancsmith]

Looks good. Maybe we should avoid duplication here, and instead use something like

valueFrom:
  fieldRef:
    fieldPath: spec.resources.requests.ephemeral-storage

That will require us updating the cache parsing code to understand kubernetes resource limit specifications, but that would be a good thing. Not for this PR, but what do you think?

[davejrt]

Thanks for this, I had to adjust this for a customer but never got back around to fixing the root cause.

[daxmc99]

I was incorrect with my first attempt at this. This requires using resourceFieldRef to specify the value.