reduce pod definition size

[tooptoop4]

Summary

The pod spec for steps run from argo is quite large and can fill etcd. Wanting to trim this down, I imagine there will be 2 parts:

1. code changes
2. docs for configuration of init vs main vs wait containers

thoughts/Qs:

1. is ARGO_TEMPLATE env variable needed on all 3 containers?
   • refactor: Only set ARGO_TEMPLATE env for init container. #13761
2. are volumeMounts needed on both main and wait containers?
   • feat: don't add configmap volumemounts in wait container. Partial fix for #13089 #13741
3. do all 3 containers need environment variables for communicating with s3 ie via artifactRepository? (i'm guessing main container doesn't)
   • seems this is already handled
4. ARGO_TEMPLATE is huge, is the ARGO_TEMPLATE env variable containing things it doesn't need? perhaps some containers need things within that other containers don't?
   • more analysis still todo
5. are ARGO_PROGRESS_PATCH_TICK_DURATION/ARGO_PROGRESS_FILE_TICK_DURATION/ARGO_INCLUDE_SCRIPT_OUTPUT/ARGO_PROGRESS_FILE env variables needed?
   • fix: only set ARGO_PROGRESS_FILE when needed. Partial fix for #13089 #13743
6. do the commands on all the containers need --loglevel/--log-format?
   • feat: don't add args with same value as default. Partial fix for #13089 #13093
7. i think within ARGO_TEMPLATE it does not need volumeMounts related to configmaps of user code like some.py configmap
   • refactor: don't add configmap volumemounts in ARGO_TEMPLATE. Partial fix for #13089  #13792
8. i think within ARGO_TEMPLATE it does not need some user provided env variables
   • refactor: don't add env variables in ARGO_TEMPLATE. Partial fix for #13089 #13793

Use Cases

ensure etcd does not fill up

[jswxstw]

• is ARGO_TEMPLATE env variable needed on all 3 containers?

ARGO_TEMPLATE may be huge, #12325 provides an optimization solution for EnvVarTemplate offload, perhaps it can be made the default logic. However, its lifecycle is aligned with the workflow, not the pod, considering delete it when pod gc?

are ARGO_PROGRESS_PATCH_TICK_DURATION/ARGO_PROGRESS_FILE_TICK_DURATION/ARGO_INCLUDE_SCRIPT_OUTPUT/ARGO_PROGRESS_FILE env variables needed?

ARGO_PROGRESS_PATCH_TICK_DURATION/ARGO_PROGRESS_FILE_TICK_DURATION/ARGO_PROGRESS_FILE are used to implement self reporting progress.
ARGO_INCLUDE_SCRIPT_OUTPUT is used to determine whether stdout needs to be saved.

I think optimizing the reuse of ARGO_TEMPLATE would be sufficient. The other aspects have minimal impact, there's no need to be overly demanding.

[tooptoop4]

@jswxstw do u know what ARGO_TEMPLATE is for? and do all 3 containers need it?

[jswxstw]

All 3 containers need ARGO_TEMPLATE to prepare inputs or save outputs, and in some types of templates like Script/Resourct/ContainerSet, it serves other purposes as well.