[SPARK-45096][INFRA] Optimize apt-get install in Dockerfile

[zhengruifeng]

What changes were proposed in this pull request?

follow the Best practices for writing Dockerfiles :

Always combine RUN apt-get update with apt-get install in the same RUN statement.

Why are the changes needed?

1, to address #42253 (comment)
2, when I attempted to change the apt-get install in #41918, the behavior was confusing. By following the best practices, further changes should work immediately.

Does this PR introduce any user-facing change?

NO, dev-only

How was this patch tested?

CI

Was this patch authored or co-authored using generative AI tooling?

NO

[zhengruifeng]

cc @Yikun @dongjoon-hyun

[zhengruifeng]

CI link: https://github.com/zhengruifeng/spark/actions/runs/6093404505/job/16533661403

[Yikun]

LGTM, but I think you should know (or maybe already know):

Always combine RUN apt-get update with apt-get install in the same RUN statement.

This best practices is for to help reduce redundat layer between update and install, but for spark infra, there are a tradeoff between image size download cost and image rebuild cost.

Move frequently modified dependencies to front means: once the Dockerfile head front dependencies are modified, the following content will also be rebuild (due to infra cache invalidated). Finally, it impacts PRs which change Dockerfile head lines or haven't rebase in time CI will cost more time (about 1h).

TL;DR: move stable dependencies to top, keep frequently modified dependencies to end.

[Yikun]

So L58 not change is intentional, right?

[zhengruifeng]

no, should be also moved in L27, let me update

[zhengruifeng]

LGTM, but I think you should know (or maybe already know):

Always combine RUN apt-get update with apt-get install in the same RUN statement.

This best practices is for to help reduce redundat layer between update and install, but for spark infra, there are a tradeoff between image size download cost and image rebuild cost.

Move frequently modified dependencies to front means: once the Dockerfile head front dependencies are modified, the following content will also be rebuild (due to infra cache invalidated). Finally, it impacts PRs which change Dockerfile head lines or haven't rebase in time CI will cost more time (about 1h).

TL;DR: move stable dependencies to top, keep frequently modified dependencies to end.

thanks for the detailed explaination, since we rarely modify the apt-install (we frequently changes the pip-install), so I think it would be fine.

[zhengruifeng]

thanks @Yikun and @HyukjinKwon , merged to master