Make snapshots expiration job leaner

[teamurko]

Summary

Snapshots expiration job skips removing files as we wanted to localize files removal to one job. The job traverses the files tree though, and that is expensive and unnecessary. As the result of this change, it updates snapshots list in the metadata without traversing the files tree.

Changes

• Client-facing API Changes
• Internal API Changes
• Bug Fixes
• New Features
• Performance Improvements
• Code Style
• Refactoring
• Documentation
• Tests

For all the boxes checked, please include additional details of the changes made in this pull request.

Testing Done

• Manually Tested on local docker setup. Please include commands ran, and their output.
• Added new tests for the changes made.
• Updated existing tests to reflect the changes made.
• No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
• Some other form of testing like staging or soak time in production. Please explain.

No change in the job effect is expected, it's purely optimization.

For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.

Additional Information

• Breaking Changes
• Deprecations
• Large PR broken into smaller PRs, and PR plan linked in the description.

For all the boxes checked, include additional details of the changes made in this pull request.

[maluchari]

Nice change to avoid unnecessary building of validDF !! Should we have a docker run to ensure if commit post this snapshot expiration look as expected?

[teamurko]

Nice change to avoid unnecessary building of validDF !! Should we have a docker run to ensure if commit post this snapshot expiration look as expected?

It's tested in

openhouse/apps/spark/src/test/java/com/linkedin/openhouse/jobs/spark/OperationsTest.java

Line 273 in cf43ae7

public void testSnapshotsExpirationJavaAPI() throws Exception {