[vpj] Fix DATASET_CHANGED tracking for Spark jobs

[nisargthakkar]

Fix DATASET_CHANGED tracking for Spark jobs

In Spark ComputeJob, some cases of DATASET_CHANGED were not getting tracked correctly, and were instead getting tracked as START_DATA_WRITER_JOB.

This was because of an incorrect assumption that Spark jobs would get executed only when the final action stage is encountered. But in practice, the spark stages were starting to get executed when the repartitionAndSortWithinPartitions call was getting invoked. This was previously done in the configure phase, and exceptions thrown in this stage are not handled as errors during the execution of the compute job.

This commit moves all Spark execution logic inside the runComputeJob method instead

How was this PR tested?

GH CI

Does this PR introduce any user-facing changes?

• No. You can skip the rest of this section.
• Yes. Make sure to explain your proposed changes and call out the behavior change.