sagemaker job return duplicates rows

[cindyangelira]

I am currently using SageMaker version 2.190 and have encountered an issue in my workflow. I am running a step function that involves three steps: prep, inference (using the sklearn estimator), and post-processing. The prep and inference jobs collectively return 1 million rows. However, when I run the post-processing job, which involves combining identifiers created during the prep phase with the inference results, it unexpectedly yields 2 million rows.
I have already debugged my code step by step when running it in SageMaker Studio, and it consistently returns 10 rows. Despite this, the post-processing job appears to generate twice as many rows. I'm curious if anyone else has come across a similar issue. Any insights or suggestions would be greatly appreciated.