Running local mode integration tests in CI.

[Ce11an]

My team is currently utilising SageMaker pipelines to coordinate model training. In the past, we encountered issues where the pipeline was misconfigured during cloud execution, resulting in challenging-to-debug errors.

I've been delving into the idea of using Localstack and SageMaker LocalPipelineSession to execute the pipeline locally prior to deployment on the cloud I've successfully implemented this on my local machine, using pytest and pytest-bdd to craft integration tests.

Building on that success, l've ventured into creating a GitLab Cl job that runs these tests upon making a merge request. l've designed a custom Docker image, enabling installation of Python, my dependencies, Docker, and Docker Compose. The job initialises LocalStack and executes the tests. The pipeline starts execution but fails with slient errors. Potentially, there is an issue with mounting the volumes to the container.

Has anyone achieve running SageMaker pipelines in local mode within CI? Any advice would be appreciated!