[BUG]: outside of artifact store bounds

[Aadik1ng]

System Information

SYSTEM_INFO: {'os': 'windows', 'windows_version_release': '10', 'windows_version': '10.0.19045', 'windows_version_service_pack': 'SP0', 'windows_version_os_type':
'Multiprocessor Free'}

What happened?

Initiating a new run for the pipeline: data_ingestion_pipeline.
Executing a new run.
Caching is disabled by default for data_ingestion_pipeline.
Using user: default
Using stack: stack_1
orchestrator: default
artifact_store: local_artifact_store
You can visualize your pipeline runs in the ZenML Dashboard. In order to try it locally, please run zenml up.

Relevant log output

Initiating a new run for the pipeline: data_ingestion_pipeline.
Executing a new run.
Caching is disabled by default for data_ingestion_pipeline.
Using user: default
Using stack: stack_1
  orchestrator: default
  artifact_store: local_artifact_store
You can visualize your pipeline runs in the ZenML Dashboard. In order to try it locally, please run zenml up.
Failed to execute data ingestion pipeline: File `D:data\artifacts\data_ingestion_step\logs` is outside of artifact store bounds `data/artifacts`

CURRENT STACK

Name: stack_1
ID: d1765057-c27f-4e5e-ac3e-e7cee6a45797
User: default / a8bf9397-512c-4f9a-9266-f24b3ea10921
Workspace: default / 5e1c6e98-5302-47d4-bbc7-b97d24d1def3

ORCHESTRATOR: default

Name: default
ID: 33520fcf-456f-438d-b894-a765938a6b5e
Type: orchestrator
Flavor: local
Configuration: {}
Workspace: default / 5e1c6e98-5302-47d4-bbc7-b97d24d1def3

ARTIFACT_STORE: local_artifact_store

Name: local_artifact_store
ID: abae7772-933e-40fb-8893-0eb1bd855a9e
Type: artifact_store
Flavor: local
Configuration: {'path': 'data/artifacts'}
User: default / a8bf9397-512c-4f9a-9266-f24b3ea10921
Workspace: default / 5e1c6e98-5302-47d4-bbc7-b97d24d1def3

Reproduction steps

@step
def data_ingestion_step(config_path: str) -> Annotated[Dict[str, str], "paths"]:
try:
# Load configuration
config = load_config(config_path)
pdf_path = config['balance_sheet_pdf']

    # Use ZenML's artifact store context to determine base paths
    context = get_step_context()
    base_dir = context.get_output_artifact_uri()

    # Construct directories within the artifact store
    # Construct directories within the artifact store
    image_dir = os.path.join(base_dir, config.get('image_dir', 'data/artifacts/images'))
    table_dir = os.path.join(base_dir, config.get('table_dir', 'data/artifacts/tables'))
    other_dir = os.path.join(base_dir, config.get('other_dir', 'data/artifacts/other_content'))



    os.makedirs(image_dir, exist_ok=True)
    os.makedirs(table_dir, exist_ok=True)
    os.makedirs(other_dir, exist_ok=True)

    # Set up logging within the artifact store
    log_file_path = os.path.join(base_dir, 'logs', 'data_ingestion.log')
    os.makedirs(os.path.dirname(log_file_path), exist_ok=True)
    logging.basicConfig(
        filename=log_file_path,
        filemode='a',
        level=logging.INFO,
        format='%(asctime)s - %(levelname)s - %(message)s'
    )
    logger = logging.getLogger(__name__)

    logger.info(f"Image directory: {image_dir}")
    logger.info(f"Table directory: {table_dir}")
    logger.info(f"Other content directory: {other_dir}")

    # Extract data from PDF
    extract_from_pdf(pdf_path, image_dir, table_dir, other_dir)
    
    # Log extraction info
    log_extraction_info(pdf_path, image_dir, table_dir, other_dir)

    # Return paths as a dictionary
    return {
        "image_dir": image_dir,
        "table_dir": table_dir,
        "other_dir": other_dir
    }
except Exception as e:
    logger.error(f"Error in data ingestion step: {e}")
    raise

@pipeline
def data_ingestion_pipeline(config_path: str):
data_ingestion_step(config_path=config_path)

if name == "main":
config_path = 'config.yml' # Adjust this path if needed
try:
# Run the pipeline
data_ingestion_pipeline = data_ingestion_pipeline.with_options(enable_cache=False)
data_ingestion_pipeline(config_path=config_path)
except Exception as e:
print(f"Failed to execute data ingestion pipeline: {e}")

Code of Conduct

• I agree to follow this project's Code of Conduct

[schustmi]

This seems like an issue with Windows paths. Can you just quickly verify whether the following code also fails for you on your local machine please:

from zenml import pipeline, step

@step
def logging_step() -> None:
  print("Some log message")

@pipeline
def p():
  logging_step()

if __name__ == "__main__":
  p()

[Aadik1ng]

FileNotFoundError: File D:\data\artifacts\logging_step\logs is outside of artifact store bounds data/artifacts
PS D:> & C:/Users/aadit/AppData/Local/Microsoft/WindowsApps/python3.11.exe D:\src\rough.py
Initiating a new run for the pipeline: p.
Executing a new run.
Using user: default
Using stack: stack_1
orchestrator: default
artifact_store: local_artifact_store
You can visualize your pipeline runs in the ZenML Dashboard. In order to try it locally, please run zenml up.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ D:\src\rough.py:12 in │
│ │
│ 9 logging_step() │
│ 10 │
│ 11 if name == "main": │
│ ❱ 12 p() │
│ 13 │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\new\pipelines\pipeline.py:1382 in call │
│ │
│ 1379 │ │ │ return self.entrypoint(*args, **kwargs) │
│ 1380 │ │ │
│ 1381 │ │ self.prepare(*args, **kwargs) │
│ ❱ 1382 │ │ return self._run(**self._run_args) │
│ 1383 │ │
│ 1384 │ def _call_entrypoint(self, *args: Any, **kwargs: Any) -> None: │
│ 1385 │ │ """Calls the pipeline entrypoint function with the given arguments. │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\new\pipelines\pipeline.py:771 in _run │
│ │
│ 768 │ │ │ │ │ │ "zenml up." │
│ 769 │ │ │ │ │ ) │
│ 770 │ │ │ │
│ ❱ 771 │ │ │ deploy_pipeline( │
│ 772 │ │ │ │ deployment=deployment_model, stack=stack, placeholder_run=run │
│ 773 │ │ │ ) │
│ 774 │ │ │ if run: │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\new\pipelines\run_utils.py:153 in │
│ deploy_pipeline │
│ │
│ 150 │ │ │ # placeholder run to stay in the database │
│ 151 │ │ │ Client().delete_pipeline_run(placeholder_run.id) │
│ 152 │ │ │
│ ❱ 153 │ │ raise e │
│ 154 │ finally: │
│ 155 │ │ constants.SHOULD_PREVENT_PIPELINE_EXECUTION = previous_value │
│ 156 │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\new\pipelines\run_utils.py:141 in │
│ deploy_pipeline │
│ │
│ 138 │ previous_value = constants.SHOULD_PREVENT_PIPELINE_EXECUTION │
│ 139 │ constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True │
│ 140 │ try: │
│ ❱ 141 │ │ stack.deploy_pipeline(deployment=deployment) │
│ 142 │ except Exception as e: │
│ 143 │ │ if ( │
│ 144 │ │ │ placeholder_run │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\stack\stack.py:853 in deploy_pipeline │
│ │
│ 850 │ │ Returns: │
│ 851 │ │ │ The return value of the call to orchestrator.run_pipeline(...). │
│ 852 │ │ """ │
│ ❱ 853 │ │ return self.orchestrator.run(deployment=deployment, stack=self) │
│ 854 │ │
│ 855 │ def _get_active_components_for_step( │
│ 856 │ │ self, step_config: "StepConfiguration" │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\orchestrators\base_orchestrator.py:187 in run │
│ │
│ 184 │ │ environment = get_config_environment_vars(deployment=deployment) │
│ 185 │ │ │
│ 186 │ │ try: │
│ ❱ 187 │ │ │ result = self.prepare_or_run_pipeline( │
│ 188 │ │ │ │ deployment=deployment, stack=stack, environment=environment │
│ 189 │ │ │ ) │
│ 190 │ │ finally: │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\orchestrators\local\local_orchestrator.py:78 in │
│ prepare_or_run_pipeline │
│ │
│ 75 │ │ │ │ │ step_name, │
│ 76 │ │ │ │ ) │
│ 77 │ │ │ │
│ ❱ 78 │ │ │ self.run_step( │
│ 79 │ │ │ │ step=step, │
│ 80 │ │ │ ) │
│ 81 │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\orchestrators\base_orchestrator.py:207 in │
│ run_step │
│ │
│ 204 │ │ │ step=step, │
│ 205 │ │ │ orchestrator_run_id=self.get_orchestrator_run_id(), │
│ 206 │ │ ) │
│ ❱ 207 │ │ launcher.launch() │
│ 208 │ │
│ 209 │ @staticmethod │
│ 210 │ def requires_resources_in_orchestration_environment( │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\orchestrators\step_launcher.py:164 in launch │
│ │
│ 161 │ │ │
│ 162 │ │ if step_logging_enabled: │
│ 163 │ │ │ # Configure the logs │
│ ❱ 164 │ │ │ logs_uri = step_logging.prepare_logs_uri( │
│ 165 │ │ │ │ self._stack.artifact_store, │
│ 166 │ │ │ │ self._step.config.name, │
│ 167 │ │ │ ) │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\logging\step_logging.py:87 in prepare_logs_uri │
│ │
│ 84 │ ) │
│ 85 │ │
│ 86 │ # Create the dir │
│ ❱ 87 │ if not artifact_store.exists(logs_base_uri): │
│ 88 │ │ artifact_store.makedirs(logs_base_uri) │
│ 89 │ │
│ 90 │ # Delete the file if it already exists │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\artifact_stores\base_artifact_store.py:151 in │
│ call │
│ │
│ 148 │ │ has_self = bool(args and isinstance(args[0], BaseArtifactStore)) │
│ 149 │ │ │
│ 150 │ │ # sanitize inputs for relevant args and kwargs, keep rest unchanged │
│ ❱ 151 │ │ args = tuple( │
│ 152 │ │ │ self._sanitize_potential_path( │
│ 153 │ │ │ │ arg, │
│ 154 │ │ │ ) │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\artifact_stores\base_artifact_store.py:152 in │
│ │
│ │
│ 149 │ │ │
│ 150 │ │ # sanitize inputs for relevant args and kwargs, keep rest unchanged │
│ 151 │ │ args = tuple( │
│ ❱ 152 │ │ │ self._sanitize_potential_path( │
│ 153 │ │ │ │ arg, │
│ 154 │ │ │ ) │
│ 155 │ │ │ if i + has_self in self.path_args │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\artifact_stores\base_artifact_store.py:133 in │
│ _sanitize_potential_path │
│ │
│ 130 │ │ │ path = path.replace(ntpath.sep, posixpath.sep) │
│ 131 │ │ │ self._validate_path(path) │
│ 132 │ │ else: │
│ ❱ 133 │ │ │ self._validate_path(str(Path(path).absolute().resolve())) │
│ 134 │ │ │
│ 135 │ │ return path │
│ 136 │
│ │
│ C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCa │
│ che\local-packages\Python311\site-packages\zenml\artifact_stores\base_artifact_store.py:98 in │
│ _validate_path │
│ │
│ 95 │ │ │ │ bounds. │
│ 96 │ │ """ │
│ 97 │ │ if not path.startswith(self.fixed_root_path): │
│ ❱ 98 │ │ │ raise FileNotFoundError( │
│ 99 │ │ │ │ f"File {path} is outside of " │
│ 100 │ │ │ │ f"artifact store bounds {self.fixed_root_path}" │
│ 101 │ │ │ ) │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: File D:\data\artifacts\logging_step\logs is outside of artifact store bounds data/artifacts

[Aadik1ng]

These are my environment variable

HOMEDRIVE: C:
HOMEPATH: \Users\aadit
LOCALAPPDATA: C:\Users\aadit\AppData\Local
LOGONSERVER: \DESKTOP-C4QBLHV
NUMBER_OF_PROCESSORS: 16
ORIGINAL_XDG_CURRENT_DESKTOP: undefined
OS: Windows_NT
PATH:
C:\Users\aadit\AppData\Local\zenml;
PATHEXT: .COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC;.CPL

ZENML_HOME: C:\Users\aadit\AppData\Local\zenml
ZES_ENABLE_SYSMAN: 1

[schustmi]

Are you using the default local artifact store? Or did you register a custom one?

[Aadik1ng]

I was using custom in my previous stack but everything is on default now
Initiating a new run for the pipeline: simple_ml_pipeline.
Executing a new run.
Using user: default
Using stack: default
orchestrator: default
artifact_store: default

[Aadik1ng]

(PS) D:\Dir> zenml artifact-store describe default
Artifact_Store 'default' of flavor 'local' with id 'd6305633-a8f6-45ea-87f6-ae469da61fcf' is owned by user '-'.
No configuration options are set for this component.
No labels are set for this component.
No connector is set for this component.

so the default artifact-store was not set properly so i change it to this
basically created a new artifact-store called balance with dir/data/artifact as path

(PS)D:\Dir> zenml artifact-store describe balance
Artifact_Store 'Balance' of flavor 'local' with id 'bc735317-765d-4e0a-99c7-dbf0728f4702' is owned by user 'default'.
'Balance' ARTIFACT_STORE Component
Configuration (ACTIVE)
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ COMPONENT_PROPERTY │ VALUE ┃
┠────────────────────┼────────────────┨
┃ PATH │ data/artifacts ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
No labels are set for this component.
No connector is set for this component.

[schustmi]

What do you mean by "the default artifact store was no set properly"? You ran into some issues when running pipelines with the default artifact store?

I think with your custom artifact store, it's somehow messing up the volumes. Did you explicitly register it with D:\data\artifacts?

[Aadik1ng]

when i was using default artifact store i was getting the same issue but the path were

Failed to execute data ingestion pipeline: File C:\Users\aadit\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\Roaming\zenml\local_stores\5b773d37-c170-4a39-b1f5-d6a20fec3d5b\data_ingestion_step\logs is outside of artifact store bounds C:\Users\aadit\AppData\Roaming\zenml\local_stores\5b773d37-c170-4a39-b1f5-d6a20fec3d5b

so i change the artifact_store to custom one, updated its path to data/artifacts. I got the same issue but this time the paths were

D:\data\artifacts..

PS D:\RAG-on-Balance_Sheet> zenml artifact-store register balance --flavor=local
You are configuring a stack component that is using local resources while connected to a remote ZenML server. The stack component may not be usable from other hosts or by other users. You should consider using a non-local stack component alternative instead.
Successfully registered artifact_store balance.