Graph-like API to traverse CMF metadata #127
Conversation
This commit adds doc strings and type annotation for CmfQuery class methods.
- References to `client` in the source code (e.g., CmfClient instead of CmfQuery) are removed.
- Several bugs related to checking input dict parameters are fixed (e.g., `d = d or {}` where it should be `if d is None: d= {}`). Now, the corrent object is returned when input dict is just empty.
- Missing doc strings and type annotations are added.
- Additional checks and log messages are added.
Fixing one possible bug related to accessing a column in a data frame when this data frame is empty. Adding key mapper classes to help map source to target keys when copying dictionaries.
The API implements graph-like API to traverse CMF metadata in a graph-like manner. The entry point is the `MetadataStore` class that retrieves from metadata store pipelines, stages, executions and artifacts. Users can specify search query to specify what they want to return (the query is basically the value for the `filter_query` parameter of the `ListOptions` class ML Metadata (MLMD) library). Each node mentioned above (pipeline, stages, executions and artifacts) havs its own Python wrapper class that provides developer-friendly API to access node's parameters and travers graph of machine learning concepts (e.g., get all stages of a pipeline or get all executions of a stage). The graph API also provides the `Properties` wrapper for MLMD's properties and custom_propertied nodes' fields. This wrapper implements the `Mapping` API and automatically converts MLMD's values to Python values on the fly.
- Adding unit tests (97% coverage). - Renaming certain classes, redesigning implementation of several methods. - Adding `Type` class that represents concepts such as ContextType, ExecutionType and ArtifactType in MLMD library.
|
This PR includes multiple work-in-progress commits that are not relevant for the project history, so this PR needs to be "squashed and merged". |
|
@annmary-roy @varkha-d-sharma This PR includes a unit test file in the PS - Unit tests in this PR require special MLMD database ( |
| """Return stage of this execution.""" | ||
| if self._stage is None: | ||
| self._stage = _graph_node( | ||
| Stage, self._db, one(self._db.store.get_contexts_by_execution(execution_id=self.id)) |
There was a problem hiding this comment.
@sergey-serebryakov Just a clarification, when calling "get_context_by_execution" will it return both the stage context and Pipeline context ?
There was a problem hiding this comment.
why do we need "one" function here ?
Related Issues / Pull Requests
This PR depends on Docstrings and annotations for CmfQuery class. #120.Description
This PR implements API to traverse CMF metadata in a graph-like manner.
MlmdGraphclass that retrieves from metadata store pipelines, stages, executions and artifacts. It implements methods such aspipelines,stages,executionsandartifactsthat return lists of respective artifacts (Pipeline,Stage,ExecutionandArtifact).Propertiesclass wrapper for MLMD'spropertiesandcustom_propertiesnodes' fields. This wrapper implements theMappingAPI and automatically converts MLMD's values to Python values on the fly.What changes are proposed in this pull request?
Checklist: