Skip to content
This repository was archived by the owner on Nov 16, 2023. It is now read-only.
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/developers/entrypoints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Entrypoints

## Background
NimbusML uses ML.NET's [Entrypoints](https://github.com/dotnet/machinelearning/blob/master/docs/code/EntryPoints.md) API to call ML.NET components from python. The Entrypoints API allows a user to working in a non-.NET language to describe a call to an ML.NET estimator or transformer in JSON format and pass the JSON to ML.NET for execution. So in NimbusML we embed the ML.NET binaries in the published package, expose a python API that constructs these estimator/transformer JSONs, and call the the ML.NET binaries via extension modules to execute the constructed JSONs.
Comment thread
montebhoover marked this conversation as resolved.
Outdated

This is implemented in NimbusML by autogenerating python classes for each of the estimators and transformers in ML.NET. These autogenerated classes do not contain the logic of the corresponding ML.NET components, but rather logic to create the appropriate JSON representation for the entrypoint API.

## Modifying Entrypoint Components and Their Docstrings
These python classes are produced by running [entrypoint_compiler.py](https://github.com/Microsoft/NimbusML/blob/master/src/python/tools/entrypoint_compiler.py), and you will see a comment noting this at the top of each of the files: `# - Generated by tools/entrypoint_compiler.py: do not edit by hand`. If you want to modify the logic of a component, you will have to modify the underlying component in ML.NET. For example, if you want to edit [kmeansplusplus.py](https://github.com/Microsoft/NimbusML/blob/master/src/python/nimbusml/internal/core/cluster/kmeansplusplus.py), you want to look at ML.NET's [KMeansPlusPlusTrainer.cs](https://github.com/dotnet/machinelearning/blob/master/src/Microsoft.ML.KMeansClustering/KMeansPlusPlusTrainer.cs).

If you want to edit the docstring for a NimbusML component, you likewise cannot directly edit the docstring in the autogenerated file. Entrypoint_compiler.py generates the docstrings in these files from a seperate docstring text file that we maintain in the source repo, and this is the file that must be modified. For example, the docstring for [kmeansplusplus.py](https://github.com/Microsoft/NimbusML/blob/master/src/python/nimbusml/internal/core/cluster/kmeansplusplus.py) is generated from [KMeansPlusPlus.txt](https://github.com/Microsoft/NimbusML/blob/master/src/python/docs/docstrings/KMeansPlusPlus.txt). After the changes are made, a modified `kmeansplusplus.py` can be generated by running `entrypoint_compiler.py`.

@shmoradims shmoradims Nov 7, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

KMeansPlusPlus.txt [](start = 497, length = 18)

This is not as easy as you mentioned. KMeansPlusPlus.txt file doesn't contain everything. A lot of the docstrings get filled from manifest_diff.json and manifest.json. Below is the order.
1- info from manifest.json. This come directly from docstrings on ML.NET side
2- Desc in manifest_diff.json
3- content of the .txt file (e.g. KMeansPlusPlus.txt)

Higher order will overwrite lower order. And there's a lot of content that's not in KMeansPlusPlus.txt but gets backfilled from #2 and #1 to create the final docstring.

So editing docstrings it is not that easy. It depends on where the content is coming from.

In general, I don't think the idea was for contributors to directly add to nimbusml. It was to add to ML.NET and have it easily show up in nimbusml.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, that info was helpful for me. I added a footnote to explain this to contributors.


If you forget and accidently edit one of these classes that are autogenerated, the validation build will catch this and fail when checking to see if the autogenerated files are consistent with the docstring text files and the corresponding ML.NET component.

So the end to end process for editing a docstring is to:
1. Edit [KMeansPlusPlus.txt](https://github.com/Microsoft/NimbusML/blob/master/src/python/docs/docstrings/KMeansPlusPlus.txt).
2. Run [entrypoint_compiler.py](https://github.com/Microsoft/NimbusML/blob/master/src/python/tools/entrypoint_compiler.py) locally which will produce [kmeansplusplus.py](https://github.com/Microsoft/NimbusML/blob/master/src/python/nimbusml/internal/core/cluster/kmeansplusplus.py) with your change reflected in the docstring.
Comment thread
montebhoover marked this conversation as resolved.
Outdated
3. Make a PR with both the edited text file and the edited autogenerated file.