Embed model identifier in events

[wochinge]

Description of Problem:
Original issue: #8815 (comment)

We need a unique model identifier in each event caused by a model in order to tie model predictions to a model

Overview of the Solution:

1. Create unique id for each trained model and embed it in the fingerprint.json (or a different file if the graph model is already underway)
2. Attach this unique model ID to the metadata field of every event caused by this model

Definition of Done:

• Models include a unique ID
• Every event caused by predictions of a model includes this ID in its metadata

[TyDunn]

@rudolfix and I were chatting about this. @wochinge will / could this unique model ID be a human readable model name?

[wochinge]

What is a human readable model name and what purpose does it have? For me that's the name of the model archive. From my perspective the unique ID in the model is some arbitrary string.

[rudolfix]

hey @wochinge, we are looking how to analyze tracker data and it's easier for humans if they can name models and then for example compare the performance of old and new model in some kind of dashboard.

here I assume that model id is a cryptographic hash of various model files or even a random string. then having this string in event in tracker store I can somehow backtrack to actual model and read it configuration and name. in RASA X, API for model could be extended to get model by model id. correct?

[wochinge]

then having this string in event in tracker store I can somehow backtrack to actual model and read it configuration and name. in RASA X, API for model could be extended to get model by model id. correct?

Exactly 👍🏻

I think the renaming whatever should be done in the dashboard then to avoid collisions.

[rudolfix]

in case of SDK I can then create the fingerprint file myself with a content I wish. In other words: this will also work without Rasa X? thanks for your time :)

[wochinge]

Why do you want to create the fingerprint in the SDK yourself?

It will work as follows

1. We assign each model a unique ID
2. Whenever we update a tracker with new events we attach the unique model ID (this includes events created in the SDK, through the API, or internally)
3. All persisted / published events will have this ID as part of their payload

How we handle it in Rasa X is up to the teams working on Rasa X. In theory they'd add a column to the model table which tracks the unique ID for each model in the database + have a column model_id on conversation_event. They then can tie each event to a specific model.

[rudolfix]

ok got it! just looked what is already in model archive and now I understand it, thanks!