Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metric context query #638

Merged
merged 39 commits into from
Dec 2, 2023
Merged

Conversation

suprjinx
Copy link
Contributor

@suprjinx suprjinx commented Nov 22, 2023

Add metric context querying in the repository layer. Metrics are filtered by a map of values, where the key is a json path, and value is what we expect at that path. For instance, given a context struct like { "key1": "value1", "key2": { "key3": "value3"}}, the context could be queried with map[string]string{"key1": "value1", "key2.key3", "value3"}. This seems to fit with the query inputs we expect from Aim client, such as metric.context.key1 = value1.
Fixes #358
Fixes #226

@suprjinx suprjinx marked this pull request as ready for review November 27, 2023 21:14
@dsuhinin
Copy link
Collaborator

Add metric context querying in the repository layer. Metrics are filtered by a map of values, where the key is a json path, and value is what we expect at that path. For instance, given a context struct like { "key1": "value1", "key2": { "key3": "value3"}}, the context could be queried with map[string]any{"key1": "value1", "key2.key3", "value3"}. This seems to fit with the query inputs we expect from Aim client, such as metric.context.key1 = value1.
Fixes #358

is it a mistake here -> {"key1": "value1", "key2.key3": "value3"} ?

Copy link
Collaborator

@dsuhinin dsuhinin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will be nice to add several cases to the GetHistoriesTestSuite_Ok test to test actual search functionality by context parameters. I can'd find it.

Comment on lines 32 to 43
func (f MetricFixtures) CreateMetric(ctx context.Context, metric *models.Metric) (*models.Metric, error) {
if metric.Context != nil {
if err := f.baseFixtures.db.WithContext(ctx).FirstOrCreate(&metric.Context).Error; err != nil {
return nil, eris.Wrap(err, "error creating metric context")
}
metric.ContextID = &metric.Context.ID
}
if err := f.baseFixtures.db.WithContext(ctx).Create(metric).Error; err != nil {
return nil, eris.Wrap(err, "error creating test metric")
return nil, eris.Wrap(err, "error creating metric")
}
return metric, nil
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can you please check this moment -> https://gorm.io/docs/create.html#Create-With-Associations. amybe we even don't need to create twice and gorm can do all the magic under the hood.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gorm is not doing the magic we need -- it could perhaps become a BeforeCreate hook on the Metric and LatestMetric model?

Comment on lines +78 to 86
if metric.Context != nil {
if err := f.baseFixtures.db.WithContext(ctx).FirstOrCreate(&metric.Context).Error; err != nil {
return nil, eris.Wrap(err, "error creating latest metric context")
}
metric.ContextID = &metric.Context.ID
}
if err := f.baseFixtures.db.WithContext(ctx).Create(metric).Error; err != nil {
return nil, eris.Wrap(err, "error creating test latest metric")
return nil, eris.Wrap(err, "error creating latest metric")
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same moment here -> maybe we can just create everything during one gorm call?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gorm is not able to do "FirstOrCreate" on the association by magic, but I think we could move to Metric.BeforeCreate hook if that's preferable (?)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's a bit strange cause they say:

When creating some data with associations, if its associations value is not zero-value, those associations will be upserted, and its Hooks methods will be invoked.
``` - upsert. if no, then ok.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my testing did not show that to work in this case, possibly because our FK relationship is inverted (ie, we'd have to create Context and then it could auto create Context.Metrics if we had that association defined).

tests/integration/golang/fixtures/metric.go Outdated Show resolved Hide resolved
pkg/api/mlflow/dao/repositories/helpers.go Outdated Show resolved Hide resolved
pkg/api/mlflow/service/metric/service.go Outdated Show resolved Hide resolved
@suprjinx
Copy link
Contributor Author

it will be nice to add several cases to the GetHistoriesTestSuite_Ok test to test actual search functionality by context parameters. I can'd find it.

yes, i couldn't add yet because the endpoint isn't done (separate story) -- I think we might have to revisit when they come together.

@suprjinx suprjinx marked this pull request as draft November 28, 2023 15:52
@suprjinx suprjinx marked this pull request as ready for review November 29, 2023 18:52
pkg/api/mlflow/dao/repositories/helpers.go Outdated Show resolved Hide resolved
pkg/api/mlflow/dao/repositories/helpers.go Outdated Show resolved Hide resolved
pkg/api/mlflow/dao/repositories/helpers_test.go Outdated Show resolved Hide resolved
pkg/api/mlflow/dao/repositories/metric.go Show resolved Hide resolved
tests/integration/golang/fixtures/metric.go Outdated Show resolved Hide resolved
tests/integration/golang/fixtures/metric.go Outdated Show resolved Hide resolved
tests/integration/golang/fixtures/metric.go Outdated Show resolved Hide resolved
).Joins(
"LEFT JOIN contexts on metrics.context_id = contexts.id",
)
sql, args := repositories.BuildJsonCondition(tx.Dialector.Name(), "contexts.json", metricContext)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can't we pass it right into Where?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above, personally I think what Geoff wrote is more readable...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also we have to ... the second return val in this approach

tests/integration/golang/helpers/arrow.go Outdated Show resolved Hide resolved
tests/integration/golang/helpers/arrow.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@dsuhinin dsuhinin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@suprjinx suprjinx merged commit 0b87a35 into G-Research:main Dec 2, 2023
17 checks passed
@suprjinx suprjinx deleted the metric-context-query branch December 2, 2023 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
3 participants