Metrics table generation from yaml

[jamesmoessis]

This PR adds functionality such that metrics can be parsed from yaml and relevant information generated in markdown.

This can regenerate the structure of the http metric semantic convention that we have today, including the "Attribute alternatives" which can be controlled via the constraints in the yaml.

There are two types of tables that must be generated from MetricSemanticConvention:

• attributes (same as before)
• metrics (id, instrument, units)

<!-- semconv metric.http.client --> - creates an attribute table as normal.

<!-- semconv metric.http.client(metric_table) --> - creates a metric table of all metrics that are under metric.http.client in the yaml file. If you have other ideas about how the yaml file should be structured, let me know (see http_metrics.yaml).

The metric prefix ID is used to prevent clash with the tracing semconvs of the same ID.

It's quite few lines of actual code, most of this PR is testing.

cc @ocelotl @jsuereth

[Oberon00]

All features require documentation in https://github.com/open-telemetry/build-tools/blob/main/semantic-conventions/syntax.md. This would also be quite useful to review the whole feature.

[jamesmoessis]

Thanks @Oberon00, I've pushed up syntax changes. Once we agree on a syntax I can add it to the JSON config too.

[jamesmoessis]

Hey @Oberon00 just bumping on this since it's been a while.

I feel like the main thing needing to be discussed is the structure of the metrics in the yaml and how the extending works. What else can I do to get this PR moving?

[Oberon00]

@jamesmoessis I went over the PR again, focusing on implementation & "formalities". I think for the actual high-level data model/YAML-schema, someone responsible for metrics needs to approve.

[jsuereth]

@jamesmoessis What's the status on this? Do you need any assistance fixing the changes suggested by @Oberon00 ? I'd love if we could get this merged soon. Let me know if a PR against your PR with fixes would be appreciated or if you'll have time to address this soon.

[jamesmoessis]

Hey @jsuereth - apologies, I've been slow with addressing those comments, I have been busy with other things. I can get to it next week.

[jamesmoessis]

Ok @jsuereth @joaopgrassi @Oberon00 I think I've addressed all comments.

The build is failing pending on this fix regarding the linter: #125

[arminru]

@jamesmoessis Thank you! I merged #125 but I think you'll have to update your branch manually to resolve the conflicts.

[jamesmoessis]

Thanks @arminru - I've resolved the conflicts.

[jamesmoessis]

@arminru @jsuereth @Oberon00 @joaopgrassi Any other comments or can we go ahead and merge?

[joaopgrassi]

@jamesmoessis not sure if the open comments are already resolved, but on my part, I think we need to agree on what approach we will take on the table rendering:

#79 (comment)

So we have a plan for the follow up.

[jsuereth]

@joaopgrassi I actually don't think we need perfect table rendering yet.

The most important aspect of this PR is that we can define metric semantic conventions in YAML and generate any form of documentation for them. I think what @jamesmoessis has now is usable, and tweakable going forward. If we want to make improvements, I suggest folks that want different views tackle that work.

For now, we really need any functionality that lets us document metric semantic conventions programatically and share attributes w/ traces & logs. If we have concerns for how that works, I'd consider it blocking.

Let's not bikeshed too much on Markdown tables, especially when those are easy to fix once we have data in YAML. Right now we don't even have our semconv in YAML.

[marcalff]

@joaopgrassi I actually don't think we need perfect table rendering yet.

+1 on that.

From the language implementation point of view (like opentelemetry-cpp), what we need is:

• metrics semantic conventions represented in yaml files,
• build tools to generate code, for metrics.

Please don't block progress because of rendering issues on table.md, if it can be resolved independently (and therefore, in parallel).

[joaopgrassi]

Yes, I agree! All my comments were "non blocking" (sorry if it was unclear) and I have been advocating to merge this for a while so we have something.

I just wanted to make sure there's a consensus going forward so two people don't do double work on the follow up.

[arminru]

I went through the remaining unresolved conversations and reached out to the authors to resolve them.

I'll also resolve the last one - https://github.com/open-telemetry/build-tools/pull/79/files#r996803040 - as we agreed that this lays a solid foundation for defining metrics in YAML and that the MD rendering can be tackled in a follow-up PR. I created #129 for that.

Thank you a lot for your work and endurance, @jamesmoessis! 😊
Also thanks to the reviewers for the multiple rounds of input, this looks good now!