[Feature] Support setting global tag to trace

[wallezhang]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

Now most applications will be deployed in the cloud environment. In order to better distinguish services in different available areas or different regions, it is often necessary to add some global tags in trace. Just like the resource concept in opentelemetry.

Use case

Inject global tags to every trace. I think it should be a better way to set a global tag through the agent config file.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

We need to discuss the detail about this. Adding tag to every trace segment is fine, but we should not add to every span.
So, what is your plan on that.

[wallezhang]

Yes, add to trace segment is enough. I think the todo list are:

• Add a field to agent.config, the value format like this: tagKeyA=tagValue,tagKeyB=valueB...
• Add a field repeated KeyStringValuePair tags to SegmentObject in proto
• In org.apache.skywalking.apm.agent.core.context.trace.TraceSegment#transform method, set global tag to SegmentObject
• OAP server saves the segment tag
• The global tag will show on span detail UI, maybe distinguish with a prefix segment.tag

[wu-sheng]

We don't have plan to change protocol, because it requires too many.

Maybe consider adding to first span?

[wu-sheng]

Actually, I am thinking why this could not be added into instance name or service name?

[wallezhang]

The service name of the same service should be uniform. But for satisfying high available, the same service may be deployed to different region. While a region has some problems, the service on that region will have errors. We can only find the service have errors from dashboard, but we don't know that there is a problem in a region. There are many dimensions like region in cloud environment.

[wu-sheng]

I think a tag would not help much in this case. All statistics today in SkyWalking are general service oriented, we can't do this for now even with added tag.

[wallezhang]

Of course, we observe the performance through the service dimension, but we can search traces by different tags. Think about this scenario: we find the service have many high-latency traces, and we also known the service was deployed to two different regions(maybe region-a and region-b). So we can search the high-latency traces through region tag like region=region-a.
Maybe add the global tag to first span of trace segment is good enough.

[wu-sheng]

Maybe add the global tag to first span of trace segment is good enough.

Yes, from your case, this seems making more sense. Injecting to first span has low payload to the core. We need to be very careful about the performance impact. Because what you proposed is going to affect all users one way or another, this is core level change.

[wallezhang]

OK, I will try my best to reduce the impact of performance.

[creeew]

Since segment supported global tag for filtering different region. Maybe we could filter metrics data by this global tag. Like we got two data center: region-a and region-b, and in this different regions deployed same service. We cannot figure out high latency service in which region if we are not support filter by global tag.
I've finshed my code filter metics data by tag in APM/Database/Web Browser and topology/trace/log. Global metrics like percentile/heatmap are calcuated by tag.
@wu-sheng If you think this make sense I can commit my codes base on @wallezhang 's global tag.

[wu-sheng]

That is why I am very concerned about this, users will begin to abuse about this kind of tag.
SkyWalking is not dealing with tenancy or cross clusters, because no concept about this, both UI and community don't support that.
So, I would like to call for a hold about this.

In the next few days, I am going to write a fully proposal about new v9.0.0 core and next booster UI (@Fine0830 leads). We are still not having plans about tenancy and cross clusters, because those two will introduce many management concepts, which are different in different countries and companies.

So, let's not push the project to a hard-to-all-users but convince-for-small-group status. SkyWalking is already a huge project, we encourage upstream first policy for common use cases, but it makes sense users have their forks and extensions.

[wallezhang]

@wu-sheng So should I pause this issue?

[wu-sheng]

I would recommend to keep this in the discussion channel for now. I am not so confident about how the community would use this. I am concerned we will be blamed due to performance impact and more customization.

[wallezhang]

OK, I will suspend development and let's take a look at the community discussion on this.