[Synthetic monitoring] Make SkyWalking as a core component when doing mock traffic testing on product env.

[stateIs0]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion
• Bug
• Requirement
• Feature or performance improvement

---

Question

• What do you want to know?
  Sorry, first of all, my English is not good, this is the use of software translation. The specific issues are as follows:
  Why doesn't Skywalking's plug-in use filter chain mode? In current mode, if you want to enhance the same class twice, you may not be able to determine the order and may even report an error.

Would it be more extension-friendly if you could support two enhancements to the same class, and control the order? In fact, in our project, we have already started to modify this piece to support filter chains.

[wu-sheng]

The key question is why? Filter and enhancement is a very expansive operation, especially in the runtime. For what purpose, you want to enhance a class twice and for a specific method?

[wu-sheng]

Current codes work well if you are targeting different methods.

[stateIs0]

Let me talk about the application scenario: the Jedis plugin for Skywalking, for example, now only has the call chain function. Now we have a new requirement, full link test(全链路压测), we need to create the shadow database, that is, during the Jedis call, we need to intercept and switch the shadow database, this logic, we do not want to put together with the previous Jedis plug-in, we want to be a new, separate plug-in.

[stateIs0]

@wu-sheng

[wu-sheng]

@tristaZero See! What I was saying :) People are using this case.

@stateIs0 If you are talking about shadow database and mock data flag propagation, that is a thing I have designed and discussed with Apache ShardingSphere. I hope they could collaborate to process this, which will be much cooler and no need for the thing you asked about repeated instrumentation.

[stateIs0]

Thank you. Now I want to discuss two issues:

1. In fact, we have made some changes on the branch of Skywalking to support the filter chain function. If possible, I wonder if we can submit PR?
2. If you are now designing a better scheme for the full link compression test, we look forward to your scheme. In addition, could you please tell us what your scheme looks like?

[stateIs0]

@wu-sheng

[wu-sheng]

In fact, we have made some changes on the branch of Skywalking to support the filter chain function. If possible, I wonder if we can submit PR?
If you are now designing a better scheme for the full link compression test, we look forward to your scheme. In addition, could you please tell us what your scheme looks like?

We haven't opened the design. @tristaZero I hope we could work together on this. Are you sure we could do this now?
But I could tell you, YES, you don't need to change SkyWalking much, and use SW with ShardingSphere proxy, you could have this out of the box. I am not sure what you have done, could you share some with us? How you do this? I am definitely willing to combine your practice with our WIP design.

[tristaZero]

Hi @wu-sheng ,

Right...I stand on your side. It is a great feature worthy of two communities' collaboration.

@stateIs0 Hi, please firstly give a look at Shadow DB;

[stateIs0]

@wu-sheng
First of all, I'm not sure what your solution with ShardingSphere is, but my feeling is that it would be better not to introduce other components.
Then, let me talk briefly about our current solution:

1. Collect all plug-ins with class as key.
2. When calling a method, find an interceptor matching the method according to method.
3. Find these interceptors and build a chain of filters. And cache the filter chain.
4. Make a call to the filter chain

The pseudocode is as follows:

@RuntimeType
public Object trace(@This Object obj,
                        @AllArguments Object[] allArguments,
                        @Origin Method method,
                        @SuperCall Callable<?> zuper) throws Throwable {
        if (cache.get(method) != null) {
            return cache.get(method).invoke(new ParamModel(obj, allArguments, method, zuper));
        }
        List<Builder.Interceptor> interceptorList = new ArrayList<>();
        for (Builder.Interceptor interceptor : interceptors) {
            List<ElementMatcher<MethodDescription>> matches = interceptor.matches();
            for (ElementMatcher<MethodDescription> match : matches) {
                // 找到能够匹配这个方法的拦截器.
                if (match.matches(new MethodDescription.ForLoadedMethod(method))) {
                    interceptorList.add(interceptor);
                }
            }
        }
        // 排序
        interceptorList.sort(Comparator.comparingInt(Builder.Interceptor::order));
        List<ChainFactory.Filter<Object, ParamModel>> list = new ArrayList<>();
        // 将每个拦截器用 filter 包装起来
        for (Builder.Interceptor interceptor : interceptorList) {
            list.add((target, arg) -> {
                // before
                interceptor.beforeMethod(arg.obj, arg.method, arg.allArguments, method.getParameterTypes(), new MethodInterceptResult());
                Object result = null;
                try {
                    // 下个过滤器进行调用
                    result = target.invoke(arg);
                } catch (Throwable throwable) {
                    interceptor.handleMethodException(arg.obj, arg.method, arg.allArguments, method.getParameterTypes(), throwable);
                }
                // after
                interceptor.afterMethod(arg.obj, arg.method, arg.allArguments, method.getParameterTypes(), result);
                return result;
            });
        }

        //  构建过滤器链, 并添加尾部节点. 尾部节点直接调用目标方法.
        ChainFactory.Invoker<Object, ParamModel> chain = ChainFactory.build(list, paramModel -> paramModel.zuper.call());

        // 发起调用.
        Object result = chain.invoke(new ParamModel(obj, allArguments, method, zuper));
        return result;
}

[stateIs0]

Hi @wu-sheng ,

Right...I stand on your side. It is a great feature worthy of two communities' collaboration.

@stateIs0 Hi, please firstly give a look at Shadow DB;

Hello, thanks for your advice. However, we are not only a database, but also Shadow libraries such as ES, Redis, Pika, MQ and so on. Shadow DB is not a universal solution

[wu-sheng]

I was not asking about this multi-interceptors support, I was asking about are you adding this because you could route the data into the shadow database?

[stateIs0]

I was not asking about this multi-interceptors support, I was asking about are you adding this because you could route the data into the shadow database?

yes， i need

[wu-sheng]

I was not asking about this multi-interceptors support, I was asking about are you adding this because you could route the data into the shadow database?

yes， i need

I know you need, my question is whether this is your only reason to do so?

[stateIs0]

@wu-sheng @tristaZero
Excuse me, I wonder if you could give me some relevant advice? Shadow DB feels that it does not meet this demand, and does the design of this filter conform to the development path of Skywalking? If not, which direction is appropriate? Looking forward to your reply.

[stateIs0]

According to my understanding, there are also some existing full-link test products on the market. For data such as REDIS, ES, MQ, etc., shadow library is required, otherwise, the online business logic will be affected.

As I said about how SkyWalking gets what it is today, you could easily say I want A to Z to unlimited, but, nothing would be there if you or someone gets hands dirty. That is the base principle of open source.
If I said I want to build SkyWalking likes today from 6 years ago, I would not know to do anything, don't I?
Talking is a super easy thing, but get it done, and getting it done in the right, cool and extendable way are the key to a widely adopted and used open source project.
So, keep your mind open, take one step at a time. Don't say, that solution is too complex, I just want to change SkyWalking's javaagent to get it done, so, a better solution doesn't matter.
I could give a short question to challenge your way? If a company has multiple language services, how it could do so? Changing all language agents again and again?

yes，you are right，Thank you for your reply. 😀

[wu-sheng]

Let's keep this issue open, if someone at ShardingSphere side is ready to join, we could initialize the process.

[stateIs0]

Let's keep this issue open, if someone at ShardingSphere side is ready to join, we could initialize the process.

Looking forward to your design proposal very much！

[daimingzhi]

Excuse me, I ran into a similar problem, but, not a full link pressure testing scenario。
We have written many javaAgent based on skywalking。At the beginning，these java agents work in isolation，like this:

-javaagent:skywalking.jar -javaagent:subEnvAgent.jar -javaagent:infoCollectAgent.jar

the subEnvAgent and infoCollectAgent is written based on skywalking。

However, as the number of agents grows, it becomes more difficult to maintain all agents.

In the end, we decided to merge all agents to one。Because of this, the problems mentioned above comed。

skywalkingAgent and infoCollectAgent both define a plugin for the same class at the same time, but skywalking 's responsibility is for trace collection, and infoCollectAgent is to collect other information。At this time, only one plugin will take effect。

To solve this problem, we made a change similar to @stateIs0 has did。

In my opinion, maybe for extension of skywalking, multi-interceptors should be supported。

[wu-sheng]

In my opinion, maybe for extension of skywalking, multi-interceptors should be supported。

Hi, if you want to discuss, more importantly, contribute this feature), please open another issue to discuss. This issue is purely tracking for synthetic monitoring. We are working on this way, check the linked issues from ShardingSphere community.

[daimingzhi]

Hi, if you want to discuss, more importantly, contribute this feature

Yes,I am glad to do this. I will open another issue later

[tristaZero]

Hi @daimingzhi ,

Thanks for your attention. It is on the way to do mock traffic test on pro env leveraging Skywalking and ShardingSphere.
To achieve it, two communities have their own work. If you want to learn more about how ShardingSphere is going, please refer to #11661.

Plus, I guess we need more future communication among all the people involved.

[wu-sheng]

Hi @daimingzhi ,

Thanks for your attention. It is on the way to do mock traffic test on pro env leveraging Skywalking and ShardingSphere.
To achieve it, two communities have their own work. If you want to learn more about how ShardingSphere is going, please refer to #11661.

Plus, I guess we need more future communication among all the people involved.

@tristan-tsl I think @daimingzhi was asking for a different thing, but matched here because there was a potential proposal having a similar feature requirement.

Besides that, I agree that, we should set up a GitHub org to host all Synthetic Monitoring related things, otherwise, people are going to confuse where they are showing up to talk about this specific use case.

[tristaZero]

I see, it is true that people will confuse when asking questions.
BTW, @wu-sheng You tagged another person, not me. ;-(

[daimingzhi]

I think @daimingzhi was asking for a different thing,

Yes,My current job is to maintain our company's javaAgents based on SkyWalking, @tristaZero I am afraid I do not have enough time to know more about ShardingSphere. What I said above is to better the extension of SkyWalking.

@wu-sheng I hope I can participate in the development or maintenance of the skyWalking kernel module in the future.Maybe this is my first step.I have opend another issue Discussion about multi-interceptors supporting in SkyWalking

[vwvm]

支持一下

[wu-sheng]

@tristaZero @mrproliu I am moving this to discussion panel, welcome to continue there.

As this is going to be a separate release project, SkyWalking community would recommend users to continue on that repo once Juan creates that.

[tristaZero]

Hi @wu-sheng and guys,

Sorry for my delayed reply, but good news, we almost finish refactoring Shadow database feature. ;-) @soulasuna will update more later. Look forward to our next step.

[tristaZero]

Cyborg, a creature that is partly human and partly machine?
Flow sounds great.
How about FlowObserver? Traffic Observer?

[wu-sheng]

Observer seems too related to SkyWalking?
Cyborg usually means hybrid or hals-half. Isn't this what we are going to do? Most traffic are from product environment, and partly are generated, routine and observing by us, right?

[tristaZero]

Exactly. ;-)
I mean observer does not only observe APP flow but includes the flow through the databases.
I am considering how to add more functional elements to its name. 🤔

[wu-sheng]

In my naming style, CyborgFlow is enough, key words included and not too many.

[tristaZero]

Sorry, I can not have a better idea, and actually CyborgFlow looks fine. So CyborgFlow Repo showed up 🎉 guys! Hi @wu-sheng your idea kicked off this project, I guess!

[wu-sheng]

Conclusion to this discussion, all following process moves to official repo, https://github.com/SphereEx/CyborgFlow

[tristaZero]

Hi, update here.
CyborgFlowTeam was created. ;-)