Refactoring plan of the shardingsphere-shadow project

[soulasuna]

Hi,community:

I am very happy to bring you the refactoring plan of theshardingsphere-shadow project. @terrymanu @tristaZero @wu-sheng

Combining the characteristics of the shardingsphere-shadow project's own functions and the configuration style of other functions of the shardingsphere.Make changes to the user API configuration.

In the future, shardingsphere-shadow will become an important part of data routing in full-link stress test scenarios.

The process of project reconstruction will not have any impact on the original functions.

After the refactoring is completed, the user API configuration changes are as follows:

• Spring-namespace xml file configuration

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:shadow="http://shardingsphere.apache.org/schema/shardingsphere/shadow"
       xsi:schemaLocation="http://www.springframework.org/schema/beans 
                           http://www.springframework.org/schema/beans/spring-beans.xsd
                           http://shardingsphere.apache.org/schema/shardingsphere/shadow
                           http://shardingsphere.apache.org/schema/shardingsphere/shadow/shadow.xsd
                           ">
    <shadow:shadow-algorithm id="user-id-match-algorithm" type="COLUMN_REGEX_MATCH">
        <props>
            <prop key="operation">insert</prop>
            <prop key="column">user_id</prop>
            <prop key="regex">[1]</prop>
        </props>
    </shadow:shadow-algorithm>
    <shadow:shadow-algorithm id="simple-note-algorithm." type="SIMPLE_NOTE">
        <props>
            <prop key="shadow">true</prop>
            <prop key="foo">bar</prop>
        </props>
    </shadow:shadow-algorithm>

    <shadow:rule id="shadowRule" enable="true">
        <shadow:data-source id="shadow-data-source" source-data-source-name="ds" shadow-data-source-name="ds-shadow"/>
        <shadow:shadow-table name="t_order" data-sources="shadow-data-source">
            <shadow:algorithm shadow-algorithm-ref= "user-id-match-algorithm" />
            <shadow:algorithm shadow-algorithm-ref= "simple-note-algorithm" />
        </shadow:shadow-table>
        <shadow:shadow-table name="t_user" data-sources="shadow-data-source">
            <shadow:algorithm shadow-algorithm-ref= "simple-note-algorithm" />
        </shadow:shadow-table>
    </shadow:rule>
</beans>

• Spring-boot properties file configuration

spring.shardingsphere.rules.shadow.enable=true
spring.shardingsphere.rules.shadow.data-sources.shadow-data-source.source-data-source-name=ds
spring.shardingsphere.rules.shadow.data-sources.shadow-data-source.shadow-data-source-name=ds-shadow

spring.shardingsphere.rules.shadow.tables.t_order.data-source-names=shadow-data-source
spring.shardingsphere.rules.shadow.tables.t_order.shadow-algorithm-names=user-id-match-algorithm,simple-note-algorithm
spring.shardingsphere.rules.shadow.tables.t_user.data-source-names=shadow-data-source
spring.shardingsphere.rules.shadow.tables.t_user.shadow-algorithm-names=simple-note-algorithm

spring.shardingsphere.rules.shadow.shadow-algorithms.user-id-match-algorithm.type=COLUMN_REGEX_MATCH
spring.shardingsphere.rules.shadow.shadow-algorithms.user-id-match-algorithm.props.operation=insert
spring.shardingsphere.rules.shadow.shadow-algorithms.user-id-match-algorithm.props.column=user_id
spring.shardingsphere.rules.shadow.shadow-algorithms.user-id-match-algorithm.props.regex=[1]

spring.shardingsphere.rules.shadow.shadow-algorithms.simple-note-algorithm.type=SIMPLE_NOTE
spring.shardingsphere.rules.shadow.shadow-algorithms.simple-note-algorithm.props.shadow=true
spring.shardingsphere.rules.shadow.shadow-algorithms.simple-note-algorithm.props.foo=bar

• Spring-boot yml file configuration

rules:
  shadow:
    enable: true
    data-sources:
      shadow-data-source:
        source-data-source-name: ds
        shadow-data-source-name: ds-shadow
    tables:
      t_order:
        data-source-names: shadow-data-source
        shadow-algorithm-names: 
          - user-id-match-algorithm
          - simple-note-algorithm
      t_user:
        data-source-names: shadow-data-source
        shadow-algorithm-names:
          - simple-note-algorithm 
    shadow-algorithms:
      user-id-match-algorithm:
        type: COLUMN_REGEX_MATCH
        props:
          operation: insert
          column: user_id
          regex: "[1]"
      simple-note-algorithm:
        type: SIMPLE_NOTE
        props:
          shadow: true
          foo: bar

[tristaZero]

@wu-sheng Hi, huge progress here.
@soulasuna Nice job.

[wu-sheng]

Hi, thanks for updating.
The term of this feature in native speaking should be Synthetic monitoring. SkyWalking is tracking through apache/skywalking#6932.

FYI @mrproliu as you asked about this.

[wu-sheng]

@soulasuna Is note-algorithmmeaning the API level rule? Which SkyWalking agent could inject?

[wu-sheng]

@tristaZero We should have a resolution where we should land the final status of this huge across-TLPs integration.
Such as which repo we should land? Because after the core level changes, we need to have a deep integration, and a separate documentation site.

[soulasuna]

@wu-sheng The API shown now only uses the configuration method of ShardingSphere.
The routing algorithm will support SQL annotations.

@soulasuna Is note-algorithmmeaning the API level rule? Which SkyWalking agent could inject?

[wu-sheng]

The routing algorithm will support SQL annotations.

Are you going to use the SkyWalking v3 standard header in the SQL comments?

[soulasuna]

Hi,community:
I am very happy that the shadow refactoring plan is completed. Thank you for your attention.