|
1 | | -<br/> |
| 1 | +<br/> |
2 | 2 |
|
3 | 3 |  |
4 | 4 |
|
5 | 5 | <br/> |
6 | 6 |
|
7 | | -## Introduction |
| 7 | +## Introduction |
| 8 | + |
| 9 | +_nTangle_ is a Change Data Capture (CDC) code generation tool and corresponding runtime. Unlike other CDC-based technologies which replicate changes to rows _nTangle_ is designed to replicate business entities (aggregate) changes. |
| 10 | + |
| 11 | +For example if a database contains a `Person` and one-to-many related `Address` table, a traditional CDC replicator would leverage the CDC-capabilities of the database as the data source and replicate all changes from both tables largely distinct from each other. Additional logic would then be required within the downstream systems to aggregate these distinct changes back into a holistic business entity, if possible. |
| 12 | + |
| 13 | +_nTangle_ tackles this differently by packaging the changes at the source into an aggregated entity which is then replicated. With _nTangle_ the CDC-capabilities of the database are leveraged as the trigger, with a corresponding query across all related tables to produce a holistic business entity. Therefore, if a change is made to `Person` or `Address` this will result in the publishing of the entity. Where transactional changes are made to both `Person` and `Address` a single holistic business entity will be published including all changes. |
| 14 | + |
| 15 | +This has a key advantage of being an excellent candidate within event-streaming scenarios where business entities are to be published based on underlying database changes. |
| 16 | + |
| 17 | +<br/> |
| 18 | + |
| 19 | +## Status |
| 20 | + |
| 21 | +[](https://github.com/Avanade/NTangle/actions?query=workflow%3ACI) [](https://badge.fury.io/nu/NTangle) |
| 22 | + |
| 23 | +The included [change log](CHANGELOG.md) details all key changes per published version. |
| 24 | + |
| 25 | +<br/> |
| 26 | + |
| 27 | +## Approach |
| 28 | + |
| 29 | +The _nTangle_ CDC approach taken here is to consolidate the tracking of individual tables (one or more) into a aggregated _entity_ to simplify the publishing to an event stream (or equivalent). The advantage of this is where a change occurs to any of the rows related to an entity, even where multiples rows are updated, this will only result in a single event. This makes it easier (more logical) for downstream subscribers to consume. |
| 30 | + |
| 31 | +This is achieved by defining (configuring) the entity, being the primary (parent) table, and its related secondary (child) tables. For example, a `SalesOrder`, may be made up multiple tables - when any of these change then a single `SalesOrder` event should occur. These relationships are also defined with a cardinality of either `OneToMany` or `OneToOne`. |
| 32 | + |
| 33 | +``` |
| 34 | +SalesOrder // Parent |
| 35 | +└── SalesOrderAddress // Child 1:n - One or more addresses (e.g. Billing and Shipping) |
| 36 | +└── SalesOrderItem // Child 1:n - One or more items |
| 37 | +``` |
| 38 | + |
| 39 | +The CDC capability is used specifically as a trigger for change (being `Create`, `Update` or `Delete`). The resulting data that is published is the latest, not a snapshot in time (CDC captured). The reason for this is two-fold: |
| 40 | +1. Given how the CDC data is retrieved there is no guarantee that the interim data represents a final intended state suitable for publishing; and, |
| 41 | +1. This process is intended to be running near real-time so getting the latest version will produce the most current committed version as at that time. |
| 42 | + |
| 43 | +To further guarantee only a single event for a specific version is published the resulting _entity_ is JSON serialized and hashed; this value is checked (and saved) against the prior version to ensure a publish contains data that is actionable. This will minimize redundant publishing, whilst also making the underlying processing more efficient. |
| 44 | + |
| 45 | +<br/> |
| 46 | + |
| 47 | +## Additional reading |
| 48 | + |
| 49 | +This [article](https://www.mssqltips.com/sqlservertip/5212/sql-server-temporal-tables-vs-change-data-capture-vs-change-tracking--part-2/) provides an excellent overview of the Microsoft SQL Server CDC-capabilities and walks through the process of setting up and using to aid in the fundamental understanding. |
| 50 | + |
| 51 | +Although throughout references will be made to [Microsoft SQL Server](https://www.microsoft.com/en-us/sql-server), the intention of _nTangle_ is that it is largely agnostic to the database technology, and therefore support for other databases will (or may) be supported in the future based on demand. |
| 52 | + |
| 53 | +<br/> |
| 54 | + |
| 55 | +## Capabilities |
| 56 | + |
| 57 | +_nTangle_ has been created to provide a seamless means to create CDC-enabled aggregated entity publishing solution. The _nTangle_ solution is composed of the following: |
| 58 | + |
| 59 | +1. [Code generation](#Code-generation) - a configuration file defines the database tables and, none or more relationships, which also includes other functionality-based properties, that are used to drive the database-driven code-generation to create the required solution artefacts. |
| 60 | +2. [Runtime](#Runtime) - the generated solution artefacts leverage a number of .NET runtime components/capabilities to support and enable. The code-generated solution then uses these at runtime to execute and orchestrate the CDC-triggered aggregated entity publishing process. |
| 61 | + |
| 62 | +<br/> |
| 63 | + |
| 64 | +### Code-generation |
| 65 | + |
| 66 | +The code-generation is managed via a console application using the [`CodeGenConsole`](./src/NTangle/Console/CodeGenConsole.cs) to manage. This internally leverages [OnRamp](https://github.com/Avanade/onramp) to enable. |
| 67 | + |
| 68 | +Additionally, the code-generator inspects (queries) the database to infer the underlying table schema for all tables and their columns. This is used as a source in which the configuration references to validate, whilst also minimizes configuration where the inferred schema information can be used. The code-generation adopts a gen-many philosophy, therefore where schema changes are made, the code-generation can be executed again to update accordingly. |
| 69 | + |
| 70 | +As stated, the code-generation is driven by a configuration file, typically named `ntangle.yaml`. Both YAML and JSON formats are supported; there is also a corresponding [JSON schema](./schemas/ntangle.json) to enable editor intellisense, etc. |
| 71 | + |
| 72 | +The _nTangle_ configuration is as follows: |
| 73 | + |
| 74 | +``` |
| 75 | +Root |
| 76 | +└── Table(s) |
| 77 | + └── Join(s) |
| 78 | + └── JoinOn(s) |
| 79 | + └── JoinMapping(s) |
| 80 | + └── TableMapping(s) |
| 81 | +``` |
| 82 | + |
| 83 | +Documentation related to each of the above are as follows: |
| 84 | +- [`Root`](./docs/generated/root.md) - defines the root configuration settings. |
| 85 | +- [`Table`](./docs/generated/table.md) - defines the primary table as being the entity aggregate. |
| 86 | +- [`Join`](./docs/generated/join.md) - defines none or more table joins to include within the entity. |
| 87 | +- [`JoinOn`](./docs/generated/joinon.md) - defines the join on column characteristics. |
| 88 | +- [`JoinMapping`](./docs/generated/joinmapping.md) - defines global identifier mappings for any of the join table columns. |
| 89 | +- [`TableMapping`](./docs/generated/tablemapping.md) - - defines global identifier mappings for any of the primary table columns. |
| 90 | + |
| 91 | +An example [ntangle.yaml](./samples/SqlServerDemo/SqlServerDemo.CodeGen/ntangle.yaml) configuration file exists within the [`SqlServerDemo`](./samples/SqlServerDemo) sample. The [`SqlServerDemo.CodeGen`](./samples/SqlServerDemo/SqlServerDemo.CodeGen) sample also demonstrates how to invoke the code generator from the underlying [`Program`](./samples/SqlServerDemo/SqlServerDemo.CodeGen/Program.cs). |
| 92 | + |
| 93 | +The code-generator will output a number of generated artefacts (see [`SqlServerDemo.Database`](./samples/SqlServerDemo/SqlServerDemo.Database) sample); these will be either database-related or additional .NET runtime components (see [`SqlServerDemo.Publisher`](./samples/SqlServerDemo/SqlServerDemo.Publisher) sample). |
| 94 | + |
| 95 | + |
| 96 | +The following [`NTangle`](./src/NTangle) namespaces provide code-generation capabilties: |
| 97 | + |
| 98 | +Namespace | Description |
| 99 | +- | - |
| 100 | +[`Config`](./src/NTangle/Config) | The _internal_ code that supports the YAML/JSON configuration. |
| 101 | +[`Console`](./src/NTangle/Console) | The code-generation tooling capabilities, primarily [`CodeGenConsole`](./src/NTangle/Console/CodeGenConsole.cs). |
| 102 | +[`Generators`](./src/NTangle/Generators) | The _internal_ code-generators used to select configuration for one or more templates. |
| 103 | + |
| 104 | +<br/> |
| 105 | + |
| 106 | +### Runtime |
| 107 | + |
| 108 | +Generally, a runtime publisher is required to orchestrate the CDC-triggered aggregated entity publishing process (see [`SqlServerDemo.Publisher`](./samples/SqlServerDemo/SqlServerDemo.Publisher) sample). This in turn takes a dependency on the _nTangle_ runtime to enable. |
| 109 | + |
| 110 | +The following [`NTangle`](./src/NTangle) namespaces provide runtime capabilties: |
| 111 | + |
| 112 | +Namespace | Description |
| 113 | +- | - |
| 114 | +[`Cdc`](./src/NTangle/Cdc) | The CDC-orchestration capabilities, primarily [`EntityOrchestrator`](./src/NTangle/Cdc/EntityOrchestrator.cs). |
| 115 | +[`Data`](./src/NTangle/Data) | The database access capabilities, primarily [`Database`](./src/NTangle/Data/Database.cs). |
| 116 | +[`Events`](./src/NTangle/Events) | The event capabilities, primarily [`IEventPublisher`](./src/NTangle/Events/IEventPublisher.cs) and [`CloudEventSerializer`](./src/NTangle/Events/CloudEventSerializer.cs). |
| 117 | +[`Services`](./src/NTangle/Services) | The service hosting capabilities, primarily [`HostedService`](./src/NTangle/Services/HostedServiceT.cs). |
| 118 | + |
| 119 | +<br/> |
| 120 | + |
| 121 | +## Samples |
| 122 | + |
| 123 | +The following samples are provided to guide usage: |
| 124 | + |
| 125 | +Sample | Description |
| 126 | +-|- |
| 127 | +[`SqlServerDemo`](./samples/SqlServerDemo) | A sample as an end-to-end solution walkthrough to demonstrate the usage of _nTangle_ against a Microsoft SQL Server database. |
| 128 | + |
| 129 | +<br/> |
| 130 | + |
| 131 | +## License |
| 132 | + |
| 133 | +_OnRamp_ is open source under the [MIT license](./LICENSE) and is free for commercial use. |
| 134 | + |
| 135 | +<br/> |
| 136 | + |
| 137 | +## Contributing |
| 138 | + |
| 139 | +One of the easiest ways to contribute is to participate in discussions on GitHub issues. You can also contribute by submitting pull requests (PR) with code changes. Contributions are welcome. See information on [contributing](./CONTRIBUTING.md), as well as our [code of conduct](https://avanade.github.io/code-of-conduct/). |
| 140 | + |
| 141 | +<br/> |
| 142 | + |
| 143 | +## Security |
| 144 | + |
| 145 | +See our [security disclosure](./SECURITY.md) policy. |
| 146 | + |
| 147 | +<br/> |
| 148 | + |
| 149 | +## Who is Avanade? |
| 150 | + |
| 151 | +[Avanade](https://www.avanade.com) is the leading provider of innovative digital and cloud services, business solutions and design-led experiences on the Microsoft ecosystem, and the power behind the Accenture Microsoft Business Group. |
0 commit comments