Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- ETL request errors now use RFC 9457 `application/problem+json` responses with a stable `errorCode`, fixed type URI, explicit 400/401/404/409/413/422/503/500 taxonomy, and no internal exception text in client responses.
- ETL requests now enforce bounded UTF-8 payload and record-count limits, prevalidate and transform the complete batch before the first JDBC call, and commit accepted records inside one Spring transaction.
- ETL transformations now preserve comma/colon-bearing values, use locale-independent text conversion and deterministic `BigDecimal` amount formatting, and retry only transient Spring data-access failures.
- The SQL Server Debezium scaffold is no longer a Spring component or production-discoverable source. `SqlServerDebeziumCdcSource` remains explicit reference/design code only, and configured `sqlserver-debezium` use fails closed as `unknown_source_type` until a maintained implementation and operational evidence exist.
- Product branding: user-facing docs and suggested image tags use **mightyETL** (formerly xtrmETL).
- Legacy Java packages (`com.xtrmetl.*`), Maven `artifactId` `xtrmETL`, and some env/topic defaults remain for compatibility.
- See `docs/rebrand-name-matrix.md`.
Expand Down Expand Up @@ -44,7 +45,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `GET /api/cdc/targets` for target SPI discovery.
- Actuator `cdcEngine` health indicator (engine running + slot details).
- SPI lifecycle: `PostgresDebeziumCdcSource.start/stop` delegates to `CdcService`.
- Scaffold CDC sources: `mysql-debezium`, `sqlserver-debezium` (discovery only).
- CDC scaffold references: `mysql-debezium` remains discovery-only; `sqlserver-debezium` is reference-only and not production-discoverable.
- Root POM `<name>mightyETL</name>` (artifactId remains `xtrmETL`).
- README honest “Supported today” matrix; compose file product-name header.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
package com.xtrmetl.cdc.spi;

import org.springframework.stereotype.Component;

import java.util.Set;

/**
* Scaffold for SQL Server CDC via Debezium. Not on the classpath / not started.
* Reference scaffold for SQL Server CDC via Debezium.
*
* <p>This type is intentionally not registered as a Spring component while its runtime start path
* remains scaffold-only. Construct it explicitly only for design/reference validation until a
* maintained SQL Server Debezium implementation and operational evidence exist.
*/
@Component
public final class SqlServerDebeziumCdcSource extends AbstractScaffoldCdcSource {

public static final String ID = "sqlserver-debezium";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,38 +9,50 @@
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertTrue;

/**
* Verifies configured CDC source resolution against the production-visible registry contract.
*/
class CdcSourceFactoryTest {

@Test
void resolvesPostgresDebezium() {
CdcSourceFactory factory = new CdcSourceFactory(new CdcSourceRegistry(List.of(
new PostgresDebeziumCdcSource(),
new MysqlDebeziumCdcSource()
)));
CdcSourceFactory factory = productionVisibleFactory();
assertTrue(factory.resolve("postgres-debezium").isPresent());
assertTrue(factory.resolve("POSTGRES-DEBEZIUM").isPresent());
assertTrue(factory.resolve("mysql-debezium").isPresent());
assertTrue(factory.resolve("oracle-debezium").isEmpty());
assertTrue(factory.resolve("sqlserver-debezium").isEmpty());
}

@Test
void describeConfiguredMarksUnknownAndScaffoldTypes() {
CdcSourceFactory factory = new CdcSourceFactory(new CdcSourceRegistry(List.of(
new PostgresDebeziumCdcSource(),
new MysqlDebeziumCdcSource()
)));
void describeConfiguredMarksUnknownScaffoldAndRetiredSqlServerTypes() {
CdcSourceFactory factory = productionVisibleFactory();
List<Map<String, Object>> rows = factory.describeConfigured(List.of(
new CdcSourceFactory.SourceSpec("pg-main", "postgres-debezium", true),
new CdcSourceFactory.SourceSpec("mysql-1", "mysql-debezium", true),
new CdcSourceFactory.SourceSpec("ora-1", "oracle-debezium", true)
new CdcSourceFactory.SourceSpec("ora-1", "oracle-debezium", true),
new CdcSourceFactory.SourceSpec("sqlserver-1", "sqlserver-debezium", true)
));

assertEquals(3, rows.size());
assertEquals(4, rows.size());
assertEquals(true, rows.get(0).get("registered"));
assertFalse((Boolean) rows.get(0).get("scaffoldOnly"));
assertEquals(true, rows.get(1).get("registered"));
assertTrue((Boolean) rows.get(1).get("scaffoldOnly"));
assertEquals(false, rows.get(2).get("registered"));
assertEquals("unknown_source_type", rows.get(2).get("error"));
assertEquals("sqlserver-1", rows.get(3).get("id"));
assertEquals("sqlserver-debezium", rows.get(3).get("type"));
assertEquals(true, rows.get(3).get("enabled"));
assertEquals(false, rows.get(3).get("registered"));
assertTrue((Boolean) rows.get(3).get("scaffoldOnly"));
assertEquals("unknown_source_type", rows.get(3).get("error"));
}

private static CdcSourceFactory productionVisibleFactory() {
return new CdcSourceFactory(new CdcSourceRegistry(List.of(
new PostgresDebeziumCdcSource(),
new MysqlDebeziumCdcSource()
)));
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import com.xtrmetl.cdc.service.CdcService;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.Map;
Expand Down Expand Up @@ -43,6 +44,11 @@ void postgresSpiStartAndStopDelegateToCdcService() throws Exception {
verify(service).stop();
}

@Test
void sqlServerScaffoldIsNotAutoDiscoveredAsAProductionSource() {
assertFalse(SqlServerDebeziumCdcSource.class.isAnnotationPresent(Component.class));
}

@Test
void registersScaffoldSourcesWhenProvided() {
CdcSourceRegistry registry = new CdcSourceRegistry(List.of(
Expand Down
10 changes: 8 additions & 2 deletions docs/cdc/any-to-any-cdc.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,17 +74,23 @@ xtrmetl:
type: postgres-debezium
enabled: true
# - id: mysql-1
# type: mysql-debezium # not registered — status will show unknown_source_type
# type: mysql-debezium # scaffold only
# enabled: false
# - id: sqlserver-1
# type: sqlserver-debezium # not registered — status reports unknown_source_type
# enabled: false
```

- **Declared** in `application.yml` / `XtrmetlProperties.Cdc.sources` and reported on `GET /api/cdc/status` as `configuredSources` via `CdcSourceFactory`.
- **Registered types:**

| Type id | Status |
|:--------|:-------|
| `postgres-debezium` | **Live** — SPI `start`/`stop` delegates to `CdcService` |
| `mysql-debezium` | Scaffold only (no connector JAR) |
| `sqlserver-debezium` | Scaffold only (no connector JAR) |
| `sqlserver-debezium` | **Not registered** — reference scaffold only; configured use reports `unknown_source_type` |
Comment thread
seonghobae marked this conversation as resolved.

- `SqlServerDebeziumCdcSource` remains explicit design/reference code only. It is not a Spring component and is not production-discoverable until a maintained SQL Server Debezium implementation, lifecycle, offset/schema-history handling, recovery behavior, and realistic integration evidence exist.
- **Live engine:** still a single Debezium Postgres path in `CdcService` (multi-source start not implemented).
- Targets remain: Kafka (live raw publish) + optional JDBC replica (`replica.enabled`).

Expand Down
Loading