Skip to content

feat: edfs refactor kafka and nats with a datasource per field#1848

Merged
alepane21 merged 123 commits intomainfrom
ale/eng-6482-edfs-refactor-filtered-datasources
Jun 6, 2025
Merged

feat: edfs refactor kafka and nats with a datasource per field#1848
alepane21 merged 123 commits intomainfrom
ale/eng-6482-edfs-refactor-filtered-datasources

Conversation

@alepane21
Copy link
Copy Markdown
Contributor

@alepane21 alepane21 commented May 12, 2025

Motivation and Context

Today changing or adding an EDFS provider requires changes to cosmo and the engine, making it harder than it should.
The main point of this PR is to make it easier to add future EDFS providers.

So I moved all the pubsub code from the engine to the pubsub package in the router and refactored it to make more sense with the new structure.

There is a README that will help implementing new EDFS providers (manually or using AI).

Important points

  • I added a test to verify that kafka publish is working, there was nothing testing it before;
  • connections operations (connection, closing connection) should happen as before;
  • switching from a previous version to the new one should just works, without any change to the configuration or schema.

Design

Interface Responsibility
ProviderBuilder Allows the router to build a Provider starting from DataSourceConfiguration and router event config.
Provider Owns provider‑wide resources for a single providerId (e.g. NATS connection pool).
EngineDataSourceFactory Implements ResolveDataSource + SubscribeDataSource for one GraphQL field.
Adapter Thin, protocol‑specific driver that actually publishes / subscribes.

Execution flow

  • When the router’s Loader (router/core/factoryresolver.go) encounters a PubSub nodes, it collects them and build the data sources and PubSubProviders by passing all the collected datasources to the PubSubProviderBuilder
  • The PubSubProviderBuilder will create a single PubSubDataSource for each root node with its adapter and event data already assigned
    • this will generate more data sources, but will reduce the operations to do after the schema has been loaded
  • The adapter will hands off the actual publish / subscribe work to the appropriate external driver (Kafka, NATS, etc.).

Checklist

alepane21 and others added 30 commits March 3, 2025 13:01
…and-nats-as-datasource-in-router

# Conflicts:
#	router/core/plan_generator.go
…and planner files, enhance NATS subject validation
…actor-kafka-and-nats-as-datasource-in-router
…ment

- Added InstanceData struct to encapsulate hostName and listenAddress.
- Updated ExecutorConfigurationBuilder and ExecutorBuildOptions to include InstanceData.
- Modified Loader and DefaultFactoryResolver to utilize InstanceData for better clarity and maintainability.
- Refactored graphServer to initialize with InstanceData, enhancing the overall structure of the configuration.
Comment thread router/pkg/pubsub/README.md Outdated
Comment thread router/pkg/pubsub/datasource/provider.go Outdated
Comment thread router/pkg/pubsub/datasource/provider.go
Comment thread router/pkg/pubsub/datasource/provider_impl.go Outdated
Comment thread router/pkg/pubsub/kafka/adapter.go Outdated
Comment thread router/pkg/pubsub/kafka/engine_datasource.go
Comment thread router/pkg/pubsub/datasource/factory.go Outdated
Comment thread router/pkg/pubsub/datasource/provider.go Outdated
Comment thread router/pkg/pubsub/kafka/pubsub_datasource.go Outdated
@devsergiy devsergiy changed the base branch from ale/eng-6482-edfs-refactor-kafka-and-nats-as-datasource-in-router to main June 6, 2025 11:27
Comment thread router/pkg/pubsub/pubsub.go Outdated
Copy link
Copy Markdown
Member

@devsergiy devsergiy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great Job!

@alepane21 alepane21 merged commit edb0ded into main Jun 6, 2025
27 checks passed
@alepane21 alepane21 deleted the ale/eng-6482-edfs-refactor-filtered-datasources branch June 6, 2025 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants