Skip to content

Pulsar: built-in Avro schema support (typed ISchema<T>) (#3213)#3214

Closed
jeremydmiller wants to merge 1 commit into
mainfrom
feat/pulsar-avro-schema-3213
Closed

Pulsar: built-in Avro schema support (typed ISchema<T>) (#3213)#3214
jeremydmiller wants to merge 1 commit into
mainfrom
feat/pulsar-avro-schema-3213

Conversation

@jeremydmiller

Copy link
Copy Markdown
Member

Closes #3213. Follow-up to the JSON schema work (#3210), completing the Pulsar schema story from #3183.

JSON used a pass-through schema where Wolverine owns the body bytes. Avro can't work that way — DotPulsar's Schema.AvroISpecificRecord<T>() owns the encode/decode, so the body must be genuine Avro on the wire. This adds a small codec seam that keeps the existing byte-oriented sender/listener (and all the ack/retry/DLQ logic) fully intact.

opts.PublishMessage<OrderPlaced>().ToPulsarTopic(topic).UseAvroSchema<OrderPlaced>();
opts.ListenToPulsarTopic(topic).UseAvroSchema<OrderPlaced>();

How it works

  • IPulsarMessageCodec / PulsarAvroCodec<T> wraps DotPulsar's built-in Schema.AvroISpecificRecord<T>(). T must be an Apache.Avro ISpecificRecord at runtime — there is no compile-time Avro dependency in the transport (DotPulsar resolves the schema reflectively from the type's static _SCHEMA).
  • UseAvroSchema<T>() sets endpoint.MessageCodec and a pass-through PulsarSchema carrying the Avro SchemaInfo, so the broker registers the Avro schema for the topic.
  • PulsarSender encodes envelope.Message through the codec (falling back to Wolverine's serialized body for non-codec messages such as a ping); PulsarListener decodes to the message object and sets envelope.Message directly — the pipeline then skips its own body deserialization (HandlerPipeline.TryDeserializeEnvelope returns early when Message is already set).

This reuses the UsePulsarSchema(...) seam direction noted in #3210; Protobuf/other schema types can plug in the same way via a custom codec.

Acceptance criteria (from #3183 / #3213)

  • ✅ Avro path demonstrated: end-to-end publish/consume under an Avro schema with broker-side AVRO schema registration, verified via the Pulsar admin REST endpoint returning type: AVRO.
  • ✅ JSON schema path unchanged — no regression (re-verified green).

Tests / build

  • pulsar_avro_schema: codec round-trip (unit) + end-to-end broker round-trip with AVRO registration — green against a Testcontainers broker. Apache.Avro added to the test project only.
  • Full wolverine.slnx builds clean in Release.

🤖 Generated with Claude Code

Follow-up to the JSON schema work (#3210). JSON used a pass-through schema where
Wolverine owns the body bytes; Avro can't — DotPulsar's Schema.AvroISpecificRecord<T>()
owns the encode/decode, so the body must be genuine Avro on the wire.

Adds a codec seam that keeps the existing byte-oriented sender/listener (and all the
ack/retry/DLQ logic) intact:

    opts.PublishMessage<OrderPlaced>().ToPulsarTopic(topic).UseAvroSchema<OrderPlaced>();
    opts.ListenToPulsarTopic(topic).UseAvroSchema<OrderPlaced>();

- IPulsarMessageCodec / PulsarAvroCodec<T> wraps Schema.AvroISpecificRecord<T>()
  (T must be an Apache.Avro ISpecificRecord at runtime; no compile-time Avro
  dependency in the transport — DotPulsar resolves it reflectively).
- UseAvroSchema<T>() sets endpoint.MessageCodec + a pass-through PulsarSchema carrying
  the Avro SchemaInfo so the broker registers the Avro schema.
- PulsarSender encodes envelope.Message through the codec (falling back to Wolverine's
  serialized body for non-codec messages such as a ping); PulsarListener decodes to the
  message object and sets envelope.Message directly, so the pipeline skips its own body
  deserialization (HandlerPipeline.TryDeserializeEnvelope returns early when Message is set).

Tests: Avro codec round-trip (unit) + end-to-end publish/consume under an Avro schema
with broker-side AVRO registration verified via the admin REST API; JSON schema path
unchanged (no regression). Docs: Avro subsection. Apache.Avro added to the test project
only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jeremydmiller

Copy link
Copy Markdown
Member Author

Superseded by the rebased branch (resolves conflicts with the merged dedup PR #3212). See the replacement PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pulsar: built-in Avro schema support (typed ISchema<T>)

1 participant