Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#nullable enable
OpenTelemetry.Exporter.OtlpExportCompression
OpenTelemetry.Exporter.OtlpExportCompression.GZip = 1 -> OpenTelemetry.Exporter.OtlpExportCompression
OpenTelemetry.Exporter.OtlpExportCompression.None = 0 -> OpenTelemetry.Exporter.OtlpExportCompression
OpenTelemetry.Exporter.OtlpExporterOptions.Compression.get -> OpenTelemetry.Exporter.OtlpExportCompression
OpenTelemetry.Exporter.OtlpExporterOptions.Compression.set -> void
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,11 @@ Notes](../../RELEASENOTES.md).
* Fixed `NullReferenceException` when exporting logs if the scope key is null.
([#7186](https://github.com/open-telemetry/opentelemetry-dotnet/pull/7186))

* Added opt-in support for gzip compression. Compression can be configured
programmatically via the new `OtlpExporterOptions.Compression` property,
or through the environment variables such as `OTEL_EXPORTER_OTLP_COMPRESSION=gzip`.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: maybe mention the signal specific env vars too here.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also, if we can update the OTLP's readme also, that'd better.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point - I forgot about the README.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README updated.

([#7055](https://github.com/open-telemetry/opentelemetry-dotnet/issues/7055))

## 1.15.3

Released 2026-Apr-21
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -103,4 +103,9 @@ internal interface IOtlpExporterOptions
/// </list>
/// </remarks>
Func<HttpClient> HttpClientFactory { get; set; }

/// <summary>
/// Gets or sets the compression method to use when sending telemetry.
/// </summary>
OtlpExportCompression Compression { get; set; }
}
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ protected OtlpExportClient(OtlpExporterOptions options, HttpClient httpClient, s
this.Endpoint = new UriBuilder(exporterEndpoint).Uri;
this.Headers = options.GetHeaders<Dictionary<string, string>>((d, k, v) => d.Add(k, v));
this.HttpClient = httpClient;
this.CompressionEnabled = options.Compression == OtlpExportCompression.GZip;
}

internal HttpClient HttpClient { get; }
Expand All @@ -52,6 +53,8 @@ protected OtlpExportClient(OtlpExporterOptions options, HttpClient httpClient, s

internal IReadOnlyDictionary<string, string> Headers { get; }

internal bool CompressionEnabled { get; }

internal abstract MediaTypeHeaderValue MediaTypeHeader { get; }

internal virtual bool RequireHttp2 => false;
Expand Down Expand Up @@ -88,14 +91,25 @@ protected HttpRequestMessage CreateHttpRequest(byte[] buffer, int contentLength)
request.Headers.Add(header.Key, header.Value);
}

// TODO: Support compression.

request.Content = new ByteArrayContent(buffer, 0, contentLength);
request.Content.Headers.ContentType = this.MediaTypeHeader;
request.Content = this.CreateHttpContent(buffer, contentLength);

return request;
}

/// <summary>
/// Creates the <see cref="HttpContent"/> for a request. Override in subclasses to
/// customise content creation (e.g. to apply compression).
/// </summary>
/// <param name="buffer">The serialized protobuf payload buffer.</param>
/// <param name="contentLength">The number of bytes within <paramref name="buffer"/> that make up the message.</param>
/// <returns>An <see cref="HttpContent"/> representing the export payload.</returns>
protected virtual HttpContent CreateHttpContent(byte[] buffer, int contentLength)
{
var content = new ByteArrayContent(buffer, 0, contentLength);
content.Headers.ContentType = this.MediaTypeHeader;
return content;
}

protected HttpResponseMessage SendHttpRequest(HttpRequestMessage request, CancellationToken cancellationToken) =>
#if NET
// Note: SendAsync must be used with HTTP/2 because synchronous send is
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,11 @@
#if NETFRAMEWORK
using System.Net.Http;
#endif
using System.Buffers.Binary;
using System.Diagnostics.Tracing;
using System.IO.Compression;
using System.Net.Http.Headers;
using System.Net.Sockets;
using OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient.Grpc;

namespace OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient;
Expand All @@ -14,6 +17,12 @@ namespace OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClie
internal sealed class OtlpGrpcExportClient : OtlpExportClient
{
public const string GrpcStatusDetailsHeader = "grpc-status-details-bin";

// A gRPC message frame header is 5 bytes:
// byte 0 - Compression flag (0 = not compressed, 1 = compressed).
// bytes 1-4 - Message length in big-endian format.
private const int GrpcMessageHeaderSize = 5;

private static readonly ExportClientHttpResponse SuccessExportResponse = new(success: true, deadlineUtc: default, response: null, exception: null);
private static readonly MediaTypeHeaderValue MediaHeaderValue = new("application/grpc");

Expand All @@ -25,6 +34,10 @@ private static readonly ExportClientGrpcResponse DefaultExceptionExportClientGrp
status: null,
grpcStatusDetailsHeader: null);

#if !NET
private static readonly byte[] GrpcFrameHeader = [0, 0, 0, 0, 0];
#endif

public OtlpGrpcExportClient(OtlpExporterOptions options, HttpClient httpClient, string signalPath)
: base(options, httpClient, signalPath)
{
Expand All @@ -37,6 +50,11 @@ public OtlpGrpcExportClient(OtlpExporterOptions options, HttpClient httpClient,
// We need the entire response content to ensure that the response trailers are received
internal override HttpCompletionOption CompletionOption => HttpCompletionOption.ResponseContentRead;

#if NET
Comment thread
martincostello marked this conversation as resolved.
// See https://vcsjones.dev/csharp-readonly-span-bytes-static/
private static ReadOnlySpan<byte> GrpcFrameHeader => [0, 0, 0, 0, 0];
#endif

/// <inheritdoc/>
public override ExportClientResponse SendExportRequest(byte[] buffer, int contentLength, DateTime deadlineUtc, CancellationToken cancellationToken = default)
{
Expand All @@ -50,6 +68,12 @@ public override ExportClientResponse SendExportRequest(byte[] buffer, int conten
// A missing TE header results in servers aborting the gRPC call.
httpRequest.Headers.TryAddWithoutValidation("TE", "trailers");

if (this.CompressionEnabled)
{
httpRequest.Headers.Remove("grpc-encoding");
httpRequest.Headers.TryAddWithoutValidation("grpc-encoding", "gzip");
}

httpResponse = this.SendHttpRequest(httpRequest, cancellationToken);

httpResponse.EnsureSuccessStatusCode();
Expand Down Expand Up @@ -173,10 +197,54 @@ public override ExportClientResponse SendExportRequest(byte[] buffer, int conten
}
}

protected override HttpContent CreateHttpContent(byte[] buffer, int contentLength)
{
if (!this.CompressionEnabled)
{
return base.CreateHttpContent(buffer, contentLength);
}

// Build a gzip-compressed gRPC message frame:
// byte 0 - Compression flag = 1 (gzip).
// bytes 1-4 - Compressed payload length in big-endian format.
// bytes 5+ - Gzip-compressed protobuf payload.
#if NET
var compressedStream = new PooledBufferStream();
#else
var compressedStream = new MemoryStream();
#endif

// Reserve space for the gRPC frame header.
#if NET
compressedStream.Write(GrpcFrameHeader);
#else
compressedStream.Write(GrpcFrameHeader, 0, GrpcFrameHeader.Length);
#endif

using (var gzipStream = new GZipStream(compressedStream, CompressionLevel.Fastest, leaveOpen: true))
{
gzipStream.Write(buffer, GrpcMessageHeaderSize, contentLength - GrpcMessageHeaderSize);
}

var compressedPayloadLength = (uint)(compressedStream.Length - GrpcMessageHeaderSize);

// Write the gRPC frame header: compression flag + big-endian payload length.
compressedStream.Position = 0;
compressedStream.WriteByte(1);

var lengthBytes = new byte[4];
BinaryPrimitives.WriteUInt32BigEndian(lengthBytes, compressedPayloadLength);
compressedStream.Write(lengthBytes, 0, 4);

compressedStream.Position = 0;

OpenTelemetryProtocolExporterEventSource.Log.CompressedGrpcPayload("gzip", contentLength, compressedStream.Length);

var content = new StreamContent(compressedStream);
content.Headers.ContentType = this.MediaTypeHeader;
return content;
}

private static bool IsTransientNetworkError(HttpRequestException ex) =>
ex.InnerException is System.Net.Sockets.SocketException socketEx
&& (socketEx.SocketErrorCode == System.Net.Sockets.SocketError.TimedOut
|| socketEx.SocketErrorCode == System.Net.Sockets.SocketError.ConnectionReset
|| socketEx.SocketErrorCode == System.Net.Sockets.SocketError.HostUnreachable
|| socketEx.SocketErrorCode == System.Net.Sockets.SocketError.ConnectionRefused);
ex.InnerException is SocketException { SocketErrorCode: SocketError.TimedOut or SocketError.ConnectionReset or SocketError.HostUnreachable or SocketError.ConnectionRefused };
}
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
using System.Net.Http;
#endif
using System.Diagnostics.Tracing;
using System.IO.Compression;
using System.Net.Http.Headers;

namespace OpenTelemetry.Exporter.OpenTelemetryProtocol.Implementation.ExportClient;
Expand Down Expand Up @@ -54,4 +55,34 @@ public override ExportClientResponse SendExportRequest(byte[] buffer, int conten
return new ExportClientHttpResponse(success: false, deadlineUtc: deadlineUtc, response: null, exception: ex);
}
}

protected override HttpContent CreateHttpContent(byte[] buffer, int contentLength)
{
if (!this.CompressionEnabled)
{
return base.CreateHttpContent(buffer, contentLength);
}

#if NET
var compressedStream = new PooledBufferStream();
#else
var compressedStream = new MemoryStream();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets leave a todo for a future optimization to see if we can avoid allocation by renting/pooling arrays.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What specifically were you thinking of?

  • Renting a buffer to pass into the MemoryStream?
  • Renting a single shared buffer (like the main exporter does) for compression and re-using it across requests to pass into the MemoryStream?
  • A custom Stream implementation that acts like a memory stream but is backed by rented buffers?
  • Something else?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got an implementation of the third option locally, I'm just testing it.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We wouldn't be able to take a dependency on those.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't recommend that for this library either.

But it has quite a few with unit tests.

I don't see a problem grabbing the code from one that fits here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a quick look, and didn't see an "obvious" one that grabbed from the array pool, plus they seemed to have a bunch of extra interfaces I'd have to prune out. As I'd already written an implementation when you commented, I'll stick with that unless there's a compelling reason to replace it with something adapted and vendored from there.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! I meant to leave a TODO for revisiting this in a future PR to keep the initial version easier to review. My suggestion is to avoid the optimization in this PR, so review can focus on just the feature, and then a follow up where we can focus on pooling/efficiency part!

Sorry I was not very clear initially!

Copy link
Copy Markdown
Member Author

@martincostello martincostello Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I've already done the work either I can remove it and just immediately open a PR as a draft that's the same plus 2 extra files to rebase later and then re-run the benchmarks without the optimization, or we can just leave it in.

#endif

using (var gzipStream = new GZipStream(compressedStream, CompressionLevel.Fastest, leaveOpen: true))
{
gzipStream.Write(buffer, 0, contentLength);
}

compressedStream.Position = 0;

OpenTelemetryProtocolExporterEventSource.Log.CompressedHttpPayload("gzip", contentLength, compressedStream.Length);

var content = new StreamContent(compressedStream);
Comment thread
cijothomas marked this conversation as resolved.

content.Headers.ContentType = this.MediaTypeHeader;
content.Headers.Add("Content-Encoding", "gzip");

return content;
}
}
Loading
Loading