thrift_proxy: introduce header transport #4082
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,299 @@ | ||
| #include "extensions/filters/network/thrift_proxy/header_transport_impl.h" | ||
|
|
||
| #include <limits> | ||
|
|
||
| #include "envoy/common/exception.h" | ||
|
|
||
| #include "common/buffer/buffer_impl.h" | ||
|
|
||
| #include "extensions/filters/network/thrift_proxy/buffer_helper.h" | ||
| #include "extensions/filters/network/thrift_proxy/transport_impl.h" | ||
|
|
||
| namespace Envoy { | ||
| namespace Extensions { | ||
| namespace NetworkFilters { | ||
| namespace ThriftProxy { | ||
| namespace { | ||
|
|
||
| // c.f. | ||
| // https://github.com/apache/thrift/blob/master/lib/cpp/src/thrift/protocol/TProtocolTypes.h#L27 | ||
| enum class HeaderProtocolType { | ||
| Binary = 0, | ||
| JSON = 1, | ||
| Compact = 2, | ||
|
|
||
| FirstHeaderProtocolType = Binary, | ||
| LastHeaderProtocolType = Compact, | ||
| }; | ||
|
|
||
| } // namespace | ||
|
|
||
| bool HeaderTransportImpl::decodeFrameStart(Buffer::Instance& buffer, MessageMetadata& metadata) { | ||
| if (buffer.length() < 14) { | ||
| return false; | ||
| } | ||
|
|
||
| // Size of frame, not including the length bytes. | ||
| const int32_t frame_size = BufferHelper::peekI32(buffer); | ||
|
|
||
| // Minimum header frame size is 18 bytes (4 bytes of frame size + 10 bytes of fixed header + | ||
| // minimum 4 bytes of variable header data), so frame_size must be at least 14. | ||
| if (frame_size < 14 || frame_size > MaxFrameSize) { | ||
| throw EnvoyException(fmt::format("invalid thrift header transport frame size {}", frame_size)); | ||
| } | ||
|
|
||
| int16_t magic = BufferHelper::peekU16(buffer, 4); | ||
| if (!isMagic(magic)) { | ||
| throw EnvoyException(fmt::format("invalid thrift header transport magic {:04x}", magic)); | ||
| } | ||
|
|
||
| // offset 6: 16 bit flags field, unused | ||
| // offset 8: 32 bit sequence number field | ||
| int32_t seq_id = BufferHelper::peekI32(buffer, 8); | ||
|
|
||
| // offset 12: 16 bit (remaining) header size / 4 (spec erroneously claims / 32). | ||
| int16_t raw_header_size = BufferHelper::peekI16(buffer, 12); | ||
| int32_t header_size = static_cast<int32_t>(raw_header_size) * 4; | ||
| if (header_size < 0 || header_size > MaxHeadersSize) { | ||
| throw EnvoyException(fmt::format("invalid thrift header transport header size {} ({:04x})", | ||
| header_size, static_cast<uint16_t>(raw_header_size))); | ||
| } | ||
|
|
||
| if (header_size == 0) { | ||
| throw EnvoyException("no header data"); | ||
| } | ||
|
|
||
| if (buffer.length() < static_cast<uint64_t>(header_size) + 14) { | ||
| // Need more header data. | ||
| return false; | ||
| } | ||
|
|
||
| // Header data starts at offset 14 (4 bytes of frame size followed by 10 bytes of fixed header). | ||
| buffer.drain(14); | ||
|
|
||
| // Remaining frame size is the original frame size (which does not count itself), less the 10 | ||
| // fixed bytes of the header (magic, flags, etc), less the size of the variable header data | ||
| // (header_size). | ||
| metadata.setFrameSize(static_cast<uint32_t>(frame_size - header_size - 10)); | ||
| metadata.setSequenceId(seq_id); | ||
|
|
||
| ProtocolType proto = ProtocolType::Auto; | ||
| HeaderProtocolType header_proto = | ||
| static_cast<HeaderProtocolType>(drainVarIntI16(buffer, header_size, "protocol id")); | ||
| switch (header_proto) { | ||
| case HeaderProtocolType::Binary: | ||
| proto = ProtocolType::Binary; | ||
| break; | ||
| case HeaderProtocolType::Compact: | ||
| proto = ProtocolType::Compact; | ||
| break; | ||
| default: | ||
| throw EnvoyException(fmt::format("Unknown protocol {}", static_cast<int>(header_proto))); | ||
| } | ||
| metadata.setProtocol(proto); | ||
|
|
||
| int16_t num_xforms = drainVarIntI16(buffer, header_size, "transform count"); | ||
|
Contributor
There was a problem hiding this comment. Is this line vulnerable to a buffer read overflow? Here's the scenario I'm worried about:
Member
Author
There was a problem hiding this comment. I don't believe so. We require that the full header (as given by header size) by available. As we read bytes from the header we decrement header_size (which is passed as a reference to drainVarIntI16). drainVarIntI16 is just drainVarIntI32 with an upper bound check. The latter requires remaining header_size to be at least 1 and then invokes BufferHelper::peekVarIntI32 which validates the buffer has enough data before reading each byte that composes the value. If the peek reports a buffer underflow (buffer doesn't have enough data to read the whole number) or requires more bytes than are left in the header (supposing the buffer has more data beyond the end of the header) we throw. That said, I'll add a test that exercises this directly. Mostly as a remdinder to myself: It's basically this one except that the number of transforms should be truncated. |
||
| if (num_xforms < 0) { | ||
| throw EnvoyException(fmt::format("invalid header transport transform count {}", num_xforms)); | ||
| } | ||
|
|
||
| while (num_xforms-- > 0) { | ||
| int32_t xform_id = drainVarIntI32(buffer, header_size, "transform id"); | ||
|
|
||
| // To date, no transforms have a data field. In the future, some transform IDs may require | ||
| // consuming another varint 32 at this point. The known transform IDs are: | ||
| // 1: zlib compression | ||
| // 2: hmac (appended to end of packet) | ||
| // 3: snappy compression | ||
| buffer.drain(header_size); | ||
| metadata.setAppException(AppExceptionType::MissingResult, | ||
| fmt::format("Unknown transform {}", xform_id)); | ||
| return true; | ||
| } | ||
|
|
||
| while (header_size > 0) { | ||
| // Attempt to read info blocks | ||
| int32_t info_id = drainVarIntI32(buffer, header_size, "info id"); | ||
| if (info_id != 1) { | ||
| // 0 indicates a padding byte, and the end of the info block. | ||
| // 1 indicates an info id header/value pair. | ||
| // Any other value is an unknown info id block, which we ignore. | ||
| break; | ||
| } | ||
|
|
||
| int32_t num_headers = drainVarIntI32(buffer, header_size, "header count"); | ||
| if (num_headers < 0) { | ||
| throw EnvoyException(fmt::format("invalid header transport header count {}", num_headers)); | ||
| } | ||
|
|
||
| while (num_headers-- > 0) { | ||
| std::string key = drainVarString(buffer, header_size, "header key"); | ||
| std::string value = drainVarString(buffer, header_size, "header value"); | ||
| metadata.addHeader(Header(key, value)); | ||
| } | ||
| } | ||
|
|
||
| // Remaining bytes are padding or ignored info blocks. | ||
| if (header_size > 0) { | ||
| buffer.drain(header_size); | ||
| } | ||
|
|
||
| return true; | ||
| } | ||
|
|
||
| bool HeaderTransportImpl::decodeFrameEnd(Buffer::Instance&) { | ||
| exception_.reset(); | ||
| exception_reason_.clear(); | ||
|
|
||
| return true; | ||
| } | ||
|
|
||
| void HeaderTransportImpl::encodeFrame(Buffer::Instance& buffer, const MessageMetadata& metadata, | ||
| Buffer::Instance& message) { | ||
| uint64_t msg_size = message.length(); | ||
| if (msg_size == 0) { | ||
| throw EnvoyException(fmt::format("invalid thrift header transport message size {}", msg_size)); | ||
| } | ||
|
|
||
| const HeaderMap& headers = metadata.headers(); | ||
| if (headers.size() > MaxHeadersSize / 2) { | ||
| // Each header takes a minimum of 2 bytes, yielding this limit. | ||
| throw EnvoyException( | ||
| fmt::format("invalid thrift header transport too many headers {}", headers.size())); | ||
| } | ||
|
|
||
| Buffer::OwnedImpl header_buffer; | ||
|
|
||
| if (!metadata.hasProtocol()) { | ||
| throw EnvoyException("missing header transport protocol"); | ||
| } | ||
|
|
||
| switch (metadata.protocol()) { | ||
| case ProtocolType::Binary: | ||
| BufferHelper::writeVarIntI32(header_buffer, static_cast<int32_t>(HeaderProtocolType::Binary)); | ||
| break; | ||
| case ProtocolType::Compact: | ||
| BufferHelper::writeVarIntI32(header_buffer, static_cast<int32_t>(HeaderProtocolType::Compact)); | ||
| break; | ||
| default: | ||
| throw EnvoyException(fmt::format("invalid header transport protocol {}", | ||
| ProtocolNames::get().fromType(metadata.protocol()))); | ||
| } | ||
|
|
||
| BufferHelper::writeVarIntI32(header_buffer, 0); // num transforms | ||
| if (headers.size() > 0) { | ||
| // Info ID 1 | ||
| BufferHelper::writeI8(header_buffer, 1); | ||
|
|
||
| // Num headers | ||
| BufferHelper::writeVarIntI32(header_buffer, static_cast<int32_t>(headers.size())); | ||
|
|
||
| for (const Header& header : headers) { | ||
| writeVarString(header_buffer, header.key()); | ||
| writeVarString(header_buffer, header.value()); | ||
| } | ||
| } | ||
|
|
||
| uint64_t header_size = header_buffer.length(); | ||
|
|
||
| // Always pad (as the Apache implementation does). | ||
| int padding = 4 - (header_size % 4); | ||
| header_buffer.add("\0\0\0\0", padding); | ||
| header_size += padding; | ||
|
|
||
| if (header_size > MaxHeadersSize) { | ||
| throw EnvoyException( | ||
| fmt::format("invalid thrift header transport header size {}", header_size)); | ||
| } | ||
|
|
||
| // Frame size does not include the frame length itself. | ||
| uint64_t size = header_size + msg_size + 10; | ||
| if (size > MaxFrameSize) { | ||
| throw EnvoyException(fmt::format("invalid thrift header transport frame size {}", size)); | ||
| } | ||
|
|
||
| int32_t seq_id = 0; | ||
| if (metadata.hasSequenceId()) { | ||
| seq_id = metadata.sequenceId(); | ||
| } | ||
|
|
||
| BufferHelper::writeU32(buffer, static_cast<uint32_t>(size)); | ||
| BufferHelper::writeU16(buffer, Magic); | ||
| BufferHelper::writeU16(buffer, 0); // flags | ||
| BufferHelper::writeI32(buffer, seq_id); | ||
| BufferHelper::writeU16(buffer, static_cast<uint16_t>(header_size / 4)); | ||
| buffer.move(header_buffer); | ||
| buffer.move(message); | ||
| } | ||
|
|
||
| int16_t HeaderTransportImpl::drainVarIntI16(Buffer::Instance& buffer, int32_t& header_size, | ||
| const char* desc) { | ||
| int32_t value = drainVarIntI32(buffer, header_size, desc); | ||
| if (value > static_cast<int32_t>(std::numeric_limits<int16_t>::max())) { | ||
| throw EnvoyException(fmt::format("header transport {}: value {} exceeds max i16 ({})", desc, | ||
| value, std::numeric_limits<int16_t>::max())); | ||
| } | ||
| return static_cast<int16_t>(value); | ||
| } | ||
|
|
||
| int32_t HeaderTransportImpl::drainVarIntI32(Buffer::Instance& buffer, int32_t& header_size, | ||
| const char* desc) { | ||
| if (header_size <= 0) { | ||
| throw EnvoyException(fmt::format("unable to read header transport {}: header too small", desc)); | ||
| } | ||
|
|
||
| int size; | ||
| int32_t value = BufferHelper::peekVarIntI32(buffer, 0, size); | ||
| if (size < 0 || (header_size - size) < 0) { | ||
| throw EnvoyException(fmt::format("unable to read header transport {}: header too small", desc)); | ||
| } | ||
| buffer.drain(size); | ||
| header_size -= size; | ||
| return value; | ||
| } | ||
|
|
||
| std::string HeaderTransportImpl::drainVarString(Buffer::Instance& buffer, int32_t& header_size, | ||
| const char* desc) { | ||
| int16_t str_len = drainVarIntI16(buffer, header_size, desc); | ||
| if (str_len == 0) { | ||
| return ""; | ||
| } | ||
|
|
||
| if (header_size < static_cast<int32_t>(str_len)) { | ||
| throw EnvoyException(fmt::format("unable to read header transport {}: header too small", desc)); | ||
| } | ||
|
|
||
| std::string value(static_cast<char*>(buffer.linearize(str_len)), str_len); | ||
| buffer.drain(str_len); | ||
| header_size -= str_len; | ||
| return value; | ||
| } | ||
|
|
||
| void HeaderTransportImpl::writeVarString(Buffer::Instance& buffer, const std::string& str) { | ||
| std::string::size_type len = str.length(); | ||
| if (len > static_cast<uint32_t>(std::numeric_limits<int16_t>::max())) { | ||
| throw EnvoyException(fmt::format("header string too long: {}", len)); | ||
| } | ||
|
|
||
| BufferHelper::writeVarIntI32(buffer, static_cast<int32_t>(len)); | ||
| if (len == 0) { | ||
| return; | ||
| } | ||
| buffer.add(str); | ||
| } | ||
|
|
||
| class HeaderTransportConfigFactory : public TransportFactoryBase<HeaderTransportImpl> { | ||
| public: | ||
| HeaderTransportConfigFactory() : TransportFactoryBase(TransportNames::get().HEADER) {} | ||
| }; | ||
|
|
||
| /** | ||
| * Static registration for the header transport. @see RegisterFactory. | ||
| */ | ||
| static Registry::RegisterFactory<HeaderTransportConfigFactory, NamedTransportConfigFactory> | ||
| register_; | ||
|
|
||
| } // namespace ThriftProxy | ||
| } // namespace NetworkFilters | ||
| } // namespace Extensions | ||
| } // namespace Envoy | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: should this be a constant (e.g.:
HEADER_LEN)?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added some constants for the various sizes this transport uses.