Skip to content
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
df6d49f
Macro 'SAFE_MEMCPY added according to comments on issue#9328'
Oct 25, 2020
6a9d849
WIP: SAFE_MEMCPY modified with memmove, and some minor changes made.
Oct 27, 2020
5a29b9a
macros.h format fixed
Oct 27, 2020
814894b
testing CI pipeline
Oct 27, 2020
45565c2
static_assert deleted from macro SAFE_MEMCPY to avoid troubles with r…
Oct 27, 2020
2ad12d4
list of dependencies changed
Oct 28, 2020
22d2bfa
label corrected in .../mongo_proxy/BUILD
Oct 28, 2020
9d72d11
casts added and deleted var val included
Oct 28, 2020
1b43b53
format corrected
Oct 28, 2020
0aab7bc
cast added to avoid using void* in MemBlockBuilder
Oct 28, 2020
d34e261
Initializing Span before calling appendData
Oct 28, 2020
6e3a257
Span constructor corrected
Oct 29, 2020
d4b3c50
Revert "Span constructor corrected"
Oct 29, 2020
34d9ecb
Span constructor corrected
Oct 29, 2020
50eee3c
style corrected.
Oct 29, 2020
1bc7916
SAFE_MACRO modified to assert input arrays' sizes
Oct 29, 2020
d68aa9e
Get size of the array
Oct 29, 2020
4d331cd
use of envoy ASSERT macro in SAFE_MEMCPY
Oct 30, 2020
5372f86
Use static_assert again in SAFE_MEMCPY. Changes in proxy_protocol.cc
Oct 30, 2020
5517c53
proxy protocol header parse changed
Oct 30, 2020
455ceb7
getSockAddr public method added to PipeInstance
Oct 30, 2020
d188a94
comparison corrected in for loop
Oct 30, 2020
b20de9a
reinterpret_cast deleted from dns_parser. Some changes added to buffe…
Oct 31, 2020
cfb8145
buffer BUILD file changed
Oct 31, 2020
87c59b3
Cast added to copy data
Oct 31, 2020
22b2ee4
proxy protocol implementation back in place. Method parseV2Header mod…
Oct 31, 2020
e1dd071
changes to mem_block_builder applied, data_ type is a raw pointer poi…
Nov 3, 2020
57916ed
format fixed
Nov 3, 2020
c71445f
check_format changed as suggested in issue #9328
Nov 7, 2020
f51819e
NOTLINT added to the line in which memcpy is used
Nov 7, 2020
b0960d3
memcpy changed for strcpy
Nov 7, 2020
74e19f1
memcpy reviewed in the dir source/
Nov 7, 2020
dc0d096
Other applications of memcpy reviewed
Nov 7, 2020
a980bbd
assert error corrected for extensions/filters/network/kafka/serializa…
Nov 7, 2020
d1a3898
SAFE_MEMCPY reviewed due to difference in array sizes
Nov 7, 2020
f594444
minor corrections
Nov 7, 2020
f75af26
Merge branch 'master' into issue-9328
rialg Nov 7, 2020
ead0075
Format fixed
Nov 8, 2020
327d8d7
Merge branch 'issue-9328' of https://github.com/grial1/envoy into iss…
Nov 8, 2020
f3b29f6
wrong file submitted. dns_parser.cc fix added
Nov 8, 2020
4e502c8
changes suggested by jmarantz were added and some tests approved.
Nov 11, 2020
42105e0
format fixed
Nov 11, 2020
d714e2e
address_impl.cc compilation error fixed
Nov 11, 2020
b7f136c
comment fixed in address_impl
Nov 11, 2020
596c784
initial set of changes. Added MemBlockBuilder in proxy protocol imple…
Nov 11, 2020
6f0305f
proposed changes applied
Nov 13, 2020
671a5f1
format fixed
Nov 13, 2020
c51bbf2
bug fixed
Nov 14, 2020
882752a
format fixed
Nov 14, 2020
fecccfe
SAFE_MEMCPY deleted from macros.h
Nov 15, 2020
3692166
SAFE_MEMCPY macro changed for safe_memcpy template function
Nov 15, 2020
e3f469b
safe_memcpy added to common/network/cidr_range.cc
Nov 15, 2020
40dc126
proxy_protocol_header.cc: function generateV2Header uses memcpy in a …
Nov 15, 2020
616f36d
format fixed
Nov 15, 2020
2a8c457
explicit types deleted from safe_memcpy, check_format.py updated, and…
Nov 18, 2020
c4211ca
format fixed
Nov 18, 2020
9488f4e
C library added for memcpy
Nov 18, 2020
3b44b88
dereference applied and safe_memcpy_lib registered for hot_restarting…
Nov 21, 2020
cdcecd3
coding style error fixed: safe_memcpy signature changed to safeMemcpy
Nov 22, 2020
6fc7264
memcpy error message changed in check_format.py
Nov 22, 2020
b7dcf04
two simple tests added for safeMemcpy
Nov 22, 2020
93622eb
format corrected and application of safeMemcpy in common/grpc/common.cc
Nov 22, 2020
ade5461
address taken from frame->ping.opaque_data in codec_impl*.cc
Nov 22, 2020
da34c9d
safeMemcpy switch back to memcpy in function parseDnsObject due to st…
Nov 22, 2020
ab19cbd
dns_filter_lib reverted due to inability to compile with MSVC
Nov 23, 2020
5f76f5e
suggested changes applied
Nov 23, 2020
95c6815
Merge remote-tracking branch 'upstream/master'
Nov 24, 2020
86d1024
Merge remote-tracking branch 'upstream/master'
Nov 28, 2020
60f3da6
Merge branch 'master' into issue-9328
rialg Nov 7, 2020
5f06fc0
Merge remote-tracking branch 'upstream/master'
Nov 29, 2020
10f4596
Merge remote-tracking branch 'upstream/master'
Nov 30, 2020
57ad370
Chnages suggested by htuch: safeMemcpy applied to proxy_protocol_head…
Nov 30, 2020
6e56f97
reinterpreting DnsHeaderFlags as uint16_t ptr
Dec 1, 2020
31f428a
typo fixed
Dec 1, 2020
f9d8c1f
Changes suggested by lizan@ and jmarantz@ added
Dec 1, 2020
b1cc18b
adding IPv6 address to the output corrected in generateV2Header
Dec 1, 2020
4a8b709
dns_parser reverted
Dec 1, 2020
d662d87
format fixed in dns_parser.cc
Dec 2, 2020
4406fa4
missing parentheses when safeMemcpy was used in dns_parser.cc
Dec 2, 2020
3dac1e3
dns_parser.cc reverted to memcpy
Dec 3, 2020
03520ed
Merge branch 'master' into issue-9328
rialg Dec 5, 2020
fff4fe1
Merge branch 'master' into issue-9328
rialg Dec 5, 2020
be23cf1
safeMemcpySrc & safeMemcpyDst refactored to match comments (drop size)
Dec 5, 2020
51c59d9
first set of changes: Do not use reinterpret_cast with safeMemcpy. Pe…
Dec 5, 2020
3f19f09
Merge branch 'issue-9328' of https://github.com/grial1/envoy into iss…
Dec 5, 2020
422b0e9
format fixed
Dec 5, 2020
23a5cf7
second set of changes: replace, where possible, memcpy for a safeMemc…
Dec 5, 2020
c789969
format fixed
Dec 5, 2020
d79baa3
address() function changed in address_impl.cc
Dec 5, 2020
3661d0a
error in address_impl.cc
Dec 5, 2020
27e3ccf
format fixed
Dec 6, 2020
83bd5dc
Revert "format fixed"
Dec 6, 2020
4263158
format fixed
Dec 6, 2020
89bc7ed
names of the variants of safeMemcpy changed
Dec 6, 2020
89e53c2
format fixed
Dec 6, 2020
08007f4
tests corrected
Dec 6, 2020
021efce
test corrected
Dec 6, 2020
9159f29
test approved
Dec 6, 2020
b561e2f
format message updated
Dec 6, 2020
ed2a892
tests corrected
Dec 6, 2020
a792eec
test/common/common/safe_memcpy_test.cc
Dec 7, 2020
f6ebfa6
tmpfile to set in motion pipeline
Dec 7, 2020
cb3ec94
temporary file deleted
Dec 9, 2020
33ef5a8
changed safeMemcpyUnsafeSrc and safeMemcpyUnsafeDst to function templ…
Dec 9, 2020
0630eee
typo fixed
Dec 9, 2020
c8f1808
void* removed
Dec 9, 2020
a2dfabb
address() function modified to use safeMemcpyUnsafeSrc
Dec 9, 2020
3b59def
Merge branch 'master' into issue-9328
rialg Dec 13, 2020
b8dd3e0
Merge branch 'master' into issue-9328
rialg Dec 13, 2020
e6760fc
Merge remote-tracking branch 'upstream/master' into issue-9328
Dec 13, 2020
4da526b
merged with master: fixed buffer_impl.cc
Dec 13, 2020
c133f5b
merge conflict solved
Dec 13, 2020
fc7aec5
merge
Dec 21, 2020
2b603f2
codec_impl_legacy.cc deleted
Dec 21, 2020
2632163
requested changes added
Dec 21, 2020
6454e2c
SafeMemcpyUnsafeDstTest modified to use basic C++
Dec 22, 2020
6d6aba8
safe_memcpy_test fixed
Dec 22, 2020
aca7254
Merge branch 'master' of https://github.com/envoyproxy/envoy into iss…
Jan 7, 2021
1151c7a
Tests coverage acceptable threshold reduced by 0.1 for source/extensi…
Jan 9, 2021
8bc497b
merge conflicts resolved
Jan 22, 2021
2cd8aaf
merge conflicts resolved
Feb 2, 2021
ddeed70
merge conflicts resolved
Feb 26, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions source/common/buffer/buffer_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ void OwnedImpl::commit(RawSlice* iovecs, uint64_t num_iovecs) {
void OwnedImpl::copyOut(size_t start, uint64_t size, void* data) const {
uint64_t bytes_to_skip = start;
uint8_t* dest = static_cast<uint8_t*>(data);
MemBlockBuilder<uint8_t> mem_builder(size);
for (const auto& slice : slices_) {
if (size == 0) {
break;
Expand All @@ -137,14 +138,14 @@ void OwnedImpl::copyOut(size_t start, uint64_t size, void* data) const {
continue;
}
uint64_t copy_size = std::min(size, data_size - bytes_to_skip);
memcpy(dest, slice->data() + bytes_to_skip, copy_size);
mem_builder.appendData(slice->data() + bytes_to_skip);
size -= copy_size;
dest += copy_size;
// Now that we've started copying, there are no bytes left to skip over. If there
// is any more data to be copied, the next iteration can start copying from the very
// beginning of the next slice.
bytes_to_skip = 0;
}
dest = mem_builder.release().get();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not look correct. Assigning this pointer is not going to move the bytes.

Really I think the problem here is I gave a suggestion for how I would build SAFE_MEMCPY that is not quite sufficient for all use-cases. Maybe we can use MemBlockBuilder for these dynamic cases, but we need to figure out how to do it without incurring extra copies, or performance will suffer.

For example, maybe we can have a variant of MemBlockBuilder (a subclass, perhaps) where the target memory and its size is provided to the constructor.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree on limiting the amount of temporary memory used to make the copy, however, would you consider that modifying the MemBlockBuilder with new methods could accomplish this task? why is it necessary a child class? I thought that MemBlockBuilder already meant the target. So, the issue is to find the an efficient way move the data in order to be accessible from the pointer in the code (i.e. dest).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't fully thought it through. But currently MemBlockBuilder owns its memory, and while you can release it, it's possible that some of the call-sites you have would be better off supplying an already-allocated buffer to MemBlockBuilder.

Determining the memory management policy is the reason I thought of using a subclass. But it's not worth making it virtual.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just looked at MemBlockBuilder again. And have two thoughts:

  1. If we do want MemBlockBuilder to populate an already-allocated buffer, I think it's not too hard. We need an extra constructor and an extra member variable. We can use the unique_ptr to hold data allocated by the current constructor. Let's call that owned_data_, and make data_ instead be a raw pointer. In the current constructor, owned_data_ will be allocated, and data_ will be set to that. And in the new constructor, a passed-in pointer will be provided for initializing data_, and owned_data_ will never be populated, thus never freed.

  2. Let's double-check that we really need to do this. Maybe we can change the classes that already allocate their own storage to simply delegate that responsibility to MemBlockBuilder, which can be used as an instance variable.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, those proposed changes seem feasible, in this case, to be sure I have got you right, the data_ member would be a pointer to an already allocated external buffer, isn’t it? Hence, the above snippet could be something like

uint8_t* dest = static_cast<uint8_t*>(data);
MemBlockBuilder<uint8_t> mem_builder(size, dest_);
// ...
//  append some data to mem_builder
// ... 

By the way, answering to a previous comment, I've started to compile the project locally, so I expected to push new changes at sometime during the next week.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data_ would point to owned_data_.get() in this case of the current constructor. And you would add a new constructor that would set data_ to a passed-in external pointer.

ASSERT(size == 0);
}

Expand Down
13 changes: 10 additions & 3 deletions source/common/buffer/buffer_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#include "envoy/buffer/buffer.h"

#include "common/common/assert.h"
#include "common/common/mem_block_builder.h"
#include "common/common/non_copyable.h"
#include "common/common/utility.h"
#include "common/event/libevent.h"
Expand Down Expand Up @@ -153,7 +154,9 @@ class Slice : public SliceData {
uint8_t* dest = base_ + reservable_;
reservable_ += copy_size;
// NOLINTNEXTLINE(clang-analyzer-core.NullDereference)
memcpy(dest, data, copy_size);
MemBlockBuilder<uint8_t> mem_builder(copy_size);
mem_builder.appendData(data);
dest = mem_builder.release().get();
return copy_size;
}

Expand Down Expand Up @@ -276,7 +279,9 @@ class OwnedSlice final : public Slice, public InlineStorage {
static SlicePtr create(const void* data, uint64_t size) {
uint64_t slice_capacity = sliceSize(size);
std::unique_ptr<OwnedSlice> slice(new (slice_capacity) OwnedSlice(slice_capacity));
memcpy(slice->base_, data, size);
MemBlockBuilder<uint8_t> mem_builder(size);
mem_builder.appendData(data);
slice->base_ = mem_builder.release().get();
slice->reservable_ = size;
return slice;
}
Expand Down Expand Up @@ -653,7 +658,9 @@ class OwnedBufferFragmentImpl final : public BufferFragment, public InlineStorag
OwnedBufferFragmentImpl(absl::string_view data, const Releasor& releasor)
: releasor_(releasor), size_(data.size()) {
ASSERT(releasor != nullptr);
memcpy(data_, data.data(), data.size());
MemBlockBuilder<uint8_t> mem_builder(data.size());
mem_builder.appendData(data.data());
data_ = mem_builder.release().get();
}

const Releasor releasor_;
Expand Down
9 changes: 9 additions & 0 deletions source/common/common/macros.h
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,15 @@ namespace Envoy {
return *objectptr; \
} while (0)

/**
* @brief Assert memory bounds to avoid copy errors.
*/
#define SAFE_MEMCPY(dst, src) \
Comment thread
rialg marked this conversation as resolved.
Outdated
do { \
static_assert(sizeof(*(src)) == sizeof(*(dst))); \
memmove(dst, src, sizeof(*(src))); \
} while (0)

/**
* Have a generic fall-through for different versions of C++
*/
Expand Down
1 change: 1 addition & 0 deletions source/common/network/address_impl.h
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,7 @@ class PipeInstance : public InstanceBase {
}
return sizeof(pipe_.address_);
}
void getSockAddr(sockaddr_un& address) { address = pipe_.address_; }
Comment thread
rialg marked this conversation as resolved.
Outdated

private:
struct PipeHelper : public Pipe {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,24 +196,28 @@ void Filter::parseV2Header(char* buf) {
std::make_shared<Network::Address::Ipv4Instance>(&la4)});
return;
} else if (((proto_family & 0xf0) >> 4) == PROXY_PROTO_V2_AF_INET6) {
PACKED_STRUCT(struct pp_ipv6_addr {
uint8_t src_addr[16];
uint8_t dst_addr[16];
uint16_t src_port;
uint16_t dst_port;
});
pp_ipv6_addr* v6;
v6 = reinterpret_cast<pp_ipv6_addr*>(&buf[PROXY_PROTO_V2_HEADER_LEN]);
std::size_t offset = 0; // Offset to iterate over buf
sockaddr_in6 ra6, la6;
memset(&ra6, 0, sizeof(ra6));
memset(&la6, 0, sizeof(la6));
ra6.sin6_family = AF_INET6;
ra6.sin6_port = v6->src_port;
memcpy(ra6.sin6_addr.s6_addr, v6->src_addr, sizeof(ra6.sin6_addr.s6_addr));

la6.sin6_family = AF_INET6;
la6.sin6_port = v6->dst_port;
memcpy(la6.sin6_addr.s6_addr, v6->dst_addr, sizeof(la6.sin6_addr.s6_addr));

for (; offset < ARRAY_SIZE(ra6.sin6_addr.s6_addr); ++offset)
Comment thread
rialg marked this conversation as resolved.
Outdated
ra6.sin6_addr.s6_addr[offset] = buf[PROXY_PROTO_V2_HEADER_LEN + offset];

for (; offset < ARRAY_SIZE(la6.sin6_addr.s6_addr); ++offset)
la6.sin6_addr.s6_addr[offset] = buf[PROXY_PROTO_V2_HEADER_LEN + offset];

PACKED_STRUCT(struct ports {
uint16_t src_port;
uint16_t dst_port;
});
ports* port_pair;
port_pair = reinterpret_cast<ports*>(&buf[PROXY_PROTO_V2_HEADER_LEN + offset]);

ra6.sin6_port = port_pair->src_port;
la6.sin6_port = port_pair->dst_port;

proxy_protocol_header_.emplace(WireHeader{
hdr_addr_len - PROXY_PROTO_V2_ADDR_LEN_INET6, Network::Address::IpVersion::v6,
Expand Down
1 change: 1 addition & 0 deletions source/extensions/filters/network/mongo_proxy/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ envoy_cc_library(
"//source/common/common:assert_lib",
"//source/common/common:byte_order_lib",
"//source/common/common:hex_lib",
"//source/common/common:mem_block_builder_lib",
"//source/common/common:minimal_logger_lib",
"//source/common/common:utility_lib",
],
Expand Down
13 changes: 7 additions & 6 deletions source/extensions/filters/network/mongo_proxy/bson_impl.cc
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
#include "common/common/byte_order.h"
#include "common/common/fmt.h"
#include "common/common/hex.h"
#include "common/common/mem_block_builder.h"
#include "common/common/utility.h"

namespace Envoy {
Expand All @@ -22,8 +23,7 @@ int32_t BufferHelper::peekInt32(Buffer::Instance& data) {
}

int32_t val;
void* mem = data.linearize(sizeof(int32_t));
std::memcpy(reinterpret_cast<void*>(&val), mem, sizeof(int32_t));
SAFE_MEMCPY(&val, static_cast<int32_t*>(data.linearize(sizeof(int32_t))));
return le32toh(val);
}

Expand All @@ -43,8 +43,10 @@ void BufferHelper::removeBytes(Buffer::Instance& data, uint8_t* out, size_t out_
throw EnvoyException("invalid buffer size");
}

void* mem = data.linearize(out_len);
std::memcpy(out, mem, out_len);
MemBlockBuilder<uint8_t> mem_builder(out_len);
mem_builder.appendData(
absl::Span<uint8_t>(static_cast<uint8_t*>(data.linearize(out_len)), out_len));
out = mem_builder.release().get();
data.drain(out_len);
}

Expand Down Expand Up @@ -88,8 +90,7 @@ int64_t BufferHelper::removeInt64(Buffer::Instance& data) {
}

int64_t val;
void* mem = data.linearize(sizeof(int64_t));
std::memcpy(reinterpret_cast<void*>(&val), mem, sizeof(int64_t));
SAFE_MEMCPY(&val, static_cast<int64_t*>(data.linearize(sizeof(int64_t))));
data.drain(sizeof(int64_t));
return le64toh(val);
}
Expand Down
4 changes: 2 additions & 2 deletions source/extensions/filters/udp/dns_filter/dns_parser.cc
Original file line number Diff line number Diff line change
Expand Up @@ -171,7 +171,7 @@ bool DnsMessageParser::parseDnsObject(DnsQueryContextPtr& context,
state = DnsQueryParseState::Flags;
break;
case DnsQueryParseState::Flags:
::memcpy(static_cast<void*>(&header_.flags), &data, field_size);
SAFE_MEMCPY(&(header_.flags), &data);
state = DnsQueryParseState::Questions;
break;
case DnsQueryParseState::Questions:
Expand Down Expand Up @@ -741,7 +741,7 @@ void DnsMessageParser::buildResponseBuffer(DnsQueryContextPtr& query_context,
buffer.writeBEInt<uint16_t>(response_header_.id);

uint16_t flags;
::memcpy(&flags, static_cast<void*>(&response_header_.flags), sizeof(uint16_t));
SAFE_MEMCPY(&flags, &(response_header_.flags));
buffer.writeBEInt<uint16_t>(flags);

buffer.writeBEInt<uint16_t>(response_header_.questions);
Expand Down
1 change: 1 addition & 0 deletions source/server/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ envoy_cc_library(
"//include/envoy/stats:stats_interface",
"//source/common/api:os_sys_calls_lib",
"//source/common/common:assert_lib",
"//source/common/common:mem_block_builder_lib",
"//source/common/common:utility_lib",
"//source/common/network:utility_lib",
"//source/common/stats:utility_lib",
Expand Down
4 changes: 3 additions & 1 deletion source/server/hot_restarting_base.cc
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#include "server/hot_restarting_base.h"

#include "common/api/os_sys_calls_impl.h"
#include "common/common/mem_block_builder.h"
#include "common/common/utility.h"
#include "common/network/address_impl.h"
#include "common/stats/utility.h"
Expand Down Expand Up @@ -36,8 +37,9 @@ sockaddr_un HotRestartingBase::createDomainSocketAddress(uint64_t id, const std:
initDomainSocketAddress(&address);
Network::Address::PipeInstance addr(fmt::format(socket_path + "_{}_{}", role, base_id_ + id),
socket_mode, nullptr);
memcpy(&address, addr.sockAddr(), addr.sockAddrLen());
addr.getSockAddr(address);
fchmod(my_domain_socket_, socket_mode);

return address;
}

Expand Down