Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide single-threaded (no-locking) implementation of IMetadataEmit family of APIs. #52

Merged
merged 52 commits into from
Jun 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
b04ad89
Add C++ RAII helper for row adds. Add declaration of IMetaDataEmit an…
jkoritzinsky Aug 14, 2023
3e967bc
Implement saving.
jkoritzinsky Aug 30, 2023
253b9f4
Implement more methods.
jkoritzinsky Aug 30, 2023
cefd384
Change md_apply_delta to take a delta image handle instead of the raw…
jkoritzinsky Sep 13, 2023
9132aed
Implement IMetadataEmit::ApplyEditAndContinue
jkoritzinsky Sep 13, 2023
3d42493
Implement most of DefineImportType
jkoritzinsky Sep 14, 2023
d978f36
Implement DefineMemberRef and fix QIs in DefineImportType to use the …
jkoritzinsky Sep 14, 2023
4ed2f5f
Stub out EnC-related methods and implement a few more methods
jkoritzinsky Sep 14, 2023
c8821b8
Implement DefineField
jkoritzinsky Sep 18, 2023
538d8bc
Implement SetFieldProps
jkoritzinsky Sep 18, 2023
108317a
Implement most of the remaining APIs on IMetaDataEmit/IMetaDataEmit2/…
jkoritzinsky Nov 29, 2023
6136ed5
Implement Param methods
jkoritzinsky Dec 12, 2023
f5cff9d
Provide a no-op implementation of SetHandler as we don't do any token…
jkoritzinsky Dec 12, 2023
0b7966d
Fix errors and correctly handle names in SetParamProps
jkoritzinsky Dec 12, 2023
48f79b6
Remove TranslateSigWithScope helper and delegate back to the correct …
jkoritzinsky Dec 13, 2023
ac00eb1
Merge metadataemithelpers and importhelpers
jkoritzinsky Jan 16, 2024
84520b6
Use helpers instead of re-implementing the algorithms and implement r…
jkoritzinsky Jan 16, 2024
0195a85
Don't use UINT32 type as it's not defined by dncp
jkoritzinsky Jan 16, 2024
51c83ff
Update dncp
jkoritzinsky Jan 17, 2024
3eceff0
Fix Clang failures
jkoritzinsky Jan 17, 2024
cdc4e8c
Define IMetaDataAssemblyEmit IID
jkoritzinsky Jan 17, 2024
69d4d30
Add override specifier
jkoritzinsky Jan 17, 2024
b14b2db
Make DefineScope result editable
jkoritzinsky Jan 18, 2024
34d3111
Correctly handle null heap entries as per the ECMA-335 spec (emit the…
jkoritzinsky Jan 20, 2024
2e24a1a
Zero-initialize each column in the ::DefineX methods when the user do…
jkoritzinsky Jan 20, 2024
8820812
Initialize currentImplementation
jkoritzinsky Jan 20, 2024
d78baa3
Fix incorrect dereference
jkoritzinsky Jan 20, 2024
3323d11
Fix bugs in DefineScope path and add first emit unit test
jkoritzinsky Jan 26, 2024
b65cda2
Add tests for some corner case scenarios.
jkoritzinsky Jan 26, 2024
c83191e
Add CMake option to enable msvc profiling to enable code coverage in …
jkoritzinsky Jan 27, 2024
ef58687
Add tests for SetModuleProps
jkoritzinsky Jan 29, 2024
df8f518
Implement tests for defining typedefs and fix bugs found in the process.
jkoritzinsky Jan 30, 2024
3c75715
Add test for SetTypeDefProps
jkoritzinsky Jan 30, 2024
061e4d5
Flip include order for non-Windows
jkoritzinsky Jan 30, 2024
28229cf
Add basic "define MethodDef" test and fix bugs.
jkoritzinsky Jan 30, 2024
9491f51
Add more methoddef tests and standalonesig tests
jkoritzinsky Jan 31, 2024
027ece6
Add tests for MemberRef
jkoritzinsky Jan 31, 2024
b611631
Add tests for TypeSpec
jkoritzinsky Jan 31, 2024
374ebda
Add Assembly test and update DNMD APIs to behave better around zero-l…
jkoritzinsky Feb 13, 2024
858aac7
Add AssemblyRef test
jkoritzinsky Feb 15, 2024
361b5e8
Add some parameter tests that also set constants
jkoritzinsky Feb 15, 2024
dbc497d
Add a test for out of order parameters at the start of the parameter …
jkoritzinsky Feb 16, 2024
ef6747d
Add tests for FieldMarshal and FieldRva table-related APIs and fix un…
jkoritzinsky Mar 12, 2024
1d4c98f
Just avoid using max
jkoritzinsky Mar 12, 2024
bc23799
Include cstddef for size_t
jkoritzinsky Mar 12, 2024
ce744d3
Fix incorrect wide-char spec
jkoritzinsky Mar 13, 2024
4661260
Add necessary const casts
jkoritzinsky Mar 13, 2024
061faf5
PR feedback excluding east-const
jkoritzinsky Jun 13, 2024
e940d12
Fix #US physical blob construction and make editor creation a little …
jkoritzinsky Jun 13, 2024
2ede98b
East const
jkoritzinsky Jun 13, 2024
8996494
Produce position-independent code to enable linking with the shared e…
jkoritzinsky Jun 13, 2024
a68f4fb
Add TestResults to gitignore so VS results from coverage/profiling ar…
jkoritzinsky Jun 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
artifacts/
.vs/
.idea/
*~
*~
TestResults/
8 changes: 7 additions & 1 deletion configure.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -55,4 +55,10 @@ endif()
#
# Agnostic compiler/platform settings
#
add_compile_definitions(__STDC_WANT_LIB_EXT1__=1) # https://en.cppreference.com/w/c/error#Bounds_checking
add_compile_definitions(__STDC_WANT_LIB_EXT1__=1) # https://en.cppreference.com/w/c/error#Bounds_checking

option(DNMD_ENABLE_PROFILING OFF)

if (DNMD_ENABLE_PROFILING AND MSVC)
add_link_options(/PROFILE)
endif()
2 changes: 1 addition & 1 deletion external/dncp
6 changes: 4 additions & 2 deletions src/dnmd/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -37,10 +37,12 @@ target_include_directories(dnmd PUBLIC $<INSTALL_INTERFACE:include>)
target_include_directories(dnmd_pdb PUBLIC $<INSTALL_INTERFACE:include>)

set_target_properties(dnmd PROPERTIES
PUBLIC_HEADER "../inc/dnmd.h;../inc/dnmd.hpp")
PUBLIC_HEADER "../inc/dnmd.h;../inc/dnmd.hpp"
POSITION_INDEPENDENT_CODE ON)

set_target_properties(dnmd_pdb PROPERTIES
PUBLIC_HEADER "../inc/dnmd.h;../inc/dnmd.hpp;../inc/dnmd_pdb.h")
PUBLIC_HEADER "../inc/dnmd.h;../inc/dnmd.hpp;../inc/dnmd_pdb.h"
POSITION_INDEPENDENT_CODE ON)

install(TARGETS dnmd dnmd_pdb EXPORT dnmd
PUBLIC_HEADER DESTINATION include
Expand Down
74 changes: 60 additions & 14 deletions src/dnmd/editor.c
Original file line number Diff line number Diff line change
Expand Up @@ -568,7 +568,7 @@ static bool reserve_heap_space(mdeditor_t* editor, uint32_t space_size, mdtcol_t
{
// Set the default heap size based on likely reasonable sizes for the heaps.
// In most images, there won't be more than three guids, so we can start with a small heap in that case.
const size_t initial_heap_size = heap_id == mdtc_hguid ? sizeof(mdguid_t) * 3 : 0x100;
size_t const initial_heap_size = heap_id == mdtc_hguid ? sizeof(mdguid_t) * 3 : 0x100;
void* mem = alloc_mdmem(editor->cxt, initial_heap_size);
if (mem == NULL)
return false;
Expand Down Expand Up @@ -628,6 +628,14 @@ static bool reserve_heap_space(mdeditor_t* editor, uint32_t space_size, mdtcol_t

uint32_t add_to_string_heap(mdcxt_t* cxt, char const* str)
{
// II.24.2.3 - When the #String heap is present, the first entry is always the empty string (i.e., \0).
// II.24.2.2 - Streams need not be there if they are empty.
// We can avoid allocating the heap if the only entry is the empty string.
// Columns that point to the string heap can be 0 if there is no #String heap.
// In that case, they represent the empty string.
if (str[0] == '\0')
return 0;

mdeditor_t* editor = get_editor(cxt);
if (editor == NULL)
return 0;
Expand All @@ -646,12 +654,21 @@ uint32_t add_to_string_heap(mdcxt_t* cxt, char const* str)

uint32_t add_to_blob_heap(mdcxt_t* cxt, uint8_t const* data, uint32_t length)
{
// II.24.2.4 - When the #Blob heap is present, the first entry is always the empty blob.
// II.24.2.2 - Streams need not be there if they are empty.
// We can avoid allocating the heap if the only entry is the empty blob.
// Columns that point to the blob heap can be 0 if there is no #Blob heap.
// In that case, they represent the empty blob.
if (length == 0)
return 0;

mdeditor_t* editor = get_editor(cxt);
if (editor == NULL)
return 0;

// TODO: Deduplicate heap
uint8_t compressed_length[4];
size_t compressed_length_size = 0;
size_t compressed_length_size = ARRAY_SIZE(compressed_length);
if (!compress_u32(length, compressed_length, &compressed_length_size))
return 0;

Expand All @@ -670,16 +687,12 @@ uint32_t add_to_blob_heap(mdcxt_t* cxt, uint8_t const* data, uint32_t length)

uint32_t add_to_user_string_heap(mdcxt_t* cxt, char16_t const* str)
{
mdeditor_t* editor = get_editor(cxt);
if (editor == NULL)
return 0;
// TODO: Deduplicate heap
uint32_t str_len;
uint8_t has_special_char = 0;
for (str_len = 0; str[str_len] != (char16_t)0; str_len++)
{
char16_t c = str[str_len];
// II.23.2.4
// II.24.2.4
// There is an additional terminal byte which holds a 1 or 0.
// The 1 signifies Unicode characters that require handling beyond
// that normally provided for 8-bit encoding sets.
Expand Down Expand Up @@ -713,31 +726,61 @@ uint32_t add_to_user_string_heap(mdcxt_t* cxt, char16_t const* str)
}
}

uint8_t compressed_length[sizeof(str_len)];
size_t compressed_length_size = 0;
if (!compress_u32(str_len, compressed_length, &compressed_length_size))
// II.24.2.4 - When the #US heap is present, the first entry is always the empty blob.
// II.24.2.2 - Streams need not be there if they are empty.
// We can avoid allocating the heap if the only entry is the empty blob.
// Indices into the #US heap can be 0 if there is no #US heap.
// In that case, they represent the empty userstring blob.
if (str_len == 0)
return 0;

uint32_t heap_slot_size = str_len + (uint32_t)compressed_length_size + 1;
mdeditor_t* editor = get_editor(cxt);
if (editor == NULL)
return 0;

// TODO: Deduplicate heap

// II.24.2.4
// Strings in the #US (user string) heap are encoded using 16-bit Unicode encodings.
// The count on each string is the number of bytes (not characters) in the string.
// Furthermore, there is an additional terminal byte (so all byte counts are odd, not even).
size_t us_blob_bytes = str_len * sizeof(char16_t) + 1;

// The string is too long to represent in the heap.
if (us_blob_bytes > INT32_MAX)
return 0;
uint8_t compressed_length[sizeof(uint32_t)];
size_t compressed_length_size = ARRAY_SIZE(compressed_length);
if (!compress_u32((uint32_t)us_blob_bytes, compressed_length, &compressed_length_size))
return 0;

uint32_t heap_slot_size = (uint32_t)us_blob_bytes + (uint32_t)compressed_length_size;
uint32_t heap_offset;
if (!reserve_heap_space(editor, heap_slot_size, mdtc_hus, false, &heap_offset))
{
return 0;
}

// Copy the compressed blob length into the heap.
memcpy(editor->user_string_heap.heap.ptr + heap_offset, compressed_length, compressed_length_size);
memcpy(editor->user_string_heap.heap.ptr + heap_offset + compressed_length_size, str, str_len);
// Copy the UTF-16-encoded user string into the heap.
memcpy(editor->user_string_heap.heap.ptr + heap_offset + compressed_length_size, str, us_blob_bytes - 1);

editor->user_string_heap.heap.ptr[heap_offset + compressed_length_size + str_len] = has_special_char;
// Set the trailing byte.
editor->user_string_heap.heap.ptr[heap_offset + compressed_length_size + us_blob_bytes - 1] = has_special_char;
return heap_offset;
}

mdguid_t const empty_guid = { 0 };

uint32_t add_to_guid_heap(mdcxt_t* cxt, mdguid_t guid)
{
mdeditor_t* editor = get_editor(cxt);
if (editor == NULL)
return 0;
// TODO: Deduplicate heap
if (memcmp(&guid, &empty_guid, sizeof(mdguid_t)) == 0)
return 0;

uint32_t heap_offset;
if (!reserve_heap_space(editor, sizeof(mdguid_t), mdtc_hguid, false, &heap_offset))
Expand All @@ -746,7 +789,10 @@ uint32_t add_to_guid_heap(mdcxt_t* cxt, mdguid_t guid)
}

memcpy(editor->guid_heap.heap.ptr + heap_offset, &guid, sizeof(mdguid_t));
return heap_offset / sizeof(mdguid_t);
// II.22 - The Guid heap is an array of GUIDs, each 16 bytes wide. Its
// first element is numbered 1, its second 2, and so on.
// So, we need to make the offset 1-based and at the scale of the GUID size.
return (heap_offset / sizeof(mdguid_t)) + 1;
}

bool append_heap(mdcxt_t* cxt, mdcxt_t* delta, mdtcol_t heap_id)
Expand Down
13 changes: 4 additions & 9 deletions src/dnmd/entry.c
Original file line number Diff line number Diff line change
Expand Up @@ -297,7 +297,7 @@ static bool initialize_minimal_table_rows(mdcxt_t* cxt)
if (1 != md_set_column_value_as_utf8(global_type_cursor, mdtTypeDef_TypeNamespace, 1, &namespace))
return false;

mdToken nil_typedef = CreateTokenType(mdtTypeDef);
mdToken nil_typedef = CreateTokenType(mdtid_TypeDef);
if (1 != md_set_column_value_as_token(global_type_cursor, mdtTypeDef_Extends, 1, &nil_typedef))
return false;

Expand Down Expand Up @@ -361,26 +361,21 @@ mdhandle_t md_create_new_pdb_handle()
}
#endif // DNMD_PORTABLE_PDB

bool md_apply_delta(mdhandle_t handle, void const* data, size_t data_len)
bool md_apply_delta(mdhandle_t handle, mdhandle_t delta_handle)
{
mdcxt_t* base = extract_mdcxt(handle);
if (base == NULL)
return false;

mdhandle_t h;
if (!md_create_handle(data, data_len, &h))
mdcxt_t* delta = extract_mdcxt(delta_handle);
if (delta == NULL)
return false;

mdcxt_t* delta = extract_mdcxt(h);
assert(delta != NULL);

// Verify the supplied delta is actually a delta file
bool result = false;
if (delta->context_flags & mdc_minimal_delta)
result = merge_in_delta(base, delta);

// Free all data for the delta
md_destroy_handle(h);
return result;
}

Expand Down
2 changes: 2 additions & 0 deletions src/dnmd/internal.h
Original file line number Diff line number Diff line change
Expand Up @@ -430,4 +430,6 @@ bool sort_list_by_column(mdcursor_t parent, col_index_t list_col, col_index_t co
// Add the heap with the specified id from the delta image to the cxt image.
bool append_heap(mdcxt_t* cxt, mdcxt_t* delta, mdtcol_t heap_id);

extern mdguid_t const empty_guid;

#endif // _SRC_DNMD_INTERNAL_H_
20 changes: 20 additions & 0 deletions src/dnmd/streams.c
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,17 @@ bool try_get_string(mdcxt_t* cxt, size_t offset, char const** str)
assert(cxt != NULL && str != NULL);

mdstream_t* h = &cxt->strings_heap;

// II.24.2.3 - When the #String heap is present, the first entry is always the empty string (i.e., \0).
// II.24.2.2 - Streams need not be there if they are empty.
// If the offset into the heap is 0, we can treat that as a "null" index into the heap and return
// the empty string.
if (h->size == 0 && offset == 0)
jkoritzinsky marked this conversation as resolved.
Show resolved Hide resolved
{
*str = "\0"; // II.24.2.3 'The first character must be the '\0' character.
return true;
}

if (h->size <= offset)
return false;

Expand Down Expand Up @@ -81,6 +92,15 @@ bool try_get_blob(mdcxt_t* cxt, size_t offset, uint8_t const** blob, uint32_t* b
assert(cxt != NULL && blob != NULL && blob_len != NULL);

mdstream_t* h = &cxt->blob_heap;

if (h->size == 0 && offset == 0)
{
// The first element must be the 0 - II.24.2.4
*blob = h->ptr;
*blob_len = 0;
return true;
}

if (h->size <= offset)
return false;

Expand Down
9 changes: 8 additions & 1 deletion src/dnmd/tables.c
Original file line number Diff line number Diff line change
Expand Up @@ -772,11 +772,18 @@ bool initialize_new_table_details(
// Set the new table's row count temporarily to 1 to ensure that we initialize the table.
table_row_counts[id] = 1;

// We'll treat any new table that has keys as sorted.
// We only want to do this for tables with keys as tables without keys
// never use the is_sorted bit.
md_key_info_t const* keys;
uint8_t key_count = get_table_keys(id, &keys);
bool has_keys = key_count != 0;

if (!initialize_table_details(
table_row_counts,
cxt->context_flags,
id,
false,
has_keys,
table))
return false;

Expand Down
Loading
Loading