(#530) Add test cases to exercise print methods for mini-allocator metadata pages and for trunk verbose logging. #531

gapisback · 2023-01-15T00:22:07Z

This commit adds new print function to drive the functions for printing keyed / unkeyed metadata pages of the mini-allocator. Existing test cases are enhanced as follows to eventually exercise these print methods.

Extend splinter_test:test_splinter_print_diags() to print just one routing-filter page hanging off trunk's root node. New function trunk_print_root_nodes_filter_metapages() will exercise the print function that will walk through routing filter metadata pages for one pivot key and eventually calls mini_unkeyed_print().
Modularize code in btree_stress_test.c to carve out code that is used to build new unit-test case, test_btree_print_diags(). First, this invokes btree_print_tree(), to see outputs of packed BTree node. Then, invokes mini_keyed_print(), to print keyed mini-allocator's metadata pages.
Add bunch of print functions in trunk.c to find out routing filter metadata pages and to invoke underlying print methods on such pages.

NOTE: This is a draft PR to discuss the changes with Alex & Rob.

[Updated 2.1.2023]: Fixed a bug. Much of the driving functions seem to work ok. The print outputs from updated test cases for keyed and unkeyed mini-allocator's metadata pages for "real-data" (generated by inserting lots of rows into trunk or BTree) seem to work fine.

netlify · 2023-01-15T00:22:11Z

✅ Deploy Preview for splinterdb canceled.

Name	Link
🔨 Latest commit	`c7f0422`
🔍 Latest deploy log	https://app.netlify.com/sites/splinterdb/deploys/63dc46027b936f0009c861fa

gapisback · 2023-01-15T00:23:16Z

src/btree.c

-                        uint64              meta_page_no)
-{
-   return root_addr + (meta_page_no + 1) * btree_page_size(cfg);
-}


Externalized in .h file so this can be called from unit test case in btree_stress_test.c:test_btree_print_diags().

gapisback · 2023-01-15T00:24:12Z

src/mini_allocator.c

@@ -337,7 +337,7 @@ static uint64
 mini_num_entries(page_handle *meta_page)
 {
   mini_meta_hdr *hdr = (mini_meta_hdr *)meta_page->data;
-   return hdr->num_entries;
+   return (uint64)hdr->num_entries;


Shouldn't really matter but just conforming to return type declared for this fn's definition.

gapisback · 2023-01-15T00:25:32Z

src/trunk.c

@@ -105,6 +105,19 @@ static const int64 latency_histo_buckets[LATENCYHISTO_SIZE] = {
 * If verbose_logging_enabled is enabled in trunk_config, these functions print
 * to cfg->log_handle.
 */
+void
+trunk_enable_verbose_logging(trunk_handle *spl, platform_log_handle *log_handle)


Externalized so this can be called directly from unit-test which is not required to go through --cmd-line-args parsing.

gapisback · 2023-01-15T00:26:16Z

src/trunk.c

      }                                                                        \
   } while (0)

 #define trunk_default_log_if_enabled(spl, message, ...)                        \
   do {                                                                        \
      if (trunk_verbose_logging_enabled(spl)) {                                \
-         platform_default_log(message, __VA_ARGS__);                           \
+         platform_default_log(                                                 \
+            "trunk_log():%d " message, __LINE__, __VA_ARGS__);                 \


Here, and on L161 above, print the line # where this trace message is generated from, which helps quickly spotting the code which got executed.

gapisback · 2023-01-15T00:29:50Z

tests/unit/btree_stress_test.c

-      params[i].start     = i * (nkvs / nthreads);
-      params[i].end = i < nthreads - 1 ? (i + 1) * (nkvs / nthreads) : nkvs;
-   }
+   load_thread_params(params,


Modularized this out so it can be used in the btree_print_diags() test-case, which also has to do the same sort of thing to insert a bunch of data.

gapisback · 2023-01-15T00:30:34Z

tests/unit/btree_stress_test.c

   // Exercise print method to verify that it basically continues to work.
+   CTEST_LOG_INFO("\n**** btree_print_tree() on BTree root=%lu****\n",
+                  packed_root_addr);


This is an existing invocation of a BTree-print method, which has been working fine (previously, too).

gapisback · 2023-01-15T00:39:55Z

tests/unit/btree_stress_test.c

+                  meta_page_addr);
+
+   // Exercise print method of mini-allocator's keyed meta-page
+   mini_keyed_print(


This is a new invocation of this print method to print BTree metadata page contents.

This invocation seems to work right, too, and prints what seem like reasonable content:

**** mini_keyed_print() BTree root=1326710784, meta page addr=1326714880 **** --------------------------------------------------------------------- | Mini Keyed Allocator -- meta_head: 1326714880 | --------------------------------------------------------------------- | meta addr: 1326714880 (2) | --------------------------------------------------------------------- | idx | batch | extent_addr | start_key | rc | --------------------------------------------------------------------- | 0 | 0 | 1326841856 | 0x00000d093a9a1700 | 2 | | 1 | 0 | 1327890432 | 0x000742d1b6583900 | 2 | | 2 | 1 | 1326972928 | 0x00000d093a9a1700 | 2 | | 3 | 0 | 1328021504 | 0x00137c3e4b314000 | 2 | | 4 | 0 | 1328283648 | 0x001df84b01da2a00 | 2 | | 5 | 0 | 1328414720 | 0x0026020907020200 | 2 | | 6 | 0 | 1328545792 | 0x0031fa2efde54b00 | 2 | | 7 | 0 | 1328676864 | 0x0039713e18992a00 | 2 | | 8 | 0 | 1328807936 | 0x0044cc7bbd592300 | 2 | | 9 | 0 | 1328939008 | 0x005006e951322a00 | 2 | [...]

and so on.

gapisback · 2023-01-15T00:42:35Z

tests/unit/btree_stress_test.c

   // Exercise print method to verify that it basically continues to work.
+   CTEST_LOG_INFO("\n**** btree_print_tree() on BTree root=%lu****\n",
+                  packed_root_addr);
   btree_print_tree(Platform_default_log_handle,


Old code relocated from existing test_random_inserts_concurrent test case to newly refactored test_btree_print_diags test case.

This commit does some minor cleanup of the mini-allocator module. Some interfaces are slightly changed to streamline init / deinit interfaces. A new unit-test is added to exercise core interfaces of this page allocation system, and to exercise / test the print methods for keyed and unkeyed page allocation schemes. Extend print methods to also print page_addr and extent_addr as page_num and extent_num, where applicable.

gapisback · 2023-02-01T20:52:02Z

src/trunk.c

+   platform_log(log_handle,
+                "\nFilter Metadata page starting from meta_head=%lu\n{\n",
+                meta_head);
+   mini_unkeyed_print(spl->cc, meta_head, PAGE_TYPE_FILTER);


Here is a sample output from this call, for a routing filter off of trunk's root node:

Filter Metadata page starting from meta_head=6200360960 { --------------------------------------------- | Mini Allocator -- meta_head=6200360960 (pgnum=1513760) |-------------------------------------------| { |-------------------------------------------| | meta addr=6200360960 (pgnum=1513760), num_entries=4 | next_meta_addr=0 (pgnum=0), pos=52 |-------------------------------------------| | idx | extent_addr | extent_num | |-------------------------------------------| | 0 | 6200492032 | 47306 | | 1 | 6200623104 | 47307 | | 2 | 6200754176 | 47308 | | 3 | 6200885248 | 47309 | |-------------------------------------------| } Found 1 meta-data pages tracking 4 extents. }

gapisback · 2023-02-01T20:53:32Z

tests/unit/splinter_test.c

+ * generated by trunk_default_log_if_enabled().
+ * -----------------------------------------------------------------------------
+ */
+CTEST2(splinter, test_splinter_verbose_print_diags)


Added new test case to exercise the calls to trunk_default_log_if_enabled() at different code points, which prints intermediate state of trunk trees while large insertions, and other activities (like compaction etc.) are going on.

…tapages This commit adds new print function to drive the functions for printing keyed / unkeyed metadata pages of the mini-allocator. Existing test cases are enhanced as follows to eventually exercise these print methods. - Add bunch of print functions in trunk.c to find out routing filter metadata pages and to invoke underlying print methods on such pages. Print pivot key and other filter-related data. - Extend splinter_test:test_splinter_print_diags() to exercise the print function that will walk through routing filter metadata pages for one pivot key off of the trunk's root node and eventually callss mini_unkeyed_print(). - Add new test_splinter_verbose_print_diags test case to exercise the verbose print logging in trunk.c (for a small data set). - Modularize code in btree_stress_test.c to carve out code that is used to build new unit-test case, test_btree_print_diags(). First, this invokes btree_print_tree(), to see outputs of packed BTree node. Then, invokes mini_keyed_print(), to print keyed mini-allocator's metadata pages.

gapisback · 2023-03-24T17:25:10Z

Draft PR. This dev work was surfaced as 2-part PRs which were internally discussed. There was no interest in absorbing this work to /main.

Abandoned.

vmwclabot added the cla-not-required label Jan 15, 2023

gapisback requested a review from ajhconway January 15, 2023 00:22

gapisback commented Jan 15, 2023

View reviewed changes

gapisback requested a review from rtjohnso January 15, 2023 00:43

gapisback mentioned this pull request Jan 18, 2023

Rework env-var interface to set log streams from unit-tests that exercise print diagnostics code. #534

Open

gapisback mentioned this pull request Jan 27, 2023

Cleanup of mini_allocator.c . Add unit-tests to exercise this module. #536

Closed

gapisback changed the title ~~(#530) Add test cases to exercise print methods for mini-allocator metadata pages~~ (#530) Add test cases to exercise print methods for mini-allocator metadata pages and for trunk verbose logging. Feb 1, 2023

gapisback force-pushed the agurajada/530-mini-allocator-print-fns branch from ec129c9 to f63bb07 Compare February 1, 2023 20:46

gapisback commented Feb 1, 2023

View reviewed changes

gapisback force-pushed the agurajada/530-mini-allocator-print-fns branch from f63bb07 to eb54892 Compare February 1, 2023 23:26

gapisback added 2 commits February 2, 2023 08:51

Bracket sensibly chunks of tree outputs in { }.

c7f0422

gapisback force-pushed the agurajada/530-mini-allocator-print-fns branch from eb54892 to c7f0422 Compare February 2, 2023 23:23

gapisback closed this Mar 24, 2023

gapisback mentioned this pull request Apr 17, 2023

Add test cases to exercise mini_unkeyed_print() and mini_keyed_print() methods found in mini_allocator.c #530

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(#530) Add test cases to exercise print methods for mini-allocator metadata pages and for trunk verbose logging. #531

(#530) Add test cases to exercise print methods for mini-allocator metadata pages and for trunk verbose logging. #531

gapisback commented Jan 15, 2023 •

edited

Loading

netlify bot commented Jan 15, 2023 •

edited

Loading

gapisback Jan 15, 2023

gapisback Jan 15, 2023

gapisback Jan 15, 2023

gapisback Jan 15, 2023

gapisback Jan 15, 2023

gapisback Jan 15, 2023 •

edited

Loading

gapisback Jan 15, 2023

gapisback Jan 15, 2023

gapisback Feb 1, 2023

gapisback Feb 1, 2023

gapisback commented Mar 24, 2023

(#530) Add test cases to exercise print methods for mini-allocator metadata pages and for trunk verbose logging. #531

(#530) Add test cases to exercise print methods for mini-allocator metadata pages and for trunk verbose logging. #531

Conversation

gapisback commented Jan 15, 2023 • edited Loading

netlify bot commented Jan 15, 2023 • edited Loading

✅ Deploy Preview for splinterdb canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gapisback Jan 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gapisback commented Mar 24, 2023

gapisback commented Jan 15, 2023 •

edited

Loading

netlify bot commented Jan 15, 2023 •

edited

Loading

gapisback Jan 15, 2023 •

edited

Loading