Skip to content

Commit 0884588

Browse files
jeanbezhoujunTheAssembler1
authored
Synchronize with develop (#242)
* Fix cache flush (#226) * Fix a thread race issue that may cause memory error when larger than cache max size data is transferred * Add a test that writes more data than server cache size * Fix CI run command * Update nersc.yml (#238) * Since PDCinit returns a uint64_t, 0 should indicate failure (#233) Co-authored-by: Jean Luca Bez <[email protected]> * Check the return value of `PDC_Client_init` in `PDC_init` (#230) * Check that return value of PDC_Client_init in PDC_init * Change return to 0 This will make is simpler when merging #233 (comment) --------- Co-authored-by: Jean Luca Bez <[email protected]> * Change `printf` to PDC logger (#232) * Changed all printf to use pdc logger Also removed large blocks of comments and chanegd the pdc logger to print the file name, function, and line number. * Change typo of LOG_INFO to LOG_ERROR * Correct grammar from fail -> failed * update grammer succesfully close -> successfully closed * switch type of LOG_INFO to LOG_ERROR * Add logging docs and fix some LOG_INFO->LOG_JUST_PRINT * update clang formatting --------- Co-authored-by: Jean Luca Bez <[email protected]> * Malloc correct size for pdc_obj_metadata_pkg (#237) Co-authored-by: Jean Luca Bez <[email protected]> * PDCregion_transfer_create validate client buf, local region, and remote regions (#236) Co-authored-by: Jean Luca Bez <[email protected]> --------- Co-authored-by: Houjun Tang <[email protected]> Co-authored-by: Noah Lewis <[email protected]>
1 parent c307e18 commit 0884588

File tree

243 files changed

+7427
-9308
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

243 files changed

+7427
-9308
lines changed

‎.github/workflows/nersc.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
name: NERSC
22

33
on:
4-
pull_request
4+
pull_request_target
55

66
env:
77
PR_NUMBER: ${{ github.event.number }}

‎.gitlab-ci.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -108,7 +108,7 @@ perlmutter-cache-build:
108108
- module list
109109
- mkdir -p ${PDC_BUILD_PATH}/perlmutter/cache
110110
- cd ${PDC_BUILD_PATH}/perlmutter/cache
111-
- cmake ../../.. -DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DPDC_SERVER_CACHE=ON -DBUILD_TESTING=ON -DCMAKE_INSTALL_PREFIX=$PDC_DIR -DPDC_ENABLE_MPI=ON -DMERCURY_DIR=$MERCURY_DIR -DCMAKE_C_COMPILER=cc -DMPI_RUN_CMD="srun -A ${PDC_PROJECT} --qos=debug --constraint=cpu --tasks-per-node=64" -DCMAKE_INSTALL_PREFIX=${PDC_INSTALL_PATH}/perlmutter/cache
111+
- cmake ../../.. -DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DPDC_SERVER_CACHE=ON -DBUILD_TESTING=ON -DPDC_SERVER_CACHE_MAX_GB=1 -DCMAKE_INSTALL_PREFIX=$PDC_DIR -DPDC_ENABLE_MPI=ON -DMERCURY_DIR=$MERCURY_DIR -DCMAKE_C_COMPILER=cc -DMPI_RUN_CMD="srun -A ${PDC_PROJECT} --qos=debug --constraint=cpu --tasks-per-node=64" -DCMAKE_INSTALL_PREFIX=${PDC_INSTALL_PATH}/perlmutter/cache
112112
- make -j
113113
- make install
114114
artifacts:

‎docs/readme.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@
6868
- Input:
6969
+ pdc_name is the reference for PDC class. Recommended use "pdc"
7070
- Output:
71-
+ PDC class ID used for future reference.
71+
+ PDC class ID used for future reference or 0 on failure.
7272
- All PDC client applications must call PDCinit before using it. This function will setup connections from clients to servers. A valid PDC server must be running.
7373
- For developers: currently implemented in pdc.c.
7474
+ perr_t PDCclose(pdcid_t pdcid)

‎docs/source/developer-notes.rst

+24
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,30 @@
22
Developer Notes
33
================================
44

5+
+++++++++++++++++++++++++++++++++++++++++++++
6+
PDC Logging
7+
+++++++++++++++++++++++++++++++++++++++++++++
8+
9+
The following macros are defined in the header file `src/commons/logging/include/pdc_logger.h` and should be used instead of `printf` for logging.
10+
11+
The available macros, which follow a similar usage to `printf`, are:
12+
13+
.. code-block:: C
14+
LOG_ERROR
15+
LOG_WARNING
16+
LOG_INFO
17+
LOG_DEBUG
18+
LOG_JUST_PRINT
19+
20+
Each of these macros automatically prepends a timestamp and log level prefix to the output:
21+
22+
.. code-block:: C
23+
[year-month-day hour:minute:second] [ERROR]
24+
[year-month-day hour:minute:second] [WARNING]
25+
[year-month-day hour:minute:second] [INFO]
26+
[year-month-day hour:minute:second] [DEBUG]
27+
28+
`LOG_JUST_PRINT` does not print a prefix and outputs the message as-is. This macro is useful when you need to print raw messages without log metadata.
529

630
+++++++++++++++++++++++++++++++++++++++++++++
731
PDC Server Metadata Management

‎examples/C_plus_plus_example/H5Timing.cc

+5-5
Original file line numberDiff line numberDiff line change
@@ -185,9 +185,9 @@ static int record_timer(std::vector<H5Timer> *timer, const char* filename) {
185185

186186
static int output_results() {
187187
#ifdef PDC_PATCH
188-
printf("total PDCwrite calls = %d, PDCread calls = %d\n", timer_class->PDCwrite_count, timer_class->PDCread_count);
188+
LOG_INFO("total PDCwrite calls = %d, PDCread calls = %d\n", timer_class->PDCwrite_count, timer_class->PDCread_count);
189189
#else
190-
printf("total H5Dwrite calls = %d, H5Dread calls = %d\n", timer_class->H5Dwrite_count, timer_class->H5Dread_count);
190+
LOG_INFO("total H5Dwrite calls = %d, H5Dread calls = %d\n", timer_class->H5Dwrite_count, timer_class->H5Dread_count);
191191
#endif
192192
if ( timer_class->dataset_timers->size() ) {
193193
record_timer(timer_class->dataset_timers, "dataset_write_record.csv");
@@ -211,10 +211,10 @@ int finalize_timers() {
211211
gettimeofday(&temp_time, NULL);
212212
timer_class->total_end_time = (temp_time.tv_usec + temp_time.tv_sec * 1000000) + .0;
213213
#ifdef PDC_PATCH
214-
printf("total program time is %lf, PDCstart time = %lf, PDCwait time = %lf\n", (timer_class->total_end_time - timer_class->total_start_time) / 1000000.0, timer_class->PDCstart_time / 1000000.0, timer_class->PDCwait_time / 1000000.0);
214+
LOG_INFO("total program time is %lf, PDCstart time = %lf, PDCwait time = %lf\n", (timer_class->total_end_time - timer_class->total_start_time) / 1000000.0, timer_class->PDCstart_time / 1000000.0, timer_class->PDCwait_time / 1000000.0);
215215
#else
216-
printf("total program time is %lf, H5Dwrite time = %lf, H5Dread time = %lf\n", (timer_class->total_end_time - timer_class->total_start_time) / 1000000.0, timer_class->H5Dwrite_time / 1000000.0, timer_class->H5Dread_time / 1000000.0);
217-
printf("merge requests time = %lf, wrap requests time = %lf, H5Dclose = %lf\n", timer_class->merge_requests_time / 1000000.0, timer_class->wrap_requests_time / 1000000.0, timer_class->H5Dclose_time / 1000000.0);
216+
LOG_INFO("total program time is %lf, H5Dwrite time = %lf, H5Dread time = %lf\n", (timer_class->total_end_time - timer_class->total_start_time) / 1000000.0, timer_class->H5Dwrite_time / 1000000.0, timer_class->H5Dread_time / 1000000.0);
217+
LOG_INFO("merge requests time = %lf, wrap requests time = %lf, H5Dclose = %lf\n", timer_class->merge_requests_time / 1000000.0, timer_class->wrap_requests_time / 1000000.0, timer_class->H5Dclose_time / 1000000.0);
218218
#endif
219219
free(timer_class);
220220

‎examples/C_plus_plus_example/multidataset_plugin.cc

+3-3
Original file line numberDiff line numberDiff line change
@@ -276,7 +276,7 @@ int flush_multidatasets() {
276276
double start_time;
277277
#endif
278278
i = 0;
279-
//printf("Rank %d number of datasets to be written %d\n", rank, dataset_size);
279+
LOG_DEBUG("Rank %d number of datasets to be written %d\n", rank, dataset_size);
280280
#if ENABLE_MULTIDATASET==1
281281
#ifdef H5_TIMING_ENABLE
282282
increment_H5Dwrite();
@@ -319,7 +319,7 @@ int flush_multidatasets() {
319319

320320
free(multi_datasets_temp);
321321
#else
322-
//printf("rank %d has dataset_size %lld\n", rank, (long long int) dataset_size);
322+
LOG_DEBUG("rank %d has dataset_size %lld\n", rank, (long long int) dataset_size);
323323
#ifdef PDC_PATCH
324324

325325
if (cached_objs.size()) {
@@ -362,7 +362,7 @@ int flush_multidatasets() {
362362
offset_length = new_end[j] - new_start[j];
363363
pdcid_t reg = PDCregion_create(1, &offset, &offset_length);
364364
pdcid_t transfer_request_id = PDCregion_transfer_create(ptr, PDC_WRITE, it->second->did, reg, reg);
365-
//printf("did = %lu, starting request offset = %lu, size = %lu, j = %d\n", it->second->did, offset, offset_length, j);
365+
LOG_DEBUG("did = %lu, starting request offset = %lu, size = %lu, j = %d\n", it->second->did, offset, offset_length, j);
366366
PDCregion_transfer_start(transfer_request_id);
367367
PDCregion_transfer_wait(transfer_request_id);
368368

‎examples/C_plus_plus_example/region_transfer_1D_append.cc

+2-2
Original file line numberDiff line numberDiff line change
@@ -58,14 +58,14 @@ main(int argc, char **argv)
5858
}
5959
sprintf(data_name, "data0_%d", rank);
6060

61-
printf("rank %d C++ example for writing %d lists of arrays with size %d\n", rank, ARRAY_SIZE, DATA_SIZE);
61+
LOG_INFO("rank %d C++ example for writing %d lists of arrays with size %d\n", rank, ARRAY_SIZE, DATA_SIZE);
6262
for ( i = 0; i < ARRAY_SIZE; ++i ) {
6363
register_multidataset_request_append(data_name, 0, data, sizeof(int) * DATA_SIZE, H5T_NATIVE_CHAR);
6464
}
6565

6666
sprintf(data_name, "data1_%d", rank);
6767

68-
printf("rank %d C++ example for writing %d lists of arrays with size %d\n", rank, ARRAY_SIZE, DATA_SIZE);
68+
LOG_INFO("rank %d C++ example for writing %d lists of arrays with size %d\n", rank, ARRAY_SIZE, DATA_SIZE);
6969
for ( i = 0; i < ARRAY_SIZE; ++i ) {
7070
register_multidataset_request_append(data_name, 0, data, sizeof(int) * DATA_SIZE, H5T_NATIVE_CHAR);
7171
}

0 commit comments

Comments
 (0)