all components either memory/disk
- copied
- duplicated
- mutex, rwmutex protected
- not in resource raced status ... anyway must be MTS MTS: multiple threads safe
- log
- macro log type: info, warn, error, debug, fatal
- SAFE: fmt log msg length is restricted
- BUG: msg truncated and overflow
- (buffer BUG in windows msys2 BUG, not mine)
- MacOS-Arm64 and Ubuntu-Amd64 is OK
- has been solved
- TODO: enable color
- hasDone: ILOG, MTS, Buffer Not Safe
- coworker
- concurrency in multithreads
- sync, async, time record
- mutex, cond signal, progress record
- param_list, return values set
- TODO: error cancel, timeout control
- wal
- normal log in app
- fd,write fd,fsync
- write ahead logging
- wal in coworker
- yyjson parse json to accelerate json parser
- yyjson read jsonstring not need to be escaping, only json format is ok
- struct of binding, serialize, unserialize
- easy search key: full_path, short_key, value
- json flatten: full_path, short_key, array_value
- TODO: how to switch tag to get type and value
- TODO: find a c lib of map any_val -> any_val
- all thirdparty library can be found on Github search by package name LIKE: yyjson
- TODO: micorsoft/mimalloc
- report BUG of opensource lib: uthash, c_hashmap
- anyway: the API design is:
- Go Server Loading a plugin of gnuc99 target .o file called plugin_json_flatten.
- go program send a original json binary bytestream to plugin_json_flatten.
- gnuc99 parse and flatten json to new json format, write a new json to a new bytestream.
- go receive a flatten json binary bytestream to parse it use predefined format in go.
- now go only to recycle garbage of original bytestream and new bytestream and new struct of.
- internal:
- use sc_map to store string to array of value by map[char*]void* -> void* reinterpret to array of real value.
use bidirectional linked list to store the order of key.- map[char*]void* to map full_path -> void* reinterpret to {uint64_t tag; void * arr;}
- write back to json_string use of mut_json
- receive from Go bytestream to char *
- send char * to Go bytestream
- free gnuc99 alloc to use mimalloc
- memory leak saved method:
- record for all alloc but maynot be freed memory
- use freelist or mimalloc in application level mimalloc and mifree for reused and preheet
- memory leak tools like: valgrind
- try to manage memory leak
- loop for collect and signal in global obj like yyjson_doc
- change the memory management thought, convert small alloc/free to middle/PAGE_SIZE block to manage
- performance-test:
- 100MB JSON: https://github.com/seductiveapps/largeJSON/blob/master/100mb.json
- 15GB JSON: curl -L http://openlibrary.org/data/ol_cdump_latest.txt.gz | gunzip | cut -f 5 > ol_cdump.json
- load of .so module plugin in golang
- .so dynamic linked failed both in GNUC99 and GoRuntime1.22
- success both in GNUC99 and GoRuntime1.22 ON ARM64-Darwin
- use microsoft/mimalloc, OK.
- find a high performance xml.Decoder and xml.Encoder
- CGO cgo -
- Linux kernel -
- CMake Cmakelists.txt-
- dynamic linked how it memory layout
- make a json_flatten allocator learned by yyjson_doc