Skip to content

Releases: Pagghiu/SaneCppLibraries

2024-11

01 Dec 21:48
Compare
Choose a tag to compare

This update brings a new functionality: Async Streams

November 2024 Update Blog Post

Async Streams

The SC::Async library has been extended with the addition of Async Streams!

Async Streams are largely inspired by node.js Streams, a very powerful tool to process large amounts of data in parallel.

The basic idea about an async stream is to create a Source / Sink abstraction (also called Readable and Writable) and process small buffers of data at time.

The state machine that coordinates this interaction handles data buffering and more importantly handles also back-pressure, that means:

  • Pausing the readable stream when a connected writable stream cannot process data fast enough
  • Resuming the readable stream when a connected writable stream is finally able to receive more data
By implementing streams on top of async operations it's possible to run many of them concurrently very efficiently.
When properly implemented for example an async pipeline can concurrently read from disk, write to a socket while compressing data.
For now only Readable / Writable File and Socket streams have been implemented, but Async Transform Streams (for compression) will be next!

Most notable differences with node.js streams are for now:

  • No allocation (designed to work inside user-provided list of buffers)
  • No object mode
  • Fixed Layout to create data pipelines (AsyncPipeline)
  • onData support only (no readable event)

Async Streams are for now in 🟥 Draft state.
It's also possible that its API will evolve a little bit to be less verbose and there is also lack of nice examples, aside from the tests.

It's better waiting for it to become stable before doing anything significant with it.

Some changes and fixes in the SC::Async library
have been made necessary to support Async Streams.

339f291 Async: Add AsyncReadableStream
11a2b92 Async: Add AsyncWritableStream
9d9d550 Async: Draft AsyncPipeline and AsyncRequest{Readable | Writable} Stream
fc97464 Async: Extend AsyncRequest{Readable | Writable}Stream to Socket Send/Receive
46174b0 Async: Make AsyncPipeline and AsyncRequest{Readable | Writable}Stream public
15ec185 Async: Make AsyncSocket{Send | Receive} buffer and handle public
a6bdc11 Async: Rename AsyncPipeline::Sink to AsyncPipeline::Pipe
3c78017 Async: Reorganize AsyncTest for clarity
c251205 Async: Use offsets only if explicitly set in AsyncFile{Read | Write}

Others

And just like every update, a bunch of fixes and improvements to all libraries have been committed.
Nothing specific stands out this update, but for completeness this is the list of related commits:

9347ca6 Build: Bypass VMWare hgfs issue setting wrong modified time for new files
a075702 Build: Support linking system libraries in XCode
a572d89 Containers: Fix compile errors under latest MSVC
ee4947b Documentation: Enable warnings as errors
596f052 Documentation: Update README.md with latest videos
b077888 Foundation: Add equality operator to Function
0ad8187 Foundation: Add HeapBuffer
c750871 Foundation: Add Span::equals
7f08864 Foundation: Add Span::get(int)
14f37f5 Foundation: Add StrongID
e80edc2 Hashing: Improve documentation
23e7b37 Meta: Ignore sync folder and icon files
fa9a48c Tools: Rebuild bootstrap when make fails

Full Changelog: release/2024/08...release/2024/11

2024-08

30 Aug 19:40
Compare
Choose a tag to compare

August has been spent doing some first round of improvements to the SC::Http library.

August 2024 Update Blog Post

Http

August has been spent doing some first round of improvements to the HTTP library.
The HTTP library is still very incomplete and fragile, but it got to the level of being able serving a simple local html web site, through the new SC::HttpWebServer class.
The SC::HttpServer / SC::HttpWebServer classes are not solid enough to be used in real internet facing application but they're in better shape than they were the previous month.
SC::HttpClient is just a very incomplete stub and it currently exists to help testing the server classes.

9f4700f Http: Add HttpWebServer to serve files from a directory
4d3a5b4 Http: Hide HttpServer implementation details
6f03e3b Http: Improve HttpServer properly releasing unused sockets

SCExample

SCExample has now a second example called WebServer that can host a website on a specific port reading files from a local folder, all integrated with a GUI interface.
The example interface now has two new methods (initAsync / closeAsync) to let samples access the main SC::AsyncEventLoop of the application.
The async io event loop is integrated with gui event loop on main thread to make sure that all examples using SC::Async can see their completions dispatched on the main (gui) thread. This means that they can modify the same state variables used by the GUI without needing mutexes.

SCExample_05.mp4

15eb5ab SCExample: Add helper to use InputText with SC::String
0893d45 SCExample: Add initAsync and closeAsync example interface methods
3814380 SCExample: Add WebServer example

Async

Implementing the SC::HttpWebServer has been putting a little bit of stress on the SC::Async library, with a few new features, and some bug fixes and QOL improvements.

Features

SC::AsyncSocketReceive and SC::AsyncFileRead report End Of File and disconnected events on all platforms, signaling when they are running out of data.

Bugs (fixed)

  • SC::AsyncSocketWrite writes were not properly handled on Posix with large buffers (and now there's a test for that)
  • Timer callbacks were called multiple times on the io_uring backend

Quality of Life improvements

It's possible now to de-allocate any SC::AsyncRequest derived request inside its own callback or just re-configure them and start them again.
That should make it a lot easier handling more complex async data flows.

1d262b7 Async: Allow AsyncRequests to be deallocated inside their own user callback
3735432 Async: Do not invoke timers callbacks multiple times on io_uring backend
1cf9c2b Async: Handle partial writes in Posix backend
34cbfd6 Async: Mark AsyncRequest as free after Cancelling it
9b1b2d0 Async: Report disconnected on AsyncSocketReceive
58aded2 Async: Report end of file on AsyncFileRead
a7c64cc Async: Simplify reporting AsyncSocketReceive disconnected on posix

Others

And just like every month, a bunch of fixes and improvements to all libraries have been committed.
The most notable bug was in the SC::Process library, that was preventing proper stdout inheritance for console applications on Windows.
Github CI was missing all output of the cl.exe and link.exe msvc executables spawned by SC::Build during compilation.
Specifically the Win10/MSVC2019 ci job was failing without printing any error, making it really difficult to figure out how to fix it!

97a687f Console: Attach to parent console on Windows
809e2b9 Containers: Add methods to obtain arena capacity
fb6877c Containers: Split ArenaMapKey in a separate header
0c0a6b6 FileSystem: Report file size in new FileStat structure replacing FileTime
12b85ed Process: Hide child process console on Windows only if parent process has no console
924a797 SerializationBinary: Prefer using loadExact when loaded schema matches source schema
1783a3a SerializationText: Support bool data types for JSON serializer
177f005 Socket: Replace SocketClient::close calls with SocketDescriptor::close
2cd0850 Time: Expose month and day as strings in Time::Absolute::ParseResult

2024-07

31 Jul 19:57
Compare
Choose a tag to compare

The focus of the month has been building a nicer showcase for Plugin and Serialization libraries.

July 2024 Update Blog Post
July 2024 Update Video

SCExample

The hot-reload system of SCExample has been tuned so that each internal example is just an hot-reloaded Plugin.
It's now very convenient to iteratively build them and experiment!
A nice serialization example has been added to SCExample, based on a very well known piece of code from the official imgui demo.
State of the canvas control is saved to binary and json using the automatic serialization provided by the reflection system.
It's also showing a somewhat advanced usage of the Reflection system, that is how to wrap a "custom" vector implementation (ImVector)

SCExample_04.mp4

There is a video where I've been recording the most important bits of this development:
C++ Serialization and Reflection (with Hot-Reload) - Sane C++ Libraries [ep.25]

And as a last nicety, SCExample has been fully ported to iOS!
Hot-reload obviously requires root access to install clang compiler and a sysroot on iOS, so it will work only if you have an active jailbreak.

Also here I've been recording a video on finalizing porting SCExample to iOS.

Hot-Reload C++ on iOS - Sane C++ Libraries [ep.24]

85e6352 SCExample: Port to iOS
c839d17 SCExample: Show compile errors when hovering examples
641c927 SCExample: Refactoring hot-reload plugins as examples
b14ef5e SCExample: Add Serialization example

Serialization

While building the Serialization example, a few fixes have been made necessary.
The first has been supporting the bool data type that for some obscure reason was missing from the primitive types.
Some helpers to read and write the schema together with serialized data have been added to make it easy supporting versioned binary serialization.
Fields can be added or removed, and the serializer will still try to load data that can be converted assuming a matching order.
There is also some support for changing field types. For example a float field that is converted to int (or vice-versa) will receive a truncated value on deserialization.

fe203bb Reflection: Add vector manipulation methods to ExtendedTypeInfo
0419d74 Reflection: Support bool primitive type
f83bb78 SerializationBinary: Add writeWithSchema and loadVersionedWithSchema
03ef3a1 SerializationBinary: Reduce boilerplate needed to wrap a custom vector container
18d871c SerializationBinary: Support bool primitive type
8981495 SerializationText: Reduce boilerplate needed to wrap a custom vector container

Plugin

Plugin library has received a few improvements, like the ability to specify a path for the sysroot under clang (needed on iOS), and it's capturing compiler and linker outputs.
This last one allows showing compile errors in the plugin host app, that is a nice addition to SCExample.
Finally some dynamic export clauses have been added to many libraries in order for Plugins to find symbols in the host application that is loading them.
This will for sure increase the executable size as the linker will not be able to trim such classes as "unused" code, even if they're not referenced by any Plugin.
Probably some macro to disable this "export by default" behavior will be added for anyone that wants to use Sane C++ Libraries without the Plugin Library or without needing to use Sane C++ Libraries type across dynamic library boundaries.

e6d21a8 Plugin: Allow overriding isysroot path
61b8928 Plugin: Capture compiler and linker output
1ffb874 Everywhere: Add export clause for use across Plugin boundaries

Build

Build system has received some improvements too, supporting generation of iOS project and application icon for Xcode app bundle.
There are still many missing features and too many hardcoded defaults but I feel that the library deserves more than a Draft status.
For this reason I've been promoting it to MVP status: maybe this will become an incentive to add more features to make it usable!

d865cc4 Build: Add iOS storyboard generation
ca0fb7b Build: Elevate to MVP status
6551c1e Build: Support application icon in Xcode app bundles
70d32b7 Build: Support iOS

Contributions

This month Jeremy Laumon has been adding natvis support for Arrays!
415dc32 DebugVisualizers: Add natvis for SegmentedItems and Array

Others

And then the usual amount of monthly fixes!
I can never stress how important is to use the libraries even just in SCExample, to trigger issues and identify missing bits.

050bad6 Strings: Export String and some SmallString for use across Plugin boundaries
1d48530 Containers: Fix some incorrect Array member functions
20f2dd9 FileSystemWatcher: Add internal header with FSEvents declarations for iOS
54fb28b Documentation: Improve README and fix descriptions of some libraries
5a72dcf FileSystem: Add read and write overloads for uint8_t
915367a Strings: Fix doubles parsing
a176a3e Everywhere: Add lifetime bound attribute for StringViews and Spans
e3121af Foundation: Add Span::reinterpret_as_array_of()
e5475af Tools: Use Github to download doxygen releases

What's Changed

  • DebugVisualizers: Added natvis for SegmentedItems and Array by @jlaumon in #20

New Contributors

Full Changelog: release/2024/06...release/2024/07

2024-06

29 Jun 21:01
Compare
Choose a tag to compare

June 2024 Update Blog

The month has been spent mainly building SCExample, a small GUI example app to showcase some Libraries (mainly SC::Async / SC::Plugin / SC::FileSystemWatcher).

SCExample

SCExample is a small GUI application based on sokol_app / sokol_gfx libraries, providing window abstraction and graphics api abstraction and dear-imgui for the UI.

Building this application has been mostly documented in the following YouTube videos:

First step has been putting together a pure cross platform GUI application (win/linux/macos), making sure to pause the render loop in absence of user inputs:
Pause Immediate Mode UI - Save CPU Time

Second step has been integrating the GUI event loop with the SC::Async I/O library (always cross-platform)
Add Async IO to Immediate Mode GUI

And lastly showcasing the integration of SC::FileSystemWatcher and SC::Plugin to watch source files that are being hot-reloaded as plugins (dynamic libraries).
Also in this case the cross-platform aspect has been preserved.

Hot-Reload dear imgui

This is the list of SCExample related commits:
6f29135 SCExample: Create (sokol+dear imgui) app re-drawing on user input only
4c72b42 SCExample: Display number of reloads and last load time of hot-reloaded plugins
96e6df4 SCExample: Implement simple hot-reload to showcase Plugin and FileSystemWatcher libraries
5be2d02 SCExample: Integrate SC::AsyncEventLoop by using SC::AsyncEventLoopMonitor
947f8a7 SCExample: Transform floating window into a toolbar

SC::Async

A small but useful addition to the SC::Async library has been the SC::AsyncEventLoopMonitor class.
This class wraps most of the required machinery to integrate SC::Async event loop in an application that has a different main event loop (GUI or IO)

784ffcf Async: Add AsyncEventLoopMonitor to poll the event loop from a background thread

SC::Plugin

SC::Plugin library has been receiving some love, with the addition of a queryInterface-like mechanism helping to define contracts between plugins and the host application.

It's possible to control linking libc / libc++, adding some custom include directories and disabling exceptions.
SC::Plugin now monitors CFLAGS and LDFLAGS environment variables (if defined) of the host environment, to allow linking the correct sysroot on posix systems.
Some fixes too have been needed on macOS where duplicated plugin symbols from different compiled .dylibs can wrongly increment their refcount, preventing from unloading them.

d6816b1 Plugin: Add PluginCompilerEnvironment to intercept and use CFLAGS and LDFLAGS
94ad0d9 Plugin: Add PluginDynamicLibrary::queryInterface
efc4863 Plugin: Add PluginSysroot and build options to allow linking libc and libc++
64ee726 Plugin: Allow multiple include paths
9466040 Plugin: Change appendDefinitions into replaceDefinitions
72354c8 Plugin: Disable exceptions under MSVC
8ad2aca Plugin: Hide Plugin symbols by default on macOS
f7dfa22 Plugin: Track number of reloads and last load time for all plugins

SC::FileSystemWatcher

Using SC::FileSystemWatcher inside SC::Example has exposed some usability issues and bugs (mainly repeated notifications), that have been promptly fixed.

05c0edd FileSystemWatcher: Filter repeated notifications on macOS
ffada00 FileSystemWatcher: Improve API usability

SC::Build

SC::Build can now generate Xcode projects for gui applications creating app bundles on macOS for the Xcode backend.
The configure phase will download required dependencies (like sokol or dear-imgui for SCExample).
This is not ideal as it requires internet connection, so it will be probably be made optional in some future update.

It can now build from paths with spaces (even when using the Makefile backend...) and generates Xcode projects avoiding any warning.
Some of the fixes have ramifications also on the SC::Tools that bootstraps SC::Build.

2be7d39 Build: Collapse all non-apple TARGET_OS to just linux
931fe7c Build: Escape quotes in Make and Xcode backends
df3eae8 Build: Fix generation of compile_commands.json file
e6acc3d Build: Fix Makefile force clean on platform specific makefiles
b225291 Build: Generate makefiles supporting building from paths with spaces
f711566 Build: Improve Xcode generator not to produce warnings under Xcode 15
0214cdb Build: Support creating app bundles on Xcode

b2e3cef Tools: Improve Makefile
0b802ba Tools: Support building a tool from path containing spaces

Others

And just like every month, working on SCExample has generated a number of minor fixes, API improvements / refactoring.
All of them have been proudly committed to the main branch!

30f07bd Containers: Fix bug in VectorMap::remove

03b81f6 FileSystem: Add writeStringAppend

95d90ab Foundation: Add Host Operating System detection

4d9d9ae Process: Avoid creating windows when spawning a new process
a7859c5 Process: Improve ProcessTest compatibility
03cd51d Process: Rename StringsTable to StringsArena

1f33202 Reflection: Disable SC_REFLECT_AUTOMATIC on older clang versions
a941264 Reflection: Fix compile error due to a GCC bug

e230df7 Strings: Add StringHashFNV
1c0fd89 Strings: Do not let StringBuilder leave unterminated string when format fails

fd24695 Time: Make HighResolutionCounter::snap return reference to HighResolutionCounter itself

1b50cb3 Documentation: Add SCExample to the Examples page
32181ff Documentation: Update README with latest video
07591f1 Documentation: Update README with May 2024 blog post
476fbf8 Documentation: Update SCExample documentation

Full Changelog: release/2024/05...release/2024/06

2024-05

31 May 20:33
Compare
Choose a tag to compare

This has been a month of steady improvements, with no big feature added.

Make sure to check out the update blog post:
May 2024 Update Blog

Async Library has been receiving some fixes and a useful refactoring.

  • AsyncEventLoop::runOnce has been split into three methods, separating request submission, polling for new events and completion callbacks dispatch.
    This allows submitting request from a "main" thread, that already has an event loop, blocking to poll for changes on a separate thread, and dispatching the callbacks for events received again on the same thread used for submission.
    The main use of this would be integrating an AsyncEventLoop with another event loop from another library that already "owns" the application main thread, including a GUI event loop.

  • Loop time that is now being updated more consistently both when using runOnce() and runNoWait()

  • Kernel synchronization was failing under Windows 10.
    The bug has been fixed and now Windows 10 is now being tested in the CI to avoid future regressions.

  • I've also been adding a new "phase" to handle re-activation that allows some requests to skip going through the submission queue, by just keeping it active.
    Re-activation happens when inside a callback for some Async Request user calls reactivateRequest(true).
    New code handles re-activation more correctly as on some backends (kqueue, epoll), where the request is always active once it has been started.
    As submissions are processed on the "next" event loop run, it can happen that such requests are "triggered" by the kernel while they were still in submission queue.
    This event breaks many assumptions made by the (kinda complex) state machine coordinated by AsyncEventLoop.

0526b5c Async: Split runOnce into submitRequests, blockingPoll and dispatchCompletions
069b4bd Async: More precise handling of GetQueuedCompletionStatusEx on Windows 10
138d1d8 Async: Allow re-activation of AsyncLoopTimeout
3d48f77 Async: Add ReactivateAsyncPhase to allow skipping submission on reactivation
af1e0f0 Async: Hide KernelQueue and KernelEvents inside Internal
b7623c5 Async: Handle different GetQueuedCompletionStatusEx behavior on Windows 10
e6a5704 Async: Update time consistently both in NoWait and ForcedForwardProgress modes

Build Library has also been receiving some love, increasing its capability while simplifying SC-build.cpp scripts:

  • Support generating VS2019 projects (needed by the Windows10 Github runner)
  • Support Objective-C or Objective-C++ files
  • Support generating multiple projects
  • Support libraries to link (Makefile backend)
  • Support creating non-console applications (Windows)
  • Consistently handle all paths (include, libraries etc) as relative to project root

2860888 Build: Generate one solution per project on Visual Studio backend
2d63b73 Build: Specify target name when invoking Makefile
35f5fc5 Build: Avoid Makefile warnings on intermediates and outputs directory creation
5879951 Build: Support Objective-C and Objective-C++ files
9f195b3 Build: Generate platform specific Makefile
a123422 Build: Express absolute include paths as relative to project dir
ad9f0ee Build: Add flag to generate Visual Studio projects using Windows subsystem
b90f05c Build: Allow selecting target for compile, build or run action
c9dd352 Build: Simplify SC-Build with more defaults
cab21d6 Build: Support link libraries on Makefile backend
df488cd Build: Simplify usage of relative paths in includes and defines
f0b3043 Build: Add basic multi-projects support

The CI has been simplified, consisting now of three github workflow file

  • windows (windows 10 + windows 11)
  • posix (linux + macOS)
  • documentation and coverage (macOS)

The GitHub CI matrix feature has been used to test multiple platforms and configurations.
On all platforms both Debug and Release configurations are now being tested.
Additionally also Windows 10 has been added to the list of platform tested by the CI.

04123ab CI: Skip documentation deployment step outside of forks
097d01d CI: Merge Linux and macOS into a posix runner
9a4d1f2 CI: Run GitHub workflows only on main branch
b59a345 CI: Fix windows pipeline
cdd5e3a CI: Run the tests on Windows 10 and Windows 11 both in Debug and Release
f8067a8 CI: Set GitHub workflow badges to the correct URL
fd1dc37 CI: Skip documentation deployment step on forks

Socket library has been cleaned up, after some "encouragement" given in a issue that was trying to expand UDP support.

645dae9 Socket: Move examples from header to snippets
9755163 Socket: Move SocketNetworking::resolveDNS into its own SocketDNS class
befe282 Socket: Require a valid socket for both SocketClient::connect overloads
cf3034e Socket: Split SocketServer::bind out of SocketServer::listen to allow creating UDP servers

First contributor of the project has appeared!
Thanks to the contributor now both Tools and Plugin will work even if visual studio is installed in a directory different from default.

89a3518 Plugin: Detect Visual Studio path dynamically
978cc4c Tools: Update 7-zip to 24.05

Some issues have been reported by users mixing Sane C++ Libraries with C++ Standard Library and they've been fixed as well.
98e7310 Foundation: Just include <memory.h> if SC_COMPILER_ENABLE_STD_CPP is defined or exceptions are used

And then just a usual bunch of fixes and minor additions.

036c5c4 Documentation: Update README with latest video
0f1cd95 Documentation: Improve commit message format and squashing sections
940b780 Everywhere: Support Visual Studio 2019
f37c814 Everywhere: Fix ClangCL build
1d0271c FileSystem: Fix Path::relativeFromTo
fde2d97 Foundation: Allow passing arbitrary number of arguments to placement new
f86ce3d Strings: Add String::owns
75819d5 Time: Add operator < and > to Milliseconds and HighResolutionCounter::getRelative
40341b4 Tools: Escape path for the .touched file in the NMAKE Windows bootstrap
d30163d Tools: Use just commit hash for SC-Package when cloning git repos
d47e439 Tools: Fix HEAD casing for "git rev-parse" used by SC-Package
d4ce9c3 Tools: Automatically retry building a tool on make failure

What's Changed

  • Tools & Plugin: Dynamic Visual Studio path detection by @silent-tech in #18

New Contributors

Full Changelog: release/2024/04...release/2024/05

2024-04

30 Apr 17:30
Compare
Choose a tag to compare

Main additions of April 2024 Release are:

Make sure to check out the update blog post or the video about SC::AsyncLoopWork to learn more:

Release highlights:

Code Coverage and Documentation
Ability to build code coverage reports and documentation, and deploy both to gitlab pages

SC::Tools
Mainly support to the code coverage reports and documentations for the CI

SC::Process

SC::Build

SC::Async

✅ Miscellaneous

Full Changelog: release/2024/03...release/2024/04

2024-03

31 Mar 22:53
Compare
Choose a tag to compare

2024-02

01 Mar 21:39
Compare
Choose a tag to compare

SC::Async got an io_uring backend!
SC::Async epoll backend got simplified
SC::Async now tracks all active requests to mark them as free on a sudden close requests.
SC::Async got all private implementation details hidden under a compiler firewall.
SC::Async can now use SC::ThreadPool to run some async ops.
SC::Async AsyncFileRead / AsyncFileWrite are now fully async also on buffered files.
SC::ThreadPool has been added, to allow running tasks in background
✅ Got CI building and running tests on Windows, Linux and macOS (using Github Actions).
✅ The CI also enforces proper formatting through clang-format.
✅ Fixed a few UB and leaks signaled by gcc UBSAN and LSAN on Linux.
✅ Improved documentation for Building as an User vs Building as a contributor

February 2024 Update Blog Post:

On the YouTube Channel you can find some videos showing some bits of the development process:

Full Changelog: https://github.com/Pagghiu/SaneCppLibraries/commits/release/2024/02

2024-01

24 Feb 14:44
Compare
Choose a tag to compare

SC::Async got an epoll Linux backend.
SC::FileSystemWatcher got an inotify Linux backend.
SC::Process has been made more solid on Posix in general (including Linux).
SC::Build got a Makefile generator for Linux (and macOS).
SC::StringConverter got UTF8 to / from UTF16 code path necessary on Linux.
SC::Plugin got necessary fixes to run on Linux.

January Update Blog Post:

On the YouTube Channel you can find some videos showing some bits of the development process:

Full Changelog: https://github.com/Pagghiu/SaneCppLibraries/commits/release/2024/01