Skip to content

2024-11

Latest
Latest
Compare
Choose a tag to compare
@Pagghiu Pagghiu released this 01 Dec 21:48
· 24 commits to main since this release
release/2024/11
a6bdc11

This update brings a new functionality: Async Streams

November 2024 Update Blog Post

Async Streams

The SC::Async library has been extended with the addition of Async Streams!

Async Streams are largely inspired by node.js Streams, a very powerful tool to process large amounts of data in parallel.

The basic idea about an async stream is to create a Source / Sink abstraction (also called Readable and Writable) and process small buffers of data at time.

The state machine that coordinates this interaction handles data buffering and more importantly handles also back-pressure, that means:

  • Pausing the readable stream when a connected writable stream cannot process data fast enough
  • Resuming the readable stream when a connected writable stream is finally able to receive more data
By implementing streams on top of async operations it's possible to run many of them concurrently very efficiently.
When properly implemented for example an async pipeline can concurrently read from disk, write to a socket while compressing data.
For now only Readable / Writable File and Socket streams have been implemented, but Async Transform Streams (for compression) will be next!

Most notable differences with node.js streams are for now:

  • No allocation (designed to work inside user-provided list of buffers)
  • No object mode
  • Fixed Layout to create data pipelines (AsyncPipeline)
  • onData support only (no readable event)

Async Streams are for now in 🟥 Draft state.
It's also possible that its API will evolve a little bit to be less verbose and there is also lack of nice examples, aside from the tests.

It's better waiting for it to become stable before doing anything significant with it.

Some changes and fixes in the SC::Async library
have been made necessary to support Async Streams.

339f291 Async: Add AsyncReadableStream
11a2b92 Async: Add AsyncWritableStream
9d9d550 Async: Draft AsyncPipeline and AsyncRequest{Readable | Writable} Stream
fc97464 Async: Extend AsyncRequest{Readable | Writable}Stream to Socket Send/Receive
46174b0 Async: Make AsyncPipeline and AsyncRequest{Readable | Writable}Stream public
15ec185 Async: Make AsyncSocket{Send | Receive} buffer and handle public
a6bdc11 Async: Rename AsyncPipeline::Sink to AsyncPipeline::Pipe
3c78017 Async: Reorganize AsyncTest for clarity
c251205 Async: Use offsets only if explicitly set in AsyncFile{Read | Write}

Others

And just like every update, a bunch of fixes and improvements to all libraries have been committed.
Nothing specific stands out this update, but for completeness this is the list of related commits:

9347ca6 Build: Bypass VMWare hgfs issue setting wrong modified time for new files
a075702 Build: Support linking system libraries in XCode
a572d89 Containers: Fix compile errors under latest MSVC
ee4947b Documentation: Enable warnings as errors
596f052 Documentation: Update README.md with latest videos
b077888 Foundation: Add equality operator to Function
0ad8187 Foundation: Add HeapBuffer
c750871 Foundation: Add Span::equals
7f08864 Foundation: Add Span::get(int)
14f37f5 Foundation: Add StrongID
e80edc2 Hashing: Improve documentation
23e7b37 Meta: Ignore sync folder and icon files
fa9a48c Tools: Rebuild bootstrap when make fails

Full Changelog: release/2024/08...release/2024/11

Assets 2
Loading