-
Notifications
You must be signed in to change notification settings - Fork 0
Home
Linea is Latin for limit, is a stream processing framework leveraged in Tau. It's based on concepts from Apache Storm however it's written from scratch without using any Apache Storm code/dependencies.
Linea also uses an Apache Storm like API to enable easy adoption and familiarity for developers who would like to leverage this framework.
Linea is intended to be light-weight (3MB shaded jar), think hybrid of Storm and Akka/Vert.x
Profiling and tuning is relatively straight forward with Linea due to it's light weight design enabling quicker development to production cycles in comparison.
You can embed this in your own microservice providing it streaming capabilities without the overhead of complex integrations with existing frameworks.
Linea was conceived out of a series of evolutions of Tau / Wraith engines for Event Correlation, since the problem domain poses some scaling and optimization challenges for guaranteed stream processing systems. These requirements demand for stream processing framework that is easy to tune, easy to deploy and doesn't have a lot of moving parts or overhead.
Even though the lineage of Linea's concepts are from existing frameworks, there are a few key differences:
Linea doesn't require a cluster, rather has the ability to auto-cluster as instances are launched, ideally it should be deployed on a docker cluster (Swarm / Kubernetes) or across VMs.
Linea will attempt to use UDP. Why? XOR ledger already provides an application level acking mechanism therefore TCP acking is redundant.
Linea allows Spouts to have implement both a Push and Pull model since each Spout is provided it's own asynchronous executor where user defined code can run.
Linea is fundamentally built with containerization in mind and it's recommended to follow a strict 1 worker per container model where a container can be a Physical Machine or VM or Linux Container.
Linea is a framework, it's not an end-product in itself i.e. you can't run a Linea cluster by itself. It's like dropwizard, spring framework, vert.x etc. therefore there is no multi-tenancy or resource sharing support. The deployment mechanism is responsible for resource multi-tenancy similar to microservices.
Here's the Developer Guide: https://github.com/srotya/linea/wiki/Developer-Guide
You can learn more about the internals here: https://github.com/srotya/linea/wiki/Linea-Internals