-
Notifications
You must be signed in to change notification settings - Fork 31
Flow of Data in the System
- Publishing - Meaning that the value gets sent into RabbitMQ.
- Subscribing - Meaning that we are listening for published values to RabbitMQ.
- Consumer - Someone/Something that is subscribing.
- Publisher - Someone/Something that is publishing.
There are two ways that a data point can enter the system and these are via the API and via the polling system. There are differences on how a data point is handled when it first arrives but later they are handled the same way.
Here we will explain these differences at arrival and then get to the common.
When polling we are polling resources and we get a blob of data in some kind of format, this then need to be parsed so that the requested data can be extracted. The relevant data should then be distributed to the resource's streams which in turn would store the data point in Elastic Search and publish it into RabbitMQ for any clients or virtual streams to subscribe to.
For now we don't have a polling system, but we have a test process that generate data at a specified interval (once per second). This test process generate only one value, i.e one resource having one stream.
The test process publish a data point into an exchange for a resource which have one or several streams processes subscribing to it. Once a stream process receives the data point it stores it in Elastic Search and publish it on its own exchange for clients or virtual streams to subscribe to.
Continue to read more in common.
When posting a data point via the RESTful API we now do it by specifying which stream we are pushing to, this allows us to store the data point directly into ES and publish it directly to RabbitMQ. The data point have to be stored successfully in Elastic Search before we publish it.
Continue to read more in common.
Once the data point have been published for the stream it have also been stored in the data store in Elastic Search. History of data points for a stream could now be retrieved, including the latest value, and also clients and virtual streams could get the data point via their subscriptions.
In RabbitMQ each client and virtual stream subscribing to a stream has its own queue for that stream. These queues are temporary and will be deleted when the consumer disconnect.
If a virtual stream subscribe to the stream it will receive the data point and apply its function to it, store the calculated value in the data store in Elastic Search and publish it on its exchange which clients or other virtual streams might be subscribed to.