You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Depending on the amount of data / processing required, starting up a server can be a long process.
Especially for testing our server infrastructure, we would like to avoid unnecessary overheads.
To do so (and as already outlined in #3), we can think about implementing a data provider to supply data to our engine without a need to recreate the datasets after a failure / for any new startup of the engine.
As outlined in #3, we should seriously consider a process that does not used shared memory but rather communication via zeromq. The reasoning here is that this process would allow for us to avoid any locking problems that come along with shared memory regions and to allow distribution of workloads onto different machine types.
A data provider should take the role of the what is currently handled in the master-service::dataset. Instead of having a dataset and allowing the creation of different data structures, the master service should hand this responsibility to a data provider that can be located anywhere. The master service handles the communication with the data provider and returns structures as we are used to.
To do so, we need to serialise all structures into PBF and deserialised them from PBF. The Provider should offer the functionality to load a raw GTFS feed from disk and put it into PBF. The PBF has then to be transferred via ZeroMQ to the MasterService which, in turn, returns the access as usual, hiding all the ZeroMQ shenanigans from the rest of the project.
add PBF serialisation / deserialisation to all timetable data structures / look-up data structures
create data provider to be started up before any server can be started (think shared memory loading in OSRM), loading data and creating pbf-forms of all structures
start off with a create-all approach?
or still create on demand (might need to add additional pbf -> datastructure steps)
add communication possibilities to data provider
replace master service data-loading and creation with request + pbf->datastructure conversion
Depending on the amount of data / processing required, starting up a server can be a long process.
Especially for testing our server infrastructure, we would like to avoid unnecessary overheads.
To do so (and as already outlined in #3), we can think about implementing a data provider to supply data to our engine without a need to recreate the datasets after a failure / for any new startup of the engine.
As outlined in #3, we should seriously consider a process that does not used shared memory but rather communication via zeromq. The reasoning here is that this process would allow for us to avoid any locking problems that come along with shared memory regions and to allow distribution of workloads onto different machine types.
A data provider should take the role of the what is currently handled in the
master-service::dataset
. Instead of having a dataset and allowing the creation of different data structures, the master service should hand this responsibility to adata provider
that can be located anywhere. The master service handles the communication with thedata provider
and returns structures as we are used to.To do so, we need to serialise all structures into
PBF
and deserialised them fromPBF
. The Provider should offer the functionality to load a raw GTFS feed from disk and put it intoPBF
. ThePBF
has then to be transferred via ZeroMQ to theMasterService
which, in turn, returns the access as usual, hiding all the ZeroMQ shenanigans from the rest of the project./cc @daniel-j-h
The text was updated successfully, but these errors were encountered: