README

CC
==

CC is ZMQ proxy for specific type of ZMQ messages (CC messages).

It listens on single ZMQ socket, and processes messages
by matching message type to handler.

It also has optional support for launching daemon processes
and providing config to them.  It is provided only for easier
administration, as any daemon can be written to be standalone.


Installation
------------

CC requires skytools 3.x for basic tooling and M2Crypto
for encryption.  After that actual install is done with:

  $ python setup.py install [--prefix=DIR]

or to build .deb package:

  $ make deb


CC message
----------

First quick intro to ZMQ messages:  ZMQ transports blobs,
with header that specifies length for the blob and whether
it is last blob or not.  So one logical message can contain
one or more blobs.  ZMQ calls it "multipart message".

Simple ZMQ request-reply pattern, implemented by zmq.REP, zmq.XREP
sockets, is to add additional blob at the start of the message
on each hop, which contains socket id from where the message
came from.  So on reply the message can be routed by over several
hops, each one removes the socket id when it sends it further.

Empty part ('') separates such socket ids from actual body parts.

CC message uses such routing, additionally it specifies meaning
to following body parts:

0 - message type (e.g. 'pub.infofile')
1 - body (json)
2 - signature (optional, can be empty)
3 - additional data blob (optional)

The message type is contained also in json, it is separated
out to make routing easier.

In case of signed message with blob, the blob is not signed,
but its hash is added into message before signing.

Message types
-------------

Message type contains dot-separated multiple ids.
So routing pattern can be applied to full id or only
some prefix parts.

Eg. id 'pub.infofile' can be routed either full
pattern or simply 'pub', which will route all 'pub'
messages to some handler.

- pub.infofile, pub.state, pub.stats
- log.info, log.error, ...
- job.*
- req.* ??


CC handlers
-----------

Handler is a Python class that is registered for a message type pattern.

Examples:

cc.handler.proxy: sends message to another CC instance
cc.handler.database: launches db function with message
cc.handler.taskrouter: keeps track of routes to task executors
cc.handler.infowriter: writes infofiles
cc.handler.locallogger: writes logfiles
cc.handler.jobmgr: local jobs [daemons / tasks] query it for config and keepalive


CC daemons
----------

These are daemons that are launched and managed by CC.
They act as ordinary clients, except they are configured
from CC config, instead of separate standalone scripts.

Examples:

- cc.daemon.infosender: reads info files, sends them to CC
- cc.daemon.taskrunner: registers on taskrouter, waits for tasks
- cc.daemon.discovery: discovers things


Patterns
--------

Neither handlers nor daemons need to be tied/managed by CC,
they can always be launched as standalone services.
They are managed with CC only for easier administration.

That also means there are few daemon/handler combinations
that make sense, and others that don't.

To avoid accidental mis-configs, the handlers are checked
against ccserver's cc-role option.

cc-role = local
- listens on localhost
- handlers: jobmgr, proxy, locallogger
- daemons: taskexec, infosender, ...

cc-role = remote
- listens network
- handlers: proxy / dbservice / logwriter / infowriter
- daemons: -


Crypto
------

CC uses the CMS/PKCS7 message format (from SMIME) for signing
and encrypting.

Currently ccserver and daemons have crypto config, handlers
all share the top-level ccserver's one.  Daemon config
is inherited from master ccserver.

cms-keystore:
  directory where certs and private keys are stored.  Private
  keys are under ./private subdir.  So common server keystore
  would look like:

  ./server.crt
  ./ca.crt
  ./confdb.crt
  ./private/server.key

cms-sign:
  Key name to sign as.  Requires files ./$key.crt and ./private/$key.key
  under keystore.  If set, all outgoing messages are signed.

cms-verify-ca:
  Cert name to use to signature verification.  Requires
  ./$name.crt under keystore.  If set, incoming messages
  must be signed under key certified by CA.

cms-encrypt:
  Cert name to encrypt to.  Requires ./$name.crt under keystore.
  Note - this should not be CA cert but some service one, like 'infoserver'.
  Must be paired with cms-sign.

cms-decrypt:
  Key and cert name to decrypt as.  Requires $name.crt and ./private/$name.key.
  Note - service key.
  Must be paired with cms-verify-ca.