Experiment with microservices using go. Truly understand how to balance scalability and understandability.
- The core of "micro" is to keep services small so you do not have to understand the full big monolith before you can be effective.
- Another advantage of this approach that natural boundaries between services make it easy to keep the system modular.
- Finally, the application that can be chopped into services is easy to scale over multiple machines.
- However, in the end you still have to understand the "whole". Clear overview of dependencies between services could help here.
- Wiring together all the loose components is hard; service registration could help
- Monitoring all the loose components is harder
- Also eventual consistancy introduces new headaches; Cheating in the UI could help
- Guaranteed and tested interworking of all combinations of all versions of all services is very hard: backwards compatible contracts or upgrading as a whole (modular monolith) could help.
Specification effort:
- Define your screens
- Based on screens, determine your commands and queries
- Cohesion and minimizing ripple effect upon change is the primary reason to devide.
- Group related commands and queries in a service if this promotes cohesion. Typically commands and queries related to an specific "aggregate" are grouped together.
- Define your events, that exchange information between services.
- Specify which events a command emits
- Specify on which events a service depends
Develop non-functional components:
- append-only event-store
- publish-subscribe bus
- Scenario based testing: "given-when-then"
Develop functional services:
- create unit tests
- implement business-logic: handle commands and events
- attach service to http and event-bus
Build screens from the provided services:
- create end-to-end tests
- develop screens using services
A tour-de-france application:
- Tour:
- As event-organiser: Create a tour for a particular year (1)
- As event-organiser: Add etappes to a tour (2)
- As event-organiser: Add cyclists to a tour (3)
- As event-organiser: Publish dayly result of etappes and calculate scores for cyclists and gamblers (4)
- As event-organiser: Mark cyclist as "abandonnee" (5)
- As anybody: View tours with their cyclists, etappes and results (6)
- Gambling:
- As gambler: Create a profile (21)
- As gambler: Compose a team of cyclists for a particular year (22)
- As gambler: View As gambler: scores of my cylists (23)
- As anybody: View All gamblers with their scores and teams (24)
- News:
- As journalist: Create a news item (31)
- As anybody: View timeline of tour events and news events (32)
- As anybody: subscribe and receive news via email (33)
- Maintenance:
- As system administrator: Keep track of everything that has happens within the system (50)
Responsible for managing tours with their etappes and cyclists (1,2,3,4,5,6)
Responsible for gamblers and their teams of cyclists and scoring (21,22,23,24)
Responsible for accepting news items and providing a timeline of tour and news events (31,32)
Responsible for sending out tour and news items via email to subscribers (33)
Non functional service that keeps track of everything that has happened. (5)
Responsible for serving the web-UI and hiding all services behind a single http-endpoint. Could be the place to apply your non-functionals (ssl-offloading, security, logging, statistics, etc).
An "application" consists of the following concepts:
- "service": An application consists of one or more loosely couples "services". each service is responsible for keeping is own data.
- "commands" and "queries"(with "attributes"): Each service supports zero or more commands and queries. Browsers interact with the system via these commands and queries on a services.
- "event" (with "attributes"): Each command emits zero or more events. An event is used to exchange of information between services in an async away.
##Technical solution
- Use a "dsl" to describe your application in terns of "services", "commands", "queries" and "events". Example: application.go. Based on this system-description some parts of the system are generated. Example: events.go and interface.go. This approach is chosen to achieve consistent approach and ease error phrone tasks.
- Use event sourcing. Store events and rebuild current state by replaying events ralated to an aggregate in the order of arrival. Currently the following aggregate are recognized: Tour and Gambler.
- Could easily use command-Query seperation if scale requires it.
- Provided implementation for "bus" (=to exchange of events between services). Current implementatio is based on NSQ.
- Provided implementation of append-ony "store" for persistence. Current implementation uses the filesystem.
- Provided implementation of http handler to process commands. Current solution is based on gin-gonic.
- Provide a mechanism for starting and configuring services. Current solution compiles into a single executable. This executable can be configured (using command-line flags) to acts a service a, b or c. This to ease the distribution and deployment.
- Declarative way of testing services. Example: logic_test.go. Based on this test spec, documentation and relationships between services can be derived. In addition to this, each test scenario (= "given", "when" and "expect") is recorded and written to file. This file describes the scenario exactly in json. Example: scenario_example.json
- Automatically derive clear, exact and up to date documentation that explains how services are related. Example: graphviz.dot and
go get github.com/MarcGrol/microgen
cd ${GOPATH}/src/github.com/MarcGrol/microgen
# fetch external dependencies:
go list -f {{.Deps}}' # to find dependencies of app
./sloc_count.sh # get overview of the application and its compoments
go test ./... # to run all unit tests
go install # to create executable "${GOPATH}/bin/microgen"
# Sync source-code with application-dsl
${GOPATH}/bin/microgen -tool=gen -base-dir=. # to generate interfaces based on ./application.go
go fmt ./... # to re-format generated code or
goimports -w=true . # requires goimports
# Start the bus
./bus/start_nsq.sh
# Start the application
${GOPATH}/bin/microgen -service=tour -port=8081 -base-dir=.
${GOPATH}/bin/microgen -service=gambler -port=8082 -base-dir=.
${GOPATH}/bin/microgen -service=news -port=8083 -base-dir=.
${GOPATH}/bin/microgen -service=collector -port=8084 -base-dir=.
${GOPATH}/bin/microgen -service=proxy -port=8080 -base-dir=.
# Fire commands into the application
curl -X POST --header "Content-type: application/json" --header "Accept: application/json" --data '{"year":2015}' "http://localhost:8080/api/tour"
curl -X POST --header "Content-type: application/json" --header "Accept: application/json" --data '{"year":2015,"id":9,"name":"VAN GARDEREN Tejay","team":"BMC"}}' "http://localhost:8080/api/tour/2015/cyclist"
curl -X POST --header "Content-type: application/json" --header "Accept: application/json" --data '{"year":2012,"id":1,"thedate":"2012-06-30T09:00:00+02:00","startLocation":"Liège","finishLocation":"Liège","length":6,"kind":4}' "http://localhost:8080/api/tour/2015/etappe"
curl -X GET --header "Accept: application/json" "http://localhost:8080/api/tour/2015"
# Open UI in browser
http://localhost:8080/static/index.html