Skip to content
This repository has been archived by the owner on Mar 16, 2022. It is now read-only.

feature/go-support #103

Closed
wants to merge 42 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
f00c69e
[go-support] made TCK test #1 happy... very raw but some circles clos…
marcellanz Aug 31, 2019
3c8dea7
[go-support] EntityDiscoveryManager happy with the EntitySpec it gets…
marcellanz Aug 31, 2019
5ca9a02
[go-support] shopping cart implemented. TCK nearly green (lacking: ev…
marcellanz Sep 5, 2019
6d7af6f
[go-support] TCK 100% passed having the shopping cart example impleme…
marcellanz Sep 8, 2019
879d93b
[go-support] return errors instead of panic
marcellanz Sep 8, 2019
336d4d4
[go-support] having the shopping cart in a separate package. travis b…
marcellanz Sep 8, 2019
9d28e64
[go-support] build go support solely in travis TCK run
marcellanz Sep 8, 2019
ec0b7cc
[go-support] added copyright
marcellanz Sep 8, 2019
d0fc04d
[go-support] refactored from PoC to a CloudState Go Client API implem…
marcellanz Sep 13, 2019
9e0a8dd
[go-support] fix go test and TCK on travis
marcellanz Sep 13, 2019
017fed6
[go-support] apply CodeReviewComments#indent-error-flow best practice…
marcellanz Sep 13, 2019
f7409c0
[go-support] removed logging
marcellanz Sep 13, 2019
ee52ef5
[go-support] fixes from my own code review :-), applying some of Go's…
marcellanz Sep 13, 2019
4b220df
[go-support] removed main.go, README cleanup
marcellanz Sep 13, 2019
da7d4fe
[go-support] changes from my own code review
marcellanz Sep 13, 2019
744c790
[go-support] having a sbt task to build the go shopping cart binary w…
marcellanz Sep 15, 2019
360e379
[go-support] the PoC is removed
marcellanz Sep 15, 2019
e5ef360
[go-support] refactored the filedescriptor registration, more documen…
marcellanz Sep 15, 2019
19bcebb
[go-support] fix copy paste error
marcellanz Sep 15, 2019
ece6c1e
[go-support] "Errorf format %w has unknown verb w" => lets go with go…
marcellanz Sep 15, 2019
dbd08e5
[go-support] fix refactoring bug. let the TCK report the config.name …
marcellanz Sep 15, 2019
29f3d4a
[go-support] fix travis build
marcellanz Sep 15, 2019
c99b53d
[go-support] EventSourcedHandler#handleEvents tries to find now a fit…
marcellanz Sep 16, 2019
4076d10
[go-support] *.proto files and .pb.go files removed
marcellanz Sep 16, 2019
69fb515
[go-support] generate *.pb.go on the fly. move generate script to ./b…
marcellanz Sep 16, 2019
26d10af
[go-support] compile protocol buffers with travis build
marcellanz Sep 17, 2019
a48158a
[go-support] having protocol buffers compilation on travis
marcellanz Sep 17, 2019
eea762d
[go-support] fix .travis.yml by adding missing shell command
marcellanz Sep 17, 2019
f38145e
Merge remote-tracking branch 'upstream/master' into feature/go-support
marcellanz Sep 17, 2019
ed14794
Merge remote-tracking branch 'upstream/master' into feature/go-support
marcellanz Sep 18, 2019
7f9288b
[go-support] documentation for go-support
marcellanz Sep 18, 2019
46506e4
[go-support] added one TODO and moved a comment
marcellanz Sep 18, 2019
0994ad3
[go-support, cherry-picked from √] Hooking the Go compilation step in…
viktorklang Sep 16, 2019
ba47ee0
[go-support, cherry-picked from √] ran sbt scalafmtSbt
marcellanz Sep 18, 2019
8cfc8b8
[go-support, PR#103] cleaned up fatals, replaced with errors. recover…
marcellanz Sep 20, 2019
c1afd6a
[go-support] added prebuild script, let protoc be built, added script…
marcellanz Sep 21, 2019
2deb37a
[go-support] added tests
marcellanz Sep 26, 2019
677151a
Merge remote-tracking branch 'upstream/master' into feature/go-suppor…
marcellanz Sep 26, 2019
2bdc3b7
[go-support] build tck with -race flag enabled
marcellanz Sep 26, 2019
35e57ae
[go-support] enabled CGO for the -race flag
marcellanz Sep 26, 2019
ee725d5
[go-support] for every call of Emit the emitted event is handled imme…
marcellanz Sep 26, 2019
31faefd
[go-support] improve tests (capture output of some, enable verbose mo…
marcellanz Sep 27, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -36,3 +36,4 @@ target-sbt
node_modules
.idea/
.nyc_output
go-support/shoppingcart/cmd/shoppingcart/tck_shoppingcart
19 changes: 13 additions & 6 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,28 +25,35 @@ jobs:
script:
- cd node-support && npm test && cd -
- cd samples/js-shopping-cart && npm test && cd -

- name: sbt tests
before_script: sbt update
script: sbt 'set concurrentRestrictions in Global += Tags.limitAll(1)' test


- name: go tests
before_script:
- ./go-support/build/prebuild_go-support_in_docker.sh
script:
- cd go-support && ./build/run_go_test_in_docker.sh && cd -

- name: docs tests
before_script:
- cd node-support && npm install && cd -
- cd docs/src/test/js && npm install && cd -
script:
- sbt 'set concurrentRestrictions in Global += Tags.limitAll(1)' docs/paradox docs/test
- cd docs/src/test/js && npm test && cd -

- stage: TCK
name: Run TCK against reference implementation
before_script:
- sbt update
- cd node-support && npm install && cd -
- cd samples/js-shopping-cart && npm install && cd -
- sbt go-support/compile
script:
- sbt 'set concurrentRestrictions in Global += Tags.limitAll(1)' tck/it:test

- stage: Deploy
name: Deploy documentation to cloudstate.io
if: branch = master AND type = push
Expand All @@ -64,13 +71,13 @@ jobs:
name: CloudState Deployment Bot
on:
branch: master

- name: Publish docker images
if: tag =~ /^v*/
script:
- echo "$DOCKER_PASSWORD" | docker login -u cloudstatebot --password-stdin
- sbt "set concurrentRestrictions in Global += Tags.limitAll(1)" "dockerBuildAllNonNative publish" operator/docker:publish

- name: Upload CloudState descriptor
if: tag =~ /^v*/
script: sbt operator/compileK8sDescriptor
Expand Down
32 changes: 30 additions & 2 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import java.util.Date
import com.typesafe.sbt.packager.docker.DockerChmodType
import sbt.Keys.{developers, scmInfo}
import sbt.url
import scala.sys.process._

inThisBuild(
Seq(
Expand Down Expand Up @@ -128,7 +129,9 @@ lazy val docs = (project in file("docs"))
"extref.jsdoc.base_url" -> ".../user/lang/javascript/api/module-cloudstate.%s",
"cloudstate.version" -> "0.4.3", // hardcode, otherwise we'll end up with the wrong version in the docs
"cloudstate.java-support.version" -> "0.4.3",
"cloudstate.node-support.version" -> "0.0.1"
"cloudstate.node-support.version" -> "0.0.1",
"cloudstate.go-support.version" -> "0.0.1",
"cloudstate.go.version" -> "1.13"
),
paradoxNavigationDepth := 3,
inConfig(Test)(
Expand Down Expand Up @@ -658,6 +661,31 @@ lazy val `java-shopping-cart` = (project in file("samples/java-shopping-cart"))
}
)

lazy val `go-support` = (project in file("go-support"))
.settings(
name := "go-support",
(compile in Compile) := {
// FIXME only recompile if any of the files in the project has changed
import scala.util.Properties.{isLinux, isMac, isWin}
val (os_arch, cmd) =
if (isLinux) ("linux on amd64", "./go-support/build/run_tck_shopping_cart_build.sh linux amd64")
else if (isMac) ("darwin on amd64", "./go-support/build/run_tck_shopping_cart_build.sh darwin amd64")
else if (isWin)
("windows on amd64", "& .\\go-support\\build\\run_tck_shopping_cart_build.ps1 windows amd64") // FIXME IMPLEMENT
else throw new IllegalStateException("Running on an unsupported OS/ARCH combination: " + sys.props("os.name"))

val SuccessMessage = "go-support compiled successfully: " + os_arch + "\n"

(cmd !) match {
case 0 =>
streams.value.log.success(SuccessMessage)
(compile in Compile).value // TODO should we produce our own compilation result instead perhaps?
case err =>
throw new IllegalStateException("go-support compilation failed: " + err)
}
}
)

lazy val `akka-client` = (project in file("samples/akka-client"))
.enablePlugins(AkkaGrpcPlugin)
.settings(
Expand Down Expand Up @@ -719,7 +747,7 @@ lazy val `tck` = (project in file("tck"))
fork in test := true,
parallelExecution in IntegrationTest := false,
executeTests in IntegrationTest := (executeTests in IntegrationTest)
.dependsOn(`proxy-core` / assembly, `java-shopping-cart` / assembly)
.dependsOn(`proxy-core` / assembly, `java-shopping-cart` / assembly, `go-support` / Compile / compile)
.value
)

Expand Down
1 change: 1 addition & 0 deletions docs/src/main/paradox/developer/language-support/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ This is achieved by having a gRPC based protocol between the Proxy and the User

* Java
* JavaScript
* Go

## Creating language support libraries

Expand Down
4 changes: 4 additions & 0 deletions docs/src/main/paradox/user/lang/go/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
# Go API docs

The Go API docs can be found [here](https://godoc.org/?q=cloudstate.io%2Fgosupport).
(TODO: supply it by godoc.org, but that needs a public available repo)
6 changes: 6 additions & 0 deletions docs/src/main/paradox/user/lang/go/crdt.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
# Conflict-free Replicated Data Types

* Explain how to use the CRDT API
* Explain how to use CrdtFactory and where it comes from
* Explain how to handle streamed calls
* Explain the APIs for each different CRDT
5 changes: 5 additions & 0 deletions docs/src/main/paradox/user/lang/go/effects.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Forwarding and effects

* Explain the ServiceCallFactory interface
* Explain how to forward replies to another service.
* Explain how to emit effects.
117 changes: 117 additions & 0 deletions docs/src/main/paradox/user/lang/go/eventsourced.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Event sourcing

This page documents how to implement CloudState event sourced entities in Go. For information on what CloudState event sourced entities are, please read the general @ref[Event sourcing](../../features/eventsourced.md) documentation first.

An event sourced entity can be created by embedding the `cloudstate.EventEmitter` type and also implementing the `cloudstate.EntityInitializer` interface. **(TODO: link to godoc)**

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #entity-type }

Then by composing the CloudState entity with an `cloudstate.EventSourcedEntity` and register it with `cloudState.Register()`, your entity gets configured to be an event sourced entity and handled by the CloudState instance for now on.

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/eventsourced.go) { #event-sourced-entity-type }

The `PersistenceID` is used to namespace events in the journal, useful for when you share the same database between multiple entities. It defaults to the simple name for the entity type (in this case, `ShoppingCart`), it's good practice to select one explicitly, this means your database isn't depend on type names in your code.

The `SnapshotEvery` parameter controls how often snapshots are taken, so that the entity doesn't need to be recovered from the whole journal each time it's loaded. If left unset, it defaults to 100. Setting it to a negative number will result in snapshots never being taken.

## Persistence types and serialization

Event sourced entities persist events and snapshots, and these need to be serialized when persisted. The most straight forward way to persist events and snapshots is to use protobufs. CloudState will automatically detect if an emitted event is a protobuf, and serialize it as such. For other serialization options, including JSON, see @ref:[Serialization](serialization.md).

While protobufs are the recommended format for persisting events, it is recommended that you do not persist your services protobuf messages, rather, you should create new messages, even if they are identical to the services. While this may introduce some overhead in needing to convert from one type to the other, the reason for doing this is that it will allow the services public interface to evolve independently from its data storage format, which should be private.

For our shopping cart example, we'll create a new file called `domain.proto`, the name domain is selected to indicate that these are my applications domain objects:

@@snip [domain.proto](/docs/src/test/proto/domain.proto)

## State

Each entity should store its state locally in a mutable variable, either a mutable field or a multiple structure such as an array type or slice. For our shopping cart, the state is a slice of products, so we'll create a slice of LineItems to contain that:

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #entity-state }

## Constructing

The CloudState Go Support Library needs to know how to construct and initialize entities. For this, an entity has to implement the `cloudstate.EntityInitializer` interface.

(TODO: provide: The constructor below shows having the entity id injected)

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #constructing }

## Handling commands

Command handlers are declared by implementing the gRPC ShoppingCartServer interface which is generated from the protobuf definitions. The CloudState Go Support library together with the registered ServiceName in the `cloudstate.EventSourcedEntity` is then able to dispatch commands it gets from the CloudState proxy.

The return type of the command handler is by definition of the service interface, the output type for the gRPC service call, this will be sent as the reply.

The following shows the implementation of the `GetCart` command handler. This command handler is a read-only command handler, it doesn't emit any events, it just returns some state:

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #get-cart }

### Emitting events

Commands that modify the state may do so by emitting events.

@@@ warning
The **only** way a command handler may modify its state is by emitting an event. Any modifications made directly to the state from the command handler will not be persisted, and when the entity is passivated and next reloaded, those modifications will not be present.
@@@

A command handler may emit an event by using the embedded `cloudstate.EventEmitter` and invoking the `Emit` method on it. Calling `Emit` will immediately invoke the associated event handler for that event - this both validates that the event can be applied to the current state, as well as updates the state so that subsequent processing in the command handler can use it.

Here's an example of a command handler that emits an event:

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #add-item }

This command handler also validates the command, ensuring the quantity items added is greater than zero. Returning a `error` fails the command and the support library takes care of signaling that back to the requesting proxy as a `Failure` reply.

## Handling events

Event handlers are invoked at two points, when restoring entities from the journal, before any commands are handled, and each time a new event is emitted. An event handlers responsibility is to update the state of the entity according to the event. Event handlers are the only place where its safe to mutate the state of the entity at all.

Event handlers are declared by either implementing the `cloudstate.EventHandler` interface

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/event.go) { #event-handler }

or implementing an unary method that matches the type of the event to be handled. Event handlers are differentiated by the type of event they handle. By default, the type of event an event handler handles will be determined by looking for a single argument that the event handler takes. If for any reason this needs to be overridden, or if the event handler method doesn't exists at all, the event is handed over to the `cloudstate.EventHandler` `Handle` method when the entity implements that interface. The by implementing the `HandleEvent(event interface{}) (handled bool, err error)` method, a event handler indicates if he handled the event or if any occurred, returns an error. The returned error has precedent and the handled flag would not be considered.

Here's an example event handler for the `ItemAdded` event.

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #item-added }

## Producing and handling snapshots

## Multiple behaviors

Multiple behaviors are not supported yet by the Go support library.

## Registering the entity

Once you've created your entity, you can register it with the `cloudstate.CloudState` server, by invoking the `Register` method of an CloudState instance. In addition to passing your entity type and service name, you also need to pass any descriptors that you use for persisting events, for example, the `domain.proto` descriptor.

During registration the oprtional ServiceName and the ServiceVersion can be configured as Options.

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #register }

```
// main creates a CloudState instance and registers the ShoppingCart
// as a event sourced entity.
func main() {
cloudState := cloudstate.NewCloudState(&cloudstate.Options{
ServiceName: "shopping-cart",
ServiceVersion: "0.0.1",
})
err := cloudState.Register(
&cloudstate.EventSourcedEntity{
Entity: (*ShoppingCart)(nil),
ServiceName: "com.example.shoppingcart.ShoppingCart",
},
cloudstate.DescriptorConfig{
Service: "shoppingcart/shoppingcart.proto",
}.AddDomainDescriptor("shoppingcart/persistence/domain.proto"),
)
if err != nil {
log.Fatal(err)
}
cloudState.Run()
}
```
78 changes: 78 additions & 0 deletions docs/src/main/paradox/user/lang/go/gettingstarted.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Getting started with stateful services in Go

## Prerequisites

Go version
: CloudState Go support requires at least Go $cloudstate.go.version$

Build tool
: CloudState does not require any particular build tool, you can select your own.

protoc
: Since CloudState is based on gRPC, you need a protoc compiler to compile gRPC protobuf descriptors. While this can be done by downloading, installing and running protoc manually.

docker
: CloudState runs in Kubernetes with [Docker](https://www.docker.com), hence you will need Docker to build a container that you can deploy to Kubernetes. Most popular build tools have plugins that assist in building Docker images.

In addition to the above, you will need to install the CloudState Go support library by issuing `go get -u cloudstate.io/cloudstate` or with module support let the dependency downloaded by `go [build|run]`.

By using the Go module support your go.mod file will reference the latest version of the support library or you can define which version you like to use.

go get
: @@@vars
```text
go get -u cloudstate.io/cloudstate
```
@@@

import path
: @@@vars
```text
import "cloudstate.io/cloudstate"
```
@@@

go.mod
: @@@vars
```
module example.com/yourpackage
require (
cloudstate.io/cloudstate $cloudstate.go-support.version$
)
go $cloudstate.go.version$
```
@@@

## Protobuf files

The CloudState Go Support Library provides no dedicated tool beside the protoc compiler to build your protobuf files. The CloudState protocol protobuf files as well as the shopping cart example application protobuf files are provided by the CloudState Protocol Repository (TODO: whats this?).

In addition to the protoc compiler, the gRPC Go plugin is needed to generate *.pb.go files. Please follow the instructions at the [Go support for Protocol Buffers](https://github.com/golang/protobuf) project page to install the protoc compiler as well as the `protoc-gen-go` plugin which also includes the Google standard protobuf types.

To build the example shopping cart application shown earlier in @ref:[gRPC descriptors](../../features/grpc.md), you could simply paste that protobuf into `protos/shoppingcart.proto`. You may wish to also define the Go package using the `go_package` proto option, to ensure the package name used conforms to Go package naming conventions:

```proto
option go_package = "example.com/shoppingcart";
```

Now if you place your protobuf files under protos/ and run `protoc --go_out=. *.proto`, you'll find your generated protobuf files in `TODO`.

## Creating and starting a server

## Creating a main package

Your main package will be responsible for creating the CloudState gRPC server, registering the entities for it to serve, and starting it. To do this, you can use the CloudState server type, for example:

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #shopping-cart-main }

We will see more details on registering entities in the coming pages.

## Interfaces to be implemented

CloudState entities in Go work by implementing interfaces and composing types.

To get support for the CloudState event emission the CloudState entity should embed the `cloudstate.EventEmitter` type. The EventEmitter allows the entity to emit events during the handling of commands.

Second, by implementing the `EntityInitializer` interface with its method `New() interface{}`, a CloudState instance gets to know how to create and initialize an event sourced entity.

@@snip [shoppingcart.go](/docs/src/test/go/docs/user/eventsourced/shoppingcart.go) { #compose-entity }
16 changes: 16 additions & 0 deletions docs/src/main/paradox/user/lang/go/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# Go

CloudState offers an idiomatic, annotation based Go support library for writing stateful services.

@@toc { depth=1 }

@@@ index

* [Getting started](gettingstarted.md)
* [Event sourcing](eventsourced.md)
* [Conflict-free Replicated Data Types](crdt.md)
* [Forwarding and effects](effects.md)
* [Serialization](serialization.md)
* [API docs](api.md)

@@@
32 changes: 32 additions & 0 deletions docs/src/main/paradox/user/lang/go/serialization.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Serialization

CloudState functions serve gRPC interfaces, and naturally the input messages and output messages are protobuf messages that get serialized to the protobuf wire format. However, in addition to these messages, there are a number of places where CloudState needs to serialize other objects, for persistence and replication. This includes:

* Event sourced @ref[events and snapshots](eventsourced.md#persistence-types-and-serialization).
* CRDT @ref[map keys and set elements](crdt.md), and @ref[LWWRegister values](crdt.md).

CloudState supports a number of types and serialization options for these values.

## Primitive types

CloudState supports serializing the following primitive types:

| Protobuf type | Go type |
|---------------|-------------|
| string | string |
| bytes | []byte |
| int32 | int32 |
| int64 | int64 |
| float | float32 |
| double | float64 |
| bool | bool |

The details of how these are serialized can be found @ref[here](../../../developer/language-support/serialization.md#primitive-values).

## JSON

CloudState uses the package [`encoding/json`](https://golang.org/pkg/encoding/json/) to serialize JSON. Any type that has a field declared with a string literal tag ``json:"fieldname"`` will be serialized to and from JSON using the [Marshaller and Unmarshaller](https://golang.org/pkg/encoding/json/#Marshal) from the Go standard library package `encoding/json`.

The details of how these are serialized can be found @ref[here](../../../developer/language-support/serialization.md#json-values).

Note that if you are using JSON values in CRDT sets or maps, the serialization of these values **must** be stable. This means you must not use maps or sets in your value, and you should define an explicit ordering for the fields in your objects. **(TODO: mention the ordering of fields here by the Go standard library implementation).**
Loading