When people start an enterprise application, one of the earliest questions is "how do we talk to the database". These days they may ask a slightly different question: "what kind of database should we use - relational or one of these NOSQL databases?".
But there's another question to consider: "should we use a database at all?"
This article presents a simple Kotlin implementation of the
Memory Image architectural pattern:
// Skeletal Kotlin TL;DR for the truly impatient:
class MemoryImageProcessor(private val system: Any,
private val eventSourcing: EventSourcing) {
init {
synchronized(this) {
eventSourcing.replay<Command> { command -> command.applyTo(system) }
}
}
fun execute(command: Command): Unit = synchronized(this) {
TxManager.begin()
try {
command.applyTo(system)
eventSourcing.append(command)
} catch (e: Exception) {
TxManager.rollback()
throw e
}
}
fun execute(query: Query): Any? = query.extractFrom(system)
}Memory image provides a reliable persistence mechanism where all application data resides safely in main memory. Yes: good ole', volatile, random-access memory.
Provided you domain model fits in main memory (which is cheap and abundant nowadays) this approach yields significant benefits over the established database-centric approach:
- Substantially faster transaction and query processing times as the system operates at RAM speeds!
- No object-relation impedance mismatch to speak of as objects only reside natively in memory. No implementation-level ORM limitations bubbling up to the design
- Much richer domain models leveraging advanced language and platform features, unhampered by persistence concerns. DDD nirvana; the cure for anemic domain models đ
Rather than persisting domain entities as such (as is done, typically, on a database) in the memory image approach what gets persisted is the sequence of application events that modify the state of the entities in the first place.
Consider the following minimalistic bank domain model:
Here, a bank holds a collection of accounts each having a balance that changes in time as it responds to events such as:
- deposits
- withdrawals
- transfers
Each such an event can be modeled as a command that, when applied to an account, modifies the account's balance to reflect the banking operation embodied by the command.
This could be modeled as:
Let's take a look at a progression of commands and the evolving system state resulting from their successive application:
The key idea behind the memory image pattern is:
Serialize all state-modifying commands on persistent storage.
Reconstruct in-memory application state by replaying the deserialized commands onto an empty initial state
Somewhat paradoxically, entity classes themselves don't need to be persisted! (But they might, for snapshotting, as mentioned below.)
If your application data fits into available memory and your event history fits into available disk space then you're in business.
A memory image processor consumes a stream of application commands by applying each incoming command to a mutable object (here referred to as the system).
Because applying commands in memory is so fast and cheap, a memory image processor can run on a single thread and consume incoming command sequentially with no need to provide concurrent access to its system. This removes much of the complexity traditionally associated with transactions as conflicts arising from concurrent mutation simply do not occur.
Individual command application, however, can fail in the midst of a transactional sequence of mutations so the memory image processor still needs a way to rollback partial changes and restore system integrity in the face of invalid data and constraint violations.
Incoming commands should only be serialized upon successful completion. Obviously, if command serialization fails, the memory image processor will stop processing further commands until command serialization is restored.
Lastly (and crucially!), a memory image processor also services queries.
A query is another type of event which, unlike commands, does not mutate system state. Importantly, queries are serviced in multi-threaded mode so querying the system is efficient and concurrent. Because in-memory access is so fast, many queries can be satisfied without indexing. However, special-purpose, in-memory indexing can be easily implemented as dictated by application requirements.
The following class diagram puts it all together:
đ Because application restarts entail reprocessing the (potentially large) history of all mutating commands, system snapshotting can be used to serialize the entire in-memory state on demand. This enables faster restarts at the expense of losing the ability to time-travel.
The above class diagram is materialized in Kotlin as:
interface Command { fun applyTo(system: Any) }
interface Query { fun extractFrom(system: Any): Any? }
interface EventSourcing {
fun append(event: Any)
fun <E> replay(eventConsumer: (E) -> Unit)
}
interface TxManager {
fun begin()
fun <T> remember(who: Any, what: String, value: T, undo: (T) -> Unit)
fun rollback()
companion object: TxManager { ... } // thread-local tx
}
class MemoryImageProcessor(private val system: Any,
private val eventSourcing: EventSourcing) {
init {
// Replay previously serialized events to restore in-memory system state
synchronized(this) {
// Any failure during initialization will be propagated
eventSourcing.replay<Command> { command -> command.applyTo(system) }
}
}
// Apply incoming command to system
fun execute(command: Command): Unit = synchronized(this) { // Single-threaded
TxManager.begin()
try {
command.applyTo(system) // Try and apply command
try {
eventSourcing.append(command) // Serialize; retry internally if needed
} catch (e: Exception) {
// Note: no attempt to rollback: this is unrecoverable
logger.severe("Error persisting command: ${e.message ?: e.toString()}")
// No further processing; start over when serialization is restored
throw e
}
} catch (e: Exception) {
TxManager.rollback() // Undo any partial mutation
val errorMessage = "Error executing command: ${e.message ?: e.toString()}"
// It's (kinda) ok for a command to fail
// Re-throw as «CommandApplicationException» and go on
throw CommandApplicationException("Error executing command: ${e.message}", e)
}
}
// Run incoming query on system
fun execute(query: Query): Any? = query.extractFrom(system) // Can be multi-threaded
}To exercise the above memory image processor, let's revisit our bank domain model:
This is implemented in Kotlin as:
/* 1) Domain entities: Bank and Account */
typealias Amount = BigDecimal
data class Bank(val accounts: MutableMap<String, Account> = HashMap())
data class Account(val id: String, val name: String) {
var balance: Amount by TxDelegate(initialValue = Amount.ZERO) {
// tiggers rollback on validation failure
it >= Amount.ZERO
}
}
/* 2) Application commands: CreateAccount, Deposit, Withdrawal, Transfer */
data class CreateAccount(val id: String, val name: String) : BankCommand {
override fun applyTo(bank: Bank) {
bank.accounts[id] = Account(id, name)
}
}
data class Deposit(override val accountId: String, val amount: Amount) : AccountCommand {
override fun applyTo(account: Account) {
account.balance += amount
}
}
data class Withdrawal(override val accountId: String,val amount: Amount) : AccountCommand {
override fun applyTo(account: Account) {
account.balance -= amount
}
}
data class Transfer(val fromAccountId: String, val toAccountId: String, val amount: Amount) : BankCommand {
override fun applyTo(bank: Bank) {
Deposit(toAccountId, amount).applyTo(bank)
Withdrawal(fromAccountId, amount).applyTo(bank)
}
}The following test exercises the memory image processor using the same sequence of commands outlined above:
val bank1 = Bank()
val memimg1 = MemImg(bank1, eventSourcing)
memimg1.execute(CreateAccount("janet", "Janet Doe"))
assertEquals(Amount.ZERO, bank1.accounts["janet"]!!.balance)
memimg1.execute(Deposit("janet", Amount(100)))
assertEquals(Amount(100), bank1.accounts["janet"]!!.balance)
memimg1.execute(Withdrawal("janet", Amount(10)))
assertEquals(Amount(90), bank1.accounts["janet"]!!.balance)
memimg1.execute(CreateAccount("john", "John Doe"))
assertEquals(Amount.ZERO, bank1.accounts["john"]!!.balance)
memimg1.execute(Deposit("john", Amount(50)))
assertEquals(Amount(50), bank1.accounts["john"]!!.balance)
memimg1.execute(Transfer("janet", "john", Amount(20)))
assertEquals(Amount(70), bank1.accounts["janet"]!!.balance)
assertEquals(Amount(70), bank1.accounts["john"]!!.balance)
memimg1.close()
val bank2 = Bank()
val memimg2 = MemImg(bank2, eventSourcing)
// Look ma: system state restored from empty initial state via event sourcing!
assertEquals(Amount(70), bank2.accounts["janet"]!!.balance)
assertEquals(Amount(70), bank2.accounts["john"]!!.balance)
// Some random query; executes at in-memory speeds
val accountsWith70 = memimg2.execute(object : BankQuery {
override fun extractFrom(bank: Bank) =
bank.accounts.values
.filter { it.balance == Amount(70) }
.map { it.name }
.toSet()
})
assertEquals(setOf("Janet Doe", "John Doe"), accountsWith70)
// Attempt to transfer beyond means...
val insufficientFunds = assertThrows<CommandApplicationException> {
memimg2.execute(Transfer("janet", "john", Amount(1000)))
}
assertContains(insufficientFunds.message!!, "Invalid value for Account.balance")
// Look ma: system state restored on failure after partial mutation
assertEquals(Amount(70), bank2.accounts["janet"]!!.balance)
assertEquals(Amount(70), bank2.accounts["john"]!!.balance)
memimg2.close()Memory image provides a simple, straightforward way to achieve high performance and simplicity without the complications associated with persisting objects on a database (whether SQL or not.)
Kotlin is a uniquely expressive language in which to implement this architectural pattern.
Interested readers can inspect the working code at the kmemimg Github repository.