Releases: urban-wombat/flattables
New flattables_sample with stock exchange end-of-week data
The previous flattables_sample
was three tables of Mickey Mouse data. Actually, Wombat and Flintstones stuff.
I think people find it challenging to come up with a use-case for flattables
. And having multiple tables may have given the impression that it's a kind of in-memory database. It's true that gotables
can be used as a flexible in-memory data store, and can easily save or transmit data. However, flattables
is more of a create-a-table-and-transmit-it kind of format. It is a fast binary constrained (tables only) FlatBuffers
that takes the effort out of designing and writing the schema and plumbing code, so long as you're happy with tabula data.
The new flattables_sample
is an end-of-week dump of data from the Australian Stock Exchange. It conforms perfectly with the flattables
tabular format. And you can, of course, add one or more other tables to the transmission to carry, say, metadata about the main payload.
Some minor cosmetic bug fixes have been made. No actual consequential bugs, that I'm aware of.
Consistent with the it's-not-a-database theme, I went through and surgically removed all the GraphQL
code. It seemed like a fine idea at the time, but that's not what flattables
(or FlatBuffers
) is, and so it had to go.
The flattables_sample
table is around 1,500 rows. Here are the first few lines to give you and idea:
[AsxMarketDataWeek20190712]
ticker date open high low close volume
string int32 float32 float32 float32 float32 int32
"360" 20190712 3.8 3.8 3.8 3.8 207325
"14D" 20190712 0.29 0.29 0.28 0.285 173125
"1AD" 20190712 0.17 0.17 0.17 0.17 25000
"1ST" 20190712 0.093 0.098 0.091 0.097 969749
"2BE" 20190712 0.395 0.395 0.345 0.36 384609
"3DP" 20190712 0.046 0.047 0.045 0.047 545265
"3PL" 20190712 0.967 0.967 0.965 0.965 243734
"4CE" 20190712 0.02 0.02 0.016 0.018 3725883
"4DS" 20190712 0.061 0.061 0.058 0.058 1845766
"5GN" 20190712 1.62 1.62 1.55 1.58 296411
"88E" 20190712 0.018 0.018 0.017 0.018 2317656
"8CO" 20190712 0.16 0.16 0.16 0.16 6500
"8EC" 20190712 0.8 0.8 0.8 0.8 991
"8IH" 20190712 0.06 0.06 0.06 0.06 3500
"9SP" 20190712 0.023 0.024 0.023 0.023 10147822
"A1G" 20190712 0.16 0.16 0.16 0.16 2300
"A1M" 20190712 0.37 0.42 0.37 0.42 73776
"A2B" 20190712 1.715 1.785 1.715 1.78 38280
"A2M" 20190712 15.81 15.94 15.79 15.85 2081197
"A3D" 20190712 0.36 0.385 0.36 0.365 74680
"A40" 20190712 0.13 0.135 0.13 0.135 1704143
"A4N" 20190712 0.115 0.115 0.115 0.115 15000
"AAA" 20190712 50.09 50.1 50.09 50.09 629689
Three fast functions for reading/writing FlatBuffers
Substantial performance improvements in functions and methods used with reading and writing FlatBuffers.
Sample code can be found here with Examples of new functions for processing FlatBuffers that are much faster.
https://godoc.org/github.com/urban-wombat/flattables_sample#pkg-examples
The new functions are:
All three functions move data between FlatBuffers and a simple Go collection of tables.
OldSliceFromFlatBuffers() reuses memory which reduces cpu and memory consumption.
To get started with gotables FlatBuffers see https://github.com/urban-wombat/flattables#a-simple-and-fast-way-to-get-started-with-google-flatbuffers
It may sound hard to believe, but all the code at https://github.com/urban-wombat/flattables_sample was auto-generated in seconds a single gotables schema file.
The objective is to define your FlatBuffers table(s) and run flattablesc to generate both Google FlatBuffers flatc code and specialised gotables table-only code.
Google FlatTables support: more methods and documentation
In a nutshell:
- Install FlatTables
- Write a data tables file
- Run
$ gotflat -f ../flattables_sample/tables.got -n flattables_sample -p github.com/urban-wombat/flattables_sample
- You get this: FlatBuffers API
- And this: FlatBuffers Tutorial
Recent work has focused on Google FlatTables support.
More methods. More documentation.
-
gotables now provides a broad introduction to
FlatTables
, thegotables
support library forFlatBuffers
. -
flattables shows how to install
gotables
,flattables
and the associated utilitygotflat
which generates Go code and a project-specific tutorial explaining how to useFlatBuffers
with theFlatTables
library. -
flattables_sample_main.go is a project-specific tutorial generated from my sample
gotables
file tables.got.
If I've missed something in the instructions that would help you get started, hit me up on: [email protected]
flatbuffers schema can now have a (deprecated) attribute.
To deprecate a gotables column (flatbuffers field) in your gotables
file you are using as a schema and code generator source:
rename the column and append _deprecated_
to the name.
myColumnName
becomes myColumnName_deprecated_
All data must be sent using the original name myColumnName
(from processes
that don't know about the deprecation change), or must send no data at all
for myColumnName
.
See previous releases (below) for how to install and use the flattables flatbuffers code generation.
Refer to flatbuffers Writing A Schema
to understand some of the benefits and gotchas of deprecated fields.
Additional functions
New functions
-
NewTableSetFromFlatBuffers.go
-
main.go
Note that with most other functions, these are generated specifically for your particular data tables,
similar to the way flatc
generates the underlying Go functions, which the above functions call into.
An easy way to get started with Google FlatBuffers using tables.
Google FlatBuffers is incredibly fast.
This library of Go templates takes a simple gotables data file and generates a conforming Google FlatBuffers
schema file.
gotft takes this schema file
and calls the Google FlatBuffers utility flatc
, and calls into the flattables
library to generate helper
functions to write and read FlatBuffers []byte arrays. It also generates tests and bench tests which you can run.
It provides a major simplification of FlatBuffers use to tabular tables. Which means your objects need to be
normalised sufficiently to write to tables, and be capable of denormalisation at the receiving end.
The code generation (using gotft
- and flatc
under the covers) is blindingly simple and easy.
The repository https://github.com/urban-wombat/flattables_sample was generated from scratch
from just one gotables data file: https://github.com/urban-wombat/flattables_sample/blob/master/tables.got
by running the following command:
gotft -f tables.got -n flattables_sample
And here's what you can run straight after that code generation step:
$ go test -bench=.
goos: windows
goarch: amd64
pkg: github.com/urban-wombat/flattables_sample
BenchmarkGetFlatBuffersAndCompareWithGotables-4 200000 9006 ns/op
BenchmarkGetFlatBuffersOnly-4 5000000 269 ns/op
PASS
ok github.com/urban-wombat/flattables_sample 3.603s