Refactor Sink Writer to accept blocks inplace of bytes #86
Conversation
|
Looks like we need to fix some tests |
Fix test cases in merge package
Fix all test cases
|
Tests are fixed. |
* Add yaml inline tag to BaseSink * Add inline tag to BaseSink * Revert sample yaml configuration
Fixed |
|
There's a few conflicts @tardunge |
will fix. |
* Add yaml inline tag to BaseSink * Add inline tag to BaseSink * Revert sample yaml configuration * fix conflict
* Add yaml inline tag to BaseSink * Add inline tag to BaseSink * Revert sample yaml configuration * fix conflict * add contributors in readme
| ) | ||
|
|
||
| // FromBlockBy creates and returns a list of new block.Row for a block. | ||
| func FromBlockBy(blk Block, schema typeof.Schema) ([]Row, error) { |
There was a problem hiding this comment.
Maybe this should be a method on a block instead?
| @@ -33,6 +35,30 @@ type Block struct { | |||
| schema typeof.Schema `binary:"-"` // The cached schema of the block | |||
| } | |||
|
|
|||
| // Create a base block for testing purpose | |||
| func Base() ([]Block, error) { | |||
There was a problem hiding this comment.
Do we need this function exported or we can move it to a test file itself?
| if err != nil { | ||
| return err | ||
| } | ||
| // buffer, err := s.merge(blocks, schema) |
There was a problem hiding this comment.
Why this is commented out?
| @@ -16,7 +19,20 @@ import ( | |||
| ) | |||
|
|
|||
| // ToParquet merges multiple blocks together and outputs a key and merged Parquet data | |||
| func ToParquet(blocks []block.Block, schema typeof.Schema) ([]byte, error) { | |||
| func ToParquet(input interface{}) ([]byte, error) { | |||
There was a problem hiding this comment.
I'm a bit wary of removing schema. Semantically taking a schema from the first block is different from providing a schema to the function itself. Why do we need to remove schema?
|
@tardunge please address comments in a separate PR(s) |
|
Sure. |
|
This MR introduced an issue that the Azure writer would not initiate. @tardunge |
Can you create an issue and post the error or behaviour you are encountering and also the expected behaviour? |
No description provided.