Skip to content

Dataframe enhancements #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
fe2s merged 68 commits into from
Nov 5, 2018
Merged

Dataframe enhancements #108

fe2s merged 68 commits into from
Nov 5, 2018

Conversation

@fe2s
Copy link
Contributor

@fe2s fe2s commented Oct 31, 2018

ntviet18 and others added 30 commits October 20, 2018 19:08
@ntviet18
Dedupe rdd tests
8cb19f8
@ntviet18
Add dataframe cluster tests
be07ba4
@ntviet18
Fix cluster test data insertion
be770f9
@fe2s
Merge branch 'dataframe' of https://github.com/ntviet18/spark-redis i…
5387c28 
…nto dataframe
@fe2s
#99 #100 changed default values of max.pipeline.size and scan.count t…
ce7ef6b 
…o 100
@fe2s
update dataframe doc: java serialization
6871200
@fe2s
refactor tests: moved tests to proper packages, renaming
0cb8ade
@fe2s
dataframe test: read data from CSV file, write to redis, read from redis
4a630d8
@ntviet18
Create convenient method to restart Redis clusters
000c56a
@ntviet18
Create key utilities class to avoid serialization issues
d083971
@ntviet18
Create base Redis benchmark suite
105ce47
@ntviet18
Create base dataframe benchmark suite
90a5c0f
@ntviet18
Create base dataframe benchmark suite for a single element
c08ad62
@ntviet18
Create base dataframe benchmark suite for multiple elements
435fb6d
@ntviet18
Create scenarios for testing hash model
5f100ca
@ntviet18
Create scenarios for testing binary model
d1f7314
@ntviet18
Test dataframe read 0 fields
dd69e5e
@ntviet18
Test dataframe read all fields
fa097ff
@ntviet18
Test dataframe read 1 fields
6e12068
@ntviet18
Remove exporting report stage
1cde031
@ntviet18
Reduce logging verbosity
c26e396
@ntviet18
Round up many values Dataframe performance test
d942160
@ntviet18
Repackage Dataframe benchmark suites
ea91fe9
@ntviet18
Create separate benchmark test profile
eea95f9
@ntviet18
Fix merge conflicts
ab35886
@fe2s
Merge pull request #104 from ntviet18/dataframe
f2d2ff7 
Initial for dataframe performance benchmarking
@ntviet18
Extend Jedis pipeline to return key
33cd2cf
@ntviet18
Cleanup k/v hgetall test
80cac05
@ntviet18
Clarify k/v builder
f6f278d
@ntviet18
Implement k/v hmget
6a62e94
ntviet18 and others added 25 commits October 30, 2018 22:25
@ntviet18
Test extract key column with prefix pattern
f9b2fa5
@ntviet18
Use explicit type casts to avoid type erasure warnings
b7f28eb
@ntviet18
Document Redis row encoding
5cf3ab8
@ntviet18
Document Redis row decoding
4c3c512
@ntviet18
Decouple schema inference from row decoding
d19ebbb
@ntviet18
Cache filtered schema while scanning rows
e019d94
@ntviet18
Prune unreachable code while persisting dataframe rows
2e16f7d
@ntviet18
Document read _id from hashes with Redis key
9328b32
@ntviet18
Loan scan rows connection
0316628
@fe2s
Merge pull request #105 from ntviet18/hashKey
c348532 
Hash key
@fe2s
change tests to use keys pattern rather table name when reading pure …
10b8ee9 
…hashes
@fe2s
docs: simplify spark session creation
a21062f
@fe2s
Merge branch 'master' into dataframe
74bf7b5
@fe2s
docs: better formatting
2b1ef35
@fe2s
docs: wording and added more details
a15f7a9
@fe2s
docs: wording and added more details
4f7b8ff
@fe2s
fix markdown table
da2f58f
@fe2s
docs: minor fix
70d5299
@fe2s
docs: fix spelling
25b1f09
@fe2s
docs: remove old known limitation, the note on pattern is described a…
4d21342 
…bove
@fe2s
docs: restructure dataframes page
f8678d8
@fe2s
docs: update dataframes page
29167e3
@fe2s
docs: update dataframes page
941d155
@fe2s
docs: update dataframes page
1a1a4fd
@fe2s
Update dataframe.md
7e1ae03
@fe2s fe2s added this to the v2.3.1 milestone Oct 31, 2018
@fe2s fe2s requested a review from gkorland October 31, 2018 14:34
@fe2s fe2s changed the title Dataframes: Dataframe enhancements Oct 31, 2018
@fe2s fe2s merged commit bab0584 into master Nov 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@gkorland gkorland gkorland approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v2.3.1

Development

Successfully merging this pull request may close these issues.

4 participants

@fe2s @gkorland @ntviet18