Code

Live & bulk loader changes#2961

Code

Merged

codexnull merged 62 commits intomasterhypermodeinc/dgraph:masterfrom javier/issue2889_live_load_jsonhypermodeinc/dgraph:javier/issue2889_live_load_jsonCopy head branch name to clipboardFeb 6, 2019

+1,287-620

Commits

Commits on Jan 12, 2019

• Accept directory as argument to --rdfs and look for .rdf and .json files there.

  

  Javier Alvarado

  committed
  6145526

  Copy full SHA for 6145526

• Merge branch 'master' into javier/issue2889_live_load_json

  

  Javier Alvarado

  committed
  6e395be

  Copy full SHA for 6e395be

Commits on Jan 14, 2019

• Add x.FileReader() function to return a reader that decompresses if necessary.

  

  Javier Alvarado

  committed
  98e0ea1

  Copy full SHA for 98e0ea1

• Forward file to the RDF or JSON processor as appropriate.

  

  Javier Alvarado

  committed
  3ca188a

  Copy full SHA for 3ca188a

Commits on Jan 15, 2019

• Working version?

  

  Javier Alvarado

  committed
  de3da8f

  Copy full SHA for de3da8f

Commits on Jan 17, 2019

• Add live test stub.

  

  Javier Alvarado

  committed
  c4291db

  Copy full SHA for c4291db

• Ensure pending nquads are submitted after EOF.

  

  Javier Alvarado

  committed
  5866bd7

  Copy full SHA for 5866bd7

• Change references to RDFs to N-Quads to be more correct.

  

  Javier Alvarado

  committed
  86c421f

  Copy full SHA for 86c421f

• Merge master.

  

  Javier Alvarado

  committed
  9cb6dca

  Copy full SHA for 9cb6dca

• Export TMPDIR in test.sh so that hopefully tests will use it. Change generic test data filenames.

  

  Javier Alvarado

  committed
  bdfcd85

  Copy full SHA for bdfcd85

• Change test data schema to reflect changes merged from master.

  

  Javier Alvarado

  committed
  1e9c530

  Copy full SHA for 1e9c530

• Live JSON load test.

  

  Javier Alvarado

  committed
  c495a01

  Copy full SHA for c495a01

Commits on Jan 18, 2019

• Auto-detect JSON in load files with no .rdf or .json extension in the name. Improve live loader messages.

  

  Javier Alvarado

  committed
  f499943

  Copy full SHA for f499943

• Fix bug reading streamed JSON. Add test of loading streamed JSON.

  

  Javier Alvarado

  committed
  7b5ad3b

  Copy full SHA for 7b5ad3b

• Live load testing improvements.

  

  Javier Alvarado

  committed
  405c60a

  Copy full SHA for 405c60a

Commits on Jan 19, 2019

• First attempt at autogenerating uid from key fields.

  

  Javier Alvarado

  committed
  37c3437

  Copy full SHA for 37c3437

• Replace RDFs in message with N-Quads.

  

  Javier Alvarado

  committed
  afe44d4

  Copy full SHA for afe44d4

• Add debugging code.

  

  Javier Alvarado

  committed
  9da9d8e

  Copy full SHA for 9da9d8e

Commits on Jan 21, 2019

• Fix bug assigning blank ids. Move more testing code to testing package.

  

  Javier Alvarado

  committed
  fce4ff6

  Copy full SHA for fce4ff6

• Add test of live loading multiple JSON files.

  

  Javier Alvarado

  committed
  fbad4cf

  Copy full SHA for fbad4cf

• Add test of live loading JSON without UID field. Testing improvements.

  

  Javier Alvarado

  committed
  8c073d5

  Copy full SHA for 8c073d5

Commits on Jan 22, 2019

• Add tests of live JSON load with auto-uid.

  

  Javier Alvarado

  committed
  f828d57

  Copy full SHA for f828d57

• Merge branch 'master' into javier/issue2889_live_load_json

  

  Javier Alvarado

  committed
  44b1a73

  Copy full SHA for 44b1a73

• Small restartCluster fix.

  

  Javier Alvarado

  committed
  cc59eed

  Copy full SHA for cc59eed

• Merge master.

  

  Javier Alvarado

  committed
  f6b25ca

  Copy full SHA for f6b25ca

• Rename fields to be more correct or specific.

  

  Javier Alvarado

  committed
  06fd748

  Copy full SHA for 06fd748

• Improved comments.

  

  Javier Alvarado

  committed
  a23a169

  Copy full SHA for a23a169

• Remove test.sh change that should be part of another branch.

  

  Javier Alvarado

  committed
  4495702

  Copy full SHA for 4495702

Commits on Jan 23, 2019

• Fix bugs catching errors.

  

  Javier Alvarado

  committed
  2a94576

  Copy full SHA for 2a94576

• Make auto-added blank uid fields per file.

  

  Javier Alvarado

  committed
  c9833fd

  Copy full SHA for c9833fd

Commits on Jan 24, 2019

• Don't bother hashing the concatenated key values to generate a blank id, just pass the concatenated string to get a real uid.

  

  Javier Alvarado

  committed
  cee8c8d

  Copy full SHA for cee8c8d

Commits on Jan 25, 2019

• Refactor processRdfFile

  

  Javier Alvarado

  committed
  3b7b648

  Copy full SHA for 3b7b648

Commits on Jan 28, 2019

• Minor changes.

  

  Javier Alvarado

  committed
  ad641a2

  Copy full SHA for ad641a2

• Add JsonToNquads to replace NquadsFromJson later.

  

  Javier Alvarado

  committed
  5b4c423

  Copy full SHA for 5b4c423

• Rename some functions. Don't dump stack trace for input error.

  

  Javier Alvarado

  committed
  4747690

  Copy full SHA for 4747690

• Move chunk test from bulk package to x package.

  

  Javier Alvarado

  committed
  6f01e18

  Copy full SHA for 6f01e18

Commits on Jan 29, 2019

• Change processRdf to use a chunker as well.

  

  Javier Alvarado

  committed
  0cb8030

  Copy full SHA for 0cb8030

• Move chunker from x package to a new loadfile package. Remove bulk chunker.

  

  Javier Alvarado

  committed
  ce62a5a

  Copy full SHA for ce62a5a

Commits on Jan 30, 2019

• Merge master.

  

  Javier Alvarado

  committed
  7286176

  Copy full SHA for 7286176

Commits on Jan 31, 2019

• Refactor more bulk/live code.

  

  Javier Alvarado

  committed
  1a409cf

  Copy full SHA for 1a409cf

• Minor testing changes.

  

  Javier Alvarado

  committed
  0454e92

  Copy full SHA for 0454e92

• Merge master.

  

  Javier Alvarado

  committed
  77d029b

  Copy full SHA for 77d029b

• Merge master.

  

  Javier Alvarado

  committed
  50f5744

  Copy full SHA for 50f5744

• Add a small visual indicator of when a long-running test starts.

  

  Javier Alvarado

  committed
  500fa12

  Copy full SHA for 500fa12

• Fix rdfChunker.Parse to parse all RDF lines in chunk instead of only first one.

  

  Javier Alvarado

  committed
  05ff572

  Copy full SHA for 05ff572

• Add check for EOF in RDF chunk parser.

  

  Javier Alvarado

  committed
  487dcc5

  Copy full SHA for 487dcc5

• Add --key option to bulk loader for parity with live loader.

  

  Javier Alvarado

  committed
  17125d9

  Copy full SHA for 17125d9

Commits on Feb 1, 2019

• Fix batching, which was broken by live/bulk refactoring.

  

  Javier Alvarado

  committed
  8e7b581

  Copy full SHA for 8e7b581

• Parse key fields earlier in the process.

  

  Javier Alvarado

  committed
  1c2c9ef

  Copy full SHA for 1c2c9ef

• Cleanup suggestions from PR.

  

  Javier Alvarado

  committed
  116a794

  Copy full SHA for 116a794

Commits on Feb 5, 2019

• PR review fixes.

  

  Javier Alvarado

  committed
  8f16e04

  Copy full SHA for 8f16e04

• Fix warning.

  

  Javier Alvarado

  committed
  7685150

  Copy full SHA for 7685150

• Remove --key support since it requires more thought.

  

  Javier Alvarado

  committed
  dd237b1

  Copy full SHA for dd237b1

Commits on Feb 6, 2019

• Remove two-line functions processJsonFile and processRdfFile.

  

  Javier Alvarado

  committed
  db7f34f

  Copy full SHA for db7f34f

• Inline nextNquads() and finalNquads() into processLoadFile().

  

  Javier Alvarado

  committed
  915eef8

  Copy full SHA for 915eef8

• Rename package loadfile to chunker.

  

  Javier Alvarado

  committed
  d09e925

  Copy full SHA for d09e925

• Move package rdf to chunker/rdf.

  

  Javier Alvarado

  committed
  94f355b

  Copy full SHA for 94f355b

• WIP...

  

  Javier Alvarado

  committed
  9f39ac9

  Copy full SHA for 9f39ac9

• Move json and rdf nquad parsing under chunker.

  

  Javier Alvarado

  committed
  6872833

  Copy full SHA for 6872833

• Move FileReader() and IsJSONData() from x package to chunker.

  

  Javier Alvarado

  committed
  ec990e2

  Copy full SHA for ec990e2

• Rename live --rdfs option to --files since it can load more than *.rdf files.

  

  Javier Alvarado

  committed
  ac41b58

  Copy full SHA for ac41b58

• Fix a bug where batch slice was being modified after sending to a channel.

  manishrjain

  committed
  6143636

  Copy full SHA for 6143636