Riak Tests for scenarios of basho/riak_kv#679 #719

russelldb · 2014-12-30T17:09:48Z

Note that the fallback test kv679_dataloss_fb will fail until that case is addressed with a riak_object format change.

seancribbs · 2014-12-31T16:50:13Z

Does this need a rebase? Looks like kv679_uid already merged.

russelldb · 2015-01-02T09:01:30Z

hmmmm, yes. Can I force push if needed?

Dataloss at coordinator Dataloss at coordinator with an old clock repaired Vnode id's not unique

add some debugging log out

russelldb · 2015-01-02T09:28:27Z

I did it anyway, sorry

seancribbs · 2015-01-02T14:42:25Z

Nice, thanks! Ping when you're ready for review.

russelldb · 2015-01-02T14:43:09Z

ping @seancribbs, ready for review

andrewjstone · 2015-01-14T23:45:51Z

tests/kv679_dataloss.erl

This timeout sticks out as something that may not be helpful for repeatability. Are we sure it's even necessary here?

It's not, vestige of early iteration. Good catch.

andrewjstone · 2015-01-15T00:09:36Z

kv679_dataloss passes with PR, fails without
kv679_dataloss code review
kv679_dataloss_fb passes with PR, fails without
kv679_dataloss_fb code review - As far as I can tell the test is correct, however it is failing as shown in a comment below.
kv679_tombstone passes with PR, fails without
kv679_tombstone code review
kv679_tombstone2 passes with PR, fails without
kv679_tombstone2 code review

andrewjstone · 2015-01-15T00:37:09Z

tests/kv679_dataloss_fb.erl

For clarity, I'd change this function name to primary_and_fallback_counts/1 to indicate it's actual function and not the expected result.

andrewjstone · 2015-01-15T00:50:34Z

Seeing the following failure with kv679_dataloss_fb

19:45:30.878 [info] Killed 2 primaries
19:45:30.883 [info] Got a preflist with coord and 2 fbs [{{319703483166135013357056057156686910549735243776,'[email protected]'},primary},{{342539446249430371453988632667878832731859189760,'[email protected]'},fallback},{{365375409332725729550921208179070754913983135744,'[email protected]'},fallback}]
19:45:31.202 [info] writing new!
19:45:31.534 [info] writing existing!
19:45:31.536 [info] VC [{<<157,62,8,202,207,232,52,51,0,0,0,1>>,{2,63588501931}}]
19:45:31.536 [info] Coord Node ID <<157,62,8,202>>
19:45:31.536 [info] Clock at 2 fallbacks
19:45:31.536 [info] Killing node '[email protected]'
19:45:31.536 [info] Killing node '[email protected]'
19:45:31.537 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev4/bin/riak start
19:45:34.764 [info] Wait until '[email protected]' is pingable
19:45:35.174 [info] Wait until '[email protected]' is pingable
19:45:35.174 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev5/bin/riak start
19:45:38.389 [info] Wait until '[email protected]' is pingable
19:45:38.776 [info] Wait until '[email protected]' is pingable
19:45:38.776 [info] started primaries back up
19:45:48.789 [info] writing existing!
19:45:48.833 [info] writing existing!
19:45:48.835 [info] VC [{<<157,62,8,202,207,232,52,51,0,0,0,1>>,{4,63588501948}}]
19:45:48.836 [info] Coord Node ID <<157,62,8,202>>
19:45:48.836 [info] Killing node '[email protected]'
19:45:48.836 [info] Killing node '[email protected]'
19:45:48.836 [info] killed primaries again
19:45:48.838 [info] deleting backend data dir for 319703483166135013357056057156686910549735243776 '[email protected]' on primary
19:45:48.839 [info] Path "/Users/ajs/rt/riak_ee/current/dev/dev3/data/bitcask/319703483166135013357056057156686910549735243776"
19:45:48.839 [info] Stopping riak on '[email protected]'
19:45:58.840 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev3/bin/riak stop
19:46:02.498 [info] Wait until '[email protected]' is not pingable
19:46:02.499 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev3/bin/riak start
19:46:05.717 [info] Wait until '[email protected]' is pingable
19:46:06.081 [info] Wait until '[email protected]' is pingable
19:46:06.081 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev6/bin/riak start
19:46:09.347 [info] Wait until '[email protected]' is pingable
19:46:09.722 [info] Wait until '[email protected]' is pingable
19:46:09.722 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev1/bin/riak start
19:46:13.026 [info] Wait until '[email protected]' is pingable
19:46:13.393 [info] Wait until '[email protected]' is pingable
19:46:13.393 [info] restart fallbacks
19:46:23.529 [info] VC [{<<157,62,8,202,207,232,52,51,0,0,0,1>>,{2,63588501931}}]
19:46:23.529 [info] Coord Node ID <<157,62,8,202>>
19:46:23.530 [info] writing existing!
19:46:23.588 [info] VC [{<<157,62,8,202,207,232,52,51,0,0,0,1>>,{3,63588501983}}]
19:46:23.588 [info] Coord Node ID <<157,62,8,202>>
19:46:23.588 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev4/bin/riak start
19:46:26.850 [info] Wait until '[email protected]' is pingable
19:46:27.237 [info] Wait until '[email protected]' is pingable
19:46:27.237 [info] Running: /Users/ajs/rt/riak_ee/current/dev/dev5/bin/riak start
19:46:30.501 [info] Wait until '[email protected]' is pingable
19:46:30.878 [info] Wait until '[email protected]' is pingable
19:46:30.878 [info] restart primaries _again_
19:46:40.888 [info] wait for handoffs
19:46:40.888 [info] Wait until transfers complete '[email protected]'
19:50:01.381 [info] Wait until transfers complete '[email protected]'
19:50:01.397 [info] final get
19:50:01.398 [error] Error in process <0.158.0> on node '[email protected]' with exit value: {{assertEqual_failed,[{module,kv679_dataloss_fb},{line,179},{expression,"riakc_obj : get_values ( O )"},{expected,[<<4 bytes>>,<<3 bytes>>]},{value,[<<3 bytes>>]}]},[{kv679_dataloss_fb,'-confirm/0-fun-16-',2,[{file,"tests/kv679_dataloss_fb.erl"},{line,179}]},...


19:50:01.398 [warning] kv679_dataloss_fb failed: {{assertEqual_failed,[{module,kv679_dataloss_fb},{line,179},{expression,"riakc_obj : get_values ( O )"},{expected,[<<"anne">>,<<"joe">>]},{value,[<<"joe">>]}]},[{kv679_dataloss_fb,'-confirm/0-fun-16-',2,[{file,"tests/kv679_dataloss_fb.erl"},{line,179}]},{kv679_dataloss_fb,confirm,0,[{file,"tests/kv679_dataloss_fb.erl"},{line,179}]},{riak_test_runner,return_to_exit,3,[{file,"src/riak_test_runner.erl"},{line,159}]}]}
19:50:01.399 [error]
================ kv679_dataloss_fb failure stack trace =====================
{{assertEqual_failed,[{module,kv679_dataloss_fb},
                      {line,179},
                      {expression,"riakc_obj : get_values ( O )"},
                      {expected,[<<"anne">>,<<"joe">>]},
                      {value,[<<"joe">>]}]},
 [{kv679_dataloss_fb,'-confirm/0-fun-16-',2,
                     [{file,"tests/kv679_dataloss_fb.erl"},{line,179}]},
  {kv679_dataloss_fb,confirm,0,
                     [{file,"tests/kv679_dataloss_fb.erl"},{line,179}]},
  {riak_test_runner,return_to_exit,3,
                    [{file,"src/riak_test_runner.erl"},{line,159}]}]}
============================================================================

19:50:01.399 [notice] kv679_dataloss_fb Test Run Complete
19:50:01.401 [info] Stopping cover

Test Results:
kv679_dataloss_fb-bitcask: fail
---------------------------------------------
1 Tests Failed
0 Tests Passed
That's 0.0% for those keeping score

andrewjstone · 2015-01-15T03:47:41Z

tests/kv679_tombstone.erl

I agree that you don't need the andalso contains_fallback(NewPL). In this case the comment should be removed and so should contains_fallback/1.

russelldb · 2015-01-15T09:20:59Z

WRT the failing kv679_dataloss_fb that scenario needs the dirty_clock stuff we discussed that is @TODO'd in riak_kv_vnode. Should I remove that test from this PR, and re-submit it when we finally fix that case?

Remove those timers and other remnants of flailing when first trying to write tests.

russelldb · 2015-01-15T16:35:00Z

Hopefully that does it. Let me know if about removing kv679_dataloss_fb until we fix that.

andrewjstone · 2015-01-15T17:18:30Z

@russelldb Great job here! I would remove kv679_dataloss_fb if it's not going to pass. Just put it in another PR for when we add the dirty clock support to the code.

+1 after that.

andrewjstone · 2015-01-15T17:55:00Z

👍

Riak Tests for scenarios of basho/riak_kv#679

russelldb added 9 commits January 2, 2015 09:02

WIP kv679 tombstone-doomstone-boomstone test

aece28a

WIP remove voodoo code, add inactivity timeout

40882ea

WIP make the test fail

4f7e9e1

WIP kv679 doomstone tests

a29d3c3

WIP kv679 tombstone test A-B-C, crash before reap

6097bda

WIP

87afd49

WIP Kv679 tests

8cbec21

Dataloss at coordinator Dataloss at coordinator with an old clock repaired Vnode id's not unique

WIP data loss test was failing for the wrong reason

a1af114

add some debugging log out

WIP

d86d122

russelldb force-pushed the rdb/gh-kv679 branch from 7c36749 to d86d122 Compare January 2, 2015 09:28

andrewjstone reviewed Jan 14, 2015
View reviewed changes

andrewjstone reviewed Jan 15, 2015
View reviewed changes

Address review comments

3657c57

Remove those timers and other remnants of flailing when first trying to write tests.

remove test for unfixed case

ccb5af7

russelldb added a commit that referenced this pull request Jan 16, 2015

Merge pull request #719 from basho/rdb/gh-kv679

c1b12f5

Riak Tests for scenarios of basho/riak_kv#679

russelldb merged commit c1b12f5 into master Jan 16, 2015

seancribbs deleted the rdb/gh-kv679 branch May 6, 2015 16:10

russelldb mentioned this pull request Mar 21, 2017

Re-add old kv679 test and add new test for further dataloss edge case #1296

Merged

Riak Tests for scenarios of basho/riak_kv#679 #719

Riak Tests for scenarios of basho/riak_kv#679 #719

Uh oh!

Conversation

russelldb commented Dec 30, 2014

Uh oh!

seancribbs commented Dec 31, 2014

Uh oh!

russelldb commented Jan 2, 2015

Uh oh!

russelldb commented Jan 2, 2015

Uh oh!

seancribbs commented Jan 2, 2015

Uh oh!

russelldb commented Jan 2, 2015

Uh oh!

andrewjstone Jan 14, 2015

Choose a reason for hiding this comment

Uh oh!

russelldb Jan 15, 2015

Choose a reason for hiding this comment

Uh oh!

andrewjstone commented Jan 15, 2015

Uh oh!

andrewjstone Jan 15, 2015

Choose a reason for hiding this comment

Uh oh!

andrewjstone commented Jan 15, 2015

Uh oh!

andrewjstone Jan 15, 2015

Choose a reason for hiding this comment

Uh oh!

russelldb commented Jan 15, 2015

Uh oh!

russelldb commented Jan 15, 2015

Uh oh!

andrewjstone commented Jan 15, 2015

Uh oh!

andrewjstone commented Jan 15, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants