Implement hardcoded sync in the light client by tomaka · Pull Request #8075 · openethereum/parity-ethereum

tomaka · 2018-03-08T15:30:36Z

Adds a hardcodedSync field to the chain specifications.
When set, contains the RLP of a header, a total difficulty, and a list of CHT root hashes. If there is no light client database, the light client will immediately jump to the given block.

After this change, the light client synchronizes in 10 to 15 seconds even at the first launch on the main network.

For now only the specs of the main network have been updated.

parity-cla-bot · 2018-03-08T15:30:39Z

It looks like @tomaka signed our Contributor License Agreement. 👍

Many thanks,

Parity Technologies CLA Bot

rphmeier · 2018-03-08T16:08:37Z

+					let cht_num = cht::block_to_cht_number(decoded_header.number() - 1)
+						.expect("specs provided a hardcoded block with height 0");
+					if cht_num >= hardcoded_sync.chts.len() as u64 {
+						panic!("specs didn't provide enough CHT roots for its hardcoded block");


rule for panics of any kind is "prove or remove" -- We may rather want to print a warning and fall back to hardcoded-less sync. Or move these checks to the deserialization phase where we can exit with an error more gracefully.

rphmeier · 2018-03-08T16:09:13Z

 	///
 	/// If the block is an epoch transition, provide the transition along with
 	/// the header.
+	#[inline]


does that actually help? generally I figure LLVM will figure it out.

rphmeier · 2018-03-08T16:10:28Z

+					let header = if let Some(header) = self.block_header(BlockId::Number(h_num)) {
+						header
+					} else {
+						panic!("Header of block #{} not found in DB", h_num);


same: do not panic.

rphmeier · 2018-03-08T16:10:44Z

+
+					let decoded = header.decode();
+
+					let entry: Entry = ::rlp::decode(&self.db.get(self.col, era_key(h_num).as_bytes()).unwrap().unwrap());


and we really never use unwrap.

rphmeier · 2018-03-08T16:12:15Z

+					let entry: Entry = ::rlp::decode(&self.db.get(self.col, era_key(h_num).as_bytes()).unwrap().unwrap());
+					let total_difficulty = entry.candidates.iter()
+						.find(|c| c.hash == decoded.hash())
+						.expect("no candidate matching block found in DB")


should rather prove based on the fact that an inconsistent but correctly-loaded database shouldn't be possible.

rphmeier · 2018-03-08T16:13:50Z

+			chts.push(cht);
+		}
+
+		unreachable!("we are after an infinite loop")


could be a loop with a break-value, either way it could follow the codebase style of proposition; proof; qed

rphmeier · 2018-03-08T16:16:26Z

 	let client = service.client();
+	// Note: uncomment this code if you want to regenerate the JSON of the hardcoded sync feature
+	// in the chain specifications.
+	/*if let Some(hs) = client.read_hardcoded_sync() {


we could make this a subcommand of some kind

generally not a fan of the commented-out code. the CLI is a PITA to work with though :\

rphmeier · 2018-03-08T16:17:05Z

needs an opt-out flag also (off by default is fine, just that it should be there).

rphmeier · 2018-03-08T16:22:15Z

looks good other than grumbles

tomaka · 2018-03-08T16:23:30Z

Should I make read_hardcoded_sync() return None in case of error? Or rather Result<Option<...>, ...>? I admit that I didn't really know what to do with it, it's an internal-ish method (which is also why I totally overlooked its quality :-/).

rphmeier · 2018-03-08T16:25:55Z

Result<Option<_>> would be better IMO

tomaka · 2018-03-09T10:29:02Z

Sorry for the bad PR quality. I think it should be good now!

rphmeier · 2018-03-20T05:20:54Z

needs merge, can you ping me for review? I'm not looking that closely at the Parity repo these days.

tomaka · 2018-03-20T10:28:03Z

@rphmeier Conflicts solved!

rphmeier · 2018-03-21T02:12:21Z

+	let on_demand = Arc::new(::light::on_demand::OnDemand::new(cache.clone()));
+
+	let sync_handle = Arc::new(RwLock::new(Weak::new()));
+	let fetch = ::light_helpers::EpochFetch {


on_demand, sync, network etc. are not necessary to start here -- a NoOpEpochFetch (I think it already exists for tests) will be enough.

rphmeier · 2018-03-21T02:15:06Z

LGTM except for grumble

tomaka · 2018-03-21T10:37:48Z

r?

5chdn · 2018-03-26T13:35:29Z

+		"header": "f90219a061d694007fbaca6e23e73e29c8c6a60099abc740ab7e27ae3cd1b9c6a47efef7a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347945a0b54d5dc17e0aadc383d2db43b0a0d3e029c4ca0a2f1bdabc1a72737de1c22a76cacc8fc162366904f759a99db7d6d19efee3090a0ac5f5b236e8977928a2ce43c7569ea5a74919643cb0b06d7540407b1ea1298f0a04356ddc5d77c83923a6541260308be167386e0969a608a017770c9e38091cfcab90100a00010002001009080011010141088000004000080081100000a002023000002204204801204084000c000010008000000000880080020c0000440200460000290005010c01c80800080004800100406003380000400402040000028084002a80087000008090a00200100544020019580022000000306100a0080100084020006809000e80000010000254810002000000a240050014200002002c10809202030006422022000203012000241089300080400000009001021020200012410348500028290230408100302000000058c0000020c08c20480081040020260004008481000080000800010010060020000e00020002140100a8988000004400201870b9af4a66df8038350a8018379d54483799eba845ab0426d984554482e45544846414e532e4f52472d3231323134313232a05eeccc80302d8fecca48a47be03654b5a41b5e5f296f271f910ebae631124f518890074810024c6c2b",
+		"totalDifficulty": "3144406426008470895785",
+		"CHTs": [
+			"0x0eb474b7721727204978e92e27d31cddff56471911e424a4c8271c35f9c982cc",


Is this anyhow possible without adding 2500 CHTs to the chain spec?

Hmmm, what would the alternative be? We're hardcoding headers in the client after all, we have to put them somewhere.

the alternative is to allow the client to be deployed w/o history or CHT roots and have the history fetched afterwards

tomaka force-pushed the hardcoded-sync branch from 52b690b to a3fc5b4 Compare March 8, 2018 15:34

rphmeier added the A0-pleasereview 🤓 Pull request needs code review. label Mar 8, 2018

Implement hardcoded sync

9e32988

tomaka force-pushed the hardcoded-sync branch from eb9d6a3 to 9e32988 Compare March 8, 2018 16:07

tomaka added the M4-core ⛓ Core client code / Rust. label Mar 8, 2018

rphmeier reviewed Mar 8, 2018

View reviewed changes

rphmeier added A4-gotissues 💥 Pull request is reviewed and has significant issues which must be addressed. and removed A0-pleasereview 🤓 Pull request needs code review. labels Mar 8, 2018

tomaka added 3 commits March 8, 2018 18:33

Fix concerns

6490333

Remove artifact

a3d6830

Fix cli tests

c4c4a61

svyatonik added A0-pleasereview 🤓 Pull request needs code review. and removed A4-gotissues 💥 Pull request is reviewed and has significant issues which must be addressed. labels Mar 9, 2018

rphmeier mentioned this pull request Mar 9, 2018

Get the light client to compile for iOS #7955

Closed

Merge remote-tracking branch 'upstream/master' into hardcoded-sync

ff483c0

Tbaut mentioned this pull request Mar 20, 2018

--light client stuck at #2088992 #8148

Closed

Fix compilation

06b84dc

tomaka force-pushed the hardcoded-sync branch from a550069 to 06b84dc Compare March 20, 2018 10:51

Update hardcoded sync block

679eae3

5chdn added this to the 1.11 milestone Mar 20, 2018

rphmeier reviewed Mar 21, 2018

View reviewed changes

Don't use any data fetch for the light service

bc7fc65

tomaka mentioned this pull request Mar 22, 2018

Allow user to reset light client to a trusted block hash #7263

Closed

5chdn mentioned this pull request Mar 26, 2018

Parity warp sync is no longer very warpy. #6372

Closed

5chdn reviewed Mar 26, 2018

View reviewed changes

rphmeier approved these changes Mar 27, 2018

View reviewed changes

tomaka merged commit 0493161 into openethereum:master Mar 27, 2018

tomaka deleted the hardcoded-sync branch March 27, 2018 11:57

5chdn mentioned this pull request Apr 25, 2018

Light clients broken on chains with genesis validator-set contract #8468

Closed

5chdn mentioned this pull request Jun 20, 2018

Update hardcoded headers #8925

Merged

5chdn mentioned this pull request Jul 12, 2018

Update light client hardcoded headers #9098

Merged


		let decoded = header.decode();

		let entry: Entry = ::rlp::decode(&self.db.get(self.col, era_key(h_num).as_bytes()).unwrap().unwrap());

Conversation

tomaka commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

parity-cla-bot commented Mar 8, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier commented Mar 8, 2018

Uh oh!

rphmeier commented Mar 8, 2018

Uh oh!

tomaka commented Mar 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rphmeier commented Mar 8, 2018

Uh oh!

tomaka commented Mar 9, 2018

Uh oh!

rphmeier commented Mar 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tomaka commented Mar 20, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rphmeier commented Mar 21, 2018

Uh oh!

tomaka commented Mar 21, 2018

Uh oh!

5chdn Mar 26, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tomaka commented Mar 8, 2018 •

edited

Loading

rphmeier Mar 8, 2018 •

edited

Loading

tomaka commented Mar 8, 2018 •

edited

Loading

rphmeier commented Mar 20, 2018 •

edited

Loading

5chdn Mar 26, 2018 •

edited

Loading