Fix BPE bonus materials

[rasbt]

Fixes issues with the BPE tokenizer to correctly handle edge cases.

Fixes #558

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[d-kleine]

@rasbt About the lowest possible merge rank defined as min_rank = 1_000_000_000, I am not sure if inf would be a better choice (just like in the original BPE implementation) to keep the code resilient. The value 1_000_000_000 might look very specific to users, although it's arbitrary. Also, with multi-modality and growing vocabulary sizes, adjacent token combinations of 1_000_000_000 might be reached at some point in the future, possible breaking the code then.

LLMs-from-scratch/ch02/05_bpe-from-scratch/bpe-from-scratch.ipynb

Lines 607 to 613 in f63f04d

	" min_rank = 1_000_000_000\n",
	" bigram = None\n",
	" for p in pairs:\n",
	" r = self.bpe_ranks.get(p, 1_000_000_000)\n",
	" if r < min_rank:\n",
	" min_rank = r\n",
	" bigram = p\n",

[rasbt]

yes, I agree!