PositionEmbedding/TokenAndPositionEmbedding guide

[DavidLandup0]

Hey! I just wrote a quick guide to doing positional/token and position embedding with KerasNLP. I think that the library really lowers the barrier to entry for many, and allows people to build NLP models much quicker and easier than ever before. As far as I can tell, KerasNLP is the first major lib to actually let people build stuff from scratch, rather than providing prepackaged Transformers.

While there's a niche for both - you're doing amazing work :)
I'll be doing a series of guides for KerasCV and KerasNLP, and this is one of the starting ones. Once a larger library of guides is finished, I'll string them together and add links between them. Any feedback on this?

• https://stackabuse.com/transformer-token-and-position-embedding-with-keras/

@mattdangerw @chenmoneygithub

[mattdangerw]

@DavidLandup0 thank you! Very much appreciate the help building resources for KerasNLP!

This looks great to me, I like that you give the peak into what TokenAndPositionEmbedding call function looks like.

One thing you could mention, though I know it isn't the focus on the article, is what tools are available for that tokenize() step you describe. Might be worth briefly saying that Keras provides TextVectorization for simple tokenization use cases, and KerasNLP has a wide range of tokenizers covering various approaches.

[DavidLandup0]

Thanks for the reply, really glad you like it! Good catch. I'll probably add a small note on TextVectorization, since it's an extremely versatile and handy class. I assumed some prior knowledge for the topic, since presumably those interested in Token and Position embedding might be already clued in but it's better to note it than not :)

Thanks!

[DavidLandup0]

Also, posted another guide with an end-to-end example of loading, preprocessing and vectorizing data and training a GPT-style decoder-only transformer for text generation at #291