This repository provides an implementation of Fourier Number Embedding (FNE) a novel method that directly maps numbers into the embedding space with their Fourier features. FNE encodes each number as a single token with only two embedding dimensions per digit, effectively capturing numerical values without fragmentation.
(a) We extract all the numbers from the input sequence. (b) For each number, we use FNE to directly map the number to its embedding. The first two entries in the embedding represent 18 mod 10, while the next two entries represent 18 mod 100. (c) We pad the FNE with zeros, add it to the word embeddings, and then feed the combined embeddings into the model. (d) For each digit, we take every two entries from the last hidden state and find the number whose representation is closest to these two entries.
On 6-digit decimal addition, FNE requires 64x less data to achieve 99% accuracy than subword and digit-wise embeddings while using 3x and 6x fewer tokens per number, respectively. Furthermore, FNE is the only method that yields 100% accuracy on over 100,000 test examples for addition, subtraction, and multiplication.
For a visualization of our method, please refer to this website.