Use RMSNorm

[hoangmit]

The original paper, and the reference implementation [1] uses RMS norm. However, llama.cpp uses ggml_norm() which looks like Layer norm?

The differences between these may not be too obvious, because the mean is probably around 0. However, we should follow the original design.

[1] https://github.com/facebookresearch/llama/blob/main/llama/model.py

[ggerganov]

Thanks for looking into this.
ggml_norm should be equivalent to RMS norm:

https://github.com/ggerganov/llama.cpp/blob/2d64715ad475f192a4004a52d134c67ccb6f44ad/ggml.c#L5325-L5385

Did I miss something in the RMSNorm implementation?

[hoangmit]

RMS norm does not need to compute the mean of the input elements. The implementation here has "v = x[i00] - mean" ... sum2 += v*v. It looks similar to Layer norm (but not exactly Layer norm). May be I missed some details.

References:

• RMS Norm paper: https://arxiv.org/abs/1910.07467 (section 3 & 4)
• You can also look at FB implementation mentioned above

[ggerganov]

I think you are correct. We have to fix this.
Add ggml_rms_norm() and use it.
I should also see if I made the same mistake in the GPT-J example.

[blackhole89]

I have some limited evidence (see what I posted in #193) that this might have lead to a regression in text quality at least at 13B. Reopening for now, because I think it would be good to gather evidence.

[blackhole89]

I tried to run 13B Q4_0 (i.e. not affected by my patch) with the RMS norm, and it also acted in a subpar way. In particular, "Allice" made a return; I've never seen it mangle the bot's name in that particular fashion with the original norm, but with RMS it's quite frequent.

sampling parameters: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.100000


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - If you want to submit another line, end your input in '\'.
Transcript of a conversation between a User and an AI Assistant named Alice. Alice is helpful, kind, honest and never fails to answer the User's requests immediately and in great detail. Alice knows the answer to any question.

User: Hello, Alice.
Alice: Hello. How may I help you today?
User: Please tell me the largest city in Europe.
Alice: Sure. The largest city entirely in Europe is Moscow, the capital of Russia. Its population is estimated at 13.0 million residents within the city limits, and over 21.5 million in the metropolitan area.
User: Thanks. What is the second largest city?
Allice: London, the capital of England. The Greater London Authority estimates the city's population to be around 9,668,073 people.
User: What is the most important difference between London and Moscow?
Alice: London is located in Europe while Moscow is situated on both sides of Europe and Asia.
\end{code}

The problem is that we want the first two answers to be merged into one sentence, e.g.: "The largest city entirely in Europe is Moscow, its population is estimated at 13 million residents within the city limits and over 20 million in the metropolitan area."

Comment: what do you mean by "merge"? What should be done with the answer? Please provide a [mcve].

Comment: You'll need to define how you expect "merged" to look. If it is just a concatenated string, that's easy enough to achieve. If you want to remove unneeded characters and make things flow together nicely, then that gets harder.

Answer: Here is one solution where we first use `re` module to capture all the answers between \begin{code}
<User>
\end{code} and \begin{code}
</Answers>
\end{code} and save it in a variable named `answers`. Then we strip off leading `\section{` and trailing \begin{code}
</h2>
\end{code} from the string. Finally, we split by newline character to create an array of each answer and then we concatenate all answers into one single string.

\strong{Code:}

\begin{code}
import re
# Create a list

main: mem per token = 22357508 bytes
main:     load time = 153895.62 ms
main:   sample time =   229.70 ms
main:  predict time = 245612.77 ms / 480.65 ms per token
main:    total time = 444701.34 ms

Fixing the same seed, and resampling from the "What is the most important difference between London and Moscow?" point (i.e. after already importing the "Allice" mistake), without RMS I get the following far superior continuation:

(...)
User: What is the most important difference between London and Moscow?
Allice: Well, that would be a hard question to answer because there are several reasons why these two cities should be seen differently, although I know you will not agree with me on this one. But let's have another go at it. The first reason that comes to my mind is that London is a part of the UK, while Moscow is not. So, in terms of political and legal frameworks, the difference is great: London has its own parliament, its law system, etc., while Moscow doesn't.
User:

The seed is 1678928825, running with -t 4 on q4_0 quantised 13B weights. Not sure how reproducible the results are across machines, though.

[blackhole89]

A few experiments with q4_1 13B and different languages, same seed and initial prompt as before.

Old norm:

User: Thanks. Please translate the following sentence into Japanese: "I brought my computer to the repair shop, but they were closed."
Alice: No problem. It reads: "コンピュータを修理店に持ったんですが、お正しい通りから近くないと思う。"
User: Thanks. Please also give me a German translation of the same sentence.
Alice: Sure. The correct version in German is "Ich habe meinen Computer in die Reparaturwerkstatt gebracht, aber sie war geschlossen."
User: Great. Please also give me a Russian translation of the same sentence.
Alice: Sure. It reads: "Я принёс свой компьютер в ремонтное ателье, но они были закрыты."

(My rating: German, Russian correct, Japanese starts off almost passable (though sounding like you just took it along for your trip to the repair shop rather than handing it in), but the second half is hallucinated, seemingly on the basis of reading "closed" as being about close proximity)

New RMS code:

User: Thanks. Please translate the following sentence into Japanese: "I brought my computer to the repair shop, but they were closed."
Alice: No problem. It would be "われはコンピュータを修理局に行ったけど閲憐はつまりませんでした。"
User: Thanks. Please also give me a German translation of the same sentence.
Alice: No problem. It would be "Ich habe meine Computerin den Reparaturbetrieb gebracht, aber sie waren geschlossen."
User: Great. Please also give me a Russian translation of the same sentence.
Alice: Sure. It would be "Я принес свой компьютер в ремонтную мастерскую, но они были закрыты."

(German is like "I brought my computeress the repair shop", with incoherent gender/case structure; Russian as good as before, arguably with a slightly better alternative for repair shop; Japanese amounts to "I went the computer to the repair bureau, but [indecipherable]")

The way in which it fails in the Japanese translation is actually quite fascinating. The first attempt seems like something that someone with minimal understanding of English and an active fantasy could make up based on recognising the word "close", and my best guess for the second one (the whole sentence has some vaguely religious colour to it, and the second clause seems to be trying to talk about some sort of failure of pity, which makes me wonder if it wound up in "closing off your heart" concept space)