Implement OLMoE architecture

[2015aroras]

Implements the recently released open-source OLMoE architecture. Tested with base model, will follow up with instruction tuned model testing. Fixes #9380.

Implementation differences from Qwen2Moe:

• no shared experts
• no bias
• QK norm
• different hyper-parameters: RMS layer norm eps, number of experts, number of active experts.

Test (base model):

// Download OLMoE weights to models/olmoe-hf
python3 convert_hf_to_gguf.py models/olmoe-hf
./llama-quantize models/olmoe-hf/Olmoe-Hf-64x577M-F16.gguf  models/olmoe-hf/Olmoe-Hf-64x577M-Q4_K_M.gguf Q4_K_M
./llama-cli -m models/olmoe-hf/Olmoe-Hf-64x577M-Q4_K_M.gguf  -p "Bitcoin is" -n 128

Output:

Bitcoin is a new currency that has been created by an unknown person or group of people under the name Satoshi Nakamoto. It is the first decentralized digital currency, and unlike traditional currencies, it is not backed by a government or central authority. This has led to a lot of debate about whether or not Bitcoin is a good investment. In this article, we will explore the pros and cons of Bitcoin and see if it is worth investing in.

What is Bitcoin?

Bitcoin is a digital currency that was created in 2009. It is the first decentralized cryptocurrency, meaning that it is not issued or controlled by a central authority. Bitcoin

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

[2015aroras]

Various gguf files uploaded to https://huggingface.co/allenai/OLMoE-1B-7B-0924-GGUF/tree/main and https://huggingface.co/allenai/OLMoE-1B-7B-0924-Instruct-GGUF/tree/main (Instruct seems to produce sensible text too). Anything else needed to get this merged?

[Meshwa428]

@ggerganov why isn't this merged yet?
Any problems?

Please merge it 🙏🏻
So that we ollama folks can use the model in ollama.