Initialization for Log Alpha

[hxu105]

Howdy, thank you for sharing this amazing work. I have some questions about the log alpha initialization. For both Llama and GPT2, you have initialized the log alphas with N(10, 0.01). Is there any specific reason for choosing this sharp Gaussian distribution? What will be the impact if I use standard normals or other initializations?