Magic constants

[platers]

This is a very clean implementation, thanks for sharing it!

I have a few questions about some constants in the code. Why is the MLP init scaled by 0.5 instead of alpha?

for name, param in self.mlp.named_parameters():
      if "weight" in name:
            init.normal_(param, mean=0, std=0.5 * (1 / config.n_embd) ** 0.5)

And for the embeddings where does the 3.3 constant come from?

init.normal_(self.embed.weight, mean=0, std=alpha * 3.3)

[cloneofsimo]

Because in the paper, they did the sweeping for transformer and found that multiplicative scaling factor for embedding is good when its something like 3 ~ 10 times, so naturally, since we dont have multiplicative factor, scale the init and lr 3 ~ 10 times each. Kinda random hyperparameter tbh, and the sweeping results of muP paper doesnt how that much difference.

[cloneofsimo]

Same goes for the initialization. look how 0.5 seems good on the left chart. so I chose that, but if anything, sweep shows you it doesn't really matter that much

[platers]

got it thanks!