architectures · poolside

Laguna

Sparse mixture-of-experts decoder from poolside, declared as model_type "laguna" (LagunaForCausalLM). Routes each token to 10 of 256 experts in the S build and 8 of 256 in XS, alongside a shared expert that every token passes through. Attention alternates a repeating full_attention / sliding_attention pattern with a 512-token sliding window, so most layers attend locally and only the periodic full layers carry global context — which is how the S build reaches a 1,048,576-token position limit at 48 layers and 3072 hidden size with 8 KV heads against 48 query heads. Grouped-query attention at head_dim 128, bfloat16, untied embeddings over a 100,352-token vocabulary. Because the expert count dominates the parameter total while only a fraction activates per token, the total figure the registry records is not the figure that determines decode throughput.