architectures · ibm-granite

Granite MoE Hybrid (Mamba + Attention)

Hybrid state-space and attention stack: config declares layer_types of `mamba` and `attention`, mixing Mamba-style recurrent layers with periodic attention layers. The recurrent layers carry fixed-size state instead of a growing KV cache, so memory at long context is dominated by the far smaller number of attention layers. Grouped-query attention on those; shared feed-forward path alongside the sparse experts. Declared as model_type `granitemoehybrid`.