architectures · mistralai

Mistral (Dense)

Dense decoder-only transformer combining grouped-query attention with sliding-window attention, so each layer attends over a fixed window rather than the full sequence and the KV cache is bounded by the window instead of the context length. Distinct from the Mixtral sparse-MoE line, which shares the backbone but routes through expert feed-forward blocks. Declared as model_type `mistral` in config.json.