architectures · tencent

HunYuan V4

Sparse mixture-of-experts decoder from Tencent, declared as model_type "hy_v4" (HYV4ForCausalLM). Routes each token to 8 of 256 experts at a 2048 expert intermediate size, alongside one shared expert every token passes through, across 78 layers at 6144 hidden size. Attention is multi-head latent attention — a 512 kv_lora_rank and 2048 q_lora_rank compress the key/value and query projections — combined with DeepSeek-style sparse attention on every one of the 78 layers, which selects 2048 positions through a 32-head, 128-dim index rather than attending densely. That combination is how it reaches a 1,048,576-token position limit with 64 query heads against 8 KV heads at head_dim 64. Because both the expert count and the latent compression sit between the parameter total and what is actually resident per token, the total this registry records is not the figure that determines either KV-cache size or decode throughput.