architectures · google

BERT

Bidirectional encoder-only transformer, declared as model_type "bert" (BertForMaskedLM). The base configuration is 12 layers at 768 hidden size with 12 attention heads and a 3072 intermediate size, over a 30,522-token WordPiece vocabulary, with GELU activations and a 512-token position limit. Attention is full and bidirectional at every layer — there is no causal mask — which is why it encodes text rather than generating it, and why the position limit is small by current standards without that being a defect. A type_vocab_size of 2 carries the segment embedding that the next-sentence-prediction objective needed. Everything else in this registry is a decoder; this is the encoder the retrieval and embedding entries descend from, and its throughput and memory figures are not comparable with a generative model of similar parameter count.