Does it fit? Yes.
Llama-3.3-70B at BF16 is about 140 GB; a 4-bit quantization (e.g. Q4_K_M / AWQ / NVFP4) lands around 35–43 GB. NVIDIA's Jetson Thor module — the T5000, part of the AGX Thor Developer Kit — ships with 128 GB of LPDDR5X and a Blackwell GPU with FP4 support (up to ~2070 sparse FP4 TFLOPS). A ~40 GB weight set leaves ample headroom for KV cache and the rest of the system. So the model fits — that part is real and unremarkable.
How fast? Bandwidth decides.
The honest constraint is memory bandwidth, not compute. Autoregressive decoding reads (most of) the weights once per generated token, so single-stream throughput is bounded by memory bandwidth ÷ active weight bytes. Thor's memory bandwidth is 273 GB/s. Against a ~40 GB 4-bit weight set that's a ceiling of roughly 273 / 40 ≈ 7 tokens/sec before overheads — and real-world will be a bit under that. You can push higher with speculative decoding, smaller draft models, MoE sparsity (not applicable to a dense 70B), or batching multiple requests — but for a single interactive stream, single-digit tok/s is the regime. Any "tens of tokens/sec for a dense 70B on a 273 GB/s device, single stream" claim should be treated skeptically; the bandwidth math doesn't support it.
That's still useful. For onboard reasoning where the alternative is a cloud round-trip (or no model at all), a few tokens per second of a real 70B — air-gapped, on the vehicle — is the capability that matters.
Specs & sources
- Model card — huggingface.co/meta-llama/Llama-3.3-70B-Instruct
- Jetson Thor — developer.nvidia.com — Introducing Jetson Thor
- Registry tier & provenance — live artifact page