VLM Inference Benchmark Explorer
Alpamayo2-Edge on Jetson AGX Thor — BF16, vLLM vision-language inference benchmark
Open this configuration in the VLM Inference Benchmark Explorer
| Model | Alpamayo2-Edge |
|---|---|
| Parameters | 2.7B |
| Device | Jetson AGX Thor |
| Quantization | BF16 |
| Image size | 720p |
| Images per camera | 1 |
| Max Cameras (TTFT ≤ 3000 ms) | 128 |
| Engine | vLLM |
| Cameras | TTFT (ms) | TPS (tok/s) | Target |
|---|---|---|---|
| 1 | 136 | 101.2 | ≤3000 ms |
| 2 | 149 | 95.3 | ≤3000 ms |
| 4 | 162 | 74.6 | ≤3000 ms |
| 8 | 212 | 138.8 | ≤3000 ms |
| 16 | 355 | 48.4 | ≤3000 ms |
| 32 | 551 | 32.4 | ≤3000 ms |
| 64 | 948 | 36.5 | ≤3000 ms |
| 128 | 1895 | 17.0 | ≤3000 ms |
| 256 | 4248 | 16.1 | >3000 ms |
| 512 | 8647 | 18.7 | >3000 ms |
| 1024 | 17902 | 14.4 | >3000 ms |
| 1280 | 21003 | 12.3 | >3000 ms |
| 1408 | 23635 | 11.6 | >3000 ms |
| 1536 | — | — | Failed |
| Images per request | TTFT (ms) | TPS (tok/s) | Target |
|---|---|---|---|
| 1 | 136 | 101.2 | ≤3000 ms |
| 2 | 185 | 728.8 | ≤3000 ms |
| 4 | 247 | 297.8 | ≤3000 ms |
| 8 | 410 | 72.5 | ≤3000 ms |
| 16 | 792 | 69.1 | ≤3000 ms |
| 32 | 1485 | 58.5 | ≤3000 ms |
| 64 | 3307 | 51.2 | >3000 ms |
| 128 | 7200 | 45.8 | >3000 ms |
| 256 | 17168 | 35.0 | >3000 ms |
| 512 | 46217 | 148.7 | >3000 ms |
| 1024 | — | — | Failed |
Alpamayo2-Edge (2.7B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 136 ms (time to first token, TTFT), then writes 101.2 tok/s.
With 512 images per camera at 720p, the answer starts after 46,217 ms. With one 2K image, after 229 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 128 cameras at once.
Alpamayo2-Edge (2.7B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 136 ms (time to first token, TTFT), then writes 101.2 tok/s.
With 512 images per camera at 720p, the answer starts after 46,217 ms. With one 2K image, after 229 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 128 cameras at once.
Alpamayo2-Edge is a driving model: from six cameras (22 frames), the vehicle's recent motion and a navigation instruction it writes a short chain of causation and plans six candidate trajectories. The table above measures only its vision-language part, served with vLLM in BF16 with greedy decoding, the way the model answers questions. Its answers to "Describe the scene." are short (2 to 48 tokens on average per cell), so its tok/s figures swing more than other rows'. This test runs the whole model with its official FP8 TensorRT Edge-LLM recipe (TensorRT Edge-LLM v0.11.1, 5 diffusion steps, 6 candidates, greedy decoding of the chain of causation) on the recipe's 19 navigation demo clips, 3 runs each.
Jetson AGX Thor at MAXN with jetson_clocks and TensorRT 11.3; the recipe's own figures come from a DRIVE AGX Thor with TensorRT 11.3. As in the recipe, each decision encodes only the newest frame of every camera and reuses the earlier ones.
| Stage | Jetson AGX Thor | DRIVE AGX Thor (recipe) |
|---|---|---|
| Vision encoder, 6 newest frames | 21 | 20 |
| Prefill, 3,261 tokens | 67 | 75 |
| Chain of causation, per token | 7.8 | about 7 |
| Trajectory expert, 5 steps × 6 candidates | 70 | 70 |
| Whole decision (22 tokens) | 331 (p90 341) | about 320 |
| Setting | ms |
|---|---|
| 120W, TensorRT 10.16 (JetPack) | 439 |
| MAXN, TensorRT 10.16 (JetPack) | 423 |
| MAXN + jetson_clocks, TensorRT 11.3 | 331 |
With all 22 frames encoded inside the timed run a decision takes 402 ms. TensorRT 11.3 brought the trajectory expert from 99 to 70 ms; fixing the GPU and memory clocks with jetson_clocks (memory at 4266 MHz instead of about 2750) brought the rest.
The recipe gives about 7 ms per token for greedy decoding but no greedy total; about 320 ms is its stages plus 22 tokens × 7 ms. On the recipe's example clip the chain of causation matches the recipe's text word for word.