VLM Inference Benchmark Explorer
How many cameras can this device watch with this vision-language model? Pick the image size your cameras send and set how long a camera may wait for its answer; the table updates for every configuration we have measured. Expand any row to see its measurements for every number of cameras, the results explained and a chart.
How to use the benchmark explorer
What a row is
A row is one deployment configuration: a vision-language model, on specific hardware, in a specific number format, served by a specific engine, measured end to end. The same model therefore appears in several rows, and two rows are directly comparable only once you know which of those settings differ between them.
Filters and the target do different jobs
Filters decide which rows you see. The device, model, parameter-count, quantization and engine filters, and the TTFT slider, only show or hide rows.
The target and the assumption decide what the numbers say. They sit under Performance Target and Assumptions. Changing one leaves the rows in place but recalculates Max Cameras and the green and red colouring of every row.
1. Describe your cameras
Two selectors describe the workload, and every number in the table is read at them:
- Cameras — how many cameras send requests at the same moment. This is the concurrency the system was measured at.
- Image Size — the size of each image a camera sends: 480p, 720p, 1080p or 2K.
Rows not measured at the selected combination are hidden, and dimmed buttons mark combinations that have no measurement. Several cameras were measured at 480p, 720p and 1080p; 2K with one camera only.
2. Narrow and sort
Filters combine, and every column sorts. The most instructive comparisons change a single setting: one model at two quantizations on one device, the same model under llama.cpp and vLLM, or one model on two devices.
3. Set your target
The target is the longest a camera may wait for its answer to start — default 3000 ms. A mean TTFT exactly on the target passes.
Beside it, Images per Camera sets how many images each request carries: one snapshot by default, or several frames of the same camera sent together, as video is often sent to a vision-language model. Several images per request were measured with one camera only, so with more than one image Max Cameras cannot be higher than 1.
4. Read Max Cameras
Max Cameras is the highest measured number of cameras at which the mean TTFT of their requests is within your target. A plus — 16+ — means the configuration met the target even at the highest number it was tested with, so its real maximum was not reached. “>3000 ms” — your target — means not even a single camera is answered in time.
5. Open a row
Click a row to see:
- on the left, the measurements at your image size and image count for every number of cameras tested, marked “≤3000 ms” or “>3000 ms” against your target, and closed by the number of cameras that failed where the camera limit was measured, and below them the same for the number of images in one request, with one camera;
- on the right, the results explained in plain language, at the default settings;
- below, a chart of TTFT as cameras are added, one line per image size, with your target as a dashed line. It appears where several cameras were measured, so with one image per camera.
The chart downloads as a PNG for reports and presentations.
Where to start
“How many cameras can one Jetson AGX Orin 32GB watch?” Select the device, set the image size your cameras send, and read Max Cameras. To see only the configurations that keep up, select the number of cameras you need and set Max TTFT to your target.
“Is 1080p worth it?” Switch the image size and watch Max Cameras, or open a row: its chart has one line per image size.
“What do quantization and the engine change?” Hold the model and the device fixed and compare the rows that differ in that one setting: Qwen3-VL-8B-Instruct at Q8_0 and Q4_K_M on Jetson AGX Orin 32GB, or Qwen3-VL-4B-Instruct under llama.cpp and vLLM on RTX PRO 6000. Changing the number of cameras shows how the gap develops under load.
“We already own this hardware; what can we run on it?” Start with the device filter, sort the remaining models by size or by Max Cameras, and narrow with the TTFT or the number of cameras you need.
What the numbers mean and how they are calculated
How the numbers were measured
Each request carries one or more photos and the prompt Describe the scene., and asks for up to 128 tokens. The photos are sixteen fixed pictures, cycled through the requests and scaled to 854×480 (480p), 1280×720 (720p), 1920×1080 (1080p) and 2560×1440 (2K), sent as base64 images through the OpenAI-compatible chat API. Hugging Face checkpoints were served with vLLM and GGUF files with llama.cpp.
Every combination of image size, images per request and number of cameras is measured on its own, after a warm-up batch that is discarded. The figures are means of the requests that completed; a request that failed during the benchmark is left out of the mean and does not count against a configuration.
- Several cameras were measured with one image per request at 480p, 720p and 1080p: 1, 2 and 4 cameras on Jetson Orin NX 16GB (up to 8 for one configuration), up to 8 on Jetson AGX Orin 32GB and its Developer Kit, and up to 16 on RTX PRO 6000, Jetson AGX Thor and DGX Spark, with 8 requests per level on a Jetson Orin and 24 on the others, never more cameras’ requests in flight than the level. The Jetson Orin configurations served with llama.cpp b11690 were measured at 1, 2, 4 and 8 cameras with 24 requests per level.
- One camera was measured at all four sizes with one, three and five images per request (1, 2, 4, 8, 16 and 32 on Jetson AGX Thor and DGX Spark, and for the Jetson Orin configurations served with llama.cpp b11690), 5 requests each (3 for one configuration). 2K and several images per request were measured this way only.
- The camera limit was measured on Jetson AGX Thor and DGX Spark, at 720p with one image per camera: 16, 32, 64 and more cameras, two requests per camera, until a request had no answer started within 30 seconds; the range between the last number that passed and the first that failed was then halved until it was at most an eighth of the passing number (at least 4 cameras). The server was set to take every camera’s request at once. A request answered with no text did not count as failed.
- The image limit was measured on Jetson AGX Thor and DGX Spark, at 720p with one camera: 64, 128, 256, 512 and 1024 images per request, 3 requests each, until a request failed — too long for the model, or no answer within 600 s.
A token is the unit a model reads and writes, about three quarters of an English word. An image is read as tokens too, and a larger image becomes more of them.
TTFT — time to first token
TTFT (time to first token) is how long a camera waits until its answer starts, in milliseconds; the table shows the mean over the requests measured. For a vision-language model this is mostly reading the images, so it grows with their size and number, and with the number of cameras sharing the device. How much it grows with size depends on the model: some turn every image into a fixed number of tokens, others into more tokens the more pixels there are. Lower is better.
TPS (tokens per second) is how fast the answer is then written, per camera. It barely depends on the image, and it is shown for reference: Max Cameras is decided by the TTFT. A long answer adds its writing time on top — 128 tokens at 25 tokens per second take about five more seconds.
Max Cameras: the number of cameras is the concurrency
A camera sends its next request as soon as the previous one is answered, so it always has exactly one request in flight. The number of concurrent requests is therefore the number of cameras.
Max Cameras is the highest measured number of cameras at which the mean TTFT meets your target. Only measured numbers count, nothing is interpolated, and if none passes the cell shows your target, “>3000 ms”. It depends on your target and moves with it: at 720p with one image per camera, Cosmos3-Edge on Jetson AGX Orin 32GB keeps up with 2 cameras at a 1000 ms target and 8 — the highest number measured, so 8+ — at 3000 ms.
Max Images per Request reads the same way for the images one camera sends in a single request: the most that still meet your target. An opened row also gives the configuration’s limits, whatever the target — the camera and image limits described above: the most cameras, and the most images in one request, that still worked before a request failed, and the count that failed. Where no limit was searched it says how far the measurement went. On Jetson AGX Thor and DGX Spark Max Cameras goes past 16 too, up to the camera limit.
The columns that describe the setup
Parameters — the model’s total number of weights, vision encoder included, as published. For a mixture-of-experts model this is the total, not the part active for each token, because all of it is held in memory.
Quantization — the number format the weights are stored in. BF16 is full precision, FP8 uses 8 bits and NVFP4 4 bits. Q8_0, Q4_K_M and Q4_0 are GGUF formats for llama.cpp with about 8 and 4 bits per weight. Fewer bits means less memory and usually more speed, at some risk to quality.
Engine — the server software that loads the model and schedules requests. It affects speed as much as the hardware does. Qwen3-VL-4B-Instruct on RTX PRO 6000 at 720p starts answering a single camera after 247 ms under llama.cpp (Q8_0) and 164 ms under vLLM (BF16); at 16 cameras the gap is 1094 ms against 461 ms.
Device — Jetson Orin NX 16GB and Jetson AGX Orin 32GB are embedded modules whose CPU and GPU share one memory. Both modules also come in other memory sizes, so the name states the one measured. Jetson AGX Orin 32GB Developer Kit is the developer kit with its own 32 GB module, listed apart from the production module. Jetson AGX Thor (developer kit) and DGX Spark also share one memory between CPU and GPU, 128 GB each. RTX PRO 6000 Blackwell is a workstation GPU with 96 GB, measured in its 600 W Workstation and 300 W Max-Q editions.
What the numbers do not tell you
The table uses one fixed workload — sample photos, a short prompt, answers of up to 128 tokens — and mean values, so that configurations can be compared without a site survey. It is not a substitute for a test with your own cameras and prompts: other image content, longer prompts or answers, other engine settings and the slowest requests rather than the mean all change real capacity. The TTFT is the wait until the answer starts; an application that needs the whole answer waits for its writing time too, and a row measured with one camera only says nothing about how it behaves with several.
Cosmos-Reason1-7B (8.3B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 7,681 ms (time to first token, TTFT), then writes 12 tok/s.
With 5 images per camera at 720p, the answer starts after 37,755 ms. With one 1080p image, after 23,305 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Cosmos-Reason2-2B (2.4B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 2,020 ms (time to first token, TTFT), then writes 30.4 tok/s.
With 5 images per camera at 720p, the answer starts after 9,935 ms. With one 1080p image, after 5,448 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Cosmos-Reason2-8B (8.8B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 6,872 ms (time to first token, TTFT), then writes 11.6 tok/s.
With 5 images per camera at 720p, the answer starts after 34,194 ms. With one 1080p image, after 21,752 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
d1-3B (3.1B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 2,365 ms (time to first token, TTFT), then writes 20.8 tok/s.
With 32 images per camera at 720p, the answer starts after 76,330 ms. With one 2K image, after 6,961 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Gemma-4-E2B-it (5.1B parameters), served in Q4_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 1,379 ms (time to first token, TTFT), then writes 29.8 tok/s.
With 5 images per camera at 720p, the answer starts after 6,306 ms. With one 1080p image, after 3,430 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Gemma-4-E4B-it (8.0B parameters), served in Q4_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 1,856 ms (time to first token, TTFT), then writes 17.2 tok/s.
With 5 images per camera at 720p, the answer starts after 8,315 ms. With one 1080p image, after 4,451 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
GLM-4.6V-Flash (10B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 6,953 ms (time to first token, TTFT), then writes 9.7 tok/s.
With 16 images per camera at 720p, the answer starts after 124,356 ms. With one 2K image, after 39,024 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Granite-Vision-4.1-4B (4.0B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 4,358 ms (time to first token, TTFT), then writes 14.4 tok/s.
With 16 images per camera at 720p, the answer starts after 84,981 ms. With one 2K image, after 4,521 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
LensVLM-9B (9.4B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 7,200 ms (time to first token, TTFT), then writes 10.3 tok/s.
With 32 images per camera at 720p, the answer starts after 230,062 ms. With one 2K image, after 56,130 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
LFM2.5-VL-3B (3.1B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 2,353 ms (time to first token, TTFT), then writes 20.7 tok/s.
With 32 images per camera at 720p, the answer starts after 76,152 ms. With one 2K image, after 6,950 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
MiMo-V2.6-Distill-Qwen-9B (9.4B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 7,204 ms (time to first token, TTFT), then writes 10.2 tok/s.
With 32 images per camera at 720p, the answer starts after 230,328 ms. With one 2K image, after 56,161 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
MiniCPM-V-4.6 (1.3B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 1,943 ms (time to first token, TTFT), then writes 43 tok/s.
With 32 images per camera at 720p, the answer starts after 62,045 ms. With one 2K image, after 2,619 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
MiniCPM-V-4.7-1B (1.6B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 4,264 ms (time to first token, TTFT), then writes 42.7 tok/s.
With 16 images per camera at 720p, the answer starts after 69,995 ms. With one 2K image, after 5,616 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen-Drive-1.0-4B (4.5B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 3,354 ms (time to first token, TTFT), then writes 11.8 tok/s.
With 32 images per camera at 720p, the answer starts after 105,020 ms. With one 2K image, after 19,483 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3-VL-4B-Instruct (4.4B parameters), served in FP8 format with vLLM on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 1,668 ms (time to first token, TTFT), then writes 14.2 tok/s.
With 3 images per camera at 720p, the answer starts after 4,004 ms. With one 1080p image, after 4,345 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3-VL-4B-Instruct (4.4B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 3,037 ms (time to first token, TTFT), then writes 14.3 tok/s.
With 5 images per camera at 720p, the answer starts after 14,623 ms. With one 2K image, after 18,387 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3-VL-8B-Instruct (8.8B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 6,916 ms (time to first token, TTFT), then writes 11.4 tok/s.
With 5 images per camera at 720p, the answer starts after 34,431 ms. With one 1080p image, after 21,785 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3.5-0.8B (0.9B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 970 ms (time to first token, TTFT), then writes 43.2 tok/s.
With 32 images per camera at 720p, the answer starts after 31,751 ms. With one 2K image, after 6,497 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3.5-2B (2.3B parameters), served in Q8_0 format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 2,094 ms (time to first token, TTFT), then writes 25.5 tok/s.
With 32 images per camera at 720p, the answer starts after 66,468 ms. With one 2K image, after 15,226 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Qwen3.5-4B (4.7B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 3,715 ms (time to first token, TTFT), then writes 15.6 tok/s.
With 5 images per camera at 720p, the answer starts after 17,250 ms. With one 1080p image, after 8,480 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3.5-9B (9.7B parameters), served in Q4_K_M format with llama.cpp on Jetson Orin NX 16GB, starts answering a single camera that sends one 720p image after 7,216 ms (time to first token, TTFT), then writes 10.3 tok/s.
With 32 images per camera at 720p, the answer starts after 231,047 ms. With one 2K image, after 56,353 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Cosmos-Reason1-7B (8.3B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 3,211 ms (time to first token, TTFT), then writes 26.7 tok/s.
With 5 images per camera at 720p, the answer starts after 15,842 ms. With one 2K image, after 18,623 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Cosmos-Reason2-2B (2.4B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 952 ms (time to first token, TTFT), then writes 57.8 tok/s.
With 5 images per camera at 720p, the answer starts after 4,449 ms. With one 2K image, after 7,073 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
Cosmos-Reason2-8B (8.8B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,710 ms (time to first token, TTFT), then writes 25.4 tok/s.
With 5 images per camera at 720p, the answer starts after 13,211 ms. With one 2K image, after 20,513 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Cosmos3-Edge (3.9B parameters), served in BF16 format with vLLM on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 510 ms (time to first token, TTFT), then writes 45.6 tok/s.
With 5 images per camera at 720p, the answer starts after 2,106 ms. With one 2K image, after 2,713 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 8 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
d1-3B (3.1B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,087 ms (time to first token, TTFT), then writes 39.9 tok/s.
With 32 images per camera at 720p, the answer starts after 32,783 ms. With one 2K image, after 3,110 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
Gemma-4-26B-A4B-it (26B parameters), served in Q4_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,341 ms (time to first token, TTFT), then writes 32.7 tok/s.
With 5 images per camera at 720p, the answer starts after 10,783 ms. With one 2K image, after 9,941 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Gemma-4-E4B-it (8.0B parameters), served in Q4_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 864 ms (time to first token, TTFT), then writes 34.2 tok/s.
With 5 images per camera at 720p, the answer starts after 3,582 ms. With one 2K image, after 2,551 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
GLM-4.6V-Flash (10B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,933 ms (time to first token, TTFT), then writes 16.1 tok/s.
With 16 images per camera at 720p, the answer starts after 50,586 ms. With one 2K image, after 15,550 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Granite-Vision-4.1-4B (4.0B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,869 ms (time to first token, TTFT), then writes 28.8 tok/s.
With 16 images per camera at 720p, the answer starts after 34,306 ms. With one 2K image, after 2,043 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
LensVLM-9B (9.4B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,733 ms (time to first token, TTFT), then writes 17.5 tok/s.
With 32 images per camera at 720p, the answer starts after 83,447 ms. With one 2K image, after 20,206 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
LFM2.5-VL-3B (3.1B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,081 ms (time to first token, TTFT), then writes 39.7 tok/s.
With 32 images per camera at 720p, the answer starts after 32,663 ms. With one 2K image, after 3,106 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
MiMo-V2.6-Distill-Qwen-9B (9.4B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,804 ms (time to first token, TTFT), then writes 17.5 tok/s.
With 32 images per camera at 720p, the answer starts after 85,598 ms. With one 2K image, after 20,413 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
MiniCPM-V-4.6 (1.3B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,116 ms (time to first token, TTFT), then writes 73.2 tok/s.
With 32 images per camera at 720p, the answer starts after 33,197 ms. With one 2K image, after 1,540 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
MiniCPM-V-4.7-1B (1.6B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,922 ms (time to first token, TTFT), then writes 76.8 tok/s.
With 16 images per camera at 720p, the answer starts after 30,405 ms. With one 2K image, after 2,587 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
MiniCPM-V-4.7-35B-A3B (35B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 3,546 ms (time to first token, TTFT), then writes 34.8 tok/s.
With 16 images per camera at 720p, the answer starts after 56,672 ms. With one 2K image, after 4,844 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen-Drive-1.0-4B (4.5B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,488 ms (time to first token, TTFT), then writes 24 tok/s.
With 32 images per camera at 720p, the answer starts after 42,941 ms. With one 2K image, after 8,678 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3-VL-4B-Instruct (4.4B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,313 ms (time to first token, TTFT), then writes 28.4 tok/s.
With 5 images per camera at 720p, the answer starts after 6,400 ms. With one 2K image, after 8,315 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3-VL-8B-Instruct (8.8B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,746 ms (time to first token, TTFT), then writes 25.2 tok/s.
With 5 images per camera at 720p, the answer starts after 13,242 ms. With one 2K image, after 20,613 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Qwen3-VL-8B-Instruct (8.8B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,636 ms (time to first token, TTFT), then writes 19.1 tok/s.
With 5 images per camera at 720p, the answer starts after 13,189 ms. With one 2K image, after 20,481 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Qwen3.5-0.8B (0.9B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 530 ms (time to first token, TTFT), then writes 76.9 tok/s.
With 32 images per camera at 720p, the answer starts after 14,854 ms. With one 2K image, after 3,064 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
Qwen3.5-27B (27B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 6,430 ms (time to first token, TTFT), then writes 6.8 tok/s.
With 32 images per camera at 720p, the answer starts after 203,226 ms. With one 1080p image, after 15,939 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3.5-2B (2.3B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 950 ms (time to first token, TTFT), then writes 47.6 tok/s.
With 32 images per camera at 720p, the answer starts after 28,006 ms. With one 2K image, after 6,986 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
Qwen3.5-35B-A3B (35B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,809 ms (time to first token, TTFT), then writes 30.5 tok/s.
With 32 images per camera at 720p, the answer starts after 86,682 ms. With one 1080p image, after 7,931 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Qwen3.5-4B (4.7B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 1,568 ms (time to first token, TTFT), then writes 33.6 tok/s.
With 5 images per camera at 720p, the answer starts after 7,143 ms. With one 2K image, after 9,038 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3.5-9B (9.7B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 2,751 ms (time to first token, TTFT), then writes 17.6 tok/s.
With 32 images per camera at 720p, the answer starts after 83,972 ms. With one 2K image, after 20,194 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Qwen3.6-27B (27B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB Developer Kit, starts answering a single camera that sends one 720p image after 5,684 ms (time to first token, TTFT), then writes 7.3 tok/s.
With 32 images per camera at 720p, the answer starts after 184,288 ms. With one 1080p image, after 14,110 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Cosmos3-Edge (3.9B parameters), served in BF16 format with vLLM on RTX PRO 6000 Max-Q, starts answering a single camera that sends one 720p image after 88 ms (time to first token, TTFT), then writes 310.3 tok/s.
With 5 images per camera at 720p, the answer starts after 343 ms. With one 2K image, after 432 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Cosmos3-Nano (16B parameters), served in BF16 format with vLLM on RTX PRO 6000 Max-Q, starts answering a single camera that sends one 720p image after 136 ms (time to first token, TTFT), then writes 88 tok/s.
With 5 images per camera at 720p, the answer starts after 739 ms. With one 2K image, after 746 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Cosmos-Reason1-7B (8.3B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 252 ms (time to first token, TTFT), then writes 98.2 tok/s.
With 5 images per camera at 720p, the answer starts after 708 ms. With one 2K image, after 613 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Cosmos-Reason2-2B (2.4B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 109 ms (time to first token, TTFT), then writes 318.3 tok/s.
With 5 images per camera at 720p, the answer starts after 562 ms. With one 2K image, after 495 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Cosmos-Reason2-8B (8.8B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 148 ms (time to first token, TTFT), then writes 92.5 tok/s.
With 5 images per camera at 720p, the answer starts after 750 ms. With one 2K image, after 680 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Eagle2.5-8B (8.1B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 258 ms (time to first token, TTFT), then writes 105 tok/s.
With 5 images per camera at 720p, the answer starts after 1,352 ms. With one 2K image, after 418 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Gemma-4-31B-it (31B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 235 ms (time to first token, TTFT), then writes 24.7 tok/s.
With 5 images per camera at 720p, the answer starts after 702 ms. With one 2K image, after 316 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Gemma-4-31B-it (31B parameters), served in NVFP4 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 271 ms (time to first token, TTFT), then writes 45.1 tok/s.
With 5 images per camera at 720p, the answer starts after 579 ms. With one 2K image, after 305 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Llama-3.1-Nemotron-Nano-VL-8B (8.7B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 292 ms (time to first token, TTFT), then writes 94.9 tok/s.
With 5 images per camera at 720p, the answer starts after 1,786 ms. With one 2K image, after 486 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
MedGemma-1.5-4B-it (4.3B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 199 ms (time to first token, TTFT), then writes 171.5 tok/s.
With 5 images per camera at 720p, the answer starts after 447 ms. With one 2K image, after 249 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Nemotron-Nano-12B-v2-VL (13B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 197 ms (time to first token, TTFT), then writes 68.5 tok/s.
With 5 images per camera at 720p, the answer starts after 777 ms. With one 2K image, after 612 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3-VL-30B-A3B-Instruct (30B parameters), served in FP8 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 161 ms (time to first token, TTFT), then writes 194.4 tok/s.
With 5 images per camera at 720p, the answer starts after 1,041 ms. With one 2K image, after 581 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3-VL-4B-Instruct (4.4B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 164 ms (time to first token, TTFT), then writes 156.6 tok/s.
With 5 images per camera at 720p, the answer starts after 640 ms. With one 2K image, after 559 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3-VL-4B-Instruct (4.4B parameters), served in Q8_0 format with llama.cpp on RTX PRO 6000, starts answering a single camera that sends one 720p image after 247 ms (time to first token, TTFT), then writes 231.4 tok/s.
With 5 images per camera at 720p, the answer starts after 761 ms. With one 2K image, after 944 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3-VL-8B-Instruct (8.8B parameters), served in BF16 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 230 ms (time to first token, TTFT), then writes 92.4 tok/s.
With 5 images per camera at 720p, the answer starts after 761 ms. With one 2K image, after 684 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3-VL-8B-Instruct (8.8B parameters), served in Q8_0 format with llama.cpp on RTX PRO 6000, starts answering a single camera that sends one 720p image after 329 ms (time to first token, TTFT), then writes 153.8 tok/s.
With 5 images per camera at 720p, the answer starts after 1,037 ms. With one 2K image, after 1,788 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3.6-35B-A3B (35B parameters), served in FP8 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 212 ms (time to first token, TTFT), then writes 220.8 tok/s.
With 5 images per camera at 720p, the answer starts after 844 ms. With one 2K image, after 586 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3.6-35B-A3B (35B parameters), served in NVFP4 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 221 ms (time to first token, TTFT), then writes 282.9 tok/s.
With 5 images per camera at 720p, the answer starts after 841 ms. With one 2K image, after 798 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Qwen3.8-27B (27B parameters), served in FP8 format with vLLM on RTX PRO 6000, starts answering a single camera that sends one 720p image after 293 ms (time to first token, TTFT), then writes 50 tok/s.
With 5 images per camera at 720p, the answer starts after 1,014 ms. With one 2K image, after 906 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Cosmos-Reason1-7B (8.3B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 4,370 ms (time to first token, TTFT), then writes 20.4 tok/s.
With 5 images per camera at 720p, the answer starts after 21,756 ms. With one 2K image, after 25,557 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Cosmos-Reason2-2B (2.4B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 1,348 ms (time to first token, TTFT), then writes 44.4 tok/s.
With 5 images per camera at 720p, the answer starts after 6,261 ms. With one 2K image, after 9,690 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Cosmos-Reason2-8B (8.8B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 4,001 ms (time to first token, TTFT), then writes 18.8 tok/s.
With 5 images per camera at 720p, the answer starts after 19,836 ms. With one 2K image, after 31,764 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Cosmos3-Edge (3.9B parameters), served in BF16 format with vLLM on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 684 ms (time to first token, TTFT), then writes 42.6 tok/s.
With 5 images per camera at 720p, the answer starts after 2,956 ms. With one 2K image, after 3,951 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 8 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Gemma-4-26B-A4B-it (26B parameters), served in Q4_0 format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 3,587 ms (time to first token, TTFT), then writes 25.1 tok/s.
With 5 images per camera at 720p, the answer starts after 24,097 ms. With one 2K image, after 15,438 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Gemma-4-E4B-it (8.0B parameters), served in Q4_0 format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 1,183 ms (time to first token, TTFT), then writes 26.2 tok/s.
With 5 images per camera at 720p, the answer starts after 5,127 ms. With one 2K image, after 3,432 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3-VL-4B-Instruct (4.4B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 1,832 ms (time to first token, TTFT), then writes 23.2 tok/s.
With 5 images per camera at 720p, the answer starts after 8,888 ms. With one 2K image, after 11,276 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3-VL-8B-Instruct (8.8B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 4,000 ms (time to first token, TTFT), then writes 18.6 tok/s.
With 5 images per camera at 720p, the answer starts after 19,933 ms. With one 2K image, after 31,702 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3-VL-8B-Instruct (8.8B parameters), served in Q8_0 format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 3,868 ms (time to first token, TTFT), then writes 17 tok/s.
With 5 images per camera at 720p, the answer starts after 19,204 ms. With one 2K image, after 31,317 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), not even one camera is answered in time.
Qwen3.5-4B (4.7B parameters), served in Q4_K_M format with llama.cpp on Jetson AGX Orin 32GB, starts answering a single camera that sends one 720p image after 2,210 ms (time to first token, TTFT), then writes 25.3 tok/s.
With 5 images per camera at 720p, the answer starts after 10,385 ms. With one 2K image, after 12,483 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Alpamayo2-Edge (2.7B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 136 ms (time to first token, TTFT), then writes 101.2 tok/s.
With 512 images per camera at 720p, the answer starts after 46,217 ms. With one 2K image, after 229 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 128 cameras at once.
Alpamayo2-Edge is a driving model: from six cameras (22 frames), the vehicle's recent motion and a navigation instruction it writes a short chain of causation and plans six candidate trajectories. The table above measures only its vision-language part, served with vLLM in BF16 with greedy decoding, the way the model answers questions. Its answers to "Describe the scene." are short (2 to 48 tokens on average per cell), so its tok/s figures swing more than other rows'. This test runs the whole model with its official FP8 TensorRT Edge-LLM recipe (TensorRT Edge-LLM v0.11.1, 5 diffusion steps, 6 candidates, greedy decoding of the chain of causation) on the recipe's 19 navigation demo clips, 3 runs each.
Jetson AGX Thor at MAXN with jetson_clocks and TensorRT 11.3; the recipe's own figures come from a DRIVE AGX Thor with TensorRT 11.3. As in the recipe, each decision encodes only the newest frame of every camera and reuses the earlier ones.
| Stage | Jetson AGX Thor | DRIVE AGX Thor (recipe) |
|---|---|---|
| Vision encoder, 6 newest frames | 21 | 20 |
| Prefill, 3,261 tokens | 67 | 75 |
| Chain of causation, per token | 7.8 | about 7 |
| Trajectory expert, 5 steps × 6 candidates | 70 | 70 |
| Whole decision (22 tokens) | 331 (p90 341) | about 320 |
| Setting | ms |
|---|---|
| 120W, TensorRT 10.16 (JetPack) | 439 |
| MAXN, TensorRT 10.16 (JetPack) | 423 |
| MAXN + jetson_clocks, TensorRT 11.3 | 331 |
With all 22 frames encoded inside the timed run a decision takes 402 ms. TensorRT 11.3 brought the trajectory expert from 99 to 70 ms; fixing the GPU and memory clocks with jetson_clocks (memory at 4266 MHz instead of about 2750) brought the rest.
The recipe gives about 7 ms per token for greedy decoding but no greedy total; about 320 ms is its stages plus 22 tokens × 7 ms. On the recipe's example clip the chain of causation matches the recipe's text word for word.
Alpamayo2-Super (34.3B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 780 ms (time to first token, TTFT), then writes 4.6 tok/s.
With 256 images per camera at 720p, the answer starts after 135,664 ms. With one 2K image, after 838 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Alpamayo2-Super (a 32B Qwen3-VL backbone and a 2.3B diffusion expert) writes a short chain of causation from six cameras (4 frames each) and the vehicle's recent motion, then plans a trajectory. The table above measures only its vision-language part, served with vLLM in BF16. To "Describe the scene." that part mostly answers in a few tokens (2 to 34 on average per cell) that no scene description was provided, although it does describe the image when asked what it sees ("The scene shows a road intersection"); its tok/s figures therefore swing more than other rows'. This test runs the whole model with the official inference path of its repository (NVlabs/alpamayo2, inference_smoke.py: BF16, 10 diffusion steps, one trajectory, top-p 0.98, temperature 0.6) on the repository's 5 validation clips, 3 runs each, Jetson AGX Thor at MAXN.
In the table above 2 requests came back with an empty answer; as in the camera limit, an empty answer is not counted as a failure, so it does not end a series. There is no TensorRT recipe for this model; attention runs through PyTorch SDPA, because flash-attn has no build for this board.
| Step | Mean | p50 | p90 |
|---|---|---|---|
| Whole decision | 13.3 | 12.6 | 17.5 |
| Input preparation | 1.0 | 1.0 | 1.0 |
| Model (chain of causation and trajectory) | 12.3 | 11.7 | 16.5 |
Each decision reads 24 images (4,580 prompt tokens) and writes about 18 tokens of chain of causation (10 to 32); the longer the chain, the longer the decision. The model takes 69 GiB of memory. Alpamayo2-Edge, with its TensorRT recipe on the same board, takes 331 ms per decision.
Cosmos-Reason1-7B (8.3B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 582 ms (time to first token, TTFT), then writes 16.4 tok/s.
With 64 images per camera at 720p, the answer starts after 56,100 ms. With one 2K image, after 2,297 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Cosmos-Reason2-2B (2.4B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 235 ms (time to first token, TTFT), then writes 62.6 tok/s.
With 256 images per camera at 720p, the answer starts after 176,283 ms. With one 2K image, after 1,018 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 64 cameras at once.
Cosmos-Reason2-8B (8.8B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 490 ms (time to first token, TTFT), then writes 15.8 tok/s.
With 256 images per camera at 720p, the answer starts after 437,057 ms. With one 2K image, after 1,906 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Cosmos3-Edge (3.9B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 241 ms (time to first token, TTFT), then writes 63.8 tok/s.
With 128 images per camera at 720p, the answer starts after 56,092 ms. With one 2K image, after 1,336 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 32 cameras at once.
Cosmos3-Nano (16B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 480 ms (time to first token, TTFT), then writes 15.7 tok/s.
With 256 images per camera at 720p, the answer starts after 436,357 ms. With one 2K image, after 1,878 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Eagle2.5-8B (8.1B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 649 ms (time to first token, TTFT), then writes 16.9 tok/s.
With 64 images per camera at 720p, the answer starts after 143,682 ms. With one 2K image, after 738 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Gemma-4-31B-it (31B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 976 ms (time to first token, TTFT), then writes 3.9 tok/s.
With 64 images per camera at 720p, the answer starts after 124,148 ms. With one 2K image, after 995 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Gemma-4-31B-it (31B parameters), served in NVFP4 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 662 ms (time to first token, TTFT), then writes 7.3 tok/s.
With 64 images per camera at 720p, the answer starts after 118,247 ms. With one 2K image, after 716 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Llama-3.1-Nemotron-Nano-VL-8B (8.7B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 752 ms (time to first token, TTFT), then writes 15.3 tok/s.
With 32 images per camera at 720p, the answer starts after 55,681 ms. With one 2K image, after 835 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
MedGemma-1.5-4B-it (4.3B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 270 ms (time to first token, TTFT), then writes 28.9 tok/s.
With 256 images per camera at 720p, the answer starts after 528,602 ms. With one 2K image, after 341 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Nemotron-Nano-12B-v2-VL (13B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 578 ms (time to first token, TTFT), then writes 10.2 tok/s.
With 128 images per camera at 720p, the answer starts after 59,143 ms. With one 2K image, after 1,355 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3-VL-30B-A3B-Instruct (30B parameters), served in FP8 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 457 ms (time to first token, TTFT), then writes 56 tok/s.
With 256 images per camera at 720p, the answer starts after 560,535 ms. With one 2K image, after 2,069 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Qwen3-VL-4B-Instruct (4.4B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 332 ms (time to first token, TTFT), then writes 28.2 tok/s.
With 256 images per camera at 720p, the answer starts after 407,744 ms. With one 2K image, after 1,261 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 32 cameras at once.
Qwen3-VL-8B-Instruct (8.8B parameters), served in BF16 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 490 ms (time to first token, TTFT), then writes 15.6 tok/s.
With 256 images per camera at 720p, the answer starts after 436,769 ms. With one 2K image, after 1,899 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3.6-35B-A3B (35B parameters), served in FP8 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 486 ms (time to first token, TTFT), then writes 53.7 tok/s.
With 256 images per camera at 720p, the answer starts after 178,088 ms. With one 2K image, after 2,037 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Qwen3.6-35B-A3B (35B parameters), served in NVFP4 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 419 ms (time to first token, TTFT), then writes 72.1 tok/s.
With 256 images per camera at 720p, the answer starts after 177,032 ms. With one 2K image, after 1,958 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Qwen3.8-27B (27B parameters), served in FP8 format with vLLM on Jetson AGX Thor, starts answering a single camera that sends one 720p image after 2,902 ms (time to first token, TTFT), then writes 8.6 tok/s.
With 256 images per camera at 720p, the answer starts after 403,624 ms. With one 2K image, after 7,013 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with one camera at a time.
Cosmos-Reason1-7B (8.3B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 511 ms (time to first token, TTFT), then writes 13.5 tok/s.
With 64 images per camera at 720p, the answer starts after 46,763 ms. With one 2K image, after 2,026 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Cosmos-Reason2-2B (2.4B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 158 ms (time to first token, TTFT), then writes 49.7 tok/s.
With 256 images per camera at 720p, the answer starts after 129,322 ms. With one 2K image, after 747 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 64 cameras at once.
Cosmos-Reason2-8B (8.8B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 390 ms (time to first token, TTFT), then writes 14.5 tok/s.
With 256 images per camera at 720p, the answer starts after 314,563 ms. With one 2K image, after 1,710 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Cosmos3-Edge (3.9B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 179 ms (time to first token, TTFT), then writes 55.4 tok/s.
With 128 images per camera at 720p, the answer starts after 41,369 ms. With one 2K image, after 950 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with at least 16 cameras at once: it still met the target at the highest number measured, so the real limit is higher.
Cosmos3-Nano (16B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 469 ms (time to first token, TTFT), then writes 14.2 tok/s.
With 256 images per camera at 720p, the answer starts after 308,947 ms. With one 2K image, after 1,613 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
d1-3B (3.1B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 175 ms (time to first token, TTFT), then writes 34.5 tok/s.
With 32 images per camera at 720p, the answer starts after 4,972 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 64 cameras at once.
Eagle2.5-8B (8.1B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 737 ms (time to first token, TTFT), then writes 15.5 tok/s.
With 64 images per camera at 720p, the answer starts after 109,194 ms. With one 2K image, after 780 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Fara1.5-27B (27B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 849 ms (time to first token, TTFT), then writes 4.5 tok/s.
With 128 images per camera at 720p, the answer starts after 127,441 ms. With one 2K image, after 3,397 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Gemma-4-26B-A4B-it (26B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 407 ms (time to first token, TTFT), then writes 24 tok/s.
With 128 images per camera at 720p, the answer starts after 40,031 ms. With one 2K image, after 445 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Gemma-4-31B-it (31B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 781 ms (time to first token, TTFT), then writes 3.8 tok/s.
With 128 images per camera at 720p, the answer starts after 228,036 ms. With one 2K image, after 740 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Gemma-4-31B-it (31B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 511 ms (time to first token, TTFT), then writes 7.1 tok/s.
With 128 images per camera at 720p, the answer starts after 208,311 ms. With one 2K image, after 539 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
GLM-4.6V-Flash (10B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 596 ms (time to first token, TTFT), then writes 10.5 tok/s.
With 64 images per camera at 720p, the answer starts after 64,534 ms. With one 2K image, after 2,813 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Granite-Vision-4.1-4B (4.0B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 303 ms (time to first token, TTFT), then writes 26.3 tok/s.
With 32 images per camera at 720p, the answer starts after 12,780 ms. With one 2K image, after 364 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
H2O-Lightning-4B (4.7B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 214 ms (time to first token, TTFT), then writes 21.2 tok/s.
With 128 images per camera at 720p, the answer starts after 34,823 ms. With one 2K image, after 946 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 32 cameras at once.
LensVLM-9B (9.4B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 370 ms (time to first token, TTFT), then writes 13 tok/s.
With 128 images per camera at 720p, the answer starts after 50,707 ms. With one 2K image, after 656 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
LFM2.5-VL-3B (3.1B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 175 ms (time to first token, TTFT), then writes 34.3 tok/s.
With 32 images per camera at 720p, the answer starts after 4,981 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 64 cameras at once.
Llama-3.1-Nemotron-Nano-VL-8B (8.7B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 888 ms (time to first token, TTFT), then writes 14.2 tok/s.
With 32 images per camera at 720p, the answer starts after 47,610 ms. With one 2K image, after 918 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Llama-4-Scout-17B-16E-Instruct (109B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 937 ms (time to first token, TTFT), then writes 18.4 tok/s.
With 64 images per camera at 720p, the answer starts after 89,062 ms. With one 2K image, after 1,392 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
MedGemma-1.5-4B-it (4.3B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 191 ms (time to first token, TTFT), then writes 24.6 tok/s.
With 256 images per camera at 720p, the answer starts after 230,121 ms. With one 2K image, after 231 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 32 cameras at once.
MiMo-V2.6-Distill-Qwen-9B (9.4B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 324 ms (time to first token, TTFT), then writes 12.8 tok/s.
With 128 images per camera at 720p, the answer starts after 49,715 ms. With one 2K image, after 1,539 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
MiniCPM-V-4.6 (1.3B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 194 ms (time to first token, TTFT), then writes 114.3 tok/s.
With 128 images per camera at 720p, the answer starts after 19,864 ms. With one 2K image, after 290 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 32 cameras at once.
MiniCPM-V-4.7-1B (1.6B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 195 ms (time to first token, TTFT), then writes 115.6 tok/s.
With 128 images per camera at 720p, the answer starts after 20,023 ms. With one 2K image, after 293 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 64 cameras at once.
Mistral-Small-4-119B-2603 (119B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 590 ms (time to first token, TTFT), then writes 31.4 tok/s.
With 64 images per camera at 720p, the answer starts after 44,930 ms. With one 2K image, after 832 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Nemotron-Nano-12B-v2-VL (13B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 570 ms (time to first token, TTFT), then writes 10 tok/s.
With 128 images per camera at 720p, the answer starts after 59,026 ms. With one 2K image, after 1,356 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
NVIDIA-Nemotron-3.5-Bolt-74B-A9B (74B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 662 ms (time to first token, TTFT), then writes 21.4 tok/s.
With 16 images per camera at 720p, the answer starts after 7,741 ms. With one 2K image, after 2,032 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
NVIDIA-Nemotron-3.5-Super-VL-120B-A12B (120B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 773 ms (time to first token, TTFT), then writes 19.1 tok/s.
With 64 images per camera at 720p, the answer starts after 34,834 ms. With one 2K image, after 2,465 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 4 cameras at once.
Qwen3-VL-30B-A3B-Instruct (30B parameters), served in FP8 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 357 ms (time to first token, TTFT), then writes 50.4 tok/s.
With 256 images per camera at 720p, the answer starts after 379,830 ms. With one 2K image, after 1,496 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3-VL-4B-Instruct (4.4B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 222 ms (time to first token, TTFT), then writes 23.4 tok/s.
With 256 images per camera at 720p, the answer starts after 279,616 ms. With one 2K image, after 1,036 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 32 cameras at once.
Qwen3-VL-8B-Instruct (8.8B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 360 ms (time to first token, TTFT), then writes 14.4 tok/s.
With 256 images per camera at 720p, the answer starts after 311,581 ms. With one 2K image, after 1,685 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3.5-0.8B (0.9B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 81 ms (time to first token, TTFT), then writes 115.3 tok/s.
With 128 images per camera at 720p, the answer starts after 12,118 ms. With one 2K image, after 340 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 128 cameras at once.
Qwen3.5-122B-A10B (122B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 603 ms (time to first token, TTFT), then writes 14.2 tok/s.
With 128 images per camera at 720p, the answer starts after 103,256 ms. With one 2K image, after 2,215 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Qwen3.5-27B (27B parameters), served in FP8 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 545 ms (time to first token, TTFT), then writes 8.1 tok/s.
With 128 images per camera at 720p, the answer starts after 97,836 ms. With one 2K image, after 2,394 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Qwen3.5-2B (2.3B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 146 ms (time to first token, TTFT), then writes 44.8 tok/s.
With 128 images per camera at 720p, the answer starts after 20,233 ms. With one 2K image, after 693 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 64 cameras at once.
Qwen3.5-35B-A3B (35B parameters), served in FP8 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 341 ms (time to first token, TTFT), then writes 53.8 tok/s.
With 128 images per camera at 720p, the answer starts after 46,522 ms. With one 2K image, after 1,365 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3.5-9B (9.7B parameters), served in BF16 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 327 ms (time to first token, TTFT), then writes 12.9 tok/s.
With 128 images per camera at 720p, the answer starts after 49,583 ms. With one 2K image, after 2,110 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3.6-27B (27B parameters), served in FP8 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 550 ms (time to first token, TTFT), then writes 8.1 tok/s.
With 128 images per camera at 720p, the answer starts after 97,355 ms. With one 2K image, after 2,397 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 8 cameras at once.
Qwen3.6-35B-A3B (35B parameters), served in FP8 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 405 ms (time to first token, TTFT), then writes 51 tok/s.
With 256 images per camera at 720p, the answer starts after 139,863 ms. With one 2K image, after 1,555 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3.6-35B-A3B (35B parameters), served in NVFP4 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 303 ms (time to first token, TTFT), then writes 78.2 tok/s.
With 256 images per camera at 720p, the answer starts after 127,970 ms. With one 2K image, after 1,342 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 16 cameras at once.
Qwen3.8-27B (27B parameters), served in FP8 format with vLLM on DGX Spark, starts answering a single camera that sends one 720p image after 775 ms (time to first token, TTFT), then writes 8.1 tok/s.
With 256 images per camera at 720p, the answer starts after 285,156 ms. With one 2K image, after 4,203 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 2 cameras at once.
Qwen3-VL-30B-A3B-Instruct (30B parameters), served in FP8 format with vLLM on DGX B300 (1 GPU), starts answering a single camera that sends one 720p image after 83 ms (time to first token, TTFT), then writes 298 tok/s.
With 256 images per camera at 720p, the answer starts after 26,497 ms. With one 2K image, after 245 ms.
At the default target of a mean TTFT within 3,000 ms (720p, one image per camera), it keeps up with 128 cameras at once.