VideoLLaMA3 2B
DAMO-NLP-SG's 2B VideoLLaMA 3 model watches a video clip and answers in English text. It skips image patches that stay the same from frame to frame, so static shots use fewer tokens.
Overview
This is the 2B video model in DAMO-NLP-SG's VideoLLaMA 3 series. Qwen2.5-1.5B-Instruct handles the language side, and the model card lists VideoLLaMA3-2B-Image as a base model, so the video weights start from the image checkpoint of the same size. The paper describes four training stages: the first three mainly build image understanding and the fourth is dedicated to video.
The model sees a clip as a run of sampled frames. Before those frames reach the language model, a step the authors call the Differential Frame Pruner compares each patch with the same patch in the previous frame and drops it when the pixel difference falls below a threshold. A tripod shot of a quiet scene loses many patches this way, a handheld shot fewer. The authors' stated aim is a compact token sequence that keeps the model on the parts of the video that move.
DAMO-NLP-SG's results table gives this model 59.6 on VideoMME without subtitles and 63.4 with them, 65.4 on MLVU (dev), 57.1 on LongVideoBench (val) and 81.1 on NextQA. MVBench is the only row of that table where a rival 2B model comes out ahead: InternVL2.5 scores 68.8 there against 65.5.
For a single photo, chart or scanned page, use VideoLLaMA3 2B Image, the image checkpoint this model was built from. Every run on this page needs a video, so pick it when the input is a clip.
What you can build
- A searchable index for a video library, built from a text description of each clip
- Direct answers about footage, such as who appears in the shot and what they are doing
- A review pass that flags out-of-place content; on this page it picks out a person vacuuming sand in a desert
- First-draft summaries of drone shots or wildlife footage for an editor to correct
Inputs
Every run needs a video, either uploaded as a file or passed as a link. The prompt box is prefilled with "Please describe this video in detail." In the advanced settings, max new tokens caps how long the answer can get; it defaults to 128 and accepts values from 10 to 4096.
Outputs
An answer in English text. For an aerial clip of a city skyline, the sample reply is three sentences about a large white-roofed stadium near the center of the frame. The desert clip gets two sentences on why vacuuming sand is unusual.
Recommended settings
The default of 128 tokens covers a short paragraph like the two samples. For a scene-by-scene account of a longer video, raise it to several hundred. Name what you care about in the prompt, such as what the person is holding or what happens after the car stops, since the default prompt asks only for a general description.
Limitations
- The model works from the frames alone. The card lists no audio input and the paper names audio and speech as unexplored, so speech and music in a clip are ignored.
- Frames are sampled at intervals, so a very brief event can fall between two of them.
- The authors point out that video training data has lower annotation quality and less variety than image data, which can hold the model back on unfamiliar kinds of footage.
Safety & compliance
The model card lists the Apache 2.0 license, but the project's GitHub page marks VideoLLaMA 3 as a research preview for non-commercial use only, under the Qwen model licenses, the terms of use of data generated by OpenAI and Gemini, and ShareGPT's privacy practices. Read those terms before any commercial use. Upload footage of identifiable people only with their consent.
Example prompts
Great starting points for VideoLLaMA3-2B.
API quick start
Run VideoLLaMA3-2B with a single API call.
{
"prompt": "Please describe this video in detail.",
"inputVideo": "https://your-cdn.com/input.mp4",
"max_new_tokens": 128
}curl -X POST "https://api.wiro.ai/v1/Run/damo-nlp-sg/videollama3-2b" \
-H "Content-Type: application/json" \
-H "x-api-key: YOUR_WIRO_API_KEY" \
--data-binary @- <<'JSON'
{
"prompt": "Please describe this video in detail.",
"inputVideo": "https://your-cdn.com/input.mp4",
"max_new_tokens": 128
}
JSONOther models you might like
Explore similar video generation models.

