Ask Video

Overview

Ask Video runs CogVLM2-Llama3-Caption, a video captioning model from THUDM. Most video footage has no written description, so THUDM built this model to write one and used it to caption the training data for its CogVideoX text-to-video model. The card lists Meta Llama 3.1 8B Instruct as the base model.

What you can build

• Caption sets for training or fine-tuning text-to-video models, the job THUDM built the model for
• Searchable descriptions for a video library, so clips can be found by what happens in them
• Alt text for short videos on websites and in apps

Inputs

• One video in MP4, WebM or MOV.
• A prompt, which is required. It defaults to "Please describe this video in detail." Edit it to ask about one part of the clip, such as the people or the background.
• Two advanced settings: the beam count (1 to 30, default 1) and temperature (0.1 to 1, default 0.1).

Outputs

A single block of text. Each sample is a four-sentence paragraph that names the subjects, their colors and the setting, then follows how the scene develops. One covers three alpacas in sunglasses whose expressions change in front of a graffiti wall.

Recommended settings

The default prompt and temperature match THUDM's example code, so keep them for dataset captions. Raising the beam count makes the model compare more candidate wordings before it settles on one.

Limitations

• The card presents the model as a captioner and makes no claims about timing or counting questions, so check those answers against the clip.
• English is the only language the card lists.
• THUDM's example code shows the model one frame per second and stops at 24 frames. With that sampling, a brief action between two frames can be missed and anything after the 24th second is never seen.

Safety & compliance

THUDM releases the model under the CogVLM2 license, which allows academic research use and requires registration before commercial use. Because it is built with Meta Llama 3, the card also asks users to follow the Llama 3 license. If a clip shows real people, make sure they agreed to have their footage processed.