Video To Text

Overview

CogVLM2-Llama3-Caption is THUDM's video captioning model, a 12.5B-parameter vision-language checkpoint from the CogVLM2 family, built with Meta Llama 3. THUDM made it because most footage comes with no written description, while a text-to-video model needs one for every training clip. Its captions became training data for CogVideoX.

You give it a clip and an instruction, and it answers with one English paragraph that follows the action in order. In the alpaca sample on this page it picks out the round black sunglasses and dark blue sweaters, then reports changing expressions on faces that barely move, and it never mentions the left alpaca coming up close to the camera. The cabin sample is one handheld walk around a single log hut. The caption keeps the order, from the metal stove to a hand holding a roof beam under a tarp, but describes it as a series of separate rooms and buildings.

What you can build

• Caption pipelines that label raw footage for training or fine-tuning a video generation model.
• Searchable descriptions for a video archive, so editors can find clips by what is on screen.
• First-draft video descriptions for accessibility work, reviewed by a person before they go live.

Inputs

• A video, uploaded as a file or given as a URL.
• A prompt telling the model what to write. The default, "Please describe this video in detail.", gives the long scene paragraphs in the samples. THUDM documents only this prompt, and a run on the alpaca clip that asked for the mood alone still returned a literal scene paragraph.
• Beams, from 1 to 30, default 1. Above 1, beam search weighs that many candidate captions and returns the one the model scores highest.
• Temperature, from 0.1 to 1, default 0.1. At one beam it has no effect, because the model then always takes its most likely next word.

Outputs

A single paragraph of plain English text, with no timestamps or bullet points. The two samples here are 96 and 112 words long.

Recommended settings

Start from the defaults. THUDM's own example runs at temperature 0.1 with a single beam.

When you caption a whole collection, send every clip the same prompt. Keep beams and temperature fixed as well, so the whole dataset comes from one setup.

Limitations

• Captions are in English, the only language THUDM lists for the model.
• THUDM's reference code takes one frame per second and stops at 24 frames. In that setup anything after roughly the 24-second mark goes unseen, and a short event between two sampled frames can be missed.
• The model reads frames only, so the soundtrack has no effect on the caption.
• Do not count on it to read on-screen text. In the cabin sample it mentions the Chinese subtitles but never says what they read.
• Captions mix what is visible with interpretation, such as "whimsically cool" or "suggesting a narrative". Review them before treating them as ground truth.

Safety & compliance

The weights are released under the CogVLM2 License, which allows academic research, asks commercial users to register first and rules out military use. Because the model is built with Meta Llama 3, the Llama 3 license applies as well, so read both before using it in a product. If a clip shows a real person's face, make sure you have that person's consent to process it.