bytedance-research / Lance-Text-to-Video

## Overview Lance Text to Video is the video-generation mode of Lance from ByteDance Research. Lance keeps text and visual context in one interleaved sequence, then routes generation through dedicated experts. It denoises a 3D latent video and decodes frames with a video VAE to produce an MP4 clip. This helps you turn a prompt into motion, camera movement, and consistent subjects. ## What you can build - Short cinematic shots from a single prompt - Product-style b-roll with clear subject focus and controlled camera moves - Character motion clips with consistent framing across the whole shot - Image-to-video animation from a still image, with motion guided by text - Video edits like background swaps, wardrobe changes, or accessory additions - Multi-turn editing workflows that keep the same subject across several edits ## Inputs - A required written description of the scene you want, in plain language. Multi-line prompts work well for camera, lighting, and motion. - An optional negative description that lists things you don’t want to see. - An optional input image to animate. Use a clear PNG or JPEG with a strong main subject. - An optional input video to edit or extend. MP4 works best. Short clips with stable camera help. - The output frame size in pixels. Pick a width and height that fit your target aspect ratio. Use multiples of 16. - The number of frames to generate. More frames can improve motion but raise failure risk. - The output frame rate for the saved video. If you don’t set one, Lance commonly saves at 12 fps. - The number of denoising steps for generation. More steps can improve detail but reduce variation. - A guidance strength that controls how tightly the output follows the prompt. - An optional seed for repeatable generations. Use the same seed to reproduce the same result. ## Outputs - An MP4 video clip that contains the generated frames at your chosen size and frame rate. - If you generate a single frame, the system can return a still image instead of a video. - For edit workflows, the output keeps the source video’s core motion, then applies your changes. ## Recommended settings - Start with 480p-class output sizes for the most reliable results. - Use 30 denoising steps as a solid baseline for quality. - Keep guidance around 4 to balance prompt following and natural motion. - For longer clips, increase frames gradually. The official Lance pipeline caps at 121 frames. ## Limitations - This is a research release, not a polished product model. Results vary by prompt. - The released checkpoint was trained up to 480p video at 12 fps. Higher settings may degrade quality. - The official inference code limits video generation to 121 frames. - Fast motion, crowded scenes, and tiny objects can cause flicker or shape drift. - Text rendering inside frames is unreliable, especially for signs and UI. - Low-quality inputs hurt edits. Heavy compression, watermarks, rapid cuts, and shaky camera reduce stability. - If your source video has lighting jumps or motion blur, edits can amplify artifacts. ## Safety & compliance - Lance weights are released under the Apache-2.0 license. - Only upload images and videos you have rights to use. - Don’t generate deceptive or non-consensual impersonation content. Review outputs before sharing.