MEMO

Overview

MEMO (Memory-guided EMOtion-aware diffusion) is an audio-driven portrait animation model from researchers at Skywork AI, Nanyang Technological University and the National University of Singapore. You give it one image of a face and one audio file. It returns a talking video in which the lips, head and expression follow the audio.

Two parts of the design matter in practice. A memory-guided temporal module keeps information from earlier frames and uses it to guide the next ones, so identity and motion stay steady across long clips. An emotion-aware audio module detects the emotion in the voice and adjusts the facial expression to match, so a cheerful line and a sad one do not get the same face.

The authors released MEMO's code and weights under Apache-2.0 as a preview model for research purposes. The bundled face-analysis models that read the identity from your portrait come from InsightFace and are licensed for non-commercial research only. The project page shows MEMO driven by speech, singing and rap in English, Mandarin, Spanish, Japanese, Korean and Cantonese, and applied to portrait photos, sculptures, digital art, animation and AI-generated images.

What you can build

• Prototype talking presenters made from one headshot.
• Lip-sync tests with illustrated or animated characters.
• Singing portraits driven by a vocal track.
• Dubbing tests: pair the same image with audio in several languages.

Inputs

MEMO needs two files: a portrait (JPG, PNG, WebP, GIF or HEIC) and the audio it should follow (MP3, WAV, M4A, WebM or Opus). A single, clearly visible face works best. If the image contains several faces, MEMO takes the identity from the largest one.

The audio can be a full song. MEMO separates the vocals from the music and syncs the face to the voice. The finished video keeps the full mix at 16 kHz, so music loses its high end.

Under advanced settings you can set the output resolution (512 by default, up to 1024), the frame rate (10 to 30 fps, default 30), the guidance scale (default 3.5), the number of inference steps (default 30) and a seed for repeatable results.

Outputs

An MP4 video as long as the audio, with your track attached at 16 kHz. The sample on this page turns a 15-second clip into a 15-second video. Frames are square, with the resolution setting as the side length, so the sample made at 768 comes out at 768 by 768.

Recommended settings

• Crop the portrait to a square before uploading. MEMO resizes the image to a square frame, so a wide or tall photo gets stretched.
• The authors' own configuration runs at resolution 512 with guidance 3.5 and 20 inference steps. The defaults here match it, apart from 30 steps.

Limitations

• The image needs a detectable face. Without one, the model runs without the identity embedding it uses to keep the person consistent.
• Everything comes from one still image, so the model has to invent anything the photo does not show, such as the inside of the mouth or the side of the head when it turns.

Safety & compliance

The MEMO authors prohibit using it for malicious, misleading, defamatory or privacy-infringing content, including deepfakes for political misinformation, impersonation, harassment and fraud. Animate only people who have agreed to it, and get proper authorization before using a public figure's likeness or voice. The authors also ask users to review outputs carefully before sharing them. Check the InsightFace license for the bundled face-analysis models before any commercial use.