InstructPix2Pix

Overview

InstructPix2Pix edits an image according to a written instruction. Tim Brooks, Aleksander Holynski and Alexei A. Efros at UC Berkeley introduced it in the paper "InstructPix2Pix: Learning to Follow Image Editing Instructions" and fine-tuned it from the Stable Diffusion v1.5 checkpoint.

Its training data was generated. Starting from real LAION-Aesthetics image captions, a fine-tuned GPT-3 wrote an edit instruction and the caption after the edit, and Stable Diffusion with Prompt-to-Prompt turned each caption pair into a before and after image. The authors report 454,445 such examples and say the model still generalizes to real photos and to instructions written by people.

On the Wiro sample, a blue-toned city skyline edited with "Add a beautiful sunset" keeps its buildings and their reflection while the sky and water turn orange and red.

What you can build

• Photo editing from a text box, where users type the change instead of masking an area.
• Time-of-day and lighting changes, as in "it is now midnight".
• Character and scene swaps like "turn him into a cyborg" or "turn the humans into robots".

Inputs

• The image to edit, uploaded or given as a URL.
• The instruction, phrased as a command.
• A negative prompt, "bad, blurry" by default.
• Steps (30 by default, 1 to 500), number of outputs (1 to 20), width and height (1024 by default, up to 2048) and the scheduler (Euler Ancestral by default).
• A seed, fixed at 123456 by default.
• Text guidance (3.5 by default, 1 to 20) and image guidance (1.0 by default, 0 to 1).

Recommended settings

Text guidance sets how closely the model follows the instruction, and image guidance sets how much of the original it keeps. Adapted from the authors' tips:

• Edit too weak: raise text guidance (the authors start at 7.5). Keep image guidance at 1.0, because any lower value switches guidance off and text guidance then has no effect.
• Edit too strong, or details lost: lower text guidance. Image guidance already sits at 1.0, its highest value here.
• Rephrase the instruction ("turn him into a dog", "make him a dog", "as a dog") and change the seed for another attempt.
• More steps sometimes help.

Limitations

• Small faces can come out distorted, because the Stable Diffusion autoencoder handles them poorly. Crop so the face fills more of the frame.
• The authors say the model struggles with counting objects and with spatial instructions such as moving something to the left of the image or swapping two objects.
• Edits can carry biases from the data and models InstructPix2Pix is built on; the paper's example is a correlation between profession and gender.

Safety & compliance

The code and weights are released under the MIT license. The authors add that the checkpoints are fine-tuned from Stable Diffusion and that its license may place further restrictions on use. If you edit a photo of a real person, make sure you have their consent.