BLIP-2 OPT-2.7B

Overview

This is BLIP-2 from Salesforce Research, built on OPT-2.7B, the 2.7-billion-parameter decoder-only language model released by Meta. BLIP-2 takes a CLIP-like image encoder and the language model as they were, keeps both frozen, and trains only what sits between them: a Querying Transformer (Q-Former) that turns the image into 32 query embeddings, plus a linear layer that projects them to the size of OPT's text embeddings. OPT reads those embeddings ahead of your prompt and writes text after it. The whole checkpoint comes to about 3.7 billion parameters.

The release is marked "pre-trained only", so it has not been fine-tuned for captioning or question answering. The BLIP-2 paper says pre-training used 129 million images, 115 million of them from LAION-400M, with synthetic captions generated for the web images.

What you can build

• Alt-text experiments that compare the model's captions with human-written ones on a test set of images.
• Labeling tests that ask every image in a set the same question, for example which animal it shows, and score the replies against known labels.
• Question answering over cartoons and drawings, like the dinosaur sample on this page.

Inputs

Upload an image (JPG, PNG, GIF, WEBP or HEIC) or paste a public link into the separate image URL field. The prompt box holds your question, and the default one asks what the dinosaur is holding. A tokens setting from 1 to 500, 30 by default, limits how much text comes back.

Outputs

Plain text that repeats your prompt in a "Question: ... Answer: ..." layout. The page has two samples:

• The dinosaur cartoon with the default question returns "A torch". The dinosaur on the right is holding a flashlight, which British English also calls a torch.
• An Eiffel Tower photo, with a prompt asking what the model sees in the image, returns "The eiffel tower".

Recommended settings

• Type only the question. The Question and Answer labels are added to the result for you.
• For a caption, ask something open, as the Eiffel Tower sample does.
• Both samples answer in two or three words, far below the default limit of 30 tokens. Raise the limit only if longer answers get cut off.

Limitations

• Meta's card for OPT warns about bias and safety problems, and about hallucination and low variety in generated text. BLIP-2 uses OPT unchanged, so those warnings carry over.
• Solved examples in the prompt do not help. The authors found no gain from them and put it down to pre-training data that held one image-text pair per sample, so ask the question on its own.
• The paper traces bad answers to wrong knowledge in the language model and to the model following the wrong line of reasoning. It also knows nothing about things that appeared after its training data.
• English is the only language listed on the card.

Safety & compliance

Salesforce lists the checkpoint under the MIT license, but the OPT-2.7B weights inside it come from Meta under the OPT model license, which grants use solely for non-commercial research. The model card states that this release is for research purposes only, in support of an academic paper. It also says BLIP-2 has not been tested in real-world applications and should not be deployed directly in any application. Because pre-training drew on internet image-text data such as LAION, outputs can repeat biases or inappropriate content from that data. If your images show identifiable people, make sure you have their consent before uploading them.