nvidia / LocateAnything-3B

Overview

LocateAnything-3B is an image grounding model from Nvidia. It combines a vision encoder with a language model, then generates a text response that encodes locations as boxes or points. It uses a block-based decoding scheme that predicts full coordinates together, instead of emitting coordinates one token at a time. This output format makes it useful for pipelines that must turn natural language into pixel-accurate regions.

What you can build

• Open-vocabulary object detection where labels come from your category list
• Referring expression grounding like “the person in a red jacket” in crowded scenes
• GUI element grounding on screenshots, such as “the search button” or “settings icon”
• Text region localization for OCR pre-processing and document workflows
• Pointing actions for robotics or agents that need a single target coordinate
• Dataset annotation helpers that produce boxes you can review and correct

Inputs

• One RGB image as a JPG or PNG. Use the original resolution when you need tight boxes.
• A task choice that controls what the model should return, such as detecting categories, grounding a single best match, grounding all matches, finding text, or pointing.
• A natural-language target description for tasks that need it. Write a short phrase or instruction that uniquely identifies the region.
• For category detection, provide the category names you want found in the image. A comma-separated list works well.
• For GUI localization, choose whether you want a bounding box or a single click point.

Outputs

• A text response that includes structured location tokens.
• Bounding boxes appear as a labeled region plus 4 integer coordinates in a 0–1000 normalized space. Convert to pixels by scaling by image width and height.
• Points appear as 2 integer coordinates in the same 0–1000 normalized space.
• When nothing matches, the model can return an explicit “none” marker for the location.
• The response may include multiple labeled regions when the task asks for many matches.

Recommended settings

• Use “Ground Single” when you want one best match for a phrase.
• Use “Ground Multi” when you expect several instances of the same description.
• Use “Detect” when you have a fixed set of categories and want all instances.
• Use “Detect Text” to get text region boxes without supplying a text phrase.
• Use the GUI “Point” output when you need a click target instead of a box.

Limitations

• Outputs are quantized to a 0–1000 normalized coordinate grid. Very small objects may need higher-resolution inputs to stay precise.
• Some scenes produce ambiguous matches. Prompts like “the biggest one” can fail when size cues are unclear.
• Dense scenes can cause missed instances or mixed labels when many objects overlap.
• Fast structured decoding can occasionally produce malformed location tokens. Your parser should validate tags and coordinate counts.
• Low-quality inputs reduce accuracy. Blurry photos, heavy compression, extreme downscaling, glare, and scanned documents with artifacts can all degrade box quality.
• Text localization depends on legibility. Small fonts, curved text, and stylized fonts can reduce recall.

Safety & compliance

• This model is released under an Nvidia research license. It is intended for research and development use, not unrestricted commercial deployment.
• Don’t upload images that include sensitive personal data unless you have permission and a lawful purpose.
• If you use it for GUI automation or document processing, add guardrails. Verify targets before taking actions like clicks or deletions.
• Follow applicable laws and site terms when using it for surveillance, monitoring, or automated interaction.