ForageBench: A Photorealistic, Physically Grounded Benchmark for Interactive Object Search

Aparajito Saha¹, Zhen Hao Gan¹, Jinjia Guo¹, Jacob Skwirsk¹, Jeremy Acheampong¹,
Anton Arapin¹, Chahyon Ku¹, Yue Hu¹, Nima Fazeli¹, Bernadette Bucher¹

¹University of Michigan, Ann Arbor

Paper Code (coming soon) PhyGS

ForageBench teaser: a robot locating and retrieving a language-specified target in a house-scale home.

A robot must locate and retrieve a language-specified target in an unseen, house-scale home — often having to physically interact with the scene to make an occluded target perceivable.

Abstract

Interactive semantic object search is the task of locating and retrieving a target object in an unseen, house-scale environment. This task requires both long-horizon spatial reasoning for navigation and fine-grained skills for manipulation, often demanding that an agent physically interact with the environment to make an occluded target perceivable. Prior benchmarks have largely isolated one capability or the other; we present ForageBench, a benchmark designed to study them jointly on a mobile manipulator developed in our newly introduced simulation framework PhyGS.

PhyGS integrates large-scale procedural scene generation with photorealism, physical grounding, and low-level robotic control. Using PhyGS, we build an evaluation dataset of 25 house-scale scenes and 100 episodes spanning a progression of difficulty, and benchmark state-of-the-art vision–language models acting as high-level skill orchestrators. Our best baseline on a simulated Boston Dynamics Spot with Arm achieves an interactive search success of only 4%. PhyGS provides a hardware-abstracted interface mirroring Spot's SDK for development transfer between simulation and the physical robot.

The Task

An agent starts at a collision-free pose in an unseen home, with no prior map, and is given a natural-language description of a target (e.g. "a ceramic mug"). It must perceive, physically retrieve, and report the target — perceiving through onboard RGB-D and acting through the robot's low-level controllers. Each scene contributes four episodes along a progression of difficulty:

Separate room, visible. The agent must open doors to traverse between rooms, but need not open receptacles.
Same room, not perceivable. The agent uses semantic cues to localize the target and manipulates receptacles to retrieve it, without leaving the room.
Separate room, not perceivable. The agent must traverse between rooms and open receptacles over the course of its search.
Separate room, occluded. In addition, the agent must move non-target objects that block its view before the target can be perceived and retrieved.

The Dataset

25 procedurally generated single-story homes (two-, three-, and four-bedroom layouts, each with a bathroom, kitchen, living room, and dining room), populated with 50–200 unique gripper-scale objects from Objaverse-XL and articulated furniture — doors, cabinets, drawers, windows, ovens, dishwashers — set closed at episode start. All scenes are generated with PhyGS.

House-scale scenes

100

Episodes

Difficulty tiers / scene

Best agent success

A sample of physically-grounded, visually diverse homes generated by PhyGS and rendered in IsaacSim.

Agent interface

A high-level skill API matched to Spot's SDK. Methods may instead use the exposed continuous control directly.

navigate(target)
open_door(ref)
close_door(ref)
open_drawer(ref)
close_drawer(ref)
place(receptacle_ref)
report_found(object)

Metrics

TS — traversal success (reach the target's location within 0.6 m).
IOSS — interactive object search success (retrieve & report the target).
Skill execution success — share of skill calls that run without error.
SPL — success weighted by path length.
SPL-M — SPL extended to include manipulation effort.

Rollouts

On the physical Spot (2x speed)

Gemini Robotics ER-1.6 driving a Boston Dynamics Spot with Arm in a two-bedroom household. We ran three sample experiments spanning the benchmark's easiest tier (separate room, perceivable) and its second-hardest tier (separate room, not perceivable); the agent knew the target's location in every episode. The white-pillow task was run twice — once with the door latched closed, and again with it left open.

SuccessSearch & Grasp Blue Bowl

Separate room, perceivable. Starting in the living room — with the kitchen not visible — the robot navigated into the kitchen and approached the countertop, where the blue bowl was partially concealed behind a small green food container. It localized the target, removed the green box to gain manipulation access, then executed a reach-and-grasp to retrieve the bowl.

SuccessSearch & Grasp White Pillow (Door Open)

Separate room, perceivable. The door was left latched open — removing the need to turn the knob — but still mostly closed, blocking the view into the bedroom. The robot drove to the door, pushed it open with its manipulator arm, identified the white pillow, navigated to the bedside closest to it, and grasped the target.

FailureSearch & Grasp Orange Ladle

Separate room, not perceivable. Starting in the living room — with the kitchen not visible — the robot moved into the kitchen, located the drawer, and pulled it open with its arm, correctly identifying the orange ladle stored inside. The manipulation planner found a candidate grasp pose, but the constrained drawer layout kept the arm from reaching it, so the grasp failed after retries.

FailureSearch & Grasp White Pillow (Door Latched)

Separate room, perceivable. Here the door was latched closed, so the robot had to turn the knob and then push the door open. It reached the door and — after a retry — detected and grasped the knob, but could not exert enough force to keep contact while turning. The gripper slipped, so it never completed the traversal into the bedroom.

Results

We benchmark two vision–language models as high-level orchestrators over the skill library, under a known and an unknown target-location condition (each over all 100 episodes). Best per column in bold.

Orchestrator	Known target location				Unknown target location
Orchestrator	TS	IOSS	SPL	SPL-M	TS	IOSS	SPL	SPL-M
Qwen3-VL-8B	13.0	8.0	5.22	3.47	2.0	2.0	0.39	0.11
Gemini Robotics ER-1.6	15.0	11.0	6.08	4.64	7.0	4.0	0.55	0.35

All values are percentages. Higher is better.

Interactive object search is far from solved. Even given the ground-truth target location, the best agent completes only 11% of episodes; with an unknown location this falls to 4%. Traversal is the dominant bottleneck — 93 of 100 episodes fail before the final manipulation stage — and path-length-weighted scores fall below 1%, showing that successful episodes are reached via highly inefficient trajectories.

The breakdown below traces where the 100 episodes fail. Only 7 reach the target's location; of these, manipulation succeeds in just 4. Within the 93 traversal failures, navigation errors dominate — the agent reaches the wrong location (38) or a suboptimal final pose (17) — while a smaller share exhaust the step budget (10) or hit low-level control errors. See the paper for the full per-tier analysis.

Sankey diagram tracing how the 100 evaluation episodes split into traversal and manipulation outcomes and their failure modes. — Where episodes fail. A flow breakdown of all 100 episodes for the best agent: traversal success vs. failure, then the dominant navigation and manipulation failure modes. Reproduced from the ForageBench paper.

BibTeX

@misc{saha2026foragebench,
  title  = {ForageBench: A Photorealistic, Physically Grounded
            Benchmark for Interactive Object Search},
  author = {Saha, Aparajito and Gan, Zhen Hao and Guo, Jinjia and
            Skwirsk, Jacob and Acheampong, Jeremy and Arapin, Anton and
            Ku, Chahyon and Hu, Yue and Fazeli, Nima and Bucher, Bernadette},
  year   = {2026}
}