PhyGS: Physically-Grounded Controllable Scene Generation

¹University of Michigan, Ann Arbor

CVPR Workshop on Multi-Agent Embodied Intelligent Systems, 2026

Abstract

The pursuit of generalist robot policies requires evaluation spanning variations of visual environments and physical interactions intractable for a robot to execute in the real world. We introduce PhyGS, a simulation framework designed for the controllable generation of photorealistic, building-scale indoor environments with high-fidelity low-level physics that bridges the gap between procedural scene synthesis and robotic control. By extending Infinigen-Indoors within the IsaacSim ecosystem, PhyGS transforms static visual backdrops into fully interactable environments through automated object articulation, the assignment of rigid-body physical properties, and the integration of ray-traced lighting.

Unlike recent agent-based generative tools that rely on stochastic language models, PhyGS employs a rule-based approach to ensure the controllability required for rigorous benchmarking. Finally, we provide a hardware abstraction layer via a unified API and USD model for the Boston Dynamics Spot robot, demonstrating a seamless transition between simulated evaluation and physical deployment to help identify critical performance gaps in state-of-the-art generalist robotic agents.

Four capabilities, one pipeline

To train and evaluate generalist robots, a simulator must be extensible for diverse environment generation, photorealistic, physically interactable, and capable of continuous robotic control. PhyGS unifies all four by bridging procedural scene generation in Infinigen with control in IsaacSim and IsaacLab, and automating object population from Objaverse-XL.

Generation overview: constraints to floor plan and furniture, part-based articulation, and manipuland population.

Constraints → floor plan & furniture (Infinigen-Indoors) → part-based articulation → manipuland population (Objaverse-XL).

How it works

Procedural scene generation

Rule-based Infinigen-Indoors builds architectural elements and large furniture within OpenUSD; config and constraint files keep arrangements physically plausible and accessible — controllable enough for standardized benchmarking.

Automated articulation

A custom exporter intercepts articulated assets (doors, cabinets, drawers) and re-injects their joints, geometry, and material properties — which the standard Blender→USD bake would otherwise collapse into static meshes.

Semantic object population

Manipulands from Objaverse-XL are placed by semantic and volumetric relevance, then given collision meshes and rigid-body physics in IsaacSim (baseline: 0.8 static friction, 1.0 kg targets — within Spot Arm's 7 kg payload).

Hardware abstraction

A unified USD model and API for the Boston Dynamics Spot-with-Arm that mirrors the official SDK, so policies developed in IsaacLab deploy to the physical robot with minimal modification.

From Blender to a physics-ready USD

Infinigen renders in Blender, but the default export to simulators discards articulations, collision meshes, and lighting. PhyGS's exporter intercepts articulated assets, adds joint parameters, gives every asset mesh colliders and rigid-body properties, and corrects the lighting scheme to retain visual fidelity.

Export process: Blendfile, intercept articulated assets, add mesh colliders and rigid-body parameters, scene USD with physics.

Blendfile → intercept articulated assets → add mesh colliders & rigid-body parameters → scene USD with physics & interactable objects.

BibTeX

@misc{saha2026phygs,
  title  = {PhyGS: Physically-Grounded Controllable Scene Generation},
  author = {Saha, Aparajito and Gan, Zhen Hao and Guo, Jinjia and
            Skwirsk, Jacob and Acheampong, Jeremy and Arapin, Anton and
            Ku, Chahyon and Hu, Yue and Fazeli, Nima and Bucher, Bernadette},
  note   = {CVPR Workshop on Multi-Agent Embodied Intelligent Systems},
  year   = {2026}
}