Bridging 3D Generative and Agentic Models for High-Fidelity and Physically-Interactive Assets
1 Peking University2 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)3 Huawei4 Wuhan University
* Equal contribution‡ Project lead✉ Corresponding authors
CameraRotating & telescoping lens
Generating interactive 3D assets from a single image requires realistic appearance, fidelity to the depicted object, and plausible physical behavior. Yet detailed geometry alone does not determine how an object should move or deform. We introduce PhysAg3D, which couples 3D generative models with agentic reasoning and tool use. A vision-language agent first infers semantic parts, candidate interactions, and physical priors from the image; a 3D generator then produces a textured whole-object mesh from the same image. A hierarchical interaction representation links these revisable priors to mechanisms and the generated surfaces they control, organizing interaction construction and refinement. Guided by this representation and observations of the mesh, the agent uses geometric and physical tools to construct parts, joints, deformable regions, and surface bindings while retaining useful generated geometry and materials. Rendered interactions and execution measurements guide targeted revisions within a fixed budget. The framework accommodates both rigid articulation and compliant deformation. Quantitative and qualitative evaluations on real photographs and external benchmarks demonstrate detailed, image-aligned assets with plausible interactions. Ablations support bounded refinement and source-surface preservation.
In motion
30 generated objects in a shared studio, with Franka Panda demonstrations across 10 interactive assets.
PhysAg3D asset showcase: 30 generated objects, a Blender-built room, and scripted interactions with an open-source Franka Panda.
Explore the collection
223 assets. Hover to preview; click Interact to open each model in 3D.
Try another name or choose a different category.
50 studio-rendered cases and 186 additional generated examples. Clips show exported articulation, recorded deformation, or labeled approximate-compliance demonstrations. This gallery is a qualitative showcase, separate from the aggregate evaluation below.
PhysAg3D
A vision-language agent proposes semantic parts, interactions, and physical priors. A pretrained 3D generator independently produces a textured whole-object mesh from the same image. Their capabilities are coupled during construction and refinement, without joint training.
Appearance & interaction
Printer and chair examples compare PhysAg3D with aDSL and PhysX-Omni. A compliant asset shows initial, pressed, and released states.
Figure 3 in the paper. Selected states illustrate appearance and visible response.
Measured results
110 predefined semantic actions
Action Completion Rate
Function-and-Visible-Stability Pass
In-the-wild evaluation, Table 1. ACR is macro-averaged over objects; FVSP requires every expected action to complete without video-visible instability.
@misc{du_physag3d,
title = {PhysAg3D: Bridging 3D Generative and Agentic Models
for High-Fidelity and Physically-Interactive Assets},
author = {Du, Bi'an and Meng, Fanpeng and Fang, Jiemin and
Yang, Chen and Li, Pufan and Liu, Daizong and
Tian, Qi and Hu, Wei},
note = {Research manuscript}
}