Robot Learning with Semantic and Physical Grounding

This is Zhengtong's PhD defense on Oct 2026 Abstract: Robot learning must connect task meaning to physical action. Manipulation requires understanding objects, relationships, and goals, while also responding to contact and generating motions consistent with physical constraints. This dissertation studies complementary forms of semantic and physical grounding across control, representation learning, visual prediction, and task planning. For physical action generation, differentiable model predictive control connects tactile observations to reactive grasping. A differentiable trajectory optimization layer first supports constrained visuomotor policy learning and is then extended to input-conditioned refinement of base-policy trajectories. For interaction perception, data-efficient tactile representation learning supports transfer to perception and manipulation, while systematic visuotactile evaluation examines the effects of sensing modalities, representations, and policy architectures. For learning from visual transitions, latent video planning predicts future observations and translates them into robot actions. RLA-WM learns residual latent actions that represent visual transitions in feature space for world modeling and policy learning. At the semantic task level, a planning-language interface connects vision-language reasoning, executable robot skills, and verification, allowing the task representation to be updated when observed execution differs from its intended effects. Together, these findings emphasize matching learned representations and decision mechanisms to the information, constraints, and feedback required at each level of manipulation. Semantic and physical grounding together provide an organizing perspective for developing robot learning methods whose task understanding remains connected to the consequences of action.