📽️ ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation

Jayjun Lee, Nima Fazeli
University of Michigan
RSS 2025
Paper Code (Coming Soon)

ViTaSCOPE is a neural implicit representation that is trained entirely in simulation to integrate vision and touch for in-hand object pose and extrinsic contact estimation.

ViTaSCOPE

How do we leverage complementary visuo-tactile feedback to infer extrinsic contact between grasped object and the environment?

Object module (O): A signed distance field (SDF) that represents the 3D object geometry.

Tactile module (T): A shear vector field that characterizes the extrinsic contact configuration.

Contact module (C): A contact field that represents the contact probability across the object surface.

ViTaSCOPE Architecture

Tactile module (T)

Tactile Module

Scaling up Visuo-Tactile Data in Simulation

We scale up visuo-tactile robot interaction data in simulation using the penalty-based tactile model using Nvidia's TacSL framework in Isaac Gym.

Zero-shot Sim-to-Real Transfer Results

We zero-shot sim-to-real transfer ViTaSCOPE to the real world and present some results below. (predicted, ground truth)

Acknowledgement

The authors would like to thank Andrea Sipos for manufacturing the Gelslim 4.0 vision-based tactile sensors for the Franka Emika Panda robot arm. We also thank William van den Bogert for his help with tracking marker displacements from Gelslim 4.0 to extract shear observations. We are grateful to Mark van der Merwe, Youngsun Wi, Miquel Oller, and Samanta Rodriguez for helpful discussions and important feedbacks. Finally, we would like to thank the authors of the TacSL paper [1] and NVIDIA for open-sourcing the framework and Isaac Gym for simulation.


[1] TacSL: A Library for Visuotactile Sensor Simulation and Learning

Team

Jayjun Lee
Jayjun Lee
Nima Fazeli
Nima Fazeli

BibTeX


      @inproceedings{vitascope2025,
        title={ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation},
        author={Jayjun Lee and Nima Fazeli},
        booktitle={Robotics: Science and Systems (RSS)},
        year={2025}
      }