SemanticPaint: Interactive Segmentation and Learning of 3D Worlds

Emerging Technologies

 

  View Video

SemanticPaint: Interactive Segmentation and Learning of 3D Worlds

SemanticPaint is a real-time, interactive system for geometric reconstruction and object-class segmentation of 3D worlds. With this system, a user can walk into a room wearing a consumer depth camera and a virtual reality headset, and reconstruct the 3D scene (using hashing-based large-scale fusion) and interactively segment it on the fly into object classes such as "chair", "floor", and "table". The user interacts physically with the scene in the real world, touching objects and using voice commands to assign them appropriate labels. The labels provide supervision for an online random forest that is used to predict labels for previously unseen parts of the scene. The predicted labels, together with those provided directly by the user, are incorporated into a dense 3D conditional random-field model, over which mean-field inference is performed to iron out label inconsistencies. The entire pipeline runs in real time, and the user stays "in the loop" throughout the process, receiving immediate feedback about the progress of the labelling and interacting with the scene as necessary to refine the predicted segmentation.

Stuart Golodetz
University of Oxford

Michael Sapienza
University of Oxford

Julien Valentin
University of Oxford

Vibhav Vineet
Stanford Univeristy

Ming-Ming Cheng
Nankai University

Victor Prisacariu
University of Oxford

Olaf Kahler
University of Oxford

Carl Ren
University of Oxford

Anurag Arnab
University of Oxford

Stephen Hicks
University of Oxford

David Murray
University of Oxford

Shahram Izadi
Microsoft Research

Philip Torr
University of Oxford