Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation

Martina Lippi*, Petra Poklukar*, Michael C. Welle*, Anastasiia Varava, Hang Yin, Alessandro Marino, and Danica Kragic

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable object. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the effectiveness of the method on a simulated box stacking task as well as a T-shirt folding task performed with a real robot.

*Contributed equally and listed in alphabetical order

Download Paper

Method

Our Method:

Use the following to cite us:

@inproceedings{lippi2020latent,
title={Latent Space Roadmap for Visual Action Planning of Deformable and Rigid Object Manipulation},
author={Lippi, Martina and Poklukar, Petra and Welle, Michael C and Varava, Anastasiia and Yin, Hang and Marino, Alessandro and Kragic, Danica},
booktitle={IEEE/RSJ International Conference on Intelligent Robots and Systems},
year={2020}
}

Folding execution videos

Fold 1

Start state

Goal state

Execution videos fold 1

Fold 2

Start state

Goal state

Execution videos fold 2

Fold 3

Start state

Goal state

Execution videos fold 3

Fold 4

Start state

Goal state

Execution videos fold 4

Fold 5

Start state

Goal state

Execution videos fold 5

Code Repository

The code used to train the vae, build the graph in the latent space, and the action proposal network including all used hyperparameter can be found on the gitrepo:

Code Repository

Contact

  • Martina Lippi; lippi(at)kth.se, mlippi(at)unisa.it; KTH Royal Institute of Technology, Sweden and University of Salerno, Italy
  • Petra Poklukar; poklukar(at)kth.se; KTH Royal Institute of Technology, Sweden
  • Michael C. Welle; mwelle(at)kth.se; KTH Royal Institute of Technology, Sweden
  • Anastasiia Varava; varava(at)kth.se; KTH Royal Institute of Technology, Sweden
  • Hang Yin; hyin(at)kth.se; KTH Royal Institute of Technology, Sweden
  • Alessandro Marino; al.marino(at)unicas.it; University of Cassino and Southern Lazio, Italy
  • Danica Kragic; dani(at)kth.se; KTH Royal Institute of Technology, Sweden