The paper by our researchers, Gabriela Šejnová, Karla Štěpánová and Michal Vavrečka, has been accepted to one of the world’s most important robotics conferences – IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2024, which will take place from 14 to 18 October in Abu Dhabi.
The scientific paper is entitled „Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks“ and addresses a key topic in the development of intelligent autonomous robots. The researchers focus on how to teach robots to perform complex manipulation tasks based on language commands, visual perception and demonstrated movements. The goal of the research is to achieve a higher level of autonomy for robots that can recognise different objects and perform new tasks with them based on linguistic instructions, without the need for pre-programming.
Research methodology
In this paper, the authors present an innovative use of multimodal variational autoencoders (VAEs) that can combine different inputs from language, image and robot motion. Each modality is processed by a separate deep neural network, and this processed data is then fused into a common space. This common latent space allows the robot to efficiently analyze each task, which includes recognizing objects, executing the desired motion, and then interacting with the environment.
During the training process, the robot receives a variety of training data, such as instructions like „move the cup to the left“ or „put the apple in the drawer“. Using this data, it gradually learns to identify which parts of the command relate to objects and which relate to motion actions. This technology allows robots to perform simple tasks, such as moving objects, as well as more complex tasks that involve multiple steps.
Practical results
In their paper, the authors showed that robots can be taught in this way, for example, to manipulate objects such as „pick up an apple“ or more complex multi-step tasks such as „put an apple in a drawer“, where multiple motion steps (opening the drawer, picking up the apple and closing the drawer) need to be combined. These advances open up new possibilities for robotics applications, especially in the area of autonomous robotic systems that will be able to respond to different stimuli from the outside world and adapt to dynamic environments.
Video – https://www.youtube.com/watch?v=IVaDvEv0oys
IROS 2024 Conference
The IROS 2024 conference, with the main theme „Robotics for Sustainable Development“, is one of the most important gatherings of scientists, engineers and experts from all over the world in the field of intelligent robots and systems. Held at the prestigious ADNEC Centre in Abu Dhabi, it brings together the world’s robotics elite each year to share the latest findings, technological innovations and future visions
Congratulations to Gabriela Šejnová, Karla Štěpánová and Michal Vavrečko on this significant achievement!
The paper by our researchers, Gabriela Šejnová, Karla Štěpánová and Michal Vavrečka, has been accepted to one of the world’s most important robotics conferences – IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2024, which will take place from 14 to 18 October in Abu Dhabi.
The scientific paper is entitled „Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks“ and addresses a key topic in the development of intelligent autonomous robots. The researchers focus on how to teach robots to perform complex manipulation tasks based on language commands, visual perception and demonstrated movements. The goal of the research is to achieve a higher level of autonomy for robots that can recognise different objects and perform new tasks with them based on linguistic instructions, without the need for pre-programming.
Research methodology
In this paper, the authors present an innovative use of multimodal variational autoencoders (VAEs) that can combine different inputs from language, image and robot motion. Each modality is processed by a separate deep neural network, and this processed data is then fused into a common space. This common latent space allows the robot to efficiently analyze each task, which includes recognizing objects, executing the desired motion, and then interacting with the environment.
During the training process, the robot receives a variety of training data, such as instructions like „move the cup to the left“ or „put the apple in the drawer“. Using this data, it gradually learns to identify which parts of the command relate to objects and which relate to motion actions. This technology allows robots to perform simple tasks, such as moving objects, as well as more complex tasks that involve multiple steps.
Practical results
In their paper, the authors showed that robots can be taught in this way, for example, to manipulate objects such as „pick up an apple“ or more complex multi-step tasks such as „put an apple in a drawer“, where multiple motion steps (opening the drawer, picking up the apple and closing the drawer) need to be combined. These advances open up new possibilities for robotics applications, especially in the area of autonomous robotic systems that will be able to respond to different stimuli from the outside world and adapt to dynamic environments.
Video – https://www.youtube.com/watch?v=IVaDvEv0oys
IROS 2024 Conference
The IROS 2024 conference, with the main theme „Robotics for Sustainable Development“, is one of the most important gatherings of scientists, engineers and experts from all over the world in the field of intelligent robots and systems. Held at the prestigious ADNEC Centre in Abu Dhabi, it brings together the world’s robotics elite each year to share the latest findings, technological innovations and future visions
Congratulations to Gabriela Šejnová, Karla Štěpánová and Michal Vavrečko on this significant achievement!
