The work of Antonín Vobecký and Josef Šivic at CIIRC CTU, who, in collaboration with Valeo.ai, focus on developing machine learning models for better data analysis and especially analysis of images from self-driving cars, has been a great success. Their methods, which will improve the safety and reliability of autonomous driving, were presented at one of the world’s most important machine learning conferences – NeurIPS 2023 in New Orleans, USA. The seven-day conference brought together more than 16,000 attendees from among the world’s leading researchers and experts in artificial intelligence.
In a recent paper, „POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images“ (/doi.org/10.48550/arXiv.2401.09413) researchers presented a method that processes 360° camera images of a car’s surroundings as input and produces a 3D semantic map of the occupancy of the surrounding environment. In addition, the user can ask the system in natural language about the location of objects in this digital 3D space. The capability to make queries in natural language allows for semantic occupancy segmentation or text-driven search in 3D, both performed exclusively from 2D images taken by the vehicle’s cameras. Questions asked might be, for example, „Where is the plastic trash bin?“ or „Where is the black truck with the trailer?“. This work was published at the NeurIPS 2023 conference, one of the world’s most important machine learning conferences.
The overall motivation is to develop methods to search for corner case situations, which are safety-critical but often very rare in the driving recordings. The goal of the research is to enable the retrieval of these captured situations in huge files of petabytes of data by simply entering instructions in natural language. Once such situations are found, they can be used as additional training data to improve existing machine-learning models and algorithms for autonomous driving so that systems can better deal with such situations and, as a result, improve the safety and reliability of autonomous vehicles.
Antonín Vobecký, who is behind the results of this research, is a PhD student in ELLIS Unit Prague at CIIRC CTU. He takes part in the industry track of the ELLIS PhD program where he closely collaborates with the Valeo.ai research team in Prague and Paris on challenging research problems motivated by real-world applications.
NeurIPS (Conference on Neural Information Processing Systems) is one of the most prestigious conferences in machine learning (along with ICLR and ICML) and is ranked among the top ten journals and conferences in all areas of science by Google Scholar. This year, NeurIPS received a total of 3,561 out of 13,321 submissions, and nearly 16,000 participants attended the conference.
This research was carried out as part of the EXA4MIND project, an EU-funded Horizon Europe project that is creating an extreme data platform that combines supercomputing and big data infrastructure to automate data analysis, management, and storage. In the EXA4MIND project, the team collaborates with one of the European automotive leaders Valeo to develop tools for large-scale management and analysis of driving data, which will in turn lead to safer systems used in autonomous cars.
Watch the interview with Antonín Vobecký about this research for EXA4MIND:
Faces of EXA4MIND | Antonín Vobecký (CIIRC CTU), machine learning researcher
ELLIS Unit Prague is based at the Czech Institute of Informatics, Robotics and Cybernetics of the Czech Technical University (CIIRC CTU). The unit brings together several lead scientists and their teams with the shared objective to address key scientific challenges in artificial intelligence and to generate innovations with a positive impact on the economy and society.
ELLIS – the European Laboratory for Learning and Intelligent Systems – is a pan-European AI network of excellence focusing on advancing science, technical innovation and societal impact. Founded in 2018, ELLIS builds upon machine learning as the driver for modern AI. It aims to secure Europe’s sovereignty in this competitive field by creating a multi-centric AI research laboratory.
The ELLIS PhD & Postdoc Program supports excellent young researchers by connecting them to leading researchers across Europe and offering a variety of networking and training activities, including summer schools and workshops. ELLIS PhDs and postdocs conduct cutting-edge curiosity-driven research in machine learning or a related research area with the goal of publishing in top-tier conferences in the field.
AICZECHIA is a Czech national AI initiative, which brings together, via leading Czech researchers in artificial intelligence, over forty academic teams and departments, mainly at universities and research institutions in the Czech Republic. Josef Sivic is a fellow of AICZECHIA.
