A new chapter in European Artificial Intelligence (AI) research begins with the launch of ELLIOT – European Large Open Multi-Modal Foundation Models For Robust Generalization On Arbitrary Data Streams. This four-year research project is funded under the Horizon Europe programme with a €25 million grant and brings together 30 leading organisations from 12 European countries. The goal is to pioneer the next generation of trustworthy, general-purpose AI models capable of working with diverse real-world data. From the Czech Republic, the Czech Technical University in Prague (CTU), specifically the Czech Institute of Informatics, Robotics and Cybernetics (CIIRC), and the deep-tech innovation start-up RoboTwin are participating. CIIRC CTU will act as the main Czech reference point for communicating the project’s results to both expert communities and industrial stakeholders.
At the heart of ELLIOT is the development of open Multimodal Generalist Foundation Models – AI systems designed to learn general knowledge and patterns from massive amounts of data of various types, including texts, images, videos, satellite imagery, sensor signals, and industrial time series from production environments. The aim is to efficiently transfer the generic knowledge learned in a generalist manner to a wide variety of downstream tasks, from robotics and autonomous driving to environmental prediction. Unlike current foundation models, which face significant challenges in terms of generalisation capabilities and multimodal data integration, the models developed within the ELLIOT network will be capable of robust generalisation and processing of multiple data types simultaneously. Crucially, they will be able to adapt to new, complex, and dynamic real-world scenarios – such as vehicle navigation, drone-based environmental monitoring, or robotic control – involving high levels of noise and significant environment variation.
CIIRC CTU is involved in a number of key research tasks within the ELLIOT project, taking the lead on several of them. The focus is on fine-tuning new large-scale multimodal foundation models to real-world scenarios and on the associated testing and evaluation of their capabilities in practical applications. A key area is robot perception, where models process human instructions given in the form of language, gestures, or images in combination with data from cameras and sensors, and translate these into actionable commands for robots. Another major area is the security and robustness of these models – ensuring that models can resist various types of attacks, such as manipulation of training data. CIIRC will contribute to the development of assessment tools that verify technical reliability and compliance with ethical guidelines and the new European AI legislation (AI Act).
„Today’s generative AI models – like ChatGPT or DALL-E – are already highly advanced and widely used by the public for text and image generation,” explains Karla Štěpánová, head of the Robotic Perception Group at CIIRC CTU. “However, most of these models still struggle to effectively process other key input modalities relevant to human perception and decision-making, such as gestures or tactile feedback. They also often lack the ability to understand changes over time, such as the motion of a robot in response to a dynamic environment, the evolution of a manufacturing process, or changes in the landscape in satellite imagery. ELLIOT aims to bridge this gap with spatial-temporal multimodal models.”
Researchers from CIIRC CTU are also contributing to a number of additional tasks, including model testing, fine-tuning, and optimization in terms of computational efficiency and energy consumption. The team brings extensive experience in using both national and European supercomputing infrastructures, having used more than 400,000 GPU hours over the past year on the Karolina supercomputer in Ostrava and LUMI in Finland, one of the most powerful systems in the world.
“Our teams will also focus on developing tools that strengthen the resilience of these new AI systems to cyberattacks and help increase their overall safety,” adds Vladimír Petrík, a robot learning and perception expert from Josef Sivic’s IMPACT group at CIIRC CTU. “Equally important is the role of our institute in transferring and adapting foundation models to specific application domains, such as robotics manipulation, in which CIIRC will collaborate with RoboTwin.”
RoboTwin will be primarily involved in the part of the project focused on collecting and generating domain-specific data essential for adapting foundation models to real-world scenarios like robotic assembly, object manipulation, and automation of manufacturing operations. The RoboTwin team will also validate the adapted models through real-world industrial robotic applications. With its focus on robotics and AI innovation, RoboTwin will help translate ELLIOT’s research outcomes into industrial practice and expand the use of foundation models in industrial robotic systems.
ELLIOT will closely cooperate with European and international open-source and open science communities like LAION and open-sci, benefiting from the strong expertise on open foundation models and datasets necessary for their creation established in those networks. These tight links will ensure staying on the frontiers of fast-progressing research and making the whole pipeline for foundation model research and development – dataset composition, training, fine-tuning, and evaluation – fully open and reproducible, such that it can be validated and easily adapted for specific industry and public sector needs.
“ELLIOT will significantly strengthen European sovereignty and independence in Artificial Intelligence, particularly in the rapidly evolving area of Multimodal Foundation Models. By leveraging Europe’s scientific excellence, computational infrastructure, and commitment to open and trustworthy innovation, ELLIOT is a major step toward ensuring that Europe actively shapes its own future in AI”, says Dr Yiannis Kompatsiaris, ELLIOT’s Coordinator at the Information Technologies Institute (CERTH-ITI).
Leveraging Europe’s world-class supercomputing infrastructure – including EuroHPC supercomputers like JUPITER, Leonardo, MareNostrum, LUMI, as well as the Alps supercomputing infrastructure of the Swiss National Supercomputing Centre – ELLIOT will combine real and synthetic data from public internet and trusted sources, including European Data Spaces, to train and deliver a family of open trustworthy multimodal generalist foundation models along with open datasets and methods. This will enable the AI community in Europe and worldwide to build, study, deploy, extend, and evaluate such models and the whole pipeline necessary for their creation in a fully reproducible manner at a level and scale that is currently not available in open public domain. In that way, ELLIOT will act as a catalyst for open-source and open science in AI, and substantially strengthen Europe’s Sovereign AI vision.
The project’s results will also enable groundbreaking applications and give Europe a competitive edge in fields such as media, Earth modelling, robotic perception, autonomous driving, computer engineering, and workflow automation. Key research activities include community building and training the next generation of European researchers in machine learning and AI, building on the internationally renowned ELLIS Society (European Laboratory for Learning and Intelligent Systems), whose Czech node ELLIS Unit Prague is hosted and led by CIIRC CTU. This will result in a complete ecosystem of European AI: built in Europe, for Europe, and adhering to European values.
The consortium is coordinated by the Information Technologies Institute of the Centre for Research and Technology Hellas (CERTH-ITI) in Greece and includes powerhouse academic and research institutions, alongside innovative SMEs, public stakeholders, and non-profit organisations with diverse expertise.
