Multi-/Omni-modal Models -- Part One (Setting the scene)

April 24, 2026

A significant amount of effort is being made across research and industry to expand on the abilities of large language models (LLMs); towards general machine ability for the understanding/perceptione.g. to ingest an image and make sense of it by providing classifications of it, answering some questions about it etc… and generatingthis is the generative element of Generative AI, e.g. to produce some new information such a text, video or speech signal. in other modalities (i.e. vision and speech). The public’s imaginings of such capabilities were piqued by two widely-watched demonstrations made in the summer of 2024;These were by OpenAI (GPT-4o) and Kyutai AI (Moshi). They were rolled out against a backdrop of growing public fascination with conversational AI (e.g. ChatGPT), and fed imaginations of how we could also interact in real-time with such AI systems by taking in audio and/or visual input, receiving audio and/or visual output we can get more naturalistic interfaces with these AI models. There has also been a widening acknowledgement of the need to ground LLMs in the physical world in order to achieve more general capabilities; see (Koller & Bender, 2020) since then, two further (and entwined) trends have emerged for these multi-/omni-modalThe term ‘omni’ is used for models that can take input in more than one modality and generate output in more than one modality, see for instance InclusionAI’s (Ant Group, Alibaba) Ming-flash-omni 2.0 and LLaDA2.0-Uni. LLMs (MLLMs; also often appearing under the umbrella terms of “Embodied AI”/”Physical AI”), and they are reinforcing the raisons d’etre of such multi-/omni-modal models.

In this post, I will briefly describe these MLLMs, try to convince you why paying at least some attention to them is a good idea, and then walk through how they are used in robotics and autonomous driving so as to give a more concrete sense of them. I also take a slight detour over an article about simulation and modeling that I really enjoyed recently, and which I found helpful to contextualise the World Models (see below) being actively pursued today. Finally, I lay out some areas I plan to write about in this space over a few more posts. For some reason I ended up adding quite a fair bit of videos clips to this post; on mobile you might have to tap the + signs to reveal them. I hope they add to the arguments I set out to assemble in this post.

2️⃣ 1. What two trends?

They are: vision-language-action (VLA) models and world models (WMs). The former (VLA) is relatively easier to define: it is essentially an LLM that can take instructions and reason (e.g. in Chain-of-Thought) together with visual input (e.g. real-time video) and typically the most important output is some set of information that can be used for controlling a robot’s movements (instead of just receiving an answer in text as we are used to with LLMs).This could be coordinates (e.g. \(x,y,z\) in physical space) for some set of robotic joints to move into resulting in some step/action being made. Visualise it like this: think of your brain sending a message to your elbow: ‘move to be horizontally up by 5 cm, rotate 10 degrees outwards…’ together with similar coordinated messages to other joints in your body, so as to fold a piece of clothing. For more details and a survey of VLA models, see this nice post by Moritz Reuss on State of VLA Research at ICLR 2026. Have a watch of the following video of the \(\pi_{0.5}\) model from Physical Intelligence for some idea of the perception and generation processes involved.

The latter (WM) is essentially a model that can give a useful prediction of some future outcome(s) given some existing conditions/state. It is an active area of research at the frontier, with contesting visions (though possibly eventually complementary) even of what the definition of the term “world model” should be; as well as what one such model should be able to do and what forms of predictions (e.g. sequence of robot actions, some group/societal responses etc) it ought to be able to give. This post touches on one form (video models, see section below on VLAs for robots and autonomous driving); A list of well-written overviews of the different visions can be found in the sidenote here.See this categorisation of the contesting visions of WMs by @zhuokaiz on X.com, to which I will add: (1) the symbolic-based effort of Moonlake led by Christoper Manning, and (2) the notions of some kind of social world models (Zhou et al, 2025). There is also interest for a Computer-Using World Model, which will help contextualise this recently announced move by Meta. For a deeper context on WMs, see also see the oft-cited 2018 work by Ha (now of SakanaAI) & Schmidhuber and the latter’s recent personal contextualisation of the subject.

🤷‍♂️ 2. Why should we care about these models?

MLLMs can also have applications beyond the ones above (that of interacting with AI models or for robots/vehicles to think, plan and act). These models (defined as being able to understand and/or generate over two or more modalities e.g. vision-language) look set to touch many aspects of our daily lives.

We have heard of how generative AI has the potential to bring advancements to healthcare, education, public safety, and MLLMs will expand those spaces. Less visible are the ways they can also be used to speed up scientific discoverySee the MicroVQA benchmark at CVPR 2025 put together by trained biologists for VQA over a highly challenging suite of tasks involving (1) visual understanding of microscope images, then (2) generating hypotheses over the observations from the images, and then (3) experiments proposal. The key insight are results (Table 3) on the benchmark showing that Claude 3.5 Sonnet can reach levels for tasks (1) and (2) that outperform a human baseline (from answers of trained biologists) on the benchmark, and near-comparable levels for task (3). (Skip this if you don’t need the details) See Appendix F.3 for the details, but these were essentially biologist co-authors of the paper but who may not have specialised sub-domain knowledge to execute these tasks (reasonable for such a large domain requiring highly specialised skills). Nonetheless, rather impressive results for VLMs. and open up new (or make feasible) ways of studying and learning about the world around us.In the physical sciences but also for instance, the work of the Earth Species Project; it was really inspiring to hear their ground-breaking vision from one of its co-founders during her keynote at a conference a few years back. See also the next video (MammAlps) for other ideas in a similar vein.

⊕
A clip from the MammAlps video dataset (Gabeff et al, 2025) of wild animals in the Alps; I came upon it looking for examples in the rather neat CV4Animals workshop.

There is also research ongoing to add capabilities such as personalisation and memory, for e.g. (Nie et al, 2026), that will give rise to personal productivity applications that are more embedded and “sticky”, which will impact us as consumers of these future products. Moreover, and unfortunately, every reasonable expand-the-pie application for MLLMs (e.g. ones that create opportunities, expand safety, widen care) could quite easily be extended towards less expansive applications (watch the Falcon Perception clip below, and you should be able to see it with a little imagination). ⊕
Source: @MaziyarPanahi on X.com; showing a third-party prototype/system demo built on the compact-sized Falcon Perception model recently launched by the UAE’s Technology Innovation Institute.

Given the potential wide-ranging reach of such models (and AI in general), the more that we each have an understanding (at the minimum, to make informed personal decisions for individual flourishing), and participate in discussions of what these technologies might look like, including who builds them, why and how they build and run them this way,For that, have a look at this well-put together series of Stanford lectures involving big tech and VC leadership. the better-positioned we are to ask for and construct a meaningfully inclusive future.Sidenote: a list of potentially interesting books along this direction can be found here.

🧱 3. Give a concrete feel about these ideas?

The Physical Intelligence \(\pi_{0.5}\) video above would have given a sense of how language and vision come together in the VLA for robotics use (user instruction understanding, task decomposition, reasoning and real-time video of the robot’s environment). Here, I will widen/narrow our field of vision (depends on how you want to see it) and step away from MLLMs for a bit. By looking into video-only generative models, we can get a look into the idea of WMs. In particular, I will examine uses in robotics and autonomous driving.There are VLA solutions for these, in autonomous driving, there is Nvidia’s Alpamayo-R1/R1.5 models.

Let’s start with robotics, with Large Video Planner proposed by (Chen et al, 2025). Think of this as Chain-of-Thought and task decomposition that we are used to with LLMs, but in a visual way. The following description and graphic explains what the video planner does and how robot controls are obtained from its output.

The action extraction referred above essentially leverages methods from “classical” computer vision such as pose or optical flow estimation, and can be rather involved (see §3.3 of their paper for how they go from predicting human hand movements to obtaining robot hand control coordinates) especially when requiring fine-grained control for hand dexterity here.The benefit of (Chen et al, 2025)’s approach is that there is a lot of human hand action video that can be collected from the web (i.e. YouTube) which can be used to train some world model that can predict hand actions to complete a task.
Sidenote: That said, there is much more to it and many edge cases to resolve for robotics, see this VC piece on robotics world models for more insights.
(Skip this if you don’t need the technical details)
In comparison, (Zhen et al, 2025) use optical flow to obtain depth and normal information that is then used to generate 4D (where the 4th ‘D’ refers to time) point cloud. An ‘inverse dynamics model’ (see §4.4 of their paper), an MLP that takes 4D point cloud at two frames (\(t\) and \(t+1\)) where \(t\) would be the current state, and \(t+1\) is the predicted state generated by the video planner. The inverse dynamics MLP gives the information for the robotic control, the degrees-of-freedom (DoF) coordinates.
Sidenote: For a sense of what a 4D point cloud is (as compared to 3D point clouds we might have come across), see the visualisation here. Click on the ‘Point Cloud Reconstruction’ tab. The many dots in the right part of the visualisation is the point cloud which capture position of the surfaces of the scene over 3D. Notice the square box moving too? That is the camera plane which is moving over time, so essentially the representation captures 4D spatio-temporal information. This type of point clouds would be useful for autonomous driving or site surveyance; similar 4D point clouds can be constructed for human-object interaction (HOI) data in robotics (see for e.g. this).

Similar ideas extend to self-driving, where the task is arguably trickier (changing light, rapid shifting of scene elements and occlusions etc). Here we take a look at the VISTA approach (Gao et al, 2024) as a reference approach. The model is intended to generate the next frame of a driving scene given the current frame (i.e. what the scene with its elements is expected to look like at some later, usually immediately after, timestep), and the application is slightly different from Large Video Planner above. It is used to generate videos with different conditioning (e.g. different prompts) which are then used for evaluating the quality of possible alternatives (what might happen should the car turn left, right, keep driving forwards etc). The purpose of such conditional video generation is to inform driving decision-making.For e.g. a judge model is used to assess the predicted scenes in ‘turn left’, ‘drive straight’ and ‘turn right’ options, and the safest (or most dangerous) option is selected (or avoided) by the self-driving controller. Some generated examples for VISTA can be seen below; if you look closely you can see various generation artifacts in the videos i.e. there is much work to get to realism.

⊕

⊕

⊕
Examples from VISTA: (Gao et al, 2024); see also (Wang et al, 2023)

Since then, there are now leaps in video generation quality, including very recent work leveraging near ubiquitous Google Street View data in major cities – see (Seo et al, 2026) and (Chou et al, 2026). Take a look below at the SimCity-like disaster videos from Seoul World Model (Seo et al, 2026).

⊕
Seoul World Model example, for text-conditioned generation of scenes with different conditions like street flooding, or 🛸a UFO! or 🦖Godzilla! etc, which is fantastic for the counterfactual in simulation. On that, I found this 🔥post by Benjamin Recht very helpful.

This should give a grounded sense of why there is the heavy interest for high-fidelity video world models.For Large Video Planner (Chen et al, 2025), the robot controls are extracted with classical CV methods, the better the quality of the generated video (details, consistency etc) the better the controls. Likewise, for autonomous driving, better quality (finer-grained details etc) videos of counterfactual states give better assessment of the best action to take. With this, hopefully the next time you see/hear the phrase “world model”, you can situate better what the video model vision of it is about.This is also what the authors of the 🔥(Vision Banana) technical report from Google Deepmind are likely referring to when they say: ‘Similarly, investigating whether video generators yield even richer, temporally-aware visual representations presents a highly promising research direction.’ Their paper is worth a read: upending thinking about modeling in computer vision, showing slightly tuned image generative model performs comparably or better than the specialist models for important vision tasks, such as the very carefully designed and trained SAM3.

4. 🔭 Contextualising and imagining futures

In fact, these generated driving videos are in a lineage of simulation that can be traced back to the last century (de Monchaux & Catalogtree, 2026). Take a look at the following “driving” video that was shot in the 1960s at the Berkeley Environmental Simulation Lab; source: (de Monchaux & Catalogtree, 2026). The marvel is that it was shot on meticulously made scale models in the 1960s, with technology, techniques and personalities that sit on a historical arc in engineering that span the Panama Canal, military radar systems and films such as 2001: A Space Odyssey and Star Wars.I highly highly recommend reading this wonderfully written and researched piece by (de Monchaux & Catalogtree, 2026) which was just published in Antikythera, itself a very fascinating journal. The article traces the arc of engineering’s interest (cybernetics, optimal control) in models for simulation, which if you look closer can see clear lines to what is being sought for in WMs. This contextualisation is valuable for perspective-taking and seeding questions for inquiry. I do hope you take some time to read the conclusion, perhaps revisiting a few times as I did, as each time might draw out further insights and connections for you.

Successful prediction is predicated on predictability. On this note, there is another thread emerging, calling for notice as to how these technology are starting to shape what our experiences will be.In this aspect, the Artificiality Institute, and the questions they are seeking answers to, are quite thoughtful.

⊕
Successful prediction is predicated on predictability. Source: Mathieu Blazy’s Chanel, Spring/Summer 2026. Watch as the models flow by in a steady unperturbed stream. Look out for Awar Odhiang (she is the last model in the file) at around 50 seconds into the clip. This moment is memorable for the purity of her joy and her ability and choice to depart from the established norms of the fashion runway. There is great beauty in the flow too. In fact, much work is needed to maintain the states (as it is in fashion with the petites mains, the atelier hands) that allow moments of such flow and for such joy to be reached. When optimising only for the flow (so easy to predict, so easy to measure!), do we narrow the space for those other moments that colour the palette of the human experience? When building is becoming so accessible so rapidly with natural language, and we can now start building what we dream up of, what we choose to build and how it is built to work matters. If you are a builder, perhaps the ethos of the Cosmos Institute might resonate with you as well (see also, The Philosopher-Builder).

Here, I would like to suggest that we might be well-served to ask how these technology(ies) can be developed/calibrated to make us all better and not worse off. Many of us cannot directly shift the arc of where we land in the future, but we can make individual informed decisions regarding our collective hopes/wishes/desires/plans that translate into attention, public discourse and quotidian actions that collectively tilt the arc. We could ask for (especially as consumers) where we wish these technologies to be and whose wellbeing they need to serve; it is better to do this sooner rather than later. In the sprint and hustle to build fast in the face of such intense competition, the path of least resistance is often key to survival; our silence is acquiescence, and systems once in place carry weight, energy and momentum of their own.

5. 👉 What’s next? What will be covered?

I hope there was something useful for you in this post. My plan is to write a series of posts on ideas around these MLLMs. Researching and writing them help me to sharpen my understanding, views and thoughts about the topics, and sharing them are my way of chipping in/adding to the virtuous cycle of knowledge, from which I have benefited so much from all, thanks to the thoughtful guidance and sharing of others.I spent the last four months doing the courses in this short NUS-ISS programme as a way to deepen my understanding of the modalities other than text (vision: image & video, as well as some audio: speech).
Sidenote: It had been very intellectually rewarding Saturdays in the classes by very qualified lecturers (with deep and up-to-date research and industry experience). The courses are part of an elective in the Master of Technology in Artificial Intelligence Systems (I took the module as a continuing education student) and it has been wonderful to learn in a classroom of highly-motivated experienced professionals from various sectors and areas in tech (making classroom discussions very rich). If you are looking for a post-graduate AI programme (or talent), I do recommend taking a look there. I do believe that language remains a wondrously rich system for storing and transmitting knowledge of our world,Some prominent computer vision researchers do not think it is necessarily so – for e.g. see here and here. They provide good explanations for why they think so, and I can see why they do. But it is hard to escape language, there is just so much encapsulated in it. For instance, listen to how language was quite important for improving image generation models like FLUX.1 Kontext. Also some concepts like ‘freedom of speech’ or ‘rule of law’ seem to require language to be encoded for transmission, as acknowledged in (Huh et al, 2024). as well as for interactions between humans as well as with machines, so I will cover this topic with connections to language throughout. I will also add some ideas I track in the humanities and the arts which interests me greatly. My intent is to draw out and connect between different perspectives that can spark curiosity, understanding and ideas in the machine understanding and generating of multimodal information.

The next post will remain high-level but start getting a bit more technical (e.g. looking into the Platonic Representation Hypothesis (Huh et al, 2024) and the emerging body of work around it to help our understanding of the ways to connect between the different modalities) and which I will try to balance any dryness with some meandering on art that I enjoyed. Following that will be at least one other post that will take a closer look into some model internals (vision-language models, VLMs; which VLAs are built upon) and examine some conditions/settings that could have impacts on safety.For e.g. how a reliance on realism in some models to secure better predictions will almost certainly at some point come up against usage in ways that shifts/weakens our experiences of reality.