Spatial Intelligence: Why AI is starting to really understand the world

Since the launch of ChatGPT at the end of 2022, the development of artificial intelligence has known no speed limit or even traffic jams. We have long since become accustomed to AI writing texts, generating images and videos, and being able to handle human voices. But despite all these capabilities, there is still one crucial limitation: AI tools operate in a flat, two-dimensional world. With spatial intelligence, however, they are now supposed to learn to understand the world in all three dimensions.

AI tools such as ChatGPT, Claude, Gemini and Mistral do not understand what gravity means, how objects relate to each other spatially, or how to move around in a room. They only know two dimensions and still have to learn to comprehend the 3D world in which we humans live.

What is spatial intelligence?

This important evolutionary step that AI still has to take is called ‘spatial intelligence’. This refers to the ability of AI systems to understand three-dimensional spaces with all their physical laws, to navigate within them and to interact with them.

To enable this step in development, the database must first be expanded. While well-known large language models (LLMs) such as ChatGPT have been trained with text, spatial intelligence also requires information that enables a comprehensive understanding of the physical world. AI developers therefore also refer to ‘‘large world models’’ (LWMs).

Figuratively speaking, they should not only be able to recognise and distinguish between objects such as a ball and a feather, but also understand how they behave in a space based on their physical characteristics. While it is perfectly logical for an adult human to understand that a ball falls to the ground faster than a feather when dropped, AI must first be trained in this logic using the new LWM.

From a technical perspective, spatial intelligence is based on the fusion of various data sources, including camera systems, lidar sensors, radar and other spatial sensors. If this sounds familiar, it's because these sensor systems already play an important role in the development of autonomous vehicles. Here, too, the aim is to give driving systems as comprehensive a view of the environment as possible. In other words, cars should learn to see in a similar way to humans.

Where spatial intelligence becomes a game changer

The practical applications of spatial intelligence are already promising. Especially in conjunction with other technological developments, scenarios are emerging that we have previously only seen in science fiction films.

Let's take robotics, for example. Scientists have already made significant progress in the development of humanoid robots in recent years with the help of AI. However, one of the biggest challenges remains imitating the upright gait of humans. Optimised sensor technology and increased computing capacity have driven development forward, but there is still a long way to go before humanoids can walk smoothly like humans.

Supported by ‘‘spatial intelligence,’’ further progress could be made in this area. Humanoid robots could then move like humans and interact with their spatial environment and the objects in it. Together with the further development of communication skills through generative AI, this opens up a wide range of application scenarios that know virtually no limits.

Humanoid robots with motor skills that are significantly closer to human capabilities could be useful in healthcare, among other areas. They could take on assistance tasks in hospitals and relieve staff by performing time-consuming activities outside of patient care. In the care sector, robots could eventually take on physically demanding tasks such as moving bedridden patients. Various companies and research institutes are already working on the next step: fully-fledged care robots. These would be a response to demographic change and the care crisis.

But even without a physical body, AI extended to include the spatial dimension can take on important tasks. Urban planning, for example, could benefit enormously from spatial intelligence. Such an AI system could analyse huge amounts of data from various sources, such as satellite images, traffic data, population statistics and environmental measurements, and derive complex correlations that would be virtually impossible for human planners to identify. It becomes exciting when concrete measures are derived from this.

Here are three examples:

1. By analysing surface temperatures and degrees of sealing, the formation of heat islands in cities can be predicted. These can be prevented by taking appropriate structural countermeasures.
2. Simulating extreme weather events reveals weaknesses in infrastructure and improves risk management.
3. Traffic control can be fully digitised through adaptive traffic light systems based on current and predicted traffic volumes.

What is still missing for the next leap in development

The progress made so far is already more than impressive. However, the implementation of spatial intelligence would be more than just a technical upgrade: it would be the logical next step in the evolution of artificial intelligence. But we are not there yet. Understanding physical laws such as gravity, friction, elasticity, acceleration and dynamics still poses a major hurdle. But there are also very pragmatic hurdles. The quality and availability of 3D data is still far from sufficient – especially when compared to the 2D data that has made large language models so powerful.

And finally, there is still a lack of suitable algorithms that can turn an AI's understanding of 3D and high-quality 3D data into a reliable and efficient large world model. Only then would spatial intelligence be a digital enrichment for our real world.

Text: Falk Hedeman