Spatial AI Framework

Building off of Proposing Spatial AI, I’m continuing to explore how to develop new artificial intelligence for reasoning through different conceptions of “space.” Here’s my newest working research definition:

Spatial AI refers to artificial intelligence for spatial reasoning, the logic of navigating, designing, and operating environments.

In a sense, spatial AI is like game theory. Just as game theory can be used to model a variety of scenarios from actual games to the world’s economies, so might spatial AI. One key difference is that spatial AI is just as much about a game’s designers (those who write the rules) as it is about the players.

By “different conceptions of space,” I mean what has been examined by mathematicians, philosophers, and geographers for centuries. I think by perusing these concepts – deliberately beyond our intuitive, everyday physical-temporal experience (and the x-y-z Cartesian coordinate system pervading 3D software) – we can invent a novel and profound lens to model the world.

And this is an opportune time because of the rapidly improving landscape of new design patterns (like encoder-decoder architectures and transformers), readily available AI tools and platforms (from PyTorch to Hugging Face), plus a renewed interest in latent fields like computational design and spatial computing.

Spatial problems are curiously complex.

Practically, from a spatial designer’s perspective, the category “spatial problems” includes activities like designing buildings, planning neighborhoods, building complicated infrastructure, or optimizing global supply chains. One common theme among them is that even the simplest ones – like fitting furniture into an office – can still involve balancing numerous complex and competing factors.

But yes, even supply chains can be viewed as spatial puzzles, as they involve managing ever-changing capacities to store and move goods through land, oceans, and airways, with fluctuating degrees of control and predictability. For companies, spatial AI might aim to generate optimal solutions for the individual behaviors of ships and trucks. For nations, spatial AI could simulate different rules of trade, including import duties, tariffs, international laws, etc., and perhaps contribute to new policies.

Board games are inherently spatial; players control pieces that traverse a finite board governed by formal rules. Different rules makes for different experiences. Spatial AI could model players’ decision-making but also recommend strategies for game designers, who explore and invent rules and mechanics for new games.

One common theme among spatial problems is that even the simplest ones can still involve balancing numerous complex and competing factors.

In urban environments, spatial AI could generate plans for directing city services, anticipating and scheduling repairs, determining levels of investment, or coordinating agencies to minimize traffic and neighborhood disruptions. Planners could consider different systems, compositions of, and incentives for urban mobility, balancing pedestrian traffic, electric scooters, bicycles, and motor vehicles, autonomous and otherwise.

Politics can be regarded spatially. Concretely, the geopolitics of an election campaign amounts to a strategic territorial puzzle. Abstractly, politicians talk about how to “navigate a political climate” and the like. If we can formally define the qualities of such a climate, spatial AI could devise strategies to influence it.

How quickly a pathogen traverses a population depends on its mode of transmission and rate of replication, but also the characteristics of the environment it inhabits, including the density of the population, presence of fomites, airflow, temperature, and other factors.

Entrepreneurs and marketers talk about “entering a space” to mean “sell to a new customer segment.” Or they say “a space is crowded” to convey a sense that “too many competitors to sustain another.” Is there something inherently “spatial” about a market then?

A framework to break down spatial problems…

To make any of these ideas actionable, we’ll need a framework to break down spatial problems systematically. The rest of this post presents this framework, which will need to:

1. leverage modern AI theory so we can utilize modern AI algorithms,
2. bridge the gap between abstract conceptions of space to notions of computable space, and
3. enable us to effectively simulate complex behaviors.

Artificial intelligence is only just beginning to grapple with space.

Today, AI treats spatial problems in rather limited ways, particularly due to how complex they are, but also to AI’s focus on intelligent agents, not intelligent environments. Spatial AI is for the designers’ of the games, not just the players.

Spatial AI is a technology-enabled way of analyzing and reasoning about not just physical space, but any kind of complex space, system, or environment that fits this framework.

So it’s worth noting the terms “space” and “environment” are related but distinct. While environment means a specific place or situation, like a particular city or building, I’ll use space to refer to the abstract qualities that characterize an environment and its relationships to its occupants.

AI today focuses on intelligent agents + dynamic-yet-dumb environments.

Here’s our starting place, a diagram showing intelligent so-called “agents” navigating an environment by utilizing some form of AI.

These agents are abstractions for autonomous actors like robots in a factory, airplanes flying in airspace, self-driving cars on a street, a computer virus through a network, a natural virus through a population, etc. And of course, their respective environments are factory, airspace, street, computer network, population, etc.

This diagram from Artificial Intelligence: A Modern Approach (Russell Norvig 55) introduces agents as employing sensors and actuators to both perceive and act within their respective environments:

Note that this article uses “agent” to talk about spatial agents, any being that navigates through a space, whether that’s a robot, human, animal, or other organic entity.

This is different from the “AI agent” and “agentic AI” terms floating around in popular media, which speak to autonomous programs that act as digital representatives for people. (That is, one that interacts with the world as you, like booking trips for you, responding to emails, etc.)

Environments can be smart, too.

When I say “dumb” environments, I don’t mean that AI theory treats them as static or unchanging. The same text by Russell and Norvig employs several productive frameworks for characterizing these “task environments,” again, usually from the perspective of an agent (e.g., whether the environment is predictable or knowable by the agent).

Environments themselves can also exhibit dynamic and active behaviors that adapt and adjust to context, occupants, resources, objectives, etc. This is one major contribution of spatial AI, supporting the reasoning underlying intelligent environments, particularly with regards to a particular conception of space.

The “smart cities” movement is an obvious example. Perhaps “smart homes” today aren’t so smart, however.

How can environments themselves utilize AI or exhibit some kind of intelligence?

Environments can exhibit dynamic and active behaviors that adapt to context, occupants, resources, objectives, etc.

Just as the agent model includes sensors and actuators, giving them abilities to respond to and affect their environments, so too can environments, affecting the affordances and impedances to spatial agents. This might mean simply locking and unlocking doors, opening or closing them, even direct signaling to pass information, like way-finding signs or audio announcements in an airport.

Enter the environment-designers.

An intelligent environment won’t create itself; someone needs to design them. In this framework, I’ll call them “environment-designers,” and later in a more general context, “spatial designers.”

These are the architects, urban planners, interior designers, industrial engineers, etc., who aren’t typically chartered with designing or operating things like robots, but they do design the environments through which humans and intelligent machines will navigate. For these designers, spatial AI is about choreographing spatial dynamics.

For spatial designers, spatial AI is about choreographing spatial dynamics.

Just as spatial problems are inherently complex, this role is inherently challenging. A transit station like Penn Station in New York City deploys systems of systems, coordinating flows of people, luggage, and trains from, among, and to each other. Roadway traffic systems coordinate pedestrians, cyclists, and motor vehicles through dense street grids with varying levels of control. Courthouses segregate flows of judges, jurors, defendants, and the general public amidst rigid bureaucratic processes.

One core question of spatial AI is how we can augment the environment-designers’ toolkits with the right kinds of AI to generate and analyze spatial scenarios. Crowd simulations, for example, are agent-based models used to determine the efficacy of evacuation routes in a stadium or large venue.

And who designs the agents? Enter the agent-designers.

Since we’ve noted the environments’ designers, it’s only fair to mention the designers of the agents themselves.

Agent-designers are the roboticists and engineers who program agent behaviors, from expert systems to modern machine learning routines. Perhaps they’re also the designers of mRNA vaccines or specialized bacteriophages to navigate the human body.

Humans in the loop: The Operators.

A peculiar class of actors, the “operators,” can intervene in ways that most other spatial inhabitants can’t; they have special permissions to control the behaviors of an environment or agent.

Environment-operators

Environment-operators facilitate an environment’s realtime machinations, like identifying and fixing problems, adapting it to new conditions, or configuring it for different functions. To perform these responsibilities, they have information that many agents won’t, utilizing sensors and systems that can monitor agents and events.

They’re like the security guards in buildings, the staff at a concert venue, or air traffic controllers at airports, orchestrating the complexities of aircraft movement to ensure safety and timely takeoffs and landings.

Agent-operators

Speaking of airports, we can look at airspace and airplanes through the lens of this spatial framework, too. This allows us to introduce yet another player, the agent-operator.

In this scenario, pilots play the role of agent-operators, controlling and guiding the aircraft, the agent. “Autopilot” is the AI that augments the performance of the aircraft and assists the pilots.

The relationship between an agent-operator and the AI is often a critical design point; the control and predictability of an AI system can be vital to reliability and safety.

The full framework…

This airport/airspace/aircraft case is our first example with all the roles that comprise this framework:

• agents → aircraft, planes, helicopters
• environments → airspace, airports, runways, atmosphere, weather
• agent-designers → aerospace engineers at Airbus, etc.
• environment-designers → architects, civil engineers, FAA, etc.
• agent-operators → pilots
• environment-operators → air traffic controllers

Or as a table:

In any given spatial scenario, each role can be played by:

1. humans (or other creatures),
2. humans augmented by AI (e.g. pilots and autopilot),
3. semiautonomous AI supervised by a human (e.g. an autonomous vehicle in self-driving mode), or
4. autonomous AI (e.g. robots, computer viruses, etc.).

Determining the AI appropriate to augment or supplant humans in the above roles is a core design challenge for spatial AI. I.e., how can AI help these roles with their spatial reasoning?

Simulation is the way.

Finally, being an agent-designer or environment-designer can be tricky. Architects and urban planners don’t get to build physical, full-scale buildings or neighborhoods as part of their design processes, but a product designer may get to prototype a new device. Likewise, agent-designers like roboticists using machine learning to develop new control systems are quickly encountering the limitations of training robots in the real world; it just takes too long.

So an essential technique for agent-designers and environment-designers to make sense of these spatial dynamics is to simulate them.

We’re now seeing large-scale commercial simulation environments made available for training models in physical-temporal space, like NVIDIA’s Isaac Sim product, which serves as a scalable training tool for robots. Virtual robots in these virtual environments can be trained en masse, much more rapidly than in the physical world. If their abilities are modeled accurately enough, the models from the virtual environment can be directly loaded into the actual physical robots with little or no adjustment.

And that’s it for now.