Spatial POV

Spatial AI is artificial intelligence crafted for spatial reasoning, the logic of navigating, designing, and operating within a particular conception of space.

Spatial AI is artificial intelligence crafted for spatial reasoning, the logic of navigating, designing, and operating within a particular conception of space.

And the manner of “spatial reasoning” depends on perspective, what vantage point or frame of reference an entity, either you or something occupying a space, takes.

Figuring your way an unfamiliar environment, like a new airport, can be daunting. And if you’ve never been to any airport before, not only can it be frustrating, you’re likely looking for help to find your way around, for things like signs, maps, information booths, or anyone who also has luggage in tow.

Luckily most airports are similar enough that after traveling through a few, you’ve learned enough about how to get around one you haven’t been to before, but designing an airport is an entirely different challenge, and from that perspective, every airport can be quite unique.

Not only are airports massive infrastructure projects with multiple stakeholders with economic and political aims, they embody a range of complicated and entangled spatial problems. Aircraft, pilots, passengers, security guards, and airline workers all have different needs. They inhabit different areas of the airport, which in turn have different spatial requirements. (Airplanes have large turning radii, but people don’t necessarily.). And the airport’s designer must anticipate these dynamics to make effective design decisions long before the airport is built.

Different AIs for Different Perspectives

This post isn’t about airports per se. But AI developed to reason about these kinds of problems, whether airports or cities or political environments, should incorporate the fact that one’s role, goals, and knowledge largely determine what reasoning needs to be employed, and thus what AI can better that reasoning.

The rest of this post presents a framework for six such perspectives.

Intelligent Agents + Dynamic-Yet-Dumb Environments

Here is our starting place, intelligent so-called “agents” occupying and traversing an environment, potentially aided by some form of AI.

Agents are abstractions for spatial actors like robots in a factory, airplanes flying in airspace, self-driving cars on a street, computer viruses through a network, a natural virus through a population, etc. We always conceive agents in relation to their respective environments (factory, airspace, street, computer network, population, etc.).

This diagram from Artificial Intelligence: A Modern Approach (Russell Norvig 55) introduces agents as employing sensors and actuators to both perceive and act within their environments:

As we’ll see, traversing an environment involves assessing not only the affordances and impedances within that environment, but the presence and absence of other agents.

Similarly, the book Algorithms for Decision Making defines “agent” thusly:

An agent is something that acts based on observations of its environment. Agents may be physical entities, like humans or robots, or they may be nonphysical entities, such as decision support systems that are implemented entirely in software. [Kochenderfer Wheeler Wray 1]

These days, “agent” is an overloaded term. This article uses “agent” in reference to “spatial agents,” any atomic actor that occupies and traverses a space, whether a robot, human, animal, bacterium, idea, or chess piece.

This differs from the “AI agent” or “agentic AI” from popular media, which speak to autonomous programs that act as digital representatives for people or companies. (For example, an AI agent that books vacations for your, responds to emails, etc.). Even so, a spatial AI can be an AI agent, and vice versa. But in other cases, a spatial agent is aided by an agent-operator, as an airplane’s autopilot is guided by a human pilot… Or is it the other way around?

locomotion ability to perceive, detect, measure space and spatial opportunities (can move here, cannot move there) a bit of self-awareness (it’s difficult for me to move, i’m expending large amounts of energy, etc.)

Environments, Dumb and Smart

When I say “dumb” environments, I don’t mean that AI theory treats them as static, unchanging, or simplistic. The same text by Russell and Norvig describes “task environments” in several ways, again, usually from the perspective of an agent (e.g., whether the environment is predictable or knowable by the agent).

The everyday environments we’re familiar with exhibit dynamic and active behaviors that adapt and adjust to context, occupants, resources, objectives, etc.

This is one major contribution of spatial AI, supporting the reasoning that underlies intelligent environments, particularly with regards to a particular conception of space, and also empower the designer of an environment with more effective predictive tools (see below).

The “smart cities” movement is an obvious example. Perhaps “smart homes” today aren’t so smart, however.

How can environments utilize AI or exhibit some kind of intelligence?

Environments can exhibit dynamic and active behaviors that adapt to context, occupants, resources, objectives, etc.

Just as an agent has sensors and actuators that give them abilities to sense and affect their environments, so too can environments affect the affordances and impedances presented to spatial agents. This might mean simply locking and unlocking doors, opening or closing them, even direct signaling to pass information, like way-finding signs or audio announcements in an airport.

Enter The Environment-Designers.

An intelligent environment won’t create itself; someone needs to design them. In this framework, I’ll call them “environment-designers,” and later in a more general context, “spatial designers.”

These are the architects, urban planners, interior designers, industrial engineers, etc., who aren’t typically chartered with designing or operating things like robots, but they do design the environments through which humans and intelligent machines will navigate. For these designers, spatial AI is about choreographing spatial dynamics.

For spatial designers, spatial AI is about choreographing spatial dynamics.

Just as spatial problems are inherently complex, this role is inherently challenging. A transit station like Penn Station in New York City deploys systems of systems, coordinating flows of people, luggage, and trains from, among, and to each other. Roadway traffic systems coordinate pedestrians, cyclists, and motor vehicles through dense street grids with varying levels of control. Courthouses segregate flows of judges, jurors, defendants, and the general public amidst rigid bureaucratic processes.

One core question of spatial AI is how we can augment the environment-designers’ toolkits with the right kinds of AI to generate and analyze spatial scenarios. Crowd simulations, for example, are agent-based models used to determine the efficacy of evacuation routes in a stadium or large venue.

Enter the Agent-Designers.

Since we’ve noted the environments’ designers, it’s only fair to mention the designers of the agents themselves.

Agent-designers are the roboticists and engineers who program agent behaviors, from expert systems to modern machine learning routines. Perhaps they’re also the designers of mRNA vaccines or specialized bacteriophages to navigate the human body.

Humans in the loop: The Operators.

A peculiar class of actors, the “operators,” can intervene in ways that most other spatial inhabitants can’t; they have special permissions to control the behaviors of an environment or agent.

The Environment-Operators

Environment-operators facilitate an environment’s realtime machinations, like identifying and fixing problems, adapting it to new conditions, or configuring it for different functions. To perform these responsibilities, they have information that many agents won’t, utilizing sensors and systems that can monitor agents and events.

They’re like the security guards in buildings, the staff at a concert venue, or air traffic controllers at airports, orchestrating the complexities of aircraft movement to ensure safety and timely takeoffs and landings.

The Agent-Operators

Speaking of airports, we can look at airspace and airplanes through the lens of this spatial framework, too. This allows us to introduce yet another player, the agent-operator.

In this scenario, pilots play the role of agent-operators, controlling and guiding the aircraft, the agent. “Autopilot” is the AI that augments the performance of the aircraft and assists the pilots.

The relationship between an agent-operator and the AI is often a critical design point; the control and predictability of an AI system can be vital to reliability and safety.

The full framework…

This airport/airspace/aircraft case is our first example with all the roles that comprise this framework:

• agents → aircraft, planes, helicopters
• environments → airspace, airports, runways, atmosphere, weather
• agent-designers → aerospace engineers at Airbus, etc.
• environment-designers → architects, civil engineers, FAA, etc.
• agent-operators → pilots
• environment-operators → air traffic controllers

Or as a table:

In any given spatial scenario, each role can be played by:

1. humans (or other creatures),
2. humans augmented by AI (e.g. pilots and autopilot),
3. semiautonomous AI supervised by a human (e.g. an autonomous vehicle in self-driving mode), or
4. autonomous AI (e.g. robots, computer viruses, etc.).

Determining the AI appropriate to augment or supplant humans in the above roles is a core design challenge for spatial AI. I.e., how can AI help these roles with their spatial reasoning?

Simulation is the way.

Finally, being an agent-designer or environment-designer can be tricky. Architects and urban planners don’t get to build physical, full-scale buildings or neighborhoods as part of their design processes, but a product designer may get to prototype a new device. Likewise, agent-designers like roboticists using machine learning to develop new control systems are quickly encountering the limitations of training robots in the real world; it just takes too long.

So an essential technique for agent-designers and environment-designers to make sense of these spatial dynamics is to simulate them.

We’re now seeing large-scale commercial simulation environments made available for training models in physical-temporal space, like NVIDIA’s Isaac Sim product, which serves as a scalable training tool for robots. Virtual robots in these virtual environments can be trained en masse, much more rapidly than in the physical world. If their abilities are modeled accurately enough, the models from the virtual environment can be directly loaded into the actual physical robots with little or no adjustment.

And that’s it for now.