Proposing a “Spatial” AI

These days, we already have remarkable technologies that help us navigate time and space.

Driving directions and GPS guide us to where we need to go by incorporating multiple data sources, shortest route by time, shortest route by destination, our driving preferences (use highways, avoid tolls), the time of day, current and predicted traffic conditions, the weather forecast.

“Autopilots” can operate planes through airspace often with comfort and safety equal to that of human pilots. Garmin’s autopilot products have even demonstrated fully automated emergency landings of small planes.

As for outer space, perhaps the flight control systems aboard the Saturn V rocket or Space Shuttle weren’t “autopilots” in the modern sense, but they performed similar functions, perhaps tuned less for comfort and more for accuracy.

A Spatial Artificial Intelligence

These are all examples of spatial artificial intelligence, at least AI that focuses on physical space. Whether at the scale of a desk or a street, they aid in sensing and reasoning through the peculiarities and complexities of how something negotiates an environment and whatever or whoever occupies them.

More recently, the tech industry has made major strides in autonomous control systems for robots, drones, and vehicles. The term "spatial computing" (Simon Greenwold, MIT, 2003) has reemerged after Apple’s embracing of it with their release of the Apple Vision Pro.

Niantic Spatial are developing “geospatial AI” products based “based on a third-generation digital map that captures the content of the world at a level of fidelity never before achieved, enabling both people and machines to understand it in new exciting ways.”

Archetype AI claim to be “pioneering novel AI and interaction technologies for the real world,” latching onto the term “physical AI.”

Meta are investing heavily in AI technologies that power their glasses-based product lines. Project Aria is an effort to develop the perceptual abilities for these glasses, through a combination of data and devices so AI can “better understand the world from a human perspective.” We can’t ignore that the devices and software of “mixed reality” are increasingly important spatial technologies.

As part of the world’s most valuable company by market capitalization, nVidia’s Spatial Intelligence Lab works to “advance foundational technologies enabling AI systems to perceive, model, and meaningfully interact with the physical world,” including world simulation, physics simulation, 4D perception, 3D/4D content creation + editing, and geometry processing.

Obviously, these companies are speculating that spatially aware commercial systems will create enormous shareholder value, although exactly what kind of value society will ultimately realize is still an open question.

And these companies are focusing on a limited set of use cases, yet there are many more spatial use cases from other fields and professions

It’s a trend that focuses on a limited set of use cases. They’re interesting, but I think there’s something even more interesting in looking

(how to get to diff concepts of space… whom am i writing to?)

(What’s the bridge here?) what these things have in common … (but why even ask the question?)

AI → only agents, not environments space angle → only physical, not other types of space (maybe this is the pathway to generalizing) profession angle → many other people have spatial problems, but not agent-related

but i think there’s more to it… (this just sounds fucking extra cheesy. is there another way?)

Why is this useful? aka “value”… Why is it valuable, and thus valuable for me to spend my time reading this fucking article? ostensibly value depends on context… in the context of architecture schools, spatial ai means … is this an intellectual pursuit?

Spatial AI comprises a class of

The most familiar ones are physical and temporal problems, that is, challenges in space and time.

where to go? lots of professions that need spatial ai, more than are serviced today linguistically → if we look at the word spatial and ai separately, what opportunities are there? solving problems with automation and autonomy, but can these be spatial? theoretically,

physical-spatial problems → how to solve them = spatial reasoning → generalize to more types of space → spatial AI includes these too creative ends, efficiency ends

physical-spatial problems → but there are more kinds of problems → how to solve them = spatial reasoning (physical and other?)

Physical-Spatial Problems

One way to frame the algorithms that power spatial AI is that they solve specific spatial problems. These can be seemingly simple tasks to more complex ones like:

Attributes of Physical-Spatial Problems

Most physical-spatial problems involve some common attributes.

Availability

There is a sense of availability, in the sense of “having enough space.” If we’re putting furniture into an apartment, there is a limit to how many chairs and sofas we can place there. And this applies to many realms, like aircraft on a runway or hangar or bits in computer memory.

Consumption

Conversely, there is a sense of consumption of that space, which reduces the available space. This makes spatial problems someone economical in nature, in treating space as a limited resource.

Occupancy

Related to consumption, there is a notion of occupancy, where space is occupied by objects or agents of different types and mobilities. A sofa occupies and consumes space like a person does, but a person can usually move on their own, exchanging a piece of consumed space for another, and a sofa doesn’t. Such self-locomoting occupants we might call “spatial agents.”

Dynamism

The fact that occupants move and thus change the availability of space, creates dynamic situations.

Frame of Reference

Space can be treated with a specific frame of reference. An apartment has some measure of available space, but so does the sofa. There is a kind of extension of available space relative to the front of the sofa that’s important for its function; we need to be able to walk up to it and sit. But the back of the sofa can be against a wall, thus inaccessible, and it doesn’t impede the furniture’s sofa-ness. A crowd of people consume the space in a concert venue, but each person has their own “personal space” that if consistently encroached upon, we might call the space “too crowded.”

Overlapping Concerns

When we lay out furniture in an apartment, we’re solving many concerns at once. It’s not only about availability – can we fit everything we want in there – the positioning of furniture is important for supporting an apartment’s functions. We need to be able to move between our furniture (walking, on crutches, in a wheelchair).

A good living room supports social activities like watching television or having conversations.

Scale

They depend on scale as well. We might call an apartment “crowded” if it has too much furniture to be comfortable for its human occupants, but airspace is “crowded” when there are too many planes to be managed safely. And the planes aren’t centimeters apart like furniture in an apartment, they may be hundreds of meters apart and still create a crowded situation.

AI aims to solve spatial problems by giving machines greater autonomy and efficiency.

AI aims to solve spatial problems by giving machines greater autonomy and efficiency.

Definitions of Space

When we think of “space,” we probably first think of the physical space around us. Or we may even think of “outer space,” our final frontier, so to speak. Depending on your profession, “spatial” might seem synonymous with “three dimensional.”

But the types of problems we’re solving and the solutions we may employ all depend on how we define “space.”

And there are many more such categories, especially if we consider that physical space isn’t the only conception of space. AI as a broad category of potential solutions, regardless of the problems’ physicality, is what comprises spatial AI.

Solving Spatial Problems

Even though the problems above are all spatial in nature, their variety implies there likely isn’t a single approach to solving them. And it shouldn’t be surprising that any given spatial problem won’t necessarily require a cutting-edge, AI-based solution.

Geopositioning (locating yourself in the wilderness) can often be solved with a frame of reference (a map or star chart), some measurements with the right tools, and a little math. (Although the original development of these original methods of geopositioning took centuries. And mapping itself is another category of spatial problems.) The modern, more practical solution is of course using the Global Positioning System (GPS), that we’re all familiar with, mostly from our phones and devices like those from TomTom and Garmin (although our smartphones use more than just GPS to determine our location).

But some of these problems certainly can be addressed with AI, and some perhaps more effectively so (well, given a proper definition of AI), and I’m obviously interested these methods, both for practical and creative ends.

The manner of AI-based solution of course depends on:

  1. the type of spatial problem (e.g., fitting, connectivity, etc.),
  2. the available technologies,
  3. our understanding of the computational reasoning required to build that AI, and
  4. our intuitive, human perceptions of spatial reasoning.

By this last point, I mean our own innate abilities, like spatial memory that enables us to remember how to get out of a building once we’ve walked through it the first time. Or even social conventions, like knowing the best place to look for a bathroom in an unfamiliar restaurant is near the bar.

Crowd management is a bit different from charting a path through a maze, especially when you can’t see the entire the maze. Maze solving is a bit different from solving a close packing problem, like fitting packaged goods into container ships or optimizing cardboard box selection on a factory packaging line. And the humanoid robots being programmed to replace humans in factories need yet another set of techniques.

But what the intuitive methods, professional approaches, and AI algorithms do have in common is that they all perform some kind of specialized reasoning to solve these problems – what we might call “spatial reasoning.”

Spatial Reasoning

Again, I’m hoping to employ “spatial reasoning” in a broader sense in order to cover a wider variety of human and computational capabilities and potential spatial problems. To build this up, if we peruse Wikipedia for related spatial terms, a few useful concepts pop up:

Spatial Intelligence

To cognitive scientists, “spatial intelligence” refers to our ability to visualize objects in our mind’s eye." Spatial intelligence is defined by Howard Gardner as a human computational capacity that provides the ability or mental skill to solve spatial problems of navigation, visualization of objects from different angles and space, faces or scenes recognition, or to notice fine details." (Wikipedia)

Spatial Ability

“Spatial ability is the capacity to understand, reason and remember the visual and spatial relations among objects or space. There are four common types of spatial abilities which include spatial or visuo-spatial perception, spatial visualization, mental folding and mental rotation.” (Wikipedia)

Spatial Cognition

“Spatial cognition is the acquisition, organization, utilization, and revision of knowledge about spatial environments. It is most about how animals including humans behave within space and the knowledge they built around it, rather than space itself.”

Spatial-Temporal Reasoning

The relevant branch of AI is “spatial-temporal reasoning,” which formulates computable representations of space and time for particular kinds of spatial problems, most notably navigation and path-finding. Quoting a bit more from Wikipedia on this:

“Spatial–temporal reasoning is an area of artificial intelligence that draws from the fields of computer science, cognitive science, and cognitive psychology. The theoretic goal—on the cognitive side—involves representing and reasoning spatial-temporal knowledge in mind. The applied goal—on the computing side—involves developing high-level control systems of automata for navigating and understanding time and space.”

These “automata” I will call “spatial agents,” that is, entities that occupy and traverse space with varying degrees of autonomy, but more about that later.

The means by which we “represent” space, that is, how we construct in a particular medium a functional analog for space, are critical to reasoning about what space might be like, its form, how it might be perceived, how it might actually function.

AI for Designers of Spaces

What makes spatial designers uniquely positioned for the problems of the modern world?

Architects in particular are most skilled at this, as they typically can’t design buildings by building buildings, in fact, they don’t build buildings at all. They produce representations of buildings that are consumed as instructions for contractors to price and construct the building. And the spatial experience, what architects are really interested in, derives from the walls, columns, stairs, etc., of the actual building.

But what about spatial problems that don’t involve creating agents per se, but rather creating the very environments that those agents occupy and traverse?

What about AI for designing the actual environments, where people, vehicles, and robots occupy and move about?

What about AI for designing the actual environments, where people, vehicles, and robots occupy and move about?

After all, creative professionals like architects, urban planners, and many engineers, employ spatial reasoning to imagine and evaluate design solutions for environments like buildings and urban space. A major part of these design decisions involves planning how people and vehicles will navigate these environments.

Airports become a complex choreography of queueing and spatial connectivity for both people and aircraft. Airspace becomes a realtime puzzle-space of managed flight paths, governed in the US by the FAA, but actively managed by air-traffic controllers.

More than Physical AI

All of these companies are really working on “physical AI,” which enables machines to productively assess and act within the physical world. But what that entails is really a kind of sophisticated automation, the classic goal of computing (and AI) that supplants typical human effort to achieve human-provided goals with machine effort to achieve the same human-provided goals.

A warehouse foreperson needs product moved from one rack to another, sorted, and cleaned, so instead of having a human do it, have a robot do it.

But what I’m really interested in are the creative opportunities afforded by expanding the definition of “spatial AI.”

How have we seen AI serve as a creative means? What is “space” and how do its many definitions present a creative opportunity?

Beyond AI as Automation

First, AI doesn’t necessarily need to be only fancy automation; fields like artificial life, evolutionary computing, and generative design already show the creative potential of computing.

Even discriminative AI methods like motion capture and pose detection assist game designers and visual effects artists replicate human-like motion for producing games, animations, and movies.

And the most recent craze of the last few years, generative AI, has produced algorithms like DALL·E and stable diffusion (under a larger category of so-called “text-to-image” methods), which can help creatives rapidly exploring rich visual ideas.

Notions of Space

This starts with diving into the nature of “space” and its many possible definitions, speculating on how “artificial intelligence” might adapt to and reflect those notions of space, and speculating on what “spatial AI” practically could do, should do, should not do, for whom, and why it matters.

In particular, I think spatial AI could be particularly interesting to “spatial designers” like architects and urban planners, for whom current notions of AI are lacking. Most ideas of spatial AI focus on building autonomous agents like robots and autonomous vehicles, not modeling and designing the complex environments that such mechanical agents (and humans) will occupy.

How can we develop AI that helps design and reason about environments (however abstract), rather than the robots themselves.

Moreover, and to me more interesting, do non-physical (i.e., abstract) notions of space provide a launching point to develop more generalized and broadly applicable ideas of “spatial reasoning,” much like game theory applies to more than games, but economics, market competition, and policy?

Expanding the Meaning of “Space”

It turns out there are many kinds of “space.”

As I noted before, I’m interested in exploring a more general conception of spatial AI that is inspired by different ideas of “space.” In particular, what about types of space that aren’t inherently physical? Is there some value in a set of algorithms that can reason spatially in a more general way?

We most often think of “spatial” as the physical space around us. For design professionals, sometimes it’s synonymous with “three-dimensional.” Or we say a representation is spatial when it portrays depth particularly well or in unexpected ways.

But the very notion of “space” has been debated by philosophers, mathematicians, and geographers for centuries. Is space equivalent to a void, that which lies between the walls of a container? Is it an extension around a location? Is it the distance between objects? If so, does it still exist when one of the objects is removed? Does it have structure or other properties?

But there are many such notions:

Could we sue these notions of space as inspiration for new AI methods? Could an AI be programmed to reason through each of these for some function, some real-world scenario that isn’t so analogous to everyday three-dimensional space?

Spatial AI is spatial reasoning by machines.

As we’ve seen earlier, the many categories of spatial problems are different enough to warrant different solutions. An aircraft’s autopilot isn’t going to solve a static packing problem. And a packing algorithm isn’t going to predict and address the ever-changing dynamics of an airport or courthouse.

Spatial reasoning is the core of spatial AI. It’s the human capability that the AI attempts first to mimic and then to exceed. In some ways, it already does.

But “spatial reasoning” involves more than just solving the problems in physical space (what a physicist might call classical mechanics) or a spatial agent’s navigation. It also involves the design and operation of space itself.

Spatial AI is the spatial reasoning of machines.

So spatial AI is artificial intelligence developed for spatial reasoning…

through different types of “space,” whether that’s the interior of a building (robots in architectural space), the streets of a city (autonomous vehicles in urban space), the ocean (geopositioning of a ship), or

I believe spatial AI should encompass more than what it first sounds like, more than algorithms for faster 3D modeling, powering virtual reality, or enabling geospatial predictions. Such methods tend to regard space rather simplistically, i.e., merely using 3D point data or latitude-longitude data.

Spatial AI is spatial reasoning by computational machines for the occupation, traversal, and design of space.

In contrast, spatial AI is spatial reasoning for the occupation, traversal, and design of space. Spatial AI embraces these existing AI methods but aspires to do more.

I taught a seminar on Spatial AI at Columbia University GSAPP in Spring of 2024, which started by exploring a vocabulary of spatial relationships (words like above/below/adjacent to/… or whole building programs and zoning codes), testing how far can we push large language models to reason through spatial rules, and authoring our own spatial semantics.