Spatial Projects

A spatial stacking game for visionOS.
Out now.

As seen on

Gesture Toys

I love experiences that embrace the limitations of the hardware to create intuitive, joyful moments. Because I find interactions that rely on precise positioning and artificial locomotion frustrating, I challenged myself to use Unity and my Oculus Quest to create a couple of digital toys which use forgiving and natural gestures instead.

Toy #1: Magic Lighter

Full hand tracking is a great VR interface because there’s no tool for the user to learn and adapt to– it makes use of our inherent sense of proprioception to deliver interactions that feel perfectly natural. It falls apart, however, when our sense of touch isn’t engaged when we might expect it to be. I want to find ways to avoid making a user reach out and “touch” something that doesn’t push back, which got me thinking about how the user could interact with their own hands to provide a more complete sensory experience.

Snap, Light, and Shake

While experimenting in Unity, I found that mapping an outcome to a snap felt super satisfying. The friction, leading to release with a real-world audio confirmation felt a bit like using a lighter, so it felt like the right fit for a flame to pop out of my index finger.
It felt nice to use the flame from one hand to light up fingers on the other – I could use both hands in easy coordination with each other for this (yay proprioception!). If fingers truly worked like matches or candles, it would follow that the easiest way to put them out would be to shake them – so I tracked the velocity of each to put out the flame if they were moving fast enough.

Toy #2: Rhythm RPS

VR games can sometimes be difficult to share with folks who don’t have experience using a video game controller. This got me wondering how I could design a game where the act of playing it is a true 1:1 map to its real-life counterpart.

Rock Paper Scissors doesn’t require the player to touch, grab, or manipulate objects — to play, you just raise your hand and mime an object. This made it a great fit, but of course the problem is that playing Rock Paper Scissors against a computer is perhaps the least fun activity imaginable.

However, with Rock, Paper, and Scissors as gesture inputs that most of us are familiar with, and with each corresponding to another as the “right answer,” all I needed was to create a challenge that felt fun and fair even with a computer on the other side.

Finding the Fun

In a regular round of RPS, the player knows when to make their move based on the rhythmic chant leading up to the climactic showdown. It occurred to me that if we show the opponent’s move up front, we can turn the challenge into “think quickly!” rather than “read your opponent’s mind!”
Taking inspiration from my favorite rhythm game series (Rhythm Heaven– it rules), I put together this 2D prototype in XD and pretended to play by making gestures at my monitor.

Making it Real

I migrated the UI and animation from XD into Unity’s Canvas, modeled the objects with the built-in ProBuilder tool, and implemented a check to determine which gesture most closely matched the hand’s current state on the 4th beat of the measure.
By tying the recognition of the gesture to a rhythm-based timer, we absolve the player of needing to indicate that they’re ready to have their gesture detected (which can be difficult without some type of controller in hand).
Lastly, I dropped in a health indicator to allow for a few mistakes, and increased the speed of the rhythm every 4 measures. It made for an intuitive challenge that was as much fun to make as it was to play.

Spatial Prototyping in 2D

I imagine a future in which relevant digital content is tied to the places we naturally turn for sensory input. In 2019, I used Adobe XD to explore how AR could act as an assistive layer adding helpful information when we need it, rather than a persistent layer that brings the clutter of our smartphones to our field of view.