People love playing video games together, whether in the same room or remotely. I just saw a demonstration of how we might soon be able to play together in augmented reality, too.

The demo was a treat because virtual reality and its cousin augmented reality—in which digital imagery is overlaid on your view of the real world—are still very isolating. You tend to use them while wearing a dopey-looking headset, and chances are that whatever digital imagery you’re seeing, you’re checking it out all alone. But last week at Microsoft’s annual developer conference in San Francisco, I and several other people put on the company’s $3,000 HoloLens headsets and interacted with each other over a wireless network.

To start, we each picked one of four little avatars (all called Poly, short for “polygon”) to hover near our shoulders; they looked like silvery robot heads glowing in different bright colors (mine was blue). I couldn’t see my Poly, but I knew it was moving with me—tilting as I tilted my head, for instance—since I could look around and see everyone else’s avatar doing likewise. The avatars tracked pretty fluidly with their owners. Even when I walked several feet away, I could still see them levitating properly in space.

Then we could use our gaze and an up-and-down index-finger gesture (Microsoft calls this an “air tap”) to aim and throw faceted silver balls at each other’s avatars. If I got a hit, a cartoony ring of stars would appear around the poor avatar’s head.

I couldn’t kill any of them off, and there was no scorekeeping. But it was pretty cool to see how accurately I could target them—it worked from six feet or so, no problem—and how promptly they moved with their human counterparts.

Making this work isn’t trivial. Your headset needs to be aware of your physical and digital surroundings, keeping track of where every other headset is in space, the actual surfaces that surround you, and where any digital objects are located. Any changes need to be communicated in real time to avoid a crappy experience.

But since HoloLens is available only to software developers and still being refined for a consumer release, my fun was very much tempered by all the work yet to be done to make mixed-reality socializing truly functional.

HoloLens is designed kind of like a pair of big black ski goggles (see “Microsoft Making Fast Progress With HoloLens”). Through its see-through visor, you have a frustratingly small field of view, which means you can’t see all that much digital imagery at once, so anything that’s too big or too close you’ll only see one piece of at a time. The headset also still feels heavy at nearly 1.3 pounds—that’s what happens when you insist on cramming all your components, including a battery and speakers, into one package. While it looks fairly compact and didn’t initially feel uncomfortable, it got annoying after a little while. While ducking and dodging virtual balls that other people lobbed at me, I started to worry it might fall off.

Even so, the social future that will be enabled by these technologies is not hard to imagine. You could play games with friends who are either with you physically or getting in on the action from far away, and these games would be far more immersive than ones on a flat screen ever could be. It will be interesting to see what developers come up with now that they have their hands on HoloLens.