Microsoft’s upcoming HoloLens headset is the first high-end augmented reality device that’s been shown to the public. Unfortunately, Microsoft has been so tight-lipped that we don’t really know much about it.
Today, at Microsoft’s Build Developer Conference in San Francisco, we got another tantalizing peek at the technology. The demo itself is shiny but not very informative. However, we can pick out some important details by reading between the lines. In this case, what Microsoft chose not to show is more informative than what they did.
Several of the HoloLens engineers came onstage to show off some of the new demos they’ve developed for the hardware. One demo takes place as a recreation of an engineer’s depressingly sparse apartment, and covers the walls and floor with virtual apps and toys. Another brings in some doctors to show a model of a human body, and then takes it apart. Finally, a kit robot grows an expressive, virtual upper body and unfolds a large status panel, allowing it to be tweaked in real time.
The Devil, The Details
So far, so impressive — but many of my initial questions about the HoloLens remain unanswered. So let’s look a little closer.
Does the demo feel a little bit staged to you? Watch the camera movements carefully. It’s not just the usual trade-show awkwardness. Watch the demo again, and look for moments where a human being passes in front of a virtual object.
This is called occlusion, and it’s a hard problem in augmented reality. In order to render this phenomenon correctly, the HoloLens needs to map the world quickly and accurately enough to cut the person’s silhouette out of the virtual image. Otherwise, you’ll have a depth conflict: your stereo vision will be telling you the virtual object is behind the person, but it’ll be displayed on top of them. As we’ve learned from virtual reality, these sorts of conflicts are upsetting, and destroy the illusion that the objects you’re interacting with are tangible parts of the world.
The demo goes to considerable lengths to prevent this. You can see the camera operator react when one of the people on stage moves towards a virtual object, moving to stop them from overlapping. The entire demo is elaborately choreographed. Getting out of the chair in the first demo causes the guy on the stage to step in front of a virtual robot — and, conveniently, the camera is pointed elsewhere at exactly that moment.
They only mess it up a few times. During the robot demo, the robot briefly passes in front of a few virtual dots on the floor, and one of the presenters steps on them. In both cases, there’s a depth conflict: you continue to see the virtual dots through the real objects, implying that Microsoft does not yet have a solution to this problem.
We can see more evidence of this in the same demo, when a visualization is displayed of the headset’s map of the environment. If you frame-by-frame through it and look closely, you can see that the map is incorrect in several ways: the table is on a raised platform, which becomes a vague bulge in the map. The robot, likewise, is visible in the map, but only as a small lump. The edges of the desk are missing. You can probably blame these problems on the limited resolution of the Kinect depth sensor inside the headset. Either way, there just isn’t enough precision here to correctly occlude virtual objects.
Let’s be clear here: this could be a showstopper. The purpose of the HoloLens is to seamlessly combine real and virtual objects in a convincing way. Occlusion is a necessary part of the illusion. Occlusion is such a powerful depth cue that adding a single white line to a video can create a powerful 3D effect by itself.
Microsoft is clearly aware of this. They show occlusion in their promotional materials. Yet, from their demos, it doesn’t look like they can actually do it. If Microsoft can’t solve this problem, they do not have a product. Users do not live in elaborately choreographed demos inside empty homes. They won’t tolerate constant, immersion-breaking rendering errors. I’m honestly kind of amazed that more of the tech press isn’t talking about this problem. I’m very curious if Microsoft has any plan to solve this issue.
Every time I watch a video showing off the HoloLens UI, the phrase that keeps going through my head was “Are we really going to do this again?”
With each new computing platform, both the PC and smartphones, we go through a cycle of bad UI before we figure it out. The problem is that the bad UIs are bad in exactly the same ways, and they’re bad in the way that the HoloLens UI is bad. I wish, for once, we could recognize the pattern and skip ahead to the good part. Here’s what the HoloLens (and every new platform) does wrong:
- First, they try to re-create old paradigms, instead of creating new ones. Think of the old Treo smartphones that were built around a physical keyboard and a mouse-emulating stylus. The HoloLens is exactly the same, using your head to emulate a mouse, and your finger to click. Heck, at one point the engineer “drags” a window by locking it to his head, something that the Oculus VR Best Practices guide explicitly warns against. The HoloLens is not a PC, and trying to use a PC UI isn’t going to work.
- Second, they get wrapped up in the shiny parts of the interface, and create ornate, useless fluff. Do you remember Clippy? Microsoft Bob? The faux leather on the old iOS notes app? This stuff is useless visual clutter. The virtual dog and fish bowl and robot are exactly the same thing. How many times do we have to do this before we realize that minimalism is the only good idea?
While we’re at it, the “holograms” themselves are a mess visually. They can’t decide whether they’re clean panes or goofy, cartoon blobs. The UI elements are way too big, requiring expansive gestures to use, up to and including physically walking around, which is bad design for a lot of reasons.
If it seems like I’m upset about this, it’s because immersive user interfaces are eventually going to be really, really cool someday. The HoloLens UI that’s been shown so far is a complete mess, and that’s a shame. A good UI is the difference between the Treo and the iPhone — between an IBM mainframe, and the original mouse-and-keyboard PC. It’s the difference between a toy for nerds and a world-changing innovation.
There is one major piece of good news. Microsoft brought “hundreds” of HoloLens devices to the show, and are showing them off to attendees. The last time Microsoft did demos, people were strapped into bulky wired headsets. The headsets used on stage were props that could display only a low-resolution black and white image. However, the presenter clearly states that the headsets are “just likes the ones you saw on stage today.” In other words, Microsoft has successfully miniaturized the HoloLens hardware to fit in a portable form factor, which is great news.
Here’s one developer’s experience with the new hardware:
“I went ‘on the construction site’ and they superimposed the blueprints on the wall. Then I could see how a door would look like and notice that a beam was in the way. We moved the door away and then added a layer with electrical and plumbing, and saw that a pipe was now in the way. I ‘gazed and air clicked’ on the pipe and left a voice note for the plumber to move the pipe away.
Then I came back in the role of the plumber and retrieved my note (it was hanging there), gazed and clicked, and heard my own recorded voice.
Wearing the device needs some getting used to. It was not super well-adjusted and pressed a little on my nose. But I think I can adjust it better in time.
The holographic area is less extensive than I thought. It’s a rectangle in front of your eyes but if you have a good side vision (like I have), it’s a bit unsettling because the hologram will be cropped when you move the head.
All in all it is extremely impressive and I can’t wait to develop for it.”
While the writer does note that the device still has a low field of view he seems impressed with the hardware overall, which is a promising sign. Eventually, AR is going to be a more powerful and flexible replacement for VR — which is already an amazing technology. The big question is — how far away are we?
Although it wasn’t mentioned in the announcement video, Microsoft also announced that, via their partnership with Unity, there will also be full support for the HoloLens in Unity, one of the most common tools used for indie game development. The support will include the ability to render through the HoloLens, and to get input from the device in various ways. Both Unity Free and Unity Pro will have free HoloLens support. While the integration isn’t complete yet, Microsoft is already giving short courses at the Build conference using the prototype Unity integrations, to give developers a first look at what coding for AR will be like.
This is a welcome but not a surprising move. Any new platform lives or dies on the quality of its launch content, and the HoloLens will be no exception. Besides, Microsoft has always been committed to developer relations.
The HoloLens is inching slowly towards commercial reality. I maintain deep reservations about some of the limitations of the hardware, but the device is undeniably cool, and something like it will one day be very useful. Are you excited for the HoloLens? Did you get a chance to try one at the conference? Just want a virtual fish bowl? Let us know in the comments!