Let's not mince words: I am visibly aroused by Valve's new Virtual Reality headset.
The Vive, created in partnership with HTC, boasts perfect tracking of your head and two controllers, a comfortable 2160 x 1200 resolution, and a ultra-sharp 90hz OLED display. It's coming out this year, and it offers interactive VR experiences that provide a sustained sense of physical presence in the virtual scene.
To get a sense of just how good the tracking is, check out this video of "The Gallery," a VR game being played on the Vive.
The Challenge of Free-Roaming Virtual Reality
The Vive is fantastic. However, if you read the reviews of the prototype headsets that are trickling in, you start to notice that one specific complaint keeps popping up.
"During my trial with the prototype, I wore a belt to keep a bundle of cords out of my way — and now that I think about it, Valve must've had someone walking around with me to make sure I didn’t trip over any slack that I created as I moved around the room."
"Because it's a fully interactive, walking-around experience, you need to clear out the furniture from the center of a room and try not to trip over the cable trailing back to the PC."
"And that's probably the main weak point with HTC Vive. It's like The Matrix in that feels real but also in that you are wired into a PC with a cable that comes out of the back of your head. That will occasionally limit your free movement and is sometimes (but not often, in my experience) a tripping hazard. It is not a very big problem but noticeable anyways."
Some of this is because the headset is a prototype with five cables (the consumer version should have just one, a combined USB and HDMI cord connecting the headset to the PC). However, the problem does seem to be fundamental to wired headsets which allow you to walk freely. Cords are a tripping hazard, and also just kind of a bummer. They prevent you from taking your VR device out of the room, much less the house.
This will become more of a problem as we start to use VR as a basic interface for our operating systems.
We didn't have PCs for very long before we wanted a mobile version of the experience (laptops, smartphones, and tablets). VR will be the same way, particularly as we begin to integrate augmented reality features along the lines of the Microsoft HoloLens.
Unfortunately, it turns out that unplugging VR headsets from PCs is a major technological challenge, and one that we still don't know how to solve. There are, however, two promising technologies on the horizon that may help to resolve the issue.
Solution #1: PC Streaming
One obvious approach to the program is to use your PC to render the virtual world, and then stream the rendered images to the headset over WiFi (like Steamlink, which streams games from your PC to your TV). The problem with this approach is latency. In virtual reality, you want to minimize the delay between moving and seeing that motion reflected on the screen. The sweet spot is generally agreed to be under 20 milliseconds.
Modern video streaming can get down to under one millisecond of latency, which is a great start. Unfortunately, an uncompressed 1440p 90hz video stream is almost 8 gigabits per second of data. Consumer WiFi routers can handle about 100 megabits. Even the highest-end industrial routers max out at about 800 megabits, eight times less than is needed. To fix this, you need to compress the images before you send them. The compression and decompression process adds about 80 ms of latency, which will make you sick.
There are a few possible solutions to this problem:
Bandwidth Improvements
One approach is to try to increase how much bandwidth is available. 5G networks, due to be available in five years, may enable speeds of around one terabit per second, which could handle futuristic 8K HMDs with no compression. In the nearer term, Li-Fi (an internet technology based on rapidly pulsing LEDS) could allow transfer speeds of 10 GB/s at very low latencies, which would allow for wireless versions of current-generation VR headsets in the next few years, though the technology isn't available to consumers yet.
This is probably the best solution to the problem, but it won't be available to consumers for at least a year or two, may be expensive, and the resolution will probably lag behind the highest-end wired headsets until proper 5G networks arrive.
Local Warping
A more complex option would be to accept the 80ms of latency, and try to correct for it after the fact, by warping the old image to the new head position. This still produces some input lag, but not the nauseating head-motion kind.
The standard approach to this sort of problem is asynchronous time-warp, which collects data from the PC and warps it using a small GPU on the headset to display the old image from a new perspective, allowing the headset to update fluidly despite the high latency.
At first blush, this sounds like a magic bullet. Unfortunately, there are some big catches. To quote Michael Antonov, an Oculus researcher,
"Positional judder is one of the most obvious artifacts with orientation-only timewarp. When you move your head, only the additional rotational component is reflected in the ATW-generated images, while any translational head movement since the frame was rendered is ignored. This means that as you move your head from side to side, or even just rotate your head which translates your eyes, you will see multiple-image judder on objects that are close to you.
[...]
One possible way to address positional judder is to implement full positional warping, which applies both translation and orientation fixups to the original rendered frame. Positional warping needs to consider the depth of the original rendered frame, displacing parts of the image by different amounts. However, such displacement generates dis-occlusion artifacts at object edges, where areas of space are uncovered that don’t have data in the original frame."
There are several techniques for making up data to fill the holes created by positional timewarp, but all of them leave behind artifacts, and the ones that leave the fewest artifacts are the most computationally expensive. The problem is similar to that faced by those attempting to display VR depth video.
In order to fix this problem perfectly, developers have to find some way of transmitting extra data to fill in the holes. A few possible approaches:
- Render far-field and near-field objects separately and stream both images
- Render a light field, which can be re-projected by large head offsets without disocclusion
- Develop extremely sophisticated head motion models, and use prediction to minimize disocclusion errors
It's not clear what the best solution is here. None of the approaches we know of are perfect, and some of them require more research. Figuring out if re-projection is a good idea boils down to how much we can minimize the artifacts. It's a promising possibility, but there are a lot of unanswered questions.
Solution #2: Local Rendering
A completely different strategy is to forget the PC, and do the rendering locally on the headset. This is how Microsoft's HoloLens and Samsung's Gear VR work. Not much is known about the HoloLens, but the Gear VR has a number of serious limitations. Its mobile GPU can handle some modest VR experiences (with intensive optimization), but barely. Heat management is an issue, the resolution and refresh rate don't rise to the level needed for presence in the scene, and the headset has no positional tracking. Some of this is due to the limitations of mobile technology, and will get better soon, buoyed by the tide of Moore's Law.
However, some of these issues are the result of having to engineer around a smartphone that wasn't designed for VR. If you were building a mobile VR headset from scratch, you could do a lot of things differently. Designers could have two high-end mobile GPUs, a very high-refresh-screen, and a big battery. They could also spread components out for better weight distribution and heat management. The resulting form factor might look a lot like the Sony Morpheus.
Or, if engineers wanted to get ambitious, they could build something more like mini-PC: a small wearable block of computronium containing one or more laptop GPUs, a fan, and a large battery. This device would have a form factor similar to a Walkman, and would drive an HMD via a single cable.
The current NVIDIA flagship GPU, the GTX 980, which is what Oculus uses internally, is only about 40% faster than its mobile counterpart, the 980M, and it's tiny. It's probably possible to fit two of them into a fanny pack, with room to spare for the rest of the guts of a small mobile PC.
Note to hardware developers: under no circumstances allow the press to describe your product as a 'fanny pack.'
This form factor would be chunkier than a self-contained headset, not to mention expensive -- back of the envelope math suggests a price point of about $2500. It's possible that modern consumers won't be interested in products that unwieldy. However, it is the only way for a mobile VR headset to deliver performance competitive with PC VR, and I suspect that hardcore gamers and other irredeemables will go for it. There's a long history of nerds driving the forefront of technology by being willing to wear products that make them look ridiculous.
Right now, mobile headsets suffer from an inability to provide positional tracking, causing a nauseating sense that the world is attached to your torso when you try to move. However, this problem may soon be solved. In the short term, Valve's "lighthouse" positional tracking technology is a great solution: the laser projectors are dumb, which means they can be tiled to cover large spaces, and can serve many devices at once. As a result, over the next few years, lighthouse tracking might become an inexpensive service that homes and businesses provide, much like WiFi is today, allowing you to use mobile VR hardware in most indoor spaces.
In the longer term, as depth cameras improve in resolution and noise level, it should begin to be possible to use them for positional tracking, with enough precision to provide a sense of VR presence. This will allow for self-contained VR products suitable for outdoor and augmented reality use.
The Future of Mobile VR
In the long run, mobile headsets are where VR technology is going.
We can do the math of Moore's law by figuring out how many doublings we are away from being able to render photorealistic virtual worlds on mobile devices. High end PCs are maybe five doublings (another console generation) away from being able to deliver photorealistic gaming experiences at 1080p. For VR, we need one doubling for 3D, and three more to render at the 8K-per-eye needed to eliminate blur and the screen door effect. To account for the mobile platform, we need about four more doublings to catch up to high-end PCs.
If you add up all the doublings and multiply by the eighteen months dictated by Moore's law, you get roughly twenty years (2035), which is the point at which nobody will be able to tell the difference between a VR scene rendered on a smartphone and one rendered on a high-end gaming PC. At that point, there'll be no reason to ever use a PC-driven headset again.
Until that day comes, I suspect we'll see both wired and wireless VR PC-driven VR headsets for a while. Wireless headsets will be more convenient, but wired headsets will offer better resolution -- at least until 5G networks are fully mature.
On the other side of the divide, mobile headsets will continue to improve, and eek away market share from the PC as we slide far enough up the curve of diminishing returns, with more and more people willing to sacrifice a modest amount of fidelity for flexibility. The competition will be fierce and, hopefully, good for the consumer. Personally, I can't wait.
Excited for wireless headsets? Know of a technology we missed? Let us know in the comments!
Image Credits: Barone Firenze Via Shutterstock, "Recording Isan's...", by Edvvc, "Phubby," by Kevin Lin, "HTC VR," by HTC, "Steam Link," by Valve, "Moore's Law," by Wikimedia