Throw away the Wii — 3D sensing technology due within a year, says SoftKinetic

May 26, 2008 | Chris Morrison | Comments |

While we’ve written several times recently about the progress of next-generation, camera-based game control technologies, including the hefty funding received by Prime Sense and an earlier update on several competing companies, there’s one detail we’ve edged around: When you’ll get to use them for yourself.

That’s because most of the companies developing gesture recognition technology aren’t sure, themselves. The firms developing the 3D cameras that make motion-sensing gaming possible have to work through intermediaries to get their products to the consumer market, and in the Byzantine world of game development, no small company can predict which way giants like Sony and Electronic Arts will lean. However, a company called SoftKinetic recently stepped up to tell me that it thinks the moment may be close.

Here’s the basic idea behind 3D gaming: A camera mounted on your TV or computer captures your movements, and with a combination of sophisticated hardware and software, extrapolates them into three dimensions (the exact details vary by company). The effect is as if you had Wii controllers strapped to your body. So in a boxing game, for example, your punches, as well as the ducking and weaving of your head, will be represented in the game world; or, outside of gaming, you could control an operating system like the one in Minority Report.

The important detail in getting this to work isn’t just good cameras. There are two other, equally important factors. One is price — the average consumer won’t pay a huge amount for any controller, no matter how cool. The second is getting console makers and game developers to run with the technology.

The latter is SoftKinetic’s business. The company makes a software development kit (SDK) and API that makes the switch to gesture recognition simple for game makers. Essentially, SoftKinetic takes the data the camera is returning and translates it into straightforward commands for the gaming people. SoftKinetic has partnerships with the four major camera companies, according to CEO Michel Tombroff, and is hammering out agreements with game development companies to get the technology on the market.

Tombroff, who says his company is the only one around acting as the software intermediary between camera makers and game developers, thinks that the breakthrough will come soon, with the first cameras hitting the market within a year. Progress is going “very quickly”, he told me, before hauling me off to a demo to show off how well the technology works.

To be clear, no major games have been developed yet with gesture recognition technology. However, SoftKinetic has been working in the field for years, for military and industrial applications, and has the consumer SDK done. No game developers themselves, they’ve nonetheless cobbled together a few rough applications to show off the goods.

The demo, done using a Prime Sense camera, impressed me with how well the technology works. One quick game involved ducking and contorting into various positions to avoid oncoming obstacles, while another had me flapping my arms to control an avian character. A third was a kung-fu demo, in which I could kick around blocks and break boards (an example of that is in the embedded YouTube video, while the others are here).

Easily the best demo, though, was a modification the SoftKinetic team did for Quake 3, a game that involves running around gunning down baddies. Running and jumping were simple; for the former, I just had to lean forward, and for the latter, of course, I just hopped in place. The rest took an odd turn. To represent my gun, I had to hold my right arm out straight in front of myself. To fire, I had to flap my left arm up and down, chicken-style. Flapping my right arm switched weapons.

Outside of dancing to 80s hits, I usually try not to let myself look quite that stupid in front of other people, but the oddball motion scheme actually made playing the game more enjoyable than using a controller. Even odder than the scheme itself, though, was the fact that it felt natural within about a minute.

That demo proved the concept to me, but also illustrated where today’s technologies will converge with tomorrow’s. Even if a player could get over the ignominy of flapping their arms around to represent gunning down enemies, they would probably rather hold a controller of some sort — either a fake plastic gun, or a controller like the one the Wii has. Exactly what the end result will look like will probably depend on the game. Tombroff isn’t bothering to speculate — designers, he says, will do a far better job at creating sensible design schemes than his company can.

As long as console makers like Sony and Nintendo can provide various technologies like the Wii controller and 3D cameras cheaply enough, they’ll probably end up combining several, not least because players may not always feel like being active. “With the Wii you can play from the couch — you can fake it. Not with this,” says Tombroff. (So no, you won’t really throw away your Wii.)

Unfortunately, even with SoftKinetic acting as middle-man, it still has to hope that big companies will take up the torch and throw their support behind gesture recognition; Tombroff says he’s hammering out a deal with a major company, but the majors are infamous for canceling plans and breaking promises. Even if they do take up gesture recognition, you won’t have a suite of games by this Christmas. At first, the technology will probably show up in arcades and advergaming (think interactive screens in stores), then move into consoles and computer games.

A possible alternate scenario involves the cameras becoming small enough to fit on, or even in, desktop displays and laptop screens. Another company, 3DV Systems, has said it plans on bringing a camera to the PC accessory market by this Fall, providing a possible work-around to the console market.
.

Next Story: Google challenge winners give Android thumbs-up
Previous Story: World Bank will raise $5.5 billion cleantech funding with the U.S., Britain and Japan

Bookmark and Share

Tags: , , ,

Photo of Chris Morrison

About the Author, Chris Morrison

Chris Morrison writes about cleantech and environmental issues for VentureBeat, with occasional forays into gaming and semantic technology. He got his start writing about tech for Business 2.0 magazine, but quickly realized new media was the ticket when that institution closed its doors in 2007. Chris has also covered public equities and regulatory issues. He originally hails from southern Virginia, graduated from Evergreen State College in Washington, and now lives in San Francisco.

  • The idea itself looks very interesting, however, I, for one, wouldn't play a game using this kind of control. The pleasure of the game is that I can dive into gameplay, then stand up off my chair and get back to my regular activities. Last night I had a chance to play "Box" on Wii for the first time. I was exhausted after ten minutes of intensive hands motion (and I'm OK to spend hour+ in the gym, mind you!). Being THAT exhausted -- is definitely not what I want from the computer game.
  • We are working on something simular but better as usual.
  • BillyVonRuple
    The exhaustion factor is tied to the current game designs that are based on the limited tracking capability of the Wii-motes. Think of the 3D depth camera more as a path to excellent full body tracking for training applications, ultimately leading to more realism in full body games. For instance, proper form for martial arts moves, tai chi, yoga, pilates, etc. could be learned - why not earn a brown or black belt at home? The WiiFit product can only detect your balance, which is important but not enough information to give complete training information. Chris Morrison's title for the piece is way off the mark. The Wii has already defined itself as the "motion" console, and is selling peripheral based products extremely well as evidenced by WiiFit. It is the logical console to target a 3D camera peripheral for. The other element that is missing in this article are the longer term patent holders and 3D tracking specialists in the gesture space, namely GestureTek. Both the camera providers and software producers will have to handle those licensing issues before launching anything in this market.
  • Realist
    Why not earn a brown or black belt at home? Because you can't actually do what you need to do to earn something like that. You can't spar with other people. You can't learn what it feels like to get hit and adjust to it so you won't freeze up in a (hypothetical) real-life situation where you need to defend yourself. Boxers won't learn how to fight body-to-body, smothering the other guy's punches and tiring him out by leaning on him slightly. Kickboxers won't know how to block kicks with their legs, or how to roll off of body kicks at the last second. Grapplers like judo and jiujitsu students, well, they can just plain forget it. The whole problem with this sort of thing is that it's not even close to the real activity because there's no physical feedback, and it's a poor training tool because the AI required to teach a person how to correct subtle errors in their movement (while tailoring the movement to their particular body type, and any health issues like stiff knees or whatever) is non-existent.

    Forget the part about the AI being a good enough teacher for a moment. The simulation isn't physically sufficient. It's fine for toys and party games, and it might even work for solo exercises like yoga, but anything that requires interacting with any real, weighted object and/or another human being cannot be taught with this control scheme.

    Nobody will ever be able to learn what it is to fire a rifle, use a firehose to douse a burning building, hit a heavy bag, or escape from an arm-bar with a system like this. Sorry, but the system simply cannot do it.
  • Billy VonRuple
    I agree with the view that activities requiring physical resistance will be difficult with any simulation technique. In the US military, they say that 5% of the motion is enough of a cue in a simulator such as a helicopter, but 0% is another story entirely.

    However, for many applications including the aforementioned Yoga and Tai Chi, and many exercises where a user could incorporate weights into the regime there could be an excellent training benefit by using depth tracking hardware to confirm that your mechanics are correct. Further, for applications like a golf swing analysis, batter's swing, and pitcher's throw, where you can actually hold onto the real device, the potential benefits are obvious in terms of relating your motion to the ideal.

    It looks like we are going to get to see what people do with it, since the rumor mill has it that there is a confirmed product release for depth on a console for fall 2009.
  • Agreed with BillyVonRuple -- if you can see this as the very beginning stages of a new form of control, there's a huge amount of room for refinement. Unpleasant or unnecessarily exhausting control schemes will be weeded out in favor of more graceful implementations.

    You'll have to forgive me for the title. Creative license, etc.
  • a kid
    This looks awsome!! If they get a console deal, imagine how combat games will be, especially those like Zelda and Prince of Persia. Who cares if there's not an actual person their, it's for fun not combat training. In fact, that might not be the best idea because we would end up with kids who had been taught the moves, but not the restraint that goes with them. But, hey, we could finally end up with a proper swordfight, like the ones we wanted from wii. Fingers crossed!