Philip R. Cohen and Scott Lind, two co-authors of "Sketch-Thru-Plan: A Multimodal Interface for Command and Control," (cacm.acm.org/magazines/2015/4/184700) a Contributed Article in the April 2015 Communications of the ACM, discuss their work.
00:00 Hearing, smell, taste, touch, sight. We communicate quickly and effectively by engaging our five senses in combination: We talk and point and touch.
00:22 But your phone, with its myriad sensors, makes you switch among them to enter information. That takes time -- time that's short in life-or-death situations.
00:35 Join us as we talk with Philip Cohen and Scott Lind, as they tell us how they've combined drawing, speech, and handwriting recognition to speed up operations planning, in Sketch-Thru-Plan: A Multimodal Interface for Command and Control.
00:53 [Intro graphics/music]
01:03 "The winning general makes many calculations before fighting." That precept is well-known to those who lead firefighters, search-and-rescue teams, and military forces. It's as true now as when Sun Tsu said it over 2,500 years ago.
01:21 SCOTT LIND: Having this much going on -- and we're talking 30-35 soldiers here -- if you're moving a batallion of 500 soldiers on the battlefield, and they all have rifles, they all have vehicles, they all have radios; there's a lot of things going on in a fairly confined space. And that's why this positive control... is essential.
01:41 Creating such "command-and-control" plans on paper or terrain boards is fast and intuitive. But then you have to translate those plans into actions and resources, re-entering the details over and over.
01:55 So the United States military started moving to a digital system called "Command Post of the Future", or CPOF, over ten years ago. While it beats out paper at data transfer and analysis, getting information in isn't nearly as easy as drawing on a map.
02:12 LIND: So this particular plan, if I were to place this in a modern command-and-control system -- of any system -- this could take me an hour if not longer... It's an 11-14 step process to enter this named objective, Objective Natasha, into CPOF.
02:30 Now a company in Seattle named Adapx believes that it has the answer. Its system, called "Sketch-Thru-Plan", allows planners to enter their instructions using a multimodal combination of speech, writing, and drawing.
02:45 DR. PHILIP COHEN: So this kind of user interface enables you to offload the data that you're planning into the modality that's most appropriate for the task at hand... Language is good for describing things, and drawing and pointing is good for entities that are on the screen.
03:04 LIND: Objective Natasha.
03:06 COHEN: Well, this began in the early '90s while my wife and I, Dr. Sharon Oviatt, were at the Oregon Graduate Institute... And the military symbology language is a very good test case for sketch recognition algorithms.
03:23 On the hardware side, Adapx has implemented Sketch-Thru-Plan on a variety of platforms. And in a full return to its roots, it can digitize plans created with Anoto's digital pens and paper.
03:35 Other planning and data-analysis systems have also grown up. So although Sketch-Thru-Plan has its own basic planning functions, it also converts the user's input into a data feed to drive legacy systems.
03:49 LIND: So we leverage the hundreds of millions of dollars in these legacy systems, regardless of what it is, and provide the user a very natural user interface to be able to quickly get data in here without having to worry about going to the four-week school to master System ABC.
04:05 It's a change welcomed by soldiers and emergency responders who got used to drawing, talking, and writing on their smartphones. Now, Dr. Cohen believes these military-inspired advances may find their way back to the phone in your pocket.
04:20 COHEN: So the fact that you have a device that has a camera pointed at the user means that it can both hear you and see your lips moving, and fuse the results of your voice and your lip motion to get better performance in noisy environments, for instance when you're outside and there's a jackhammer or whatever. So this concept of combining modes to overcome their weaknesses is very powerful... And the idea of being able to speak and draw or sketch, or combine other modes of communication on your cell phone, will dominate in the next decade.
04:57 VO: Find out more in this month's Communications of the ACM, in the contributed article, "Sketch-Thru-Plan: A Multimodal Interface for Command and Control".
05:08 [Outro and credits]