CW: You've also demonstrated shopping in a three-dimensional virtual world that maps to a real-world location. What is the significance of that?
Mundie: That demo showed what it would be like to navigate through a model of the real world. I call it First Life because many people have experimented today with Second Life. That world is a synthetic one. It did not get a lot of broad usage because in reality, there are not that many people who want to build their own world or operate in some alter ego.
The technology allows us to build models of the real world at a resolution that allows them to become a navigational metaphor. You get another element of a natural user interface. In this case, it's not natural in that it is simulating human sensory input and output, but rather it uses those senses that you have to interact with a model that is a world you already understand. It gives us another natural way to get stuff done in cyberspace.
What I demonstrated was the idea that I could walk down a [virtual] street in the way I would normally navigate it in the real world, enter a shop and look around and interact with things much as I would if I were at the physical shop. The goal was to maintain as much fidelity between what the 3-D model of the world was and what the actual world was, such that the two became increasingly interchangeable.
If I know how to walk around in the real world, I know how to get around in cyberspace. If that lets people do a lot more things without having to perfect using an Xbox game controller to navigate in 3-D, then that will further expand what these cloud-plus-client applications will look like and how many people will be able to use them.
CW: What role will robotics play in this world of the future?
Mundie: People think of some anthropomorphized machine like Terminator 2. I don't think of robotics that way at all. Today, industrial robots perform dedicated tasks, and they're machines that are designed to optimize that [task]. We're moving to an environment where the kind of synthesis that goes on assembling a robotic system will be done whether the robot is a physical realization -- a machine -- or a virtual realization like Laura the robot receptionist.
The design and operation of these systems is virtually identical in the two environments. Building the tool kits that will allow these highly distributed, highly concurrent systems to be built is the next big thing in robotics. As we perfect that and people become more accustomed to having these artificial helpers around, either physically or logically, we will start to see more acceptance of [robots] performing tasks that today we associate only with people.
What tasks do you see these robots taking on? My near-term dream is Laura the robot doctor or physician's assistant. If we had that humanlike ability to communicate with the computer in natural language and yet embody an expert system like what's required to do medial diagnostics and prescribe treatments, we'd be creating the only scalable solution I know of [that is capable of] delivering health care to another 5 [billion] to 7 billion people in the next few decades.
There's no way to scale the current model of health care to the people who don't have any today. Something's going to have to change, and I think information technology is it. Many times you say that and people think, "I want better electronic medial records." But I think that's the uninteresting part of the problem.
CW: The idea of electronic medical records is uninteresting?
Mundie: Yes, in the way that people have historically talked about them. In fact, this Amalga product that we've been building and deploying really moves beyond the idea that you need a prior, specified medical record. It just ingests all of the historical medical information, synthesizes metadata, flattens it out and creates a new abstraction for dealing with all for the medical information.
The technology we demonstrated bypassed the traditional primary view of what information technology should do next in medicine. But even if you solved all of those problems of medical record-keeping, you still don't have a way to scale heath analysis, diagnosis and delivery to the billions of people who don't have any at all.
The same may hold true to some degree in education. Maybe if you had Laura the teaching assistant that complemented teachers in a rural village, as well as Laura the physician's assistant or doctor in a box coupled with sensing technologies, many forms of nonacute medical care could be rendered to people who today have none at all. So many of these things are important in dealing with some of the societal challenges we have on a global basis, and they ought to be big businesses to boot.
CW: In the era of client plus the cloud, what will be the role of Windows? Will the operating system be as relevant to the end user as it is today?
Mundie: How relevant was Windows when you thought the world was DOS? The answer is it became pretty relevant. That's the way I think about this problem now. We're going to move to a new platform with new models of human interaction solving new problems at a higher level of abstraction. The operating system will be the thing that creates the mapping between the physics of the computing environment and our ability to write these applications and portray them for people.
In some of the things I described, there's no shell, no graphical user interface model, [but] there's still an incredibly important role for the operating system.
You may not have the same direct association of the operating system as a part of the application. This doesn't mean you won't have clients in screens. That correlation remains. I'm describing a world where there's no less of a requirement for controlling complex hardware that arguably will get even more complicated. But the boundary between what the user associates as the app, what part lives in the cloud, what part lives on the device in their hand -- those boundaries will be blurred.
To make the machine work, there's going to have to be operating systems on the devices that make up the cloud and on the devices that make up the client. The world I'm describing is one where those two things will be operating in some very symbiotic relationship.
There will be a new class of apps, and I think that those will be as different as the difference when we moved from the command-line interface to the Windows model. That's the way I see the future.
CW: Microsoft has had some famous flops, such as Microsoft Bob and Clippy, the infamous animated paper clip in Office. Why is it so hard to make things easier?
Mundie: Why is it that we haven't been able to create the starship Enterprise yet? We see it on Star Trek, we have a fairly clear model of space travel. What's so hard about it? The answer is, it's just a big engineering problem and it takes time.
Whether you're a science fiction person, a visionary in computing or someone who's just trying to think about a practical way of making something like television better, it turns out that it's a lot easier to envision what that future thing should be like than it is to actually build it. Then, even after you build it, there's hysteresis in the system that resists the change. So you have to have some big thing, whether it's economics or whatever, that forces you past that hysteresis point where you have stake in the new model.
We called [Microsoft Bob and Clippy] the social user interface. We were trying to make the machine add a more social view to its interaction. Those things may be anticipating something that we still ultimately want to get to, and the natural-user-interface things we discussed here may be a stepping stone to that day.
Those were a valiant attempt, at a time when the computers really weren't powerful enough, to introduce some of those types of contexts. But in both of those cases, in contrast to the [Microsoft Office] Ribbon, for example, what we were trying to do was introduce them as an adjunct thing without changing the primary model of the user's experience.
Both were intended to help you. It was more like saying, "We have this character-based social interaction model to help you." But in many cases, people said, "I don't really want the help. How do I make that annoying guy go away?"
That's very different from saying that ultimately, we're going to change the complete model of how people interact, not just in an adjunct function like help but in the fundamental way in which you interact with the machine. But that was certainly not possible at the time. So you could say it just failed because it was a dream for starship travel and all we could build was automobiles.
CW: Was the Microsoft Office Ribbon an adjunct technology as well?
Mundie: No. We realized that people were having a tough time discovering in a natural way what the system could do for them. We saw that [Clippy] didn't work. The Ribbon was a way of creating a user interface that in the natural course of usage would expose and let people trial, at zero risk, the rest of the features of the product.
It morphed in a task-oriented way to expose the features that were the most useful, even if you didn't know they existed. For example as you roll across the formatting buttons, your document is immediately -- but only temporarily -- reformatted. You don't have to click it and then decide you didn't like it and then undo it. As you scroll across, you see the one you like, you stop, you click, it's done. It becomes a risk-free, low-cost way of learning in the process of doing.
This story, "Microsoft's Mundie on the Next Big Thing" was originally published by Computerworld.