How will the convergence of 5G and AR and artificial intelligence lead to the creation of experiences that no one’s ever imagined? Join Forbes columnist and author Charlie Fink to find out!
Alan: Hello everybody. Today’s guest is one of my good friends and someone I consider a mentor. XR consultant, columnist, speaker, and author, Mr. Charlie Fink. Charlie Fink is a Forbes columnist and an author of two AR-enabled books – Charlie Fink’s Metaverse: A Guide to VR and AR, and Convergence: How the World Will Be Painted with Data. Charlie is a former Disney, AOL, and AG Interactive executive, famous for coming up with the idea that turned into The Lion King. Charlie was EVP and COO of VR Pioneer Virtual World Entertainment. He was an SVP of AOL Studios and president of the American Greetings Interactive. Charlie founded and exited to venture-backed startups and has produced over 30 award-winning Broadway musicals. Charlie is leading the way in XR for business by covering and reporting on everything XR-related. With that, let’s welcome Charlie Fink.
Charlie:
Wow, I don’t know if I can live up to an introduction like that.
Thank you Alan.
Alan:
Oh you’re amazing. You’re amazing and thank you so much for being on
the show. First of all I want to congratulate you on the launch of your new
book Convergence, an AR-enabled book about AR. Maybe you can tell us a little
bit about the book, and why it came to be.
Charlie:
Well, we were looking at doing a second edition of the first book
which sold very well, but it soon became apparent that the AR topic in
particular had been sort of glossed over in the first book, which was largely
about VR. So, I started to think about the idea of doing a new book, and I got
a lot of support for it in the community. In fact, it is a sponsored book,
meaning I got enough donations that I could afford to write the book and get it
printed. It’s printed on premium paper and it’s priced high like a textbook at
fifty dollars – of course it’s filled with an hour of animation – so it’s not a
normal book and it doesn’t adhere to book economics. So to make it work I
really needed to build a community of about 100 people around the book, many
providing augmented scenes and many actually contributing thought leadership
and reporting. The topic of the book kind of evolved because I, you know,
originally I was thinking the title would be something like ‘the many modes of
AR,’ or ‘how the world will be painted with data,’ which is a phrase I coined,
which suggests a fully-scaleable AR cloud. Which would make the devices we
use a whole lot more useful and interesting than they are today, where we use
something called marker-based AR.
And as I got into the book and editing the
work of my collaborators, it was clear everyone was talking about the same
thing, which is this world of ubiquitous, wearable computing. What the final
form factor will be we can debate, but this idea of a magic-verse based on a
mirror world became the theme of the book, which is essentially the convergence
of 5G and AR and artificial intelligence that would manage all of this data for
you so that it is wanted and contextual. Otherwise, if it’s interrupting you, I
don’t see how that’s possibly a welcome augmentation, right? It can’t be showing
you advertising; it has to show you what you want when you want it or where you
need it. You don’t need to see the weather in front of your eyes spatially,
although it would be fun from time to time. But the truth is you know what the
weather is.
So that became the title of the book,
Convergence. And if you look at history you know there have been a number of
moments of sort of platform innovation that launched heretofore unconceived
ideas. So
if, for example, look at GPS in a phone, right, there would be no Uber, no
Seamless. There’s a whole range of services, and GPS is getting better now that
it’s being married to computer vision, so you can get highly-localized Google
directions – really accurate, down to a couple of feet.
So convergence is beginning, right? The combustion
engine is an example of convergence. What did it enable? Look at steam: it
enabled factories, it enabled transportation and railroads in the same way that
GPS and the Internet has essentially created five of the biggest companies in
the world. So that’s the topic of the book. And of course, it’s filled with
augmentations, as opposed to the first book, which had kind of a funny cartoon
series that was made by living pop-ups. This has 50 augmentations, mostly from
the sponsors. There is a little living pop-ups layered in just because it’s so
funny and so incredibly well-done.
Alan:
It is amazing.
Charlie: On the topic of technology, so, it’s great to work with them and all the collaborators. We made this book together… although, of course, I had to do that horrible 12 weeks of doing nothing else and sleeping five hours a night in order to ultimately pull it all together with all the pictures; the book is beautifully designed. You know that’s an award-winning design that Porteramic came up with. So I’m feeling good about it, people seem to really like the book. It’s much denser than the first book. I think the first book was 40,000 words; this is 90,000 words. So I would say it’s the most in-depth book about what’s happening in AR and what the trends are today.
Alan:
Awesome. So, Charlie, you know, your book really covers a lot. So
it’s got in there everything from kind of social and social augmentations and
things like this, but really, for the listeners of this podcast, we want to
drill down into the business applications of this. In the book, you discuss
military, telepresence, design and architecture, medical, retail, marketing,
journalism, social media, entertainment, gaming, and education. Which of these
do you feel right now holds the greatest business potential and why?
Charlie:
Well, enterprise AR just in its infancy, but it transforms anyone
who’s involved with either the manufacturing process or logistics, which is
just about every company in the world. So AR is going to make the jobs that
people are already doing much easier. There will be adoption issues. It’s quite
inexpensive compared to other methods; it’s no more expensive than an RF gun,
and it enables all sorts of new functionality like, as you mentioned, see what
I see. So you can call a remote expert. And again, that’s an innovation that is
being driven by the camera. You can also do that on a pad or a smartphone, and
of course the use of the camera and the digitization of real environments
spatially, you know, is going to enable a whole range of functionality. They’ve
already got, you know, in your Apple phone, RF detection is already built in.
That’s the beginning.
Alan:
Interesting. So then I think one of the questions that people are
going to have is, at what point do we go from a mobile device – so, a phone or
a tablet – and then go to glasses, and where kind of is that divide, and is
there a solution that basically bridges the gap between that, and what do you
see as the uptake of glasses versus devices such as iPads?
Charlie:
There are two things. One is, glasses allow workers to keep their
hands free. Glasses allow workers to see schematics and the handwork in the
same field of view, so that many fewer mistakes are made. When you’re wiring a
jet, mistakes are…inspection and correction takes as long as the actual
wiring of the jet. So you’ve got companies like Boeing and Airbus saving tens
of millions of dollars per jet – I don’t know, Boeing makes 3,000 jets a year –
so that’s quite extraordinary in terms of the amount of savings, and they have
a system that they can use themselves and change out the schematics. The
workers can take a picture of their work when they’re done, so that if a
mistake is found, the picture could be reviewed, or their picture could be
reviewed in real time.
Alan:
So interesting.
Charlie:
Many companies are adapting it for that reason. But, you know,
companies should select the right tool for the job. Is this a job for Google
Glass? Is this a job for the Hololens? Or is this something that we can just
have a worker do on a smartphone?
Alan:
It’s interesting that you talk about these different headsets. And
for the listeners who don’t understand the difference between them, maybe you
can walk us through kind of the beginning of it. So, you know, maybe a
phone-based, right up and through to something like the Hololens, and kind of
what are the differentiations between them?
Charlie:
Well, first let me set this up by talking about the difference
between AR and VR.
Alan:
Sure.
Charlie:
VR has a lot of fantastic applications in training and simulation.
That is its most powerful use. And it is quite effective at those things. Of
course it also has telepresence, so you could be virtually present in a place
like Egypt – not in real time, but in space and simulation. And a lot of
companies – anybody who’s got vehicle-based businesses, machine businesses –
there was a fantastic example, up South by Southwest. The Raymond Corporation,
which makes forklifts, has created a simulator module that you can clip onto a
real forklift. You can have that in your warehouse, and whenever you’re
training a new worker, you have them spend a few hours on the simulator.
Alan:
That’s awesome.
Charlie:
So that’s what VR is good for, right? It fully includes the world
and creates a new reality in which you can be present. So that’s a very
powerful quality that it has, and again, it’s a platform and people are
innovating. But the success seems to be, right now, limited to both medical
applications and training applications. There are some games and social
experiences. I mean, there are millions of people going into Rec Room. So, it has some
social benefits, you know, because that’s not a game – that’s an experience.
Not to say that games can’t be experienced. But in my opinion, I don’t think
the killer app of virtual reality is games.
But let’s now switch to the topic of
augmented reality, which is really what this conversation is about. It’s much
more broadly applicable to business than VR. They’re both technologies that
that share some things in common, but largely are separate. Because you need
the real world in order to augment it.
Alan:
Absolutely. What you’re saying here is virtual reality is great for
kind of those before you’re on the site – so, training, and then maybe remote
collaboration for design, maybe? And then AR is more on the practical side of
your everyday work.
Charlie:
Yeah that’s one way to think about it. In other words, think about
it on a spectrum of immersion… although that doesn’t really make sense. Let’s
start out with the basic device, right? You can do a lot of the things we’re
talking about – see what I see with remote experts, documentation, geo-located
instructions – you can do all that with a smartphone, right? That’s cheapest,
because the worker already has a pad or a smartphone with them. So no
retraining required. This is not a worker who necessarily has to have their
hands free in order to read the information that they need.
Charlie:
So the cheapest and the simplest is, of course, in many cases, all
that’s necessary. You have to look at the segment specific-elements of it that
can be augmented. Not everything in the business process is going to be
benefited by augmented reality. Of course, anybody who has a warehouse can be
hugely benefited, because you take the functionality of that pad, the barcode,
the RF gun, and you put it all in a head-mounted display. So that’s actually a
fantastic use case, and there are a lot of companies focused on exactly that: warehousing
and logistics. Because augmented reality is so effective, and they use
something called the monocular micro display, which is really a tiny TV set
hanging out there in your peripheral view, and you’re mostly looking past it,
but you can focus on it when you need it.
Alan:
Kind of like heads-up instructions.
Charlie:
Yeah it’s Google Glass. You know, there are a lot of
companies that probably people on this podcast have never heard of, nor do they
need to know their names, that make monocular micro displays. Some of them,
like Vuzix, are as popular as Google Glass. They make something called the
M300, which is very popular. So, the next kind of augmented reality is called
Reflective AR, and this is where you’re using your smartphone, but you’re
balancing the image or the information on some kind of projection surface in
front of your face. And that is actually similar to the way the Hololens works,
except the Hololens is filled with sensors, and is aware of your environment, and
it has advanced optics which can place 3D objects in your field of view.
Alan:
So give us an example of why that would be important, where you need
to be contextually aware and why you would need 3D data.
Charlie:
Let’s go back to the wiring of the jet.
Alan:
OK.
Charlie:
And I’ll talk about something that Lockheed does. You put the
Hololens on, and it can place on each peg what wire goes in it. Why can it do
that? It has much better, more sophisticated sensors. So that’s a better tool
for that job in a certain respect. But it’s much more intrusive, it’s heavier,
it’s harder to wear for a long period of time. Whereas a monocular micro
display can clip one to your helmet.
Alan:
Interesting. So give us an example of, you know, maybe some
companies – you said Lockheed Martin is using spatially-aware headsets to kind
of overlay instructions and data on top of the real world. Give an example of
maybe a company that you know of that’s this monocular display. You talked
about Vuzix and I think there’s you know there’s other–
Charlie:
Vuzix is one. Kopin is one, but they made a reference design that
they’re hoping other companies like RealWear will adapt and manufacture and
market.
Alan:
Maybe describe RealWear to everybody.
Charlie:
RealWear is a company that is associated with Kopin, and uses their
reference design to create a monocular micro-display. But they house it…
their trick is the way they house it, is industrial-grade. So it clips on to,
for example, work glasses. Let’s say you were doing welding: you know, it
incorporates easily, those kinds of practical uses.
Alan:
I’ve actually tried it at CES, they had it in a work helmet.
Charlie:
Yes, I saw that too, and also when they walked on stage at AWE last
May, one of the guys was in a hazmat suit.
Alan:
Interesting.
Charlie:
So it’s a very versatile, rugged device that they made using
somebody else’s technology, but their industry knowledge and their
understanding of the customer needs.
Alan:
It’s interesting, because I think we’re going to see a lot of this,
where you have hardware manufacturers like Kopin, and for people who don’t
know, Kopin a display manufacturer that makes micro displays for these types of
glasses and headsets; I believe they make parts for the for the Vuzix Blade,
and they make different parts for different people’s glasses. But there’s gonna
be the makers of the of the actual hardware, but then there’s going to be
solutions providers, like companies like Upscale, where they’re actually
providing kind of the software back-end and other companies–
Charlie:
They are Reflect. There’s a whole class of companies that I call
“consulting integrators.”
Alan:
Got it.
Charlie:
So they’ve got a platform that they’re going to leave behind them.
They’re going to work with the client to identify the need, the right tools for
the job, and then they leave them with a SAS platform that’s object-oriented,
meaning you don’t need to be a programmer to update the app.
Alan:
Absolutely.
Charlie:
Of course, then they get recurring revenue, and that’s really what
everybody wants in media today.
Alan:
So maybe you can, in the interest of the listeners and getting
straight to the point of value creation, what are some of the companies – you
mentioned a couple there – what are some of the companies that are offering
this service? Because I know coming up in other episodes of this podcast we’ll
be interviewing these different companies, so what are some of the companies
that you are seeing that are–
Charlie:
Depends on the vertical, right? They’re sort of dividing it up by
vertical.
Alan:
Got it.
Charlie:
So you’ve got a company like Visualix out of Germany that is
specifically looking at geo-location in warehouses; micro geo-location. So it’s
using edge computing, it’s identifying a box down to six inches. So you’re
driving the forklift and it’s giving you directions to the box and everything
is automatic. And it tracks the changes in the environment. So that’s one
example. You have Reflect, which is focused largely on the industrial
aftermarket and large equipment and integrating CAD. Occipital does that as
well. Scope AR, you know, is a broad integrator, like Upscale, that has clients
in a lot of different verticals. And then of course the big daddy that really
doesn’t get talked about as much is PTC, which is a 10-billion dollar,
publicly-traded company focused, I would say, probably 65 per cent, two thirds,
on augmented reality. And then they have a legacy CAD business that has
different forms for them, and still is a lot of their revenue. Right? But a lot
of what they did was say, ‘why can’t we bring these CADs into the real world?’
Think of the application for architecture and design on-site. To see their
buildings in place before they break ground.
Alan:
Interesting. We’ve talked a lot about kind of the industrial and
enterprise applications, but you were just at South by Southwest for the launch
of your book. Can you maybe talk about some of the the marketing applications,
or some of the things that you saw there, that were maybe not as industrialised
for listeners that are in marketing or in sales or in training or HR; what are
some of the other aspects of this technology that are being used that you’ve
seen?
Charlie:
Anything that uses your camera is inherently social. The way companies
market today is they get their customers to do it for them on social media.
Obviously they do microtargeting and use Facebook and Twitter and Instagram to
try and find people in context, but they also use different strategies for
getting people to make content for them. And this is an example of the camera
as a platform enabling things that were heretofore impossible.
Amazon Prime has a series coming up called
Good Omens, which is about angels and devils among us in the days before the
apocalypse. And it’s a comedy. And they set up a compound that people waited in
line hours to get into, and it was sort of a Garden of Earthly Delights, if you
will, where you could post it in different stations and take selfies, and you
uploaded them to get prizes. They had a big leaderboard of all the social posts
that were being made, and they made it as visual as possible. Including when
you left, having a machine that would take a selfie and post it for you. And
then, they were walking all around, had costumed characters as angels and
devils all walking around the convention center, for some experiential
marketing. It was fantastically well-done activation, it probably cost millions
of dollars.
Alan:
Is this for a new TV show, is that what it’s for?
Charlie:
Yeah it’s it’s a series on Amazon.
Alan:
Amazing.
Charlie:
But companies don’t have, you know, TV commercials don’t really do
it today, and they’re still very expensive, so that was a way to take 80,000
influencers, and people were active – you know, generally people who go to
South by Southwest are thought leaders of some sort in their own social circle.
Alan:
This is true.
Charlie:
So you see a lot of big companies doing activation. There are always
the Vox Media, and Vevo, and Dell, and Deloitte all have, you know, big sort of
houses that they rent, both by the convention center and over on Rainey St.,
which is across the street from the convention center. There is a street full
of bars, you know, a lot of little houses that were converted into bars, and
those get rented by companies – probably, I’m going to guess, for about $25,000
a day. And they have events there, and they host parties there. Everyone is
trying to, you know, get in front of the people who are there. And some of them
are really popular, and they’re free, right? People wait in line to get their
drink tickets and hear a band. The other thing about Southwest is you do hear a
lot of amazing music by accident. I was at Flatstock, which is the paper print
exhibition – posters and big bands. So it’s called Flatstock. And there was a
Flatstock stage, where there was a band playing: Jonathan from Croatia.
Alan:
Okay.
Charlie:
He was really pretty good. So, those are the delights of South by
Southwest. Those things that happen accidentally that weren’t on your schedule
that you just stumble into.
Alan:
It’s interesting.
Charlie:
Brands will that too to establish themselves. Now, you asked about
specifically marketing applications for AR, because all this started with with
me saying it has to be social. So, Burger King has an app where it’s doing
computer vision to recognize the advertising of a competitor, and sort of
deface it in an amusing way.
Alan:
I actually posted that the other day and it got 50,000 views in two
days.
Charlie:
Cool. So it’ll get out and, of course, people are using it socially.
You know, it’s on their camera. They’re record it.
Alan:
Really a fun one. I love that.
Charlie:
They think it’s cool. So that’s a tremendous amount of marketing on
Burger King’s behalf that is being done by its customers.
Alan:
Absolutely. It’s interesting that you brought that one up because,
you know, of all the crazy things we’ve seen people do with AR from, you know,
selfie images and all of this, the AR – and for those of you who don’t know,
Burger King, what they did was they created an app so that if you pointed it at
their competitor’s advertising, whether it be a billboard or a poster or
whatever, it burst into flames and gave you a free Whopper, a flame-grilled
Whopper.
Charlie:
They’re using
computer vision. It’s a really good application.
Alan:
So smart. I’m assuming they’re just looking for the logos of their
competitors and then blowing them up. So what a great–
Charlie:
Or they are, you know, texture-mapping or depth-mapping the images
so that they can trigger the hyper location, right? They know where they are
physically; they need a kind of mesh to recognize it, if they’re not using a
marker of some kind, or the logo. Some logos will scan better than others,
depending on what technology they’re using.
Alan:
Absolutely. So let me ask you a question: of all the things that
you’ve seen – in your new book here, you’ve gotten military applications,
you’ve got retail applications, you’ve got – what do you think is that
so-called ‘killer app’ of augmented reality, what is that? And you know it’s
funny, because I wrote an article about augmented reality’s first killer app,
but I’m interested to know what your thought is on the first killer app of
this, in business.
Charlie:
I don’t think we…well, I mean, in business, remote experts and
work instructions and wayfinding in warehouses are the three really big ones.
Alan:
So maybe let’s let’s unpack that. So the first one is remote
instruction.
Charlie:
Well that’s where you, remote instructions, right? That was my
wiring diagram from earlier.
Alan:
So one of the things that–
Charlie:
Any worker who’s looking at schematics can benefit from having the
schematics in the field of view of the handwork.
Alan:
One of the things that we saw early on was kind of a
‘see-what-I-see,’ being able to call in an expert and–
Charlie:
Yeah, take a low-skill worker and turn them into a high-skill
worker. So, you know, the low-skill workers is on the factory floor; the
high-skill worker is in the office. So call your supervisor. He doesn’t have to
walk 10 minutes to find you and, or, you know. So the see-what-you-see is
powerful and useful.
Alan:
I agree.
Charlie:
And also the ability to document the work in a hands-free way.
Alan:
That’s very important. So shifting gears slightly, you know, and
this is interesting, it’s in your book as well: A recent study commissioned by
Microsoft in the Harvard Business Review showed that 87 per cent of
correspondants that they interviewed are currently exploring, piloting, or
deploying mixed reality in their company workflows. What I want to know is, is
this congruent with what you’re seeing and hearing in the field?
Charlie:
At big companies that have innovation offices? It is true. And there
are a lot of integrators out there. The big consulting companies that are
evangelizing this in various companies, it’s starting to be something that they
do.
Alan:
So there’s, you know, the Deloittes and the Accentures of the world,
is that you’re talking about?
Charlie:
Correct. Correct. But they’re looking at, you know, because they
have Fortune 500 clients. And their clients aren’t super cost-sensitive. Again,
look at the savings that Boeing yielded working with Upscale. They liked them
so much they invested in the company, because it was obvious to them how many
applications there would be, and in fact they’re going factory-by-factory and
figuring it out. But, you know, the pace of change in a company like Boeing is
extraordinarily small…er, extraordinarily large! So that’s partly what the
Innovation Office does, they kind of figure out what’s out there and try and
evangelize it to colleagues. Sometimes on their own, sometimes with the help of
the CIO, or just depends on how the company is approaching it. But I think the
companies that Microsoft included in the survey were those large companies that
have innovation offices. But, you know, you’re talking about, you know, one per
cent of their workflow is being assisted by augmented reality. This kind of
augmented reality that we’re largely talking about with micro displays are no
longer, we’re no longer calling it ‘augmented reality’ – we’re calling that
‘assisted reality.’
Alan:
Got it.
Charlie:
The difference being it doesn’t go through the camera.
Alan:
Got it. OK, so that’s just an assisted reality, a heads-up display.
You know, interestingly, the person who wrote your forward for your new book
Convergence, Jay Samit, he was the kind of chairman or–
Charlie:
He’s still, right now, the non-executive vice chairman.
Alan:
–Non-executive vice chairman of Deloitte Digital. And in their–
Charlie:
Yeah, they deploy him when they need him.
Alan:
Interesting.
Charlie:
He’s like a weapon in their quiver, and he knows a lot of people. I
would expect Jay is actually helping them with sourcing clients.
Alan:
Absolutely. He’s a very, very well-known innovator.
Charlie:
He also is a fantastic speaker and one of the nicest guys you’ll
ever meet.
Alan:
Yeah. I’m hoping to have him on the podcast.
Charlie:
–who I used to work for years ago. He seems to be living a few
years into the future, so it’s always exciting to get his perspective of what’s
going on.
Alan:
So speaking of looking into the future, somebody else who’s kind of
looking to the future that you’ve been covering in your Forbes articles is
Magic Leap, and their CEO, Rony. Maybe you can discuss kind of what their big
vision is – because I know you’ve spent some quite a lot of time with them, and
you’ve written several articles about their vision for the future – and how do
you see them relating to kind of the business applications of XR technologies?
Charlie:
Well, a few things about Magic Leap, just to describe it for your
readers who don’t know: they use a multi-planer system to place objects in-depth.
That is its most unique aspect of the optical technology that they have, right?
They know how far away things are. That’s part of the beauty of their sensors.
They haven’t yet–
Alan:
Be honest; when you first put it on in and you hear a sound come
from behind you, and it sounds like it should be 10 feet away, and you look and
something is 10 feet away – it’s incredible.
Charlie:
Right. You place digital objects and they remain anchored where
you’ve placed them. It does games where they, you know, anchor on a wall and
then things come through the hole in the wall. It’s a very, very clever system
that took them over five years to pull together because there were so many
issues that had to be solved. Along the way they raised over 3 billion dollars,
which is quite remarkable since most of their plans were secret.
Alan:
Let’s just kind of… you know, 3 billion dollars they raised as a
startup.
Charlie:
Right, and they raised it. And part of the story there is who they
raised it from. Google put in 500 million dollars. Once they did that, it
activated a lot of large investors, including Disney and Warner Brothers, and
AT&T, and the sovereign wealth fund of Saudi Arabia. They also have the
largest investment companies in the world investing in them and they don’t
expect to make money for at least 10 years.
Alan:
Incredible. So they’re taking the long route.
Charlie:
They’re taking the Amazon route.
Alan:
Do you think that…
Charlie:
Amazon is basically still not profitable.
Alan:
Do you think 3 billion is enough?
Charlie:
Probably not. But they’re getting a lot of traction, so they’ll be
able to raise more money at a more realistic valuation because so much more
will be known by the market. But I think they could go several years on the
money they have now. Magic Leap a lot of great people running off doing
different things. They’re in education. They’re pushing hard for enterprise.
They recognize that those kinds of applications in the near-term, because
they’re more expensive, are going to be the ones used most frequently. In the
long term they see a consumer platform that has some powerful A.I. in it and
can function well in a fully-5G environment. I think the beauty of Magic Leap
is this whole idea of spatial computing, which you describe very well a couple
of minutes ago. They have, the past several years, been evangelizing this
vision. It hasn’t always been clear to people what the heck they were talking
about, but their vision is a 3D world in which digital objects and information
cohabit the world with us. So imagine walking down Fifth Avenue and everybody
walks past you as a cartoon animal and every building is, you know, like Toon Town,
just to use an example. Those are the kinds of apps that Magic Leap wants to
enable. But you need ubiquitous, zero-latency wearable computing to do it. So
that’s their vision. And they’re thinking, the final form factor they’re
thinking is very light and not terribly different from the glasses you’re
already wearing. But they’re also thinking, it’s not necessarily the device,
right? It’s the layers. The Magic-verse is where the value is. And you might
project that from a watch onto a projection surface. We don’t know what the
form factor is going to be. And they’re hedging, because they know they don’t
know. But they’re using the optical system and the infrastructure that they’re
building to build this 3D twin of the world in which content can be realistically
anchored and persist for the next person who sees it.
Alan:
It’s interesting you mention that, because, you know, recently Kevin
Kelly released an article on Wired called ‘Mirrorworld,’ talking about how we
would have an exact duplicate of the entire planet, and if you look at what
Google’s done over the last 20 years, or 10 years I guess, they’ve really taken
a depth map and full, comprehensive view of the entire external world that we
live in…
Charlie:
Google Maps maybe one of the
biggest digital assets in the world.
Alan:
Absolutely. And the one thing that people don’t realize is that
Google Maps has a map of the entire planet on the outside; they don’t have the
inside world. So I think there’s gonna be a huge opportunity for companies like
Magic Leap, and maybe Apple and Amazon and Google, to capture a point cloud or
a version – a mirror world – of every part of our lives, from the inside of our
house, to our offices, to buildings. What are your thoughts on that, and
companies like 6d.ai that are kind of pioneering this as well?
Charlie:
6d.ai is an interesting company. They have an augment in the book
and we’re one of the sponsors; the CEO and co-founder of the company Matt
Miesnieks, a former Samsung executive who also is part of a well-known AR
venture capital fund, Super Ventures. In his travels, he became exposed to this
technology and started thinking about this notion of using everybody’s depth
data and point clouds, and stitching them together on the back end to create a
digital twin of the world, that mirror world, that you could place augmented
reality on top of a within. So there are two things that we’re really talking
about, right? One is the mirror world, and the other is the AR which gets
anchored to it spatially.
Alan:
Interesting.
Charlie:
And Magic Leap calls that the Magic-verse, or Ian Barr calls it the
Wikipedia for the real world, and I call it a world painted with data. All the
data, which will be like anchoring pictures of food where you take them right?
So this is the end of the news feed, which goes by at 100 miles an hour.
Instead, it will be the age of contextual content. So you want to take a
picture of food and you’ll add a little review a friend who’s in the same
restaurant a year later.
Alan:
Incredible. Absolutely.
Charlie:
So it’s pretty exciting. But we’re… again, a mirror world is
talking about the twin, right? It’s talking about infrastructure. The AR cloud,
or the magic-verse, or the layers that Magic Leap refers to as the content of
the services that sit on top of that. Again, let’s go back to the combustion
engine enabling, you know, the car and the airplane. So the combustion engine
is the thing – the person who made the combustion engine didn’t think, this is
going to make air travel possible.
Alan:
No, absolutely.
Charlie:
Right? But but it was extremely disruptive new technologies that,
you know, like the camera, like GPS, enabled fantastic new services.
Alan:
One of the things that I think is really intriguing is this idea
that, once you have the magic-verse, or this, you know, the capture of the
mirror world, who actually owns that three dimensional space? And you know,
it’s interesting, we’re doing a virtual and augmented reality association
meet-up on the legal aspects, or the looking through the legal lens of this
technology, and you know something like the Burger King example where Burger
King is using their competitors’ advertisings to advertise their products. And
I mean, what are the legal implications around using three dimensional space,
and will it be more… give more agency to the end user, so that they can
decide what they see versus what advertisers want them to see?
Charlie:
Well anything that you can see in public is public domain, right?
You can’t say, the front of my building is so unique that you may not
photograph it, right?
Alan:
That’s true, but I can’t go over it…
Charlie:
When you get inside a building, now I think you’re in a legal gray
zone, right? Is the inside of a mall a public place? Or does the mall owner own
that data?
Alan:
Interesting. That’s a really good point.
Charlie:
Of course, in your private place, there’ll be all sorts of security
protocols that you can opt in and out of. But anybody who walks through your
house can make a map of some sort, so that’s a very different matter, right? I
don’t think people are going to be comfortable with other people knowing where
everything is in their house. But any kind of a magic-verse is going to be
governed by a filter. That’s where the AI comes in. In order to see the burning
McDonald’s sign, you have to activate that filter, say, ‘yes you can show me
that.’
Alan:
Interesting.
Charlie:
But it has to be a permission-based system. Let’s say, I mean,
there’s going to be a ton of graffiti. There will be graffiti apps and, you
know, graffiti layers, if you will. Maybe people will turn them on or off for
entertainment. They’ll be in a park. They’ll see it’s there. Right? There has
to be a detection system that tells you when content is proximate to you and
ask if you want to see it. But it would be so many questions, it would be so
intrusive. The machine has to know; the computer has to know. So that’s
artificial intelligence, where it’s integrating what it’s learning with what it
knows.
Alan:
So, Charlie, you’ve mentioned a couple of technologies. You’ve
mentioned 5G, AR, and now artificial intelligence. How do they all work
together? Because I think what people don’t realize is that, as we enter this
kind of exponential age or phase of technology where they all kind of coalesce
together, you’re gonna have all of these different technologies working
together, and you can’t really look at augmented reality without 5G, without
IoT sensors (or Internet of Things sensors)…
Charlie:
Right, and what you’re talking about is what the technology people
call a stack; what could be in the stack? Computer vision, geo-location…
sensors, sensors, sensors. It’s all about sensors. And what I would say is, a
sensor does not need to be head-mounted, right? A sensor could be on your
lapel, and you could be wearing some Bose spacial sound glasses that have your
prescription in them, and it could be telling you what you need to know rather
than disrupting your field of view. And they’re quite extraordinary, because
you’re the only one who hears them, even though you don’t have your earplugs
in. And they are spatial, as we were describing the Hololens, or the Magic Leap
device; the creator’s edition, which is their developer edition. So the current
version of the Magic Leap is $2,300 and it’s focused on programmers and pro-sumers.
Alan:
Interesting. Shifting focuses a little bit to something that I saw
on the Hololens 2 launch that, I think, is personally going to be a really
killer app for businesses, and that’s telecommunication or remote
communication; something like Spatial – the company Spatial – that kind of came out of stealth recently and
introduced this amazing ability to have virtual people in your space. So I
could have a meeting with you in virtual space and you and I could see each
other, interact with each other, look into each other’s eyes because it now has
eye tracking. So maybe talk to the business applications around Spatial and
what they’re doing.
Charlie:
You’re talking about a new startup called Spatial, which is
well-funded and staffed by big technology company veterans – although they’re
still quite young – and it creates an avatar for the user, no matter what
device they’re using. And the avatar is generally based on your social media
presence. So, they built my avatar, when I went to visit them, out of my
Twitter picture. And then they use a real-time AI to kind of predict, based on
that picture, what the rest of you looks like.
Alan:
Amazing.
Charlie:
So, a very uncanny resemblance – although it’s not photographic,
it’s graphic. It’s cartoon. Just like Facebook does something similar in
Facebook spaces for virtual reality.
Alan:
But is it creepy? Because you mentioned uncanny, and for those of
you who don’t know, there’s a term called…
Charlie:
No, it’s not like, you know, those realistic Hanson robots, whatever
that–
Alan:
Yeah, what’s her name?
Charlie:
That one is a little creepy, but it’s a really dumb person. They’re
trying to make Robbie the Robot, right? Or that alien helper, or the cyborg
helper in the Aliens movies. That’s never going to happen. That’s a
technology… that’s alien technology. We’re so far away from that, you know,
my grandchildren won’t see it.
Alan:
So do you think Spatial, the solution that they’re providing, where
you can have collaboratively–
Charlie:
It’s very compelling. I mean, you do get a genuine sense of
presence, no matter what device you’re using. Obviously it’s best in a more
immersive headset like the Magic Leap or Hololens But even people who are using
their pads, know, feel like they’re there. And of course, you then have the
point-of-view of your avatar; you’re not seeing yourself. You can go into 3-D
mode, but typically you’re yourself. You’re present. And it uses the tone of
your voice to create gestures, and is generally pretty good as a remote
collaboration platform. You could bring in YouTube, you could move the screens
around, you could bring in 3D objects, you can walk around them, you can walk
closer to someone. So it really does have a very, very strong sense of
presence, especially for an augmented reality device where you’re still
anchored in your home or office.
Alan:
Do you think it’s going to replace travel for business? Or not
replace, but maybe decrease–?
Charlie:
It will augment it.
Charlie:
What we’re seeing, right, is that it’s slowly creeping in to apps we
use every day, such as social media filters, and people don’t say that it’s
augmented reality. They say it’s face filters. They don’t say it’s facial
recognition filters. Those are words that consumers don’t need to know. I mean,
every extra word is explaining, and explaining is bad.
Alan:
So, I know you, I know your opinion on this, but since you just hit
on it, what do you think about this term ‘XR,’ as a catch-all phrase for
virtual, augmented, and mixed reality?
Charlie:
I think it’s useful, to be honest. I have my…objections to it,
which is it just conflates all these technologies, which is super unhelpful,
because there are many, many modes of AR. On the other hand, if you’re writing
about it, or writing a book about it, it’s convenient to say XR instead of
AR/VR. And is AR/VR even that accurate? I have been railing against it, because
I really think there’s so much misunderstanding out there. Another XR, or
another ‘R,’ I should say, and now we’ve included diminished reality, and as I
said earlier, assisted reality. So I don’t like all these R’s, I don’t like
‘XR,’ but I’m done fighting about it? Ori Inbar, in his keynote at AWE 2018,
said, ‘go XR or go home.’
Alan:
Yeah, I read that in your book, actually.
Charlie:
Once he said that, I mean, I kind of lost. I know other people feel
as I do, so I’m not alone. I think I’m in a distinct minority, but it’s over.
There seems to be a rear guard action I may have ignited. But that’s just going
to lead to even more confusion.
Alan:
Exactly. So–
Charlie:
I don’t know about this, other than this is all entre nous. This is
all inside baseball. Most people don’t even need to know that we’re arguing
about this because, all they care about is what it does–
Alan:
What it does for them, and I think that leads me into, kind of, the
last part of this interview, which I really want to thank you so much for
taking the time to enlighten our listeners here about the different business
applications and what’s coming. The question that I have is, then, what do you
see for the future of virtual, augmented, and mixed reality, or XR, as it
pertains to business? Where do you see as, kind of, the ultimate future of this
technology?
Charlie:
We’d have to look at the development of the personal computer, which
took 20 years. For a long time, people didn’t think they applied to them at
work, and eventually you had to have them. And then people started to learn
about the Internet and e-commerce, and they said, oh it’s it’s got an
application that is for me personally. And don’t forget, in the mid 90s people
started to feel that if they didn’t have email, and if they didn’t have a
website if you were an entrepreneur, that you didn’t exist, or you feared you
wouldn’t exist. So that was creating a lot of FOMO. But I think part of it was
driven by email, and part of it was driven by e-commerce, and both of those
enabled by consumer access to the Internet, through dial-up at first and then
through broadband, which really came around very quickly; you know, by the end
of the 90s, everybody had a cable modem.
Alan:
It’s interesting, I was just mapping kind of the–
Charlie:
By the way, the end of the 90s was 20 years .
Alan:
Yeah! Oh my goodness.
Charlie:
So this could easily take 20 years. I know people are saying five.
And you know, we’ll see. But it’s going to take some convincing to get a lot of
people wearing a head-mounted display. I don’t know what the benefit is yet,
that is powerful enough to get people to do that.
Alan:
I agree. I haven’t seen it yet either.
Charlie:
But in business, of course, it’s a tool. You just take it off when
you’re done. It’s not… I will also say this, both about consumer AR and AR
for industry: AR is not a thing to like or dislike, or do or not do. AR as a
tool. AR enables tools. AR augments tools, like the personal computer that has
the schematics on it; well, AR would would make that a little better. You look
at consumer apps like Google Maps – we talked about the combination of computer
vision and geo-location, which enables local directions, walking directions, to
be much more effective than they are right now, because it can tell you where
to go but… it could tell you where you are, but not how to get there. So that’s
going to change. Will people walk around saying, oh my god, computer vision and
augmented reality have made Google Maps much better? Or will I just say, this
is better Google Maps? Yeah that is cool.
Alan:
Exactly.
Charlie:
Because that’s how much thought they’re going to give it. So, you
know, that’s an example of, you know, augmented reality taking things we’re
doing everyday and making it better, and that’s how it integrates itself into
your life. And so I think most of the growth of augmented reality, both in
enterprise and on the consumer side – which will take much longer – you know,
is going to happen without people calling it anything. It’s just going to be
better computers, and better tools, and better apps. So augmented reality is a
quality, or a tool that apps use to be better. So I’m not sure that there’s
going to be a big parade for augmented reality.
Alan:
Definitely not.
Charlie:
It’ll be a big parade for better Google Apps.
Alan:
It’s interesting…
Charlie:
Again, a lot of this conversation is baseball. If you’re not in the
industry, you don’t care.
Alan:
It’s interesting because, you know, Sunder Pichai from Google said,
you know, the idea is that the very idea of the device is going to fade away.
So it’ll be less–
Charlie:
Invisible computing, that’s true! The device becomes your glasses;
you’re not even aware of them being special. That’s certainly something that
our grandchildren will see.
Alan:
So, speaking of our grandchildren and looking quite that far out,
one last thing that I think is an interesting segue into the future is
brain-computer interfaces, and it’s not something that people are talking
about, but I think, you know…
Charlie:
Elon Musk funded the company to the tune of 40 million dollars
that’s doing that. We do have implants, right? People who have epilepsy and
Parkinson’s are getting implants. People have been getting implants in our
hearts for 30 years. So we already have implants now and we have some in the
brain. There’s still not a lot known about the brain; it’s usually to treat a
disease. But, ultimately, you know, there’s a company working on contact
lenses. My understanding is right now they can put basically a tweet in contact
lenses. So they may be a ways away…
Alan:
Kind of like the North glasses.
Charlie:
Well, North glasses are just a reflective system that tells you what
the weather is and…
Alan:
In a tweet.
Charlie:
…But, my problem with North – and I love the form factor – is
just, are going to want to be interrupted like that?
Alan:
I have a pair and–
Charlie:
I don’t see the benefit.
Alan:
–they are a little distracting. I was walking down the street and I
got a message and thought, how cool is this? I was reading my message, and I
almost walked into somebody because, even though it’s monocular – meaning one
eye – both eyes kind of look towards it and I guess over time your brain, kind
of, you know, can can do both things? But I had just got them, and I’m walking
down the street, and I’m looking at this message and I literally almost walked
into some poor woman on the street. So I can see this as being something maybe
dangerous in driving, especially the first little bit, until you get used to
that.
Charlie:
Yeah, well, I agree. The final form factor is that… that’s another
argument for sound, by the way. I happen to think Apple’s stealth way into
augmented reality is not going to be optical going to be based on airpods.
Alan:
Yeah it’ll be interesting to see what Apple comes up with. They’ve
hired a lot, a lot of people. They acquired Matteo and Nirvana and a number of
other companies. I believe it was Eyefluence or one of the eye-tracking
companies.
Charlie:
Well I mean, my guess is what they’re going to do is going to be a
spatial AR/VR inter-operable device. But the technology is really not quite
there yet, in terms of miniaturization and form factor. And, you know,
Microsoft is really leading the way with sensors, and I don’t think we can
overemphasize the importance of sensors versus optics.
Alan:
Really excited to actually try the new Azure Connect, which is a
cloud-based connect system. So yeah, you’re–
Charlie:
Of course, it’s a way for Microsoft to charge you extra money every
month.
Alan:
They love that.
Charlie:
They love that Windows license fee level for enterprise.
Alan:
Well, Charlie, thank you so, so much, and everybody, thank you for
listening. This podcast was another amazing example of how XR technologies are
revolutionizing business across every industry. You can learn more about
Charlie and his new book by buying the book Convergence today, by going to
www.ConvergenceAR.com. And Charlie, thank you again so much for being on the
show. It’s been an amazing honor.
Charlie:
Thank you.
Alan:
Thank you so much.
Our TOPPODCAST Picks 
Stay Connected
Golf on your tablet!
SenseGloves in motion
A rad bike helmet designed using ThreeKit's 3D configurator.
