Taking Core Location Indoors

Session 708 WWDC 2014

iOS 8 makes it possible for an iOS app to determine its precise indoor position in supported venues. Learn best practices on how your app can take advantage of indoor positioning. Discover how indoor positioning and iBeacon complement each other, and understand the best use cases for both technologies. As a venue, find out how you can get involved and signup to enable indoor positioning.

Good afternoon everyone.

I'm Nav Patel.

I'm a software engineer on the Core Location team, and today I want to tell you about the next great leap we're taking in location services.

We're taking Core Location indoors.

So we'll start off by covering what indoor positioning is, how does it work, how does it fit in with the other Core Location technologies?

We'll also cover how you use it, what APIs you'll use, and we also will talk about indoor positioning and iBeacon technology and how they work together.

Finally, we'll cover some next steps on how venues can get signed up and how you guys can start looking at the documentation and the APIs.

So with today's technology, Core Location on iOS is the best location service you can get on a mobile device.

We have the best accuracy.

We have the best response time.

We have the best Time To First Fix, and this comes from the tight hardware and software integration we have.

The hardware team and the software team work with each other to make sure we get the most out of the hardware we have on our devices.

So our Core Location technologies today start off with cellular.

Now, cellular is great for iPhones and iPads with data connectivity because it's essentially free.

You're already connected to these cell towers, and it's good enough to give you an approximation of which area of a city you're in.

Now, because you're already associated with a tower, you're not spending any more energy or power to get this fix and it's usually available.

So, for cases like: you're landing in an airplane and you turn on your phone, this is the first fix you'll get, and it's good enough to set your time zone.

Next we have GPS technology.

Now, GPS is based on satellite, and it was the first big revolution in location services.

It was accurate enough to do outdoor navigation and responsive enough to do outdoor navigation.

Now, we didn't stop there.

GPS was introduced a little while ago, but we've been constantly improving the accuracy and the Time To First Fix of GPS as well.

What's great about GPS is we launched satellites into space.

We use both the American Navstar and the Russian GLONASS satellite constellation, and this gives us really great global coverage.

Now, the issue with GPS is urban canyons.

When you're in a city with tall buildings, they can obscure your view to the satellite, and so you might not get a signal, and when you do get a signal, that signal can suffer from multipath.

So the signal will bounce off in between buildings, and you might not get a very good fix at first.

So to solve this problem, we use Wi-Fi.

Now, Wi-Fi is good enough to give you position within a city block.

Often it will put you in front of the right building, and this is really for those cases like New York and San Francisco where you have tons of Wi-Fi already out there where you can use this to give you position in urban canyon environments.

What's great is that this also gives you location services on Wi-Fi only devices.

So if you have an iPod or an iPad Wi-Fi, you can get location using this.

So their entire location experience is based on Wi-Fi.

So the existing technologies work great.

They'll let you navigate outdoors.

They also work well globally.

You don't need to put up any infrastructure.

You don't need to launch satellites as an app developer, and the Wi-Fi AP's already out there, and so it actually works very well.

But there's an issue, and that's when you move indoors.

Today we do give you some position indoors, and you might have had that experience where you pull out your phone in a mall at a Starbucks, and it will tell you, "Hey, it actually found my exact spot.

How are they doing that?"

And that is using a combination of GPS that may have come in through the windows or skylights or using Wi-Fi, and that can work indoors, but it's really not good enough to do any sort of navigation indoors.

Also, Core Location gives you an altitude, and when you're thinking about the context of being in a building, you really want floor information.

Altitude is hard to use.

So we heard you guys really want to build applications for indoor venues, and for that we're announcing Indoor Positioning.

So here's a look at what it looks like.

This is me a couple of days ago in the California Academy of Sciences in Golden Gate Park.

Now, you can see me walking around the planetarium here.

The playback has been sped up by 10x so you can actually see me move through the space.

Here I'll walk past the giant swinging pendulum towards the swamp, the albino alligator, and as you can see, this is pretty accurate, and it works amazingly well indoors.

It's also very responsive.

You know, hypothetically, you could build a game of Pac-Man in your office, and a bunch of engineers could run around and piss off HR...

[ Laughter ]


But it's quite good.

So indoor positioning works using RF parametric data that's in your venue already.

We also use motion sensors.

We use the M7 chip, and these sensors and these RF signals that are in the air is what we use to give you your indoor position.

So how does this all work?

Well, we use your existing technologies: your cell, GPS and Wi-Fi, and these are running, and they'll put you close to or inside a venue like we just talked about.

When you approach a venue that's indoor position enabled, we'll start scanning your Wi-Fi and using more of your motion sensors.

We use your motion sensors at this point because it gives us information for our estimation algorithms of where you're moving indoors, but it also gives us key information about how quickly you're moving.

If you're moving at 40 miles per hour in a car, you're probably not indoors.

Hopefully, you're not indoors.

So when we get a good fix and we're confident, hey, you are indoors, we'll also turn down your GPS chip.

So this is the hardware and software integration I was talking about earlier.

We do this to save power.

So that indoor positioning, even though we're using RF more, we're not consuming more power.

Now, all of this is really cool and it looks really great.

But why indoors?

Why do we care?

Well, there are a lot of amazing applications for large indoor venues today.

We have things like directories.

So here's the Heathrow application, and it's great.

It shows you all the shops and restaurants that are throughout the entire airport.

We also have venue maps.

Now, venue maps will tell you: "Oh, the Nordstrom's is on this side, the Macy's is on the other side," and it gives you useful information, and we have some level of interactivity.

So this is the MOMA application, and a user can browse the catalog and pick a particular thing that they're looking at and listen to commentary about it.

But there's only limited context over here.

What we want to do is more.

With indoors you can enable things like way-finding.

So here you can imagine you're going through the airport, now we can route you to the right gate.

You can also imagine the impact this can have on the visually impaired community.

They get great GPS services outdoors today and that helps them get around.

Venues can also build this experience into their applications so that you can navigate indoors inside a mall.

And not just for the visually impaired, but think about when you're visiting a new large indoor venue for the first time.

If you're at a hospital, you don't want to follow a 10-step direction you got at the reception.

You don't want to parse floor plans.

You just want to find your loved one's room, and this is where indoor positioning and navigating with way-finding steps in.

Now, with location we can also do other really interesting things.

We can help you find each other.

So you can think about applications like Find My Friends.

Here you can imagine a kid gets dropped off at the mall and pulls out his phone and says, hey, where are all my friends hanging out?

Okay, they're at that the food court on the ground floor.

I can head there.

That's pretty cool.

You can also find you.

So the venue in this case could be aware of where you're using your application.

This is one of my favorite use cases.

Imagine you're in a Vegas casino.

You're in your hotel room.

You pull out your phone, and as you're leaving, you order a drink.

You go downstairs, settle down on a poker table, and the drink arrives to exactly where you are.

That's a pretty awesome experience.

Now, these use cases are pretty location centric, but I want you to think about how you can use indoor positioning to be the last piece of the puzzle in a problem.

So here you can imagine something like an airport application.

Now, I have my flight leaving soon, but I need to grab a quick bite.

So I'm going to walk over to a restaurant and sit down.

My application knows which gate I'm leaving from, knows what time my flight is boarding, and it knows how far away I am from the gate now, and so we can start incorporating this information to, say, give you a quick buzz and say, "Hey, you better start heading back to your gate if you want to board this flight."

So these are some pretty exciting use cases you can build with indoor positioning.

So how do you start using it?

Well, you need to have your Wi-Fi on.

We depend on RF parametric data in these venues.

We also want your users to be actively engaged in the application.

We use aggressive Wi-Fi scanning and motion sensors, so we want to make sure we're not just running this if the user isn't actually using that indoor positioning.

Now, in terms of API, what API do you need to use for indoor positioning?

Well, you can use the exact same Core Location API you've already been using.

Now, this is really cool because if your application today is requesting best accuracy, you'll start seeing indoor fixes when your user wanders into an indoor-enabled environment.

You won't have to make any code change for that.

Now, one problem we talked about earlier was altitude versus floors and if you want to build an indoor app, you really want to know the floor number.

So in iOS 8 we're adding floor number to the Core Location API.

It looks something like this: in the CLLocation object you'll see a "floor" property.

This property will usually be nil, but when you walk into a venue and we're serving you indoor fixes, we'll populate this number with the level of floor that you're on, zero being the ground floor.

Positive numbers up.

Negative numbers down.

We'll still show you 13.

[ Laughter ]

So we have your floor number, but Core Location is giving you a latitude and longitude, and you're usually displaying your floor plan on a flat image.

So how do you deal with this?

How do you go from your latitude and longitude onto your X, Y on your floor?

To help you figure this out I'm going to call my colleague Vitali.

Thanks, Nav.

Hi, I'm Vitali Lovich.

I'm from the Core Location team, and I'm going to show you how you can integrate your existing Core Location API with an existing app that you have that shows an indoor floor plan.

So a geographic coordinate system, as you may be familiar with, is a latitude and a longitude.

It's very common and convenient.

Two numbers represent any place in the world; however, there are some challenges that you may have when you work with them.

For instance, finding the distance between two latitudes and longitudes or a direction between them is very complicated math.

But when you show a floor plan image, you want to show a floor plan image to the user, and it's very easy to display.

It maps to the display coordinate frame system very well, and its in pixels, and it's X is positive to the right, Y is positive down.

And so we want to somehow bridge the two by converting and so there are three coordinate systems that you want to keep in mind when doing these conversions: There's the geographic coordinate system, which is spherical; There's the floor plan coordinate system, which is in pixels and it's planar; And then there's the display coordinate frame system, where you show what part of the floor plan you're interested in, and for this talk we're going to cover the conversion from geographic to floor plan coordinate frames.

So you want to make sure that you account for spherical distortion, because any time you go from a sphere and you try to map it onto a rectangular plane you'll have all sorts of distortions where straight lines aren't straight lines anymore .

So to help us accomplish this task, we're going to make use of some amazing MapKit helper functions, and so the first one I want to call out is MKMapPointForCoordinate.

Now, what this does is it takes a geographical latitude and longitude and converts it to a MapKit point, which is just an X, Y coordinate on a Mercator projection, and so positive so, X and Y represent east and south and it's in points.

MKMetersBetweenMapPoints is a very accurate and very precise way of finding the distance between two MK map points in meters, and it accounts for your spherical distortions, and then we'll take a look at MKMetersPerMapPointAtLatitude a little bit later.

So now that we have this primer, what do we need to get started?

Well, we want to find two points on our map, and we'll call them anchor points, and an anchor point is simply a latitude and longitude mapped to a floor plan pixel so we know how the floor plan exists in the real world.

So what you want to do is you want to find two identifying features in your floor plan that are very easy to find on an aerial map, like you might find in Apple Maps, and then you find the corresponding latitude and longitude for that floor plan pixel.

So now that we have these two anchor points, we need to figure out how the floor plan in pixels maps to the real world.

So we need to figure out a scale.

So how many pixels represent a single meter?

So to accomplish this we're going to use the MapKit helper functions I pointed out earlier and we're going to convert our anchor points' geographical coordinates into Mercator projection, and then we use MapKit again to figure out the distance in meters.

So now we have distance in meters.

And then the distance in a floor plan is very easy to calculate.

It's just the Pythagorean Theorem, and we're just going to call hypot to do it for us, and now we have a distance in pixels.

So now we have pixels.

We have meters.

We know how many pixels represent 1 meter.

Orientation is a little bit more complicated, and so your floor plan, when you show it to the user, is usually aligned to some logical orientation.

You might show it to them so that X is positive on the screen, and Y is down.

That doesn't necessarily correspond to your north and east coordinates, and so we need to figure out how we would rotate to that floor plan to make it show up correctly.

So what we're going to do is we're going to align the east and south to the X and Y, and we can see now that our two anchor points in yellow are relative to the east and south, but they're somewhat different.

They're rotated relative to the green, which is relative to the floor plan, and so what we're going to do is try to calculate the angle between the green line and the yellow line using some existing angles that we know about, and so the way to do that is you simply subtract the two angles.

So now we know how much to rotate the floor plan by to get into your logical orientation that you want.

So now we have these two precomputed values.

What do we do with them?

Well, we're going to get a user position in geographical coordinates, latitudes and longitudes.

As you may have guessed, we convert it into the Mercator projection, and so that gives us a user point, and now what we're going to do is subtract the user points from one of our anchor points, and that gives us how many MapKit points we are east and south from the floor plan.

But we want to account for the fact that MapKit points are not the same thing as meters.

So we want to account for the fact that MapKit is a projection of the globe, and so it has distortion, and so the distortion changes depending on which latitude you're at.

And so, by multiplying by this meter scale, we know how many meters east and south the user is on your floor plan, and so now we just apply the scale we precomputed previously.

That gives us how many pixels east and south, and then once we apply the rotation, we get pixels in X, Y of your floor plan.

And so, if you were to apply this backwards, you would have a mechanism like, literally, the exact same steps backwards you would have a mechanism converting your floor plan pixels into geographic coordinates.

So I hope I've demonstrated a way for you to incorporate the existing geographic coordinates that you get from Core Location into your indoor floor plan.

Thank you.

So with that information you can take your Core Location fix, map it to your floor plan and start displaying it to your user.

Now that's really cool; you can build an application with that.

Where can you start building these applications?

Well, in terms of availability, I'd like to tell you about three venues.

There's the California Academy of Sciences that I demoed to you earlier, in Golden Gate Park in San Francisco, which is a big natural history museum.

There's the Westfield SF Mall a couple of blocks from here.

Has many floors, and it's a quintessential mall in a big city.

You may have also flown in from the San Jose International Airport, and so we're going to be enabling that soon, and all of these will be enabled in upcoming seeds of the developer build along with sample code that walk you through the process that Vitali just described.

So keep an eye out on developer.apple.com for when that sample code comes out, and you guys can run out and start, and we'll enable these venues, and you guys can start using indoor positioning.

Now, these are not the only set of venues.

There are very many, many more that are going to be coming up in the upcoming months, and so we'll be announcing them as that happens.

So you build this application.

How do we get people to start using this application?

How do we get people to install this application?

Well, there's a traditional way, which is just advertise at your venue.

Put up a giant sign with an ugly QR code, but we don't like that very much.

So last year we introduced the Near Me feature in the App Store, and that allowed you to launch the App Store, look at that particular view, and it would tell you the relevant and popular applications near you.

But that still relies on the user to launch the application and look for nearby venues, which they might have never thought about.

So as a continuity feature, we're bubbling up that popular application onto your Lock screen.

So now when you open your phone and wake your device, you'll see this in the bottom left corner.

I'll show a little App Store icon so that you can launch the App Store directly to the page that will tell you the popular applications near you, and once that application is installed, we'll actually show you the icon for that particular application.

And when the user swipes this up, we'll launch your application directly from the Lock screen.

Now that's really cool.

Now people can show up at your venue, pull out their phone and launch the appropriate application without ever having to go through multiple steps.

But let's say I have a specific venue with a really great app and I want to notify people when they walk into my venue.

Well, here we're thinking about when we're close to, let's say, the entrance of a venue, and we're thinking about notifying your application, and that's when we want to think about iBeacon technology.

So, indoor positioning and iBeacon technology really complement each other.

They complete the indoor experience that we expect.

Now, iBeacon technology, we covered this last year in the "What's New in Core Location" talk.

Devices with iBeacon technology, these little beacons, can run on little coin batteries and last a few months.

You can also take any iOS device with Bluetooth and advertise beacons with them.

So if you have a point of sale device, you can turn that into a beacon as well, and we can use these beacons to advertise that, "Hey, I have a region over here," and your application can register to get notified when the device sees that region.

So when we're thinking about indoor positioning and iBeacon technology, indoor positioning is really about your position in the geographic space, where you're on the floor plan, and it's about navigation and getting you from one place to another.

iBeacon technology is about proximity.

What is near me?

It's also about notification.

Because it's based on Bluetooth for energy, you can register for beacon regions, and we'll notify your application even if it's in the background when the device sees this.

So let's ground all of this in an example.

I want you think about an art gallery, and in this art gallery we have an amazing application that really creates an immersive experience for our visitors, and we love this application so much, we want to notify our users when they enter into this art gallery.

Well, you could use a CLCircularRegion.

Now, this is a region with a point in the middle in lat/long and a radius, and a CLCircularRegion is a great API that lets you notify the same way beacons would in the background.

But CLCircularRegions rely on Wi-Fi, and so we really want to make the radius about as small as 100 meters, not much less than that.

Now, this would work great if you have a colossal building that's perfectly round, but that might not be the case.

So for our floor plan, if we have a round region, this might be an issue.

Now, if someone is walking into your parking lot, they'll get notified.

If you're on the second floor of a building and you're just walking on the first floor, they'll get notified.

Here's where we want to use the power of iBeacon technology.

What we can do is install beacons at the entrances of your floor plan, and the great thing about these beacons is we can really tune them and drop them exactly where we want and with what power we want them to emit their signal so that we can control, "I want to notify at this point."

And now the device can say...

when you have your application, the application can register with Core Location and say, "when you see this beacon, notify me."

And when the device enters your region, it will trigger that API.

So we use the startMonitoringForRegion API with a beaconRegion where you specify which unique identifier you are looking for on that iBeacon technology device.

And when you see that region, we'll trigger the didEnterRegion delegate, and this is when you can start notifying your user when appropriate.

At this point the user sees the notification, and they think, oh great, I do have this application.

Let me launch it.

And here you can create your indoor positioning experience.

So you can display your user's position on the map, and as the user walks around, the little dot will follow them, which is pretty cool, and while they're browsing your catalog, they can pick something that they really want to see, and you can navigate them to that particular exhibit.

This can provide step-by-step direction exactly where you want to go.

So once we're at this place, we want to also show the user relevant commentary about the exhibits that are around them.

And again, here we're talking about proximity, and when we're thinking about proximity, we want to think about iBeacons.

We need fine-grain proximity.

So what we can do now is deploy beacons at all the various exhibits where we have commentary on our floor plan, and when a user is using your application and enters the region, we can start ranging from that beacon, and we can say, "Oh, am I just seeing it or am I very close to it?

Am I right up next to it?"

And when you are, you can start doing things like providing commentary or showing the appropriate item in the catalog.

Now, this uses the Ranging API.

So you can call startRangingBeaconsInRegion, and then your delegate will get called with all the beacons that it sees.

Here you can use the proximity value to filter for how close you want to be from that beacon.

Now, these beacons will also advertise a major and a minor code.

Now, this major code can encode something like, "Is this an entry beacon or is this an exhibit beacon," and the minor code can tell you exactly which exhibit this exhibit beacon is advertising, and then you can do a local lookup and show relevant information for that.

So we can build these amazing features with beacons and indoor positioning, and this really completes that experience of what your application needs to do for an indoor venue.

Now, when you're thinking about these amazing features, you have to keep the user in mind.

You have a lot of power with these features, and with great power...

comes strict security and privacy guidelines.

[ Laughter ]

So we want you and your users to be on the same page here.

You, as the app developer, need to anticipate how am I going to be, how is my user going to be using their location?

Your user also needs to be able to anticipate, "How is this application using my location?"

So forming this trust between the user and the app developers is what allows users to trust you, as the app developer, with their location.

So there's a couple of things you can do.

You can request location only as you need it.

So if you have a particular view in your application that's not using location and you're just logging in, don't start requesting location immediately.

Wait till you actually launch a view that has a map on it or you need to use some location services.

Also use When In Use authorization.

Now, this is something new that was introduced in iOS 8.

If you missed the "What's New In Core Location" talk that just happened prior to this one, be sure to check out that video.

There's some amazing new features, and When In Use authorization really makes it clear to the user that I can tell when this application is using my location so I can trust it.

Also have a clear purpose string.

Tell your users what you're doing with your location.

This is a really great opportunity to communicate with them so that they're on the same page as you.

So let's talk about some next steps.

We have the Maps Connect website, and here, if you're a venue owner or if you're building an application for a particular venue, you can head on over to this website and tell us all about it.

Tell us what you plan on doing at this venue.

Tell us what kind of applications you plan on building, and we can get you started on the pipeline so that we can enable indoor positioning in your venue.

There's also the Maps page, which is a great resource for maps and Core Location APIs, and there's also a new iBeacon page, where it will tell you about iBeacon technology, again, all the API and documentation that you'll need, and also licensing information if you're interested in manufacturing iBeacon technology.

So as a summary, we talked about indoor positioning today.

We talked about some amazing use cases that we can enable with this.

We also talked about Core Location APIs and how there's very little you need to add to your application to make them indoor enabled.

We're just using indoor position to create a new amazing experience for your users, and with indoor positioning and iBeacon technology, these two technologies really work perfectly well with each other, and they complete the indoor experience that you expect from your application so that you start breaking out of the barriers of, "Hey, this is what's happening in my phone, and this is what's happening around me."

Now the environment and the device can interact with each other.

For more information, be sure to check out the developer forums and the documentation.

There was a related session, as I mentioned, of "What's New In Core Location".

So please make sure to check that out.

Also there's a "User Privacy in iOS and OS X" talk on Thursday, and here we'll discuss how you should be using location, how you can communicate appropriately with your users.

Thank you very much.

[ Applause ]

Apple, Inc. AAPL
1 Infinite Loop Cupertino CA 95014 US