Advanced Testing and Continuous Integration

Session 409 WWDC 2016

Take an advanced class in Testing and Xcode Server. You'll gain an in-depth knowledge on the lifecycle of a test, how they're are hosted, and how using modern observation and expectation-based APIs can help you make bulletproof tests for your app. Then, learn about changes in Xcode Server that make continuous integration easier than ever, including configuring your own user for testing, enhancements to issue tracking, email notifications and support for crash logs.

[ Music ]

[ Applause ]

Good afternoon, everyone.

My name is Zoltan.

Later my colleague, Eric, will join me on stage.

We're both engineers with the Xcode team.

Software development these days is a lot like conducting an orchestra.

You and I maintain suites of tests, and when one of them is not performing well, we need to quickly understand the issue and correct it.

Okay. Maybe you're a testing maestro with hundreds of thousands of tests.

Well, Xcode has tools and techniques to conduct even the largest test suite, and today we're going to show you how.

First, we're going to introduce some concepts that haven't been addressed in previous sessions, and then, we'll introduce some new features in Xcode, in Xcode Server, in particular, the configurable integration user.

And then, we'll conclude with some new features in xcodebuild.

Let's get started.

So, as a brief recap of testing, you can think of testing as these four characters.

There's XC test.

That's the framework for your tests for both Objective-C and for Swift.

And your tests are compiled to bundles before they're run.

Xcode, that's the IDE for authoring your tests, and you can also run individual test while you're developing.

Xcode is also where you review reports from both local test runs and from Xcode Server.

And talking about Xcode Server, that's the continuous integration solution for your tests.

You set up bots to periodically run your tests, and Xcode Server will generate reports for you.

And if something goes wrong, it'll notify you.

There's a great way to keep track of your project over time.

So, Xcode Server is built on top of xcodebuild, and you can use xcodebuild on the command line, too.

You can run tests and see results in the console.

Xcodebuild is the building block for custom, continuous integration systems, as we'll see later.

So, you can learn more about these four characters in our previous sessions.

But, today, we want to look at some concepts that we haven't covered before.

We want to take you behind the scenes and show you exactly how your tests are running.

And to do that, it helps to think of a timeline.

After your tests are compiled, they must be hosted.

That gets the test started in the first place.

And then, once the tests are running, you can see detailed progress in your tests using a technique known as observation.

So, we're going to look at these two concepts, hosting and observation.

Let's look at hosting first.

So, the hosting story, it's different depending on whether you have unit tests or UI tests.

So, for unit tests, your test bundle is loaded directly into your application.

And, in this case, we call your application the host application.

But for your UI tests, your test bundle is going to get loaded into a UI test runner which is separate from your application.

And in this case, we refer to your application as the target application.

Now, this has some implications.

For your unit tests, you will have direct access to your applications data structures and API, but for your UI tests, you have to access your application using accessibility and send events and see your application as a user would from the outside.

For your unit tests, all your tests are going to run in the same launch of the host application.

So, you should be careful to clean up between test invocations.

But for your UI tests, your tests can terminate and relaunch the application.

So, that's great if you want to test how your application starts up.

Okay. That's hosting.

That's how your test gets started in the first place, and once your tests are running, then you can use a technique known as observation to see detailed progress in your tests.

So, let's zoom in on that timeline.

Here you can see two test cases, one running after the other.

And these test cases belong to a test suite.

Test suite corresponds to a test class you've written.

And you'll have multiple test suites in your test bundle.

And maybe you're interested in doing some setup work before any of these tests run.

Or perhaps doing some tear down work after all the tests have finished.

Or maybe you're interested in doing some custom logging while the tests run.

Well, you can do that with the XCTestObservation Protocol.

You write an object that conforms to this protocol, and after registering it with a shared observation center, your object will receive call backs.

So, for instance, before any of the tests start, you'll get this BundleWillStart call back.

And then, before the suite starts, you'll get a SuiteWillStart call back.

Then, for every test case that runs, you'll get this testCaseWillStart call back and testCaseDidFinish call back.

If something goes awry, you'll get this testCaseDidFail call back.

And then, as the tests are wrapping up, you'll get this testSuiteDidFinish call back, and your final chance to do any work is in this testBundleWillFinish call back.

So, here's an example.

This object conforms to XCTestObservation Protocol.

In the object initializer, I'm going to register with a shared observation center.

Then, as the tests progress, I'm going to log events, such as this one.

When things go wrong with the test, I'm going to log this.

And then, once the show is over, I'll log that, too.

So, an ideal place to set up this observer is in your test bundles info.plist.

So, you do that using Xcode's info.plist editor.

You add this NS principal class entry, and this is a test specific instantiation for your observer.

It's independent of the class load initializer.

Okay. That's hosting and observation.

They're useful concepts to help understand exactly how your tests are running behind the scenes and to help you diagnose issues with your tests.

Let's look at some new features in Xcode 8.

Crashes are a frequent source of failures in tests, and the crashes can be in both your host application and the target application.

So, normally, Xcode will relaunch your host application to complete your test suite.

But it's up to you to gather the diagnostics necessary to resolve the crash.

I'm please to say that this year, Xcode is going to help with this issue.

We will now gather the crash logs for you in the test report.

So, this is both for UI and unit tests, for local and server runs of your tests.

The crash logs will be collected in the test report.

And you can see the textual representation of the crash there, or you can choose to see the crash in the context of your source.

And I'd like to show you that now.

Eric and I have been moonlighting.

We've been working on a tvOS application in our spare time.

So, I'll run the application by control clicking on the Run button.

It's an application defined nearby concerts with some strangely test related band names.

I can move in the simulator using the keys on the keyboard here.

So, we're adding a new feature to this application to support users who have location services disabled.

Let me show you what it's like for those users.

I'll go to the settings menu, and in privacy, I'll disable location services.

So, now, back in Xcode, we've added a new ViewController to support those users.

They can enter in a zip code.

Let me just add one more test for this ViewController.

So, I want these tests to run in a scheme on their own.

So, I'm going to go to the scheme menu, and I'll duplicate this existing scheme.

I'll call this something meaningful.

And, in the settings for the test action, I'm going to disable the debugger.

So, that's a technique to avoid interrupting the tests.

It allows all the tests to run to completion without breaking into the debugger.

And it's similar to how your tests run on Xcode Server.

So, I'll disable tests that are not related to location services.

I'll share the scheme.

And then, I'll run the test with Command U.

So, here's a new ViewController for users who have location services disabled.

They can choose to enter a zip code.

So, right now, the tests are entering a zip code, and that's an unexpected crash.

So, right now, Xcode has gathered the crash log, and it's reported the test failure.

And, let's have a look at that.

So, here's the failed test, and I can click to jump to the test report.

I'll disclose the test transcript.

Here's the events as the tests were entering in the zip code, and here at the bottom is the crash.

Now, I can click to see the textual representation of this crash, but in this case, I want to show you the crash in the context of the source.

So, I'll click on this arrow here.

So, here's the exact line that the crash occurred on.

And you can see in the top left, the stack frames in the debug navigator.

Now, I can see the crash in the context of my source and diagnose the issue.

So, I happen to know that I'm referring to the wrong ViewController here.

Instead of parent, this should be presenting ViewController.

So, I'll make that change, and now, I'll rerun the test by going to the test navigator and clicking on this icon here.

So, earlier, I showed you the test report for a local test run, but that report would look exactly the same if it came from Xcode Server.

So, now the tests are reentering a zip code.

Great. So, these tests have passed.

We would now check in this tests and build confidence that we're supporting users who have location services disabled.

But, for now, let's go back to slides.

So, you've seen how Xcode will now gather crash logs for you and capture them in the test report.

And you can choose to see the textual representation of those crashes, or you can see the crashes in the context of your source.

So, that's a great way to diagnose the issue and to make the fix right in the context of your source.

So, we have some new features in Xcode Server, and to show you more, please welcome my colleague, Eric Dudiak, on stage.

[ Applause ]

Good afternoon.

I'm Eric Dudiak, and I'm going to talk to you a bit about Xcode Server and what we have new in Xcode 8.

So, let's go over a little bit of an overview of what we're going to talk about today in Xcode 8 with Xcode Server.

So, we have custom environment variables that you can now set per integration.

We have advanced trigger editing workflow that we've improved in Xcode 8.

We also have some enhancements to issue tracking and blame to make sure you get notified of issues as they come up on your bot.

And we'll see how that plays with upgrade integrations, a new feature we have for you.

And finally, we'll talk about the configurable integration user which is new in Xcode 8, as well.

So, let's jump right in, and let's talk about custom environment variables.

Now, this is a little bit of a cheat.

It's actually new in Xcode 7.3, and it allows you to configure the exact environment that's passed to xcodebuild on your bot.

And it controls how your integrations are run there, so you can configure any number of settings that you might need for your bot to run on your server different than you might have locally.

This is a great way to customize how your server runs your integrations, and it's a great way to do that without having to create a lot of extra schemes in your project.

Now, on to what's actually new in Xcode 8.

So, we've significantly improved the trigger editing experience.

So, we have two types of triggers, scripts and emails.

First, let's talk a little bit about trigger scripts.

These run either before your integration or after your integration, and they're normal shell scripts.

So, we've improved the editor by giving you a lot more space to see the scripts, and you can actually see exactly what's in them, so you have a lot more visibility into that.

Along those lines, we also now let you name triggers.

This is great if you work on a team.

Your other teammates can see exactly what each trigger is supposed to be doing, and if you have particularly long running scripts that are part of your integration triggers, you will see that called out in the status as Xcode's integrating.

So, when you see a project integrating into status UI, you can actually see which script it's currently running.

Finally, if you have a script that you come up with later that you actually really wish was running before all your other scripts, you can now reorder triggers in this UI.

Simply add a new script trigger, drag it to the top, and it will become the first to run on your bot.

Now, let's talk a little bit about the other type of trigger we have.

We have email notifications.

Historically, in Xcode Server, this was always a one email per integration setup which can leave you with a lot of spam.

So, in order to help you and reduce the amount of email in your inbox, we've split these types of triggers up into two different types.

We still have the report triggers or report emails that come out every time you run an integration.

Of alternatively, you can now schedule them to run just once every day or once every week.

This way, no matter how often your bot runs, your email isn't flooded, or your inbox isn't flooded with a lot of emails.

And we think this is great for managers who want to check up on the health of a bot continuously but don't want all their inbox to be filled with emails.

Additionally, we also, now, let you configure certain fields on the email.

So, you can configure your cc field as well as your reply to fields.

This lets you have a lot more control over exactly the types of emails you're sending out.

Now, I hinted a little bit earlier that we actually, now, have two different types of email triggers.

The other is issues.

So, new issues come up.

We will now send you an email to the people of interest for that particular issue.

We'll go over a little bit of that in a minute.

But, if you do have more committers that are part of the code that you're integrating than you necessarily want to email, we do let you filter recipients right here to make sure you're emailing just the people you want to be.

So, if we take a look at that.

If you have multiple repositories, you can choose to only send emails to committers from certain repositories, or if you know exactly which domains all the email addresses will be from that you want to send emails to, you can add those right here.

So, let's take a minute and talk a little bit about issues or build issues anyways.

And, nobody is perfect and writes perfect code every time.

That's the whole reason we have continuous integration.

That's why we have unit tests.

We have unit tests because we know that they will inevitably fail.

That's why we write them in the first place.

We also, occasionally, commit code that just doesn't build, and some of us don't check it before we commit, and that's exactly the kind of thing that continuous integration is great at catching.

And, in these types of cases, Xcode will send you an email and notify you that you broke your build.

But those aren't the only issues that can come up.

Sometimes, even if you write absolutely perfect code, things can change around you.

That can be when you install a new Xcode, you get a whole bunch of other new features, too.

You get new SDKs that might have new deprecations.

You might get new issues from language improvements.

And, we like to make sure that every Xcode that we ship is smarter than the Xcode we shipped before.

So, you might see issues that we didn't find before that have always been there that Xcode now tracks such as static analysis issues.

So, the important thing when you see one of these emails is we really want to make sure it's actionable, and let you know that you're receiving this email, for the issue emails, when it is because something happened that you can do something about.

So, the first type is if you introduced an issue.

And that's because you broke it.

So, this will call you out, as you see right in the email.

You'll see an email like this that says you introduced an issue.

And we know this because the issue showed up on or near a line that you just recently modified.

We also might know if, between the two integrations where the issue showed up, you were the only person committing.

In that case, that's pretty much a fair guess that it's you that broke it.

Now, that's not always the case.

Take, for example, a application that's built on top of a framework.

A change in the framework might cause breakage in the application without anyone committing to the application.

So, in this case, you'll see an email a little bit more like this, and it'll be a little bit less condemning of your work and will simply say that you might be able to help fix an issue on the bot.

And we know this because you commit frequently to the area where the issue showed up.

So, we, essentially, assign ownership to various areas.

And, keep in mind, that this is a bit fuzzier matching than when we're directly blaming someone.

So, expect this email to go out to a bit of a wider audience.

It casts a very wide net when trying to figure out who might be of interest for a particular issue.

Now, that's great for issues that come up in your code, but sometimes, you can actually get issues that come up regardless of code changes.

One of the easiest ways to see that is actually by reconfiguring your bot.

So, Xcode 8 or Xcode Server in Xcode 8 will now track changes to your bot configuration and actually call those out.

Whenever possible, we will attribute any new issues that come up to changes in the configuration, such as if you enable testing or enable static analysis.

Those types of issues may have been in your code forever, but we can actually attribute them specifically to changes in the configuration of the bot rather than a change in your code.

We also make sure to include this information in emails for the next integration, so that when you do see one of these emails, you know that some of these changes might be because you're picking up a configuration change.

Let's go back to something we were talking about earlier of installing a new Xcode and getting a bunch of new features in the process.

Well, when that happens, Xcode Server on Xcode 8 will actually reintegrate your entire project, and we call this an upgrade integration.

We take the exact same revision of the previous integration of all your repositories and simply rerun all your tests, rebuild everything, rerun the static analyzer, and when this happens, we know that the issues that come up in your project at this point, any new issues, are specifically because of the upgrade since we took the exact same commits as before.

This saves you a ton of time trying to track down changes in your source code that just aren't there because the issues came up from changes around your code.

So, that's some of the new features and issues and blame, but I want to talk a little bit about my favorite new feature in Xcode Server and Xcode 8 which is our new, configurable integration user.

So, we now give you full control of the macOS user that's running your integrations.

And this gives you a lot of improved visibility into how your integrations are being run, and more to the point, it allows you to configure exactly how your integrations run.

Historically, in Xcode Server, there's a hidden macOS user that was running all of your integrations in the background.

You wouldn't have access to the password, and you wouldn't be able to log in as them.

This meant that you were basically getting a stock user no matter what, and you couldn't make any changes.

Now, you'll own and manage this user yourself.

You will be given the password.

This user will be a completely normal macOS user.

It'll be any user on the system.

Can be anyone you want.

We suggest a new user.

But it will available at login window, and it's fast user switching.

So, you can log in to that user.

You will notice you're running as that user through a menu extra that shows the Xcode Server integration icon.

We'll see that in just a minute.

Let's go over how you would set up this user.

So, here I've opened the server app to the Xcode service pane, and in order to enable the Xcode Server, I need to go ahead and choose an Xcode.

So, I'll choose a new Xcode 8 that I have installed, and I'll be presented with this dialog, asking me to set up which integration user I want to use, and I'm going to go ahead and create a new user just for running integrations.

I'll go ahead and give it a name and a password, normal user stuff.

And I'm going to leave it as a basic user.

And when I push "create user," Xcode's going to go ahead and do a little bit of work in the background while it gets everything ready to run integrations, and once that user's mostly set up, we're going to be asked to login as them.

Now, we are asked to log in as them because we're going to go ahead and run a little bit of the setup assistant.

Remember, it is a real, normal macOS user.

So I can log in, I can sign into a test iCloud account, for example.

I can stage any data that I want to have for my integrations.

Anything like that.

When Xcode's ready to run integrations, you will see a notification like the one you see in the top right here, and in the top right in the menu bar, you'll notice that there is a little hammer indicating that this is the Xcode Server integration user.

With that all done, I can go ahead and switch back, and I'll be back in server app, and I'll have the indication that the user is ready to go, logged in, and integrations can begin running.

So, now, we've seen how to set up that user.

Let's take a look at some of the things that we can do now that we have access to the integration user.

And for that, let's go to the demo machine.

All right.

Here we have that same project we saw earlier, and we saw an issue where it was crashing because it didn't have access to location.

And we had to go in manually to the simulator and turn off location access to test it locally.

But, as we're developing our application, we might not want to do that.

We probably want to leave location services turned on on our Apple TVs and on our simulators, so that we're running, more or less, the way most of our users will see it and how we want to see the app.

But, now that we've fixed the issue we saw earlier, we definitely don't want it to come up again, and we want to be notified if it does.

So, it just so happens that this machine I'm actually on right now, happens to be my server machine.

So, I don't have to go anywhere to go to my server.

I can actually just go to the fast user switching menu, and I see that have a configurable, continue, integration user called Xcode Server.

And I'm just going to go ahead and select that and enter my super-secret password that's only four characters long.

And we will log in as this user.

So, I've gone ahead and changed the desktop background so that I know that it's the build service user.

And, here we see the hammer icon of Xcode that indicates that this is Xcode Server.

We see it's on and waiting for integrations.

So, it's all configured, but I want to go ahead and go into Xcode.

So, I can actually configure any settings I want in Xcode locally on this user, and they will be picked up in my integrations.

So, I'm going to use a neat little trick.

If I go to the devices menu, I see all the devices and simulators that are plugged into this particular machine.

Now, being generous at Apple, we give you one of every device we've ever made, but if that's not enough for you, for absolutely free, you can get more simulators just by clicking this Plus [+] button.

Now, the Apple TV 1080p simulator's the built in one that I got when I installed Xcode.

I've gone ahead and created another one called Apple TV no location, and I've actually got it booted here.

And if I go in, I can double check, and it is, indeed, has location services turned off.

So, I go to settings, general, privacy.

Location services is turned off.

So, this simulator is all configured for use, and my other one's still usable for when I want location services turned on.

Now, let's do a quick switch back, and just log back into my normal user.

Normally, I'd be walking across the room to my actual work machine, so this is a little faster.

And here we see all the UI tests.

Now, I already have a bot that's integrating this normally, but I want to go ahead and create a bot that will integrate just the case where we don't have location data.

So, I'm going to go product, create bot, and use the Harmony no location scheme on this server.

Going to go ahead and give it location, sorry, access to my repository so it can check out the project.

I'm going to disable the archive action because this will be the same as my normal bot.

So, I don't care about that, but I do want to run tests.

And, let's leave static analysis on.

And I want to be notified as soon as I break this, but probably once a day is good enough.

So, I'll know within 24 hours if we ever break this again, our UI test will catch it on our server.

So, we'll just run it every day at 1:00 AM, and here I get to select the devices.

By default, every integration runs on all devices of that OS.

So, this is a tvOS project, so it'll run on all tvOS devices and simulators.

Instead, let's do specific TV devices, and I'm just going to select the Apple TV no location.

This is that environment variable view we talked about earlier.

I don't have any environment variables I need.

I'm going to go ahead and make sure that I get emailed when a new issue comes up in this particular case, and go ahead and create the bot.

So, that is going to go ahead and kick off an integration which we see running here.

It's checking out building.

We've already seen the UI test run a little bit, so let's actually look at an integration that I prebaked a little bit earlier.

And if we go to the test, we can see that it did run all our tests including this zip code test.

And we can see, if I look at some of the screenshots from it, it is, indeed, running without location because it's opening that ViewController that we saw earlier.

So, we saw how you were able to get significantly improved visibility into how your user runs.

You can log in as them and see everything that's happening.

We saw how you could customize different settings such as simulators to run exactly the operations that you want to run on your integrations.

We saw this was a completely normal macOS user that I was able to switch to in fast user switching and that I had the password to.

And, we got to see that menu extra that shows you a little bit of your integration status in that integration user.

Now, with this great new power comes a little bit of responsibility, and so we have some best practices for you.

First, we highly recommend you dedicate a new user.

You want this to be as similar to your customer's experience as possible, so you don't want all your settings to be causing impact on your bots.

We also recommend you avoid administrator accounts.

Keep in mind that anyone who can create or edit bots on your serve will have access to this account through triggers, so this goes the same for any private or customer data.

Try to avoid storing that in this user.

If you want to stay logged in as this user in the background, use fast user switching.

Integrations can continue to run in the background just as they always have.

However, if you do want to run the integrations as the front most user, make sure that you turn off screen lock.

Just like iOS screen lock blocks testing, so does macOS screen lock.

And finally, make sure to customize this for whatever needs you have.

That includes, as we saw earlier, any simulators.

If you have any specific networking configuration that you need for your integrations, any stage user data or settings that you want for UI tests, go ahead and configure those.

And finally, if you have any advanced provisioning, such as if your administrator gives you a provisioning profile, however you would configure on Xcode, you can actually configure that in your configurable integrations user, now, and make sure that code signing will work for you.

So, with that, I'm going to bring Zoltan back to talk about some new features in xcodebuild.

[ Applause ]

Thank you, Eric.

So, you've seen new features in Xcode and Xcode Server, and now we have new features to show you in xcodebuild.

xcodebuild has the test action for custom continuous integration systems.

You give it a workspace, a scheme, and a destination, and xcodebuild will dutifully build your sources.

It will install built products onto devices as needed.

It will run your tests, and then report results to you on the command line.

So, this year, we're introducing two new options for this action.

With the only testing option, you can effectively constrain the set of tests that the action will run.

So, you can specify TestCases or TestSuites or TestBundles to run to the exclusion of other tests.

And you can also use the skip-testing option to specify TestCases to exclude from the tests while running everything else.

And the big news this year is that we're introducing two new actions to xcodebuild.

So, we're effectively splitting the test action in two.

These actions are already available in the Xcode IDE, but we're going to bring them to xcodebuild, now.

And so, we'll look at each in turn.

Build for testing is just the building portions of the test action.

So, you give it a workspace, a scheme, and a destination as before, and it will build your sources, perhaps making certain symbols visible, and then it will output the built products to derive data.

It also produces this xctestrun file.

It's a kind of manifest for everything your test needs to run.

We'll come back to that in a minute.

Test without building then, is the second part to the story.

You give it a workspace, a scheme, and a destination, as before.

And, xcodebuild will find the build products in derived data.

It will install those onto devices as needed, and then run your tests and report results just as before.

But the cool thing is, you no longer need to provide a workspace.

Instead, you can just provide this xctestrun file, and Xcode will run your tests from binaries alone.

So, it will ingest the xctestrun file.

It will find binary products relative to that file, and then it will run the tests and report results in exactly the same way as before but without having any access to your sources.

So, this is ideal for distributed testing environments.

You would build your tests on machines up to optimized for building, and then move those built products onto machines optimized for testing.

And then, in parallel, you can gather the test reports.

So, this is all made possible by the xctestrun file, that manifest for your tests that specifies which tests to run and which to skip and which test machines.

It provides environment variables and command line arguments to your tests, and you can read more about the format in our man pages using this command.

[ Applause ]

So, we covered a lot of ground today.

We first reviewed testing in terms of these four characters, and then we looked at some new concepts in Xcode.

They were great for diagnosing issues with your tests, and then we introduced new features in Xcode, in Xcode Server, and in xcodebuild.

I hope these insights and these new features help you to develop test suites which perform well, and, after all, that will help you deliver great applications.

You can read more about this session at this URL.

Thank you.

[ Applause ]

Apple, Inc. AAPL
1 Infinite Loop Cupertino CA 95014 US