Check out RailsClips on Kickstarter!!

 

01:56 - Testing and Test-Driven Development (TDD)

03:23 - Panel Experiences with TDD

08:10 - Value Objects

09:08 - How To Do TDD

11:28 - Jaim’s TDD Process

13:44 - Value and Getting Started with Testing

21:58 - Writing Tests First

27:18 - Testing on a Team

32:47 - Higher Level Testing

36:54 - KIF

38:00 - Other Ways of Testing UIs

39:44 - Who Writes the Tests?

44:06 - Test Data and Environments

46:50 - Lower-level to Higher-level Tests Transition

51:51 - Recording User Interactions

Picks

John Reid: UIViewController TDD [Screencast] (Jaim)
Test-Driven iOS Development (Developer's Library) by Graham Lee (Jaim)
WatchKit FAQ (Alondo)
This Idea Must Die: Scientific Theories That Are Blocking Progress (Edge Question Series) by John Brockman (Alondo)
Martin Fowler: The Test Pyramid (Pete)
Working Effectively with Unit Tests by Jay Fields (Pete)
Avery Brewing IPA (Pete)
A Wizard of Earthsea by Ursula K. Le Guin (Chuck)
80/20 Sales and Marketing: The Definitive Guide to Working Less and Making More by Perry Marshall (Chuck)
Miracles and Massacres: True and Untold Stories of the Making of America by Glenn Beck (Chuck)

 

If there’s noise in the background I can frown furiously.

Good day.

How directional is the mic, did you just hear Linda say good day?

Yes.

Yes.

Okay, not so directional.

Bloody hell.

Hey everybody and welcome to episode 95 of the iPhreaks Show. This week on our panel we have Alondo Brewington.

Greetings from snowy North Carolina.

Jaim Zuber.

You stole my line [chuckles].

Pete Hodgson.

Good morning from sunny Los Angeles.

I’m Charles Max Wood from devchat.tv and this week we’re going to be talking a little more about testing and TDD in particular. So we talked a lot about the tools that people mostly for more of a BDD set of tools, kind of like our spec in Ruby last week with Natasha or two weeks ago. We thought we’d talk a little bit more about the process of testing and some of the things that you can do to make that easier. I’m curious, how many of you guys are TDD practitioners?

All the time. Are we talking like a purist TDD; write your test always first or – yeah, we write tests. You get into semantics with this [crosstalk 02:32].

I think we should start with the definition of what does TDD mean, might be a good place to start, before we decide if we’re doing it or not doing it.

It probably means you’re doing it wrong.

I can definitively say I’m doing neither [chuckles].

We got one clear answer, that’s good.

So Chuck, what’s your definition of TDD?

TDD? Well, it’s the process of specifying what your code is supposed to do before you write it, by writing tests.

That’s a fairly good definition. I suspect that very few people are doing pure TDD. I certainly don’t do that all the time, but I think there are a lot of people doing some form of automated testing part of their coding. There’s the distinction of when you do it, and how much you and what type that you do.

I think we discussed in a couple of different episodes the benefits of having tests. I’m curious though, you guys who have at least tried TDD or do TDD, what has your experience been? Was it hard, you gave up or was it “I stuck with it and I learned to love it.”? I’m curious.

I’ve seen a bunch of teams or individuals go down the path of starting of TDD, the advice that I now give to people is start with getting some understanding of how to do test automation, how to write unit tests, and what a unit test is. Get that figured out first before you try and learn how to do the TDD part of writing a test first because both of those are really hard skills. If you’re trying to solve or learn both of those at the same time it’s quite easy to get discouraged because it’s really hard.

When I normally see people stop, I suggest they start with just learning how to do a unit test of light simple stuff and then do a test for hard stuff. Then start thinking about how to write the test to define functionality before the functionality. I think it’s a really hard skill; it’s like chess it’s easy to learn the basic s but you can spend years and years learning the intricacies of how to do TDD correctly, in my opinion; how to write tests in general.

So if we’re talking about a unit test, what is that exactly? In a can of worms [chuckles]

My definition of the unit test is an automated test that gives you a binary yes or no answer that the computer can say, “Yes, that passed,” or “No, that didn’t.” It’s focused on the very small unit of code, so that’s like an individual method or maybe an individual class. Then it’s also something that’s – I lost the third part, I had a third piece but I can’t remember what it was. Basically it’s small – oh, it doesn’t touch anything apart from. It doesn’t touch the date space, doesn’t touch the FAT system, doesn’t need anything to be standing up, doesn’t need other than the class under test. So it’s an isolated, automated test of a small unit of code. [Crosstalk 05:22]

I think that’s a good definition. So it’s just in code, you can create a method that creates your object, calls some function on it, test some properties and verifies what happens; like going to a database, like going to a network, just one small little unit.

Yeah. Then the thing is that definition then gets extended so that you wind up with – where it does reach in to the outside world, mocking and stubbing, which complicate the discussion around testing, a bit.

Right, so that was my definition of the unit test; that it’s an isolated unit a single method or class, but that’s partly my definition because I come from more of the mock-ist school of aggressively isolating the code under test by using mocks and stubs. There's are a lot of folks who come more from the classical school of TDD or the state-ist school where they’re okay with including instances of other objects and still defining that as a unit test. So even though you’re not totally isolating your test to just a single class, in that school of unit testing that would still be considered a unit test.

44] or not.

Okay so if I hear you correctly, for a mock-ist point of view, you want one class under test at a time.

Anything that the class would depend on, a dependency, you would create stub or a mock for.

Yes, apart from a value object. So the way that I think about it is not in terms of, “I’m now going back to what I said of having a hard and fast rule”. So let’s say for example you have a method on a class which is responsible for a user’s name. So you give it a user object and it will give you the user’s name ready to be displayed, so it just adds the first name and last name together. If I wanted to test that functionality and I was super strict about only including the code that I’m testing, then I would give a stub version of the user object to that thing I’m testing to verify that; when the first name is Pete and the last name is Hodgson, it gives me X but I would create a fake user using a stubbing library.

In reality, it almost always makes sense. You can create a real instance of that user object and that user object’s a value object that doesn’t drag a bunch of other dependencies along with it, then in my opinion that’s the preferred approach. It’s more about isolating the focus of your test than it is about only having one real object in play. I don’t know if that made sense.

I think it did. So you’re not exclusively focused on one class but you’ve also brought the term “value objects”. What are you talking about with the value object?

Basically, what I’m thinking of with that is a thing that represents an object in the system which you can create an instance of that object without having a bunch of dependencies attached to it.

39] is probably going to want to have some natural views so It references and it has all of these dependencies on other parts of a system, whereas just a really straightforward value type thing like a user or an event or a money object. You can create it in isolation. One way of thinking of it is you can new up an instance of this class, just giving it a primitive value or other value object.

We’re talking a little bit about how you do this. I want to step up a level though, and instead of talking about how to write the test, I’d like to talk about how you do the TDD. So the mantra of TDD is Red, Green, Refactor. In other words, you write a test that doesn’t pass, and then you run your tests and it comes back where maybe [inaudible 09:30] it didn’t pass, and then you write code to make it pass and then you refactor the code so it’s not ugly.

I’ve actually had clients – I get red, green. Then I show them something work and they don’t want me to refactor which is another discussion. I really like the approach; one thing that I have to say is as I get into to TDD, I tend to drill down a little bit. It’s usually red, then I go and I start writing the code. So I write more tests to specify the lower level stuff because I’ll write the test for the use case or the outer edge of things. Then I’ll write testes for the next level in.

I’ve seen that referred to as the BDD style where – and there’s a lot of confusion here because there’s BDD; this syntactic kind of style of it blocks and describe blocks like we were talking about with Natasha a few weeks ago. Then there’s BDD the kind of philosophical thing of defining the behavior from the top level. Then use that to drive out the lower level tests that you build. I think it’s an interesting thing that these two things have got conflated when really they’re quite separate; one it’s just a style of writing tests and the other is a philosophical approach to how you develop your software. I’ve seen that referred to as the BDD cycle where you have these high level tests that describe the behavior of your system. Then you use that one failing high level test to drive out a bunch of smaller unit tests and you’re doing like a nested Red, Green, Refactor cycle inside of another one.

I think the Cucumber book – either the RSpec book by the Prag Programmers or the Cucumber book; I don’t know which one – is the one where I first saw that formally defined. And there’s a good diagram in there that shows this cycle within a cycle.

So Jaim, did you say that you’ve done TDD?

I have.

What does your process look like? Is it generally that or is it less formalized than that?

it’s different every time because I’m not always operating in a situation where I understand [inaudible 11:45] what things are supposed to look like at the end. I’m building up small parts possibly; I don’t maybe have clear requirements. But in the cases where I have a reasonable idea of what I’m supposed to be building, it’s definitely possible to get into a routine where; write a test, get it to pass, Red, Green, Refactor. It’s possible to do it, if I’m disciplined that’s a decent way of doing things. A lot of times I’m just trying to figure out what’s going to work.

If we’re doing UI stuff you don’t always know what’s going to look correctly, those two can be difficult to test in a pure TDD cycle, but business logic things that lowdown where you can actually test logic that makes sense to do things for.

33] other people using it, how easy the code is to break. So something’s going to break easy, I’ll write a test just to keep it working but it’s definitely possible to do PD on iOS, just a little bit hokey. I think one of the things that is glossed over when people start people start talking about TDD is and how great it is. The people that have already gone through that learning cycle are the people who are talking about how great it is and you sit down, “Hey, I’m going to start writing tests.” You have no clue how to navigate the different things we’re talking about – you start at the top level, you start at the bottom level. If you have three or four classes working together, is it okay to test at that level? And depending on the person you ask questions from, you get different answers.

24] stuff for years and years but writing testable code is definitely a different mindset. It takes a while of trying something new like, “That’s terrible, who would ever go this way?” I’m like, “Yeah, here’s the only way I can do it.” There are definitely things you can do and it’s valuable to learn it but it’s not an easy task if you’re not used to it.

That’s the thing I’m curious about. As someone who’s actually attempted to do this a couple of times, I’ve never able to stick with it. I’m really looking for someone who’s interested and think there’s value in it but I keep running into that road block and it’s easier to just go back to the old way. How do I beat that?

It’s okay to take a run at it and get a little bit better and then take another run at it. I don’t think people should give themselves a hard time because they failed getting test effective the first time they tried. For me personally, it took me probably two or three years of off and on playing around with this and doing it on my own time, unlike my own projects. Then not doing it at work and then trying to do it at work and not being able to figure it out.

It took me a few years of that before it really clicked for me. I think part of it is just trying it out for a while and muddling through how to solve these problems and being okay with it being actually quite a long haul to get comfortable solving these problems.

Yeah, look for quick wins. If you start, just take a random block of code that we had to write in iOS, maybe we’re talking about core data, maybe we’re talking with notification center, and we’re talking to views. Start with your testing in a code like that?It’s difficult, but it’s easier to test code that the just plain old [inaudible 15:08] subjects the Ponzi [crosstalk 15:10] – the Ponzo.

Start with a model object that is just code that formats a string that just does some business logic. Start with things like that. With view controllers, you can write tests to make sure your views are connected in the nib. John Reid did some CADAs for doing things like that. So start with small sections of code, you don’t want to do everything because it’s hard to test everything, I don’t test everything.

I’d also say like maybe it’s a good advice when you’re starting – I totally agree with that sentiment of find easy places to get familiar with annex of writing your tests and get quick wins at it where to feel like you’re making some progress. And part of that is avoiding initially testing stuff that requires a lot of mocking and stubbing, using all of those power tools because partly it’s just harder. It’s just an extra thing to learn at the same time as everything else, and there’s a bunch of syntax you have to learn and APIs and it’s confusing and partly because that’s where people really can – that’s the first stage of failure with people trying to get in to unit testing where they’ll write tests using lots of stubs that just absolutely shackle their code.

This is kind of trough of despair with TDD, or with unit testing for a lot of teams where about three months in after doing it, they suddenly realize that every time they want to change a line of code they have to change 500 lines of unit tests. Normally that’s because they drank the Kool Aid too much on mocking and stubbing and they’ve over specified the behavior of the internal software. It’s a very easy trap to fall into. So I would say initially focus on the long game and start off by getting good tests around things that are easy to test. Like business logic is divorced from framework code-tac kind of thing. Then you’re less likely to get into a big tangle of mocks and stubs where you feel like the test is slowing you down instead of speeding you up.

That’s a great point. From the mock-ist school of testing, I was scarred from that for a long time because the first time I sat down and that have been thoroughly inter-tested, the mock-ist style I did. I sat down and got to this code, it was all tested. I’m like “Oh, great.” I did the simplest [inaudible 17:38] just extracted the method, things in there and I wrote 40 tests and I was completely lost.

There are different approaches. I think the important thing, what you talked about earlier Chuck, is to refactor. You refactor your code but you also refactor your tests and you need to know how to do that.

Refactoring can include deleting stuff. It’s okay to delete a test that’s no longer providing any value.

I love deleting stuff.

Delete all the tests, here we go.

It’s true though because I do this a lot with – I guess we’re skipping to once you’ve become really good at TDD. For myself now, I’m very comfortable with the mechanics of how a TDD flow goes, I will very often start off a new work day or a new problem that I need to solve by just writing a really stupid test that I’m planning to delete in ten minutes just to get past that writer’s block.

So I’ll write a stupid test like the class exists. The test provides no value other than to give me a place to start driving towards building out that class’ functionality. I’ll write that test, then I’ll get it to pass then I’ll move on and then about a day later or an hour later, I’ll just go through and be like, “Oh that test is serving no value anymore.” And I’ll delete it. There are no sacred cows in terms of like, “If I’ll delete this test then I’ll lose some kind of protection.” That’s true but you’ll also lose some overhead in terms of the amount of tests you need to change when you change your code.

The point where you have a lot of redundant functionality that you’re testing a lot of test that are testing the same thing in maybe slightly different ways. It may slow down your ability to change which goes against what we’re trying to do, we’re testing it in the first place.

I think that’s really true. The tests are there to give you the confidence to move forward and that speeds up development. That’s something I picked up from Martin Fowler when we talked to him about refactoring on Ruby Rogues. So your test enables you to refactor without worry; it enables you to add things because you understand that you have the safety net that’s going to catch most of the things that you can make go wrong.

One of the benefits I’ve heard about this approach is that it helps you with design that you’re making. You’re designing classes in a more efficient way, in ways that makes sense. So you find that you’re experiencing that or not?

Absolutely.

I would say that I do. The reason is that basically what you do is when you’re doing the test step or you’re writing the test that’s not going to pass, you’re specifying the criteria for success, for whatever it is you’re writing. And at the same time, you’re also usually specifying some kind of API; I’m going to call this method, I'm going to do this kind of thing to this object.

And so you have an idea of what you want to do, of what your outcomes need to be. So that’s the design piece. And so, when I sit down to write some code and I write the test first, that’s what I’m thinking about is the client or the customer says that they want a button that sends an email to the person who clicked it. So when they tap that button on their screen, I already have their email address because they entered it into the app so now I’m boiling that down. So I need to check and see that an email was sent, that it was sent to this address, that it had this text in it – you know those kinds of things. And so, then when I sit down to write the code, I can say, “Okay, I need the button wired up to this action on the view controller,” and then I can just test that action on the view controller and make sure that it can see the outcomes that I specified.

When I go to write the code then I know what the method needs to be called, I know what it needs to do, I know what I’m going to be looking for in the end. So if I forget a step or miss something, then it gives me that reminder. But it also helps me just think about and specify what exactly needs to happen in enough detail to where I can put it in to a test and tell the computer what to look for.

I think the analogy that I make when I’m trying to describe that, why does testing first improve that design of your code? The way that I look at it is when you’re writing your test first, the first you’re doing when you’re defining a new piece of functionality is you’re thinking about how is someone going to use this piece of functionality? Just like really good quality iOS apps of user-focused, and we’re thinking how is the user going to achieve their goal. And that’s the thing that drives our decision making, in my usability point of view.

It might be interface for the user. If you’re writing your tests first, then you’re forced to think about how people are going to use the internal APIs of your classes and your objects first. So you’re always driving out the design of your application based on usability. It doesn’t mean that the code is inherently high a quality; what it does mean is the code is easier to change over time.

Your design has better ergonomics so when you need to change things in three months’ time, two weeks’ time, it’s easier because the classes are more loosely coupled, more cohesive because your tests have been driving them in that direction. So, that for me is the reason I do TDD, it’s because it helped me write code that’s easier to maintain in the long run because most of the time we’re changing code, we're not writing code.

I agree. I think he stumbled on a truth that I’ve heard before. If code is easy to test, it’s easy to change which is very important to be for code quality. If you get to a point where this code you’re writing is hard to make a test for, that’s probably a sign that maybe it’s time to refactor.

And you get that feedback while you’re writing the code versus a couple of months down the road when you’re trying to change something else. So you get the feedback about your design, how easy you code is to use upfront which is very valuable. Your head’s in the right place where you can actually start working with it.

It’s kind of like to labor that analogy; that user interface a bit more is like a difference between creating the entire UI based on how you know the internals of the application work. Like, “There will be a list of events”, so I’ll create a table with events inside of them and I will sort them in the order that are in the database because that makes the most sense to me based on the internals. If you do that then you’ll build the UI that’s functional but not really usable.

And if you then take that mediocre UI and try to turn it into a good UI it takes a lot of time and effort. Whereas if you start with the focus of how is someone going to use this app occasionally, then you tend to build it right the first time. For me TDD gives me that same aspect for the internal design of my classes rather than the user interface of the app.

Yes definitely. Another benefit is if you’re writing test before you write code you stop with the test pass. My default is always to add more functionality than I need into a class. I’m creating a new class, “Here’s all the things I might need.” Then I’ll start writing them out. A lot of times I’ll start writing tests; I actually have a third of the functionality I thought I needed and I’m done. So that’s one third of the moving parts that can break in the future. So the most important thing you can do to improve code quality is to have less of it. Keep the honest.

I think that’s where Chuck’s thing is the BDD cycle of starting with what’s the functionality I need from the top level and then that defines what lower level tests to use to drive out the internal functions; that’s when that really kicks in, right? Because when we start doing something – if we start at the bottom level an individual class – and let’s say we got a user a place to store users and we need a way to create users and we need a way to edit users.

15] stuff you don’t have to maintain as a win. [Inaudible 26:19]

We should put a caveat on the truism that testing makes your code better because at certain extreme especially with static languages, if you’re insistent in testing everything like framework calls, you can get your code where it’s harder to read. I learned TDD in doing C#. So if you want to access something from the framework, you had to create a wrapper for it and pass it in as a dependency versus just using it. You guys are more Ruby people where you don’t really have to do that. You can just mock around it and use some run time food to write the test for it.

But if it gets to the point where you’re testing every little framework call and having to wrap that functionality and pass it in, that can make code harder to read. So at the extreme ends, they can have a negative effect on readability in my opinion. That can be more of a concern with Swift versus Objective-C because of its static nature.

So a follow-up question on that. So I'll give it another shot and I go down that path, but I work on a team where not doing that as a practice. Is that more or less difficult or is it irrelevant, whether or not I’m the only person in the team of developers taking this approach?

That’s the tough question with iOS because testing is not integrated into the culture like in different communities. Still can be valuable to write tests and have them work, because it’s not hard to run them if people remember to. If you write your test, that’s the line in the sand saying, “This is how my code is supposed to work, no one break it.” If they break it, you can go to them and say, “Hey, don’t break this.”

It is difficult. If you’re the only person trying to write tests, it’s hard to make that a culture. It has to be allowable. So people up top that are making the decisions should be at least on board. But it is valuable. Just be the beachhead where you’re writing some tests and say, “Hey, don’t break these tests.”

It’s definitely challenging. I think the basic tests are easy to write and easy to fix the kind of beginner level unit tests. I think there a reasonable argument to be made that you could just introduce those on your own. Discuss it with the team and let them know; don’t just sneak them in there. Do it as something that, “Let’s try this out.” Ask the team to keep an eye on them, and if they break then at least let you know and ask you to help fix them.

I’ve seen that can go wrong is if the whole team isn’t on bought in to the idea of trying to do this. These very mock heavy tests that mean that it’s harder for the team as a whole to change code without breaking stuff. You can sour everyone to the idea of tests in general and make it a lot harder to get the rest of the team on board. So I think that’s part of why I would say stop with some easy, entry level testing and try and get enough of the team interested and sold on that. Then you can say okay great as a team we like this now, let’s try and step it up and get more of our code on the test.”

One other thing that I see really help with getting teams on board with testing, and I’ve gone through this with a couple of different teams, is automation as much as you can. In Ruby and Rails, there’s guard, so you just set it up so that it’s like, “Look I have to run this command and it’ll run your test for you.” There are test runners in most systems. I haven’t explored what’s available in iOS or in Cocoa in particular. I’m sure there’s something out there that’ll do it.

In some ways, it’s even easier for a language like Objective-C [inaudible 30:00] like Swift. Because there’s already that idea of compiling your code and so you’ve already got the idea of after I’ve done of making a change to my source code, I run the build step and get some feedback on whatever it compiles and I get some feedback from the linter or from clang telling me this variable isn’t used.

So we’re already on the mindset of getting the feedback on the quality of that code when we hit a button. So for me when you want to get started unit testing you just include the unit tests in hitting that button.

It’s easier to get an iOS team on board with that idea than it is to get for example the JavaScript team on board because they used to save the file, reload the browser and that’s the feedback route for them.

One other thing that I’ve seen really helps is continuous integration. There are systems out there that are varying degrees of easy to set up. So if you can get something set up in an hour or something that notifies the team when something breaks via email or via having some dashboard up in your office. It’s just another reminder, “Hey, somebody broke something. Let’s go fix it.”

All it has to do is save your team a whole bunch of trouble because the CEO is pissed off because there’s a bug. “Oh, we caught that, it’s a good thing we caught that,” or a quick reminder that somebody did something that was a little bit different from the team’s norms, and saves somebody time down the line. It starts to pay itself forward, it starts to pay for itself.

I would say if you’re an iOS team today and you’re saying, “Should we start writing unit tests?” And you don’t have CI, then stop thinking about writing U-tests and set up CI that just checks whether your code compiles. You’re going to get more bang for your buck in that. Once you’ve got that in place ,then you can start unit tests. You’re leaving a ton of money on the table. If you’re a team of more than one person, and you’re not doing CI, you’re not having the computer do the work for you to check that the code that you checked actually compiles, then you’re just leaving money on the table.

Yeah, it’s completely easy to get something working for your machine that doesn’t work for anyone else. You know you had a framework from the wrong position that only exists on your machine. And you go home for the night, and everyone else is out of luck. So a CI catches that right away.

When you merge in someone else’s code and there’s a merge conflict. So you fix the merge and then push and you forgot to check where it compiles and it turns out that there’s some stupid error that doesn’t compile. Everyone’s made those kinds of mistakes and it’s nice to have the computer tell you, you made the mistake rather than someone be grumpy at you the next day.

Or call you at three a.m. [chuckles]

Or show up at your house at three a.m. [chuckles]. I hate when they do that.

So what about other areas in which we can test? We’ve talked a lot about testing at the unit level and using BDD, but are there some other practices that use to improve the overall app quality by testing it at higher levels or more comprehensively?

For me that’s why I like to have that distinction of what’s a unit test because then we can talk about what other types of testing we can do. So if we say that a unit test is this low level test of an individual class or maybe a few classes collaborating in isolation. Then you can also have high level test, and testing the way that groups of classes or the whole system works together. So for example you can have tests that check that when the view controller tries to do something to a domain object, then it does something, saves it to core data and then refreshes some other part of the UI.

So you can write these tests that are more high level and then not really testing so much the functionality of individual things but more the way that these things integrate with each other. So the way the different parts of the systems, the different parts of the application integrate with each other. Then you can go higher level than that and start talking about UI based test where you’re actually prodding automatically, pretending to be a user and poking a button on the screen. You’re making sure that when I tap “add user”, and I fill in my first name and my last name and I hit save, then that user gets save to some backend service. That’s the highest level test you can do on a UI level acceptance test.

I think one of the things that people really want to do is do a lot of tests. Once you give someone the ability to do those high level UI automation test, they almost always start wanting to write more of them because they feel more valuable, because they’re testing more of the code at once. They’re kind of in some ways easier to write because you don’t have to change the way that you’re writing code. You just pretend to be a user and write some little scripts and verify that they do stuff. The problem is those tests are also very prone to failure, very prone to break when you change your code and very slow to run. So they’re actually less valuable than you expect.

That intermediary area that you mentioned, one layer up above unit tests testing a collection of things. Say for instance one when an action happens in a view controller, am I using the same tool kit to write those tests than I am at the unit level or is it requiring using some additional libraries?

That’s a good question. I’m a really big fan of just using the same tools so if you write in ObjectiveC you can use Kiwi for example as the test runner and the test framework for your unit test. Then you could also use Kiwi to write these high level tests. You end up having to do more set up and tear down because by definition, you’re setting up a collaborating cluster of classes and then poking at them and prodding at them and verifying what they do.

You can still use the same libraries. What’s really nice about iOS in general is there’s no magic with the way the UI works. View controllers adjust classes they derive them and a subject indirectly obviously. But they’re just objects that you can instantiate and that you can mess around with, and the same is true with everything else in the system. It’s very possible. Just like Jaim is saying with John Reid’s example of testing how view controller binds to nibs.

It’s very easy for you to do that programmatically using a regular unit test, test runner like Kiwi and still verify that high level behavior. You can even do that all the way up to the UI automation level. So if you’re using KIF which is like one of the automation, UI automation frameworks available to iOS. I don’t know if they’ve changed this now; it used to be that they had a separate test runner but there’s no reason you couldn’t use Kiwi for example to drive KIF test which simulate user interactions. So it’s still running in process, still running as a Kiwi test suite but it’s testing when I swipe this button then I should see this text in this area in the UI.

So how does KIF work? How does it simulate what’s happening in the UI? Is it – some clicks, what’s going on?

The way it used to work is it would actually simulate the events. I think it’s the way it still works haven’t looked at it for a while. It basically simulates the low level touch events that occur in the system. So there’s three parts of any kind of testing – UI testing framework. There’s simulating interaction with the UI, there’s verifying output from the UI, and then there’s a third part which I forgot. I had a whole conference talk about this, I should know this stuff.

56] just programmatically just by getting subjective views [inaudible 37:58].

What are some other ways of testing UI's?

Well, I'm the maintainer of that. [Chuckles 38:06] There’s two schools of UI level testing; there’s essentially two types of tools. There’s the KIF type of tool that’s in-process. You write your test in Objective-C and it’s just directly injecting stuff into your UI.

The other technique, or the other kind of style of these UI level tests is more independent where you have a little thing that’s embedded inside of your app that’s listening for commands over an http connection. And you write your text in whichever language you prefer. So you might have a test that says when I swipe this thing, then I should see this thing on the UI. And under the covers, that test turns into a series of commands that you’re sending to your application say, “Hey, simulate this swipe. Hey, verify this text at this value.” Internally your app is using similar techniques in order to do the simulation that KIF uses, but you don’t have to write your tests in Objective-C. So, a very popular version of this is Calabash. The one I maintain is Frank, and then there’s another one called Appium.

The standard one – I forgot the official one that Apple has. Apple has UI automation which does similar stuff. It’s a similar style to that second style, like out process style but it uses some secret API that we’re not allowed to use to really get under the covers into the application. And in that case, you use Javascript – a rather clunky mechanism to run the test. There are very well set up for a CI, for example.

So given that your team wants to start writing UI tests like this, who writes the test? Is the developer writing the test before they code? Is QA writing the test? What has worked with teams you’ve been on?

So it’s super context specific, really depends on the type of teams. For the standard type of thought works [inaudible 40:03] shaped team where we have devs that are very into test automation, then usually a dev will write these high level tests in collaboration with a QA or BA, so they’ll sit together. Like Chuck was alluding to earlier with the BDD cycle, we might get together before we play a story and say, “How can we verify this behavior with a test?”

And then we’ll write the test or at least sketch out the high level test that we want to use. And then quite often, it would be a dev that would actually implement that test. That’s one way of doing it. In other teams, you might have test automation QAs whose sole job is to write these tests. That tends to be less effective because the QA doesn’t have the ability to change things inside of the application to make it easier to test. It also doesn’t really – it divorces the idea of writing the tests – it makes the team feel like there’s the real work and then there’s the test work, which I don’t think is true because your job is to build quality, your job’s not just to build stuff and then hope that it works. You can have separate people doing that, but it doesn’t work as well.

So being the maintainer of a testing tool, one of the questions that I used to get asked quite regularly and probably still get asked quite regularly is, “Hi. I don’t have access to the source code of the application I’m testing. How do I instrument it together under test?” And my response is normally, “Don’t even bother trying to do this. If you can’t talk to your developers enough to get them to insert this testing tool into your framework, then that’s the problem you need to fix before you start talking about test tools.”

That’s a good point. I would say the second approach that having the QA write the test, I’ve been on teams where that’s been used pretty effectively. We we’re on an Agile cycle, where we have two week scrums and we figure out the functionality we wanted to do. We all agree on what it’s supposed to do, the test [inaudible 41:57] is also test. And if you’re used to working with the same people you get into the cadence where you guys know how to talk in each other’s language. [Inaudible 42:05] “If I win, okay I’m going to break your test now.” So you need to update things. I’ve had success doing it with the other way, having QA writing it but I think [crosstalk 42:15] is solid.

One thing that I think is really important is, if you’ve got the point that QA's are writing these tests, you start seeing this thing where the test accumulates over time because someone’s job is to build these tests so they accumulate over time. What’s really critically important if you’re doing that is just set up someone to have the responsibility to also delete those tests over time or merge them into large tests or do something to maintain the size of that test. Because those tests are really – they give you very good high level feedback if something works or not but in terms of an investment, they’re really low Return on Investment compared to lower level tests.

So you need to be really quite focused on keeping that test suite very small in my opinion. I’ve heard some people be as extreme as saying, “No more than ten high level tests per application.” Maybe that’s not a very extreme example but it’s really important to do that because otherwise the test suite becomes fragile and people stop paying attention to the tests. At that point, they have zero value, or they have negative value because you still have to run them and you still have to look at why they failed but they’re not giving you any useful feedback because you’re ignoring the feedback. The test goes red and you say, “Let me rerun them and hopefully they’ll pass this time.”

Yeah, definitely. If your team doesn’t trust your test suite, it’s next to useless.

Yeah, I would argue it’s less than useless. You’re expanding energy to maintain this test suite, your waiting on the test to pass before you do something else. You’re paying the tax of having the test suite but you’re not getting the value, because when the tests say something’s broken you turn around to the test and say, “I don’t believe you, I’m going to run you again.” So where’s the value in that?

Yeah, definitely. So you mentioned brittle test being one factor where UI tests aren’t as valuable or give you a low Return on Investment, what other factors play into that?

The biggest one that you see when you get to test at this level is test data and environment. So usually you’re running these tests integrated so it’s not just the application – the iOS app – that’s in play, but it’s also like some backend services plus the data that’s involved like the user that’s logged in.

Now let’s say you’re testing a ticketing application and you need not just users in the system, you need events in the system and different events of different types and some of them have to be sold out, some of them have to not be sold out in order to reproduce all these scenarios. If you don’t have a really solid mechanism for managing that in one way or another then you’ll see loads and loads of flaky tests. Because the environments down, because someone deployed something to it or the data changed because somebody needed to test some other scenario in the system. So you need a lot of maturity about how to deal with those things. There’s a few different ways to solve those problems, and you need to know that they’re problems and be working on solving them.

That’s a good point. I was also going to add that UI tests can take a while to run, so if you have a bunch of them, and it takes hours to run, you don’t have the feedback that, “Oh, I broke something,” and you’d know right away. If you run a unit test, compile it in a test suite – if you have a ton of tests, it’s still under a minute but I tell you, I want a lot less than that. Generally, it's 10-15 seconds which gives you instant feedback that something is wrong and you can fix it right away. If you have to wait a long time to get the feedback that you broke something, that lessens how useful it is.

Yeah, and it goes back to that Return on Investment thing. A unit test is just a lot more valuable; it gives you more bang for the buck. It will fail in a lot more specific way; it will fail not in a maybe this works, maybe this didn’t. It will fail in like, “I expect 2+2 to equal 4, but 2+2 equals 5”. It’s very unambiguous; you don’t need to run the test again to see whether it’s still is broken. It’ll fail a lot faster and you’ll know where in the code base to look. A unit test gives you rapid feedback, gives you focused feedback and it gives you unambiguous feedback.

All those three are things that you don’t get from a UI level test. Takes ages to run, you don’t really trust it all the time and if it breaks it could be anywhere from like some service you don’t even control, all the way up to the way that you’re rendering stuff in the UI. That’s why I always advocate, why I keep yammering on here about focus way more on unit tests and integration tests than on those UI level tests because you’ll get way more bang for the buck on your lower level test.

So if you want to be focused more on the lower level test, because they have more value, what point do you make the transition into the higher level tests? I have my own answer for this, but I’m curious what you have to say.

So what do you mean; how do you decide whether to do a low level test or a high level test?

So for example, I write unit test mostly because they’re easier to write, they don’t take as long to write, they don’t take as long to run, and it pinpoints where the problem is. I don’t write the higher level tests as often because they take longer to write, they take longer to run. And ultimately they don’t really tell me where the problem is, they just tell me which work flow it’s in. So if you’re writing in the higher level test, it’s usually tap here, tap here, and click here, click here, enter this, into this field, and then do your thing. But sometimes it is worth writing.

One lens to look at this through is what’s the value of trying to achieve of writing these test? With unit tests, I’m writing a test because I want to verify the functionality of my code. I want to verify that when I go to format a user’s name and their last name is longer than 50 characters, I just use the first name. The focus of the unit test – the value that you’re getting from it is verifying the functionality of your system. As you go higher up, the value you’re looking for is not, “Does my app do the right thing?” It’s more, “Do all the pieces plug together correctly?”

So that plays into things like, “Do I test happy path cases versus sad part cases.” So at the unit test level, I want to test all of the ins and outs, all of the nooks and crannies of the functionality. So I want to check what happens when the user name has zero characters, what happens when then user name has five thousand characters – all of these sad path edge cases. On the extreme, on the level of the UI based test, I should be testing the very happy path through the system because if I want to test all the corny cases of that level, I’m going to have a huge amount of tests, and again like you’re paying a lot of tax for each of those tests.

That’s one lens to look at it through is, “Why am I writing this test, what is the value I want out of it? And at the UI level, all you are really testing is, do all of the moving parts tie up together correctly in a very broad sense like am I able to save the passwords, or am I able to validate the password somewhere?

My take on this is it goes back to value. And so what are the most valuable things to test? In other words, if I spend four hours writing some kind of top level acceptance test on things – you tap here, you tap here, you do this, you do this, you do this – is it worth that amount of time, and is it worth the hassle of then having to run these longer running test that automate the UI? And usually what it boils down to is will I lose lots of money if this path doesn’t work.

So if it’s something like in an iOS app, some feature in the app I could conceivably push an update and fix that versus people can’t buy any of my in-app purchases. Well, I’m not making any money until I get that out there. So, I may test one path but not the other based on that. In the web app, it’s typically the payment process and then the couple of critical features that people are going to leave and not pay me anymore money because it doesn’t work. Or it’s going to cause me some legal issue because it doesn’t work.

I think that’s a really good way of thinking about it. It’s essentially Return on Investment.;is this test going to pay for itself?

Yep.

Another thing along the same line in terms of what’s the value I’m getting out of this is just from a regression point of view, it’s nice to have some kind of very loose broad test coverage around parts of the code that don’t touch very often. Because you're not going to – the log in flow, if you break that, yes, critical that you fix it, but also you’re going to find out pretty quickly because no one can log in to your application. Not displaying the legal disclaimers, you probably won’t notice for months because who cares about the legal disclaimer, right? But it is important for you to display them then you should probably have some tests that cover your butt there basically.

Right, then it’s not in depth, “Does it have all these texts?” It’s just, “Is it there?”

I’m perfectly happy to do some happy stuff there. Like say, load this view, or click through to where the legal disclaimer should be then look for a specific phrase somewhere in the UI. That’s going to cover me most of the time. It’s an easy test to write, it’s an easy test to change if he legal disclaimer’s change. It gives you some loose level of safety net that you haven’t screwed something up and not noticed.

Yep, alright. I’m not if I have anything else to add or ask. Do you guys have anything else to get into before we get into the picks?

I don’t.

No, it’s a good show.

Yeah, it’s been great.

I think I’ve got one last thing to add because this comes up a lot as well with UI based tests is someone always says, “Isn’t there some way to just record the user interactions, like record my manual testing process and then just play it back? In that way I don’t have to write these tests?” They answer is yes, it’s possible, but no it’s not a good idea so don’t do it. There are tools out there that you do record and playback, but if you want to talk about fragile tests, imagine a test that doesn’t understand anything about your application at all.

So anytime you change anything in your application all the test breaks and you can’t just go into the code and fix it, you have to rerecord the interaction by hand. These are very tempting path because it feels like, “All I have to do is interact with the application. I don’t need to write any code, I don’t have to have any specialist skills. And they always end up a weight that drags you down rather than something that gives you any benefit.

On the web I have seen it where it captures the idea or class of the element and then attempts to build out the go find this element with this specifier [inaudible 52:58] later, but they’re never perfect. In a lot of cases, I would say around half or more, you wind up having to go in and tweak those scripts, to a point where you may as well have written it yourself because the generated code is

just awful.

And that’s in the web world where these tools are relatively mature. In iOS world, you can guarantee that they’re not going to produce anything that you can maintain in the long run. It would be like trying to design a UI by just taking a Photoshop mock-up and cutting it up into pieces and just slapping it directly into your application. It will get the first version done really quickly but as soon as someone asks for a change, you’re going to have to start from scratch all over again.

Yup. Alright, let’s go ahead and do picks. Jaim, do you want to jump in and do picks?

Alright, I can do some picks. So I’m going to do two picks and they’re probably the same two picks I do every time we talk a lot about TDD. First one’s going to be the John Reid; he’s got a Screencast on testing new controllers. Alondo you talked about how you’re going to start. One thing is just write some test for your controllers, make sure that the views are created correctly. I think we’ve talked about this in the past where that seems a little bit anal to test that way but if you’re code lives on for a while, you do refactoring, you have changes around your nib, and it’s easy to break stuff. So these are valuable tests; I recommend writing them. This is one of the things that help me understand how you can start writing test for iOS.

The second one is – at least the first book I’m not sure if there are other ones but Test Driven iOS Development by Graham Lee, is another thing that really helped me understand how to write unit test in iOS land. It helps you understand things like how do you test a table view? I don’t know, but this book tells you. So Test Driven iOS Development, good book. Those are my picks.

Alright. Alondo, do you have some picks for us?

Yes I do. My first pick is actually and FAQ that just came out on the Rey Wenderlich site, a WatchKit. So if you are interested in learning about WatchKit, what it can do right now, and have some ideas about some new feature that you may want to add to a new or existing app. This is a really good starting point to get an idea of what’s available and what the status of WatchKit is at this point.

Scientific Theories that are Blocking Progress”. It’s basically about a long held conception that we have in Science or Cognitive Psychology. And probably the scientists are speaking to why we should abandon those ideas. I’m finding it really interesting so far, it’s a really cool read. Those are my picks.

Alright. Pete, do you have some picks for us?

My picks are going to be very on topic which is slightly less exciting than you got. My first pick is going to be kind of around the same thing I’ve just been talking about; less UI tests, more unit tests. And so a well-known concept in this area called The Test Pyramid: Martin Fowler. As is often the case, has a very nice article that describes briefly what the test pyramid is and why it’s important. That’s my first pick.

My second pick is a book in a timely enough manner. I actually just finished reading it. It’s a book by Jay Fields called Working Effectively with Unit Tests. This is a really good read from someone who’s been in the trenches doing TDD for probably over a decade. Very good pragmatic guide on how to pull this off. Definitely well worth the read, even though I disagree with some of the stuff he says. And some of the stuff he says are that’s just really awesome. Well worth the read, short book and pretty cheap.

That and my last pick is a beer I’m going to pick Avery Brewery IPA. I had this last night and it tasted pretty nice; pretty good, standard, but not as ridiculous as your West coast IPAs but not as boring as your East coast IPAs.

Alright.

Just kidding about the East coast IPAs, I love the East coast IPAs. [Chuckles]

Alright. I got a couple of picks, I been in tuned to marketing, and socio-political kick lately with my book. So I’m going to pick a couple of books that I’ve read. The first one is the Wizard of Earthsea by Ursula Le Guin. I’m going to pick the fiction book first because that was just a fun read. It’s an older fantasy book but I really enjoyed it.

The second pick is 80/20 Marketing by Perry Marshall, and it was just a tremendous book. I really actually want to go back through the book, probably with a bunch of people as kind of a book club. The other one that’s on my list to do that with is Think & Grow Rich by Napoleon Hill. So I may actually pull something together; I’m still debating on that.

The last pick I have is Miracles and Massacres and it’s a bunch of stories from American history. So if you’re from the US and you want to get some stories from the Revolutionary War, the early days of our country. The story I am in the middle of right now is Thomas Edison fighting the Westinghouse, the company over having DC power transmitted, electricity transmitted into the grid versus AC power being transmitted into the grid. So the people involved are Thomas Edison and Nicola Tesla, and it’s really fascinating.

That book is by Glen Beck and I know that some people have political leanings that make them not like him, but I didn’t really pick up on a lot of politicizing on these stories, it was just mostly interesting stuff that happened. Those are my picks.

I don’t think we have anything else. I would really appreciate it if you went and checked out the Kickstarter campaign that I started if you like this show it’s at devchat.tv/kicktarter. If you’re not into Ruby or Rails you can ignore all the Ruby on Rails stuff and just go look at the rewards, or if you want to support the shows, I would appreciate you pledging anyway. Those are all my picks.

I guess we’re done. We’ll wrap up and we’ll catch on next week.