01:20 - Neal Ford Introduction

02:16 - Efficient Engineering Practices for Software Projects

08:42 - Feature Branching

12:13 - Bad Things About Feature Branching

17:02 - Trunk Based Development

24:10 - Continuous Integration (Cont’d)

26:58 - Hypothesis-Driven Development (Experiment-Driven Development)

32:08 - Pull Requests 

35:32 - Feature Branching (Cont’d)

40:32 - Implementing Feature Toggles

44:47 - Automatic Machine Provisioning

49:15 - Consulting Judo - Michael Nygard

54:28 - Eliminating Pain Points; Automation & Delegation

01:00:33 - The Out-of-town Consultant Effect

Picks

Apple TV (Jaim)
Pebble Time Steel (Chuck)
iOS Remote Conf (Chuck)
LaunchDarkly (Neal)
Grasp (Neal)

 

And we’re live.

Hey everybody and welcome to episode 136 of the iPhreaks Show. This week on our panel we have Jaim Zuber.

Hello, from Minneapolis. I see snow.

I’m Charles Max Wood from Devchat.tv. Yes, it snowed last night up here, too. Kind of crazy. It’s not terribly cold out though so it’s not bad.

[Inaudible] five flakes so we’re not getting too much right now.

Oh. I’ll ship you some. We also have a special guest this week and that is Neal Ford.

Hello. I’m at Atlanta and I’m sweating.

Now you’ve been on the show twice I think, Neal.

I could be, yeah.

This could be the third time. Do you want to quickly introduce yourself though?

Sure. I’m happy to. I’m Neal Ford. My title is director, software architect and meme wrangler at ThoughtWorks which is an international software consultancy. We specialize in really hard software projects. We’re known for being on the cutting edge of interesting stuff in software.

Martin Fowler’s our chief scientist. We’re really into micro services and all the new, cool stuff out of the world.

Nice. Can you get micro services on Amazon Prime?

Don’t know yet. [Chuckles] There’s actually a video. You joke that there is a video but I think it’s on [inaudible] or not that it was on prime yet. [Chuckles]

Free startup ideas from Charles Max Wood.

There we go.

There we go.

Here’s s fresh, new micro service. We don’t know what it does but it got to you in two days.

Alright, we are talking this week about efficient engineering practices for software projects. I’m thinking to myself maybe I can hand wave over this just a little bit more. It’s a rather broad topic. Do you want to give us some idea of what you're thinking here for efficient engineering practices?

Yes. I have a story to tell that will lead me nicely into this topic.

Back in the 60’s and 70’s on big software projects it used to be considered the best practice that everyone will go often work in isolation, work on their part of the system and at the very last part of the project was the integration phase of the project where everybody take all their code and try to get all the work together.

Not surprisingly, that was a pretty horrible part of a lot of projects. If you look at software engineering texts of the 60’s and 70’s, there’s always this big, giant, amorphous, ill-defined integration phase at the end where we try to get everyone to work together.

So I worked on team like that in the 90’s. At that type – the project, I just want a vacation. [Chuckles]

Yeah, exactly. That’s the time to go on vacation because that was no fun.

I was going to say my brothers played football and they had this concept of hell week where they do four workouts a day for two hours each. I think I’d rather do that than try and integrate with a whole bunch of people’s software.

Absolutely. As projects get more and more complicated of course, it just goes up – the complexity

of that goes up exponentially.

Yup.

So in the early 90’s, there were a bunch of software developers that collectively worked on hundreds of software projects, and they were frankly kind of depressed about the state of software development. One of these guys said that whichever software developer process you pick was not a good predictor of project success but you could flip a coin and that was a good predictor of project success. You're in one of these dozen different processes they had floating around.

What they did was say, “Well, we’ve got a lot of collective experience on software projects; why don’t we build software just based on the stuff that we know worked really well in the past. We’ll just go as far the extreme as we can on that.” That was, of course, the extreme programming guys.

So they were the guys who had identified test-driven development because they realized when projects would be really aggressive about testing, we tend to have higher quality code. We use the correlation there, so what’s the most aggressive, possible version? Let’s write test before we write code; we’ll call it test-driven development.

Integration is one of the things that these guys has identified because they noticed that on projects where they were really eager to do integration, a lot fewer integration-based headaches [inaudible] projects where they put it off for a really long time, so they coincide. You have continuous integration; everybody commits the trunk at least once a day.

People who were in traditional projects heard that and thought that was the worst idea they’ve ever heard. Like you said, everybody knows integration is the worst part of the project that you force everybody to do that every day. It makes every day the worst day ever.

Well, it’s not terrible if you're the first person to integrate.

Exactly, but you don’t want to be the last.

That’s true. [Laughter] That is true. Somebody move your cheese unless something else’s there for you.

Exactly. So what the XP guys stumbled on to is this really interesting principle that pervades a lot of the engineering practices in the software world that it seems like a software project, as it gets bigger, the amount of effort that you expend to do things like merges or things like that should get linearly bigger as time goes by as you get literally more code. But the longer you put stuff like that off, it seems like it grows at an exponential rate, so the longer you put those things off, they fester.

I have a theory as to why that is because if you look at the real world in physics, there’s actually a very narrow degrees of freedom that things have in the real world [inaudible] buildings. There are a

lot of options; boards get suddenly joined in a way that you’ve never seen before because they’re a little of the physics. The software components are very soft and squishy, and so they interact with one another at all sorts of unusual, unexpected ways. So the more you allow them to interact in an uncontrolled way, the more pain grows up.

What the XP guys stumbled on to is this idea of continuous integration; let’s integrate early and often, and that ends up solving a bunch of these integration-based text. But people very quickly built machinery to handle this chore for them because it’s such a useful thing on projects so they built continuous integration servers.

ThoughtWorks, the company I work for, built the first and open sourced the very first continuous integration server which was CruiseControl. It was based on pain they were feeling on a specific project they’re working on. They couldn’t get good, consistent builds so they said, “Let’s build an integration machine that handles making the official build.” Now there’s an entire product category around this idea of continuous integration servers.

We don’t even think of continuous integration as a practice so much anymore; we think of it as a tool that you apply on a project to automatically run test and to make sure all your code works together, etc. But I do a lot of work in the Continuous Delivery space.

Continuous delivery is, of course, as its name suggest, about being able to deliver software at a more or less continuous fashion using a bunch of engineering practices. But really, what continuous delivery is about efficient engineering practices for software projects. Because part of what continuous delivery was to do is to turn software more into an engineering discipline which means predictability, which means that we have well-known best practices or cross different kinds of projects. So that’s part of the effort that continuous delivery puts in.

One of the things continuous delivery tries to address is this idea of continuous integration of practice and places where projects have accidently thrown it away.

So one of the things we look for on projects is [inaudible] have the engineering practices that people have taken on started interfering with what their ultimate goals are that they’re trying to achieve. Because a lot of times, you can micro-optimize one thing but overall, it’s hurting the overall workflow in your project because you're too focused on one small thing.

Let me give you an example of that. This is a visual that I sent to you guys and [inaudible] point people to this. It’s a picture on Martin Fowler’s website because he’s written about this topic quite a bit. If you do the search for Martin Fowler and feature branching, it’s his blog and [crosstalk].

Oh! This is the one I want to fight with you about. Go ahead.

Yes, okay. So let’s talk about feature branching. So if you look at – so here’s the problem, and you don’t have to have the visual to picture this because many of you live this problem on a daily basis.

So let’s say that you're using some sort of same version control that handles things like branches, [inaudible] and merges, etc. So you know, some subversion, or a commercial one like Perforce or [inaudible] those guys. You have a main line of development that’s going on. There are a couple of features that may or may not make it into the next release of your software.

One of the ways you can model this problem is this engineering practice called feature branching. Use the branching feature in your version control so that you can have those in-flight features live on branches so they’re not part of your main line. At the end of the project, you can decide will we merge those in or not depending on whatever the business decides about the legitimacy of this feature.

Let’s say that you start this process and you have two in-flight speculative feature; one that’s being managed by Professor Plum and the other – this thing managed by Reverend Green. Now as time goes by, both Professor Plum and Reverend Green know that you should periodically pull from the trunk because we’re living on a branch and things are still happening on the trunk. So they now that periodically, I should pull from trunk and handle any little merge conflicts that come up there.

Let’s say a few weeks go by, maybe a couple of months go by and it’s decided, “You know what, we love Professor Plum’s feature. We want it in the next version of the code base.” So Professor Plum pushes to trunk and performs what we refer to as a merge ambush on Reverend Green [chuckles] because Reverend Green didn’t do anything wrong. He just came in to work on Monday morning; he did SVN up and now slam! He’s got two months’ worth of merge conflicts to deal with from Professor Plum.

Yup.

So he got ambushed by this which is bad because it’s not the only bad thing about this. Because what happens if on the very first day of that branch, Professor Plum started making profound changes to the customer class. On his first day of his branch, Reverend Green started making profound changes of customer class in the opposite direction. They’ve essentially been coding in isolation from one another for a couple of months, and now Reverend Green has a terrible decision to make – do I go back and undo all of my changes to customary and reconcile it with Professor Plum since I’m doing all that work, or this is what I’m terrified of; do I just create my own copy of the customer class that preserves all of my changes, and we’ll let those two things diverge into the future forever on after?

Here’s the problem – and this is finally, after fifteen minutes, gets to the thrust of my topic here today about efficient engineering practices. What we’ve done here is traded one engineering practice for another one. We’ve traded a feature branching for continuous integration. Turns out, it is a bad bargain because not only did we give up continuous integration, we gave up three really valuable things by doing feature branching.

So let’s talk about the three bad things about feature branching then I’ll talk about some alternatives.

Okay.

The first bad thing is the merge ambush, the inevitable big, scary merge that happens when you finally merge your feature branch back into your trunk.

Can I tell a story?

Sure.

I had a client that I was working for last summer. It might’ve even been – no, it wasn’t this summer; it was last summer. Anyway, what happened was I kind of pretty quickly became the pet project’s guy for the CTO that I was working for. What that meant was if he wanted something specific and outside the normal development of thing that went into the core of the project, I got settled with those, which was fine because a lot of more fun.

One of those in particular was statistics on travel information because it was a travel website and he wanted all of these graphs built up based around the data that they were pulling in from all of their sources. Well, the issue was that I was moving ahead on my stuff, but when I was done, I had to merge it back from my branch into the other code. This would happen every week and I wasn’t working for more than 15 hours a week.

So I would get things done and then I’d go merge it, then when I go to merge it, things would have changed out from under me on the master branch so I’d get that merge ambush that you're talking about. Next week, I would go and I would fix the merge issues because I’d merge everything and I’d fix the merge issues, then I would actually go and make some new changes throughout that week. By the time I got done, I’d hit a new merge ambush. After about a month of that, they came to me and said, “Why aren’t you getting anything done?” Because it was never working state on the main branch for them to actually go in and do something with it and see that it was actually working and behaving the way they wanted it to.

This gets exactly the right point; you were doing a lot of work that just wasn’t really efficient.

No, not at all because the work was essentially fix what was already working before because somebody else moved it.

That’s the big first problem with feature branching is the merge ambush and you constantly changing that particularly when you're in a situation like what you were talking about. I’ll give you solutions to that problem, a direct solution to that problem in just a second.

Let’s talk about the other two problems that feature branching induces. The second one is no opportunistic refactoring. So if you're an Agile engineering shock, one of the things you're supposed to do is if you run across something a little flawed in the code, you're just supposed to refactor that and fix it as you move along and leave the world a better place than when you found it.

But if you're living on a feature branch, you get punished for those kind of refactorings. Because if you're on a feature branch then you discover that somebody’s misspelled the customer class, then you go and change that everywhere. You incidentally touched 40 files which is going to make your merge a problem that much worse, so it actively discourages you from doing this thing I would really like to encourage people to do but that discourages people.

In fact if you use one of the ancient horrible tools that still imposes that pre-90’s workflow on you – there’s a tool by IBM called ClearCase that forces every developer on their own branch and then you have to handle the merge stuff at the very end. Nobody that uses ClearCase ever refactors anything because they’re punished so hard for doing the mechanical factoring.

Well, a lot of those renames, too, are handled like they have built-in refactorings in a lot of the IDEs and stuff so it’s mostly automatic. But yeah, then you have that merge issue and all you really did was make one little change with the tool and you don’t even know the impact, because it really doesn’t matter. It’s all incidental; it’s just a name.

Right, but [inaudible] you have to merge it all to get back to [inaudible].

The third thing – the third problem is a problem that you were also running into. Let’s say you’re in the Professor Plum/Reverend Green world and the big boss comes to you one week and says, “Okay, I want to see a version of the product – just the trunk, just Plum, just Green,” and both Plum and Green go. Now you have to handle three merge conflicts because you have trunk, you have to merge in Plum, you have to merge in Green and to merge in both of them and get that all to work.

If you come back a week later and he wants to see the same thing, you have to do that exact same thing over and over again.

So let’s talk about the way to solve this problem. The fundamental problem we have here was we made a bad trade. When we trade away continuous integration for feature branching, let’s not do feature branching. So let’s talk about a whole set of engineering practices that have become more and more popular called trunk based development. You can do a Google search and [crosstalk].

No!

No? [Chuckles] You can do a Google search on trunk based development. There’s a lot of writings that’s been done about this. Facebook very famously does this. In fact, a lot of big places do trunk based development. In fact, trunk based development in conjunction with – let me talk about one of the things that we do in trunk based development that solves exactly the problem that you were just having and this is the idea of feature toggles or feature flags.

Ah, there we go. See, that is one of the issues that we’re talking about because ultimately, my issue with trunk based development – and to be perfectly honest, this is mostly what I do these days because it’s myself and maybe one other developer working on the same code base, not necessarily at the same time even.

Feature branches is more overhead than it’s really worth for us but – and they’re my projects so I can do whatever I want there.

The flipside there is that the issue that I have is yeah, you get anything experimental in there that’s going to break the page or break the build or make people’s stuff not work the way that is should and it breaks everything for everybody everywhere.

But that’s part of the magic trick of feature toggles is you don’t leave something toggled on that’s broken yet.

Okay.

Part of the show – let’s talk about feature toggles. The idea behind the feature toggle is you have a configuration file that goes along your application that basically says this feature is on or off. Now, every time your application needs to do something, it checks to see ‘should I show this versus that?’

For example, the user interface. You might have an if condition that says ‘if this feature’s on, show this this or do I show this other thing’. On the server side, you have a conditional that says if this feature’s turned on and execute this workflow. Otherwise, execute this other workflow.’

So let’s talk about feature toggles in the Reverend Green/Professor Plum world that we’re just living in; in fact the world that you were just living in in the story that we were just talking about. Let’s say rather than doing a feature branch, they have both the unfeatured toggles of Reverend Green featured toggle and Professor Plum feature toggle. The other one always leaves the other guy’s feature toggle turned off so as long as their code compiles, then everything’s cool.

Professor Plum never has to interact with Reverend Green’s code because it’s always behind conditionals that they never execute. As long as everything compiles and all the unit test passed, everything’s cool. But on the very first day, they both touched the customer’s class; they get a merge conflict because they’re still living on trunk. So continuous integration is back and forth.

So I will argue that feature toggles and trunk based development help you find those conflicts as early as possible.

Now I’m partially playing devil’s advocate here but two things come to mind here. One is, doesn’t this make your code a little bit more complicated because now you have these conditionals built in there that are just turn this on, turn this off?

It is. So here’s our perspective on feature toggles. Feature toggles are purposeful technical debt to support engineering practices by continuous delivery.

One of the rules of feature toggle is as soon as you figure out yes or no, you strip out all the other conditionals. What we do on our projects is anytime we place story card that has a feature toggle on it, we also insert a downstream customer card that says ‘remove this feature toggle’.

We don’t know exactly when that’s going to be played but it is now in a set of playable story cards now. As soon as we – as eagerly as we can, we’re going to strip that out. This is purposeful, temporary technical debt that we’ve added to support a bunch of other beneficial things. [Crosstalk]

I want to restate how very important it is to remove this feature toggles because I’ve worked with code bases that have had 20 years of feature toggles in them. [Crosstalk] So when I first heard about – not feature branching but developing this way, I just threw up a little bit in my mouth because I lived through that where people did that and did not clean up the [inaudible] themselves.

That’s the classic blunder years – putting them in there and then getting too much of a hurry to take them out. That’s a bad thing.

There is a certain level of engineering maturity which required to do this well, but that’s part of what continuous delivery is really trying just to promote is this idea that, look, we’re never going to be respected engineering professionals if every time we say something’s hard, we don’t do it. This is one of our pet peeves about software is people say, “Well, that’s hard so we don’t want to do it.” You know what else is hard? Open heart surgery is hard. [Chuckles]

If doctors were like software people, we wouldn’t have open heart surgery now because it’s hard. So it does require a certain level of engineering maturity to be able to put it in and also take them out.

You said classic blunders and my brain went straight to Princess Bride, “You fell for one of the classic blunders! The first well-known is they’ll put in feature toggle then let you plan to take them out!”

That’s exactly right. That’s the first blunder of feature toggles.

The other thing that is related to this – and this is the other thing that came through my mind is that in some cases, you're going to have to put feature toggles into your data models and into your controllers and into your views. This is an iOS show and you have all of these pieces to your app; I could see that you could wind up with a few up to a dozen or so feature toggles and placed in different places to make sure that these things don’t turn on and impact things until you're ready for them, too. So you're doing that; how do you keep track of all of that?

That’s another part of the idea of continuous delivery is that the projects that are doing this that have an appointment pipeline which is like the continuous integration server on steroids that will – as soon as you make a change to something, we’ll pick up and run that against all the possible effective ecosystems to make sure it runs such a test against it and make sure you haven’t broken stuff.

A couple of other things about feature toggles – don’t create feature toggles that depend on each other because that’s the road to insanity. [Chuckles]

The other thing is you don’t want a bunch of these at a time. This is like three, four or five at a time, not 20 or 30 at a time because they are hard to keep track of and the last thing you want to do is not understand the code base.

There are also places that have taxonomies of future toggles so operations will have one set of feature toggles they manage; application development are another completely different set of feature toggles that manage. But you're right; in a lot of cases, because you have very distributed applications now, is that coordination across all these guys about what things are toggled on versus what things you're toggle off.

One of my former colleagues, Paul Hammond has actually written a piece of open source software that you store configurations on a Git repo, but when the application starts up, it checks that Git repo to see what kind of configuration it should take on. So everything is always consistent because you have a single server truth, or what should be turned on versus turned off and you just manages that stuff with tags.

That makes sense. I want to go back to continuous integration just for a minute.

Sure.

I think most people, when I talk to them about continuous integration, usually it’s on the web so there really isn’t a build step in the same way there is for iOS though I could definitely see that as part of continuous integration in iOS.

Essentially, when we talk about continuous integration, it’s code for ‘we have something that automatically runs our test. Is there more to it than that?

Well, really what continuous integration is supposed to be is the official canonical build. This is the official; if you ever get – well it works one way on my machine versus another machine, the only one that it counts on is continuous integration server.

The idea was that’s the pristine environment that doesn’t have email and browser bookmarks and other junk on it. It just has stuff that should be part of the software so that it would be an official canonical build. But then you get into deployment pipelines and continuous delivery and that actually is a whole series of steps that you could put your code through. So it acts like continuous integration in that it triggers upon change, but then after it runs a bunch of unit test, it will promote that and try to build out some of these Puppet and [inaudible] to build out a realistic environments that run test over those, and try to build more and more sophisticated, gated process where every gate is more sophisticated stuff you're doing to your code to get it live.

That’s more common in these continuous delivery environments is to have not just a continuous integration server but a deployment pipeline. It manages a lot of these coordination around abnegated change here; let’s make sure [inaudible] if you had a continuous deployment environment, you want to make sure as you make changes that you're not breaking stuff that’s writing in production; it’s just going to go live. It would not work to production. I want some good reassurances that I’m not breaking the world.

Right. The other thing that I want to put out there is just that what we’ve been talking about with efficient engineering practices is mostly just not causing other people pain. But in my experience, to be the most efficient engineering team, it really boils down to communication. I’m wondering, do these practices tie in to communication some way other than just tell people this is turned off, or am I missing something there? Or are those other practices that we can talk about in a minute?

There’s a lot of practices around that we can certainly use for that. You see, a lot of our projects, we just assume that your work is going to be done under a future toggle, so it’s like a common thing to see what are the work strings that going on that are being dead end feature toggles. It makes it much easier to do things like continuous deployment because you can just roll stuff out.

Because nobody’s interacting; people are not seeing that feature yet because it’s still toggled off.

So you do feature toggles and then trunk based element?

Yes. Yup.

Let’s talk about once you switched over to that mindset – and it really is a mind shift to start thinking that way about doing that. Let’s talk about some super powers that this gives you because one of the nice things about adopting some of these engineering practices like this is suddenly you can deliver new super powers to your business, like AB testing now is very easy.

Right, because you can just turn the test on and off.

In fact, most of the feature toggle frameworks will allow you to do AB testing so that you can do a gradual release to your users so that 10% of your user gets the feature toggle on, butt 90% get it toggled off so that you can do experimental development and in fact support what we’re talking more about is this idea of hypothesis driven development.

You guys are all familiar with the Agile world and the Agile way of thinking about things. One of the things that’s always bothered me about Agile is what are you supposed to do when you start an Agile project? You get a whole bunch of people in a room and they all create the big, giant backlog. You prioritize the backlog and you start hacking away at it.

Oh, you should see my shirt. [Chuckles] Let me see if I can turn on my video. It says the backlog is a lie.

Yup. Actually that’s – [chuckles].

It has cakes scratched out. So if you’re familiar with portal, yeah.

Yup.

Yeah. Anyway.

So what is that backlog generation but a big, giant design upfront exercise?

Yeah.

Once you have things like feature toggles in place, you can actually start approaching things differently and this is this idea of hypothesis driven development. Rather than create this big, giant backlog, create an environment that makes it really easy to do experiments with your users. Instead of this big brainstorming exercise, what you're doing is coming up with hypothesis, do you think users will like this more or this? Put it out live and find out what they actually like better. This allows you to experiment with things and if people don’t like it, you can just shoot it in the head and move on to something else, but if they really like it, you can double down on it.

Or if they like it, a slightly different version could pivot and change what you're doing, but having an environment like this with feature toggles where you can do gradual releases or do AB testing releases gives you the ability that you probably didn’t have before from when interacting from a user’s standpoint.

The other thing is – and this is the thing that I like. I’ve also heard it called the experiment driven development.

Right.

Effectively, the idea is yeah, it’s then not ‘well, we’re going to see if this feature makes sense’ because that whole hypothesis doesn’t make any sense. We’re going to wave our hands at it and we’re going to build a future because we want it. That’s what that means.

Instead, what we’re talking about is we’re saying, “Okay, I need to increase user engagement by at least five percent in order to meet these other targets in the business.” So my theory is that if I add this feature, then the people who are using – who have that feature are going to use the application 5% more.

Yup. That’s a good example of one of those. Let me give you another one. This is actually a great story that came out probably eight months ago on the awesome Radiolab Podcast. It was about Facebook and it was about – I can’t remember the year, I think it was 2013, between Christmas Day and New Year’s Day. People posted more photos to Facebook than all of Instagram. They had a problem because they have this thing in Facebook where you can flag photos as being offensive, and over a million of those photos were flagged as offensive. They went back and started looking at them and it was like pictures of puppies.

Oh, I hate puppies.

And these things that were really – did not seem to think of it all said they couldn’t figure out why. The big problem from a business endpoint they had, they have committed to sorting through all these pictures to make sure they weren’t offensive and removing an offensive one. They just literally did not have enough staff to handle the volume of pictures they were getting.

But Facebook has a really good experimental setup for big [inaudible] deploy stuff and so they started asking more nuance question about the picture. They just didn’t say flag it as offensive, they started asking questions why. They found out that puppies were offensive because they have been lost in a divorce and people want to be reminded about that. So they kept asking more and more nuance questions and what they do is put up several different versions of the questionnaire, and the one that gave the better results, they doubled down on that and kept doing that.

Before long, they had it down to a few thousand pictures per cycle and they could easily hire their staff to handle that many truly offensive pictures. But without having this platform for experimentation, it was taking them a really long time why people were flagging these things that they didn’t believe were offensive as being offensive. That’s the real advantage of having an environment that allows you to do this kind of experimentation; you can engage in a way that you’ve never even been engaged before.

I like it.

One of the things that I’ve – I’ve been working on a team that’s been doing feature branching. It’s been going pretty well; we keep the features that we want to do the way [inaudible] merge, pretty small. That’s the practice that we’ve developed. But one thing I do like is the concept of a pull request. You have a request, people look at it; how do you replicate that when you're committing to trunk?

Pull request is basically a voluntary ambush. The pull request says, “I’m ready for you to ambush me now,” versus being ambushed by surprise. So it’s still merged so you're still dealing with merge conflicts there. The same thing is true to the feature branch. The longer that code has been at sync with one another, the worse the merge conflict’s going to be. So I actually do feature toggles as an alternative to pull request. Pull request are really just – is different in what I do in feature branching in some ways.

Okay. That’s part of how you get to the merge but the [inaudible] is that people are looking at it. So how do you replicate that in the trunk based development?

Well in trunk based development, the code’s all there and just under feature toggles.

Okay. It’s there but as a framework to get the other people on your team understand what you're doing – check for bugs, doing a code review.

There’s nothing specific when you build in the feature toggles to [inaudible] all that code reviews or to [inaudible] that. You just have to do that as part of – will you decide to turn this feature on like you already [inaudible] a code review you can deal with at that time. That’s an advantage that pull request have; it’s a nice way to change where you could see the entire change set at one thing.

You don’t get that unified view in the feature toggle kind of world.

Okay.

So I’ve been travelling the world, trying to talk people out of doing feature branching. [Chuckles]

How’s that working for you?

It’s actually working pretty well once you have a chance to talk to people about it because I have this theory that I’ve developed overtime. This theory is that meta work is more interesting than work; that in a lot of places, the way they use version control is way more about meta work than it is work because here’s the observation.

I see a lot of developers and that – when you really boil the job down to its assets, everybody’s going to Christmas soon so they’re going to be talking with their grandparents – when you boil it down, take things from websites and put them in databases, take some databases and make them appear on websites, that’s not all that terribly exciting if you’ve done that for 20 or 30 times. You know what developers do when their day job is boring? They invent cool puzzles to solve so that they have a really meaty, cool puzzle to solve when they come in to work every day.

That’s the way I see version control being used more often [inaudible] a way to make things work more efficiently. It’s the cool puzzle that we’ve built that we all have to interact with on a daily basis. I’m not really about making things better but kind of as an end unto itself. That’s the thing I tell people to be careful about is make sure that all your engineering practices are pulling in the same direction, that you're not overly optimizing one thing and it’s really killing your productivity somewhere else because you're spending a giant amount of time doing merges over and over again or something like that. Try to look at the overall [inaudible] of how efficient you are from an engineering standpoint. Don’t try to micro-optimize; try to macro-optimize.

That makes sense. So I’d like to comment about the AB testing because the team I was working with has been doing an AB testing in a feature branch environment.

We developed a lot of the same practices that you’ve talked about by putting the code in there, make sure it’s tested. You do the AB test and you could run tests in both cases but we also advocate for – okay, we need to clean this up when the test runs and that was a huge thing we had to advocate for.

Yup.

I forgot my question. [Chuckles]

I think you were going to say that feature toggles are awesome and you're never going to do feature branching ever again. [Chuckles]

Well, I still enjoy the feature branch workflow. I’m finding it – you can definitely reduce a lot of the down size by keeping your feature branches small if it takes [crosstalk].

If I were going to do feature branching, really short, little feature branching we’re not so mad at is the problem of getting really long, little feature branching because basically, the way you think about this is the entire time you're living on the feature branch, you disable continuous integration. That’s just something that’s not happening anymore. The longer you live in that world, the more painful it’s going to be to rejoin that world.

So what if you have your CI server running on the feature branch and you’ve just merged or rebased it?

You can do that and just deal – but you’re going to hit the same merge conflicts over and over again. Whereas if you're doing – the beautiful thing about feature toggles is you merge the code at the earliest possible time we could merge it which is the first time we get a conflict between those two. Any other practice is going to cause you more pain than doing a feature toggle.

Yeah. One thing that my continuous integration system uses – I’ve been using CircleCI – is that it’s pretty good about running those tests because it’s web so there’s no real build step. It’s pretty good about running those tests whenever I commit to a separate branch. It’ll say, “This is the master branch and then this is the other thing branch.”

Yup. In a deployment pipeline, you can set up a whole suite of test and other things you run on each branch. [Crosstalk]

But I still have to manually merge it, is that what you're saying?

Exactly. You eventually have to pay that price to merge it. Whereas if you're doing feature toggles, you merge it as eagerly as you possibly can.

Right.

That’s really the thing about feature toggles that ultimately – it’s not the perfect – I’m advocating for it here – it’s not the perfect solution to every problem. It certainly does add complexity to your code which is why we encourage you to take it out as quickly as possible but it just allows some of the other nice things to happen like optimistic refactoring comes a lot easier.

We could argue that it causes the least amount of merge headaches you're ever going to have on a project if you use that because you hit merge conflicts as eagerly as possible.

Right.

Yeah. I’ve done trunk based development for four years and I’ve enjoyed it. It’s been as fast as possible and get that feedback right away multiple times a day and that blows people’s mind. But it’s an effective way to work even though most of my team based stuff is doing feature branches.

Trunk based stuff definitely works well.

I do a lot of these continuous delivery workshops all over the place. Essentially, I’m going to bring this topic up because most of the reaction I get from this is either, “Oh my gosh, it’s the craziest thing I’ve ever heard. I can’t believe anybody would do that,” or, “Why are you talking about something that’s so old. We’ve been doing it for a decade now.” [Chuckles] It seems like there’s very little in between there.

You look at the company like Facebook – the Radiolab episode is really fascinating because he talks about doing – it’s really about experimenting on a platform like Facebook and they are running up to 20 experiments at any given time at different regions for different purposes. The interviewer asked them, “What’s the chances – as [inaudible] Facebook user – what’s the chance that I’ve been in one of your experiments?” And they said, “Oh absolutely, 100%. You’re probably on three of them right now. [Inaudible] has it.” Because that’s the way big scale, mature code bases work is they do – they don’t try to do branches and try to manage 20 different branches and what’s going on. They added feature toggles where everything is already integrated and just turn on toggles for different purposes.

It was a good example of how Facebook [inaudible] out chat when they decided that Facebook users chat with one another. They did a continuous delivery practice called dark launching where they dark launch all the code and all their servers underneath the feature toggle. They toggled on that feature per region so they could look at the scale problems. The hit on New Zealand first and then Australia which was the first big-scale test. It basically followed [inaudible] across the world until you got the east coast and the US, figured out that, yup, we’re going to go ahead and turn on to US so they toggled on all US regions.

Over the course of 24 hours, they rolled out this new feature very slowly over time and monitored continuously to make sure it was wound up over [inaudible] the structure. That’s a good example of modelling DevOps plus engineering practices working together.

So a bit of a question about how to implement feature toggles, especially unit testing. Because when I encountered feature toggles way back when it was in C++ with if [inaudible] and it was a nightmare. I ran away screaming. I will never do that again. What are some ways that we can get this code tested? [Crosstalk]

I was going to ask this, too.

Okay, that’s a good question. So if you're in a C++ world, those are build time feature toggles. So [inaudible] which means you're literally compiling different versions of the code base. If you're in a more modern environment like Java for example, there are a bunch of feature toggle frameworks for Java that are one time feature toggles which means you can actually get a console or toggle them on and off as it’s running. This is the thing that allows you to do that partial releases and other things like that because there’s a front server that sits in front of your application that just decides that they get this toggled on or off and you just interact with that server [crosstalk].

On this show our listeners are in the dark ages as far as any of that.

So build time feature toggles are with the ifs and the [inaudible] and things like that. The trick there is you basically need a deployment pipelines pro-feature because you're building a different version of a code base every time. If you build at the end, it’ll be more dynamic.

For example, when your application starts up, read a text file that has named value pairs in it and then decides to dynamically turn those things on and off, that’s actually an easier environment than unit testing is because your unit test can also ready your feature toggle file and decide should I try to test this or not based on when that toggle is turned on or not.

That’s trickier in the build time feature toggle world because then, you basically have to have two unit tests – one for the feature on and one for the feature off.

So what you're saying is that you [inaudible] your test where you can essentially say ‘run the test with variations on these different sets of toggles’. So it just runs your test suite four times essentially with the two that you have in there, on and off, on and off.

Yup. You could set it up to do that, particularly run time feature toggles. You just run the test and then tell the framework ‘turn this feature on now that we’re on this other test’. So run time feature toggles are a lot easier to do with a [inaudible] stand point.

Definitely. I’ve been doing similar code with feature toggles implementing AB tests and we ended up doing things where we do a lot of it in code because in the iOS world, we really don’t have a lot of frameworks we don’t have the maturity of the Java world. But we’ve been doing a lot of things where we decided AB tests are first-class citizens of our code to we code classes that are actually [crosstalk] doing AB test.

The beautiful thing about feature toggles though is as long as you can read external configuration, as long as you have an end state, you're good. O it’s actually quite easy to build a [inaudible] simple feature toggle framework instantly like in iOS. The really common thing of course [inaudible] an iOS app it’s talking to a backend that’s basically JSON [inaudible]. You could certainly make that backend deliver a feature toggle information to the front end if you want it to.

Definitely. Yeah, I think there are a number of frameworks that do similar things like that.

Yup, absolutely. It’s not hard to implement at all. That’s one of the nice things about this is it’s very easy to get working and [inaudible] a platform. There’s a relatively ancient web integration platform called webMethods that I recently got feature toggles working on because it can read external configuration and it had enough statements. The barrier to enter is really low.

So in a new test, you have separate test that just don’t run based on the feature flags so you have feature [inaudible] on the test and the code.

Yup, exactly. There’s a little more of a testing [inaudible] so again, this is another reason that you don’t want a lot of these at any given time. You [inaudible] the toggle away chokes the code not individual lines of code all over the place.

Uh-hm.

You could have one takeaway from the show. [Crosstalk] feature toggles. [Chuckles]

Yes, that’s it. Hygiene is important. One of the things we said a continuous delivery [inaudible] is you must be this tall before you start doing this thing and that’s one of those things that you have to be able to have enough discipline to clean up, things like old feature toggles or your code will get messier and messier because you're just imposing more and more technical data.

Have I lived through that mess? Yes. [Chuckles]

Yup.

What are some of the other things that we can do to add efficiency to our engineering process besides trunk based development and feature toggles?

There is certainly a lot of stuff in the continuous delivery world about automatic machine provisioning. So that’s one of the things that would try to do a continuous delivery world is try to get your code to a realistic production-like environment as quickly as possible. A lot of times, that means running on simulators and things like that but trying to get it in this production-like world as possible, as quickly as you can so you have less things you have to fight about as you go forward.

Another thing that we talked about in this world is – another thing that I think projects don’t do well is trying to target the engineering practices round tests toward the target audience that benefits from them.

Let’s project or look at unit tests versus functional tests. We’ll make this distinction in the continuous delivery world. Unit tests are really about structure and getting the structures correct, like you have a foundation for [inaudible] refactor code and other kinds of code hygiene to have.

Functional tests in the continuous delivery world are the ones that capture things like requirements, so there are several functional testing libraries like Cucumber, behavior driven development frameworks that people use to capture requirements versus capturing scaffolding. So when you look at those, suddenly you have those two kinds of tasks; all those small projects that we often have, more levels of testing, user acceptance testing, etc. Let’s just look at those two as a good example.

Unit test are important because they are part of the developer’s normal workflow. So the normal workflow for a developer is you check some code in and you make sure all the unit test pass before you start working on the next thing. What that means is that the target audience for unit test are developers and the best thing you can do for them is to make those unit tests really fast. So speed is more important even than comprehensiveness.

One of the things that we’ll do very often in unit test is for unit test, never reach out of process. Don’t go reach for advice on data bases; don’t go after web implements, etc. Mock instead everything inward so that we’ll get really super-fast unit test because that’s really beneficial to the audience that consumes those tests.

On the same project, we said functional test never mock or stab anything. They get real values because we’re trying to verify requirements. We’re willing to make a much longer time hit on functional tests and get the much deeper level of verification than we were on unit test. So that’s an example of honing the engineering practices around things like tests toward the audience and how they benefit on it.

I was on a project not too long ago. For example, it was a Ruby on Rails project and unit test ran at about 41 seconds and the functional test ran at about five minutes. We did exactly that strategy because now, developers have the fastest feedback loop that it can possibly have on being able to get to work again. But then because we’re harming the verification there, we’re beefing it up on the backend. That’s an example of honing the engineering effort toward which audience benefits from it.

We actually do that a lot for testing in the continuous delivery world is to look at who’s consuming these tests and how can we optimize that to make it the most beneficial for that audience.

Another good example of this is we’re really big fans of being able to automatically provision environment using Puppet or Chef or Ansible or one of the DevOps tools. A really common practice is to make it so that if you have some manual that’s [inaudible] function that – one of the big problems you have in a lot of companies is [inaudible]. So it’s okay, we’re ready for the next build that we need to verify. And then come the developers and developers saying, “Okay, we’ll get you a new build. We have to eat a developer, a live chicken and a voodoo doll just to be two hours.” Now I’ll promise I’ll get you [inaudible] if you can run your [inaudible] stuff on.

Have you been working on the same teams that I have? [Chuckles] The voodoo doll is critical.

That’s right. It won’t work otherwise. We want a provision environment automatically so that you could click a button somewhere and you have an environment split up so people get to work on it.

This is actually a really good example of this thing that [inaudible] who’s really well-known in the DevOps space refers to as a consulting judo. Judo, as a martial art, uses your opponent’s weight against them. So Mike found himself as the de facto CTO of this big box retailer. He was trying to convince them that they should invest more effort on these DevOps practices and it was just falling on deaf ears.

What Mike did was found a pain point. The pain point of this particular organization was they never had enough QA environments. People always fighting over them, people try to share them and then they get conflicts with each other. There’s all this messiness around QA environments, so he use that as a way to convince the powers [inaudible] to let him automate the creation of QA environments.

That showed them that A, it was way easier than they thought it was going to be. All of a sudden, all these noise and grief that was happening just melted away and because anyway can get a QA environment just by pressing a button. This is a really good example of this [inaudible] that I talked about in the continuous delivery world.

This is particularly true. We have developers who are trying to convince operations people they need to change the way they do stuff or they’re going to your QA people or your DBAs and try to let you know about how to do their job better. You should stop doing that. My mantra is demonstration trumps discussion. Don’t talk about it; you need to show them how their world’s going to be a better place.

Mike [inaudible] walked through that place until he’s blue in the face trying to explain to people how the world will be a better place if you could just speed up environments more easily, but until he had the opportunity to demonstrate that to them, he didn’t have any attraction. But as soon as he showed them how it worked, everybody got on board right away.

So that was demonstration trumps – [crosstalk].

Discussion.

Discussion. There we go. Meme Wrangler in action right here.

There you go. [Chuckles]

I really like it though. There a couple of – that’s why skunk works, right?

Yup. Absolutely.

So there were so many places that I worked at where it was – we need a continuous integration server and we need this, that and the other. We want to be able to refactor these couple of different things. It was always, “No. We’re not going to pay for the time that it’s going to take to do that.” “Why should we fork over so much money for continuous integration system? Or why should we fork over for this tool or that tool when you seem to be doing your work just fine the way you are.”

But we can actually go in and say, “Hey look, we set this up on our own and it’s given us this kind of increase in speed, efficiency, productivity. It got this and this and this problem out of our hair and now we can move ahead twice as fast.” They we’re like, “Oh, we’ll pay for that.”

Absolutely. So continuous delivery is one of those things where operations, people and DBAs and QA people are saying that this is just an example of developers wanting to steal their jobs but that’s not true at all. The problem is their job has started to suck so much, it’s starting to make my job suck and that’s [inaudible]. So I’m going to have to fix the things that are broken about your job so that it doesn’t end up breaking my job. That’s where demonstration trumps discussion comes in because it’s really hard to talk people out of – I’ve been doing it the same way for a decade to show them how this is a much better place and it’s much easier to change their mind.

Absolutely. Especially if it’s a pain point that you both have shared from one end to the other. For example, I worked at a company where the deployment to production, or staging even, was super painful. So when we found ways to short circuit some of the things that the operations guys had to do, for example, packaging up gems – because security was a big deal when we were dealing with government data. So there were all these rules around what we could and couldn’t do even with our test data.

So they couldn’t just gem install something on the server. They couldn’t just use Bundler to put it up there; they actually had to basically put it into an RPM package and then push it up to the server and then have it installed that way. So by creating processes that made some of those things easier, it made our lives easier but it also made their lives easier. And it wasn’t that we were trying to save – steal their job, and they never felt that way; it’s just we’re tired of begging for this and we know you're busy. It totally came through and it worked out nicely that way.

The beautiful thing about automating something like that is this is this constant pain you feel from doing that manually over and over. Once you automate that way, that pain just evaporates. Now all of a sudden, that time that you spent doing that manual thing over and over, you’re freed up to do something else.

Yup.

That’s one of the things that we really aggressively look for in projects. In fact, that’s where the continuous delivery book came from. It was written by two ThoughtWorkers and we’re really adamant about looking around in projects and thinking about what could be – how could we automate that way, what can make that more efficient. That just frees up more and more time to do the interesting stuff of the job, not the mundane parts of the job.

So ThoughtWorks – I believe you folks were [inaudible] rather large projects. But I also want to just throw out there that a lot of the projects that I work on are comparatively much smaller. We’re talking about two or three developers, we’re talking about maybe a few hundred thousand lines of code. It’s not this giant project; it’s a smaller project and things are pretty modular so that we can pull things apart and put them back together when we need to.

Even then, a lot of these practices – now I’ll admit, we still do some feature branching but it’s mostly trunk based development. We haven’t done the feature toggles and definitely looking forward to trying that. But some of these other things – with the way that we approach our continuous integration and the way that we are now looking at continuous deployment – it just takes these pain points away. Usually, they are the pain points where I’m the bottleneck because I’ve held on to control of those things. So my developers and other folks I’m working with informed me ‘you don’t have to worry about that stuff’.

For me, what it’s really about is some of the pain is going to be the way that I interact with a code versus another developer interacts with a code, but a lot of it really is the way that I do things and whether or not I’m willing to give up control. Therefore, I need to automate that away so that it’s not as much of an issue.

Yup. And small projects are a slippery slope because every project starts out small.

Yes. [Chuckles]

But at some point you ask, “What happen if you're doing this over and over?” I wrote a book a long time ago called The Productive Programmer. One of my mantras in the productive programmer was ‘every time you find yourself doing something for the third time, it’s time to automate it’ because if you do it mainly three times, you're going to do it a thousand times.

Yeah. I heard an interview on the Eventual Millionaire with Jaime Tardy, and this is a business podcast. She interviews millionaires and talks to them about basically what makes them millionaires.

I think it’s a million dollars.

Yeah, but it’s how did you get there and how can the audience of eventual millionaires get to be millionaires. She had a productivity guy on there and he jumped in and he basically said that you should be willing to spend up to 30 times – and I’ve said this on other shows – 30 times the amount of time that takes you to do a job once to either automate it or delegate it and to get somebody else to do it. The reason is because if it’s something you do every week in a year, you’ll have done it 50 times. And if you spent 30 times the amount of time to show or teach somebody else to do it or automate it, then you saved yourself – you’ve netted another 20 times whatever that time is.

So if it’s a half hour, that’s ten hours that you save that year and next year, it’s fifty hours. If you do it every day or every work day – let’s say there are roughly 200 work days or 20 work days in a year, it’s that much more, because then it’s 220 times the amount of time this year and 250 times next year.

So if you're doing it over and over again, it makes a ton of sense to look for ways to automate it because eventually, you're going to be more efficient because it’s just a push of a button now instead of running a script and babysitting it.

And [inaudible] to that point, I was on a project before where the unit test took 20 minutes to one and we moved that down to 15 minutes. So every developer – we’ve got 10 developers – every developer checks in ten times a day, you save a huge chunk of time now just by that little tweak. Seems insignificant but you start adding it up all the time, it becomes a huge amount of time and money.

The other thing is David Brady from the Ruby Rogues podcast has said over and over again that his threshold for running tests is his Twitter threshold. In other words, how long can I sit here and watch these test run before I’m over on Twitter. Then we’re not just saving the other minute and a half that you spent running the test, but the other ten minutes it takes for him to goof off on Twitter and then come back and get back to work.

Exactly. Way back in the day, I use to teach classes on C++. I notice this guy – C++ compile times were brutal – then there’s this guy; every time we tick off compile, he was doing something really intricate. I realized, he was doing these little origami animals. So this is [inaudible] mechanism to being a C++ coder. [Chuckles] The origami kicked off the compiler; he build a little Godzilla zoo of origami animals there to keep himself occupied.

He must’ve been an origami master.

He was by that time.

Yeah. There was an xkcd that I’ve seen co-opted this couple of different things where two guys were standing on their office chairs, fighting swords and of course, it’s stick figures and stuff because it’s super high quality. But anyway, the boss walks by and is like, “What are you doing?” And they’re like, “Waiting for the build,” or “the one I saw for Ruby was bundle installed. Well that was slow.” And yeah, you're totally off in Neverland.

It’s easy to get there.

Oh yeah.

You’ve got [inaudible] runs a really long time. It’s easy to get really distracted and wander away.

Yeah, and it’s funny, too, because I hear so many entrepreneurs or sometimes programmers that say ‘I swear I have ADD’, and that’s not necessarily the case. I’m sure some people – you could diagnose some. I sometimes wonder if I have it. My son has ADHD but what that really boils down to is I have so many things going on that all it takes is one notification to pop up on my computer when I have some downtime and I’m gone.

Yup. [Chuckles] That’s a good motivation; turn all notifications off on your computer.

Yeah.

One of my pieces of advice in The Productive Programmer.

Yeah. That’s another thing I have to – I have a beef with you now, Neal, because I was going to write a book called The Productive Programmer and now I have to change title.

Yup. I beat you to that one. [Chuckles]

Now I have to get creative.

Yeah.

Alright. Well, we’ve been talking here for about an hour. Are there any critical things that we haven’t talked over that we should go over? I’m sure there are a million productivity tips we could give, but is there maybe one more that is just ‘if you're not doing this, it will change your life’ kind of thing?

One of the things that I encourage everybody to do? I’m a consultant, and there’s this phenomenon that we recognize called the out-of-town consulting effect which is if you rode on an airplane to get to a meeting, you automatically have more credibility than the people who rode in cars to get to that meeting.

I can vouch for that.

But there is a little bit of truth to that because – there’s a famous story about boiling frogs – that was in the Pragmatic Programmer book. Apparently, if you want to boil a frog, you can’t throw them in a pot of boiling water because they will jump out, but if you put him in a pot of cold water and turn it up slowly, he’ll cook without realizing it. Turns out that’s not true because frogs are not that stupid. [Chuckles]

Big companies are that stupid. So as the out-of-town consultant, you can come in and go, “You realize all you frogs are boiling here,” and they go, “Oh, yeah. We haven’t noticed that because they weren’t looking at it with an objective eye.”

So my best piece of advice for everyone listening to this is show up at work tomorrow and act like an out-of-town consultant. Look at all the new processes you're doing with an objective eye and say, “Why are we doing that? Is that the best way to do that? Could we just remove that step entirely or is there a much better way to do that?”

The continuous delivery book has a great suggestion exercise which is to go look at – there’s this thing called a value stream map which is just a fancy way of saying lines and boxes where you delineate or link the times of things. What they say is go look at your core deployment process from the time that you start working on code until it’s available to your users. Delineate each step that your code goes through and ask yourself ‘what could I do to make that way more efficient and shortening the amount of time it spends on all these engineering practices.

If you look at that with an objective eye, you could probably find a whole bunch of low hanging fruit that you could either change or just get rid of entirely or change the order of to greatly enhance the productivity of your engineering team.

Yeah, that’s interesting. I can also say that when I moved over – that last full-time job that I had, I was just talking about it. When I got there, I basically came in and said, “We used continuous integration with Jenkins when I was at the other place,” and the kind of talked about it but they haven’t implemented it.

So just me coming in and saying ‘I’ve done this before’ and ‘I think it would really benefit us here and here’s why’; just having that perspective as new person, not even that I had flown in, it made a huge difference because all of a sudden, it was a different perspective where somebody actually had some proof that something would pay off.

I’m pretty sure that I’ve seen this work out even with junior people in a new position on a team of senior people. They come in and they start making suggestions and backing it up with their experience and you can have a tremendous effect just by coming in and being that set of fresh eyes. So don’t just count on it just from the standpoint of ‘oh, I’ve been paid big bucks to show up here’ or from the standpoint of ‘I’ve been doing this forever and so I know the right ways to do it’. If you're new but you know something works that they’re not doing, see if you can get them to try it.

Absolutely. We tried to take suggestions from all corners at ThoughtWorks. There’s no role within ThoughtWorks that feels like they’re not allowed to make a suggestion to make things better.

It’s really hard to foster that but I’ll tell you, it pays off big time.

Yup. The first time I was a team leader of a project and a junior developer changed one of the lines of code that I checked in the day before, I first bristled at it but then I looked at it, realized he’d make an improvement so I shut up and got over my day.

Yup. Alright, well before we get to picks, what are the best ways for people to follow up with what you're working on these days, Neal?

I’m doing a bunch of stuff around software architecture and how it intersects with continuous delivery. The Continuous Deliver book is certainly where a lot of the stuff comes from.

My website nealford.com is where all my upcoming appearances and most recent videos and books and things show up. I just started working on a book about evolutionary software architecture so I’m spending a lot of time working on that. More to come about that in the near future.

Alright. Well, let’s go ahead and get to picks. Jaim, do you have some picks for us?

Yeah. I’ve got one pick. It’s kind of an obvious pick. One of the people who were listening on the show got an Apple TV for free when they sent them out. I wasn’t on that list but I did buy one when they were released. Went to the store and picked one up like an animal. [Chuckles] I’m enjoying it so we use it for streaming Netflix and I’ve got the NHL package so I can watch hockey everyday which is fantastic, except for Christmas for some reason but anyway.

Is that a real sport?

It is. At some places, the water freezes and they can skate on it with [inaudible] called ice skates. It does exist. Apparently water doesn’t freeze in Minnesota anymore, [chuckles] not this year. No, I’m liking it. I previously used an Xbox for basically watching Netflix and NHL and that kind of stuff and it was really a [inaudible] interface. I’m glad to not have to pay Xbox live so I’m pretty happy with my Apple TV.

So if you haven’t – if you didn’t got one for free from Apple and you do like to use your TV for stuff, I recommend Apple TV. I’m pretty happy with it.

Very nice. I got a couple of picks. I don’t know if I picked this last week or not so I’m just going to mention it.

My Pebble Time Steel. So I have that – we did an episode with – I think it was with Neal where we talked about the information on the watch and all that stuff.

So yeah, we definitely had you on a couple of times because I think we also had you on for the tech radar. Anyway, it now has health tracking data which was the one thing that I was like, “Man, I

really wish that I had health-tracking data. I wish I had an Apple watch instead of a Pebble Time.” But now I have that on my watch.

Now I’m sure it’s not the same as it is on the Apple watch but it also costs $400 less. So if you’re looking at smart watches, I am now even more delighted with my Pebble Time Steel. Battery life is a good week between charges. Anyway, it’s awesome.

When I moved my arm too quickly in the movie theatre though – I went and saw Star Wars yesterday – it does light up but anyway, super happy with it. Definitely going to pick that.

I also just want to put out there – for this show, I am doing an iOS Remote Conf next year in April. So if you're in the iOS development, which you probably are if you listen to this show, the call for proposal is open until some time in March. I’m probably going to move that up because people want to see the talks earlier and I think it helps get more people excited about the conference. If you want to speak, yeah, definitely get that in by the end of February, early March. The early bird tickets are on sale as well. They’re $100 a piece; the price doubles when early bird ends so you definitely want to be getting those tickets now as well.

You can also go to allremoteconfs.com and get a package of tickets. If you see some of the other ones in there like Git or robots or things like that, you can get a package of tickets and then you can use that to get into multiple conferences. Anyway, go check that out. The full list of conferences is at allremoteconfs.com. Those are my picks.

Neal, what are your picks?

So I’ve got a couple. I was talking about feature flags, that kind of stuff here. So I will point you to a website called launchdarkly.com which is feature toggles as a service so you can actually have these guys deploy your app and they’ll manage feature flags for you. It’s pretty slick. I haven’t tried it yet, just discovered his but it looks really slick if you're into that feature toggle world.

The other really slick thing that I found recently – this is a JavaScript thing but it is so cool.

Graspjs.com – it’s a command line utility that lets you search and replace refactorings in your JavaScript code but it lets you do really funky placeholders and that kind of stuff so you can do really amazing structural refactoring with JavaScript.

The most powerful refactoring JavaScript code I’ve seen and it’s a command line tool. Very slick. It’s called Graspjs.com.

Nice. Alright, I think we’ve covered as much as we’re going to cover here. Thanks for coming again, Neal.

My pleasure.

We’ll go ahead and wrap up this show. We’ll catch you all next week.