The panelists discuss handling client launches.

 

Hey guys! Anything new?

No?

Reporting live from the bunker –.

Hey everybody and welcome to episode 124 of the Freelancers’ Show. This week on our panel we have Eric Davis.

Hello.

Curtis McHale.

Ola!

Reuven Lerner.

Hi, everyone!

I'm Charles Max Wood from DevChat.tv, and this week we’re going to be talking about client launch development, or launching clients’ apps once we have done them. It’s kind of interesting; we were talking a little bit about what we meant by launched development earlier, but is there something that you do when your client is getting ready to launch something that you built?

Well I book a dance party sometimes first, depending on the client, right?

Yeah, strobe lights, colored organ.

[Laughs] It depends too on was it a good client? Do you want to work with them again, or was this a project you finished where ‘it wasn’t the right client for me’ or they were just a terrible client? I do different things for each style, outside of actually turning the features on.

So if it’s a good client, you make it so they can’t deploy on their own so they call you up, right?

Never.

Oh my God. Starting even many years ago, I would just be shocked that there were people who really did that. If you have a client who wanted – I would sometimes come to these people as a new client and they would say, “Oh well, we were kind of happy with the people we were using, except for this one little thing – anytime we wanted to change anything on our website, we had to go through them. We only wanted to change one word, and it becomes a change request, and we had to go through them.” I just was shocked that people would put up with this, but they just didn’t know better.

Yeah, I've seen that where it was language in the contract and I've seen it where it was just technical expertise are lacked thereof [crosstalk]. Come on, don’t be a douchebag, okay?

I never thought [inaudible] the contract, oh my God.

Actually, for the employment process, I do set up something with git branches and I use Beanstalk to deploy that many of my clients can’t handle on their own, so I do that on my own. But they have their own FTP – it’s their site; they can do whatever they want with it. And then I'm building on WordPress so they can also do whatever they want with the admin of WordPress, right?

Yeah. I’ll talk through a situation I'm in right now specifically with deploying and stuff. I have a client that is using Instructure Canvas and they needed it updated, and so I spent six hours rebasing crap [inaudible] in their custom setup, and then I went and set it up on a VPS so that they had kind of a staging setup that they could see before they actually launched, which is actually relevant, I guess, to this conversation, having a staging setup so they can verify changes. Anyway, I tried to walk my client through managing the deployment through git because that’s more or less how Canvas works; that’s how they tell you to set it up in the first place. When you have changes, you just do a git pull on your git repository and they copy the files into place, but with his situation, he has a custom branch in git. If you're not a technically person, essentially what it is is within the repository, within the place where we have all the code, there's a separate branch or fork that has the differences in the code between the canonical version and his version. Anyway, so I have to pull that down and then copy the files over. And then there are some other stuff that have to be run in order to get everything setup properly, and so I'm trying to walk him through this process and it’s all command-line stuff, and he’s trying to figure out how to do all this stuff from his Windows machine and shell into the server and stuff. It’s just hard, and so I'm trying to think of a better way to make it so that he can just push a button and deploy it.

That’s the thing: there's a difference between letting your client have access to everything, and then the client actually being able to do it.

Yeah.

With my clients, I make sure they have access to everything, so if I literally get hit by a bus or if I'm gone, they can do stuff that might be hard and they might have to install stuff, or maybe even hire someone who knows the technical aspects to do it, but they have all the tools there for them to use – they just might not know which end of the wrench to pick up.

Right.

Yeah, and I do the same thing. I give my clients access to Beanstalk, because that’s where all their code would be, so if I got hit by a bus they could run with it, or get someone else to run with it. Chuck has asked me in the background, Beanstalk is like a git repository, but it also does deployments so you can set it up on a branch to automatically send stuff over to your servers, and then you can roll back that way. WordPress doesn’t do well with stuff like automated deployment like Capistrano; Beanstalk does a lot of that for you. They also have a service that works with GitHub and Bitbucket as well that’s all just the deployment portion. I don’t remember what it’s called, though, but they have the same service if you don’t want to use their git stuff as well, if you're on Bitbucket or GitHub.

Yeah, that makes sense. I'm looking at writing them a deploy script of some kind – Capistrano or something – so that essentially, worst case scenario, they have to get into the server and then run it, so that’s one thing. Do you guys usually setup staging servers before they do the actual launch?

In 90% of cases, yes. There's the odd project that’s super tiny that isn’t going to crash the site if something’s not quite right, say, in admin-only feature, that just helps their workflow then we may not if it’s small enough. But if I'm sending over a thousand dollars with a time on it, then there's a staging server involved as well.

Right.

I mean, you definitely want to have a staging server at some point. This is one of those lessons that I've learned the hard way in some way, where I would say, “Oh, it’s such a small site. Do we really need it?” but eventually, you're going to want to show them what's going on or test the next version of things, and so not having a staging server at some point is just going to hurt.

Yeah, and I develop locally even on those little ones. I develop it all locally, and I've usually actually screencasted how it works for them so they can see it once, and they can say, “Oh, this isn’t going to work because of these three things,” or “That looks great,” which is usually what they say when it’s small enough. “That looks great, let’s get this on the site now.”

I've actually gone a different route. I used to do staging stuff, like I'd have a completely identical server and deploy stuff up there. I’d have my machine, I'd have a staging one, and then I'd have the actual production deployment, so there are at least three copies. Recently, I've been getting away from the staging; for the apps and stuff that I'm working on, it’s so much work, so much setup to do, and the benefits – they're just not worth it anymore. Most of the work I'm doing now is very fast and I can get it into production and people are using and they’ll know within five minutes, ten minutes, an hour that stuff’s not right and we can roll it back. That happens because I built up a lot of the automation around it and my clients are very quick with the feedback and getting back to me on things. There are some tools out there where you can kind of hide things, so new things go on to the server live, but regular users can’t get to it. You can select your power users or maybe your clients’ admin accounts that they can use and test it out on the live server, and I found that works pretty good. A lot of those are called feature flippers or stuff like that. I just found, for staging, the apps that I worked on in the past, it was a whole another process to manage the staging, and it felt like I was pushing code up there and then the clients were like, “Okay, we reviewed it a little bit, but it might work a little bit different in production,” and the value just wasn’t there. One client I had, they were using one of the top two URL hosts, and I don’t know how many hundreds of dollars they're spending a month just on their staging servers. It was insane.

Yeah, I think you have a valid point there where if you had quick feedback and not major changes, then the way that you're talking about doing things definitely makes sense. In this case, they hadn’t updated the app for a year and a half, and so by rolling those changes in, it was a fairly major upgrade, and so I felt like I needed the staging server.

Yeah, and I think you have to be flexible with that. I've done some development where I'm actually doing all the development work on a remote server, like it works as my desktop, I guess, and there have been times where a client is back and forth on a UI or how a UX flow works, and what they'd actually do is I would set them up so they would connect to my development machine and see kind of a demo of the code. I would be editing it live underneath them so they could compare how it works completely, and so by the end of that session, they would be like, “Oh yeah, we got it perfect now. Let’s lock this in and then move on.” It was kind of a staging server in a way that the client was poking at it, but it wasn’t an official one. It was like a temporary, ‘for this time, we’re going to use it.’ That works good, and being able to do that when you need it was nice because it was more of a higher risk thing than just standard bug fixes or something.

I think that having a staging server is one of those things that’s important to educate clients about, to show them that it’ll improve the reliability of their software, it’ll improve their ability – as Eric just said – to be involved in the process of developing it as well, and they're just going to get a better product out of it. But not everyone, and in fact I would say most of the companies, especially the small ones that I deal with, don’t think about this at the beginning. It’s part of the process of talking to them and helping them figure out how is this going to become a reliable, integral part of their business.

Yeah, I guess that’s a way to look at it. I see a staging server as a tool, and a tool is used to increase collaboration, especially on the QA and confirming if this works. If the company is able to collaborate without that, you don’t have to pay the cost of having it. But some companies – I know some, like I would force them to have a staging server because they would never look at it until it got in production, and it would sit there for weeks before they discover that, “Oh, our database isn’t saving anything” or something really bad. That’s because there's no strong collaborative relationship there.

Yeah, how do you build that kind of strong collaborative relationship?

Slowly.

[Chuckles] Carefully.

I mean, it depends on the client, I guess. You know going into how technical they are, how collaborative they can feel, and I think a lot of it is just [inaudible] you're building trust with the client. Instead of you disappearing, doing your work and coming to them with the entire piece, you come to them for a little bit, get their input on it, take their input and actually use it, maybe even change it around a little bit and bring it back to them. They’ll feel like, “Oh, this guy knows me; he knows what I really want and he’s able to tweak it and he adds his own experiences” and so that trust builds up and that’s the foundation for collaboration. It also depends on the client; some clients, I know them outside of the project and they're great at collaborating but on a specific project, they might not be good because it’s not a high-priority project or they're pulled in another direction, and so you don’t have the time to put in to really collaborate.

One thing that I'm really curious about – and this is something that I've seen off and on with some of my clients, and by off and on I mean either it comes up or sometimes it just doesn’t work out – is that in some cases, I'm a tech guy but I have more experience in launches than they do. And so what happens is they start heading down the road of getting to launch their product, and I'm trying to give them suggestions, “Do these things: build the mailing list, whatever” and they kind of just see me as the tech guy, and so they're not so interested in my opinion even though I feel like I could help out in that camp too. Is there a way to make that better? Because some folks get it, and it’s like, “Oh, that’s a good idea. I’ll do that” and other folks are just like, “Okay, whatever.”

I think that steps way back to how you position yourself at the beginning, right? Are you coming in as just the guy who programs, or are you coming as the business consultant who solves problems who happens to also write code?

If they hire you as the business consultant who writes some code too, then they're hiring to listen to you if they're hiring you as the guy who writes the code they told you to write and make some technical decisions then they want you to write your code.

I'm working on a project now; we’ve just finished. The first one was actually of a product and I was sort of insulated from the company. This guy knows the CTO – he’s actually the CTO in the contract, but basically, I talk to him; I never talk to the company at all, and he talks to them on my behalf. Now it looks like he’s probably going to be stepping aside because they hired a full-time CTO and then I’ll start talking to them, and then I think the relationship will change because they’ll start to understand that I have more to offer than just coding. But for now, I've been presented as “the coder” and that's all they think of me as, and that’s a little frustrating to me. It’s also frustrating to me that the current CTO keeps saying to me, “Oh, you're so lucky you're not talking to them, because then you're involved in all the product development stuff and so the politics.” I keep thinking, “Actually, I like being involved in that, and I think it’s useful for me to be involved in that.”

Positioning is important with that.

So on the day of the launch, and then again I know we’re talking a lot of the type of work that we do, but I think there are some parallels and correlations to other industries. If there is a problem when you launch, are you basically on-call or do you just tell them to plan ahead and use a staging server? How do you handle that?

It depends on the launch: what kind of launch it is and how big. If it’s a big, public, [inaudible] our business is starting on this day, then you're pretty much going to be on-call or close to being oncall. I've had days where I would take it off from business and just go to email and administrative stuff so I could jump on the phone or jump on chat or whatever with them in case problems came up. There are other launches, like if it’s an internal launch where it’s just in the company; you can do certain things, you can roll it out differently; if you're just launching not a full website but [inaudible] features on it, launches are typically not that big of a deal. You might have actually launched the changes on a Tuesday, but it’s actually promoted to the users till next week, so it gives you time to get used to it, make sure everything’s good, so there's a lot there. In some launches, if it’s internal tools or just improvements to existing systems, I don’t even consider them as launches; it’s just a release like, “Hey, here’s an update.”

Yeah, when I've already tested four or five times. Usually, I actually spin up a quick, secondary server to make sure all the launched stuff is automated. I've done it twice locally, and I know that the launch is essentially me pushing the code and everything else is automated past that.

I would say, don’t launch on a Friday or even on a Thursday unless you want to run the risk of working on it over the weekend.

I never launch Fridays. I actually had one client who said, “We always launch Fridays” and I said, “You need to find someone else.” I'm not launching Friday ever. That is only a recipe for me to work all weekend.

That depends, because I actually found for internal apps, launching on Friday is nice because it’s just the select users who are working over the weekend to test it out before the big rush when everyone’s on the workweek. It depends on who’s using it. I think it should be never launched when you have a high level of activity on the site whatever the product is.

Yeah, okay. That’s fair. [Crosstalk]

Yeah, the companies that I worked for, we usually did our launches in the evening and it was usually in the middle of the week. The reason was that we would have fewer users overnight, and so we could monitor it, and then if we were comfortable with it, we had our automated systems that would report errors and things like that – I guess we can talk about those in a minute – but we would be made aware if there was a major problem. We just watch it for a little bit with diminished traffic, and if we wound up being up all night fixing it, then we just didn’t have to come in Thursday morning. It’s like, “Hey everybody, go sleep; we’re going to buy you all something nice” kind of thing.

[inaudible] where you spent the night under your cube Thursday and went home Thursday night.

Yeah. That’s definitely a thing that I've done, and usually it’s overnight with a workday the next day so that we can make up the time at home, sleeping or whatever we need to do to make up for the fact that we were there till 2 AM because something went wrong. [Crosstalk]

I often end up launching at the end of my day. Most of my clients are in Eastern Time, so if I launch at 3 o’clock, it’s already 6 o’clock their time so it’s in their evening time. A lot of the stuff I'm working on, it’s a business – a business-centered site, so most of their users are not on it at 6 PM Eastern Time.

Yeah, I definitely take advantage of the 7-hour difference between Israel and Eastern US to launch things in a softer way and try it out before too many people get on there. All of these seem to agree that launches will have bugs; they will have hiccups. Again, I think it’s an important thing to talk to clients about, that they shouldn’t expect, “Oh, we’re launching. You're done testing all your software; you're done with all these testing? Great! It’ll be super smooth.” It might be super smooth, but it would be wise for us to assume that there would be problems and have contingency plans.

Yeah.

So here’s a question though, Reuven? I dealt with a client today and they said, “How can you guarantee that our launch will go smoothly? Because this other agency did.” [Laughter] Chuck, that’s what I started with. I actually laughed out loud.

I can see the future!

That’s what I said. I laughed and then actually said, “Well, they're lying to you.”

[Laughing] Sorry.

I might couch it in slightly more diplomatic terms. [Laughter]

[Crosstalk] I maybe have said, “They're lying to you” straight up.

Right, right. Even though that is an accurate way to describe it, but I think I would say, “I don’t think I would ever make such a claim because software is inherently complex and buggy, and until it actually launches, we can’t guarantee anything. But basically, they're lying.” [Laughs]

I mean, here’s how you make that claim – and this is what I was talking about earlier – is you launch the software weeks earlier and you’ve been testing it live so your launch is just marketing that, “Hey, we have new stuff that you might have not seen yet.” That’s how you do it; you get the risky stuff out there, you get a select amount of people using it, get it working good, and then you bring in the 90% of the rest of the user base. I found that works really good for the stuff I've worked on where there's a large amount of users.

I like that, and I can buy that for a totally new product. But if you're doing upgrading, it gets a little trickier.

Yeah. There are technical challenges, depending on how stuff works. It can get hard, but if you look at it that way – I look at it as, “What are the risky things? What are the things that we can isolate in a way that it’s not going to affect the rest of the system, that we can launch earlier and get them out there?” It’s all about reducing the total risk, and that might be by spreading it out over a longer term than just one second that you hit the button.

Yeah, I do definitely like that approach. I know that GitHub and Etsy and some of the other ones out there, they push the new features out with a flag on that says, “Show this to a certain percentage of users” or “Show this to users that have this flag set or meet this other criteria.”

Yeah, the feature flags [inaudible]. I've done it with clients where basically everything changed. It was a web app, but the entire web app design changed: the domain name changed, the underlying architecture did a major shift, and we did that by using different domain names. My client and the developer team and some of the power users – we all set up custom DNS routes, we would go to the website we wanted to and it would point to a different server. We were able to actually really test it, do some kind of - not live, but the users would use it and get back to us really quickly, like feedback sessions – and we got most of the major stuff out. We actually got to test, like, “Okay, let’s roll up this big update” multiple times because we were able to just completely wipe out that new server and start from scratch. We had confidence in our process, we had confidence in the code that the bugs that were going to come up were going to be minor or we weren’t going to be able to find them without having a lot of people using it.

Have you guys tried things like Vagrant or Docker for releasing in a more stable way? Or even Chef or Puppet where you don’t reuse servers – you just roll out a new server each time?

I use Puppet; for most of this stuff I have used Puppet to do the automation. I'm looking at using Docker for this. Docker and Vagrant are interesting because they're – just the way they work. They might be good, but you have to be careful. I think it requires discipline on the part of the team, but yeah, that’s what I mean. There's tools, like you can get really deep into technical stuff to make it completely automated and the continuous deployment idea where you make a change and then ten seconds later it’s actually on the production servers, like how Etsy has it set up or whatever. But that introduces some risk and you just have to make sure you're client’s comfortable with that. There are some projects that – it doesn’t matter, like it’s an internal app that you can just SSH in and rsync it or just copy it across FTP and that’s good enough for a deployment.

Yup.

I think one thing that helped – I haven't updated it, but I actually have in my company wiki, I have a page like “When doing a side launch, here are things to check.” It’s like a checklist of “Is DNS setup how it’s supposed to be, especially if it’s moving stuff around? Is the DNS TTL turned down so you can move quickly? Is your RSS feed setup? Are there Cron jobs set up? Is there a privacy policy? Are you going to be doing advertising? Are your landing pages setup? Is the support email working? Do you have all the launch marketing stuff if you're going to do a blog?” It’s basically a checklist of any launch I can go through with any client and say, “Yes, this is done. No, this isn’t done. or “No, it doesn’t matter for what we’re launching here” and I give a copy of this to my client so they can see what you're talking about earlier, Chuck, “I'm not just a coder; I know analytics stuff, I know marketing, I know there's more to it than just putting code on a server.”

Yup. I want to go back to the Chef discussion just to answer Reuven’s question in the way that I handle things. In a lot of cases I have a basic setup for mostly Rails and Rails-related stuff, and so I just add the recipes in and yeah, I can spin up a new server and then customize it for the client. I

can also add recipes or settings or different things into Chef to make it work, but I found that in a lot of cases, beyond the basic recipe, it’s actually not worth the effort to do it, so I just plug in Nginx Passenger or whatever and then just run with it. But it is nice to be able to spin up a server, a brand new server, and just try it out.

Yeah, and like I said awhile back, I was working – my desktop environment was like a remote VPS server. I would have clients come in and demo stuff, and that was set up through Puppet. There were times where, as a developer, stuff  kinda just get sideways and things would get installed weird or whatever, and normally you'd have to debug it, figure out what's wrong, find out you’ve installed one thing you shouldn’t have and it screwed everything up. In that case, I actually just deleted the entire server and started over, because the time to actually debug and figure out what happened to my development environment was less than the time if I would just build a new server and start from scratch. Because I had to run the application, it was in a way kind of this deployment of “Oh, this one server screwed up; I'm going to throw it away and start it over.” It’s the same idea, but it was used in the development sense instead of a live site sense.

I'm curious – as part of the prep for launching or talking to people that are launching, I don’t think I've done this very much. Have you ever talked to them about what sort of bugs people might encounter, or technical support, or what sorts of issues they should expect?

Yeah.

Yeah, I mean it depends on the changes. If you're doing a lot of backend changes, you're going to have to talk to them about there might be a down-time as the database gets upgraded, or they might be data issues that [inaudible] come to support of what we lost –. It looks like the end-user lost data, but it’s there, just not showing up. If it’s like design changes or whatever, then you kind of bring up – like your support’s going to be burdened by people not being able to click in the same place as they used to; they can't find those menu items they're looking for. We at least talk about it so they're prepared for it; we don’t actually do any kind of hard, physical documentation or instruction manual of version 2.1 rollout or whatever.

Yeah, I was just thinking – this project that I'm working on now where one of the questions was, since we’re using Rails, so there are migrations to change in the database, their question was, “Wait a second, isn’t it possible there's going to be some point in time when the application will be upgraded and the database won’t be migrated to the new version?” I said, “Yeah, it’s possible. But it’s probably not a big worry, and probably it’ll be fine. He said, “Okay. That’s fine, as long as I know.” But you just have to talk to people [inaudible] about someone might encounter this sort of issue.

Yeah. I tend not to go and say, “Here’s where the issues may be.” I will do that if I know something’s not completely baked, and so I'd just give them an indication, but for the most part it’s, “Look, I've done everything I can to make sure that there's not going to be a problem and if there is one, here’s what you do. Here’s how you find me.”

Yeah, you just set downtime expectations, right? But normally it’s like, “Yeah, you might have 60 seconds where there's a little bit of down-time or our user might get a random issue. Past that, it’ll all be good.”

Coming from a system administrator background, I’ll usually ask depending on the app, like for 30 minutes or 60 minutes of downtime – larger if it’s big moves. The system might be down or a minute at the most while stuff is getting changed, but I’ll tell them that during those times, expect it to be unreliable, expect stuff to happen, because I might have to deploy, find a bug, have to fix it, redeploy, or as Reuven was saying, there are database changes so the code might be there but the data’s still getting upgraded or a whole number of things. But I try to use a window of ‘here’s the time I need to get this stuff out there’ and I do a smoke test, make sure stuff works, kind of the last test before I tell people. When I'm done, I'm like, “Hey, it’s out there; it’s done” or if I run out of window, which has happened once or twice, I tell them, “Hey, I had to back it out there; there's too many things. We need to go back into development to fix these.”

So do you guys use any kind of error reporting, send something to your pager, whatever kind of thing?

Homing pigeons and monkeys.

In that order?

[Laughs] You’re assuming we have errors.

Yeah, like, really.

I know.

Eric, get with the program. Now we just use flying monkeys; we’ve merged the two.

I don’t have anything that’s emergency like that, but I do build into most of my work a logging routine so if we get failure cases that are kind of unbelievable, like if we miss everything else, then it logs everything that happened in the system at the time and I don’t have those email me or anything. I would have to go check them.

And I think it depends on the app and everything. Curtis, you're doing WordPress stuff which you have to build that yourself, versus most of the other ones here, we’re doing Rails which comes with logging and there's half a dozen or a dozen different automated error-reporting, software as a service subscriptions. We can give people our credit card and basically get a lot of that for free. But I think it depends on what the app is – what are the needs, what are the possible failure points, and if you can’t fix it and work around it, how can you at least be notified when a failure happens.

Yeah. I use a pre-built class for WordPress that allows you to enter log entries really quick, so I basically just wrap an if statement and throw in my logs. There are not automated systems that I'm aware of to capture the WordPress logs.

[inaudible] calls me or pages me or SMSs me or anything, but I've [inaudible] addition to log files, so New Relic is pretty good about that stuff. You can go in, take a look and see how many errors you’ve got or how bad things are, and so when something first launches for the first two hours, I’ll be a little obsessive with checking that, checking the errors and checking the server in a bunch of different ways to make sure that nothing’s going horribly wrong.

I'm laughing off-mic right here because my phone has gone off three times during the show with pager duty telling me I have server problems. It’s not my problem, and it’s not really a big server, but on the other end, I have the system admin level infrastructure set up – and this is my own, personal server. This isn’t even client stuff, but I have stuff on this server, like all the logs get sent to a centralized logger that will notify me if it sees a thousand attempts to SSH in at once or something. I have Nagios, which connects all of my servers, all of my services and checks, like [inaudible] use is good, are there too many users logged in, are our processes crashing, are the database servers – everything I could think of – and then I have a system that checks Nagios, so I have someone checking the checking system. I have pager duty that if Nagios says something’s wrong, I get emails, phone calls, SMS messages, and it continues until I say it’s fixed. If Nagios sees, “Oh, you fixed it but it’s still broken” it starts the cycle over again. This is my own, personal stuff; I set this kind of thing for one client, and then I actually hooked a second client up into my system while we were launching just so that they had an extra security blanket, but I've done this. There's New Relic, which will watch some things; Honeybadger, I use for error tracking, so if there's a bunch of errors on a site, the whole team gets emailed. You can have stuff going to team chats, stuff like that. It depends on what you're launching and how close you want to be to it. In the PaaS, in the Rails community, there used to be a plugin that would email you every error, so if you launch and you have a thousand users all getting the same error you just got a thousand emails. That used to be the best thing to do and I actually still use it on some projects that don’t have a high volume. But you got to fit your monitoring tools with what you want to monitor.

Yup.

Yeah, I know Beanstalk does automatically send me, if there's a problem with the deploy that it has, then it will email me that there's a problem with the deploy and it actually rolls itself back at that point too, to the last stable [inaudible].

Yeah, I use a lot of the same tools that Eric mentioned – Honeybadger and New Relic. I'm not so keen on Nagios just because I'd have to set it up and maintain it myself, but a lot of these other tools are nice and they’ll notify you via email or however you want to do it in order to keep that information straight.

Yeah, and the Nagios – it’s because it’s a pain to get set up, but once it’s set up, adding new checks or systems or services to it is very easy, and so for my stuff I have it set up and then I plug my stuff in. Like I said, for my client, their app was very similar to the other things I already had in there, so 98% of their configuration was just copying the file across and then putting it in a server that I know already works. I told them, “Yeah, it’s in my Nagios. It will call my cellphone is something goes down, so you don’t have to worry about it as much” and then I would forward emails off to them or call them, help them wake up their team.

Mm-hm. So when you set this stuff up, do you set it up so that it’ll notify the client if you set this kind of stuff up for the client?

It depends on the type of build

Yeah, I follow what we said earlier.

Right. It depends on how technical they are.

I give them access, like I actually hook their stuff to the Nagios thing#. I make sure my client gets New Relic; I make sure they sign up for Honeybadger, all that, so they own the account and they let me in, so they have access to get notified if them want. Some of them will get the first few notifications to see how it works, but then they have the control to turn that off if it’s annoying them or it’s not relevant to them.

I had the opposite problem. I had one client where I set it up to send them some error reports, error messages, so they would get email every time there's a problem on the site. But it wasn’t just problems in the site; it was – we had some sort of a PHP script that tried to break into the site, and it encountered a URL that on the Rails side, it’s not going to do anything; it’s just going to give you an error of some sort. So they would then get the reported error and email me and say, “Oh my God, what's going on with the site?” and I have to explain to them, “Okay, those 10 URLs are not something to worry about.” At a certain point, I feel like [inaudible] the little boy who cried wolf, because they were so worried for so much the time that in the end it turned out to be a big nothing, so now they never ask me about any of the errors.

One thing I did about that was I was getting all the errors from my client because they didn’t want them, and I would put that into our bug tracker, and I'd have the full error. I'd put it in there and then I'd explain [inaudible] “Okay, this is an error because of x, y, z; it only happens when there's a blue moon, and mercury is in retrograde or whatever.” I would track also how many times it occurs, based on [inaudible] they could see when it updates, they could look at it and it’s like, “Okay, this is something that never occurs. It might be kind of annoying, but it’s not hurting the data; we’re just going to ignore it” versus something like “Every user’s getting this error” or “They can’t use the app.” It’s just like real bugs – they can actually prioritize them, put budget into it, we can estimate it and it goes through the whole process like any other development feature. And I found that was great because it’s transparent, like they could see the exception, they could see how often it was occurring, see the impact and they can make the business decision of what to do with it.

That’s a great idea.

So Eric posted a subtopic, aborting a launch.

Yeah, this is something I've had to do a few times, but it’s also something I see a lot of when I work with other teams. They don’t think about it ahead of time, but having the process – even something simple or a script or any way to back out of a launch that goes bad –. I've had launches go bad because the code was bad; it wasn’t tested correctly. I've had it go bad because there's some weird network connection I added, like an OS provider, and so we can’t download the things we need to install or just things that are completely out of our control. The problem happens when there's no process to abort a launch or to back out of it, and you're kind of stuck with this halfway, not functional system. I think that something that everyone needs to think about, at least have a short discussion and figure out if we do need to abort, what do we do.

Most of the time for my WordPress projects, there's no data migration that’s really happening before I've automated that portion. It’s the code, so I really just deactivate a plugin, usually, not gets us back to where we were before.

Yeah, I've had this happen as an employee working on teams, and yeah, we made it pretty easy to back stuff back out. I don’t have a whole lot to say on the topic, because I haven't had to abort any releases for any of my clients. Occasionally, there's a bug but I usually wind up fixing the bug as opposed to aborting the release.

Yeah, and that might be one thing. I talked about how I have a deployment window: if the problems are minor, it might be better just to do another bug fix supplement# right away than to roll it back and start over. That’s a conversation I have with my clients; if I can’t fix it, I think my window is actually – I'm not going to be able to do it at all. This is what I'm going to do: I'm going to roll it back and then we’ll reschedule for another time or whatever.

Yeah, I find the bugs that do crop up are usually ultimately fairly benign. No matter what the client thinks, they're fairly benign.

Yeah, the big ones – I've said [inaudible] we were in the process of deploying and the host went down. Something went down on their network’s storage, so you could log in but anytime you ran a command or whatever, the command line would hang for minutes on end, and obviously when you're running a deployment script and putting in a couple thousand files on the server, that doesn’t quite work. Then I've had problems like that, or the database update didn’t go smoothly so you have to get a backup – it’s these weird things that you thought you tested but just something, like a spare electron jumped a piece of RAM and all of a sudden everything crashes.

Yup. Any other aspects of client launches that we should talk about before we wrap up?

Well what about all the soft stuff? We talked a little bit about it. Like, once I tie up a launch – what

do you do after you tied up a launch, it’s all good, the client’s happy?

It depends on the contract. Most of mine – a launch is just a small event in the contract, like I continue on doing the next development or maintenance or whatever. But if your contract is basically to launch the product, launch is done, your contracts over – that’s a different circumstance. You have to figure out closing out the contract, or if there are warranties or stuff like that. For me, I've been making launches in even smaller events as possible. Launch is just another task that has to be done for the project.

Okay, so what if you hit a milestone? I know for me, when I say I'm going to launch – I've got product configurations sweep for someone coming up, and sort of how you build an Apple laptop, right, you can pick your options. Once we get that point, there are lots of other things we’ll do, but once I get to that point, I actually circle back with them and I send them a questionnaire about the project. “How was the launch? How did the overall project management go? If there's one thing we could do to improve it, would that be for you guys?” That’s part of my launch sequence as well.

Most of the time, it’s hard for me to remember a time when this did not happen – although I'm sure it did – but as Eric said, the launch is just the first milestone in a longer series of smaller launches or smaller projects or tickets that we’re going to take care of. Today, with this launch that we just did for this company that I've mentioned before, the guy who was the acting CTO asked me, “Are you interested in moving forward and doing more work with them?” I said, “Absolutely, yes.” To me, I thought it was obvious that I was interested in continuing a nice and interesting client, nice and interesting work, but he said, “Look, the chemistry doesn’t always work, so I want to give you the chance to evaluate that” and I actually appreciate that.

Yeah, I also do that on my side too; I just have my own questionnaire. Do they still hit the clients – my idea client, are the still in that? If not, are they so far down the thing that I need to cut them or are they kind of in the middle where they're decent but not amazing.

Yeah, I do that, but that’s more of when a contract’s wrapping up and not necessarily a launch event. I've had a couple of projects in the past where it wasn’t a launch like we were talking about, but my contract was to build a piece of software for them and I built it, gave it to him, sent him a zip file of the software and that was the end of the contract. I followed it up with him to make sure there are no bugs, or if he needed any last-minute changes, but that was very low-key. They had it; they could review it on their own and get back to me. It was more of closing out the contract than an actual launch.

Yeah, and so I guess my launch would be the first contract with this client; it will be for the product configuration stuff. My second would be like a net future contract with the other things. Again, I'd circle back with them about the project management, because that also helps you be better, right? Circling back with them what was good and what was bad; if we could do one thing to improve or you what would it be.

Alright, well any other things that we should talk about before we get into the picks?

Like I mentioned a couple of times, half a process – it might be useful is to come up with your own ideal process ahead of time of “This is how a launch should go, here’s a checklist” and go to a client. Give that to them, say, “Here’s what I'd like to use for my launch process. We’re free to change, but this is basically the template that I give to all clients.” That way, you can have a discussion with them, make sure you're covering all the items, all the things you forgot last time that you added to the list, but the client feels like they actually have a hand in it, that they can collaborate and actually customize it for their project.

Alright, let’s do some picks! Curtis, you want to start us with picks?

Sure! I'm going to pick Git Tower 2.0 for those times you actually need a git GUI. Tower has just released it; their version 1.0 was really good and their version 2.0 looks even better. It has full git flow, so that looks really good. I bought it, and there's an upgrade if you have a version 1.0 license. I usually don’t use their git GUI, but it is helpful sometimes digging back through old commits and old stuff to look through and then to stash or to pull those out so that you can see a new branch.

Cool! Eric, what are your picks?

Alright. For years now, I've been using an ergonomic keyboard. The most recent one has been the Microsoft 4000, which is kind of their big one that they had that’s wired and all that, but a couple of weeks ago I bought the new Microsoft Sculpt Ergonomic Keyboard. This is the one where it’s wireless; the number pad is actually a separate unit in all that and I've been using it for a little while and I've actually finally gotten over how the keys are different and I'm really enjoying it. Not only is it lighter and easier to use, but it also works with my iPad. It’s wireless, but it’s not Bluetooth; it’s like some proprietary thing, so I have to plugin the USB. I have a lightning adapter, a camera connection kit that has a USB, and then my little dongle for the keyboard. It’s nice because now I actually have an ergonomic, wireless keyboard for my iPad, which has been my goal for a couple of years now, so I've enjoyed it. If you're using ergonomic stuff, it’s a good one to get. It’s nice being able to detach the number keypad because it lets you bring your mouse closer to the keyboard.

That’s my pick.

Awesome! Reuven, what are your picks?

Okay, so I've got two picks for this week. One of them is a book and I think the title says it all. The title is, Surviving Your Stupid, Stupid Decision to Go to Grad School. [Laughter] And it’s written by someone who finished his PhD, who decided, “You know, I could make some money selling a book, laughing at people who go to grad school to get PhDs.” [Chuckles] It’s actually pretty amusing; it’s not like fall-on-the-floor-funny but it’s still pretty good and not inaccurate. The second thing is, as long-time listeners know, I've been working on my Mandarin Chinese, which is just a fun, fun language to learn. A friend of mine recommended up Pimsleur, and they have these recordings, and it’s based on this scientific method by this Professor Pimsleur from Princeton from a number of years ago. I don’t know if it would actually work just on its own, but in combination with other techniques, I found it’s really surprisingly effective. I would not expect a recording to really be able to get you to learn words and sentence structure as well as it does, and this friend of mine told me that it’s especially good for pronunciation although I'm not convinced that’s true; I think I just have a horrible American accent and everything. But that’s definitely worth it if you're interested in languages. They have a bunch of languages there; it’s worth taking a look. So those are my picks.

Very cool. I've got a couple of picks. I picked up a couple of docking stations for my Macbook Pro; it just seemed like the development went a little smoother on it and mostly it was because I hated going to the café or whatever and having to reset up my laptop after working on my desktop or working on something in the cloud and then having the connection be so bad that it just couldn’t connect to the cloud, so I've moved most of my development onto my laptop. Anyway, so I got the docking station. I got one and it’s a Belkin – I’ll put a link to it in the show notes here in a second. There's the Belkin one and it says it has two Thunderbolt adapters; be aware that one of those Thunderbolt adapters is the one that connects to your computer, so you don’t get two Thunderbolt adapters for plugging your one into this device. I was looking around at some of the other ones and I wound up getting another one and it is a – I’ll put a link to it in the show notes too – it’s about the same; it’s a little more expensive, but the thing that I like about it is that it has the same number of USB and everything else connectors. Both of them have the wired Ethernet adapter on it, which my Macbook Pro actually doesn’t, so when I plug it in at home I get full connectivity through a wire. The reason I have two is because I have a standing desk, so I can pull the plugs out of it when it’s sitting on my desk, and then plug it back in on the standing desk. I have two monitors over there too, so it hooks up and displays and everything’s nice. The other adapter though, the second one, the reason why I like it a little bit more because it has an HDMI and two Thunderbolts. While it says you get two Thunderbolts, one of them, again, is dedicated to your laptop, but it has the other Thunderbolt adapter and the HDMI adapter so I can plug it in and get two monitors with only taking up one Thunderbolt spot on my laptop. So it’s been really nice, and then I just use the wireless keyboard and mouse, or keyboard and track pad, and it works pretty nice. I'm going to pick those. And then here in Utah, on the 24th of July, we have a state holiday called Pioneer Day. It’s the day that the Mormon Pioneers came out of the canyon and Brigham Young said this is the right place, and so they settled in Salt Lake Valley, and incidentally, all around here including in Idaho, Wyoming, Colorado, New Mexico, Arizona and Nevada. A whole bunch of those are originally Mormon settlements. There might be a few in California and there might be a few in Montana, but I'm not completely sure. Anyway, incidentally, Las Vegas was originally a Mormon settlement. Side note. Anyway, some of my ancestors are among them, and I get a lot of information about them off of FamilySearch.org. It’s a website that’s run by the Church of Jesus Christ of Latter Day Saints or the Mormon Church, but the church has an extensive genealogical program. If you're interested in genealogy at all, it’s free and it’s worth looking at to see what they have. It’s not just information on your ancestors; they actually go through and index records. What that means is they’ll look at things like army enrolments, or burial certificates – I mean, all kinds of stuff from all over the world. They actually go and photograph old church records, and then they index them, meaning that somebody actually goes and then types it in because it’s indexed in a database that you can go look it up in. If you're interested in that, I would definitely go check out FamilySearch. If you want to be part of that work, they do have volunteer indexing, and what it is is you get a program on your computer and you can then read it and you just fill in the fields. They have this in all kinds of different languages and countries, so if you live inside or outside the US, you can definitely get that information. One thing that was interesting that I ran across – and I'm just going to put this out there – that was kind of funny was that it turns out, I had never heard of the time my wife was from before I married her, but it turns out that one of my ancestors actually lived and died in that little town. It was funny too, because I was looking at the marriage records and it showed that he’s my great-great grandfather, or great-great-great grandfather, and so it showed my great-great-great grandmother, and then a little bit further down it showed that he married another woman 12 years later, and I was like, “Wait a minute, she didn’t die until 50 years later.” And then I realized this was back when the Mormons were practicing polygamy, so I am descended from a polygamy. It’s just a funny side-note there; there's a bunch of stuff there. Anyway, so my picks are FamilySearch.org and FamilySearch.org/index if you want to help out with indexing. Anyway, that’s all I've got. We are going to be talking to Daniel Pink about the book To Sell is Human here in a couple of weeks, so look forward to that. Besides that, thanks for listening and we’ll catch you all next week.