The Rogues discuss Square with Jack Danger Canty and Zach Brock.

Special Guests: Jack Danger Canty and Zach Brock.

 

Dumm, Dum, Dumm

What just happened? [Laughter]

I’m trying to encourage the intro music to start. I don’t know.

Alright here we go.

Are you cargo culting the intro?

Hey everybody and welcome to Episode 53 of the Ruby Rogues Podcast. This week on our panel we have David Brady.

Hi I’M THE GREAT… nope can’t do. Just make face, I’m Dave.

We also have Avdi Grimm!

Good Morning.

Josh Susser.

Ok, proudly leading team San Francisco this morning.

James Edward Gray

How the heck did we end up with this many Californians on one call? What happened?

DAVE: yeah some got outsourced.

I know it’s scary. And I’m Charles Max Wood from Teach me to Code.com. We also have 2 guest rogues. That is Jack Danger. What’s your last name Jack? I get stuck at Danger.

Sure. Sure. It’s “Canty”. It’s Irish for happy.

Oh Ok! And also…

Your name is “Happy Danger”?

That is amazing. [Laughter]

“Jack Danger Happy” actually.

Jack Danger Happy. That’s like, that’s like the Chinese ideogram for opportunity is crisis plus opportunity.

So I just want to know what the words in Irish are for sleepy, dopey, dark, bashful, sleazy? Who am I forgetting? Well, grumpy?

Well, grumpy is Zach. Dopey is Brock. So I think we’ve actually covered the next guest. [Laughter]

That’s right we have another guest and that’s Zach Brock.

Also from team San Francisco.

Yeah, they both work at Square and you guys can tell people what Square does and then introduce yourselves. That would be great.

So it’s this filtered photo sharing application. We think it’s going to be really big. [Laughter]

Square is a start-up it does payments. , our goal is to change the way that payments happen. So we start off with card processing and now we’re doing other really exciting stuff.

Our goal is to carry every transaction.

Which sounds crazy, except for the fact that the second largest advertising network on the Internet started off as away to figure out if your neighbor in the Harvard dorms is single or not. [Laughter]

I have a neighbor in the Harvard dorms?

When somebody from Square walking around a supermarket with me putting the cereal box into my cart, you guys will have won. [Laughter]

I’ll do that for you Josh.

I’ll hold you to that.

So did you just say that your business plan is based on the concept that because other business plans have become insane, we must be ok?

Well, it’s that going for an audacious goal is you know something that we might look at as crazy but it’s crazier things have happened and we think that we have a really you know compelling case and story around it. You can remake huge industries.

David, you’ve been in San Francisco less than a day, you’re gonna get used to this.

Yeah. There’s an insanity field here and it’s awesome!

Yeah. It’s exciting. It’s fun that people are willing to try all kinds of crazy stuff and a lot of it actually ends up sticking.

Jack gets to say something, right?

Yeah, yeah, about Square. That’s our large goal. More specifically, we’ve recently have been written up as having the talent to do Apple better than Apple. I’m not sure I agree with that, but we’re really focused on design and simplicity which fits in really well as well as iOS fits into really well. And we’re an interesting engineering company that focuses so deliberately on aesthetics and experience over features that it’s pretty fun to actually build these products.

Cool! So what about you guys in particular. What’s your experience? Where were you before Square and what kind of cool stuff do you do outside of work?

My last company, two companies in a row, I founded with a co-founder that I really appreciate and enjoy and we have almost enough skill to make the company work, but not quite. My last one was, would you believe it, online deals? We thought it was gonna be huge, not a lot of fun.

Yeah, none of those sites work.

Yeah that’s a lot of pain, a lot of pain. When I’m not at work or not writing code I’m a feminist. That’s pretty much my off time right there. That’s a different podcast though so ask me questions if it relates to Ruby Gems.

I think Ruby Gems can use some more feminism.

I’ve worked on it. I’m wrote one called feminizer, what it does is takes a string of text input and swaps all the gender of pronouns in the string output. You just hook it up to any website and it has a proxy and there’s one called the artofmanliness.com. I created the artofwomanliness. It’s the exact same copy; it’s just with the pronoun switch so it makes as much sense.

That’s awesome!

I’m afraid of what it would do to somebody if they ran the string “I am happy” through that? [Laughter]

Well I’ve checked a version of the Bible that is ran through this which leads to some pretty interesting passages.

That’s awesome. So you guys kind of explained you’re in the financial transaction and stuff. To me, looking around on the site, it seems that one of the really cool things is that you basically have this iPhone adaptor here that you put in and that you can swipe a credit card through it, right?

Right.

Our big goal is to make payments easier without taking credit cards it’s kind of a hassle.

Don’t you plug it into the headphone Jack?

True, yeah we couldn’t get the license in the e-lays for the 30 pen adapter so we just it’s basically cassette tape technology. It just sends the audio straight down the headphone jack and then our software decodes whatever the reader encoded and then sends it across the Internet to our servers which then sends it to some sort of payment processing gateway deep in the sort of payment network globally and then all the way back at the stack.

Holy crap! So you couldn’t get the 30 pen adapter so you encoded the credit card as audio and decode it on the other side? That’s freakin awesome!

Original part of it turns out that the bitrate on mag stripes happens to be about the same as like the audio frequencies that smart phones can record at. So you just sort of patch it straight in and it kind of works out.

Well I can hum my credit card number. Can’t you guys?

Yes, I know someone who can replicate the modem sounds.

I can actually hum Chuck’s credit card number as well.

Please do so that we can record it. [Laughter]

37 DAVE: So do you guys actually send an audio tone out to the device to feed it like AC power?

No, the readers are broken up and actually power down. The news ones that we’re sending out have a battery built in.

The old ones, I find this exciting, were kinetic energy. So that’s why its self cause the power to send the audio signal download worldwide.

Wow! That is awesome.

So, you go real slow, you don’t get a signal.

So this would never work for Wal-Mart then. “Try it faster. Try it slower. Try it again. Here, let me try it. I want to put a bag on it.”

So, what was the desire to switch from kinetic energy to battery then? It seems like kinetic would be superior.

We get a lot more data, a lot cleaner signal out of powering it. We also doubt to encryption in the device which is pretty cool and we can get information about which direction it was swiped in so we can use that in debugging problems literally for quality control and the readers, stuff like that.

How programmable is the device? I mean can you upload new software to it in the field or is it?

No definitely not. I think it nukes itself if you try and do that. Our security did a lot of pretty cool stuff, I think.

But you send me one and you tell us what you think.

Try to put something there but don’t tear it apart and try to reclaim it for some other use. That makes our routine really intact.

I’ve actually got Android Linux running on mine but I wasn’t able to get Ice Cream Sandwich so it doesn’t really count. [Laughter]

They’re too nervous to laugh.

I’m starting to visualize. Actually I’m imagining how long it would take to do it off of a tiny micropropeller.

Well, you know that they’ve sat in there trying to power the thing by swiping a card through it over and over and over again. [Laughter]

27 JACK: In a stack of ten cards.

It’s almost, oh men! I got to start over!

Okay, so that, how big is your engineering team at Square now? How many Rubyists do you have there?

Well, those have two different answers.

Well, that’s why I asked two questions.

Well, about 90 engineers and I’d say, two thirds, actually I’d say everybody touches Ruby in some way. Because of nothing else there are internal tools like our stuff for getting a new machine instead, uses Ruby and often times there’s like press and stuff you want to make to that. People who do Ruby more or less full-time is probably half that and 90 engineers sounds like a lot we also have 16 engineering teams so the average team sizes like 4 people. Like I said, Square is very much the iceberg. We just have to do a lot of stuff to get like, but you get to offer your credit card and do your bank account.

And do you manage all of the engineering teams there or just some of them?

No, so right now I’m in the teams responsible for the logged-in website and for international and dev tools and I think our current ratio is like 1 to 10 for managers to engineers.

Are you guys hardware guys yourselves or did you bring someone in to clean up the hardware stuff?

I did Electric Engineering in college but I have not touched anything at Square those guys are much, much better than I am. We have an amazing hardware team and a lot of them came from Apple actually. Were they building although like a cool way plastic accessory stuff that you use.

Everybody comes from Apple.

They’ve had some extremely talented people. Well, less now since we’ve been hiring more. [Laughter]

It doesn’t work that way Jack.

I’m done!

Here, here! Why don’t you give me your credit card number and run it through Square and then…

I’ll send it to you.

It’s 4 folds and 15 ones.

That’s mine too! How did you know?

Amazing! So the, tell us, if you want to run every transaction in the world, tell us what kind of scaling problems that’d give you.

Oh God! Okay. So the actual request per second is not that high. They’re a lot of Rails apps that can handle much more just simple requests per second but those are html publishing apps where a lot of it is cached or in some way easier to manage. What we do is take the card swipe, it hits our system, and it goes through a series of systems and eventually hitting a third party service to basically decide whether this card is legit or not and whether it should be authorized and then all the way back out. And that has to be incredibly fast, which means the scaling is that we would be able to do that in every market we need to enter but we need to do it with incredibly low latency, which is a different problem than incredibly high throughput.

And then, we need to be able to scale every accessory service, like our risk team with all the machinering they do, turn on and off bad users, and all the settlement work you do to get money into the bank accounts. We just announced that we do next day deposit which where if you took a payment at 5pm it would be in your bank account the next day, which is extraordinarily hard and trying to scale that to lots and lots of users without having a hundred thousand analysts, actually we’re gonna need per users is actually tough.

Yeah, this approach that Paypal took is they basically hired most of Omaha Nebraska to basically support a risk team. They had something like 8 or 900 employees there. All they did was like answer phone calls and look at the transactions. So, our goal is to scale sort of sublimely with the number of transactions.

Have you thought about using mechanical turk for that?

We’ve looked at a couple of the problem solving things but.

Yes, we’ve thought of it. We’ve thought of it.

It’s a fulltime job to manage quality on crowds or stuff. There’s actually some start-ups that do a really good job of it but we haven’t got to the point where it makes much sense.

We plan to start like that. We don’t so much as use our service as try to take them.

Yeah, we just try to buy them.

Looks like a mechanical server can set the, you can set the requirements for each worker must have these qualifications. You just say you’re just a bonded and licensed CPA. [Laughter]

And we’ll pay you a nickel for every transaction you have then.

So to what Jack said, I think our scaling profile is very different from Facebook or Twitter, like if you love Facebook and you have a white screen after a refresh, it’s not a huge deal. If you load a website and your money doesn’t show up, people start freaking out. So we have like availability and correctness problem instead of like a throughput and like a scale building problem.

So, have you built any interesting tools or framework internally to deal with that or you just couldn’t find anything that we needed to do?

I think, yeah actually. Our goal is to open sources as much as stuff as possible. So hopefully in the next few months we’ll have some things to show off. But one of the things, one of the challenges we’ve run into is that to give some contacts for sort of the overall engineering story. We started off as a single Rails App because it’s a great way to build a future set. We had an API, we had a website and it was in a really, really fast. We basically figured what our business was and what we wanted to do and as we’ve gone bigger, we’ve started pulling out chunks of that app, I guess this started almost a year ago as we start bringing up some new services. And moving towards service architecture is challenging, but the payoff is so worth it. Like we have clean interfaces, separate code bases and separate test sweeps and separate deployment pipe appliances. Really, really nice.

So the team’s not like stepping on each other’s toes.

So, one thing I’m wanna ask is that you were talking about having these transactions go all the way down the stack and come all the way back up really quickly. What are some of the tricks to doing that? I mean, you’re saying that you can’t cache makes sense to me. So, what are you doing to make those super responsive because I mean people want to see that it’s coming and saying, this credit card through properly or not.

Okay, there’s like 2 things like design patterns that we use. One is degrading gracefully where we have often redundant services at various stuff in our stuck like, for whatever reason the service it relies on goes down or something is broken we just fall over to the back-up so it always goes through. The other is just a decorator pattern, where kinda like Rack works at middle where each piece in the pipeline gets to add something or remove something or make an alteration. We use them a lot for our synchronous calls just because it’s actually something we can reason about and it’s something that prevents bugs.

And just because it’s just a simple idea is that this particular service is that turns our encrypted card data into unencrypted and the people downstream from that to get unencrypted. Just goes to show that how many to that degrading gracefully and having fall backs and keeping a clear idea of who gets to do what and you just simply modify a stream going in and out, works really well. And specifically from the point in we have a lot of the complex logic in Ruby, but we have to deploy this in jRuby because we need people who can tool to JBM in a way that it doesn’t have downtime at all and ever, and goes really fast to be able to get this reliable uptime.

Right, so my next question is about your SOA, your Service Oriented Architecture. Do you just use the kind of, it seems like the standard model is typically to use some kind of queue for processing jobs and then if you need some kind of synchronous response then you can make an HTTP request. Is that how you’re doing it or are you doing something different?

Yeah we use HTTP for a lot of the stuff. The queuing thing is actually something we’ve debated for a long time we’ve worked our way up for coming with a solution we are happy with. There are lots of problems with queues but you’re dealing with money especially some of them have some guaranteed fail case so they just lose data.

By the way, I want a money queue, I really want a money queue.

Money queue!

We had a fire hose in our office use to have a sign out that says money hose only turn on in case of emergency. [Laughter]

So you were saying about queues?

The strategy we’ve gone with is actually more like a feed so each app responsible basically for publishing a feed of interesting events that happened to it and everybody else can just come back and read through that feed to sync themselves up with the state of the world. Which means, as long as your particular service is open everybody else can sort of figure out what your state is almost have been on.

Yeah that makes sense, then if there is something on the feed that I need to do something about, then I can report that back then the service they’re providing the feed can update the feed.

That’s the model we actually think about it in terms about it is the PubSubHubBub, which in terms means we can pull as sort of the --- case but actually provide basically notifications when something happens so it should be brilliance, genius.

Sounds like a decentralization of the Linda model which should be redundancy. The Linda model. They used to call it the black board model now they call it the white board model. It’s basically if you have worked for somebody you throw it up on a common bulletin board or black board and then someone comes along and marks it and says I’m working on this, It’s basically a big shared work queue that has some defying states for how it gets managed.

The trick to this of course is your event feed has to be immovable. You can never go back and update something in the past. So you just basically stream events, and did I ever say about is that if a new server comes up basically you get completely caught up on something else by starting a cursor position you know it’s just rolling forward.

Yeah, It’s also really great for anytime you need to recreate the state of the world, right? You can always just replay what happens, right?

Yup! The cool thing about this is we sort of a solution with them and now I just see it everywhere. I just came through reading from a source for how it does, slave synchronization, and that’s pretty close to what happens as it basically they get dump of a data and they’ll replay the log in order to catch up like a stream of events.

I heard a couple of years ago that the way e-Bay was managing their transaction was that they were not using transactional systems. They were just logging everything and then they would go back and make everything consistent slightly later.

There was actually a talk on this kind of approach at Rails Conf last week.

The events streaming thing. It’s cool. There’s also, there’s a great article on the LMAX Architecture which I think came out of bed fair. I think Martin Fellow wrote it. But it talks basically about putting this into practice and how they have a trading systems that handles like thousands of requests a second.

There’s millions. There’s like twenty million or something.

So you’ll be open sourcing Ruby Gem for this next week?

Yeah, games are the first few systems up and running with this thing but, the goal is to extract into something if we’re going to use it all over the place so hopefully yeah.

So, your feeds standard like our session feeds or are they a little more specialized than that.

That’s a no. And probably a custom schema for each service for now just because we don’t know exactly how we’re gonna be using this and we have a lot of really weird services. So, we’re just very strange services publishing things. They’re not certain concepts that we’ve encountered before so, maybe Zach has an idea but I don’t know if it’s protocol.

No I think, we have to get the first few of these up in production. Then we’re gonna find out the ways that it doesn’t work and probably write it 2 or 3 times before we manage to source it.

So, you guys took basically mildly weak application. You divided it up into service oriented architecture and it sounds like you took a pretty small view of service like you mentioned at one point that having a service that just does the encryption, decryption and basically you pipeline those services, right? So this service feeds to that, feeds to that on down the line.

Yeah, it depends on the sort of what layer of fashion you look at, for example there is a payment service that exposes like an HTTP interface internally even if they say, I would like you to charge this credit card this amount and says in a whole lot of seconds it comes back with a response. So that’s what the rest of the company sees but within that payment service it’s actually half a dozen other services like a gateway for Amex. We get a way to talk to a payment tech. There’s a decryption service. There’s a fueling things as well. They expose like a single interface but actually has a pipeline sort of behind the scenes. I think that’s like, it’s the same principles of like good software architecture just applied to systems architecture. It’s like half a good interfaces, half a good extractions, don’t expose private state to people.

That’s what Avdi said in our panel last week. He said that pretty much every large software system becomes object oriented over time.

So, one other thing that comes to mind with the feed approach is if you’re not using a queue usually what happens is you pull off something off a queue. You work it and then nobody else pulls it off the queue because it’s not there anymore. So, how do you handle concurrency if you have to scale it horizontally in order to manage… (I just lost my train of thought because Josh just said, I just quoted Avdi to his face and trying not to laugh) but if this feed is out there for multiple people or multiple services to subscribe to and handle things on that queue, or on that feed then if you scale horizontally to manage the traffic cue, how do you manage concurrency? How do you make sure that everybody’s not doing the same job?

It’s all on the consumer side. As a consumer you have some set of processes or workers. It’s your job to coordinate that. It’s not the job of the person publishing the feed. It’s very similar like when you hook something up to the Twitter fire hose, you’re responsible to figure out if you’ve processed the Tweet or not.

Right, that makes sense.

50 JACK: We still do have queues and we’re using rescue pretty heavily so we still do have workers turning through things, where once it happens it’s removed in addition to our feeds.

I have a question about, you have a pretty sophisticated distributed system you’ve constructed here. Do you ever find yourself wanting better linguistics support for doing this kind of distributed processing?

Are you asking them why they haven’t switched to Erlang yet? [Laughter]

No, but I guess that would be one way of asking the question.

Jack, Well, I tell you I’ve do more and read than play with Clojure than all of Square ran on Clojure. It’s not as fun as Ruby, but there’s some nice, yes. Linguistics support will be beautiful.

Can you say in particular?

So in Clojure you have to explicitly state when you’re gonna change an mutable state and it’s like an awkward invocation. When you say swap with a bang and then you’re allowed to change something and in Clojure at the runtime, it will do the locking or insert the new text you need to make sure that there’s no race conditions. And everything else, like every normal thing you write is assumed to be concurrency and I think it’s beautiful that the edge case the special case is when you have state, and everything else is it’s just pure functional, whereas in Ruby it’s the opposite.

Whereas in Ruby you can reach in and do whatever you want to somebody else’s ---.

You can hack your existing runtime code. You can open a backdoor. We don’t do this. We can open a backdoor where you send the right TCP packets and change you’re running code from any host.

It’s like but that’s cool but we’re a payments company and cool and reliable are sometimes at odds.

We have a enlightened security team otherwise they would probably tell us to search a job.

True. [Laughter]

We’re not gonna do that.

Okay, so the other question I have about service oriented architecture is, how you manage your common code? I assume that since you had some big monolithic application or maybe a small monolithic application, you break it apart. You need to have, you wanna be sharing code with among them, do you have like private Ruby Gems that you would use internally or some other thing?

Sure, first I’ll clarify that we have, we had a big moment with the gap and now we have a bunch of services contained and a big monolithic app. [Laughter]

Unfortunately, there’s not a switchy flip where you’re like “Ah! Our monolithic problems are solved.”

I half expected you to say now we have a small monolithic app.

Ooh! And that too! They’re pretty well contained, but the old Rails app is still there and it has a variety of functions which is a terrible pattern.

It’s a junkyard pattern.

It’s the junkyard pattern. It has a wrapping where sometimes a big or control used to do a bunch of stuff with all the models that it managed and all that stuff is now handled by a service, but the control is still routing proxy for it. So, requests comes in, the Rails app says oh ok, it’s just the mg shell right there and just sends off to a service and then comes back but that there are other things were we prototype new features and there’s no more convenient place to put it than in this Rails app and we need our services to get a little more mature. And sort of slice this Rails app down smaller and smaller, until it’s either it’s just a routing layer or more likely just the signed in website.

And I think this is going to be an ongoing task for a while. We’re very much like the beginning of this SOA process. It’s going to be a pretty cool challenge for the next year or two to sort of finally kill the Monorail. It’s pattern is actually the more we talk to people about it, the more I’m noticing it in other companies as well. Like Twitter runs this exact same process sounds like group on where they usually produce somewhere.

Did you say the Monorail.

Monorail.

Is that a term for a Monolithic Rails application or is that something else?

Yeah! I wish we could take credit for it but we actually stole it from Twitter.

It has a lot of deep connotation. Anyone who’s works with one hears that and think, ah yes! The monorail, the source of all my pain but also there’s.

Disneyland! [Laughter]

Gee Simpson’s monorail song.

Yes absolutely. But you must have some sort of shared functionality among several pieces you your code. You run on different servers.

We definitely, we use a lot of intro to Ruby Gems for that sort of stuff. We have a GitHub enterprise installed which makes it really easy. And do all the stuff that we share is like the things like the feed consumer will be like a Ruby Gem and stuff like we don’t generally share a lot of like business logic between apps.

Do you break that up into a service and just talk to that over the wire?

Yeah! That’s much less hassle.

But you bring a good point. Documentation about this service is actually very hard. We’ve explored a few different ways to keep things in sync like just writing the documentation for the interface on the wiki, which means it’s always out of date or we have one thing we created at iMac regenerates the documentation, but it’s really complex and most engineers don’t know how to automatically generate the documentation even though really it’s a good idea. Yahoo uses this idea where they have a, every service ships its own client but also has a separate library of its API that the client and the service server uses. So the API is actually its own concept and it is shipped so that every client server can use it but I don’t know if we’ll go that direction but I like it.

Okay. So the thing I love most about the Square office aside from the meals there, is that when you walk in there you feel like I’m in the situation room. There’s like all of this everywhere I look there’s some television monitor mounted high in the wall which makes me feel like I’m being overloaded with information. And it feels like anywhere I go in that office, there’s no way to avoid being aware of what’s going on with the company’s information systems.

Well I was hoping they had Netflix.

We also have Netflix.

A lot of us are plugged into Apple TV so you can probably watch Netflix. [Laughter]

But anyway.

Having the monitors is fantastic. It’s really nice to get a big picture of what’s going on and you can set alerting and you can have Nagios checks and everything else but you’re actually seeing a graph. Often times you’ll cut things before they would turn into a real problem. We’re in notice, like weird trends or ask questions sort of entices people to be curious.

So how much of an investment, I mean the hardware side? How much of an investment internally do you make in monitoring and reporting all of the activity in your systems?

Hardware is not the hassle. We’ve done a few TVs with Apple TVs hooked up was the small part. We had an analytics team full of analysts or both people who are good at come up with general analytics and one guy who just writes amazing analytic software, Mike Bostock.

We have actually open sourced one of this stuff like Cube is a time series database and cubism is a friend for doing a time series visualization. So the way we have it set up right now is a lot of app --- get fed in the cube so we can see over the course of days and weeks how many card authorizations are processing how many tabs we’re gonna go in, how many dollars we’re moving. And then cubism visualizes that, so the system metrics which go into graphite. So we really need to see the correlation between database load and dollars moving into the system or you see interesting spikes and you line them up with slow down to the job queue, or something like that.

And it really is designed not to be just fun sort of form but actionable information, where when you’re on call and which we have an on call rotation using pager duty and you’re in charge of the current deploy or something going on, then the task you have is to make sure that card authorizations are still processing, that the system is healthy as you like you’re doing some sort of tricky database migration or something. And how would you do that if you don’t have a rich set of dashboards telling you the state of all of our hosts?

So that’s interesting, have you noticed like times where you caught some information on the screens and it led you to something that was happening?

Totally, yeah. Like I think it was a couple of weeks ago. We’re doing a deploy and there was some sort of database migration. And we noticed that as the deploy was about to start, database load went up like a lot up and so we had to stop the deploy because we knew that something had just happened in the system that triggered a massive rise to one of our primary databases. That if we were to then try to do the database migration, MySQL would lock on a big table and everything would just grind to a halt. And you probably have some leeway where things where things would sort of stay in some sort of temporary queue or passengers stack up. But if something locks for 20 or 30 seconds, that means a card authorization that would normally take 1 second, now takes 30 seconds and that is totally unacceptable. So we just shut it down, we check out the source of the problem, it was some sort of periodic job. Okay, it doesn’t have to run right now. Let’s shut that down. Things go back to normal. Push our code out.

So when you guys said that you’ve got a team of analysts, do you mean statisticians, like actual people that are sitting down, sifting through the numbers and trying to come up with interesting questions to ask the numbers?

Yeah. I mean we have a lot of people like that and one of the things that we try to do is beat metric disturbance as much as possible so hopefully everybody is looking at numbers all the time but the analytics team in particular, is mostly focused around risks and fraud and we’ve got guys who are like heavy machine learning backgrounds are also great engineers and I requested. They do everything from looking for patterns in the data and working to come up with machine learning models to actually building and deploying the services and infrastructure to run those in real time.

I have to say, I really like the whole idea of having the metrics available to the company at large just in the sense that, you can get in there and you can say “Hey folks! This is the vision and this is how we’re doing on the vision and move forward that way.” So everybody is kind of on the same page instead of well, “All I really care about is my one little corner and as long my one little corner works the rest of you guys can be as stupid as you want”.

No, I think it helps everybody make better decisions also, like you can very clearly see both like what’s going on also the impact that you have like you roll something out and you look at the day over day graph and like wow! A lot more reasons to feel all of a sudden. It’s really cool.

It’s also a side effect of our very open culture. Like every employee knows how much money we’re processing, and how much money is in the bank account. We know all the details about Square, every employee’s tribute to. It just makes sense that we would have access to information on how all the performance is happening to.

So do you ever look up there and go, “We’re rich!”

All the time!

Look up there and go, “We still have a long way to go, we have millions of people we have to go out and serve.”

Okay, both of you guys keep using the “D” word, deployment. Can you talk a little about your deployment magic? I imagine it must be pretty crazy to be able to do zero downtime on your system which seems like that would be like a big requirement.

There’s 2 parts to it. There’s deployment for our monorail and then there’s deployment for all the new services. The new service stuff is fairly because we thought about how it is your time is really important and make this really easy. We actually open sourced one of the tools that we use this which is called Jetpack which lets you hook up or rack up to Jet really easily and then we have a corresponding script that I don’t think we’ve ruled out yet. That makes the deployment services.

So for all the new stuff it’s really easy, you have like 3 instances running, you rotate it about one at a time and sort they rule into play. Deploying the monorail is a bit more complicated, but we have it down to a one liner. There’s a script that you run in. You basically watch the graphs and stuff goes out. It takes about 10 minutes from when you decide to do it to stuff being alive on every server. But basically you have to do them sort of asynchronously. MySQL is annoying, problems doesn’t exist. We’re switching everything to PostgreSQL. It’s really nice.

It’s good to hear. I was just telling someone about the asynchronous or the concurrent index construction on PostgreSQL, which is awesome.

And the next version of PostgreSQL has a full JavaScript runtime. So now you know it’s hipster.

Yeah well that plus H store means who need MongoDB anymore. [Laughter]

MongoDB is really nice. It’s just like, the guys at MongoDB do some really amazing stuff with it. They have some, they definitely push the limits with what you can do with like a database and push a lot of stuff that look like it’s in the database which is neat. So for the deployment stuff we actually take servers out of the load bouncer deploy into them warm them up with a few requests so nobody is stuck with the slow passenger problem and then we rotate them back into the bouncer.

You warm them up with Zach’s card? [Laughter]

Otherwise just hit the home page but you should turn your card for I think so too.

So that’s interesting. So I’ve heard of people doing cutovers this sounds like you’re doing cutovers in small batches?

Yeah. Well we’d like to move towards is have a full replica of production. And we are going to have a full replica of production and actually build a sort of Green/Blue deployment. So that’s probably 6 months out. Don’t tell our production team that I said 6 months. They’re probably gonna hear me.

They’ll be done tomorrow.

Don’t worry I’m sure nobody listens to this podcast. [Laughter]

Six months, plus or minus two years, right?

We have three listeners and four of them are actually the panelists so[Laughter]

Yeah.

And the other one’s Dave’s mom.

Mm hmm.

Yeah, so ah, the deployment is basically like rolling restarts and sort of ruling stuff out and doing migrations offline. it does mean a little bit more complex in your code. You have to be backwards and forwards compatible. but the, the sort of, both the safety of not having to worry about it when stuff goes out and also the ability to deploy at any time or for any reason is really nice.

Alright so, so that sounds like an interesting topic for, like a, a talk at a conference.

Oh really?

It does. It really does. In fact you should go back in time, go back… invent an—

Interesting…

[Laughter] you should. Just saying.

ZACH: some sort of San Francisco Ruby Conference coming up.

Yeah, yeah, I’m just saying, you know especially –

Yeah, forget, forget that one. There is one in Salt Lake too but it’s already over. [Laughter]

It’s okay. They got to a time machine built in time. [Laughter]

How else do you build a time machine?

(Hard Laugh)

Was that—What was the protest sign, “What do we want? Time Machines. When do we want it? Okay, you do not understand the question.”

(Hard Laugh)

Hey guys I’ve got to drop off. I got to figure out how to get across San Francisco to, my new contract gig and it’s been great talking to you guys.

London dude, text us if you need help with directions, we’re locals.

Fantastic. Will do.

Yeah, Dave’s our not-so-local-anymore dropout.

Yeah, for this week, I’m actually tipping the balance so we actually have more San Franciscans on the call than anybody else that’s just-- that’s awesome.

Yeah, so... I had a fantastic day you guys. Thank you.

Bye.

Bye.

Bye Dave.

Bye Dave.

38 CHUCK: See Ya. Alright. Well, it’s about time we get to the picks anyway—

He’s going, Yeah.

Already?

One more question.

Okay, go for it.

So you guys have to work with banks a lot right?

True. Yeah.

So my question is… Is that awful? Or is it the worst? [Laughter]

Is that like Dave’s CoffeeScript question?

On the awful scale…

It’s like “worst++” It’s something like the bank and its software that was noted, it wasn’t like, it’s not that it’s like it was written in the 70’s, it WAS written in the 70’s. [Laughter]

It’s actually, the more you begin to like the financial system the more, sort of, terrifying it is? You can move a shocking amount of money with an FTP account at a flop file because that is basically how easy the system runs.

Oh my God.

And you have to hide out your lines with zeroes because it’s the time for punch cards.

36 CHUCK: Oh, man. I’m in the wrong profession. I should learn how to hack FTP. [Laughter]

Absolutely. Yeah, we get FTP under the bank and can we send this multi-gigabyte flat files, and then this money goes places. It’s a terrible system.

Wow. That shakes my faith in the system right there.

It all works and it works pretty consistently. Probably what we’re trying to do is make the banking system better; being able to make moving money to have a more pleasant experience.

What astounds me is how much the banking system is antiquated crap like that. And then the other half is things like smartcards and space-aged technology.

Yeah. Or jabber card, right? [Laughter]

Yeah. Every now and then, raise your iPhone, you’ve got jabber card in it.

Snapping pictures of a check with your phone and it can not only verify that it’s a real check but it can actually go and deposit it in your account.

Right. It all adds up to be, we live in the future right? We don’t have flying cars but we do have things like device in your pocket that tells you where you are in the world and could get you in contact with almost anybody.

Yeah. It basically is telepathy, right?

Right. Yes, the reason that we can do such “modern stuff” with banking is because we put in the work and deal with the pain of dealing with the “ancient stuff” about banking. And we’re that abstraction layer, in so many ways between the crap with actual banking system, and how people want to interact with payments, it pays off.

So I’m wondering if you guys have exposed any kind of API for your stuff.

Do you want the official answer or the real one? [Laughter]

Whichever one won’t get you fired.

Severely reprimanded is acceptable but fired, isn’t.

[Laughter] The real answer is, we don’t offer an API because we want to craft the span of your transaction around payments and you need control in order to do that. Because anybody can just move money around but not everybody can make you have an enjoyable experience when you’re paying or receiving money. So that’s really our value at and that’s what can make the company worth so much whereas, simply being just another banking platform is not advantageous to us. That said, we have an API because our clients use it and we have a set of curl scripts internally that hit the API to test it and we’re just not going to tell you about it but it exists.

Yeah. We have sort of tried some API Strategies and we haven’t called to the one that we’ve been really happy with. But we do have some partners that are using some of our earlier tempts. We announced our partnership with Salvation Army and Obama for America and with the Mirami Campaign. We’re definitely looking for ways to expand the number of people who can use Square.

And even when really unusual direction with those partnerships when rather than saying, “Here’s API, use it”, we have enough talent in house that we just haven’t built them an app. Those are rom applications. Like anytime we partner with somebody we’ll just go ahead and roll them their own version of the Square card reader app and shoot it off them. So it’s unusual to do that kind of thing to an API but it means a lot to us to able to crash the whole experience.

Great. So how do I get an app with my face on it where people can send me money? [Laughter]

Send me an e-mail, let’s see what I can do.

Alright. Cool. One more question I have and then we’ll get into picks and that is, “Is it really hip to be Square?”

Absolutely. The Engineers wear plaid, nothing hipper. [Laughter]

Jeans, plaid...

Alright. Let’s go ahead and get into the picks. We’ll make James start us off.

Alright. I’ve been playing around with different themes for text editor and terminal lately and the one that everybody seems to be in love with these days is Solarized. So, if you haven’t checked that out you should definitely look at it. Solarized is pretty cool in how it is designed. It’s got some cool science behind it. My problem with it is I think it’s ugly. I’ve never been able to get—

I agree.

For some reason it’s just—

I think their contrast is just way too low and it just looks like shades of brown to me.

Yeah. So, I’ve had trouble getting into it. I really appreciate the design behind it, but I’ve had trouble getting into it. But I found one recently that’s not exactly along the same lines but it kind of with the same level of detail. But really am liking and that’s “Tomorrow Theme”. What’s cool about Tomorrow Theme is that it has several variations of it so if you like a light one, there’s that…and if you like the pastels or the super vibrant or blue. Anyways, I found that’s been great and there are iTerm versions of it so I put it in my terminal and Emacs and Textmate and whatever your editor is it’s in there. There are some co-themes I’ve been playing with lately and those are my picks.

Nice.

Avdi, what are your picks?

I think just one today. I got started on a client project with some Facebook open graft development which, if you don’t know about it, requires Facebook to be able to crawl your pages for the tags that are embedded in them which becomes a little bit problematic when you’re developing locally and tying to manually test locally and because Facebook can’t see your local app. So, I looked at a few solutions for that, a lot of people set up their own tunneling proxies and stuff like that or they use dynamic DNS.

I really wanted as brainless solution as possible because I didn’t really want to manage a tunnel and I found a bunch of services for this. There’s stuff like showoff.io and Tumblers. I did a little blog post where I listed out the ones that I found. But anyway, the one that seemed like it had the most features is Pagekite and I’ve been using Pagekite quite happily. It lets you set up as many pseudo sub-domains as you want, which is nice when you’re testing big apps that actually have multiple apps involved in them. Like one app delegates to another app for the authentication or something like that. I can set all that up so it works. They even have a Deviant Package that it’ll set it up so it will start up when your computer starts up. And they have a nice little local gooey that does stuff like show you that Pagekite is active. Even actually the task bar icon shows a little blinky when the tunnel is being hit.

And of course, the nice thing about these tunneling things is that, you’re not actually exposing a local port because they’re doing something like SSH tunneling up to the service. So once you shut it down, you don’t have to turn off the local port on your router or anything like that. So, Pagekite.

Cool. Josh what are you picks?

Well, I had a great time at Rails Conf last week and the of Ruby Rogues in person finally happened. We didn’t transform into a giant robot but we did the next best thing.

That’s unfortunate.

We’re working on it.

We did the next best thing of what?

Of all wearing great hats.

Oh, yes, yes.

But I have a couple of travel-related picks. My first one is the quirky pearl cable wrapper. It’s this wonderful little piece of flexible, rubbery, plastic stuff that you pop your MacBook power supply brick into and then it has two grooves for wrapping up the cord that goes in the wall and the cord with the mug MagSafeplug that goes into your mac and it’s just great. It’s like I wrap up the cord and shove it in my bag and I don’t have cables all over the freakin’ place anymore in my bag, Goodness great.

And he was showing it off to us and we were all jealous.

It is cool. I got to say, I love how the groove for the MagSafe adaptor is thinner because that cord is thinner.

Yeah I guess quirky does these things like crowd sourcing product design and then producing them so it’s kind of awesome. So I like that and the hotel I was staying in was kind of loud the first night I was there, so I ran this mac application called Mac White Noise. I bought it in the Mac app store and it’s just a white noise generator and I played it in my mac all night, made it a lot easier to sleep. I tried a couple of them and the one I liked was from a company called TM soft.

And then I had a self indulgent pick – there is new Ruby Conf from the map this year. It’s the Steel City Ruby Conf, because Pittsburgh is the Steel City, not the Motor City. Some people get confused about it. So Steel City Ruby conf, its first weekend in August, I think the 4th or the 5th. And I’m going to be doing the talk there which makes me incredibly happy because I grew up in Pittsburgh and being able to go home and speak at the first Ruby Conference in my home town is pretty exciting for me. Our friend of the show, Steven Klabnik which is one of the organizers of the conference, and they’re setting as the ideal first conference for Ruby Developers.

Awesome. Yeah. That one looks like fun. I guess I’m next because I’m the last regular for the show. While we were in Austen, David actually– Little bit back the story here, people kept asking if the theme song for the podcast was Kryptonite by Three Doors Down and No, it’s not. It’s actually a little song that I bought off of JewelBeat and so, that’s one of my picks, is JewelBeat. You can just get really inexpensive songs licensed for one podcast or whatever. But anyway, So David had this picking on, series that he’d been listening to and one of them was pickin’ on Three Doors Down and had kryptonite on it and it was stuck on my head for two days after that. I really, really like them and they actually have a pickin’ on U2 and a couple of others. I grew up listening to country music and I have this kind of love for Blue Grass and country music and at the same time they cover a lot of the rock bands that I really enjoy. So I went and looked them up in iTunes. I’ve been listening to those for a week now and I really, really like them. So, that’s my other pick, is that deal. Anyway, Jack do you have some picks for us?

I sure do. One I mentioned earlier is Clojure. I wouldn’t have tried something in the JBM, I also like Erlang, except that Phil Hegelberg of the Seattle RB wrote the Package Management System for Clojure and so I feel like its Rubyist approved. And that gives me the boost that I need to take it seriously. I like it. I like playing with it. I think that it’s a great piece of tech for building a Service in particular. And I’ll place that to our notes so everybody could see it.

And the other one is, I don’t know but last summer, that kind of changed things for me in terms of my career and professional development that I didn’t expect. My last start-up, I mentioned in the beginning of this podcast, I tried my best. My partner tried his best, but we didn’t have the actual skills necessary to accomplish what we were trying to do. And I was wondering for months after, “What exactly went wrong?” Like, sure, we didn’t have all the experience we could have had but we were pretty smart guys, we knew the technology. And then we read, Richard Rumelt’s book, Good Strategy, Bad Strategy, he’s a professor at UC Berkeley if I’m say correctly, and he was Engineer at NASA’s JPL. He has serious technological credentials and he studied strategy for the last few decades.

Effectively, the book is about how people come up with really terrible strategy or buzz words and think it’s strategy, and how you can avoid that and how you can actually come up with a plan for an acting change in the world around you or for accomplishing measurable goals. It was pretty awesome. I haven’t really done much with it (which is embarrassing), but I’m going to rule the world eventually—not really. I will supply these skills in my next commercial endeavor. I really recommend the book. At least it helps you not do lazy thinking.

Right. Alright, Zach. Do you have some picks for us?

Yeah. As about a comic book geek, I think I have to start about the new trailer of Dark Knight Rises because it looks amazing. I think it just came out last night. I mentioned earlier that like post crest is amazing and there are all kinds cool stuff you can do and Ryan Smith, Ace Hacker, over at Heroku has a Queuing library called Queue classic which basically gives you a similar background queue to rescue built on top of Postgres. It handles stuff around, jobs and serializing things and shoving them into background and does it using the database that hopefully you are already using. It’s kind of cool. You have the right to set up and one other process in order to do this. I guess Postgres in general, the more I learn about it the more amazed I am at how awesome it is. So if you haven’t tried it out yet, you definitely should.

Both if these technologies I interviewed on the Teach Me to Code Podcast, both Ryan Smith about Queue Classic and I also interviewed Josh Berkus who is the team lead on Postgres so if you want some inside information on those, then go look those up. So I guess we’ll go ahead and wrap this up.

Chuck, I have this one last bonus pick and I’ll just put it in the show notes, it’s a link to an answer on Quora. It’s worth checking. [Laughter]

You know it, Josh.

Yeah, Watch out. Okay that’s it.

Alright I just put it in this Quora story so you guys are going to have to click on it to see what it is about. A few announcements, First off – We announced at Rails Conf last week that you can go to sign up for the Ruby Rogues Parlay, which is basically you get on a mailing list with us and you can ask us questions and stuff. I have actually got it set upon the website, but I haven’t redirected the parlay.rubyrogues.com domain yet.

So if you go to Rubyrogues.com over on the right you’ll see that you can select 10$/year, $10/month or $50/month and basically that’s just going to support the show and help cover the cost here and maybe give a little bit back to the Rogues. So if you wanna go and join up, any of those levels get you into the mailing list. Doing more than the minimum just helps the show.

And how much do they have to donate to come be a guest to the show? [Laughter]

E-mail me and we’ll talk.

Okay, Okay.

Being a guest is totally worth it, the pre-show is pretty clutched. [Laughter]

Yeah, we’ll charge extra for that.

Honestly, we invite the guests because we think they have interesting things to say and I have yet to be disappointed. But if you want to be a guest on the show, if you feel like you have something interesting to share, you can let us know. We’ll discuss it and see where we think you fit as far as our timeline and things like that.

The easiest way to get yourself to be a guest on the show apparently, is to challenge us. You know, call us out in public on Twitter, tell us we’re wrong. [Laughter]

It worked for Steve Klabnik didn’t it?

Yeah, yeah. Then come fight with us.

Yeah. So other things—we are doing a Book Club. I’ve got a tentatively scheduled for May 23rd with Jesse Storimer and we are going to be reading his book “Working With Unix Processes”, you can get that at workingwithunixprocesses.com and when you buy the book, use the code book club, I think it’s $5 off.

Another good thing about that book, it’s growing and changing over time. It recently just added another chapter to it, which was really cool because when I read it I was like, “Oh man! My biggest complaint is when you didn’t even cover IPC.” And then he just added the chapter that covers IPCs.

So if you haven’t stayed up with the updates you should do that.

Cool. Good to know. Now I’m not 75% done, I’m 65% done, is that what you’re telling me?

That’s right. You’re falling behind. Catch up, Chuck.

Alright. So other than that, you can find us in iTunes. Just do a search for “Ruby Rogues”. I think we’re still in the “what’s hot” category in technology which is really great. I really appreciate all the reviews and spreading the word that we’ve gotten from our listeners so you guys rock. Leave us review if you haven’t and you can also get us on Android and other devices simply by using one of the pod catchers on there. And if we’re not somewhere where you want us to be, we just got into Stitcher, send us some e-mail, leave us feedback in the feedback form and thing like that just to let us know where you’re looking for us so that we can make sure we’d get in there.

And finally, if you do have topic ideas, things you want us to talk about then go to Rubyrogues.com and click on request a topic, it’s up at the top. That takes you to the form where you can put things and you can also vote for topic ideas. That’s where we gotten some of the ideas like “Objects on Rails Part 2”. We also did the getting into open source and all of those came out of the forms. Those were things people `that people asks us to talk about. We do try and hit those at least once a month and get interesting guests the rest of the time.

And finally, if you were in the audience at Rails Conf, just Thanks again for the warm reception that we got. That was a lot of fun and it’s so great to see you guys there and get to interact with you in person. So, we’ll wrap this up. We’ll be back next week and we are talking to somebody. Who are we talking to?

I just looked and actually it’s “Ruby Quizzes, Exercises” and something else. Was that a pick from our website?

Yes. It’s from the topic form.

Gotcha.

So I guess we won’t be back with a guest but that’s what we’ll be talking about and we have a resident expert here on the panel. So excited for that. We’ll wrap this up. We’ll see you next week!