The Rogues talk about monitoring with Joseph Ruscio.

Special Guest: Joseph Ruscio.

 

Hey everybody and welcome to Episode 62 of The Ruby Rogues podcast. This week on our panel we have David Brady.

Oh! My gosh Are we still doing this?

Yeah! We have Josh Susser.

[laughs] Wow Flashbacks!

We also have Avdi Grimm.

Hello from Pennsylvania

James Edward Gray.

I feel compelled to point out nobody has funded me for a hundred million dollars of anything. [Laughs]

I’m Charles Max Wood from devchat.tv and this week we have a guest and that is Joe Ruscio, did I say that right?

Ah close, yeah Ruscio.

Ruscio, okay. It looks Italian and so my brain just kicked over to that pronunciation.

Yeah it’s six letters but three syllables. That’s a lot in just a short space.

[Laughs] Kinda like you.

Yup [Laughs]

Alright well, you want to introduce yourself for people who are new to who you are?

Yeah, yeah, so I’m the CTO of a company called Librato and we do monitoring that’s our thing, measure everything. I also organize the San Francisco Metrics Meet Up and we once a month talk about metrics monitoring stuff and I have been a Rubyist for about, I guess it’s like three years now which I think makes me relatively new here, but I enjoyed it.

So, Metrics Meet up, that’s like everybody comes with a ruler or?

No. A meter stick, that’s the standard.

Yard sticks are not allowed.

Gotcha!

Take those units and get out.

There’s really like, so like tell me what you guys typically discuss there that’s cool?

Yes, we’ve done.. Well actually our third one is tonight and so basically what we do is we just get people together, to talk generally on a couple topics. Usually it’s either some tooling someone has put together, like some of outsource project like we’ve had from D3.js, Mike Bostock came to talk about Horizon Charts, I think cubism.js is a library he gave a talk on, and one of my colleagues did a talk on act a support notifications. And also people come in and talk about how they do monitoring and practice. Mark McGranaghan from Heroku came in and talked about how they do monitoring, Aaron Quint from Paperless Post AQ is going to be there tonight talking about what those guys do. It’s a really broad. Anything from in practice to implementation details visualization, collection, storage, aggregation, analysis, anything to do with measuring. And also we had a talk actually on doing tracking business drivers from the guys at [Lacien] how they track unique visitors to their site. Really cool stuff, connecting saying “Oh! this guy signed up with his Gmail address and then this company signed up for an account and we can put together that it was the same person”. Right?

That’s awesome.

Yeah!

Cool.

Well that’s pretty interesting; I should put that to my calendar for something we go to. I can’t go to tonight but maybe next month.

Yeah the responses have been really positive. We got a couple a hundred of people on the meetup lists already and the meet ups have been full. So, the biggest thing I think is just keep finding people things you should have to talk about, but I think we should really do that.

Well, are you measuring the metrics of attendance versus topic?

I don’t know if other meet-up organizers or if you’ve ever signed up to one of these meet ups, particularly in San Francisco, it’s interesting because you’ll have you know, order hundred or how many people signed up to come and there is some percentage that doesn’t show up. I think there is a set of people who just sign up for all for them on a chance they want to go. But that’s the one thing, meet up, there’s not a good way actually to connect, who signed up and who actually attended rather than having someone at the door taking names.

So, if you sign up and then don’t show up three times in a row, you’re out? [Laughs]

Well, yeah, that’s a kind a thing as an organizer. You can’t do that but in your ideal world, you are like “Oh other people aren’t coming because of you”. I think what you otherwise do is to just say there is enough room for some here at stake, say twice as many people as you can actually fit and figure out it will be about right.

Yeah I’ll let you in on a secret; that’s what happens with the Ruby meet up. There is a lot more seats than the actual capacity.

Yeah, that’s what we’ve been doing and it’s been working pretty well.

Southwest Airlines does the same thing. Have you ever had to bump somebody to like a later Ruby Meet up? Like, people outside standing by.

Actually, yes. [Laughter]

You just have to make that next meet up free, can you do that?

Great, great

Upgrade them to first class.

Put them in a hotel in the mean time. So, Joe, earlier you said in your introduction, you said measure everything. That’s your thing. So can you talk about that? Why?

Yeah, usually I want to talk about monitoring, I like to say, I think monitoring is new again. So if you’ve been even somewhat paying attention kind of this pays the last I think two years or so and there’s really been this kind of news Zeitgeist around monitoring. And I started looking to see like “ why is that right? So, why people have been to observing monitoring for decades” right? So, what’s the big deal now? And if you look at their companies, I feel who really kind of pushing them to bleeding edge like ETSI for sure I mean John Allspaw and his crew there, have been doing amazing stuff, GitHub, what do these guys have in common? And the one thing that kept on coming back was continuous deployment.

I think what’s interesting is if you look at continuous deployment there’s kind of like five things we really need to do it. Like continuous integration, one click deploy, feature flagging but then the next two pieces, the last two you need monitoring and alerting, and that’s really because your biggest risk is a change boundary, right? Like everything about operationally speaking, the biggest.. the time where we have the most risk for things going wrong is we need to choose change boundaries, kind of like changing code right? And that’s why you need unit tests for your code, when you’re in the old model of “Oh, we ship once a month” right? We throw the code over the fence to the ops team once a month they deploy it, you have this very easy to see change boundaries and you don’t need a lot but when you’re shipping multiple times a day like continuous deployment, the guys at ETSI, talk about shipping the average. Every time they give a talk they’d say, what their current average ship rate is, and it’s usually in twenty to thirties, occasionally highest fifty times a day, they ship to safe. Change is constant, right? And if you don’t have monitoring, you basically are flying blind to that point.

So the reason your saying you need the monitoring and alerting and stuff is that , you may shift that feature, something goes wrong you need some kind a feedback on that and then you would use like you said, feature flags to flag that feature back off or something?

Alright, I think anything you do is you ship the code on production so you decouple, actually deploying the code with the deploying the feature, feature flagging ideally. So then you flag it on, you watch your graphs on your dashboards and you’re looking for one or two things, if it’s a performance improvement you’re looking for, your graphs to go down the right way or if it’s not performance specific you’re looking at least for your primary indicators not to change.

Well. Its seems like the fundamental stuffs you want to be monitoring all the time but in terms of the continuous deployment, are there strategies for particular things that you want to monitor relative to the certain features that you’re pushing out? You’re just talking about like performance type things, are there other kinds of strategies for particular things you want to monitor?

I think there is generally at least always two levels, right?  You’re always going to have your most important statistics, your business drivers and also your App performance. How your site feels and see that kind of thing. You want to make sure none of that shifts, but then given to ideally with the new feature, part of specing the feature just like writing tests, unit tests should be understanding, “Ok, what should I be looking for? What should I be monitoring when this feature goes out? right? And generally like where are the risks that performance could degrade? What particular actions or routes? And also, a good thing in these monitors are how often people using the new feature? a lot of times so, in our App and API references when we add some new API call and part of what we do is how many times that call that action is actually used. Every time we decide to implement a feature, your kind of hypothesizing that “Oh my customers will find use in this.” So you should be measuring. Our people are actually using it, after we push it out.

That’s a good point. You talked earlier about, you said monitoring and warning. So you’re calling those two different things, can you talk about the difference?

Yeah. I’m offering an alerting…. So the issue thinks monitoring is, in that sense always when I say that what I mean is monitoring is the “okay, I know that I am introducing a change boundary right now” like I’m getting get pushed to Heroku or I’m doing the Capistrano deploy or whatever. So I’m going to look at all of my dashboards and verify for the next couple of minutes that everything went alright. And a good example of this… this wasn’t a change boundary we introduced, was the leap second last weekend I think, I’m still kind of having problematic stress about it. That was something that, you know it’s coming, so we basically had our boards up and a lot of people had their boards up basically watching waiting for the leap seconds to hit and see what happened.

There’s a lot of stuff went bad [laughs] it’s what happen. I’ve got some interesting graphs where you can see JVMs and stuff growing sideways we are on the dot adds, right when the leap second hit. There’s that. The problem though is going into the alerting that, there are all these latent bugs right? So there are bugs that aren’t going to manifest themselves immediately, Right? And so what the alerting does, if that’s actually active so your systems are watching, checking thresholds the simplest one, right? Something like watching for a disk to run out of space or running out a memory on a server are two simple ones but also, checking, we have a bunch of threshold set around our API. We’re very API driven service, so we have a bunch of threshold set around the API response. So we’ll get paid your duty alerts get hooked up if our API response time goes over a certain amount. And that’s often a great leading indicator that something’s wrong somewhere.

So, when you said “boards”, you mean like a dashboard kind of thing that where your monitor…?

Yeah. So dashboards, that’s another thing I kind of like to elevate. So, there’s monitoring and being able to hook graphs and I guess that includes… the important thing also is I think near real time monitoring, right? So having basically as close to real time feedback about your systems as you can as you can get. But Dashboards I think deserves special mention because they’re really a step forward. So, it’s one thing to have and say “ok well I can go, when I can look up and I can go on my web browser and pull up a graph and look at it, right? or I can be on alert.

A dashboard is a kind of more building and sharing and understanding and sticking it up on the wall somewhere, right? I think that’s really important that you can get your subject matter experts, so you can get your best sys admin guy who knows everything about the servers and he can build up server dashboard, right? And he can say actually these are the ten most important thing about a server. And then when something goes wrong you can just go look at that dashboard and maybe you don’t even know what those ten things are but you can see one of the graphs that’s gone sideways and that gives you something to drill into.

And even just to ensure understanding about the health of the business, there’s this term “Information Radiator” which I actually just came out of.. I saw some quote from Alistair Cockburn, ten years ago or something. So, I guess it came out of Agile Development where, new Information Radiator has you’re your sprint stats on it, but the same goes for operations, your Information Radiator should have the whole health of your service, so anyone walking by can see that on the wall.

Have you been on the Square office?

I have, yeah. They’re everywhere.

Yeah, I just had lunch there the other day and it’s just amazing, you can’t look at any direction without seeing a big monitor up on the wall telling you about something how their systems are doing. It’s pretty impressive.

That’s awesome.

There’s no more sophisticated aberration detection mechanism than your marketing guy with a cup of coffee staring at a TV on the wall [laughs]

That’s awesome [Laughs]

Anything about computers or your service but I’ve had it happened where, they sit with my back, hacking on my computer and our marketing guy will go, “Joe, what’s that? Why did that graph just do that?”

Why are the lights flashing? What does this mean?

Is this red light bad? [Laughs]

Actually Joe that leads me to one of the questions that I had for you about monitoring which is a… I worked at a shop that we really did have “that guy” only it was a CEO who had kind of like OCD about flow through the site it was a lead gen site, it was like a lead flow. And it was the number of people coming in, the number of conversions, number of lead flow out etcetera and first of all when I first got there we didn’t have a really strong test suite and so, the unit test suite was that Chandler would come up stairs and say “hey guys the server is down“.

So there is a real professional triumph to actually know that the server is down before he showed up. But then he had this soft metrics he would come up and he would say “Guys there’s something wrong with the server” and we’d go look “No sever is up, everything is fine” He said “No we haven’t had any leads in seven minutes” and were like, “is that wrong?” And he’s like “this time of day we should be getting twenty leads a minute.”

Yeah. I actually worked on a project one time where there was kind of a cultivated a data base kind of thing and then there were probably in the early days 150,000 entries in this data base and I would seriously get e-mail sometimes and they would be like “something is wrong I can’t find these three guys” And they just had a finger on it that well that they knew when, people were missing for whatever reason. Yeah. No matter how small, it’s strange.

Yeah. So there’s like this expert system right? That’s.. it’s okay …well, in this case its actually the expert and no system. So how do you monitor for that kind of stuff Joe?

I think the question is… well I think one thing you need to do when you got kind of a human, when you have someone like that, that turns into a reliable indicator, it’s having a system in place that it’s easy to add new metrics to. The other things I like to talk to people about is the most important thing to do is to drive the cost of metric collection as close to zero as possible, right?

That’s awesome.

Yeah. And so that’s… what’s really cool that going on right now is system like a graph, open TST and services is like the one I run in Librato. We’re kind of focused on decoupling the act of collection from storing, analyzing, visualizing all the data up. I think historically systems, they would say “oh, like this data they’d write an agent” And then you’d say “oh I need to store it somewhere” and then grab RD tool and put on the server and then I’d say “oh well I need to visualize it” and then write some little web server front end to it. And then the next gaps to collect the different piece of data, does the same, right? And you end up with this… we can’t correlate data across them.. once you add data… I mean service data. So you should ideally have a common repository or store that you can just push all you’re time through this data to, do your visualization analysis alerting all that.

And then collection is just like “Oh how hard is it to put a little in my code and just ship the data off to wherever it goes. And I think that’s a new paradigm that’s bringing up and its cool because now all the people like the actual coders they just want an of code. All you have to do is right tiny little libraries and like in Ruby that’s springing up all over the place. There is a stuff like Active Notifications in Rails which is awesome. So, I think the important thing is, when you find that expert, whatever he is coming up with, you should just try and capture that in the metric and in the end I think you are asking was, what if he is wrong? right?

Well no, mostly I was asking like “How do you capture the really really hard stuff that requires an expert? And it sounds like your answer is, instead of worrying directly about the hard stuff, focus on making the easy stuff go away”.

Yeah. Definitely that.

Well it’s also sounded like he was saying too that “If you can collect all the little pieces they may have a sense of the overall system, without actually seeing all the little parts that go into what they kind of getting and so they kind of intuitively see these trends and can pick them up” And if you can figure out what those little pieces are that give them their “feel” then you can start to quantify when things are going bad.

It’s going to be just you have to iterate with them and kind of like, it’s almost like five why’s well why do you feel that? And then where are you getting that data from? Break it down and you have all those little pieces of data in a counter repository that’s really easy to [hold], correlations. Correlations are incredibly important, right? Then you can say, “Okay this is my leading indicator, response rate has gone to the roof let me just start iterating through things that could be causing that and see which of it correlates to it.”

I liked what you said about trying to reduce the collection of metrics to as close to zero as possible. If you’re able to get that data then it’s encouraging to do so and then if it’s decoupled from how you work at that day and then maybe it turns out that three day the points you gave didn’t even really know were related can be the answer to the question you’re looking for.

I think a big part of what’s is going on right now shifting the [ownness] of implementing monitoring in metrics from operations over devs. What we’re going to need to talk about dev ops. I don’t know how familiar you guys are with that term, you do all show on that, but yeah, if it’s a super easy for a programmer as he is writing, you can literally say “Okay, when you write this code in addition to writing tests you have to write metrics and it doesn’t ship without test of metrics, right?

, It’s an interesting idea.

So, I’ve seen that in a writing performance tests that go along with your features. And Rails actually, how you send support for that in the framework although I’ve never gotten that stuff over pretty well, but I like the idea of putting that stuff into real time monitoring as well as just development time.

I think it’s usually important, I mean the Lean Start-up stuff are one of the things that I really liked about the Lean Start-up stuff is going back and say “When you actually choose to build the feature that cost time and money you should validate that actually got what you thought out of it, right?”

Yeah there’s been so many times when I’ve worked so hard on the feature and put it out there and then realize six months later “Oh, there is no way anybody ever used this because it never worked.” CHUCK:

You can get value out of every other line; you just to put a logger there.

Nice. So, Joe have seen people doing a certain monitoring for innovation accounting metrics? I don’t know how familiar with the Lean Start-up stuff you are, but….

I would like to call for a definition. [laughter]

So innovation accounting is Eric Ries’ term for paying attention to the metrics that actually affect a business rather than the vanity metrics like the number of uniques that I had on my site because great, you can get a gazillion uniques a day but they don’t actually mean anything, they don’t correlate with anything in my --- how quickly I am earning money or generating customers.

I agree with that. And one like most important thing that comes out of a.. that’s a Lean analysis, what he wrote about that is like Cohort Analysis, which is a powerful thing where instead of just monitoring uniques, you actually track your user is based on when they signed up. And we’ve seen that and definitely we’ve been doing that in hours and the people who’ve signed up very early in our service life time when things were lot more polished, their life time behavior is much different than with people who would sign up more recently, right? And so it translates like “Oh, my turn plummeting much better than if you’re just looking at all my global turn for the life of the service”.

When you do that is that actually… does that have an affect in the data collection phase of things?

Or is it more in just the reporting side of things?

I think it’s more in the reporting side of things. Those kind of metrics they’re very interesting and we use a service to track that kind of stuff. There is a couple, we use Kiss Metrics, there is also a new one called [To Tango] which is pretty interesting. And I think next time I will probably have some support for that kind of stuff. So, those kind of things were interesting because you can, just put a little JavaScript on your page and it records all the different events and then you can start to build things in a reporting phase. You could definitely be build that yourself still if you want it too, but you have to be able to track things back to “a user” and when that user joined your service, because you have to really put them in cohort, right?

Right.

So, I’m.. let say I am bootstrapping some little service.

Are you?

Well, yes.

Let’s say it. Avdi is bootstrapping a little service.

But, I don’t know, I think a lot of people are in this position, there’s so much involved with that, if your bootstrapping it’s just like you and your spare time and there’s no funding or.. you only have like a part time for it. Monitoring and metrics feels like this whole other realm that it’s like I’ve got all this.. I’ve got marketing to worry about; I’ve got getting my features out to worry about, I’ve got my site design to worry about. All these things. And then there’s metrics which is like this whole other realm. Well, I guess I’m curious about like getting over that hump but we’ve talked about that a little bit just like how it’s easier now. Mostly curious just about at what point like where is that threshold where I say “Okay, I need to care about this now” because at some level, at some early level you really do just need to focus on getting that minimum valuable product out and not unlike tweaking , tweaking is so that.. just the right shade of blue gets a few gets more conversions. Where do you say “A h! It’s okay, is time to start thinking about collecting metrics so that I can tweak things a bit more.”

There’s a couple of things, one is I thought was interesting there’s a sack of news thread about a month or two ago where someone was talking about “Oh, you have the dashboards” and a lot of the comments evolved into this like “Oh well, That is a luxury item and you can only afford to have those when you’re a big team and you have all this man power” and there’s this perception that, it’s a huge hurdle to get over to have good metrics and visual feedback.

Oh! That’s my perception and it’s not just even just the set up of getting Dashboard set-up, it’s also just I know that… I feel like I would have to devote a lot of time and thought to figuring out what to track and that’s the time that I can’t afford.

Is it just me or does it feel like its 2005 and were talking about unit testing?

I’m serious, I’m not trying to kick Avdi’s tires here, because I don’t have good metrics either, but I really feel like this is just comment about making metric collection the difficulty go to zero, it really sounds like what we’re trying to deal here is hammer these patterns into people’s brains to the point that two years, three years from now, we’re going to see people coming out saying “the very first thing you should do is set-up your CI server or your continuous deployment” and then go write your first spec and then.. I don’t know, now I’m putting words…..

Anyway, I interrupted you talking about where that threshold is, so I’m sorry about that.

Yeah. One of the point about unit testing 2005.. I’m not on biased but I absolutely believe that that’s going to be a case, right? That’s within the next couple of years, were going to get to the point where people are going to be, if you are shipping code with no insight into what’s going on, right? And so I won’t say that “Oh, the very first line of code you should write should be a instrumentation statement” I think that threshold ideally should be really well and if you go back you could come by Eric, if you talk to Eric Ries I think that’s kind of what he hammers on in that book. If you’re not measuring, you can’t validate your hypothesis, right?

Yeah, let me interrupt there for just a sec because I have this friend who has this T-shirt that I love, and I always flash to this T-shirt whenever we’re talking about this but it’s… “You can’t brag about it until you measure it”.

Nice.

Right? But it’s true, right? I mean if you know you can say well I did this, you did what? If you you don’t have any numbers, if you can’t prove that what you did had some kind of effect, right? Then so what?

The one I like is ”It’s not in production until it’s monitored and graphed.”

like what John says. He says, “If you don’t have a plan to fix it, there’s no point watching it.”

Right.

Yeah, it’s like, why would you ever have your pager go off at three in the morning if you didn’t have a plan to correct the thing that you were alerted about?

Right.

I’ve worked at that company and my job was to lie awake worrying.

Pretty much. [Laughs]

Not enough, not enough

I think it’s important you mentioned that, so that the barrier for collecting metrics has gone away down. I mean, before we even get to like Librato or something. How hard is it to throw that Google Analytics’ JavaScript in your page? I mean, which it would at least give you some insight to visitors, which pages they’re doing, how people are finding you.

When you’re doing your MBP it’s probably just those, what I call business level metrics. It’s probably the only thing need to care about, right? Just throw your set up on Heroku, whatever, don’t worry about Ops and just worry about “Am I getting anybody here? What are they doing?” and then you can, I want to put the performance step off too far because basically there’s a ton of research that shows how snappy your site feels that’s very important to converging some stuff, but…..

Oh Okay, I think a lot of Rails developers; the first thing we do for monitoring is New Relic.

Absolutely, yup.

Cause it’s super easy to get it going, its supporter all over their place and you get a pretty good level of data collection and its relatively real time, it’s not instantaneous, but you got results within minutes. So, at what point do you need to start caring about more than that?

Yeah, I mean one of the reasons I think the Ruby Community is maybe more in tune with the need to do this and then the movement is I think we need to go pass that and I don’t want to speak to out of turn about their service but where you want to start doing more of custom stuff to your application, because the thing with the solution like that is, you install it and it’s to get going it’s nothing, but you’re basically playing in their sand box then, right? So, your color in between their lines, and the visualizations and dashboards and stuff you have are the ones that what they think matches your data.

So, I think we see people maybe augmenting that with other stuff is when they say “why have all these custom metrics just that only my App cares is about like the number of this widget’s moving through this queue, right? And then, I also wanted to correlate that against, this business level driver like the number of people signing up, right? And so, when you want or pulling all kinds of custom metrics from different layers of your stat I think that’s where you need to add in solution were you can just push in whatever data you want and build arbitrary correlations and dashboards after the fact.

That’s an ambitious thing, I’ve seen a couple of companies trying to do that and mostly they’re not around anymore. I’m not spelling doom for you guys but it’s just… I think it’s until recently it hasn’t been something that you could.. There was a lot of attention on.

Yeah. I definitely think awareness is growing and we’re trying to raise that. I think part of the view is, I think what a lot of companies are trying to do before and failed is they want to get support but they still try to go through an agent based approach, right? Where they said “Oh, here is our agent, you install it on your servers, it’ll collect some stats and then you can dump in all these little plugins that will add custom stats”. But, for something right in the plug-in and put in a new file in a directory somewhere in the server is expensive for a lot of stuff. I just want them actually timed these six lines of this Ruby block, right? I just want to see what these Ruby block cause and want to write a plug-in and somehow track that over. So I think we want to look for what is going to be successful I think, and this is the approach we are taking is decoupling that at the level of an API so they… some integration point to our service and some other open source solution that do the same thing as.. if you can [0:32:08.9], you can get data into our system, right?

So, yeah, tell us what… how does that change, change for example compared to the agent with the plug in?

Well, for example using that Active Support Notification. So, Rails, it’s one thing of many Rails people that’s in this, take away, they should check that out, but what active notifications does is it act all this instrumentation of Rails out of the box, right? And in your code you can just hook up what they call like subscribers to that, for instance when you’re in dev mode, all the plug-in output you see now from Rails actually all comes from Active Support Notification. So, they’re just subscribing to those notifications out to implement the logger there. If you can hook into that and then [0:33:00.0] and then you have the ability to just dump that over some API. So, HTTP or even UDP we haven’t even talked about ---. We should talk about [0:33:12.0] that used in UDP.

Would you say, “Hey, I’m just going to grab this timing and I’m just going to dump it all over the network to somewhere to an end point” right? Then that makes that instrumentation cost in your code get very close to zero cause you just have a library that you just have one line of code somewhere and “Boom, I’ve got a new metric”, right? So..

And then using, but your service Librato, then you expose that metric in various ways and allow us to relate that to other metrics etcetera.

At service like.. All you have to do is get the data to us and then the rest is, point and click. I want to build this correlation; I want to build this dashboard. And it’s a service, so you’re only paying for want you used, going back to the funding. So you don’t have a lot of.. It’s like using Heroku for hosting, right? You can make something an operational expense instead of a capital expense.

That’s a good point.

Okay. So, is there’s any.. Let’s talk about the Stack a bit, you’ve mentioned the Stack a couple of times and are there any things to be concerned about or at any particular ways to approach it, that make a lot more sense?

You mean rendering your application stack, or do you mean…

Well, okay. So, let’s say I’m running a Rails Application on a server, I have all the Ruby code that’s running, there’s been all these stuff that sits between that and the internet and I’d probably care about how those are operating as well. I remember in 2009, GoGaRuCo, Nick Kalen gave a talk about logging and trying to keep track of correlation between a user’s activity and the log- on. Like here’s your Apache log vs. your Rails log. And you need to have some sort of unique idea that ties all these things together, and your Twitter now has this snow flake like open source project that generates unique IDs quickly, right? And I think that’s related to that, that they need that be able to have these unique IDs and track users across all the different layers of their stack. Is that particularly challenging for to set something like that up or is that as long as you have the pieces, it’s easier to drop in.

I think as long as you have the pieces, as long you have something, whether It’s something like outsource Librato or you set up something like Graphite or open TSTB, as long you’ve got a central place you can push data, not that focuses on the collection, our logs are good we haven’t mentioned that yet, but another important thing your logs are an incredibly rich source of monitoring data, right? And the guys, the team Heroku, have definitely been pushing this, but I think, you’re thinking of your logs as opposed to files as event streams, right?

36:22.4]right? And so this is a very common pattern so [0:36:25.4] down there is [0:36:28.9] two open source projects, you can do something like it. There’s Paper Trail, is a service that you can point your logs to, but they also tell [0:36:37.9] of “Hey, here is a search” right? “Here is a [0:36:40.0] like HTTP400, right? So, I want to count the number of 400 areas my API is getting. And the log processor will match every log line for that and then report account like every minute, right? So now you got a map trick that you drive that to your logs that you can correlate against other things.

And the real example that going back to the leap second when were watching our board, the one thing that kind of… because we have turned off time, we’ve turned off the time updates on all our machines about an hour prior to that, we’re still watching and the number of failed API request coming in from costumers like shot though the roof. Like our little graph about four hundred areas went crazy, so we had to dig in and figure out what was going on, which is then where we’re actually going to your logs and it turned out just one of our customers who’d, they’re up and gone hay wire and just started throwing mouthful of requests at us. Incredible to me.. I did not want to go and tend, but that whole episode.. because basically anyone running the JBM, their life went upside down and there’s no such schedule for those things. That’s the most amazing part, some guy or committee just every once in awhile says “Yeah, a time for another leap second“ You can actually subscribe for e-mail notification.

That’s right.

That’s awesome .

Would you like to be part of our news letter from CERN?

Today, we discovered the Higgs Boson on and we’ve decided to add another leap second.

Yeah. That’s right.

: This second we’ll have mass.

[laughs] Oh Geez.

Three nerds out there laughing and hysterically.

You wish.

Yes, okay.

Okay, so Joe, in terms of monitoring and instrumentation all that is there is or anything else to think about when you’re sort of crafting your deployment, setting up your servers?

Something I would recommend, we do this in practice but I just mentioned earlier there’s this project called Status D which the team at --- had done and I think it’s really important and useful. Because one the thing that is tricky, when we build web apps, right? We’re primarily interested in the request response cycle which ideally is order, the hundred seconds, right? If you want something at the request level, the raw data stream your getting is like fifty millisecond resolution and even worst if you’re spread across, obviously multiple servers handling request simultaneously, right? And that’s way too dense for a human to make sense of.. “So, what do you do with that?” And there’s concept of aggregation, where you have a point that you take all the pool request measurements and merge them in to a more human consumable form like, “Okay, what was the number of these request that happened in this minute?” And the other tricky thing is, you actually need to take that measurement without actually impacting the performance of request response.

Yeah. That’s tricky sometimes.

Really, tricky, yeah.

Professor Heisenberg says, “That’s impossible.”

That’s right.

Yeah, you don’t want to get.. but there’s a cat involved but.. yeah… So, but with Stats D is a little Node.js Daemon, it sits somewhere and listens on the UDP port. And so there’s this UDP wire protocol that you can stuff in the UDP packet a key value pair and a time stamp and then you can just send this UT packet to the Stats D Daemon. And because of UDP because there’s no connection set up or anything, the only price you pay is the memory copy from user’s space in the kernel buffer, right? So your cost on your request is almost nothing than that packet shipped off. And then the Stats D Daemon aggregates all these packets on and whatever you’ve set it to, like once a minute, then it can digest them and push out a single , permanent measurement up to wherever you’re storing your data. And so we actually deploy and instances we have a setup, as we have Stats D a little Daemon running on every instance. So anything on that box whether it’s a Rails App, we have engine access plug in to talk to it, they can install and dump data over the loop back to the local one and then he can push it off into our service where we track all the measurements

Sounds a little bit like D trace.

Yeah, there is definitely some parallels there but I think that’s a very useful tool because it gets you from the per request level to the per minute level. And it does interesting things too, like if per timings attract percentile, so you can configure the percentiles and that’s really important too. Like averages, averages suck, it’s a short way to put it but being able to give you up and say “Okay, well 90% of my request finished in this much time and 75% of my request finished in this much time” you surface your outliers, right? A lot more obviously.

So, I have a question and this is more of a general question. But what are the top mistakes that you see people making with their monitoring?

Oh. That’s a good question. One of the things I’ll say is that they’ll dive in and write something where they say “Oh well, I’ll just slap this on literally,  this controller action that’s per user by some other dimension, by some other dimension” right? So we’ll see in our service trances someone signed up for a trial and then all of a sudden they have something like forty, fifty thousand metrics pouring in, with very, very low number. And then, I think they come and they start looking around and they’re like” Oh My goodness” right?

So it’s, I think you want to spend a little bit of time and actually say “Okay “ a little bit of planning ahead of time and say “Okay, what are the business drivers actually care about? What are their performance indicators actually care about?” And make sure you start with a tractable set of things and then drilling when you need to write. If there’s something strange going on and you don’t understand it, that’s an indication you need to drill on more. That’s definitely one, not doing the map and saying “Oh, if I measure this dimension by this dimension by this dimension what’s the multiplicative product of all of those?

Right.

And then, what we were just talking about another thing is trying to.. people will come in and will want to post the measurement to out of a single request response which, again you don’t want to do you don’t want to connect and say “Oh, I’ll just send this directly to wherever” on every request response and that adds a tunnel latency, sets another bad one.

So you just want a reasonable sampling, is that what you are saying?

Yeah, you want a reasonable sampling and you want to pay attention to what performance impact your instrumentation.

CHUCK Right. Okay. Well I think, were about to the point where we need to get to the picks.

Okay, I have one more quick question, I hope it’s a quick question. All the cool kids these days are building Java Script heavy apps with they really light weight back-ends. Does it make sense to do monitoring at the Java Script level? Or do you mostly just monitor your API’s and let the Java Script kind of evolve its own stuff for later?

No, [0:43:40.0] do monitoring at JavaScript level.. And so [0:43:46.5] doing a lot of cool presentations on the Chrome teams in building in all these timings data, right? Where you can actually perform not too much of something we’re getting into, but you can actually get an amazing wealth of data from the browser breaking down your Java Script between the fetch time resolving DNS for different assets, where all the time is being spent in the browser. And that is super important, that’s something we as a community actually need to get a lot better about and I just think that even the collection tooling is still --- there.

And, also, any metrics you would connect collecting JavaScript can have a very well impact on your application, right?

Yeah, As long as you’re not pushing your stat bundles to your web server, right?

Yeah. Right.

Why would that be a problem?

I don’t know.

Alright, well, let’s get to the picks we’ll make Josh go first.

Well, thank you very much.

You’re welcome.

There’s someone whose last name is towards the end of the alphabet, so I see as to being at the end of the line in school.

Yeah, tell me about it

They work backwards here.

Well, we’re all a little backwards here.

Okay, so being a little backwards, let see if you haven’t gone and seen The Avengers, you should but that’s not my pick, even though it’s the best movie I’ve seen all year, it was great. So, you  gotta stay through the credits at the end then you see this, evil smiling face glowing over the closing credits or something. And if you’re not real Marvel Comics junkie, you probably didn’t recognize that that character was “Thanos” one of the internals of Titans. But he’s a major  Marvel bad ass, but not many people know about him. So, I picked up a copy of a graphic novel of the Silver Surfer called “The Rebirth of Thanos” because I wanted to catch up on some of Thanos’ history in preparation for the movie which is coming out in only, what? Two or three years or something? So I’m kind of reading it now.

So, but this was a great read, I haven’t read much of these stories before and the great thing is there’s three or four other graphic, not all is following it, that I need to read now. But it’s a good read and I like the old Silver Surfer stuff, Silver Surfer. So, it’s a lot of fun to read so that’s that one “The Rebirth of Thanos”.

Since we’re doing episode on monitoring which is kind of like measuring but in real time, I have a pick for a little widget called “Free Ruler” and this is something I started using a couple of years ago. The version of it is.. don’t be scared when you download because the version is 1.7B5 and the last time it was updated was in 2007. And it’s made with real BASIC but all that aside, all that aside it’s an incredibly handy tool it runs in the Mac and you’re on this program and it puts up a horizontal and vertical ruler that you can set in terms of various resolutions, I like to run it in pixels. Then you just move the cursor around on the screen and it tells you what the measurements are. So, it’s really great when I’m doing UI work and I’m trying to get my lay-outs to look right. So, I like that, it’s just, it’s very handy, so “Free Ruler”.

I think I’ve had every old mentioned except Hyper Card you needed to work Hyper Card, in there somewhere.

Oh Well, yeah. I really should’ve, really should’ve tried to figure out some way. Maybe there’s a documentation stack written. Okay, and then I’ll just take the opportunity to plug my conference, Golden Gate Ruby Conference registration is open, we’re announcing speakers. Right now we have a couple of really good speakers we’ve announced and more will be announced later this week so by the time you heard this podcast there may still be some spaces available. Don’t tattle we always sell out. And then I got requested the last minute saying “Aren’t there more tickets?” Then I have to say no.

They have speakers with great hair.

Yes. And in fact Avdi is going to be one of our speakers which is very exciting for me.

Right on.

Awesome.

Looking forward to it, yes!

Have you been to San Francisco before?

Never.

You gotta show me around

Yeah, we’ll set something up after the conference, not before? Okay. So I’m done, next.

Awesome, David what are your picks?

Alright, I always say I’m gonna be quick with my picks and then I take forever so I’m gonna do one pick, and it’s gonna be quick and that is, whatif.xkcd.com

Yes!

Randall Munroe has just started a weekly column where people kind of sending in questions and say, “What if?” and he comes up with physics or statistical answers. So like the current one is “what if everyone who took the SAT guessed on every multiple choice question? How many perfect scores would there be?” And he gives you the statistical likelihood and it comes out to some number of how many heat depths of the universe you would need to ever get all the answers right on the SAT just by chance,

Wow.

And…

That first word was like, what happens if you throw a pitch at 90% the speed of..

DAVID: of light?

Light, yes. And it basically ends up in like almost an atomic bomb like explosion.

It’s…flat out is. I guess it’s it. Well it is a fission reaction.

Yeah, fission and it’s crazy.

And the thing is it’s a full article explaining with little XKDs, it’s a little pen in the engineering notebook style cartoon style. He draws and explains why there would be a nuclear explosion and so, yeah.. what-if.xkc.com there’s only two episode up, so it’s a real quick read right now. But they’re funny and they’re really intelligent and they’re a lot of fun, and that’s my pick.

Awesome, James, what are your picks?

I have chose simple ones, first of all just a cool monitoring related pick, if you ever even checked it out do check out “Scout” as far is like Rails monitoring goes one of my favorite super easy to get to that full disclosure I did work on Scout at a couple of points in the past, so I’m a little bit biased there, but it’s run by the nicest guys in the world and it’s really super easy in monitoring. It isn’t perfect for everything that’s what we are talking about earlier kind of the agent with the plug-ins and stuff. Which it sounds at some things and not at others, but it is really easy to get set up with and super great. So, check that out.

Aand then for fun stuff I found this project on Kickstarter, thanks to Peter Cooper, followed him on Twitter and it’s called OUYA and its basically this open platform console hardware’s open, software’s open it’s based on Android and build your own games in your bedroom while not wearing pants, everything you want in a console..

I am in.

I know, so it’s amazing, right? So, I just think this is one of the coolest Kickstarter projects I’ve seen and in a while for gaming geeks like me and you gotta check it out so, that’s it Scout and OUYA, those are my picks.

Pantless Driven Development huh?

Absolutely.

I could develop how I podcast.

That’s my life.

Alright, Avdi, what are your picks?

Well, let’s see in the spirit of monitoring, I have a just an analytic service that I’ve been using for a while now that I really, really love it’s called Clicky, it’s getclicky.com and back when I first started thinking about monitoring like web analytics as I was starting to sell some products and stuff, I started getting into the Google Analytic stuff and was just completely overwhelmed and realized that it would basically.. it’s like this whole other area of expertise that I’m pretty sure there are books on like mastering Google analytics. And it was just like “I don’t wanna become an expert on this” and I ran across Clicky and it was just like a billion times simpler and exposed all the interesting information that I actually cared about in ways that were SO much simpler than Google analytics and they were Real Time long before Google analytics when Real Time.. which Real Time seemed like a gimmick to me at the time I was really more interested just the fact that it was simple to use and the JavaScript is simpler but it turns out that Real time is actually pretty cool because if you change some kind of analytics configuration, you want to get some immediate feedback that you didn’t screw it up. And that enabled me to just immediately, see you know, “Okay yeah, I configured it correctly, I’m seeing the information come in that I expect to see. So, get clicky.com, you’ll really love them.

Yeah, I use them too and I have to second that, it also makes it.. You can get selective links that you can share with specific people and there are just a million different things you can do with it that are just really, really nice.

Yeah.

And it’s easy to read and see what is going on.

Yeah. There’s a million things you can do but it’s just so much simpler, even just like the JavaScript that you put in the pages there’s.. know that the customization of the Java script that you’d have to do with Google, it’s great and it’s.. for those of us in the Northern Hemisphere, it’s the hot season and so I thought it’s been a while since I did booze pick, so by booze pick today is a “Bell’s Oberon Ale” it’s an American Wheat Ale and just a wonderful refreshing summer beer, just make sure to get it good and cold.

And finally a little plug for my own stuff, I recently decided to spend a lot more of my time focusing on my own projects. One of those which has been kind of on a back burner for a long time is my stab at a screen casting subscription service and which is called “Ruby Tapas” basically little bites and tidbits of an intermediate to advanced Ruby knowledge in screen cast form. If that sounds interesting to you please head us over to “Rubytapas.com” and sign up for the mailing list, that’ll let me know that people are interested in checking in having that developed into a real thing. So, I think that’s it for me.

Awesome. Alright well, I’ll go next. I’ve got a couple of picks. This last week was the fourth of July and while I was driving down to my in laws’ house, I got about as far as you can get from civilization, if you’re driving in between Spanish Fork, Utah and Price, Utah, this is a big long canyon and that’s about where my car broke down. So, and it broke down.. Yeah, I’m idiot.

Did you break down by Thistle?

Almost.

Okay.

Yeah, Sheep Creek, if you know where that is.

Yep

Anyway, so….

You’ll love Sheep Creek.

That’s right. Yeah there’s a gas station there in an RV park and both of them were closed so, at least I can get off for the road, but I have my ten month old and my three year old with me. And so, that was even more fun and I had my laptop so I just put a movie on for my three year old and then let my ten month old crawl all over the steering wheel. Anyway, yeah it helps if you keep the oil levels up where they’re supposed to be in the car¸ it was probably my fault that it broke, but I spent the last four days with my father in law putting a new engine in the car.

Yeah, anyway, while we we’re doing that he listens to talk radio all day and which in some ways does two different for me, listening to podcast all day except the stuff I listen to is typically more interesting than what he listens to. But one of the programs he listens to is the “The Rush Limbaugh Show” and Rush kind of bugs me a little bit, but he had this guy filling in form named Walter E. Williams, who’s just this brilliant economist and he was talking about the economy and most of what he was talking about really made sense, really kind of resonated with things and he could explain that in a way that you could actually make a logical, you could follow a logical progression to get to where he was talking. And so that’s my first pick, he’s Walter E. Williams, I looked at some of the stuff and I think that he is just brilliant. Incidentally he also has a

proclamation of amnesty and pardon granted to all persons of European decent, his black, his African American. And basically he has this pardon for people of European decent for anything that they or their ancestor may have ever done to him or his ancestors, just so that people will not act like damn fools that’s what he says in their relationship with that Americans and African ancestry. So in other words you can talk him like normal people without worrying about whether or not you’re crossing some racial boundary which I thought was just funny. So..

What am I suppose to do with all this white guilt?

You’ve got a pardon; you don’t need to worry about that anymore.

Well, that’s great we’re now in post racial podcasting.

Anyway, I thought it was interesting anyway, so Walter E. Williams is my first pick and I’ll put a link up to his blog. My second pick is Eric Davis sent me his Monitor Mounting Arm and I sent him a little bit of money for it, but It’s an Ergotron mounting monitor arm and so, you mount your monitor up and then you can just move it around and kind of floats in the air. Honestly I love it, I actually ordered another cheaper mounting arm for my other monitor and it’s just so nice that it kind of clean up my desk, I feel like I have a whole lot more space there and I can push these monitors around to wherever I need them so, since we’re speaking about monitoring I thought I’d……

Can you raise them up and down?

Yes, it’s a little more involved because actually you have to move the.. There’s a bar and there’s a little collar there that I can tighten and loosen with a wrench but I can move up and down that way.

So, the hinges are vertically arranged so you can move the monitor in a horizontal plane easily?

Yes, the Ergotron monitor I can actually move up and down because it has a horizontally mounted hinge, so I can move it up and down and right and left, but the other one is a little cheaper so I actually I have to move that one up in the back but.. I have gotten more about where I want them so, I’m not too worried about it so..

Okay, Chuck, so we’re doing an episode on “monitoring”, your pick was a “monitor stand”?

Yeah.

Okay. I have another pick it’s for this lizard called the monitor; it’s the largest lizard around.

I was trying to be little tongue and cheek, you’ve been pointing that out.

Josh, I also would’ve accepted an alerting stand.

My biggest CRT monitor is this huge, Oh! never mind.

It’s like a TV, like the old kind.

Yeah. I guess that’s over. Never mind.

I got this plasma display thing that uses this factor graphics.

I really want to get a CRT monitor and then put wood paneling on the side, just so its way retro, right?

Very, steampunk.

Let’s get back from back in the days when portable meant to have a handle on it.

That’s right.

Anybody ever have one of those three hundred foot televisions?

No, no, no wheels, wheels.

I had like a two hundred television and it had a handle on the top of it, so it was considered portable.

Oh, there you go.

Should we let Joe make some picks?

Yeah, go ahead Joe.

Alright. So fun pick, I saw Amazing Spiderman this weekend.

Yes!

Was it amazing?

Yeah. It was good and I actually want to bring this up because I saw DHH just tearing it down the other day and well there was a couple of groan inducing clock points which I don’t think they can make a Super hero movie without I guess appealing to the lowest common denominator to some extent. Well, I felt as a marble nut from way back when I felt after the whole Spiderman thing better than the Toby Maguire movies.

Totally, agree with you there.

I haven’t seen it but I don’t think they could… I don’t think the bar is set very high if you want to do them Toby Maguire movies.

Exactly, so I think this doesn’t spoil anything but there’s actually web shooters mechanical devices again, which is nice and then I just felt like Andrew Garfield I think is the actor that did a much better job of capturing Spiderman’s kind of right on that balance between being a wise guy and without being obnoxious, right?

So, wait a minute he gets bitten by a radioactive spider and the spider grows mechanical web shooters?

No, he builds his own Chuck.

Oh!

He’s a Genius

Yeah. I’ll write these comments. I deserve my geek points.

I did like that they went with Gwen Stacy instead of MJ.

Yeah. So, I would definitely… especially if you’re a Spiderman geek or used to be, it’s definitely worth seeing. AND then I’ll pick first a service real quick. We’ve been starting an experiment with this new service called “Customer.io” and it’s kind of cool because their goal is to automate the email engagement life cycle, right? Where you want to try and keep your users engaged or drive engagement by sending them this automated e-mails periodically so they reach some new… they start using some new feature in your service they haven’t been using before and you can detect that and have an e-mail sent that says “Hey, sorry we doing this, you should look at this docs” right? Something like that, which I think typically you have to build into your app yourself like both tracking the events and then checking the e-mails off when they happen. So, we’ll be starting to use an app that looks pretty promising.

And then for a book, private.. the tech books I’ve been getting the most use at that lately it’s called Third party Java Script. It’s been written… there’s been early access eBook available for I think after six months or something at least. A couple of guys they have Java Script running on almost any popular blog or places where you could put comments, but it tells in all the details about like how you defensively package your JavaScript and all your little things you wouldn’t necessarily think of and stuff that are really useful to me because for our service we’ve been basically building JavaScript SDK that people can use to embed graphs on their sites and that kind a thing for now.

So, the Third Party JavaScript. I’ll put a link up for that.

Alright. is that all your picks?

I think so. That was pretty good right?

Yup.

Yeah. Absolutely, so we’re talking about how awesome Avengers is and how awesome Spiderman is, I guess we’re not going to mention Prometheus, right?

Well, I’ve haven’t seen it yet, so I have nothing to say.

It’s got very entertaining science in it.

Oh, Cool.

Josh, if you’re interested more in Thanos star, remember The Infinity got was like a 6 issue crossover series I collected that at that time.

Yeah. That definitely comes after the re-birth of Thanos.

Yeah. It was like the late 90’s I think it came out.

Alright. No, go ahead you can keep talking if you want.

If you have those, we could do a lone swap sometime.

Alright. Well, let’s go and wrap this up. Few announcements first off we are reading “Growing Object Oriented Software Guided by Tests”, we have confirmed it’s gonna be the 22nd of August I believe that we’ll be talking to the authors.

Yes. Procrastination pays off again!

Right on.

The last time I was on the show I mentioned to everybody that we had not forgotten about the Ruby Golf, for the Rogues Golf contest.

I’m upgrading that to… we actually forgot about it and I’m promising to our listeners to get after to the rest of the hosts on the shows sometime this week or next to talk about those maybe offline.

We’ll make some arbitrary picks on which ones we liked and why and we’ll pick some winners.

Yeah, we did run headlong into a holiday so..

We did.

It’s kind of hard, but anyway, yeah we’ll take care of that, we’ll get the winner on the show and give them their book and all that good stuff. Other than that, you can find us in iTunes feel free to leave us a review, really really appreciate that. We have quite a few reviews and we’re always happy to get more. You can also leave comments on this episode on the blog that’s Rubyrogues.com and sign up for Ruby Rogues Parlay at Rubyrogues.com/parlay and that’s it. Thanks.

Bye everybody!

Thanks guys.

See yah!