Jillian_Rowe:
Hello everybody, we are back for another week of Adventures in DevOps and today we're going to talk about logs and long streamings and I don't really know a whole lot about that topic so we're going to kick it right on over to the guest as we like to do. So with us today we have Chris Cooney, did I say your name right?
Chris_Cooney:
Yes, mom, perfect.
Jillian_Rowe:
Alright, great. Will Button.
Will_Button:
Hey, what's going on?
Jillian_Rowe:
and Jonathan Hall.
Jonathan_Hall:
Hi, I heard we were talking about streaming. This is all about Twitch, right?
Jillian_Rowe:
Yes, video games, Twitch, the new Zelda
Jonathan_Hall:
Awesome.
Jillian_Rowe:
game, that's it. That's what we're
Jonathan_Hall:
That's
Jillian_Rowe:
here
Jonathan_Hall:
why
Chris_Cooney:
wish.
Jillian_Rowe:
for.
Jonathan_Hall:
I'm here.
Jillian_Rowe:
Me too. Alright, Chris, so with that, um, you know, completely, uh, derailed
Will_Button:
Yeah.
Jillian_Rowe:
your topic for a moment there, why do you- Why do
Chris_Cooney:
Well,
Jillian_Rowe:
you tell
Chris_Cooney:
now I'm...
Jillian_Rowe:
us a little bit about yourself and whether or not you play video games and how that relates to streaming?
Chris_Cooney:
Yeah, okay, this is quite a challenge. Unfortunately, I'm now the guy that's not talking about gaming, which is the worst position to be in. So I'm Chris, I've been a software engineer for 10 years now. I began my career as a Java engineer, and then, which was fun. And then I moved into sort of front engineering, React, Angular, that kind of thing. And then eventually ended up maybe four or five years ago, getting into the sort of SRE and DevOps space requirements as well. So that was all the sort of work I was doing at Sainsbury's. Now I am out of the retail space which is nice and I am working with CoreLogix. CoreLogix is a full stack of availability provider. So we process logs, metrics, traces, security data and we render that in new, fun and interesting ways and we hope to, the big things that we're focused on challenging is breaking data silos. So people typically look at their logs, all their metrics, all their traces, as we blend the experience together. So you can jump from your logs, your metrics, your traces, and you can kind of move between them really fluidly. And helping customers deal with scale. So helping them approach the challenge of, you have a vast, vast amount of data. Holding onto all of it is extremely expensive. How do you archive it? How do you store it, compress it, do whatever you need to do with it? And moreover, how do you do that in a way that's completely accessible using the lowest cost storage very highly performant and effective queries and maintain data discoverability. So it's a big sort of space that we're in. And yeah, the reason for the topic today is that one of our big differentiators were built on a streaming-based architecture. So using Kafka, Kafka Streams, and unfortunately Twitch streaming kind, so sorry to disappoint anyone who's excited. And as an ancillary topic, yes, I do play games. Hence the chair for anyone who's watching the inevitable video. Mae'r cyfnodd yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdwe
Jonathan_Hall:
What?
Chris_Cooney:
Oh, good.
Jillian_Rowe:
I'm, mine still seems to be going and has this recording button.
Jonathan_Hall:
Mine is also claims to still be going. I don't know if that means anything.
Jillian_Rowe:
Yeah, we all have red buttons except for you, Will. It only kicked you out.
Jonathan_Hall:
Maybe you just need to rejoin.
Jillian_Rowe:
Maybe daylight savings time, like screwed up the, maybe
Chris_Cooney:
the
Jillian_Rowe:
there's
Chris_Cooney:
Y2K.
Jillian_Rowe:
like multiple podcasts. Yeah, but I don't either, but that doesn't mean it doesn't screw things up because some people do and some people don't and my whole week it's just, okay. Well, why don't we keep going and then the editor will edit out that fun little segue for us. Oh, never mind. He's back.
Chris_Cooney:
Click.
Will_Button:
I am. I jumped right into the recording mode, so okay.
Jillian_Rowe:
Cool.
Will_Button:
Um
Jillian_Rowe:
Alright.
Jonathan_Hall:
I think it might be useful to define what streaming means in this context. Because I mean, we're talking about logs, right? What
Chris_Cooney:
and
Jonathan_Hall:
other
Chris_Cooney:
look.
Jonathan_Hall:
way of logging is there than streaming it?
Chris_Cooney:
Yeah.
Jonathan_Hall:
Help me understand what we're talking about.
Chris_Cooney:
So there's it's it's logs, metrics, traces and security data. It's the full sort of gamut of observability and telemetry data that one might expect. And so the typical approach that people, most companies take is they take, you send your data to them in some way. So via an API call via some sort of queue that sent kinesis or whatever. And then you, they take it, they store it in a giant data store. They will index it. some sense of it and then they will trigger downstream alarms, they will trigger, they will update metrics, they will update dashboards, they will train machine learning models, they will do all these things but ostensibly it's ingest, store, index and then process. This is a pretty normal data processing model. The problem with this model is twofold. One, maintaining this giant store at the start is really difficult. It actually means just from a pure a given byte of data is higher if you go with this model because you've got all of the additional processing that's associated with indexing and querying those indexes and rationalization normalization and this is if then the customer might decide actually I just want to drop this data you know it's really really common for this and actually block is often a delete process it's not actually I don't store this in the first place it's actually a delete process because the company is simply architected to store up front so the way streaming differentiates from that is is you ingest data into CoreLogix, immediately it goes onto a stateful Kafka stream that has persistent backup. This enables a few things. One, really, really early and very fast decision-making. So if you don't want data, it will never be stored, it will never be held in the database, it'll never be indexed, no metadata will be held about it, it will just disappear into the
Jillian_Rowe:
Thank
Chris_Cooney:
ether.
Jillian_Rowe:
you.
Chris_Cooney:
The second thing is that it enables you to make decisions about where you'd like that data to be stored. So what a lot of archiving, if you'd like as an example, a lot of archiving solutions do is you take your data, it goes into high tier storage or something, and then after a certain amount of time, you can send that off to the archive. Whereas actually what we found is that if data is useless or if data is very verbose and you never query it, holding it for two weeks isn't gonna change that fact before it goes into an archive. And so what we allow, what streaming observability allows you to do with that early decision making is it's been ingested by CoreLogix, route it directly to your archive. And the thing that CoreLogix can do differently the archive directly and the archive is actually held in S3 in the customer's account. And so you can query the archive directly and without having to re-index or decompress or anything like that. And so these are all things that are enabled by a sort of stream processing based architecture. And there's a series of non-functional things as well that aren't necessary for the customer. So for example, if you do a store first kind of approach, if your storage dies, features straight away. And if storage layers can be temperamental, there's also the scariest part to change, nor is there any architecture, is the stateful parts, because if they get corrupted, if the memory goes wrong, or if the actual drives die or something, you're in serious trouble. Whereas, so if your storage dies, it kind of goes. With a streaming-based architecture, all of that processing happens up front. Storage kind of happens at the end. your dashboards will still update, your metrics will still be generated. And then if the storage layer does have a problem, it might not be stored at the end, but all of the key features along the way are still functional. It's much more resilient, much less reliant on centralized storage. That kills that single point of failure that might exist in a sort of storage first architecture. This is a few of the reasons. So it's more cost optimal, it's more resilient. It's faster, alarms fire faster in streaming-based architectures than they do in storage first architectures. Hopefully they made the case well though.
Jillian_Rowe:
I think so. Do you have any like cool stories where streaming logs has saved the day? Like somebody who was able to very quickly be like, oh no, something is not right here and go and find it. Oh wow, look at this,
Chris_Cooney:
Yeah,
Jillian_Rowe:
it's just us now.
Chris_Cooney:
yeah, so dropping people.
Jillian_Rowe:
Yeah.
Chris_Cooney:
Is there something wrong with the thing or the...
Jillian_Rowe:
I know it is a little bit temperamental, I've noticed.
Chris_Cooney:
and I'm going to be back
Jillian_Rowe:
I don't know why. It still seems to be recording for my end in years and it records
Chris_Cooney:
Yeah.
Jillian_Rowe:
like separate, it records like separate audio streams and everything anyways.
Chris_Cooney:
like everyone's their own thing.
Jillian_Rowe:
Yeah, yeah, everybody has their own audio and video, so it should be okay. Like we've had people drop off in the editor just, oh, now we're all back. And the editor just is brilliant and pieces it together. I guess I'm
Chris_Cooney:
Oh
Jillian_Rowe:
not exactly
Chris_Cooney:
cool.
Jillian_Rowe:
sure how that works.
Chris_Cooney:
Yeah, just the magic of video editing.
Jillian_Rowe:
and his mantra.
Chris_Cooney:
Yeah. Yeah. You guys okay?
Jillian_Rowe:
We were just talking about how like storage is the point of failure. It was Jonathan's point of failure.
Chris_Cooney:
Perfect. Perfect.
Jillian_Rowe:
Ah, that's funny.
Chris_Cooney:
Yeah. I feel like when you ask me, do I have any good stories? There is a story currently unfolding.
Jillian_Rowe:
Ooh, can we
Chris_Cooney:
That's
Jillian_Rowe:
know?
Chris_Cooney:
good.
Jillian_Rowe:
Can we know? Or is it like still
Chris_Cooney:
No, I
Jillian_Rowe:
everybody's
Chris_Cooney:
mean this,
Jillian_Rowe:
too sensitive about it?
Chris_Cooney:
this one right now, this is happening right now, stop
Jillian_Rowe:
Oh
Chris_Cooney:
relying
Jillian_Rowe:
yeah, okay.
Chris_Cooney:
so much on storage. This is great,
Jillian_Rowe:
Alright.
Chris_Cooney:
I can't do anything better. Stream your data somehow. In this context, it doesn't really work because they have to store it somewhere.
Jillian_Rowe:
Yeah, it gets stored locally and then uploaded. I think,
Chris_Cooney:
Yeah.
Jillian_Rowe:
I don't know, I'm not like totally familiar with what's happening.
Chris_Cooney:
way for Jonathan to get back. How long have you guys been running this podcast for? All right.
Jillian_Rowe:
Yeah, but I'm kind of off and on. Depends on how my meetings and whatnot are structured. So I'm like a part-time panelist.
Chris_Cooney:
how it goes.
Jillian_Rowe:
Oh no, whatever, whatever, think such a thing. Why is that? What is this thing doing to me? Oh, something else.
Jonathan_Hall:
I don't see any sign of problem on this side. on shows that's recording again in 1990.
Jillian_Rowe:
been blacklisted well.
Jonathan_Hall:
Can you, should I log in as the host and see if it works for me? Okay, you might have to log out, I'm not sure, but I'll try it here.
Chris_Cooney:
That's okay, stuff happens. Suffer engineer for long enough to see enough things break. So it's cool.
Jillian_Rowe:
It's all dumpster fires. It's all just varying degrees of on fire.
Chris_Cooney:
Yeah, yeah, yeah, basically, as long as you're not the direct cause of the file, you know, still there with title and matches, then it's all good.
Jillian_Rowe:
Even if I am, at this point it's like, ah, you know.
Chris_Cooney:
I'm human, what do you want from me?
Jillian_Rowe:
Was it like data I'm gonna get sued over deleting? Probably
Chris_Cooney:
I know.
Jillian_Rowe:
not. Like hopefully not.
Chris_Cooney:
It
Jillian_Rowe:
It might
Chris_Cooney:
got.
Jillian_Rowe:
be, but then...
Chris_Cooney:
Yeah, that's all the one way to find out, you know. I remember the first outage I ever caused and I panicked so much, panicked so much when it happened. And now it's like, this again.
Jonathan_Hall:
Okay.
Jillian_Rowe:
No.
Jonathan_Hall:
I see everything
Jillian_Rowe:
It sends
Jonathan_Hall:
recording
Jillian_Rowe:
like
Jonathan_Hall:
but
Jillian_Rowe:
an alert to Slack.
Jonathan_Hall:
I have a message that says Will Buttons browser preventing recording ask Will Buttons refresh the page.
Jillian_Rowe:
Yeah, well. Alright, you got the red dot. Let's see if this works.
Jonathan_Hall:
Looks like it might be working now. I'm not sure. I see that you're recording, it says 0% uploaded. No, it says 99% uploaded, I think we're golden.
Jillian_Rowe:
cool. So we refreshed the the Riverside happenings. Where
Jonathan_Hall:
So where
Jillian_Rowe:
were
Jonathan_Hall:
were
Jillian_Rowe:
we?
Jonathan_Hall:
we?
Jillian_Rowe:
We're, yeah, we were talking about storage being a point of failure. And then I asked you
Jonathan_Hall:
Thank you.
Jillian_Rowe:
if we have any cool, do you have any cool stories besides the one that we're currently experiencing
Chris_Cooney:
Ha
Jillian_Rowe:
that's
Chris_Cooney:
ha.
Jillian_Rowe:
like a little bit too existential for me at the moment. I don't think that we can do that one.
Chris_Cooney:
Yeah, sometimes the universe just conspires against you.
Jillian_Rowe:
Yeah,
Chris_Cooney:
So
Jillian_Rowe:
it really does.
Chris_Cooney:
I obviously talking about the internals goings on in the technology and core logistics might be, might be NDA breaching grounds, but I suspect that I can talk a little bit about my engineering background. We've built, so we internally, it's actually what led me to an observability company in the first place as an engineer was we built a Kafka cluster to process a great deal of sort of stream log data. mostly logs that we were trying to process. And because it was just such a volume of them coming from all these different teams and products, it was crazy. And what we found was, as soon as we introduced this cluster and we, the Kafka cluster, and we added stateful backups onto the topics, all of these missing logs were no longer missing anymore. Like that just stopped happening. And we did, there was never an intention. The intention was just, we just accepted a certain amount of logs were gonna be missing And that just stopped happening. Every single log was in place. It was never an intention. So we were trying to work out what's going on there. But obviously, the obvious answer now in reflection was really straightforward. Kafka has good mechanisms for making sure that message has been delivered where they should be delivered. And having that buffer in the way and not relying on a central storage to constantly say, keep up with what's coming on, make it more of a pull mechanism was much more resilient, much more consistent data. was far, far higher with a far higher quality of service that could offer, because we were part of the platform team, we were essentially servicing 2000 engineers, we could say to them we're absolutely confident, yeah do your worst, like set every application to trace level, it's fine, do what you want, like the level of confidence we had in the system went way, way higher, and it meant, it also meant interestingly that doing things like short patching of elastic search nodes, It was like, well, the nodes will go down, the queues will fill up for maybe a minute, a minute and a half, then it's everything recycles, they'll just start consuming from the queue again. And it wasn't massive message going out on Slack saying, hey, everyone, you would have experienced massive errors and then having to deal with that sort of fallout. Those are just the example there where it was literally just better, like it was just far, far better experience for the customer, it was better experience for us. Fewer wake-ups in the night, fewer scary deployments and that kind of thing. That was awesome.
Jillian_Rowe:
I want to say I like the response when like there's no data lost and things are working. What's wrong here? Like
Chris_Cooney:
Yeah.
Jillian_Rowe:
there must be some that's this
Chris_Cooney:
What
Jillian_Rowe:
that's
Chris_Cooney:
do we
Jillian_Rowe:
like a
Chris_Cooney:
do?
Jillian_Rowe:
very typical engineering response. That's kind of worked correctly
Chris_Cooney:
Yeah,
Jillian_Rowe:
the first time like
Chris_Cooney:
exactly.
Jillian_Rowe:
no way. What's really happening here?
Chris_Cooney:
It's very true. Yeah, it was a general confusion of like, it was too good. It was one end where, well, we need to deal with the sheer number of errors and how slow and mapping indexing problems and all sorts of different issues that are occurring in open search. really inconsistent missing logs. It was like, wait, what? How has that happened? No one trusted it because we couldn't track the root cause. Once we did a course, it was fine. And like, I think because we were so skeptical at the start, it wasn't just like, oh, yeah, if you put a, if you put a scalable sort of cloud, like a scalable queuing or streaming mechanism before your, your data storage, it makes your data storage a bit more resilient. So, okay, well, that's relatively well understood computer science, but we just didn't trust it at the time. But yeah, of the before and after and experience in that directly. And obviously when I did the Chrologics interview and they were like, hey, so we do the streaming observability thing, I was like, I know all about that. I don't know. It was great to see a product actually form out of that. And it's a big product as well, it's doing really well.
Jillian_Rowe:
That's great. That does sound much more reliable than my typical cat log file grip. This thing went
Chris_Cooney:
Yeah,
Jillian_Rowe:
wrong.
Chris_Cooney:
yeah,
Jillian_Rowe:
So
Chris_Cooney:
yeah.
Jillian_Rowe:
it sounds much better. Are there certain types of industries that you tend to work with?
Chris_Cooney:
So we tend to do best, I think, with medium-sized companies. They're approaching the problem of they have too much data, and they need a sort of scalable partnership. And potentially the ones that have been burned by some of the less attractive aspects of the existing SaaS providers. So one example of this is we offer a 15-second support response time. So if a customer asks us a question, time is 15 seconds. We sign SLAs for two minutes just to be safe, but two minutes is still
Jillian_Rowe:
Wow.
Chris_Cooney:
everybody else is like half a day a day. We have a customer support team that will respond. We have a median resolution time of about 57 minutes I think this week. So it's like we tend to respond much faster, we tend to resolve much faster than people respond. And when you couple those two things together, you've got these mid-sized companies that are scaling rapidly, they're trying to find their own way, they're trying to learn how they're going to go about. and growing their infrastructure, growing their product, having that kind of support is just really, really awesome. Excuse that, I just knocked over a card. This arm here is currently like cat defense. Yeah, sorry. Anyway, so the most important thing is recognizing that a lot of the customers have experienced two really unpleasant things. One is really bad support. And actually, especially with the SaaS observability, the solution to support is terrible. I can't name names as long as I get sued, but there are some big vendors out there that will leave tickets for weeks. And it's this Byzantine system of you have to have this specific kind of issue in order for a human being to look at it. Otherwise, you have to traverse some crazy AI I bought that the intern made last summer that was supposed to say winter money or something. It's nuts. So this is like one big issue. The other one is overages. The sort of prevalence of overages amongst SaaS observability. And it's like their business model at points. It's like, well, oh yeah, the game is this great rate, but every single year we get hit with an overage that's five times our base billable. And it's like, well, that's crazy. That's not the only way to do business. So we don't charge overages. And what we found yw'r gweithio'r gweithio, a'r gweithio'r gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n g
Jillian_Rowe:
Cool. I guess the more you guys grow, just don't bank with SVB, I guess.
Chris_Cooney:
Oh God, no. That
Jillian_Rowe:
No.
Chris_Cooney:
was terrifying. Luckily it was zero exposure for us, but it was... Yeah, I saw that. I was like... Obviously I'm from the UK, and I knew about the existence of SVP, but I didn't know just how far-reaching the consequences of that would be. And then I started hearing that the Prime Minister here is Um, so the prime minister here is like is currently in talk within talks over the weekend with Joe Biden and I think the I can't remember who else was the third person And they were just trying to like make sure that there was People were buying off the different arms of svb to make sure there wasn't this massive Bank run that occurred. It was crazy. I didn't really I didn't I never realized it was that It was going to be that bigger thing Yeah, since I was Was it was it like a national news kind of thing or?
Jillian_Rowe:
Yeah, I was trying to convince my husband that I should be a stay at home like kid cat mom. He was
Chris_Cooney:
Okay.
Jillian_Rowe:
not, he was not going for that plan. I was like,
Chris_Cooney:
Okay.
Jillian_Rowe:
yeah, you know, might be out of business, might not be, we don't know. I guess we'll find out Monday.
Chris_Cooney:
I like how you tried to seize opportunity from that though. You were like, yeah, but I could just be, I could just chill now.
Jillian_Rowe:
Yeah,
Chris_Cooney:
SPV
Jillian_Rowe:
it's
Chris_Cooney:
trash,
Jillian_Rowe:
just
Chris_Cooney:
yeah, but awesome.
Jillian_Rowe:
my new lifestyle.
Chris_Cooney:
Yeah, absolutely.
Jillian_Rowe:
Especially like my kids are in school too, so it's like, ah, you know.
Chris_Cooney:
Yeah. It's motor. Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha
Jillian_Rowe:
That's a good way to put it.
Chris_Cooney:
It's amazing.
Jonathan_Hall:
What kind of pushback do you ever get when you start talking about streaming? Do you get naysayers or people who don't believe that it's the right solution? What kind of things do you hear there?
Chris_Cooney:
I'll be honest, I think that stream processing is still, I know Kafka is becoming more and more normalized in the industry and more and more people are using it, but I actually think that most people don't really know what the difference of a streaming architecture versus a sort of stateful or a batch architecture might be. And so most of the time actually what we find is an educational challenge. It's like a, they're like, great, you're streaming. What the hell does that mean to us? And so, and that's actually, I think would be the biggest thing that we come up against. And so to the point now where we were in internal sessions with people who aren't necessarily technically in the company, so that everybody in the company knows really well and really clearly what the difference is, what the pros and cons are, and why a streaming component in an architecture is generally a good idea from both a resilience perspective but also from a cost optimization perspective as well. And so yeah, I would say that's the biggest thing, it's the education piece. Mae'r problemau yn ymweld yn ymweld. Ond yw'r ysgrifennu sy'n gwybod.
Jonathan_Hall:
You said pros and cons. What are some of the cons? Are there times when it doesn't make sense to do streaming?
Chris_Cooney:
So speaking from a sort of, in the past, I would say, introducing a sort of streaming component assumes that you're comfortable with eventual consistency. You know, you've got like a series of events piled up and you have to just chew through them at the speed that you can. And sometimes there are systems where, and also that you can assume the precedence of the data that you've got. So like first come, first served as a reasonable model, that might not be the case. if you've got the price of something constantly changing. You don't wanna queue those things because it's the latest price that is the most relevant. You don't wanna process any of the previous prices. And so you can get around that with different solutions, but a basic queuing model in that case wouldn't really work for you.
Jonathan_Hall:
Yeah, yeah.
Chris_Cooney:
So I think it's possible to build any architecture efficiently using a streaming model. It's just about there are certain nuances that appear that you have to be very aware of depending on your engineering challenge, depending on what you're trying to achieve. Luckily for logs and metrics and things, most of the time it's about the story of what happened. It's not just about what is the latest current CPU of this box, it's what has been the trend in CPU of the past hour, where has that spike come from? Those are the more common questions that one tends to ask and therefore eventual consistency models work really well for that. well but those are like additions to the, they're some of those new ones that they talked about essentially.
Jillian_Rowe:
That's really interesting. Cause I remember earlier you were saying, oh, it's not just like logs, but it's observability and metrics. And I don't think that like quite got into my head until you were starting to talk about CPU.
Chris_Cooney:
Yeah.
Jillian_Rowe:
And then my brain went like, but that's not, how would you get that from a log? So yeah, it is, that
Chris_Cooney:
Yeah.
Jillian_Rowe:
is like a nice intersection.
Chris_Cooney:
Yeah, it's the whole work. It's logs metrics, tracing and security. We started life in 2017 as a company and we were primarily log analytics. And our big sellback then was that we could use machine learning to process logs. We could cluster them together. We could find trends and variances across different values. of a certain log from a certain application and alert you when you deviate from that normal flow.
Will_Button:
Thank you.
Chris_Cooney:
And those were some of the earliest differentiators. Now it's a much more full, larger platform that's all kind of catered around the, as they say, the three pillars of observability,
Will_Button:
say the three
Chris_Cooney:
logs,
Will_Button:
plus.
Chris_Cooney:
metrics and traces, and then security as well. We have a security offering for CEM, CSPM, and then we have a sort of managed services thing on the security side as well. You can, they will do like vulnerability scanning
Will_Button:
with those.
Chris_Cooney:
Yeah, and this is
Will_Button:
Yeah,
Chris_Cooney:
what I mean.
Will_Button:
and
Chris_Cooney:
I think
Will_Button:
this
Chris_Cooney:
one
Will_Button:
is
Chris_Cooney:
of
Will_Button:
what
Chris_Cooney:
the things
Will_Button:
I mean.
Chris_Cooney:
that people
Will_Button:
I think
Chris_Cooney:
do
Will_Button:
one
Chris_Cooney:
is
Will_Button:
of the
Chris_Cooney:
they,
Will_Button:
things that people do is they...
Chris_Cooney:
with the metric side of things. So if you think about a typical metric solution is like Prometheus, maybe you're sophisticated and you've horizontally scaled Prometheus using Thanos and you've got that deployed across multiple clusters or something like that. And immediately, almost immediately with Prometheus, you run into the problem of like, all of these nodes are running out of memory, like straight away, like before you do anything, all of the nodes have run out of memory. And so and this is why I quite like a streaming architectures and be how we leverage s3 So we actually have a system for storing metrics in s3 and querying them absolutely blazing fast. It's really really quick But because it's s3, it's much much much cheaper than holding things in RAM holding things in You know a stall threads. I saw the day people using CPU for storage. It was the craziest thing we've seen Well, it was like the worst of all worlds. It was more volatile and more expensive But yeah, so it's much cheaper than RAM, far cheaper than SSD storage, far cheaper than even cheaper than most magnetic storage, S3 is by far the cheapest way of doing it. Problem is the throttling and the query limits. And we've kind of worked out quite clever system we're getting around that. So we can lift, I think one of our query things about 10 terabytes of data from S3 about 10 seconds. It's pretty awesome. And that's actually archived data in CoreLogix. So it's super fast, super powerful.
Jonathan_Hall:
I'm really curious, can you talk about how you query S3 so quickly or is that a trade secret?
Chris_Cooney:
No, so I can talk a little bit about the, so we contribute heavily to a project called Apache Arrow, which is all about columnar data formats,
Jonathan_Hall:
I know
Chris_Cooney:
making
Jonathan_Hall:
that
Chris_Cooney:
sure
Jonathan_Hall:
one, yeah,
Chris_Cooney:
that,
Jonathan_Hall:
yeah.
Chris_Cooney:
yeah, so we contribute to that. We use the Data Fusion query engine, and it comes down to using a clever format. So we use a slide variant of Parquet, called CX data, Chorologix data. So those things optimize performance, And then it's a great deal of parallelization, essentially a great deal of using utilizing serverless infrastructure, issuing a lot of parallel requests to S3. Because the one thing S3 is, it's very, very consistent at high parallel scale.
Jonathan_Hall:
Yeah.
Chris_Cooney:
But it doesn't allow these massive one off queries, or it doesn't perform particularly well on those. It usually batches them, and you end up having to do
Jonathan_Hall:
However.
Chris_Cooney:
lots and lots of them in chain. So we utilize a great deal of parallelism to pull a huge amount of data. And then we have a custom language called Dataprime. language and the real beauty of it is that it allows you can do data discovery so you can process and change. You can aggregate data in stream. So for example you can pull out data and say tell me the three availability zones, the top three availability zones that have the most logs coming from them to give you an idea of activity coming from your different availability zones for balancing. You can do that on the fly and it will aggregate the data on the fly for you. All of that is enabled by essentially just a huge amount of parallelism, coupled with a great deal of cost optimization built into things like AWS Lambda and Fargate and things like that, so we kind of just pay for usage and nothing more. Yeah, I think that's a fair summary without...
Jonathan_Hall:
Yeah,
Chris_Cooney:
yeah.
Jonathan_Hall:
that's really cool. I wouldn't have thought of all that. Yeah, that's pretty sweet. I recently had the privilege of interviewing Matthew Topol from Voltron Data who works on Apache Arrow
Chris_Cooney:
Cool.
Jonathan_Hall:
on a different podcast. And
Chris_Cooney:
Oh
Jonathan_Hall:
it was,
Chris_Cooney:
wow.
Jonathan_Hall:
I mean, I didn't know about the project until then and it was quite fascinating to hear
Chris_Cooney:
Yeah.
Jonathan_Hall:
how that works and it's
Chris_Cooney:
Yeah, it's
Jonathan_Hall:
a,
Chris_Cooney:
awesome. It's awesome.
Jonathan_Hall:
yeah.
Chris_Cooney:
I guess not. It's certainly wouldn't be my area of expertise. But we have some engineers in the company who they talked to me about this stuff. And I'm like, sure, yeah, absolutely. Like, great sounds fantastic. Like,
Jonathan_Hall:
Yeah.
Chris_Cooney:
I say that as a 10 year engineer, I like, I have a clue about 35% of what they're saying. And then they just lose me. They start talking about I was never good at data structures and algorithms. It's never been my like,
Jonathan_Hall:
Right.
Chris_Cooney:
never been my topic. But
Jonathan_Hall:
Yeah.
Chris_Cooney:
those guys are hot on it. They're really, So yeah, it's a great deal of computer science going on at Carologix actually, which has been a nice but somewhat haunting refresher of my degree, my university
Jonathan_Hall:
If anybody's
Chris_Cooney:
years.
Jonathan_Hall:
interested in listening to that episode, by the way, it's the Cup of Go podcast, the episode from February 13. If anybody wants to hear that interview with Matthew Topol, just for the listeners who might be interested.
Will_Button:
So if someone, hey, I just wanna check is my mic working okay?
Jillian_Rowe:
Yep.
Jonathan_Hall:
Your mic is working, but I keep getting this notification that it's not recording. I don't know if that's still valid or not.
Jillian_Rowe:
and there's Echo.
Will_Button:
I think it might be resolved. I switched computers.
Jonathan_Hall:
Okay,
Chris_Cooney:
Oh.
Jonathan_Hall:
awesome.
Will_Button:
So hopefully that's resolved. Although I feel bad for our editor at this point because
Jonathan_Hall:
Yeah,
Will_Button:
he's going to be
Jonathan_Hall:
he's
Will_Button:
like, dude,
Jonathan_Hall:
earning his due this
Will_Button:
I've
Jonathan_Hall:
week.
Will_Button:
got like 47 copies of your audio.
Chris_Cooney:
Ha ha ha.
Jillian_Rowe:
What are we doing
Will_Button:
Um,
Jillian_Rowe:
here?
Will_Button:
I wanted to ask what's, um, what's the onboarding process? Like if I, if I have a, an existing logging solution from some unnamed other party,
Chris_Cooney:
Yeah.
Will_Button:
what's the transition look like to switch over to Coralogix?
Chris_Cooney:
Yeah, so it obviously depends on the tooling you've decided to utilize in the first place. But there are a few key things about our customer service. One we offer free onboarding. So if you have an account, you can go straight to customer support and say, hey, I need help integrating with CoreLogix. In 15 seconds you get a response. Usually very, very shortly after that you'll get an engineer and the engineer will sit with you and work with you to sort of install software in your cluster and deploy things If you have installed anything like Fluent D, Fluent Bit, Open Telemetry, File Beat, any of the really normal open source solutions, Prometheus, Open Search, then it's just a matter of shifting a bit of config and you will natively integrate with all those things. We natively integrate with all the cloud provider metrics, just like CloudWatch and AWS, for example. We have firehose integrations, direct integrations with CloudWatch, Lambda. Lambda layer, which will automatically export your CloudWatch metrics, all sorts of different stuff. So obviously it depends on your tech stack in terms of the actual technology you'll employ, but you'll do all of it with an engineer who spends all of their time helping people integrate and so you'll do it with an expert basically. And that's one of the things that we, I personally am quite proud of talking about because I've seen this functionality develop in the company and how they've trained the people, how they train the engineers on the job to customers on board and the platform is awesome. So the onboarding journey is super smooth because you're not really doing it on your own. There are, as we say, people that just sign up, start a plan, do their own thing, and they're quite happy. We have a bunch of documentation. There's a great deal of me speaking because I make all these videos that are embedded in the documentation. So I imagine somewhere out there, there are a series of people who are sick of the sound of my voice. But maybe you guys will end this podcast, we'll see. Yes, it's embedded in the documentation and also the platform itself has all these nice little tooltips. You hover over something and it'll tell you all about it. Cheat sheets buried everywhere for people to use. We have native integrations like 200 different tools. The chances are if you use it, we integrate with it. It's pretty smooth at this point from both the support and the engineering perspective.
Will_Button:
Yeah, I do have to say as a CoreLogix user, like the in-app tooltips and help menus and links to documentation is actually really, really well done, which is hard to do in an application of that scale. And
Chris_Cooney:
Yeah.
Will_Button:
there's numerous examples of seeing it done incorrectly. But inside of CoreLogix, it's pretty intuitive to get where you need to be and get help for the things get there.
Chris_Cooney:
Yeah, that's awesome. I'll definitely make sure the feedback gets back to the engineers and the product guys. They are great. They actually agonize over that
Will_Button:
Yeah.
Chris_Cooney:
stuff a lot. So every time the new feature discuss, the big conversation is, you know, how cohesive is this with the other features, where this confused people? And then obviously, there's an old phrase, what is it? There's only two hard problems in computer science, like cash invalidation and naming things. And we, the naming things is we really agonize over it. a'n meddwl i'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r d
Will_Button:
It makes a material difference to the experience that this has had. Oh, for sure. Yeah. It's awesome
Jillian_Rowe:
naming
Will_Button:
to hear
Jillian_Rowe:
things
Will_Button:
that.
Jillian_Rowe:
is terrible,
Will_Button:
Naming things
Jillian_Rowe:
I agree.
Will_Button:
is terrible, I agree.
Chris_Cooney:
It's a real source of stress. I don't like it. So there's always somebody with a better vocabulary than you as well who's like, did you think to use this word? And I'm like, no, I didn't think to use this word. But great.
Will_Button:
Yeah.
Jillian_Rowe:
You blocking the source as you? No.
Will_Button:
see you then.
Chris_Cooney:
Thank you. Yeah. Yeah. Or when engineers typically use very, very niche words that are technically correct, what you need to use addiction. It's like C++ programming and then a dictionary on top of it I learned a word, munging, was the very first word, my very first role in software engineering. And I was like, what is munging? What is that verb? And then it turns out it's like merging two things together.
Will_Button:
again.
Chris_Cooney:
And it's sometimes used in data. And I never knew this. I
Will_Button:
I
Chris_Cooney:
think
Will_Button:
never
Chris_Cooney:
that's
Will_Button:
do
Chris_Cooney:
the way the worst sounding
Will_Button:
this. A,
Chris_Cooney:
word
Will_Button:
the
Chris_Cooney:
I've
Will_Button:
worst
Chris_Cooney:
ever heard.
Will_Button:
sounding
Chris_Cooney:
I've never heard that word
Will_Button:
word. I've
Chris_Cooney:
used
Will_Button:
never heard
Chris_Cooney:
ever.
Will_Button:
that word
Chris_Cooney:
Like
Will_Button:
used
Chris_Cooney:
not joining
Will_Button:
ever. Like, not
Chris_Cooney:
or
Will_Button:
joining
Chris_Cooney:
merging or
Will_Button:
or
Chris_Cooney:
anything
Will_Button:
merging,
Chris_Cooney:
that everyone knows
Will_Button:
or
Chris_Cooney:
about
Will_Button:
anything
Chris_Cooney:
munging
Will_Button:
that everyone
Chris_Cooney:
for
Will_Button:
knows
Chris_Cooney:
some
Will_Button:
about
Chris_Cooney:
reason.
Will_Button:
munger. For some reason.
Chris_Cooney:
But when you actually look at the definition,
Will_Button:
But when you actually look
Chris_Cooney:
precisely
Will_Button:
at the definition,
Chris_Cooney:
what it
Will_Button:
it's
Chris_Cooney:
was
Will_Button:
precisely
Chris_Cooney:
doing, like exactly
Will_Button:
what it was
Chris_Cooney:
what
Will_Button:
doing.
Chris_Cooney:
it was doing.
Will_Button:
Exactly.
Chris_Cooney:
So you can't argue with the correctness,
Will_Button:
you with the correctness.
Chris_Cooney:
but yeah, the usability of the word is a lot lack.
Will_Button:
I've always felt like munging was this word that saying the word kind of describes exactly what it's doing once you know that.
Chris_Cooney:
Yeah. Yeah. All of this I agree with. And that didn't stop me from spending like two weeks as a as a brand new software engineer frantically trying to work out is it this kind? What is munging me? What is munging and not wanting to ask anyone because I just seemed everybody else knew it turned
Jonathan_Hall:
What's the
Chris_Cooney:
out that
Jonathan_Hall:
algorithm
Chris_Cooney:
half the
Jonathan_Hall:
for
Chris_Cooney:
floor
Jonathan_Hall:
munging?
Chris_Cooney:
had
Jonathan_Hall:
I don't
Chris_Cooney:
no
Jonathan_Hall:
want
Chris_Cooney:
idea.
Jonathan_Hall:
to get it wrong.
Chris_Cooney:
Yeah.
Will_Button:
Great. Or as the new guy, you're like, these guys are screwing with me. They just made that word
Jonathan_Hall:
Yeah.
Will_Button:
up.
Chris_Cooney:
Yeah, yeah, yeah. They
Jonathan_Hall:
reminds
Chris_Cooney:
thought
Jonathan_Hall:
me
Chris_Cooney:
it
Jonathan_Hall:
when
Chris_Cooney:
crossed
Jonathan_Hall:
I was
Chris_Cooney:
my
Jonathan_Hall:
a
Chris_Cooney:
mind.
Jonathan_Hall:
when I was a waiter we would tell new new hires to go to the basement and find a can of steam.
Chris_Cooney:
Oh, yeah.
Jonathan_Hall:
There was
Will_Button:
Yeah.
Jonathan_Hall:
no basement and what's a can of steam. Come on.
Chris_Cooney:
We used to tell, so this was back when I was a bartender a long, long time, maybe 11, 12 years ago now. And yeah, if we ever got new people in, it would always just be like, what was the one? It was like, can you go get the non-alcoholic vodka from the back? And they'd go into the back and they'd be there for ages, just hunting around
Jonathan_Hall:
Yeah,
Chris_Cooney:
for
Jonathan_Hall:
yeah,
Chris_Cooney:
non-alcoholic
Jonathan_Hall:
yeah.
Chris_Cooney:
vodka. And then they'd come up to the front and be like, I can't find any. And it'd be like, you spent 45 minutes.
Jillian_Rowe:
Ha
Chris_Cooney:
Well,
Jillian_Rowe:
ha ha!
Chris_Cooney:
what would you do?
Jonathan_Hall:
Huh.
Will_Button:
That's
Chris_Cooney:
That's
Will_Button:
what he
Chris_Cooney:
really
Will_Button:
does. He
Chris_Cooney:
trying
Will_Button:
does
Jonathan_Hall:
We're
Will_Button:
everything.
Chris_Cooney:
to
Jonathan_Hall:
just
Chris_Cooney:
find
Jonathan_Hall:
about
Will_Button:
That's
Jonathan_Hall:
to
Chris_Cooney:
it.
Jonathan_Hall:
leave.
Will_Button:
what he does. He doesn't really
Chris_Cooney:
Yeah.
Will_Button:
love. Yeah.
Chris_Cooney:
It was brilliant.
Will_Button:
So what kind of tips can you give to us that, like streaming logging and or just logging in general, like that's the back end of it. What kind of tips can
Chris_Cooney:
Yeah.
Will_Button:
you give us to make it, to make your logging better or more useful? Like what kind of application side things can or should you be doing?
Chris_Cooney:
Yeah, so look, let's say you go with a build your own DIY pretty typical stack, open search for logs and permutative metrics and stuff. I think there are some pitfalls that everybody does. And I can say that I said one before, which is the typical model is hold the data in Elastic Search, for example, for two weeks, and then archive it. And like I said before, the problem with that is it makes the assumption that if you just
Will_Button:
Thanks for
Chris_Cooney:
hold
Will_Button:
watching.
Chris_Cooney:
onto data
Will_Button:
on the point.
Chris_Cooney:
And so
Will_Button:
So
Chris_Cooney:
that model I think is an
Will_Button:
that
Chris_Cooney:
anti
Will_Button:
model
Chris_Cooney:
pattern.
Will_Button:
I
Chris_Cooney:
If
Will_Button:
think
Chris_Cooney:
you
Will_Button:
is
Chris_Cooney:
hope
Will_Button:
not something
Chris_Cooney:
if it's
Will_Button:
that's
Chris_Cooney:
useless at day one, the chances are that it's not going
Will_Button:
chances
Chris_Cooney:
to be useful
Will_Button:
are that
Chris_Cooney:
at day
Will_Button:
it's not
Chris_Cooney:
14.
Will_Button:
going to be useful for day
Chris_Cooney:
And
Will_Button:
14.
Chris_Cooney:
so you just you've just spent
Will_Button:
And so you've just
Chris_Cooney:
two
Will_Button:
spent
Chris_Cooney:
weeks worth of money
Will_Button:
two
Chris_Cooney:
essentially,
Will_Button:
weeks worth of
Chris_Cooney:
and it's
Will_Button:
money,
Chris_Cooney:
not
Will_Button:
essentially.
Chris_Cooney:
it's not a huge
Will_Button:
It's
Chris_Cooney:
amount per
Will_Button:
not
Chris_Cooney:
log,
Will_Button:
a
Chris_Cooney:
but
Will_Button:
huge
Chris_Cooney:
it
Will_Button:
amount
Chris_Cooney:
adds up
Will_Button:
per log,
Chris_Cooney:
really,
Will_Button:
but it adds
Chris_Cooney:
really
Will_Button:
up
Chris_Cooney:
quick. And
Will_Button:
really, really quick.
Chris_Cooney:
when you add on to
Will_Button:
And
Chris_Cooney:
that, the overheads
Will_Button:
when you add
Chris_Cooney:
of
Will_Button:
on
Chris_Cooney:
maintaining
Will_Button:
to that, the overheads
Chris_Cooney:
a large cluster
Will_Button:
of maintaining a
Chris_Cooney:
and
Will_Button:
large cluster
Chris_Cooney:
operating these
Will_Button:
and
Chris_Cooney:
sort of extremely
Will_Button:
operating
Chris_Cooney:
complex
Will_Button:
the system
Chris_Cooney:
open
Will_Button:
extremely
Chris_Cooney:
search
Will_Button:
comfortably.
Chris_Cooney:
clusters is a skill into itself.
Will_Button:
scale into itself. That would be the first thing. So as a general characteristic
Chris_Cooney:
as much
Will_Button:
is
Chris_Cooney:
data as
Will_Button:
keep
Chris_Cooney:
possible
Will_Button:
as much data
Chris_Cooney:
out
Will_Button:
as possible
Chris_Cooney:
of your open
Will_Button:
out
Chris_Cooney:
search cluster.
Will_Button:
of your
Chris_Cooney:
Like
Will_Button:
open source
Chris_Cooney:
don't think that
Will_Button:
Don't
Chris_Cooney:
everything needs to
Will_Button:
think
Chris_Cooney:
go in
Will_Button:
that
Chris_Cooney:
there.
Will_Button:
everything is still in
Chris_Cooney:
Try and use your
Will_Button:
there.
Chris_Cooney:
high-performance
Will_Button:
Try and
Chris_Cooney:
storage
Will_Button:
use your
Chris_Cooney:
for
Will_Button:
high-performance
Chris_Cooney:
your things that you
Will_Button:
storage
Chris_Cooney:
really, really
Will_Button:
for your
Chris_Cooney:
need.
Will_Button:
things that
Chris_Cooney:
So
Will_Button:
you need.
Chris_Cooney:
error level logs and things of that nature, really important stuff. And try and find another low-cost place that you can hold stuff. So one example
Will_Button:
So,
Chris_Cooney:
that people
Will_Button:
for
Chris_Cooney:
have used
Will_Button:
example,
Chris_Cooney:
in the past that
Will_Button:
the
Chris_Cooney:
we've
Will_Button:
people
Chris_Cooney:
seen
Will_Button:
using
Chris_Cooney:
is
Will_Button:
the class
Chris_Cooney:
storage
Will_Button:
that we've seen
Chris_Cooney:
necessary,
Will_Button:
is storage necessary
Chris_Cooney:
indexed into
Will_Button:
index into
Chris_Cooney:
a parquet format,
Will_Button:
a
Chris_Cooney:
index
Will_Button:
parquet
Chris_Cooney:
it using
Will_Button:
format, indexing
Chris_Cooney:
AWS
Will_Button:
using
Chris_Cooney:
Athena,
Will_Button:
the
Chris_Cooney:
for example,
Will_Button:
query, for
Chris_Cooney:
and query
Will_Button:
example,
Chris_Cooney:
it that way.
Will_Button:
and query it that way. The query
Chris_Cooney:
seconds,
Will_Button:
is maybe
Chris_Cooney:
40
Will_Button:
30
Chris_Cooney:
seconds,
Will_Button:
seconds,
Chris_Cooney:
sometimes
Will_Button:
40 seconds,
Chris_Cooney:
a lot longer
Will_Button:
sometimes
Chris_Cooney:
if you got a lot of
Will_Button:
a
Chris_Cooney:
data
Will_Button:
lot longer
Chris_Cooney:
in there.
Will_Button:
if you know what they are
Chris_Cooney:
And
Will_Button:
doing.
Chris_Cooney:
that's where it gets really the limits of this theory. But try and find
Will_Button:
try
Chris_Cooney:
ways
Will_Button:
and find
Chris_Cooney:
of cost optimizing
Will_Button:
ways
Chris_Cooney:
early
Will_Button:
of possible
Chris_Cooney:
and build that into your architecture, because otherwise you're going to rapidly run into some problems, especially around performance, but
Will_Button:
throughout
Chris_Cooney:
later
Will_Button:
the performance
Chris_Cooney:
on down the line around
Will_Button:
board.
Chris_Cooney:
cost. And then in terms of,
Will_Button:
in terms of
Chris_Cooney:
for us, we'll go up the stack a bit. So
Will_Button:
Felly, yn ystod y cyfnod, rwy'n credu y Llorys Tosh.
Chris_Cooney:
in terms of your infrastructure, try and find the lowest touch maintenance freaking nightmare and be ready to trade off
Will_Button:
a'r ffordd yw'r ffordd yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch
Chris_Cooney:
on some features for that. In my experience, especially on the collection side, you don't need super, super, super fancy, you can do all all singing or dancing sort of agents and things like that. Nine times out of ten, you're going to use the most basic features in them and hopefully never never think about it again, except the odd patch that you've got to do, you know, when there's like a log for shell update or something that you've got to
Will_Button:
So, I strongly recommend it.
Chris_Cooney:
vendor, open source collectors, definitely, definitely. It gives you more purchasing power. You know,
Will_Button:
interesting
Chris_Cooney:
if
Will_Button:
power.
Chris_Cooney:
you don't like the vendor you're with, your code isn't absolutely littered with their code, so you can go and change. You can just switch config and move to
Will_Button:
Thank
Chris_Cooney:
somebody
Will_Button:
you.
Chris_Cooney:
else. But also because they're usually
Will_Button:
also
Chris_Cooney:
better.
Will_Button:
because they
Chris_Cooney:
They're
Will_Button:
usually better
Chris_Cooney:
more well
Will_Button:
than
Chris_Cooney:
maintained,
Will_Button:
the
Chris_Cooney:
longer
Will_Button:
one
Chris_Cooney:
lived,
Will_Button:
in
Chris_Cooney:
battle
Will_Button:
the
Chris_Cooney:
tested,
Will_Button:
team, the
Chris_Cooney:
there's
Will_Button:
one
Chris_Cooney:
no weird
Will_Button:
in the left,
Chris_Cooney:
memory
Will_Button:
that's what
Chris_Cooney:
leaks,
Will_Button:
has to
Chris_Cooney:
they
Will_Button:
be done.
Chris_Cooney:
perform more reliably. And then if we go up the level again,
Will_Button:
ac ydych chi'n gwybod i'r gael, yn ymdweud, ychydig i'r meddwl ychydig i'r gael.
Chris_Cooney:
to the as possible into the log documents themselves. So don't
Will_Button:
Felly, ydych chi'n gwybod i'r gael, mae'n gwybod i'r gael yn ymdweud i'r gyflwyno, yn ymdweud i'r gyflwyno.
Chris_Cooney:
just have a log that's like, hello, obviously log in like a JSON format, you know, make sure it's a machine readable format as nice and easily indexable, and then include lots of metadata
Will_Button:
and include
Chris_Cooney:
fields
Will_Button:
lots of metadata
Chris_Cooney:
in that
Will_Button:
fields
Chris_Cooney:
log.
Will_Button:
in that
Chris_Cooney:
So
Will_Button:
map.
Chris_Cooney:
what that does
Will_Button:
So
Chris_Cooney:
is it means
Will_Button:
what
Chris_Cooney:
that your
Will_Button:
that
Chris_Cooney:
data
Will_Button:
does
Chris_Cooney:
can
Will_Button:
is
Chris_Cooney:
be
Will_Button:
it means that
Chris_Cooney:
sliced
Will_Button:
the data
Chris_Cooney:
from
Will_Button:
can
Chris_Cooney:
lots
Will_Button:
be
Chris_Cooney:
and lots of
Will_Button:
sliced
Chris_Cooney:
different ways,
Will_Button:
from lots
Chris_Cooney:
it improves
Will_Button:
and lots
Chris_Cooney:
the
Will_Button:
of
Chris_Cooney:
cardinality
Will_Button:
different maps.
Chris_Cooney:
of your data. And what this in turn does
Will_Button:
this.
Chris_Cooney:
is it means that your data can be analyzed in lots and lots of different ways.
Will_Button:
ac mae'n ddweud y ffordd yw'r ffordd o'r cyfransor.
Chris_Cooney:
And it's actually the thing that tools like open source, is dealing with high cardinality data. So those would be sort of three of the stacks of tips that I would say. There's obviously a whole
Will_Button:
C'est un bon moment.
Chris_Cooney:
dark arts of sort of getting this stuff right, but really be aware
Will_Button:
really
Chris_Cooney:
of the
Will_Button:
be
Chris_Cooney:
three
Will_Button:
aware of
Chris_Cooney:
major
Will_Button:
that.
Chris_Cooney:
sources of cost. The storage,
Will_Button:
storage,
Chris_Cooney:
hauling onto all this stuff,
Will_Button:
holding
Chris_Cooney:
compute,
Will_Button:
onto all this
Chris_Cooney:
keeping
Will_Button:
stuff,
Chris_Cooney:
your queries
Will_Button:
and Q.
Chris_Cooney:
fast. And the third one is the operational and overhead of just keeping this thing alive once it's big enough. Those are the three places where the cost is really going to get you. If you go with your own build, you build your own solution. With the SaaS
Will_Button:
success,
Chris_Cooney:
world,
Will_Button:
success,
Chris_Cooney:
I would
Will_Button:
world,
Chris_Cooney:
say
Will_Button:
and
Chris_Cooney:
it's all about
Will_Button:
it's
Chris_Cooney:
the overages that
Will_Button:
all
Chris_Cooney:
you want
Will_Button:
about
Chris_Cooney:
to
Will_Button:
the
Chris_Cooney:
avoid
Will_Button:
number of
Chris_Cooney:
to
Will_Button:
people.
Chris_Cooney:
be really, really aware of what data you're sending and
Will_Button:
and
Chris_Cooney:
making
Will_Button:
make
Chris_Cooney:
sure that your deal
Will_Button:
me happy.
Chris_Cooney:
is reflective of the actual usage of your data. That's something we see a lot with people who've come over towards some other companies. As they say, well, our base amount was, let's say, it was $100,000
Will_Button:
some hours.
Chris_Cooney:
a year. $1000 a year. And we're like, well, what was your base
Will_Button:
what
Chris_Cooney:
deal?
Will_Button:
was
Chris_Cooney:
And
Will_Button:
your
Chris_Cooney:
then you
Will_Button:
base
Chris_Cooney:
look at their
Will_Button:
deal?
Chris_Cooney:
base deal,
Will_Button:
And when
Chris_Cooney:
and it's
Will_Button:
you
Chris_Cooney:
like,
Will_Button:
look
Chris_Cooney:
well,
Will_Button:
at the
Chris_Cooney:
your
Will_Button:
base
Chris_Cooney:
base deal
Will_Button:
deal, it's
Chris_Cooney:
was
Will_Button:
like, will
Chris_Cooney:
a
Will_Button:
your
Chris_Cooney:
quarter
Will_Button:
base deal
Chris_Cooney:
of what
Will_Button:
work?
Chris_Cooney:
you actually sort of use. So this was always going to happen. This was always like, your true cost was always going to be $400,000 a year. It's just that for
Will_Button:
It's just
Chris_Cooney:
some reason,
Will_Button:
that...
Chris_Cooney:
it's been divided into a base cost and an overage. And so we find that that's the kind of stuff, especially with SaaS
Will_Button:
stuff
Chris_Cooney:
providers, that
Will_Button:
especially
Chris_Cooney:
you need to be
Will_Button:
as
Chris_Cooney:
aware
Will_Button:
well.
Chris_Cooney:
of. Make sure your bill is really, really good.
Will_Button:
Make sure that's
Chris_Cooney:
because
Will_Button:
the
Chris_Cooney:
there's
Will_Button:
police.
Chris_Cooney:
nothing worse than leaving a ticket for a week and a half and wondering whether anyone's listening to any other side, especially when you're paying a lot of money. Yeah, those are just some of the
Will_Button:
it.
Chris_Cooney:
things.
Jillian_Rowe:
Cool. I'm wondering,
Will_Button:
Cool.
Jillian_Rowe:
like,
Will_Button:
I'm
Jillian_Rowe:
what
Will_Button:
wondering,
Jillian_Rowe:
kind of machine learning
Will_Button:
what
Jillian_Rowe:
models
Will_Button:
kind of machine
Jillian_Rowe:
are you
Will_Button:
learning
Jillian_Rowe:
guys
Will_Button:
models
Jillian_Rowe:
using?
Will_Button:
are
Jillian_Rowe:
What
Will_Button:
you guys
Jillian_Rowe:
is their
Will_Button:
using?
Jillian_Rowe:
purpose?
Will_Button:
What is their purpose?
Jillian_Rowe:
I know you
Will_Button:
I
Jillian_Rowe:
said something like
Will_Button:
know
Jillian_Rowe:
whether
Will_Button:
you
Jillian_Rowe:
or
Will_Button:
said
Jillian_Rowe:
not
Will_Button:
something
Jillian_Rowe:
a log
Will_Button:
like
Jillian_Rowe:
is important.
Will_Button:
whether or not a log is important.
Chris_Cooney:
Yeah.
Jillian_Rowe:
Is there anything else?
Will_Button:
Is there anything
Chris_Cooney:
So there's
Will_Button:
else?
Chris_Cooney:
a bunch of stuff we do. We do anomaly detection. So we have machine learning models that are detecting flow anomalies. So it will, if for example, at 3pm on a Tuesday, you typically are receiving, I don't know, 100 error logs. The machine that the CoreLogix platform will learn that that is your normal profile. And you can set an alarm, a certain amount so I get a thousand more logs than I normally expect, tell me about it. That's one of the machine learning aspects that we have, what that enables us to do is it
Will_Button:
do
Chris_Cooney:
answers
Will_Button:
is.
Chris_Cooney:
the quite complicated question of dynamic alerting. Static alerting would be tell me when I have more than a thousand errors or tell me when I have more than four percent errors, but maybe for your platform on Friday at 4pm, four percent error rate is actually okay
Will_Button:
and error rate is actually okay because you have so much traffic.
Chris_Cooney:
for some reason. And so what we say is you can set
Will_Button:
What
Chris_Cooney:
a
Will_Button:
we
Chris_Cooney:
more
Will_Button:
say
Chris_Cooney:
than
Will_Button:
is
Chris_Cooney:
usual
Will_Button:
you can say
Chris_Cooney:
alarm.
Will_Button:
a lot.
Chris_Cooney:
And you say actually, what I've
Will_Button:
Ie, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, a'r ddweud yma, mae'n 8% yn ymgyrchol.
Chris_Cooney:
got, what I've had this week, that's good. Let's roll with that. And then next week you get 8%, you'll be told about it. And so that's one of the ways
Will_Button:
Mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae
Chris_Cooney:
that we utilize machine learning, super powerful. What we often recommend is that that is kind of like your catch all safety net at the bottom of all of your alarms. That's the thing that's going to catch the weird stuff
Will_Button:
i'w ddweud yw'r ddweud yw'r ddweud yw'r ddweud?
Chris_Cooney:
you didn't consider the unknown unknowns, people often
Will_Button:
defined much better.
Chris_Cooney:
thresholds higher up. You can say, well, if this exceeds this, or this
Will_Button:
ac mae'n dweud y gallwch chi'n gwybod ychydig yn ymgyrch.
Chris_Cooney:
exceeds this, or whatever. And that way, you have a lot of known problems that you can cover with alarms, and then this nice safety net underneath all of it, using machine learning. And the second thing, there's
Will_Button:
yw'r gweithio'r cyfnod?
Chris_Cooney:
a bunch of different examples, just two that immediately come to mind. The second one is clustering the logs together. So you run a, OK, how many errors have I had in the past hour? 250,000. That's great. What do I do with that? 250,000 alarms. One of the things that we found,
Will_Button:
One of the things that we found, we've run like some meta-analysis of how to do it.
Chris_Cooney:
we've run like sort of meta analysis of how customers are using the platform and one of the things that we found is that 95% of all errors in a customer's dataset are caused by the same five problems. So 95%
Will_Button:
So
Chris_Cooney:
of all
Will_Button:
95%
Chris_Cooney:
alarms, error
Will_Button:
of
Chris_Cooney:
alarms,
Will_Button:
all
Chris_Cooney:
error logs,
Will_Button:
errors
Chris_Cooney:
everything are all from
Will_Button:
are
Chris_Cooney:
the same
Will_Button:
all
Chris_Cooney:
five
Will_Button:
from
Chris_Cooney:
root causes.
Will_Button:
the same file reported.
Chris_Cooney:
So we use something
Will_Button:
So we
Chris_Cooney:
that's
Will_Button:
used
Chris_Cooney:
called log aggregation
Will_Button:
to do the typical
Chris_Cooney:
or
Will_Button:
propagation.
Chris_Cooney:
a feature called log aggregation where
Will_Button:
of similar looks together.
Chris_Cooney:
logs that look almost identical with some minor variations, it will group all them together
Will_Button:
MBC 뉴스 김성현입니다.
Chris_Cooney:
and you can click in and see, well, this
Will_Button:
and
Chris_Cooney:
is the
Will_Button:
see
Chris_Cooney:
normal flow
Will_Button:
this
Chris_Cooney:
of
Will_Button:
is the
Chris_Cooney:
this
Will_Button:
normal
Chris_Cooney:
type
Will_Button:
flow
Chris_Cooney:
of error
Will_Button:
of
Chris_Cooney:
log
Will_Button:
this
Chris_Cooney:
and this is
Will_Button:
type
Chris_Cooney:
what you're
Will_Button:
of
Chris_Cooney:
currently
Will_Button:
our life.
Chris_Cooney:
seeing, superimposed on top. And likewise, you can click on each individual field and you can see, well, for this field
Will_Button:
will
Chris_Cooney:
in a
Will_Button:
pull
Chris_Cooney:
log,
Will_Button:
this
Chris_Cooney:
for
Will_Button:
field
Chris_Cooney:
this key
Will_Button:
in
Chris_Cooney:
in
Will_Button:
along
Chris_Cooney:
a log, for example,
Will_Button:
this field.
Chris_Cooney:
you know, Kubernetes.host, here is the number for this
Will_Button:
Here
Chris_Cooney:
host,
Will_Button:
is the
Chris_Cooney:
here
Will_Button:
number
Chris_Cooney:
is the number for
Will_Button:
of this.
Chris_Cooney:
this host, here's the number of logs that came out of this host and so on. And the power of that is that
Will_Button:
that is rather than strolling through and being
Chris_Cooney:
But
Will_Button:
okay. I think that
Chris_Cooney:
2,000 of them came from
Will_Button:
2000
Chris_Cooney:
this host,
Will_Button:
when
Chris_Cooney:
so
Will_Button:
it came
Chris_Cooney:
maybe
Will_Button:
to me.
Chris_Cooney:
that's a place to investigate. You just skip all of that.
Will_Button:
skip
Chris_Cooney:
How
Will_Button:
all
Chris_Cooney:
do
Will_Button:
about
Chris_Cooney:
you construct that
Will_Button:
how
Chris_Cooney:
query
Will_Button:
do you
Chris_Cooney:
stuff?
Will_Button:
construct
Chris_Cooney:
And you
Will_Button:
the
Chris_Cooney:
get straight
Will_Button:
query
Chris_Cooney:
into
Will_Button:
stuff
Chris_Cooney:
that. Okay, 2,000 from this host suite. This is our start, that's a problem, this host is broken. So yeah, those are two ways
Will_Button:
So yeah,
Chris_Cooney:
that we
Will_Button:
it's
Chris_Cooney:
utilize
Will_Button:
those.
Chris_Cooney:
machine learning. Like I say, that was our original, that was the original thing that made us, differentiated us back in 2017. Well before my time at the company. Now it's 250 so scaling quick
Will_Button:
in the wake. So is it safe to say that you munch those logs together?
Chris_Cooney:
Oh, absolutely. 100%. Nice use of the word
Will_Button:
Yeah.
Chris_Cooney:
munch, by the way. Well, I like it. It was good.
Will_Button:
Nothing else, I'm good for comic relief on the podcast.
Chris_Cooney:
Ha ha ha
Jillian_Rowe:
Alright,
Will_Button:
Well, cool,
Jillian_Rowe:
well...
Will_Button:
sir. head Jillian.
Jillian_Rowe:
I was just gonna say, this
Will_Button:
I'm
Jillian_Rowe:
is really
Will_Button:
just
Jillian_Rowe:
cool.
Will_Button:
gonna say,
Jillian_Rowe:
I
Will_Button:
this
Jillian_Rowe:
don't
Will_Button:
is really
Jillian_Rowe:
really
Will_Button:
cool.
Jillian_Rowe:
think
Will_Button:
I
Jillian_Rowe:
I
Will_Button:
don't
Jillian_Rowe:
have any other
Will_Button:
really
Jillian_Rowe:
questions
Will_Button:
think I have
Jillian_Rowe:
though.
Will_Button:
any other questions
Jillian_Rowe:
Is
Will_Button:
though.
Jillian_Rowe:
there anything else that you wanted
Will_Button:
Is there
Jillian_Rowe:
to
Will_Button:
anything
Jillian_Rowe:
add
Will_Button:
else
Jillian_Rowe:
or
Will_Button:
that you wanted
Jillian_Rowe:
anything
Will_Button:
to
Jillian_Rowe:
that
Will_Button:
add
Jillian_Rowe:
you didn't get
Will_Button:
or
Jillian_Rowe:
to talk
Will_Button:
anything
Jillian_Rowe:
about?
Will_Button:
you didn't get to talk about?
Chris_Cooney:
No, all good. Happy to do some great questions there, so thank you very much. But yeah, all good otherwise.
Jonathan_Hall:
I've enjoyed it too. Thanks for coming on and talking about it.
Chris_Cooney:
I hope you enjoyed it.
Jonathan_Hall:
Shall we go to pics?
Will_Button:
I think we should do some pics.
Jonathan_Hall:
He wants to go first today.
Will_Button:
I can go as soon as I get my phone open to get to my pick.
Jonathan_Hall:
Pulling mine up right
Will_Button:
Please
Jonathan_Hall:
now
Will_Button:
hold.
Jonathan_Hall:
too.
Will_Button:
All right, so my pick this week, I'll just jump right in. David Goggins, if you aren't familiar with him, he is a Navy Seal, Ultramarathoner. Just has a pretty incredible backstory. He released a new book about a month ago called Never Finished that kind of picks up where his first book left off. And it talks about continuing his Ultramarathon career to become a smoke jumper, which a smoke jumper is someone, a wildland wilderness firefighter that jumps out of the plane into the backwoods and fights the fire from within and works their way out. And so it's his quest to become that. It's a really cool book, very inspirational and motivational for me anyway. So if you like that kind of stuff, book. And I read the book but I've also heard that the audiobook is really really well done because he's actually in the recording booth with the narrator and then adds a lot of colorful context that's not in the printed copy of the book.
Jonathan_Hall:
When you said
Jillian_Rowe:
That
Jonathan_Hall:
it was
Jillian_Rowe:
does
Jonathan_Hall:
called
Jillian_Rowe:
sound
Jonathan_Hall:
never,
Jillian_Rowe:
fun.
Jonathan_Hall:
when you said it was called never finished, I thought you were talking about my personal life and how I never get anything done.
Will_Button:
Yeah,
Chris_Cooney:
Thank you.
Will_Button:
right. It's like, wait, David Goggins is a scrum master now?
Jonathan_Hall:
Yeah.
Chris_Cooney:
Ha ha.
Will_Button:
That guy
Jonathan_Hall:
Well I
Will_Button:
can
Jonathan_Hall:
can...
Will_Button:
do it all, man.
Jonathan_Hall:
Yeah.
Chris_Cooney:
If
Jonathan_Hall:
I can
Chris_Cooney:
he
Jonathan_Hall:
go
Chris_Cooney:
was
Jonathan_Hall:
next.
Chris_Cooney:
the scrum master for the team, I would, that would be terrifying. Absolutely terrifying.
Will_Button:
Right?
Chris_Cooney:
It'd be great. I get more work than ever, but I would, I would, yeah, the stress would be intense.
Will_Button:
Yeah, you'd be closing tickets just out of sheer fear.
Chris_Cooney:
Yeah.
Jonathan_Hall:
All right, so my pick for this week, actually I have two related picks. The first is a book I'm close to done audio reading. It's called A World Undone, the story of the Great World War, 1914 to 1918. So it's kind of an all-encompassing history of World War I. And the author starts out by explaining that there's been a lot of books and movies and other stories about World War I, but they're usually very narrowly focused on a particular battle or a particular person or a country. So this book tries to sort of be all-encompassing, explaining what led up to the war from the assassination until the war actually started a couple months later. So it's really fascinating, but one of the things that, and
Will_Button:
things
Jonathan_Hall:
it's long,
Will_Button:
that
Jonathan_Hall:
it's like the audiobook is 27 hours or something like that, so it's a long book, as you could expect for a book that tries to be all-encompassing about such a big topic. But it's a really good book,
Will_Button:
But it's a really good book.
Jonathan_Hall:
but it's also such a sad story.
Will_Button:
That's
Jonathan_Hall:
course.
Will_Button:
the worst sound, of course.
Jonathan_Hall:
But so often
Will_Button:
So
Jonathan_Hall:
throughout
Will_Button:
often
Jonathan_Hall:
this
Will_Button:
throughout
Jonathan_Hall:
war,
Will_Button:
this war,
Jonathan_Hall:
both sides made
Will_Button:
both
Jonathan_Hall:
horrible,
Will_Button:
sides
Jonathan_Hall:
horrible,
Will_Button:
made horrible,
Jonathan_Hall:
terrible mistakes,
Will_Button:
horrible, terrible mistakes.
Jonathan_Hall:
you know, like
Will_Button:
You know,
Jonathan_Hall:
assuming
Will_Button:
like, assuming
Jonathan_Hall:
that they
Will_Button:
that
Jonathan_Hall:
could just
Will_Button:
they could just
Jonathan_Hall:
charge the enemy and
Will_Button:
charge
Jonathan_Hall:
with
Will_Button:
the enemy
Jonathan_Hall:
enough willpower,
Will_Button:
in with
Jonathan_Hall:
they
Will_Button:
enough
Jonathan_Hall:
would
Will_Button:
willpower
Jonathan_Hall:
break through the front lines and or ignoring warning signs that the enemy was advancing or just whatever. Just terrible, terrible, terrible mistakes. I'm sure that happens in all
Will_Button:
happen
Jonathan_Hall:
wars,
Will_Button:
in
Jonathan_Hall:
but
Will_Button:
all wars
Jonathan_Hall:
it's just
Will_Button:
but it's just
Jonathan_Hall:
terrible. How many,
Will_Button:
terrible
Jonathan_Hall:
you know,
Will_Button:
how
Jonathan_Hall:
you
Will_Button:
many you know
Jonathan_Hall:
read
Will_Button:
you
Jonathan_Hall:
about 80,000
Will_Button:
read about
Jonathan_Hall:
people
Will_Button:
a battle
Jonathan_Hall:
died
Will_Button:
80,000
Jonathan_Hall:
because
Will_Button:
people died
Jonathan_Hall:
the
Will_Button:
because
Jonathan_Hall:
two people in charge
Will_Button:
the two people
Jonathan_Hall:
made
Will_Button:
in
Jonathan_Hall:
stupid
Will_Button:
charge
Jonathan_Hall:
mistakes.
Will_Button:
made stupid
Chris_Cooney:
Yeah.
Will_Button:
mistakes
Jonathan_Hall:
And so that leads to my second
Will_Button:
and so that
Jonathan_Hall:
pick,
Will_Button:
leads to
Jonathan_Hall:
which
Will_Button:
my
Jonathan_Hall:
is
Will_Button:
second
Jonathan_Hall:
going to
Will_Button:
pick
Jonathan_Hall:
be
Will_Button:
which is going to be
Jonathan_Hall:
any English
Will_Button:
any
Jonathan_Hall:
translation of
Will_Button:
english
Jonathan_Hall:
Sun
Will_Button:
translation
Jonathan_Hall:
Zoo's Art
Will_Button:
of
Jonathan_Hall:
of
Will_Button:
Sun
Jonathan_Hall:
War.
Will_Button:
Zoo's
Jonathan_Hall:
If
Will_Button:
art
Jonathan_Hall:
any
Will_Button:
of war
Jonathan_Hall:
of these people had
Will_Button:
if
Jonathan_Hall:
read
Will_Button:
any
Jonathan_Hall:
that
Will_Button:
of
Jonathan_Hall:
book,
Will_Button:
these people had
Jonathan_Hall:
World
Will_Button:
read
Jonathan_Hall:
War
Will_Button:
that
Jonathan_Hall:
I
Will_Button:
book
Jonathan_Hall:
probably would
Will_Button:
war
Jonathan_Hall:
have been
Will_Button:
war
Jonathan_Hall:
a lot
Will_Button:
one
Jonathan_Hall:
shorter.
Will_Button:
probably would have been a lot shorter
Jonathan_Hall:
So those are my two picks, A World Undone by G.J. Meyer and Son Zews Art of War.
Will_Button:
Thanks.
Jillian_Rowe:
I've always been amazed at sort of like the human
Will_Button:
so it's
Jillian_Rowe:
psychology
Will_Button:
sort of like the human psychology
Jillian_Rowe:
of war.
Will_Button:
of war.
Jillian_Rowe:
Like how does this
Will_Button:
Like,
Jillian_Rowe:
work? I'm going
Will_Button:
how does
Jillian_Rowe:
to convince
Will_Button:
this work?
Jillian_Rowe:
thousands
Will_Button:
I'm gonna convince
Jillian_Rowe:
of people
Will_Button:
thousands
Jillian_Rowe:
to just
Will_Button:
of people
Jillian_Rowe:
go
Will_Button:
to
Jillian_Rowe:
run
Will_Button:
just
Jillian_Rowe:
towards
Will_Button:
go
Jillian_Rowe:
death
Will_Button:
run
Jillian_Rowe:
like
Will_Button:
towards
Jillian_Rowe:
that
Will_Button:
death,
Jillian_Rowe:
way,
Will_Button:
like
Jillian_Rowe:
go
Will_Button:
that
Jillian_Rowe:
that way.
Will_Button:
way, go that way.
Jillian_Rowe:
How does that happen?
Will_Button:
How does that happen?
Chris_Cooney:
drugs
Will_Button:
Drugs.
Chris_Cooney:
mostly, alcohol was the weapon of choice for World War I, certainly World War II. Almost all wars
Will_Button:
Almost, almost.
Chris_Cooney:
after the brewing of beer, it's pretty much, it's amazing actually,
Will_Button:
Mae'n gweithio'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r
Chris_Cooney:
I'm a big, I'm a bit of a history fan as well, a big fan of British history and there's an endless set of examples
Will_Button:
endless set of examples
Chris_Cooney:
in modern war,
Will_Button:
in
Chris_Cooney:
Renaissance
Will_Button:
modern
Chris_Cooney:
war,
Will_Button:
war, in the Acehons
Chris_Cooney:
in antiquity,
Will_Button:
war, in antiquity,
Chris_Cooney:
even when the British
Will_Button:
even when
Chris_Cooney:
were
Will_Button:
the British were fighting the Romans, everything where we consistently messed up
Chris_Cooney:
absolutely
Will_Button:
strategies because we were not able
Chris_Cooney:
wasted
Will_Button:
to.
Chris_Cooney:
while we were trying to conduct like 10,000 person formations and things and we're just falling all over ourselves and it's genuinely fascinating
Will_Button:
It's genuinely
Chris_Cooney:
how and
Will_Button:
fascinating
Chris_Cooney:
in World War I,
Will_Button:
how,
Chris_Cooney:
World
Will_Button:
and
Chris_Cooney:
War II
Will_Button:
World
Chris_Cooney:
how
Will_Button:
War
Chris_Cooney:
much
Will_Button:
I,
Chris_Cooney:
amphetamines
Will_Button:
World War II, how much
Chris_Cooney:
like
Will_Button:
I'm
Chris_Cooney:
played
Will_Button:
fatigued,
Chris_Cooney:
a role
Will_Button:
like,
Chris_Cooney:
is
Will_Button:
played
Chris_Cooney:
basically
Will_Button:
a role.
Chris_Cooney:
just like
Will_Button:
It's basically
Chris_Cooney:
here
Will_Button:
just
Chris_Cooney:
take
Will_Button:
like,
Chris_Cooney:
this
Will_Button:
hey,
Chris_Cooney:
you feel
Will_Button:
take
Chris_Cooney:
really
Will_Button:
this,
Chris_Cooney:
good
Will_Button:
and
Chris_Cooney:
and
Will_Button:
you
Chris_Cooney:
run
Will_Button:
feel
Chris_Cooney:
that
Will_Button:
really
Chris_Cooney:
way
Will_Button:
good,
Chris_Cooney:
like
Will_Button:
and run that
Chris_Cooney:
ignore
Will_Button:
way. You
Chris_Cooney:
the bullets
Will_Button:
know the
Chris_Cooney:
enjoy
Will_Button:
bullets,
Chris_Cooney:
yourself you know it's crazy
Will_Button:
enjoy
Chris_Cooney:
how
Will_Button:
yourself,
Chris_Cooney:
much that
Will_Button:
it's crazy how
Chris_Cooney:
it's
Will_Button:
much that.
Chris_Cooney:
got people going.
Jillian_Rowe:
so it's a bar fight at
Will_Button:
So
Jillian_Rowe:
scale
Will_Button:
it's a bar fight
Jillian_Rowe:
is what
Will_Button:
at
Jillian_Rowe:
you're
Will_Button:
scale.
Chris_Cooney:
Oh,
Jillian_Rowe:
telling
Chris_Cooney:
basically,
Jillian_Rowe:
me.
Will_Button:
Is what
Chris_Cooney:
yeah,
Will_Button:
you're
Chris_Cooney:
essentially,
Will_Button:
telling me?
Chris_Cooney:
yeah,
Will_Button:
Yeah,
Chris_Cooney:
for
Will_Button:
essentially,
Chris_Cooney:
sure.
Will_Button:
yeah,
Chris_Cooney:
Especially World
Will_Button:
yeah,
Chris_Cooney:
War
Will_Button:
yeah,
Chris_Cooney:
One. The World
Will_Button:
for sure.
Chris_Cooney:
War One
Will_Button:
Especially
Chris_Cooney:
was
Will_Button:
World War
Chris_Cooney:
our
Will_Button:
I. World
Chris_Cooney:
first
Will_Button:
War I
Chris_Cooney:
attempt
Will_Button:
was the
Chris_Cooney:
at truly
Will_Button:
first
Chris_Cooney:
industrialized
Will_Button:
attempt to
Chris_Cooney:
warfare.
Will_Button:
truly industrialize
Chris_Cooney:
Like, before
Will_Button:
warfare,
Chris_Cooney:
then
Will_Button:
like,
Chris_Cooney:
it was
Will_Button:
before
Chris_Cooney:
in Napoleonic
Will_Button:
any of
Chris_Cooney:
Wars,
Will_Button:
that.
Chris_Cooney:
right? And that was just not even the same kind of, we're even close to that kind of scale, that kind of technology that was being employed. And so the idea, like, you mentioned that
Will_Button:
I
Chris_Cooney:
the
Will_Button:
don't
Chris_Cooney:
generals,
Will_Button:
imagine
Chris_Cooney:
like,
Will_Button:
that
Chris_Cooney:
if we just,
Will_Button:
the generals
Chris_Cooney:
we can just beat
Will_Button:
are like,
Chris_Cooney:
them with willpower. So like, yeah, kind of, I think during
Will_Button:
Yeah.
Chris_Cooney:
the Napoleonic War, we started to realize that willpower was less useful than particularly well-organized troops
Will_Button:
to be
Chris_Cooney:
in
Will_Button:
well
Chris_Cooney:
superior
Will_Button:
organized
Chris_Cooney:
technology.
Will_Button:
truth since
Jonathan_Hall:
Yeah,
Will_Button:
you're carrying technology.
Jonathan_Hall:
yeah.
Will_Button:
But technology
Chris_Cooney:
quite
Will_Button:
things have been
Chris_Cooney:
landed
Will_Button:
quite
Chris_Cooney:
there
Will_Button:
long
Chris_Cooney:
because guns and weapons and bombardments were still so temperamental whereas now
Jonathan_Hall:
Yeah.
Chris_Cooney:
first world war of course that becomes they're really reliable
Will_Button:
that they're
Chris_Cooney:
they're
Will_Button:
really
Chris_Cooney:
really
Will_Button:
reliable,
Chris_Cooney:
really efficient
Will_Button:
that really
Chris_Cooney:
really reliable
Will_Button:
do the fishing really well.
Jillian_Rowe:
Alright, well I'll go next. I'm gonna pick the third season of This Dark Materials.
Will_Button:
on.
Jillian_Rowe:
It's on Hulu, and I normally don't like book-to-TV adaptations, but I did really like this one. It, I don't know, it was really good. It stayed pretty true to the series, I thought. And... I had something else too. Oh,
Will_Button:
something else
Jillian_Rowe:
I'm gonna
Will_Button:
too.
Jillian_Rowe:
give my usual
Will_Button:
I'm
Jillian_Rowe:
disclaimer.
Will_Button:
gonna give my usual
Jillian_Rowe:
Don't watch
Will_Button:
disclaimer
Jillian_Rowe:
any of
Will_Button:
don't
Jillian_Rowe:
it
Will_Button:
watch
Jillian_Rowe:
if you
Will_Button:
any
Jillian_Rowe:
cry
Will_Button:
of it
Jillian_Rowe:
easily
Will_Button:
if you cry
Jillian_Rowe:
and you
Will_Button:
easily
Jillian_Rowe:
have stuff
Will_Button:
and
Jillian_Rowe:
to
Will_Button:
you
Jillian_Rowe:
do
Will_Button:
have
Jillian_Rowe:
that
Will_Button:
stuff
Jillian_Rowe:
day.
Will_Button:
to
Jillian_Rowe:
Like
Will_Button:
do
Jillian_Rowe:
just
Will_Button:
that day
Jillian_Rowe:
don't do
Will_Button:
like
Jillian_Rowe:
it.
Will_Button:
just
Jillian_Rowe:
That's
Will_Button:
don't
Jillian_Rowe:
my
Will_Button:
do it
Jillian_Rowe:
pro
Will_Button:
that's
Jillian_Rowe:
tip.
Will_Button:
my appropriate... That's fair.
Jonathan_Hall:
I guess I won't be watching that one then. I cry very
Jillian_Rowe:
No.
Jonathan_Hall:
easily.
Jillian_Rowe:
His dark materials
Will_Button:
No.
Jillian_Rowe:
has
Will_Button:
His
Jillian_Rowe:
to
Will_Button:
dark
Jillian_Rowe:
be crying
Will_Button:
materials
Jillian_Rowe:
like a
Will_Button:
has
Jillian_Rowe:
little
Will_Button:
to be
Jillian_Rowe:
girl.
Will_Button:
crying like
Jillian_Rowe:
Like
Will_Button:
a
Jillian_Rowe:
it's
Will_Button:
little
Jillian_Rowe:
just,
Will_Button:
girl.
Jillian_Rowe:
it's not
Will_Button:
Like
Jillian_Rowe:
okay.
Will_Button:
it's just,
Jillian_Rowe:
And then my
Will_Button:
it's
Jillian_Rowe:
kids
Will_Button:
not okay.
Jillian_Rowe:
come and
Will_Button:
And
Jillian_Rowe:
like
Will_Button:
then my
Jillian_Rowe:
troll
Will_Button:
kids
Jillian_Rowe:
me about
Will_Button:
come and like
Jillian_Rowe:
it.
Will_Button:
tell me about it.
Jillian_Rowe:
Would
Will_Button:
Oh
Jillian_Rowe:
you
Will_Button:
no.
Jillian_Rowe:
watch
Will_Button:
You just
Jillian_Rowe:
mom,
Will_Button:
found
Jillian_Rowe:
the
Will_Button:
the
Jillian_Rowe:
little
Will_Button:
little mermaid.
Jillian_Rowe:
mermaid?
Chris_Cooney:
Thank you.
Jonathan_Hall:
Oh.
Jillian_Rowe:
No. No, I didn't.
Will_Button:
No, I didn't.
Jillian_Rowe:
It's something else making
Will_Button:
It's
Jillian_Rowe:
me
Will_Button:
something
Jillian_Rowe:
cry
Will_Button:
else
Jillian_Rowe:
today.
Will_Button:
making
Chris_Cooney:
Okay.
Will_Button:
me cry today. Ha ha ha. Torrible.
Jillian_Rowe:
Ha ha ha! Alright, Chris, your turn.
Will_Button:
Chris your turn.
Chris_Cooney:
I'll try not to cry. So I I Would pick two the first is a book
Will_Button:
him.
Chris_Cooney:
by general Stanley McChrystal, which is called team of teams Which is a fantastic book all about Work, man and controls sit in any kind of scaled organization of people How how much sort of localized independent decision-making teams should be able to make? that I was partially responsible for helping to organize in some form. It was a very very good
Will_Button:
It was
Chris_Cooney:
guidebook
Will_Button:
a very, very
Chris_Cooney:
on
Will_Button:
good guidebook
Chris_Cooney:
backing
Will_Button:
on
Chris_Cooney:
off and letting people
Will_Button:
backing
Chris_Cooney:
do
Will_Button:
off
Chris_Cooney:
their
Will_Button:
of the...
Chris_Cooney:
thing and encouraging people to
Will_Button:
encouraging
Chris_Cooney:
be able to do their thing
Will_Button:
people
Chris_Cooney:
rather
Will_Button:
to be
Chris_Cooney:
than
Will_Button:
able to do the
Chris_Cooney:
me
Will_Button:
same
Chris_Cooney:
being
Will_Button:
thing
Chris_Cooney:
the
Will_Button:
than
Chris_Cooney:
sort of composer
Will_Button:
me being
Chris_Cooney:
or orchestrator
Will_Button:
so.
Chris_Cooney:
all the time which was certainly the first instinct. So really great book, strongly
Will_Button:
.y'r ffordd yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch
Chris_Cooney:
strongly recommend it or read if anything because General Stendham and Crystal was the guy that completely against Al Qaeda. So he was the one that turned it from American,
Will_Button:
Thank you.
Chris_Cooney:
essentially losing people left right and center and constantly losing skirmishes and really poor intelligence.
Will_Button:
and
Chris_Cooney:
He was
Will_Button:
I'm
Chris_Cooney:
the
Will_Button:
really
Chris_Cooney:
one with some of
Will_Button:
looking
Chris_Cooney:
the some
Will_Button:
forward
Chris_Cooney:
of the ideas
Will_Button:
to
Chris_Cooney:
that
Will_Button:
seeing
Chris_Cooney:
he details
Will_Button:
what we
Chris_Cooney:
in
Will_Button:
can
Chris_Cooney:
the
Will_Button:
do.
Chris_Cooney:
book
Will_Button:
So we had these
Chris_Cooney:
turned
Will_Button:
in the
Chris_Cooney:
that into
Will_Button:
book
Chris_Cooney:
really, really
Will_Button:
tonight,
Chris_Cooney:
efficient
Will_Button:
so
Chris_Cooney:
intelligence
Will_Button:
really, really
Chris_Cooney:
supply
Will_Button:
efficient
Chris_Cooney:
lines
Will_Button:
and efficient supply lines.
Chris_Cooney:
and incredibly,
Will_Button:
They're
Chris_Cooney:
incredibly
Will_Button:
incredibly
Chris_Cooney:
collaborative
Will_Button:
efficient.
Chris_Cooney:
teams that were spanning the tens of thousands of people with disparate missions, disparate
Will_Button:
for
Chris_Cooney:
goals,
Will_Button:
mission.
Chris_Cooney:
sometimes conflicting goals and somehow made that into a sort of self reconciling system.
Will_Button:
I
Chris_Cooney:
Absolutely
Will_Button:
think it's the first modern law
Chris_Cooney:
The
Will_Button:
in
Chris_Cooney:
second
Will_Button:
the world.
Chris_Cooney:
is a book that I've just finished rereading, Don Quixote, which I think is the first model novel ever written, but it was by Miguel de Cervantes, and it is generally just an affirmation
Will_Button:
generally
Chris_Cooney:
of
Will_Button:
just an affiliation
Chris_Cooney:
doing stupid
Will_Button:
of
Chris_Cooney:
stuff is
Will_Button:
doing
Chris_Cooney:
sometimes
Will_Button:
stupid
Chris_Cooney:
good, so
Will_Button:
stuff
Chris_Cooney:
just
Will_Button:
is
Chris_Cooney:
do
Will_Button:
sometimes
Chris_Cooney:
it anyway,
Will_Button:
good so just do it and living
Chris_Cooney:
and
Will_Button:
your own life.
Chris_Cooney:
live in your own fantasy for a while, why not?
Will_Button:
and see
Chris_Cooney:
Like
Will_Button:
for
Chris_Cooney:
be
Will_Button:
a
Chris_Cooney:
a
Will_Button:
long
Chris_Cooney:
bit delusional,
Will_Button:
while.
Chris_Cooney:
it's good for you. And that's the thing I take with my boy. I strongly recommend it, especially
Will_Button:
Felly, mae'n rhaid i'n ffawr i'r ffawr o'r ffawr, ond mae'n fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r f
Chris_Cooney:
if you're feeling like you're embarking on some stuff, and you need someone just to tell you to take the risk and give it a go. It's a long book, like a thousand pages, but it's great, awesome. But you finish it and you feel like you take this whole
Will_Button:
I'm going
Chris_Cooney:
new
Will_Button:
to finish it and put things
Chris_Cooney:
perspective,
Will_Button:
all in
Chris_Cooney:
a much more optimistic
Will_Button:
perspective,
Chris_Cooney:
outlook,
Will_Button:
which are
Chris_Cooney:
and
Will_Button:
optimistic.
Chris_Cooney:
ultimately that's what the book's about. So yeah, those are my picks.
Will_Button:
Yeah, those are my picks.
Jonathan_Hall:
Very cool.
Will_Button:
Right on.
Jonathan_Hall:
I agree that Team of Teams is an excellent book. It's one of my favorites.
Chris_Cooney:
Yeah.
Will_Button:
Cool. I believe we have a podcast.
Chris_Cooney:
Masha'Allah.
Jonathan_Hall:
too.
Jillian_Rowe:
I think so.
Will_Button:
I think so.
Jonathan_Hall:
I guess I control the button, don't I? I have to hit stop now. All right, here it goes.
Will_Button:
Yeah.