Chris Cooney is a Developer Advocate who works at Coralogix. He joins the show to talk about "Stream Processing and Why it's so Important for Observability". He begins by explaining what is stream processing and its significant role.

Hello everybody, we are back for another week of Adventures in DevOps and today we're going to talk about logs and long streamings and I don't really know a whole lot about that topic so we're going to kick it right on over to the guest as we like to do. So with us today we have Chris Cooney, did I say your name right?

 

Yes, mom, perfect.

 

Alright, great. Will Button.

 

Hey, what's going on?

 

and Jonathan Hall.

 

Hi, I heard we were talking about streaming. This is all about Twitch, right?

 

Yes, video games, Twitch, the new Zelda

 

Awesome.

 

game, that's it. That's what we're

 

That's

 

here

 

why

 

wish.

 

for.

 

I'm here.

 

Me too. Alright, Chris, so with that, um, you know, completely, uh, derailed

 

Yeah.

 

your topic for a moment there, why do you- Why do

 

Well,

 

you tell

 

now I'm...

 

us a little bit about yourself and whether or not you play video games and how that relates to streaming?

 

Yeah, okay, this is quite a challenge. Unfortunately, I'm now the guy that's not talking about gaming, which is the worst position to be in. So I'm Chris, I've been a software engineer for 10 years now. I began my career as a Java engineer, and then, which was fun. And then I moved into sort of front engineering, React, Angular, that kind of thing. And then eventually ended up maybe four or five years ago, getting into the sort of SRE and DevOps space requirements as well. So that was all the sort of work I was doing at Sainsbury's. Now I am out of the retail space which is nice and I am working with CoreLogix. CoreLogix is a full stack of availability provider. So we process logs, metrics, traces, security data and we render that in new, fun and interesting ways and we hope to, the big things that we're focused on challenging is breaking data silos. So people typically look at their logs, all their metrics, all their traces, as we blend the experience together. So you can jump from your logs, your metrics, your traces, and you can kind of move between them really fluidly. And helping customers deal with scale. So helping them approach the challenge of, you have a vast, vast amount of data. Holding onto all of it is extremely expensive. How do you archive it? How do you store it, compress it, do whatever you need to do with it? And moreover, how do you do that in a way that's completely accessible using the lowest cost storage very highly performant and effective queries and maintain data discoverability. So it's a big sort of space that we're in. And yeah, the reason for the topic today is that one of our big differentiators were built on a streaming-based architecture. So using Kafka, Kafka Streams, and unfortunately Twitch streaming kind, so sorry to disappoint anyone who's excited. And as an ancillary topic, yes, I do play games. Hence the chair for anyone who's watching the inevitable video. Mae'r cyfnodd yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdweud yn ymdwe

 

What?

 

Oh, good.

 

I'm, mine still seems to be going and has this recording button.

 

Mine is also claims to still be going. I don't know if that means anything.

 

Yeah, we all have red buttons except for you, Will. It only kicked you out.

 

Maybe you just need to rejoin.

 

Maybe daylight savings time, like screwed up the, maybe

 

the

 

there's

 

Y2K.

 

like multiple podcasts. Yeah, but I don't either, but that doesn't mean it doesn't screw things up because some people do and some people don't and my whole week it's just, okay. Well, why don't we keep going and then the editor will edit out that fun little segue for us. Oh, never mind. He's back.

 

Click.

 

I am. I jumped right into the recording mode, so okay.

 

Cool.

 

Um

 

Alright.

 

I think it might be useful to define what streaming means in this context. Because I mean, we're talking about logs, right? What

 

and

 

other

 

look.

 

way of logging is there than streaming it?

 

Yeah.

 

Help me understand what we're talking about.

 

So there's it's it's logs, metrics, traces and security data. It's the full sort of gamut of observability and telemetry data that one might expect. And so the typical approach that people, most companies take is they take, you send your data to them in some way. So via an API call via some sort of queue that sent kinesis or whatever. And then you, they take it, they store it in a giant data store. They will index it. some sense of it and then they will trigger downstream alarms, they will trigger, they will update metrics, they will update dashboards, they will train machine learning models, they will do all these things but ostensibly it's ingest, store, index and then process. This is a pretty normal data processing model. The problem with this model is twofold. One, maintaining this giant store at the start is really difficult. It actually means just from a pure a given byte of data is higher if you go with this model because you've got all of the additional processing that's associated with indexing and querying those indexes and rationalization normalization and this is if then the customer might decide actually I just want to drop this data you know it's really really common for this and actually block is often a delete process it's not actually I don't store this in the first place it's actually a delete process because the company is simply architected to store up front so the way streaming differentiates from that is is you ingest data into CoreLogix, immediately it goes onto a stateful Kafka stream that has persistent backup. This enables a few things. One, really, really early and very fast decision-making. So if you don't want data, it will never be stored, it will never be held in the database, it'll never be indexed, no metadata will be held about it, it will just disappear into the

 

Thank

 

ether.

 

you.

 

The second thing is that it enables you to make decisions about where you'd like that data to be stored. So what a lot of archiving, if you'd like as an example, a lot of archiving solutions do is you take your data, it goes into high tier storage or something, and then after a certain amount of time, you can send that off to the archive. Whereas actually what we found is that if data is useless or if data is very verbose and you never query it, holding it for two weeks isn't gonna change that fact before it goes into an archive. And so what we allow, what streaming observability allows you to do with that early decision making is it's been ingested by CoreLogix, route it directly to your archive. And the thing that CoreLogix can do differently the archive directly and the archive is actually held in S3 in the customer's account. And so you can query the archive directly and without having to re-index or decompress or anything like that. And so these are all things that are enabled by a sort of stream processing based architecture. And there's a series of non-functional things as well that aren't necessary for the customer. So for example, if you do a store first kind of approach, if your storage dies, features straight away. And if storage layers can be temperamental, there's also the scariest part to change, nor is there any architecture, is the stateful parts, because if they get corrupted, if the memory goes wrong, or if the actual drives die or something, you're in serious trouble. Whereas, so if your storage dies, it kind of goes. With a streaming-based architecture, all of that processing happens up front. Storage kind of happens at the end. your dashboards will still update, your metrics will still be generated. And then if the storage layer does have a problem, it might not be stored at the end, but all of the key features along the way are still functional. It's much more resilient, much less reliant on centralized storage. That kills that single point of failure that might exist in a sort of storage first architecture. This is a few of the reasons. So it's more cost optimal, it's more resilient. It's faster, alarms fire faster in streaming-based architectures than they do in storage first architectures. Hopefully they made the case well though.

 

I think so. Do you have any like cool stories where streaming logs has saved the day? Like somebody who was able to very quickly be like, oh no, something is not right here and go and find it. Oh wow, look at this,

 

Yeah,

 

it's just us now.

 

yeah, so dropping people.

 

Yeah.

 

Is there something wrong with the thing or the...

 

I know it is a little bit temperamental, I've noticed.

 

and I'm going to be back

 

I don't know why. It still seems to be recording for my end in years and it records

 

Yeah.

 

like separate, it records like separate audio streams and everything anyways.

 

like everyone's their own thing.

 

Yeah, yeah, everybody has their own audio and video, so it should be okay. Like we've had people drop off in the editor just, oh, now we're all back. And the editor just is brilliant and pieces it together. I guess I'm

 

Oh

 

not exactly

 

cool.

 

sure how that works.

 

Yeah, just the magic of video editing.

 

and his mantra.

 

Yeah. Yeah. You guys okay?

 

We were just talking about how like storage is the point of failure. It was Jonathan's point of failure.

 

Perfect. Perfect.

 

Ah, that's funny.

 

Yeah. I feel like when you ask me, do I have any good stories? There is a story currently unfolding.

 

Ooh, can we

 

That's

 

know?

 

good.

 

Can we know? Or is it like still

 

No, I

 

everybody's

 

mean this,

 

too sensitive about it?

 

this one right now, this is happening right now, stop

 

Oh

 

relying

 

yeah, okay.

 

so much on storage. This is great,

 

Alright.

 

I can't do anything better. Stream your data somehow. In this context, it doesn't really work because they have to store it somewhere.

 

Yeah, it gets stored locally and then uploaded. I think,

 

Yeah.

 

I don't know, I'm not like totally familiar with what's happening.

 

way for Jonathan to get back. How long have you guys been running this podcast for? All right.

 

Yeah, but I'm kind of off and on. Depends on how my meetings and whatnot are structured. So I'm like a part-time panelist.

 

how it goes.

 

Oh no, whatever, whatever, think such a thing. Why is that? What is this thing doing to me? Oh, something else.

 

I don't see any sign of problem on this side. on shows that's recording again in 1990.

 

been blacklisted well.

 

Can you, should I log in as the host and see if it works for me? Okay, you might have to log out, I'm not sure, but I'll try it here.

 

That's okay, stuff happens. Suffer engineer for long enough to see enough things break. So it's cool.

 

It's all dumpster fires. It's all just varying degrees of on fire.

 

Yeah, yeah, yeah, basically, as long as you're not the direct cause of the file, you know, still there with title and matches, then it's all good.

 

Even if I am, at this point it's like, ah, you know.

 

I'm human, what do you want from me?

 

Was it like data I'm gonna get sued over deleting? Probably

 

I know.

 

not. Like hopefully not.

 

It

 

It might

 

got.

 

be, but then...

 

Yeah, that's all the one way to find out, you know. I remember the first outage I ever caused and I panicked so much, panicked so much when it happened. And now it's like, this again.

 

Okay.

 

No.

 

I see everything

 

It sends

 

recording

 

like

 

but

 

an alert to Slack.

 

I have a message that says Will Buttons browser preventing recording ask Will Buttons refresh the page.

 

Yeah, well. Alright, you got the red dot. Let's see if this works.

 

Looks like it might be working now. I'm not sure. I see that you're recording, it says 0% uploaded. No, it says 99% uploaded, I think we're golden.

 

cool. So we refreshed the the Riverside happenings. Where

 

So where

 

were

 

were

 

we?

 

we?

 

We're, yeah, we were talking about storage being a point of failure. And then I asked you

 

Thank you.

 

if we have any cool, do you have any cool stories besides the one that we're currently experiencing

 

Ha

 

that's

 

ha.

 

like a little bit too existential for me at the moment. I don't think that we can do that one.

 

Yeah, sometimes the universe just conspires against you.

 

Yeah,

 

So

 

it really does.

 

I obviously talking about the internals goings on in the technology and core logistics might be, might be NDA breaching grounds, but I suspect that I can talk a little bit about my engineering background. We've built, so we internally, it's actually what led me to an observability company in the first place as an engineer was we built a Kafka cluster to process a great deal of sort of stream log data. mostly logs that we were trying to process. And because it was just such a volume of them coming from all these different teams and products, it was crazy. And what we found was, as soon as we introduced this cluster and we, the Kafka cluster, and we added stateful backups onto the topics, all of these missing logs were no longer missing anymore. Like that just stopped happening. And we did, there was never an intention. The intention was just, we just accepted a certain amount of logs were gonna be missing And that just stopped happening. Every single log was in place. It was never an intention. So we were trying to work out what's going on there. But obviously, the obvious answer now in reflection was really straightforward. Kafka has good mechanisms for making sure that message has been delivered where they should be delivered. And having that buffer in the way and not relying on a central storage to constantly say, keep up with what's coming on, make it more of a pull mechanism was much more resilient, much more consistent data. was far, far higher with a far higher quality of service that could offer, because we were part of the platform team, we were essentially servicing 2000 engineers, we could say to them we're absolutely confident, yeah do your worst, like set every application to trace level, it's fine, do what you want, like the level of confidence we had in the system went way, way higher, and it meant, it also meant interestingly that doing things like short patching of elastic search nodes, It was like, well, the nodes will go down, the queues will fill up for maybe a minute, a minute and a half, then it's everything recycles, they'll just start consuming from the queue again. And it wasn't massive message going out on Slack saying, hey, everyone, you would have experienced massive errors and then having to deal with that sort of fallout. Those are just the example there where it was literally just better, like it was just far, far better experience for the customer, it was better experience for us. Fewer wake-ups in the night, fewer scary deployments and that kind of thing. That was awesome.

 

I want to say I like the response when like there's no data lost and things are working. What's wrong here? Like

 

Yeah.

 

there must be some that's this

 

What

 

that's

 

do we

 

like a

 

do?

 

very typical engineering response. That's kind of worked correctly

 

Yeah,

 

the first time like

 

exactly.

 

no way. What's really happening here?

 

It's very true. Yeah, it was a general confusion of like, it was too good. It was one end where, well, we need to deal with the sheer number of errors and how slow and mapping indexing problems and all sorts of different issues that are occurring in open search. really inconsistent missing logs. It was like, wait, what? How has that happened? No one trusted it because we couldn't track the root cause. Once we did a course, it was fine. And like, I think because we were so skeptical at the start, it wasn't just like, oh, yeah, if you put a, if you put a scalable sort of cloud, like a scalable queuing or streaming mechanism before your, your data storage, it makes your data storage a bit more resilient. So, okay, well, that's relatively well understood computer science, but we just didn't trust it at the time. But yeah, of the before and after and experience in that directly. And obviously when I did the Chrologics interview and they were like, hey, so we do the streaming observability thing, I was like, I know all about that. I don't know. It was great to see a product actually form out of that. And it's a big product as well, it's doing really well.

 

That's great. That does sound much more reliable than my typical cat log file grip. This thing went

 

Yeah,

 

wrong.

 

yeah,

 

So

 

yeah.

 

it sounds much better. Are there certain types of industries that you tend to work with?

 

So we tend to do best, I think, with medium-sized companies. They're approaching the problem of they have too much data, and they need a sort of scalable partnership. And potentially the ones that have been burned by some of the less attractive aspects of the existing SaaS providers. So one example of this is we offer a 15-second support response time. So if a customer asks us a question, time is 15 seconds. We sign SLAs for two minutes just to be safe, but two minutes is still

 

Wow.

 

everybody else is like half a day a day. We have a customer support team that will respond. We have a median resolution time of about 57 minutes I think this week. So it's like we tend to respond much faster, we tend to resolve much faster than people respond. And when you couple those two things together, you've got these mid-sized companies that are scaling rapidly, they're trying to find their own way, they're trying to learn how they're going to go about. and growing their infrastructure, growing their product, having that kind of support is just really, really awesome. Excuse that, I just knocked over a card. This arm here is currently like cat defense. Yeah, sorry. Anyway, so the most important thing is recognizing that a lot of the customers have experienced two really unpleasant things. One is really bad support. And actually, especially with the SaaS observability, the solution to support is terrible. I can't name names as long as I get sued, but there are some big vendors out there that will leave tickets for weeks. And it's this Byzantine system of you have to have this specific kind of issue in order for a human being to look at it. Otherwise, you have to traverse some crazy AI I bought that the intern made last summer that was supposed to say winter money or something. It's nuts. So this is like one big issue. The other one is overages. The sort of prevalence of overages amongst SaaS observability. And it's like their business model at points. It's like, well, oh yeah, the game is this great rate, but every single year we get hit with an overage that's five times our base billable. And it's like, well, that's crazy. That's not the only way to do business. So we don't charge overages. And what we found yw'r gweithio'r gweithio, a'r gweithio'r gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n gweithio'n g

 

Cool. I guess the more you guys grow, just don't bank with SVB, I guess.

 

Oh God, no. That

 

No.

 

was terrifying. Luckily it was zero exposure for us, but it was... Yeah, I saw that. I was like... Obviously I'm from the UK, and I knew about the existence of SVP, but I didn't know just how far-reaching the consequences of that would be. And then I started hearing that the Prime Minister here is Um, so the prime minister here is like is currently in talk within talks over the weekend with Joe Biden and I think the I can't remember who else was the third person And they were just trying to like make sure that there was People were buying off the different arms of svb to make sure there wasn't this massive Bank run that occurred. It was crazy. I didn't really I didn't I never realized it was that It was going to be that bigger thing Yeah, since I was Was it was it like a national news kind of thing or?

 

Yeah, I was trying to convince my husband that I should be a stay at home like kid cat mom. He was

 

Okay.

 

not, he was not going for that plan. I was like,

 

Okay.

 

yeah, you know, might be out of business, might not be, we don't know. I guess we'll find out Monday.

 

I like how you tried to seize opportunity from that though. You were like, yeah, but I could just be, I could just chill now.

 

Yeah,

 

SPV

 

it's

 

trash,

 

just

 

yeah, but awesome.

 

my new lifestyle.

 

Yeah, absolutely.

 

Especially like my kids are in school too, so it's like, ah, you know.

 

Yeah. It's motor. Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

That's a good way to put it.

 

It's amazing.

 

What kind of pushback do you ever get when you start talking about streaming? Do you get naysayers or people who don't believe that it's the right solution? What kind of things do you hear there?

 

I'll be honest, I think that stream processing is still, I know Kafka is becoming more and more normalized in the industry and more and more people are using it, but I actually think that most people don't really know what the difference of a streaming architecture versus a sort of stateful or a batch architecture might be. And so most of the time actually what we find is an educational challenge. It's like a, they're like, great, you're streaming. What the hell does that mean to us? And so, and that's actually, I think would be the biggest thing that we come up against. And so to the point now where we were in internal sessions with people who aren't necessarily technically in the company, so that everybody in the company knows really well and really clearly what the difference is, what the pros and cons are, and why a streaming component in an architecture is generally a good idea from both a resilience perspective but also from a cost optimization perspective as well. And so yeah, I would say that's the biggest thing, it's the education piece. Mae'r problemau yn ymweld yn ymweld. Ond yw'r ysgrifennu sy'n gwybod.

 

You said pros and cons. What are some of the cons? Are there times when it doesn't make sense to do streaming?

 

So speaking from a sort of, in the past, I would say, introducing a sort of streaming component assumes that you're comfortable with eventual consistency. You know, you've got like a series of events piled up and you have to just chew through them at the speed that you can. And sometimes there are systems where, and also that you can assume the precedence of the data that you've got. So like first come, first served as a reasonable model, that might not be the case. if you've got the price of something constantly changing. You don't wanna queue those things because it's the latest price that is the most relevant. You don't wanna process any of the previous prices. And so you can get around that with different solutions, but a basic queuing model in that case wouldn't really work for you.

 

Yeah, yeah.

 

So I think it's possible to build any architecture efficiently using a streaming model. It's just about there are certain nuances that appear that you have to be very aware of depending on your engineering challenge, depending on what you're trying to achieve. Luckily for logs and metrics and things, most of the time it's about the story of what happened. It's not just about what is the latest current CPU of this box, it's what has been the trend in CPU of the past hour, where has that spike come from? Those are the more common questions that one tends to ask and therefore eventual consistency models work really well for that. well but those are like additions to the, they're some of those new ones that they talked about essentially.

 

That's really interesting. Cause I remember earlier you were saying, oh, it's not just like logs, but it's observability and metrics. And I don't think that like quite got into my head until you were starting to talk about CPU.

 

Yeah.

 

And then my brain went like, but that's not, how would you get that from a log? So yeah, it is, that

 

Yeah.

 

is like a nice intersection.

 

Yeah, it's the whole work. It's logs metrics, tracing and security. We started life in 2017 as a company and we were primarily log analytics. And our big sellback then was that we could use machine learning to process logs. We could cluster them together. We could find trends and variances across different values. of a certain log from a certain application and alert you when you deviate from that normal flow.

 

Thank you.

 

And those were some of the earliest differentiators. Now it's a much more full, larger platform that's all kind of catered around the, as they say, the three pillars of observability,

 

say the three

 

logs,

 

plus.

 

metrics and traces, and then security as well. We have a security offering for CEM, CSPM, and then we have a sort of managed services thing on the security side as well. You can, they will do like vulnerability scanning

 

with those.

 

Yeah, and this is

 

Yeah,

 

what I mean.

 

and

 

I think

 

this

 

one

 

is

 

of

 

what

 

the things

 

I mean.

 

that people

 

I think

 

do

 

one

 

is

 

of the

 

they,

 

things that people do is they...

 

with the metric side of things. So if you think about a typical metric solution is like Prometheus, maybe you're sophisticated and you've horizontally scaled Prometheus using Thanos and you've got that deployed across multiple clusters or something like that. And immediately, almost immediately with Prometheus, you run into the problem of like, all of these nodes are running out of memory, like straight away, like before you do anything, all of the nodes have run out of memory. And so and this is why I quite like a streaming architectures and be how we leverage s3 So we actually have a system for storing metrics in s3 and querying them absolutely blazing fast. It's really really quick But because it's s3, it's much much much cheaper than holding things in RAM holding things in You know a stall threads. I saw the day people using CPU for storage. It was the craziest thing we've seen Well, it was like the worst of all worlds. It was more volatile and more expensive But yeah, so it's much cheaper than RAM, far cheaper than SSD storage, far cheaper than even cheaper than most magnetic storage, S3 is by far the cheapest way of doing it. Problem is the throttling and the query limits. And we've kind of worked out quite clever system we're getting around that. So we can lift, I think one of our query things about 10 terabytes of data from S3 about 10 seconds. It's pretty awesome. And that's actually archived data in CoreLogix. So it's super fast, super powerful.

 

I'm really curious, can you talk about how you query S3 so quickly or is that a trade secret?

 

No, so I can talk a little bit about the, so we contribute heavily to a project called Apache Arrow, which is all about columnar data formats,

 

I know

 

making

 

that

 

sure

 

one, yeah,

 

that,

 

yeah.

 

yeah, so we contribute to that. We use the Data Fusion query engine, and it comes down to using a clever format. So we use a slide variant of Parquet, called CX data, Chorologix data. So those things optimize performance, And then it's a great deal of parallelization, essentially a great deal of using utilizing serverless infrastructure, issuing a lot of parallel requests to S3. Because the one thing S3 is, it's very, very consistent at high parallel scale.

 

Yeah.

 

But it doesn't allow these massive one off queries, or it doesn't perform particularly well on those. It usually batches them, and you end up having to do

 

However.

 

lots and lots of them in chain. So we utilize a great deal of parallelism to pull a huge amount of data. And then we have a custom language called Dataprime. language and the real beauty of it is that it allows you can do data discovery so you can process and change. You can aggregate data in stream. So for example you can pull out data and say tell me the three availability zones, the top three availability zones that have the most logs coming from them to give you an idea of activity coming from your different availability zones for balancing. You can do that on the fly and it will aggregate the data on the fly for you. All of that is enabled by essentially just a huge amount of parallelism, coupled with a great deal of cost optimization built into things like AWS Lambda and Fargate and things like that, so we kind of just pay for usage and nothing more. Yeah, I think that's a fair summary without...

 

Yeah,

 

yeah.

 

that's really cool. I wouldn't have thought of all that. Yeah, that's pretty sweet. I recently had the privilege of interviewing Matthew Topol from Voltron Data who works on Apache Arrow

 

Cool.

 

on a different podcast. And

 

Oh

 

it was,

 

wow.

 

I mean, I didn't know about the project until then and it was quite fascinating to hear

 

Yeah.

 

how that works and it's

 

Yeah, it's

 

a,

 

awesome. It's awesome.

 

yeah.

 

I guess not. It's certainly wouldn't be my area of expertise. But we have some engineers in the company who they talked to me about this stuff. And I'm like, sure, yeah, absolutely. Like, great sounds fantastic. Like,

 

Yeah.

 

I say that as a 10 year engineer, I like, I have a clue about 35% of what they're saying. And then they just lose me. They start talking about I was never good at data structures and algorithms. It's never been my like,

 

Right.

 

never been my topic. But

 

Yeah.

 

those guys are hot on it. They're really, So yeah, it's a great deal of computer science going on at Carologix actually, which has been a nice but somewhat haunting refresher of my degree, my university

 

If anybody's

 

years.

 

interested in listening to that episode, by the way, it's the Cup of Go podcast, the episode from February 13. If anybody wants to hear that interview with Matthew Topol, just for the listeners who might be interested.

 

So if someone, hey, I just wanna check is my mic working okay?

 

Yep.

 

Your mic is working, but I keep getting this notification that it's not recording. I don't know if that's still valid or not.

 

and there's Echo.

 

I think it might be resolved. I switched computers.

 

Okay,

 

Oh.

 

awesome.

 

So hopefully that's resolved. Although I feel bad for our editor at this point because

 

Yeah,

 

he's going to be

 

he's

 

like, dude,

 

earning his due this

 

I've

 

week.

 

got like 47 copies of your audio.

 

Ha ha ha.

 

What are we doing

 

Um,

 

here?

 

I wanted to ask what's, um, what's the onboarding process? Like if I, if I have a, an existing logging solution from some unnamed other party,

 

Yeah.

 

what's the transition look like to switch over to Coralogix?

 

Yeah, so it obviously depends on the tooling you've decided to utilize in the first place. But there are a few key things about our customer service. One we offer free onboarding. So if you have an account, you can go straight to customer support and say, hey, I need help integrating with CoreLogix. In 15 seconds you get a response. Usually very, very shortly after that you'll get an engineer and the engineer will sit with you and work with you to sort of install software in your cluster and deploy things If you have installed anything like Fluent D, Fluent Bit, Open Telemetry, File Beat, any of the really normal open source solutions, Prometheus, Open Search, then it's just a matter of shifting a bit of config and you will natively integrate with all those things. We natively integrate with all the cloud provider metrics, just like CloudWatch and AWS, for example. We have firehose integrations, direct integrations with CloudWatch, Lambda. Lambda layer, which will automatically export your CloudWatch metrics, all sorts of different stuff. So obviously it depends on your tech stack in terms of the actual technology you'll employ, but you'll do all of it with an engineer who spends all of their time helping people integrate and so you'll do it with an expert basically. And that's one of the things that we, I personally am quite proud of talking about because I've seen this functionality develop in the company and how they've trained the people, how they train the engineers on the job to customers on board and the platform is awesome. So the onboarding journey is super smooth because you're not really doing it on your own. There are, as we say, people that just sign up, start a plan, do their own thing, and they're quite happy. We have a bunch of documentation. There's a great deal of me speaking because I make all these videos that are embedded in the documentation. So I imagine somewhere out there, there are a series of people who are sick of the sound of my voice. But maybe you guys will end this podcast, we'll see. Yes, it's embedded in the documentation and also the platform itself has all these nice little tooltips. You hover over something and it'll tell you all about it. Cheat sheets buried everywhere for people to use. We have native integrations like 200 different tools. The chances are if you use it, we integrate with it. It's pretty smooth at this point from both the support and the engineering perspective.

 

Yeah, I do have to say as a CoreLogix user, like the in-app tooltips and help menus and links to documentation is actually really, really well done, which is hard to do in an application of that scale. And

 

Yeah.

 

there's numerous examples of seeing it done incorrectly. But inside of CoreLogix, it's pretty intuitive to get where you need to be and get help for the things get there.

 

Yeah, that's awesome. I'll definitely make sure the feedback gets back to the engineers and the product guys. They are great. They actually agonize over that

 

Yeah.

 

stuff a lot. So every time the new feature discuss, the big conversation is, you know, how cohesive is this with the other features, where this confused people? And then obviously, there's an old phrase, what is it? There's only two hard problems in computer science, like cash invalidation and naming things. And we, the naming things is we really agonize over it. a'n meddwl i'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r ddweud o'r d

 

It makes a material difference to the experience that this has had. Oh, for sure. Yeah. It's awesome

 

naming

 

to hear

 

things

 

that.

 

is terrible,

 

Naming things

 

I agree.

 

is terrible, I agree.

 

It's a real source of stress. I don't like it. So there's always somebody with a better vocabulary than you as well who's like, did you think to use this word? And I'm like, no, I didn't think to use this word. But great.

 

Yeah.

 

You blocking the source as you? No.

 

see you then.

 

Thank you. Yeah. Yeah. Or when engineers typically use very, very niche words that are technically correct, what you need to use addiction. It's like C++ programming and then a dictionary on top of it I learned a word, munging, was the very first word, my very first role in software engineering. And I was like, what is munging? What is that verb? And then it turns out it's like merging two things together.

 

again.

 

And it's sometimes used in data. And I never knew this. I

 

I

 

think

 

never

 

that's

 

do

 

the way the worst sounding

 

this. A,

 

word

 

the

 

I've

 

worst

 

ever heard.

 

sounding

 

I've never heard that word

 

word. I've

 

used

 

never heard

 

ever.

 

that word

 

Like

 

used

 

not joining

 

ever. Like, not

 

or

 

joining

 

merging or

 

or

 

anything

 

merging,

 

that everyone knows

 

or

 

about

 

anything

 

munging

 

that everyone

 

for

 

knows

 

some

 

about

 

reason.

 

munger. For some reason.

 

But when you actually look at the definition,

 

But when you actually look

 

precisely

 

at the definition,

 

what it

 

it's

 

was

 

precisely

 

doing, like exactly

 

what it was

 

what

 

doing.

 

it was doing.

 

Exactly.

 

So you can't argue with the correctness,

 

you with the correctness.

 

but yeah, the usability of the word is a lot lack.

 

I've always felt like munging was this word that saying the word kind of describes exactly what it's doing once you know that.

 

Yeah. Yeah. All of this I agree with. And that didn't stop me from spending like two weeks as a as a brand new software engineer frantically trying to work out is it this kind? What is munging me? What is munging and not wanting to ask anyone because I just seemed everybody else knew it turned

 

What's the

 

out that

 

algorithm

 

half the

 

for

 

floor

 

munging?

 

had

 

I don't

 

no

 

want

 

idea.

 

to get it wrong.

 

Yeah.

 

Great. Or as the new guy, you're like, these guys are screwing with me. They just made that word

 

Yeah.

 

up.

 

Yeah, yeah, yeah. They

 

reminds

 

thought

 

me

 

it

 

when

 

crossed

 

I was

 

my

 

a

 

mind.

 

when I was a waiter we would tell new new hires to go to the basement and find a can of steam.

 

Oh, yeah.

 

There was

 

Yeah.

 

no basement and what's a can of steam. Come on.

 

We used to tell, so this was back when I was a bartender a long, long time, maybe 11, 12 years ago now. And yeah, if we ever got new people in, it would always just be like, what was the one? It was like, can you go get the non-alcoholic vodka from the back? And they'd go into the back and they'd be there for ages, just hunting around

 

Yeah,

 

for

 

yeah,

 

non-alcoholic

 

yeah.

 

vodka. And then they'd come up to the front and be like, I can't find any. And it'd be like, you spent 45 minutes.

 

Ha

 

Well,

 

ha ha!

 

what would you do?

 

Huh.

 

That's

 

That's

 

what he

 

really

 

does. He

 

trying

 

does

 

We're

 

everything.

 

to

 

just

 

find

 

about

 

That's

 

to

 

it.

 

leave.

 

what he does. He doesn't really

 

Yeah.

 

love. Yeah.

 

It was brilliant.

 

So what kind of tips can you give to us that, like streaming logging and or just logging in general, like that's the back end of it. What kind of tips can

 

Yeah.

 

you give us to make it, to make your logging better or more useful? Like what kind of application side things can or should you be doing?

 

Yeah, so look, let's say you go with a build your own DIY pretty typical stack, open search for logs and permutative metrics and stuff. I think there are some pitfalls that everybody does. And I can say that I said one before, which is the typical model is hold the data in Elastic Search, for example, for two weeks, and then archive it. And like I said before, the problem with that is it makes the assumption that if you just

 

Thanks for

 

hold

 

watching.

 

onto data

 

on the point.

 

And so

 

So

 

that model I think is an

 

that

 

anti

 

model

 

pattern.

 

I

 

If

 

think

 

you

 

is

 

hope

 

not something

 

if it's

 

that's

 

useless at day one, the chances are that it's not going

 

chances

 

to be useful

 

are that

 

at day

 

it's not

 

14.

 

going to be useful for day

 

And

 

14.

 

so you just you've just spent

 

And so you've just

 

two

 

spent

 

weeks worth of money

 

two

 

essentially,

 

weeks worth of

 

and it's

 

money,

 

not

 

essentially.

 

it's not a huge

 

It's

 

amount per

 

not

 

log,

 

a

 

but

 

huge

 

it

 

amount

 

adds up

 

per log,

 

really,

 

but it adds

 

really

 

up

 

quick. And

 

really, really quick.

 

when you add on to

 

And

 

that, the overheads

 

when you add

 

of

 

on

 

maintaining

 

to that, the overheads

 

a large cluster

 

of maintaining a

 

and

 

large cluster

 

operating these

 

and

 

sort of extremely

 

operating

 

complex

 

the system

 

open

 

extremely

 

search

 

comfortably.

 

clusters is a skill into itself.

 

scale into itself. That would be the first thing. So as a general characteristic

 

as much

 

is

 

data as

 

keep

 

possible

 

as much data

 

out

 

as possible

 

of your open

 

out

 

search cluster.

 

of your

 

Like

 

open source

 

don't think that

 

Don't

 

everything needs to

 

think

 

go in

 

that

 

there.

 

everything is still in

 

Try and use your

 

there.

 

high-performance

 

Try and

 

storage

 

use your

 

for

 

high-performance

 

your things that you

 

storage

 

really, really

 

for your

 

need.

 

things that

 

So

 

you need.

 

error level logs and things of that nature, really important stuff. And try and find another low-cost place that you can hold stuff. So one example

 

So,

 

that people

 

for

 

have used

 

example,

 

in the past that

 

the

 

we've

 

people

 

seen

 

using

 

is

 

the class

 

storage

 

that we've seen

 

necessary,

 

is storage necessary

 

indexed into

 

index into

 

a parquet format,

 

a

 

index

 

parquet

 

it using

 

format, indexing

 

AWS

 

using

 

Athena,

 

the

 

for example,

 

query, for

 

and query

 

example,

 

it that way.

 

and query it that way. The query

 

seconds,

 

is maybe

 

40

 

30

 

seconds,

 

seconds,

 

sometimes

 

40 seconds,

 

a lot longer

 

sometimes

 

if you got a lot of

 

a

 

data

 

lot longer

 

in there.

 

if you know what they are

 

And

 

doing.

 

that's where it gets really the limits of this theory. But try and find

 

try

 

ways

 

and find

 

of cost optimizing

 

ways

 

early

 

of possible

 

and build that into your architecture, because otherwise you're going to rapidly run into some problems, especially around performance, but

 

throughout

 

later

 

the performance

 

on down the line around

 

board.

 

cost. And then in terms of,

 

in terms of

 

for us, we'll go up the stack a bit. So

 

Felly, yn ystod y cyfnod, rwy'n credu y Llorys Tosh.

 

in terms of your infrastructure, try and find the lowest touch maintenance freaking nightmare and be ready to trade off

 

a'r ffordd yw'r ffordd yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch

 

on some features for that. In my experience, especially on the collection side, you don't need super, super, super fancy, you can do all all singing or dancing sort of agents and things like that. Nine times out of ten, you're going to use the most basic features in them and hopefully never never think about it again, except the odd patch that you've got to do, you know, when there's like a log for shell update or something that you've got to

 

So, I strongly recommend it.

 

vendor, open source collectors, definitely, definitely. It gives you more purchasing power. You know,

 

interesting

 

if

 

power.

 

you don't like the vendor you're with, your code isn't absolutely littered with their code, so you can go and change. You can just switch config and move to

 

Thank

 

somebody

 

you.

 

else. But also because they're usually

 

also

 

better.

 

because they

 

They're

 

usually better

 

more well

 

than

 

maintained,

 

the

 

longer

 

one

 

lived,

 

in

 

battle

 

the

 

tested,

 

team, the

 

there's

 

one

 

no weird

 

in the left,

 

memory

 

that's what

 

leaks,

 

has to

 

they

 

be done.

 

perform more reliably. And then if we go up the level again,

 

ac ydych chi'n gwybod i'r gael, yn ymdweud, ychydig i'r meddwl ychydig i'r gael.

 

to the as possible into the log documents themselves. So don't

 

Felly, ydych chi'n gwybod i'r gael, mae'n gwybod i'r gael yn ymdweud i'r gyflwyno, yn ymdweud i'r gyflwyno.

 

just have a log that's like, hello, obviously log in like a JSON format, you know, make sure it's a machine readable format as nice and easily indexable, and then include lots of metadata

 

and include

 

fields

 

lots of metadata

 

in that

 

fields

 

log.

 

in that

 

So

 

map.

 

what that does

 

So

 

is it means

 

what

 

that your

 

that

 

data

 

does

 

can

 

is

 

be

 

it means that

 

sliced

 

the data

 

from

 

can

 

lots

 

be

 

and lots of

 

sliced

 

different ways,

 

from lots

 

it improves

 

and lots

 

the

 

of

 

cardinality

 

different maps.

 

of your data. And what this in turn does

 

this.

 

is it means that your data can be analyzed in lots and lots of different ways.

 

ac mae'n ddweud y ffordd yw'r ffordd o'r cyfransor.

 

And it's actually the thing that tools like open source, is dealing with high cardinality data. So those would be sort of three of the stacks of tips that I would say. There's obviously a whole

 

C'est un bon moment.

 

dark arts of sort of getting this stuff right, but really be aware

 

really

 

of the

 

be

 

three

 

aware of

 

major

 

that.

 

sources of cost. The storage,

 

storage,

 

hauling onto all this stuff,

 

holding

 

compute,

 

onto all this

 

keeping

 

stuff,

 

your queries

 

and Q.

 

fast. And the third one is the operational and overhead of just keeping this thing alive once it's big enough. Those are the three places where the cost is really going to get you. If you go with your own build, you build your own solution. With the SaaS

 

success,

 

world,

 

success,

 

I would

 

world,

 

say

 

and

 

it's all about

 

it's

 

the overages that

 

all

 

you want

 

about

 

to

 

the

 

avoid

 

number of

 

to

 

people.

 

be really, really aware of what data you're sending and

 

and

 

making

 

make

 

sure that your deal

 

me happy.

 

is reflective of the actual usage of your data. That's something we see a lot with people who've come over towards some other companies. As they say, well, our base amount was, let's say, it was $100,000

 

some hours.

 

a year. $1000 a year. And we're like, well, what was your base

 

what

 

deal?

 

was

 

And

 

your

 

then you

 

base

 

look at their

 

deal?

 

base deal,

 

And when

 

and it's

 

you

 

like,

 

look

 

well,

 

at the

 

your

 

base

 

base deal

 

deal, it's

 

was

 

like, will

 

a

 

your

 

quarter

 

base deal

 

of what

 

work?

 

you actually sort of use. So this was always going to happen. This was always like, your true cost was always going to be $400,000 a year. It's just that for

 

It's just

 

some reason,

 

that...

 

it's been divided into a base cost and an overage. And so we find that that's the kind of stuff, especially with SaaS

 

stuff

 

providers, that

 

especially

 

you need to be

 

as

 

aware

 

well.

 

of. Make sure your bill is really, really good.

 

Make sure that's

 

because

 

the

 

there's

 

police.

 

nothing worse than leaving a ticket for a week and a half and wondering whether anyone's listening to any other side, especially when you're paying a lot of money. Yeah, those are just some of the

 

it.

 

things.

 

Cool. I'm wondering,

 

Cool.

 

like,

 

I'm

 

what

 

wondering,

 

kind of machine learning

 

what

 

models

 

kind of machine

 

are you

 

learning

 

guys

 

models

 

using?

 

are

 

What

 

you guys

 

is their

 

using?

 

purpose?

 

What is their purpose?

 

I know you

 

I

 

said something like

 

know

 

whether

 

you

 

or

 

said

 

not

 

something

 

a log

 

like

 

is important.

 

whether or not a log is important.

 

Yeah.

 

Is there anything else?

 

Is there anything

 

So there's

 

else?

 

a bunch of stuff we do. We do anomaly detection. So we have machine learning models that are detecting flow anomalies. So it will, if for example, at 3pm on a Tuesday, you typically are receiving, I don't know, 100 error logs. The machine that the CoreLogix platform will learn that that is your normal profile. And you can set an alarm, a certain amount so I get a thousand more logs than I normally expect, tell me about it. That's one of the machine learning aspects that we have, what that enables us to do is it

 

do

 

answers

 

is.

 

the quite complicated question of dynamic alerting. Static alerting would be tell me when I have more than a thousand errors or tell me when I have more than four percent errors, but maybe for your platform on Friday at 4pm, four percent error rate is actually okay

 

and error rate is actually okay because you have so much traffic.

 

for some reason. And so what we say is you can set

 

What

 

a

 

we

 

more

 

say

 

than

 

is

 

usual

 

you can say

 

alarm.

 

a lot.

 

And you say actually, what I've

 

Ie, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, a'r ddweud yma, mae'n 8% yn ymgyrchol.

 

got, what I've had this week, that's good. Let's roll with that. And then next week you get 8%, you'll be told about it. And so that's one of the ways

 

Mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae'n ddweud yma, mae

 

that we utilize machine learning, super powerful. What we often recommend is that that is kind of like your catch all safety net at the bottom of all of your alarms. That's the thing that's going to catch the weird stuff

 

i'w ddweud yw'r ddweud yw'r ddweud yw'r ddweud?

 

you didn't consider the unknown unknowns, people often

 

defined much better.

 

thresholds higher up. You can say, well, if this exceeds this, or this

 

ac mae'n dweud y gallwch chi'n gwybod ychydig yn ymgyrch.

 

exceeds this, or whatever. And that way, you have a lot of known problems that you can cover with alarms, and then this nice safety net underneath all of it, using machine learning. And the second thing, there's

 

yw'r gweithio'r cyfnod?

 

a bunch of different examples, just two that immediately come to mind. The second one is clustering the logs together. So you run a, OK, how many errors have I had in the past hour? 250,000. That's great. What do I do with that? 250,000 alarms. One of the things that we found,

 

One of the things that we found, we've run like some meta-analysis of how to do it.

 

we've run like sort of meta analysis of how customers are using the platform and one of the things that we found is that 95% of all errors in a customer's dataset are caused by the same five problems. So 95%

 

So

 

of all

 

95%

 

alarms, error

 

of

 

alarms,

 

all

 

error logs,

 

errors

 

everything are all from

 

are

 

the same

 

all

 

five

 

from

 

root causes.

 

the same file reported.

 

So we use something

 

So we

 

that's

 

used

 

called log aggregation

 

to do the typical

 

or

 

propagation.

 

a feature called log aggregation where

 

of similar looks together.

 

logs that look almost identical with some minor variations, it will group all them together

 

MBC 뉴스 김성현입니다.

 

and you can click in and see, well, this

 

and

 

is the

 

see

 

normal flow

 

this

 

of

 

is the

 

this

 

normal

 

type

 

flow

 

of error

 

of

 

log

 

this

 

and this is

 

type

 

what you're

 

of

 

currently

 

our life.

 

seeing, superimposed on top. And likewise, you can click on each individual field and you can see, well, for this field

 

will

 

in a

 

pull

 

log,

 

this

 

for

 

field

 

this key

 

in

 

in

 

along

 

a log, for example,

 

this field.

 

you know, Kubernetes.host, here is the number for this

 

Here

 

host,

 

is the

 

here

 

number

 

is the number for

 

of this.

 

this host, here's the number of logs that came out of this host and so on. And the power of that is that

 

that is rather than strolling through and being

 

But

 

okay. I think that

 

2,000 of them came from

 

2000

 

this host,

 

when

 

so

 

it came

 

maybe

 

to me.

 

that's a place to investigate. You just skip all of that.

 

skip

 

How

 

all

 

do

 

about

 

you construct that

 

how

 

query

 

do you

 

stuff?

 

construct

 

And you

 

the

 

get straight

 

query

 

into

 

stuff

 

that. Okay, 2,000 from this host suite. This is our start, that's a problem, this host is broken. So yeah, those are two ways

 

So yeah,

 

that we

 

it's

 

utilize

 

those.

 

machine learning. Like I say, that was our original, that was the original thing that made us, differentiated us back in 2017. Well before my time at the company. Now it's 250 so scaling quick

 

in the wake. So is it safe to say that you munch those logs together?

 

Oh, absolutely. 100%. Nice use of the word

 

Yeah.

 

munch, by the way. Well, I like it. It was good.

 

Nothing else, I'm good for comic relief on the podcast.

 

Ha ha ha

 

Alright,

 

Well, cool,

 

well...

 

sir. head Jillian.

 

I was just gonna say, this

 

I'm

 

is really

 

just

 

cool.

 

gonna say,

 

I

 

this

 

don't

 

is really

 

really

 

cool.

 

think

 

I

 

I

 

don't

 

have any other

 

really

 

questions

 

think I have

 

though.

 

any other questions

 

Is

 

though.

 

there anything else that you wanted

 

Is there

 

to

 

anything

 

add

 

else

 

or

 

that you wanted

 

anything

 

to

 

that

 

add

 

you didn't get

 

or

 

to talk

 

anything

 

about?

 

you didn't get to talk about?

 

No, all good. Happy to do some great questions there, so thank you very much. But yeah, all good otherwise.

 

I've enjoyed it too. Thanks for coming on and talking about it.

 

I hope you enjoyed it.

 

Shall we go to pics?

 

I think we should do some pics.

 

He wants to go first today.

 

I can go as soon as I get my phone open to get to my pick.

 

Pulling mine up right

 

Please

 

now

 

hold.

 

too.

 

All right, so my pick this week, I'll just jump right in. David Goggins, if you aren't familiar with him, he is a Navy Seal, Ultramarathoner. Just has a pretty incredible backstory. He released a new book about a month ago called Never Finished that kind of picks up where his first book left off. And it talks about continuing his Ultramarathon career to become a smoke jumper, which a smoke jumper is someone, a wildland wilderness firefighter that jumps out of the plane into the backwoods and fights the fire from within and works their way out. And so it's his quest to become that. It's a really cool book, very inspirational and motivational for me anyway. So if you like that kind of stuff, book. And I read the book but I've also heard that the audiobook is really really well done because he's actually in the recording booth with the narrator and then adds a lot of colorful context that's not in the printed copy of the book.

 

When you said

 

That

 

it was

 

does

 

called

 

sound

 

never,

 

fun.

 

when you said it was called never finished, I thought you were talking about my personal life and how I never get anything done.

 

Yeah,

 

Thank you.

 

right. It's like, wait, David Goggins is a scrum master now?

 

Yeah.

 

Ha ha.

 

That guy

 

Well I

 

can

 

can...

 

do it all, man.

 

Yeah.

 

If

 

I can

 

he

 

go

 

was

 

next.

 

the scrum master for the team, I would, that would be terrifying. Absolutely terrifying.

 

Right?

 

It'd be great. I get more work than ever, but I would, I would, yeah, the stress would be intense.

 

Yeah, you'd be closing tickets just out of sheer fear.

 

Yeah.

 

All right, so my pick for this week, actually I have two related picks. The first is a book I'm close to done audio reading. It's called A World Undone, the story of the Great World War, 1914 to 1918. So it's kind of an all-encompassing history of World War I. And the author starts out by explaining that there's been a lot of books and movies and other stories about World War I, but they're usually very narrowly focused on a particular battle or a particular person or a country. So this book tries to sort of be all-encompassing, explaining what led up to the war from the assassination until the war actually started a couple months later. So it's really fascinating, but one of the things that, and

 

things

 

it's long,

 

that

 

it's like the audiobook is 27 hours or something like that, so it's a long book, as you could expect for a book that tries to be all-encompassing about such a big topic. But it's a really good book,

 

But it's a really good book.

 

but it's also such a sad story.

 

That's

 

course.

 

the worst sound, of course.

 

But so often

 

So

 

throughout

 

often

 

this

 

throughout

 

war,

 

this war,

 

both sides made

 

both

 

horrible,

 

sides

 

horrible,

 

made horrible,

 

terrible mistakes,

 

horrible, terrible mistakes.

 

you know, like

 

You know,

 

assuming

 

like, assuming

 

that they

 

that

 

could just

 

they could just

 

charge the enemy and

 

charge

 

with

 

the enemy

 

enough willpower,

 

in with

 

they

 

enough

 

would

 

willpower

 

break through the front lines and or ignoring warning signs that the enemy was advancing or just whatever. Just terrible, terrible, terrible mistakes. I'm sure that happens in all

 

happen

 

wars,

 

in

 

but

 

all wars

 

it's just

 

but it's just

 

terrible. How many,

 

terrible

 

you know,

 

how

 

you

 

many you know

 

read

 

you

 

about 80,000

 

read about

 

people

 

a battle

 

died

 

80,000

 

because

 

people died

 

the

 

because

 

two people in charge

 

the two people

 

made

 

in

 

stupid

 

charge

 

mistakes.

 

made stupid

 

Yeah.

 

mistakes

 

And so that leads to my second

 

and so that

 

pick,

 

leads to

 

which

 

my

 

is

 

second

 

going to

 

pick

 

be

 

which is going to be

 

any English

 

any

 

translation of

 

english

 

Sun

 

translation

 

Zoo's Art

 

of

 

of

 

Sun

 

War.

 

Zoo's

 

If

 

art

 

any

 

of war

 

of these people had

 

if

 

read

 

any

 

that

 

of

 

book,

 

these people had

 

World

 

read

 

War

 

that

 

I

 

book

 

probably would

 

war

 

have been

 

war

 

a lot

 

one

 

shorter.

 

probably would have been a lot shorter

 

So those are my two picks, A World Undone by G.J. Meyer and Son Zews Art of War.

 

Thanks.

 

I've always been amazed at sort of like the human

 

so it's

 

psychology

 

sort of like the human psychology

 

of war.

 

of war.

 

Like how does this

 

Like,

 

work? I'm going

 

how does

 

to convince

 

this work?

 

thousands

 

I'm gonna convince

 

of people

 

thousands

 

to just

 

of people

 

go

 

to

 

run

 

just

 

towards

 

go

 

death

 

run

 

like

 

towards

 

that

 

death,

 

way,

 

like

 

go

 

that

 

that way.

 

way, go that way.

 

How does that happen?

 

How does that happen?

 

drugs

 

Drugs.

 

mostly, alcohol was the weapon of choice for World War I, certainly World War II. Almost all wars

 

Almost, almost.

 

after the brewing of beer, it's pretty much, it's amazing actually,

 

Mae'n gweithio'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r hyn o'r

 

I'm a big, I'm a bit of a history fan as well, a big fan of British history and there's an endless set of examples

 

endless set of examples

 

in modern war,

 

in

 

Renaissance

 

modern

 

war,

 

war, in the Acehons

 

in antiquity,

 

war, in antiquity,

 

even when the British

 

even when

 

were

 

the British were fighting the Romans, everything where we consistently messed up

 

absolutely

 

strategies because we were not able

 

wasted

 

to.

 

while we were trying to conduct like 10,000 person formations and things and we're just falling all over ourselves and it's genuinely fascinating

 

It's genuinely

 

how and

 

fascinating

 

in World War I,

 

how,

 

World

 

and

 

War II

 

World

 

how

 

War

 

much

 

I,

 

amphetamines

 

World War II, how much

 

like

 

I'm

 

played

 

fatigued,

 

a role

 

like,

 

is

 

played

 

basically

 

a role.

 

just like

 

It's basically

 

here

 

just

 

take

 

like,

 

this

 

hey,

 

you feel

 

take

 

really

 

this,

 

good

 

and

 

and

 

you

 

run

 

feel

 

that

 

really

 

way

 

good,

 

like

 

and run that

 

ignore

 

way. You

 

the bullets

 

know the

 

enjoy

 

bullets,

 

yourself you know it's crazy

 

enjoy

 

how

 

yourself,

 

much that

 

it's crazy how

 

it's

 

much that.

 

got people going.

 

so it's a bar fight at

 

So

 

scale

 

it's a bar fight

 

is what

 

at

 

you're

 

scale.

 

Oh,

 

telling

 

basically,

 

me.

 

Is what

 

yeah,

 

you're

 

essentially,

 

telling me?

 

yeah,

 

Yeah,

 

for

 

essentially,

 

sure.

 

yeah,

 

Especially World

 

yeah,

 

War

 

yeah,

 

One. The World

 

for sure.

 

War One

 

Especially

 

was

 

World War

 

our

 

I. World

 

first

 

War I

 

attempt

 

was the

 

at truly

 

first

 

industrialized

 

attempt to

 

warfare.

 

truly industrialize

 

Like, before

 

warfare,

 

then

 

like,

 

it was

 

before

 

in Napoleonic

 

any of

 

Wars,

 

that.

 

right? And that was just not even the same kind of, we're even close to that kind of scale, that kind of technology that was being employed. And so the idea, like, you mentioned that

 

I

 

the

 

don't

 

generals,

 

imagine

 

like,

 

that

 

if we just,

 

the generals

 

we can just beat

 

are like,

 

them with willpower. So like, yeah, kind of, I think during

 

Yeah.

 

the Napoleonic War, we started to realize that willpower was less useful than particularly well-organized troops

 

to be

 

in

 

well

 

superior

 

organized

 

technology.

 

truth since

 

Yeah,

 

you're carrying technology.

 

yeah.

 

But technology

 

quite

 

things have been

 

landed

 

quite

 

there

 

long

 

because guns and weapons and bombardments were still so temperamental whereas now

 

Yeah.

 

first world war of course that becomes they're really reliable

 

that they're

 

they're

 

really

 

really

 

reliable,

 

really efficient

 

that really

 

really reliable

 

do the fishing really well.

 

Alright, well I'll go next. I'm gonna pick the third season of This Dark Materials.

 

on.

 

It's on Hulu, and I normally don't like book-to-TV adaptations, but I did really like this one. It, I don't know, it was really good. It stayed pretty true to the series, I thought. And... I had something else too. Oh,

 

something else

 

I'm gonna

 

too.

 

give my usual

 

I'm

 

disclaimer.

 

gonna give my usual

 

Don't watch

 

disclaimer

 

any of

 

don't

 

it

 

watch

 

if you

 

any

 

cry

 

of it

 

easily

 

if you cry

 

and you

 

easily

 

have stuff

 

and

 

to

 

you

 

do

 

have

 

that

 

stuff

 

day.

 

to

 

Like

 

do

 

just

 

that day

 

don't do

 

like

 

it.

 

just

 

That's

 

don't

 

my

 

do it

 

pro

 

that's

 

tip.

 

my appropriate... That's fair.

 

I guess I won't be watching that one then. I cry very

 

No.

 

easily.

 

His dark materials

 

No.

 

has

 

His

 

to

 

dark

 

be crying

 

materials

 

like a

 

has

 

little

 

to be

 

girl.

 

crying like

 

Like

 

a

 

it's

 

little

 

just,

 

girl.

 

it's not

 

Like

 

okay.

 

it's just,

 

And then my

 

it's

 

kids

 

not okay.

 

come and

 

And

 

like

 

then my

 

troll

 

kids

 

me about

 

come and like

 

it.

 

tell me about it.

 

Would

 

Oh

 

you

 

no.

 

watch

 

You just

 

mom,

 

found

 

the

 

the

 

little

 

little mermaid.

 

mermaid?

 

Thank you.

 

Oh.

 

No. No, I didn't.

 

No, I didn't.

 

It's something else making

 

It's

 

me

 

something

 

cry

 

else

 

today.

 

making

 

Okay.

 

me cry today. Ha ha ha. Torrible.

 

Ha ha ha! Alright, Chris, your turn.

 

Chris your turn.

 

I'll try not to cry. So I I Would pick two the first is a book

 

him.

 

by general Stanley McChrystal, which is called team of teams Which is a fantastic book all about Work, man and controls sit in any kind of scaled organization of people How how much sort of localized independent decision-making teams should be able to make? that I was partially responsible for helping to organize in some form. It was a very very good

 

It was

 

guidebook

 

a very, very

 

on

 

good guidebook

 

backing

 

on

 

off and letting people

 

backing

 

do

 

off

 

their

 

of the...

 

thing and encouraging people to

 

encouraging

 

be able to do their thing

 

people

 

rather

 

to be

 

than

 

able to do the

 

me

 

same

 

being

 

thing

 

the

 

than

 

sort of composer

 

me being

 

or orchestrator

 

so.

 

all the time which was certainly the first instinct. So really great book, strongly

 

.y'r ffordd yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch yn ymgyrch

 

strongly recommend it or read if anything because General Stendham and Crystal was the guy that completely against Al Qaeda. So he was the one that turned it from American,

 

Thank you.

 

essentially losing people left right and center and constantly losing skirmishes and really poor intelligence.

 

and

 

He was

 

I'm

 

the

 

really

 

one with some of

 

looking

 

the some

 

forward

 

of the ideas

 

to

 

that

 

seeing

 

he details

 

what we

 

in

 

can

 

the

 

do.

 

book

 

So we had these

 

turned

 

in the

 

that into

 

book

 

really, really

 

tonight,

 

efficient

 

so

 

intelligence

 

really, really

 

supply

 

efficient

 

lines

 

and efficient supply lines.

 

and incredibly,

 

They're

 

incredibly

 

incredibly

 

collaborative

 

efficient.

 

teams that were spanning the tens of thousands of people with disparate missions, disparate

 

for

 

goals,

 

mission.

 

sometimes conflicting goals and somehow made that into a sort of self reconciling system.

 

I

 

Absolutely

 

think it's the first modern law

 

The

 

in

 

second

 

the world.

 

is a book that I've just finished rereading, Don Quixote, which I think is the first model novel ever written, but it was by Miguel de Cervantes, and it is generally just an affirmation

 

generally

 

of

 

just an affiliation

 

doing stupid

 

of

 

stuff is

 

doing

 

sometimes

 

stupid

 

good, so

 

stuff

 

just

 

is

 

do

 

sometimes

 

it anyway,

 

good so just do it and living

 

and

 

your own life.

 

live in your own fantasy for a while, why not?

 

and see

 

Like

 

for

 

be

 

a

 

a

 

long

 

bit delusional,

 

while.

 

it's good for you. And that's the thing I take with my boy. I strongly recommend it, especially

 

Felly, mae'n rhaid i'n ffawr i'r ffawr o'r ffawr, ond mae'n fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r fawr i'r f

 

if you're feeling like you're embarking on some stuff, and you need someone just to tell you to take the risk and give it a go. It's a long book, like a thousand pages, but it's great, awesome. But you finish it and you feel like you take this whole

 

I'm going

 

new

 

to finish it and put things

 

perspective,

 

all in

 

a much more optimistic

 

perspective,

 

outlook,

 

which are

 

and

 

optimistic.

 

ultimately that's what the book's about. So yeah, those are my picks.

 

Yeah, those are my picks.

 

Very cool.

 

Right on.

 

I agree that Team of Teams is an excellent book. It's one of my favorites.

 

Yeah.

 

Cool. I believe we have a podcast.

 

Masha'Allah.

 

too.

 

I think so.

 

I think so.

 

I guess I control the button, don't I? I have to hit stop now. All right, here it goes.

 

Yeah.