Welcome to another exciting episode of Top End Devs! In this installment of "Adventures in DevOps," we dive into the world of cloud architecture and engineering with a fascinating discussion led by our hosts Warren Parad and Will Button, and joined by our special guest, Matt Lee. Matt, hailing from Wisconsin, is the driving force behind innovative projects like CloudWarGames.com, a platform designed to enhance DevOps training and hiring through engaging problem-solving scenarios. As we explore his journey, from coaching gymnastics to developing digital training ecosystems, you'll discover how Matt's experiences shape his unique perspectives on technical challenges, team dynamics, and the ever-evolving landscape of cloud solutions. Whether you're curious about the technical intricacies of infrastructure or seeking inspiration for your own career path, this episode offers a captivating look at the intersection of technology, creativity, and human connections. So, sit back, relax, and get ready to explore the world of DevOps in a whole new way! 

Will Button [00:00:01]:
There we go. What's up, everyone? Welcome to another episode of Adventures in DevOps. Warren, thanks for joining me again.

Warren Parad [00:00:10]:
Yeah. You know, I think you don't need to thank me every time I I decide to show up. I I know we said that we're gonna have, like, a maybe a cool fact at the beginning of every episode, and I try to come up with one. Although, I do believe last time, Jillian committed to bringing the fact. And I don't maybe it's a coincidence that she's just not here today. So, I will have to step in and, share something. I forgot the pin for my backup phone. And I didn't remember how many digits it was.

Warren Parad [00:00:37]:
And on Android, apparently you can keep going forever. So I was on a 50 attempts. I was I was trying to guess if this pin is and it wasn't like like, I have no I don't care about factory resetting it. But, you know, I just wanted to see, like, can I get it? And, actually, a 51, I finally got it. I had upped the pin the pin size to 12 digits, and, like, that was, like, a realization for me. So, you know, if you forget your Android pin, just keep trying. There's there's no there's no time I mean, there's a thirty second time out prep, per attempt, but other than that, you just keep going.

Will Button [00:01:09]:
So is that, like, manual brute force attack? Is that the category this would fall into? Yeah.

Warren Parad [00:01:16]:
For sure.

Will Button [00:01:17]:
Right on. Cool. I'm excited about today's episode. Joining us today, we have Matt Lee from Schematical and from CloudWarGames.com. Looking at your background here, Matt, cofounder of drawnby.ai, author of shiporgetoffthepot.com, and then gymnastics coach for eighteen years and out of Madison, Wisconsin. Welcome, Matt.

Matt Lee [00:01:46]:
Thanks. Thanks for having me. Yeah. Yeah. The shipper, getoffthepod.com, I'm not doing anymore, but, the, I actually did write a book on coaching gymnastics called hacking fear tips, tricks, and strategies how to help high performing athletes through mental issues. So fun fact.

Will Button [00:02:03]:
Right on. My youngest son did gymnastics from, I think, from when he was about 10 all the way up until his senior year and high school and his senior year. I mean, you're probably well aware of this. Like, the number of collegiate positions for men's gymnastics is very, very small. And so he knew that that wasn't gonna get him anywhere, and he wanted to participate in some kind of activity at school because from for his entire education, he would, get out of school. His mom would be waiting in the parking lot, and she would drive him an hour to the gymnastics facility, and he would work out for four hours doing his homework in the car. And that's that was his his life growing up. So his senior year, he wanted to do something that was school related, and that was the end of his gymnastics career.

Will Button [00:02:55]:
But it's a commitment, and and there's that so much mental to it.

Matt Lee [00:02:59]:
Mhmm. Absolutely. Yeah. The it's not an uncommon story. Unfortunately, men's gymnastics is just not doing as well these days. Too many competitors to this sport are cooler and other things.

Will Button [00:03:10]:
Right. For sure. Yeah. And and the I think the the commitment to it just kind of kinda just, like, self filters, you know, so that when you do get to those people that are there, like, you've got some very committed individuals.

Matt Lee [00:03:25]:
Yeah. They you don't you play football. You play soccer. No one plays gymnastics.

Will Button [00:03:30]:
Right. Fair point. Cool. So tell us a little bit about, actually, first, you it looks like you've been in Wisconsin for a long time. Are you a Wisconsin native?

Matt Lee [00:03:45]:
Yeah. Born and raised, traveled the world, but I keep coming back here. And and, about four years settled down, bought a house out of the countryside, a little north of Madison, and, been loving

Will Button [00:03:55]:
it. Right on.

Warren Parad [00:03:57]:
Madison is fantastic. I I lived there for two years, and I will say that I love being there in the summer. That's qualifier that

Matt Lee [00:04:06]:
I Summer and fall. Yeah. You can get this time of year, not not as nice.

Will Button [00:04:13]:
What's your current temperature today?

Matt Lee [00:04:15]:
Oh, it's actually really nice. I was outside earlier. I let the dogs out and I do some jumping jacks to wake up, get myself ready for the podcast, and that would be around 10 degrees. So it's it's actually very pleasant compared to the year 20 we were close to a week ago.

Will Button [00:04:30]:
Yeah. That's balmy. Right on. So, cloud I'm gonna mess up this the name. Damn it. Sorry about that. Cloudwargames.com. Tell us a little bit about that.

Matt Lee [00:04:43]:
And don't feel bad about messing with the games. I keep calling it CloudWatch games because I'm sure you just have to stare at CloudWatch all day long, AWS, Amazon Web Services. And, you know, with Cloud War games. Yeah. So this, goes back a little bit if you don't mind me going back. I had a team. I was running a I think I had, like, 11 people on staff, and I was like, I needed to come up with a training, idea, a project for that. So I I've come up with a couple different games over the years and get their tools to help out my devs.

Matt Lee [00:05:12]:
And so one of them was the predecessor to cloud board games, which is where I would take a whiteboard and draw a network map. They're like you know? And so I would I would do this Dungeons and Dragons style game with them. And what I did besides just straighten down all these different voices, I had hundreds of scenarios of things that had gone horribly wrong over the years that I would then throw at them. And I'd say something like, you know, give them a brief, which should be like, all of a sudden, your website's timing out. You know? Why is that? And they'd have to walk through and tell me how they want to solve it. And so this one, this case, security group switch. Somebody removed a security group, which is basically a firewall rule from one of the various instances or resources running there and started to sign out. So I've got all these other ones incident cannot deploy.

Matt Lee [00:06:07]:
And so it's you know, you don't get a lot of information a lot of times when people come to you saying, there's a problem. At the time, this customer service person saying, our guys you know, our our shopping cart is disappearing randomly for our ecommerce store. People all of a sudden, the shopping cart's empty. You know, so, using that and just their wits. And and we just walked through it as a dungeon master. And it was a pretty good training tool. You could really figure out who was, you know, thinking outside the box, who was, like, really analytical, or who just kinda I don't know. You know? But it turned out to be a great sales tool as well.

Matt Lee [00:06:44]:
You walk the client through this. They're like, that can go wrong. Yeah. That's good. Like, is

Speaker D [00:06:50]:
it like, that can happen?

Matt Lee [00:06:51]:
You're like, yeah. I can. What would you do? I don't know. I guess I'd call you. Alright. So that is how it started. Then, literally last Thursday, I got a mastermind group, and I was talking about some goals I got. I was gonna launch a community, for people to wanna learn AWS.

Matt Lee [00:07:08]:
I was telling them about my idea for war games, which is where we spin up actual infrastructure, database infrastructure. We got infrastructure's code. It's not like I have to manually do this, so I just spin it up. And then I basically knocked down a part of it or fill up a database or remove security, you know, something or DDoS attack myself in front of it. And, then people will the the participants, the HFC participants, which people with their cameras on ideally, will be sitting there and they'll be like, hey. They're they'll they'll they'll try and fix it. You know? And there's a couple different variables. I wanna I've got a beginner, intermediate, and advanced.

Matt Lee [00:07:45]:
So, like, the beginner one, maybe they would just talk to me and I'll fix it. I'm like, do you screen the advanced ones? I'll give them actual AWS keys and say, here you go. Go ahead and fix it. So there's just a yeah. I'm I'm trying to make it available to all ages. So I'm getting creative with it. But, yeah, that was the impetus of, oh, yeah. So Friday, I bought the domain cloudmoregames.com, and then I set it up and sent out a mailing list to my measly hundred subscribers, and I had the biggest reaction I've ever seen.

Matt Lee [00:08:14]:
I've got people signing up, coming out of the woodwork. I have no idea who they are. And, then I emailed you guys, and then you guys were like, hey. You wanna be on? So here I am five days later, and, this is it's going well.

Warren Parad [00:08:27]:
Well, I'm glad it's going, you know, so quickly for you. You know, to have a hundred different scenarios, that must have taken quite a long time. Like, I'm just thinking back throughout my career. And, like, I don't know if I could come up with, like, a hundred things that had gone wrong and have the whole scenario. I'm like, for sure more than a hundred things have gone wrong in my career. It's just, you know, having remembered them and or have written them down, like, that must have taken some effort on your part.

Matt Lee [00:08:48]:
I am a compulsive. I mean, I think I showed you guys in the preview. I am a compulsive note taker and journaler and whatnot. I actually have multiple journals that have journaled by the side of my bed journal here, journal there. I've got business notebooks, I guess. So I piece it together. And, you know, if I'm not if I write it

Speaker D [00:09:07]:
down in another scenario, if I

Matt Lee [00:09:08]:
if, something goes wrong, I also make a comment about it sometimes. So

Will Button [00:09:13]:
Right. Yeah. You shared some of your comics with us, when we were emailing back and forth. Is or is that are those something you publish regularly?

Matt Lee [00:09:25]:
I try and do them I was doing them twice a week, but now that my content schedule's changed and doing more video content publicly again, I'm probably only gonna do once a week, but we'll see. Right on.

Warren Parad [00:09:37]:
That's still a lot. That's, you know, 52 a year. That's still a lot of content.

Matt Lee [00:09:42]:
I'll I'll be honest. They're not all bangers, but some of them. So they can take off. You know, I printed out this this was the first one that broke a hundred thousand views on Reddit. So Oh, wow. I was I was pretty happy about that. This one was a banger too, but, don't get the coffee cup yet. I gotta work out the details on those.

Warren Parad [00:10:03]:
Do you find that through through your I mean, having having having journals of all of the different scenarios and what had happened, what went wrong, and how it was fixed, like, did that have a beneficial impact for your career, or did you just find that you are able to consume that information now to make the cloud war games?

Matt Lee [00:10:23]:
I I mean, I had thought and this this idea, I thought about coming back to you for a long time. So I I have been keeping I mean, a lot of times, now I mean, I would just write a deal to my daily emailing list. I if there's something that goes wrong or something I fear about, there's one that I want really run a snare through the, s three ransomware attack that's out there right now. I'm gonna I'm gonna attack myself with that. That should be fun. But, yeah, I write it down and and and mail it out, you you know, people. So now, yeah, I guess collecting all that data has been helpful. I mean, in the last year, I've been better I I hired a business coach, and I've been better at, sharing that information.

Matt Lee [00:10:56]:
You know? Before, I write it down just because I obsessively take notes. And then now it's like I'm actually finding and helping other people, get updates on Amazon, you know, get up there, you know, on on things I'm an expert in that I I enjoy doing. So, yes, I think it's got a beneficial impact, but I didn't really capitalize on that until fairly recently.

Warren Parad [00:11:16]:
Yeah. I mean, the fact you just mentioned this, like, I can't believe I haven't been keeping track of every single bug that I run into that has some sort of, you know, out as it turns out, you know, and then a story because I feel like there is some novel interesting thing there that is worth sharing. And, now I'm starting to think like, would I give this advice to even experienced senior engineers? Like, oh, you know what? Like start keeping a personal journal of these sorts of things. I mean, I usually talk about a business context. Like it's great to keep track of what you're doing from, like, a brag standpoint so that you can report on it during your, hopefully, quarterly performance evaluation. Like, you have a list already of everything you've done. And I think there's a lot to be said there, but I think there's another level here of, like, the interesting technical challenges that you had to face and how you sort of tackle them because, like, you've already found a a new opportunity based off of what you've written down to, you know, turn into a business.

Matt Lee [00:12:13]:
Yeah. I mean, it's it's a good habit to do after action reports, especially when you're training a junior or something like that. You know, they make mistake, you know, a fairly expensive mistake maybe. You know, they delete something they shouldn't. And then you got two options. You could slap on the wrist to say, you know, you're in trouble. Got it. Or you could say, what did you learn from this? We just you know, let's say I got a client, you know, their website goes down.

Matt Lee [00:12:34]:
Let's say it costs them a hundred thousand dollars an hour. Junior takes down that website for a while. They can look at it like, oh my god. We lost all this money. Or you just say, oh, we invested this much into our junior running the board lesson. Let's have them present that in an after action report to the whole team. So we got, you know, this is positive from it. Here's you're writing the SOPs for the business, standard operating procedures for the business.

Matt Lee [00:12:56]:
I don't know what not to do. You know? As as somebody try and push towards my, developers, no fail forward faster. Know where to fail at, where you can make mistakes at. You know, fail on staging. Nuke staging on the water and slow down our dev for a day, or, you know, identity for a day still costs us less than if you knew production for even a few minutes. You know? So learn where to fail, learn how to fail. And so, this is one of those ranks that you're worried about.

Will Button [00:13:26]:
Wanna come back to something. There's been a couple things you've said that have stood out to me, and I wanna come back to those even though they have nothing to do with cloudwargames.com. You mentioned that you're in a mastermind, and then you also talked about your journaling. And those are two of, like, the consistent things that like, if you study, like, high impact, high performing people, like, those two things come up over and over again. So how did you get into a mastermind, and what does that look like for you?

Matt Lee [00:13:59]:
I'm actually in several, and, we could do a whole other podcast in the consulting side of things and structure your business, which I'm gonna save for another time. But, there there's one more I am, a group coaching, where where they do live q and a's and stuff like that that aimed at solo, consultants expertise. People wanna put themselves as expert. I would never call myself the expert, but I've I've been doing this a long time. So that's a good one. I have direct access to the main community guy there. And the other one is the room of round robin, approach where I found that one, I think, through another community. Actually, a third community where it's actually got very limited interaction, but they do, things.

Matt Lee [00:14:43]:
And I can give you guys the name of some of the stuff you want later, but, I the the I'm actually starting. I'm hoping to start another mastermind group, but I've kinda deprioritized that for, CTO for, cloud of war games. You know? I don't even remember the name of that. Because that's that's got, well, you know, a lot of traction right now. But I would strongly advise that, especially if you're you're you're going out there and starting on your own and you're not part of a big team, if you're if you're starting your own consulting, you need that sounding board. I'm very fortunate to have a informal business partner that we have just we are always bouncing ideas off each other because it could be, you know, in a vacuum when you're trying to get clients. It can be, you know, seem very daunting, you know, seem very dark.

Will Button [00:15:27]:
Yeah. For sure. I've I belong to a mastermind that I've been in for well over ten years now. And it's it's been I it's been the difference in my career because, like, our sole purpose in that group is to call each other out on our bullshit and hold each other accountable, and it it's made a difference.

Warren Parad [00:15:51]:
Yeah. We actually we actually do a full episode dedicated to, Will's being called out on his bullshit. I mean, his

Will Button [00:15:57]:
because there's a lot of it. Like, those the guys in my mastermind, they have a full time job calling me out on my bullshit.

Matt Lee [00:16:05]:
Yeah. It's my most valuable people are the people that are the blunted with me. You know? They're willing to be like, stop building developer tools, you fool.

Warren Parad [00:16:15]:
Right. And and giving them away for free. I think that's Yeah. The the important caveat there.

Matt Lee [00:16:21]:
Yeah. So For sure.

Will Button [00:16:23]:
And then on journaling, is that a daily exercise for you?

Matt Lee [00:16:28]:
Yep. Now let the dogs out, and then it's pretty much straight to journaling. And then I've recently been trying to do more journaling in the evening. Actually, the, close the thought loop. I have a lot of trouble sleeping. And I I wake up in the middle of my at night, and I'm like, oh, all these things I should do is to break forward to bed. I try and write down anything that's that's open ended here and on a piece of paper, and I got sticky notes on the side of the bed that I'll, like, write up to do to do to do. And that way, I could be like, okay.

Matt Lee [00:16:56]:
Tomorrow, Matt, we'll get to this. I don't have to be thinking about it. It lets me sleep at night. So it's it's a very strong tool for me.

Warren Parad [00:17:03]:
Yeah. I think there were a nontrivial number of times where in order to sleep well at night, I had to write up a whole email, regarding a a situation and then just not send it.

Matt Lee [00:17:13]:
Yeah. Yeah. Not send it that I am. That that's

Warren Parad [00:17:17]:
the important, factor there. Yeah. For sure. When when you're when when you're frustrated, definitely don't click send. But, you know, it's because those those thoughts are going around in your head, and it's like, I'm constantly focused on whatever that is, whatever problem it is, whether or not I did the right thing in that situation. And sometimes just I mean, I know from a fact now I have to terminate those threads, whatever the thread is that I picked up and I started just getting it done, whether it's writing an email, I use, like, every single kind of scheduling tool available. Like, I will delay emails to myself. I will snooze them.

Warren Parad [00:17:50]:
I will send myself future emails. I'll use calendar events, anything to get stuff off of my radar and into some sort of, temporal delay so I can come back to it later when I'm, like, at the right time.

Matt Lee [00:18:03]:
Yeah. That's a big one. You talk about, you know, you're frustrated and you write that email. I I when I get frustrated with something or possibly even a colleague or something like that, I write it on a sticky note, and a month later, it's a comic. So I just you know, that's how I and I I don't send it to him necessarily unless it's, you know, funny.

Warren Parad [00:18:20]:
Yeah. No. That's great.

Will Button [00:18:22]:
Right on. So cloud we'll come back to Cloud War Games. You've got people signing up for it. Is it like, are you operating, like, scheduled head to head matches, or is it a a single player experience? What's the interface look like?

Matt Lee [00:18:39]:
I like to do it with small teams. So I I'm thinking right now, we've got four people in the hot seat. We're gonna experiment with our initial batch. So that means I've got two events. Four people could be on camera. Either mic's hot. And, well, hopefully, anybody else that wants to be in the wait list or whatever the gallery, they can chat quietly. But like I said, there's I'm gonna do three, tiers of it.

Matt Lee [00:19:01]:
And in the future, I might do four people sign up for big dinner, four people sign up for intermediate, four people sign up for advanced or some. But the you can kinda just hang around if you do the beginner one and you can listen to the intermediate and the advanced. But I don't want the advanced people solving the intermediate one instantly there. And so, the way I do it, is there'll be a pre kind of, let's let's just say it's an hour long session. First fifteen minutes, you get in there, you get your bearings. You can I'll send out, network diagrams using my pixel art network diagram software that I use for my YouTube videos and ahead of time, and I'll set up TerraForm ahead of time so then people can get a feel for that. But then, you get in there, maybe you set up your dashboards. You know, you figure out your firefighting tools for the first fifteen minutes.

Matt Lee [00:19:43]:
And then I do the brief, and I say customer service just called and said, you know, the that we we just sent out. This This is actually this is probably my first one. So preview for those of you guys I don't know when this is coming out, but preview for the first advanced one. Customer service shows up and says we just sent out, a lot of emails from our primary email address that you will get, and it's for, pharmaceutical ad of some nature. It's like that. And we don't do that. So what just happened? So and that happened to me. That did happen to me, and it also is a conference.

Matt Lee [00:20:17]:
So, you know, and then after that, let's just say fifteen minutes to a half an hour diagnosing. So you gotta do diagnosis first. And then from there, they gotta come up with solution. And, I'm not I'm gonna give read access to my AWS account to anybody on there, but I'm not gonna give you write access. So you get to tell me the solution. I'm not never give anybody else write access to that. And then from there, last fifteen minutes, after action report. You know, I kinda give them deep behind the scene, what did I do, and how could you have improved, you know, the speed at which you diagnose the situation.

Matt Lee [00:20:50]:
So

Will Button [00:20:51]:
Oh, that's cool. That's cool. Are you recording these for, like, a dish like, creating YouTube channel where people can watch it after the fact?

Matt Lee [00:21:04]:
Yes. I mean, one thing I thought about is that well, part of the need for this thing is that there are so many AI generated resumes out there. It's tough for people, software engineers, DevOps people to stand out in that stack, resumes. Also, so many people using AI code. And I'm not against that, but, you know, the coding standard coding tests don't work so well. So this, I want them to record it, and I want them to share that be able to share this with perspective, hiring people. You know, on the other side of this, if you're hiring people, how do you know that they're actually able to problem solve? Don't you wanna kinda get a feel for it? Don't you wanna see if they are collaborative in nature or are they more like, I'm gonna just go heads down and not tell anybody what I'm thinking. You know? You you you wanna get a feel for that.

Matt Lee [00:21:44]:
Maybe there's, you know, there's you want one or the other, but this could be a really powerful hire tool, I think.

Will Button [00:21:50]:
Oh, yeah. For sure. Yeah. Yeah. Because then you get, like, a little insight into what this what it's like to work with this person under pressure, which is everything that you wanna know before hiring a person but have no way to test it.

Matt Lee [00:22:03]:
Yeah. And then on the flip side, it's a great way to inoculate yourself to stress. I see so many developers and the, you know, the the CEO comes running in saying, we're losing money. We're losing money. They're screaming and everything, and they're just like, oh, they lock up. You know? And this inoculates you from the stress. You know? I mean, we talk about I mean, back to my gymnastics career. I I'm pretty good at dealing with this type of stress because in my gymnastics care career, if you're spotting somebody and you mess up, that you could significantly injure or harm an athlete.

Matt Lee [00:22:31]:
You know? This some rich guy loses a bunch of money, but no one dies. And so if you put that in that perspective, it makes it easier to handle this stressful situation.

Warren Parad [00:22:41]:
I mean, there's there's not a lot of opportunity to practice emergency situations in the Yeah. Attack. Right? And so the only time we're in that situation is when we really mess up. I mean, some companies do like, there is this idea in the industry. Oh, you should, you know, practice your runbooks, you know, databases being down, regional failures, etcetera. But I I think that a majority of companies, the ones who need to practice it the most because they're often too much in firefighting modes are exactly the ones who don't practice it. They can so, I mean, this this could be more than just a hiring tool realistically. This could be, like out of band practice for engineers at any company to sign up and be like, hey, you know, go through this equivalent, you know, live training.

Warren Parad [00:23:25]:
Right? Continue education on how to do these things. You know, what how you're gonna respond in that situation and then potentially be graded or get, like, feedback on, how they can improve.

Matt Lee [00:23:36]:
Yeah. I mean, I I I could I mean, I just see myself being bringing this into a large company and then looking at their infrastructure, setting up a drill similar to their infrastructure and knocking it over and doing trainings internally. I mean, there's there's all sorts of options. I mean, back to the kind of individual side, I'd love to have a leaderboard and championships eventually. I've got ideas for all sorts of fun stuff to do. But, yeah, there's I think there's a lot that can be done on this.

Will Button [00:24:02]:
Could be like the next next iteration of, like, e gaming, like the professional gamers.

Matt Lee [00:24:10]:
Yeah. You're actually not not just I am a gamer, obviously, but, you know, a little more productive than just,

Warren Parad [00:24:17]:
you know,

Matt Lee [00:24:20]:
Fortnite. Yeah. But I so I think I saw, Will, you're you're in the CrossFit. I can see the CrossFit games for, you know, ops people, for infrastructure people.

Will Button [00:24:30]:
Oh, yeah. For sure.

Warren Parad [00:24:32]:
Let let it well, I'm sure no one ever heard me say this before, but, you know, when I'm done with work, what I wanna do is go home and play another game. Like, play a game that's exactly the same as what I do for work every day, you know, with with high higher stakes potentially than what I'm doing at work.

Matt Lee [00:24:49]:
Yeah. That's that. It's I'm hoping people do that so far. I mean, it seems like people are interested.

Warren Parad [00:24:56]:
Yeah. No.

Will Button [00:24:57]:
I think it's exciting. Been

Matt Lee [00:24:58]:
great. Yeah.

Will Button [00:25:00]:
And I I think it's like well, I'm I'm gonna make that a little stronger statement. I don't think I know that there are a lot of people who are early in their career, and DevOps is one of those industries that it's so hard to get into because you can't get into DevOps without having DevOps experience, and you can't get DevOps experience without working in DevOps. And this actually gives you a way to solve that problem.

Matt Lee [00:25:29]:
That is Thursday's comic, by the way. You have to have five years of experience for this entry level job or whatever. How do you have chicken and egg? This might solve that problem, though. You know?

Warren Parad [00:25:41]:
Yeah. So I understand. I I think it's sort of the same thing, though, where, like, people don't leave the university, and go directly into executive role where realistically part I think one of the most important aspects of of DevOps is sort of understanding the connection of the the business and what you're trying to execute and running it and how that impacts the technology and what it actually means for uptime and, providing that as a value add to whoever your users or customers are. And I think, you know, one of the huge challenges there is that's just not really something you're taught in any level of education system. Like, you don't you don't go to your chemistry class or your physics or, you know, calculus and be taught about how that provides value to end users somewhere and how critical that is. I mean, because if you were, then you could sort of maybe make the jump there. But I feel like that's one of the biggest things, the lack of experience, but also, the lack of attention to how to utilize those skills in a real world environment. And I think that, like, the email sent out that has a business impact starts to get like, scratch that.

Warren Parad [00:26:50]:
It's not just that there's a emergency situation. It could just be about a debugging experience, like, how did this happen? But understanding that there is a real impact to real customers somewhere, that someone will care and money is getting lost. I think that's the sort of thing that a lot of people are insulated throughout their academic career.

Matt Lee [00:27:07]:
Yeah. Yeah. It's the the I see engineers making a lot of decisions based on ones and zeros and not dollars and cents. But I like that tagline. Great. But I see

Speaker D [00:27:18]:
it I see it happening. And as a business owner, you know,

Matt Lee [00:27:21]:
I I I have to take into account the bottom line. Recently, I had a major outage, for one of my clients right before the holiday season, and we had a triage. We actually had to decide to have a bunch of different domains kind of it's not Coke or Pepsi or anything, but think about how, like, Pepsi owns Mountain Dew and it owns KFC or whatever. It owns, Taco Bell. One of those mister Pibb was down and it was causing this to the traffic mister Pibb was causing every other site to have problems. It ripples across the whole system. And we had to triage that and say, listen. We don't sell that much, mister Pibb.

Matt Lee [00:27:57]:
Shut it down. We don't care. You know? So you gotta it's it's interesting. We need to do that in real time, little firefighting. That that was a pretty exciting day. But same type of concept I see with the open source software where versus paying Amazon. If you're a big company, you've got thousands of people, and you can save millions of dollars. It makes sense to host your own cluster of x y z.

Matt Lee [00:28:17]:
But if you're a smaller company and you you you forget the the engineers forgetting that, oh, yeah. My man hours are being wasted maintaining this thing, and we're only getting an ROI of one third of what my man hours are spent. You know, what they're paying me to maintain this cluster, then what are we doing it for? You know? So that's that that's a very important skill that, you know, you're they don't teach in school.

Will Button [00:28:40]:
Yeah. For sure. I think it's something I've said on this podcast at at least a billion times. Like, no company cares about how tight and compact and efficient your Dockerfile is. And because they only care about what value it's providing to the customers who are paying money to that business. And I think that's the biggest career advice anyone in this industry can learn is your technical skills are a tool used by the business. They're not the product.

Matt Lee [00:29:15]:
Mhmm. And and and to be clear, I'm not saying there's one right answer for the other. You gotta take into account the bigger picture and say, you know, we're a company this size. You know, how much time should I be spending on this, or or is that moving the needle for the company? You know? Right. Because you always gotta pick what, you know, is important there. So sorry. I didn't mean to interrupt you.

Will Button [00:29:35]:
No. I was done. That was it.

Warren Parad [00:29:37]:
I mean, I I think this goes in another direction as well, and I'm sort of curious your perspective, Matt, is that right now, I know there's a allegedly, the job economy is not great for knowledge workers, and I'm I'm curious if you see particular opportunities in some way or really, like, how do you stand out? What what are there particular things that do help, engineers in this area or throughout the industry stand out compared to you know, we talked about AI generated resumes, or, you know, other aspects that they could really lay on. Any any thoughts on that?

Matt Lee [00:30:16]:
Yes. Tons. The the first thing is that with with I'm actively talking to several companies that are creating large language models, LLMs, to basically replace, me and and my designs so they can just put in words and then it's been it's up Terraform and it goes. There's they're not there yet. They're a long way from it, be able to maintain it, but they'll get there. So that means I'm out. That's my whole job there. But just like elevator operators back in the day.

Matt Lee [00:30:49]:
When when's the last time you saw an elevator operator? Been quite some time. You know? So the who wants to stand in a hot box with People's, you know, enclosed space like that, especially after COVID and everything. Oh my gosh. No. So you have no no more elevator operators, but they need engineers to maintain those things. You know? You don't those things fall apart. That could be real bad for you. So you've got to evolve.

Matt Lee [00:31:12]:
That's something that I can't emphasize enough is to evolve and and change your skills. You know, holding on to that position. Like, okay. I'm I'm gonna be the one typing in the code for CSS or pixel pushing. You know? No. You want to be a step above that, and you wanna find that next step. So, things I do to stand out, I mean, I I make sure they see my face. I do a ton of content on stuff.

Matt Lee [00:31:36]:
I try and, you know, basically, I mean, I'll be honest with my content. Sometimes I'm just reading in the news and that way they see my face and they could recognize me there. But that's something I personally had to get over. I'm an introvert oddly enough. I probably don't seem it but I introvert by nature, extrovert by necessity. So, you know, don't be afraid to make videos about stuff. Some of my videos one of my videos from it's always bugged me. I did a video where I was trilaterating Wi Fi signals, in between projects just to track down printers and different things, just because I was bored out of my mind, and I put it on YouTube.

Matt Lee [00:32:10]:
And it's still one of my most popular videos, and I don't do that. I mean, I have no cap at all. So Absolutely. Stuff like that. You know, the communication, that's a huge skill. You need to I mean, even if you're typing into an LLM, some guy in a suit is gonna come up and say, hey. We need this bigger picture thing. And then you can sure go ahead and type to our army army of LLMs and have it generated.

Matt Lee [00:32:30]:
But if they can't do that prompt, there's values to you doing that. So find a way to bridge that gap between human and machine. It's just we're just changing the way that works. You know? So find ways to add value to other people. That's one thing I obsess over my clients. It's like, I'll help my clients, like, source other developers. Now that this tool here, my winners, I'm gonna send right to my clients. Be like, you need a guy? You know, I need a girl.

Matt Lee [00:32:53]:
You need a person, whatever. Just here you go. This is this one just won. This one solved my hustle in two seconds flat. So, yeah, that's that's what I would do. Yeah. Communicate. That's it.

Matt Lee [00:33:04]:
You know? You you give them a wide service there and don't communicate the same way everybody else is. There's a reason I use pixel art everywhere. People remember it. My my I love my clients' death. They they gained as kids. You know, we're perfect as a game as kids. They remember, you know, Mario Bros. And so they see that, and they remember it's a standout and and and and communicate, you know, quite a bit.

Warren Parad [00:33:27]:
I I love the perspective on spend some time creating content, because, you know, it really does resonate with the there's only so much you can do of practicing LeetCode interviews or learning a new programming language, but, you know, really take that next step of doing literally anything else. You know, if everyone's focusing on tech, focus on creating content or making some artwork or, you know, using LMS or whatnot or hand drawn or, you know, making comics, because that is some way that you can really stand out. You can put on your resume. Oh, yeah. And also all of these comics are me. Right. You know, this is something about me that really stands out. That's unique for who I am compared to everyone else.

Warren Parad [00:34:06]:
Who's trying to compete for the, for the same job. I think right now it's it's, I think this is always true. It's always now is the best time to say, even start your own company. I see like a lot of resumes are like practice projects by engineers who are interested like, oh, I made a recommendation engine or I made this other thing. And the most important question I always ask is why did you do that? Like, what was your end goal here? And it's like, oh, I just wanted to do a project. I'm like, okay, well that doesn't really mean anything to me. You had no bounds, no, no restrictions, no, constraints on what you were doing. You just went wherever you felt like it was way more interesting as, oh, I was actually trying to make a business or I was actually trying to automate my, my pictures being uploaded to LinkedIn or whatever.

Warren Parad [00:34:47]:
And like, I had a real problem that I actually needed to solve, and this is how I went about solving it. That's a way more interesting thing, to hear as a potential interviewer. So, you know, the content's great. The webcomics are great. You know, anything that really makes you, stand out as an individual.

Matt Lee [00:35:02]:
Yeah. Another another thing that's popped in mind is, you ever hear the t skill set? You know, you don't wanna be a generalist. You know, if you can find a niche, a niche that's in demand, you know, then the folks are I think my AWS niche for Microsoft and business is too wide. I'm trying to figure out how to, like, zero that in. Like, maybe I'm just gonna do WAF, you know, or now that I've got these trainings, I'm probably gonna focus on that. But, you know, I was thinking about just being the WAF command or web application firewall because I fight DDoS attacks all day long, and I could do a whole course on that. I know we'll teach a whole course on that. So, but, yeah, find a niche that's in demand and and dive on in.

Matt Lee [00:35:39]:
And, I I I've got a discord full of, hodgepodge of brand developers, and they have these really cool projects. But they that are niche, and they they just don't showcase them. They tell you about it in tax. It's like, you've got cameras everywhere. Your computer, you can record the screen. Just screen record it and show this cool project, you know, off as much as you can.

Will Button [00:36:03]:
Yeah. Yeah. It's definitely, we're we're moving from, like, a a technical marketplace to a creator marketplace, and it's I think what both of you are saying really highlights that. It's that the technical skills are the foundation, but then as the creator, you have to to showcase and and market those to make you stand out and then identify, like, here's the thing that I do. Like, on the the t, like, you've got the top line of the t here where everyone has the same skill set, and then here's where I go deep. Mhmm.

Matt Lee [00:36:41]:
Yeah. I'm not saying don't, you know, don't learn things outside of your purview, but don't make that your public facing persona. You know, you I I I know how to do a lot of things. I could do WordPress. And actually not to anybody ever, but, you know, stay with I I publicly, I I have a very particular thing I talk about, you know, and you got a very short amount of time to get people's attention right now with, you know, all the spam and garbage out there. So you've got to figure out a way to do that very quickly and and find people that have your problem. Not everybody you know, I I solve a million dollar problem for some of my clients. Not everybody has those problems.

Matt Lee [00:37:18]:
You gotta find the people that have the that you are the ass into their head. You know? So so be be very aware. Don't just cast a wide net and try and pitch to everybody. Find the businesses that are suffering from your stuff. I now have that criteria that I use for lead gen where they either run AWS. Great. You know, I have a very specific size requirement. I don't wanna be too big.

Matt Lee [00:37:39]:
Got an HR department probably bigger than you don't need me. And then if you've gone down recently, then I'm talking to you because you've you've got the pain. You've, like if your site's slowing down because you're growing too fast, I wanna talk to you right away. You wanna talk to me. So if you have that criteria, then it makes it so you're focused. You're not just, you know, getting disheartened because you sent, you know, a thousand resumes out to Joe's that, you know, doesn't have you know, doesn't have the need that you can scratch. You wanna be like, this is your problem. This is my this is my expertise.

Matt Lee [00:38:13]:
This is how I can solve that problem. This is how I've solved it hundred times case studies. It's a big thing I'm working on this year is getting more case studies on my website. Show them how you how you solve that exact problem. So if I put a coin in me, I'm gonna keep going.

Warren Parad [00:38:25]:
Yeah. No. Definitely. I know. I'm and I'm now I'm gonna ask you another question. Like, the sort of issues that you you see come up for each of your customers and from your experience, do you feel like they are sort of repetitive in any way? Like, oh, you've seen this problem before? Or do you see that some of these problems, like, they are actually nuanced and new in some way that you haven't experienced?

Matt Lee [00:38:48]:
Yes and no. The the people problems I see, that pops up quite a bit. So I I don't do a lot of hands on work for my clients. I do advisory and oversight quite a bit. So they pair me with project manager, and they say we've got this problem. You know, we we want we wanna scale up to this thing. I do a design, and then I work with PM to hand that off to other people to do. So I train other people.

Matt Lee [00:39:10]:
I see a lot of people problems that are repetitive. The technology problem, yes. Nine times out of 10, it's a security group problem.

Warren Parad [00:39:19]:
So that's true.

Matt Lee [00:39:21]:
But with with that said, past that, there's always new technologies. We're getting detoxed in ways I've never seen before. It's it's quite fascinating. Actually, I got this if you guys will indulge me, I've got one kind of more prop like the game that you guys might find interesting real quick. I'm sorry.

Warren Parad [00:39:37]:
Get the prop.

Matt Lee [00:39:38]:
I'm gonna digitize this. Oh, no. It's spinners. I lost the spinner. I will. So you said you talk about repetitive stuff. When I had some mid level junior devs, a lot of times they get stuck on things, and I, they'd be like, I I don't know what to do. I don't know what to do.

Matt Lee [00:39:55]:
They'd come up to me, and and a lot of times it became repetitive. So I I took the top, like, 10 or 20 things that, anything that did that was happened, including check the security group. Just You know? And I put a spinner on this thing, and I said, you can't you can't bug me till you spun it at least, you know, I think I said three three times tried three different things on here. I'm like, you gotta tell me you've already checked the logs. You've already RTFM. Okay? I'm not gonna explain that abbreviation, but, Yeah. Add more logging. You know? So this is actually coming soon to, to cloud at WarGames.

Matt Lee [00:40:35]:
This will be if they get stuck, we'll spin it for the beginners. I won't take this away for the advanced people. This is, like, a rare chance.

Warren Parad [00:40:41]:
You got a free spin. You know? Spend a while on the

Matt Lee [00:40:45]:
Exactly. I phone phone a friend. Ask a like, ask your peer before you come asking me. Like, another junior might actually have this little bit. Work together, collaborate people.

Warren Parad [00:40:54]:
And not just in the same company. You know? I I find that, especially above the senior level, a lot of engineers haven't realized that they can go outside the company to solicit assistance. Like, you don't need to pay someone, but there are communities out there that you can go to. And it's so ridiculous to me that at so, like, SMBs, like small medium businesses, like, not giant, model as you know, the FAANG companies, etcetera. You know, at smaller sizes, you just don't have anyone that's more experienced than you in the company. You have no one to mentor you, no one to sort of bounce ideas off of. And they're like, well, what do I do? And I'm like, well, how many how many Discord servers are you in? Like, I'm like in 30 or 40 or something. Like, there are those out there where people, you know, are willing to provide assistance if you just ask the company publicly.

Warren Parad [00:41:40]:
And the idea that they're just not even they don't even see this as an opportunity is, like, like, really great additional source of valuable information that you're missing out on.

Matt Lee [00:41:51]:
Yeah. I mean, that's part of the reason I am starting a community for for my niche is that there there's they're absolutely people are, you know, where do I go? Where do I go? I I love that I've got these communities I can go to about my business and all that other stuff. But if you if you're an engineer and and, you know, for a smaller company at least and you're running on Amazon, like, there's probably no one else that knows more than you, and you need answers. And it's like, yeah, you could take a course, you know, or take a bunch of tests and get certified, but it's almost better to have that master mind group, that community you can you bounce off. There's there's some additional value to it. So that's with all the AI going on, you can just chat to PT all day. But, you know, to have somebody else that actually breathes and has a pulse that understands your pain, even sometimes you just event to about, like, oh, it's the the security group for the hundredth time. Why did I make that mistake again? You know? So, I truly agree with you.

Matt Lee [00:42:43]:
I think that's a huge value. If you're not in one of those, you know, I would strongly recommend it. Yeah.

Will Button [00:42:51]:
Yeah. And it seems like Cloud War Games is gonna be like a foundation for helping you identify those groups because you get in, you start working with people that you don't know, and then you just start chatting from there. It looks like those are just gonna naturally form as a result of this.

Matt Lee [00:43:07]:
I I would love that. I hope I hope there's some because you you you get some, good relationships that are formed in foxholes, so to say. Right. You're in the fire there. It's like, you know, you you if, yeah, someone cracks under pressure, you know, but yeah, I I've got some guys that I'm pretty tight with because we sweated it out, you know, in in bad times. And, you know, that's I I there's there's you'd be amazed how quickly you forge those relationships. Sorry. Hopefully, no idea was still good.

Warren Parad [00:43:34]:
Well, I mean, the the worst the situation is, the more interesting and, honestly, the thing that you can sort of look back at later to sort of enjoy it, the better the story it is that you can, you know, walk through. Like, oh, there was these decisions that were made, and there were problems there. And then and then you're able to tell that story later. Like, it's a much better story. If everything just worked right all the time, like, your job would be boring, and you would leave after when you retire, be like, you didn't do anything. But the fact you can look back and be like, oh, like, there was one time the website was down because someone did this and it took forever. Like, that's a great story that you can tell.

Matt Lee [00:44:08]:
Yeah. Yeah. I I tell my parents this stuff. My mom's like, you know, yeah, I don't know how you can handle this stuff. This sounds so bad. I'm like, we will if this stuff didn't go down, occasionally, we didn't get sorry. We didn't get attacked like this, then I wouldn't have a job. So, you know, it's kind of like that's there's yeah.

Matt Lee [00:44:22]:
I I find these kind of outliers. The problem is you design a system that's so solid, it doesn't go down. So and they don't need you anymore. But luckily, the Internet's full of evil people attacking us. So gives me a job. Sorry. That sounds weird to say it now.

Will Button [00:44:37]:
It's always opportunity somewhere. Yeah. So, you've been you're literally just a weekend at Cloud War Games. Right?

Matt Lee [00:44:47]:
Less than a week. Yeah. Yeah.

Will Button [00:44:49]:
What was the trigger that made you decide to to to start this?

Matt Lee [00:44:57]:
So I had thought of it as a side part for my kind of community I was creating. It was a bit I I I wanted to help. I'm trying my mission is trying to find as many people that I can add value to their lives. And, you know, some of my big clients, you know, that's that's great. But what if a mom and pop tiny little shop comes to me and needs something? I can't offer them as much one on one time. So I was like, I'll I'll build a community. And then they can, you know, share information, just look who they're talking about, and I can help more people. And and it works out for all of us.

Matt Lee [00:45:26]:
But then I was in the mastermind group, and I was like, what are they adventure gonna do? It was the war gaming. And they're like, no. No. No. That's the front and center thing. Like, that's I could see companies hire you just to go in and do war games or all these other things. And then they got me hyped up, and I'm like, I'll have it launched I'll have it launched by tomorrow. And I said, I'll have it.

Matt Lee [00:45:42]:
I'll have a post. I'll have a video, and I'll have a landing page. And, you know, I I I tied my hands because I am the same way with the I I was like, no coding. K? You can't you know, no over engineering. Luckily, I'm a horrible designer, so I was like, I'm not gonna design. I'll just use a template. And I I set up this, on convertkit.com, this landing page, and I used Calendly to schedule the first couple. And I just broke some emails there, so there's no coding because I I I do have a tendency to over engineer if I allow myself.

Matt Lee [00:46:11]:
And, yeah. So, here we are Monday morning, and I had, five people signed up, and now it's doubled already. It's over doubled since then. So it's moving. And and now I'm on the spot. Yes. So, yeah, that was just the mastermind that basically they focused me. It really was good.

Matt Lee [00:46:27]:
Just cut away the other stuff and focus. You know?

Will Button [00:46:31]:
Yeah. For sure. That's, like, another, benefit of that is having people who will well, like you said, just focus you, you know, because we tend to over engineer it. And you think, oh, man. I've gotta have a website, and I need, a database to store the people's information and get their profiles, and I need authentication. And but, honestly, by having someone hold you accountable to it, you have the MVP in less than five days without having to do any of that stuff that's not actually gonna test your idea or not.

Matt Lee [00:47:09]:
Yeah. And I will say there's also some other factors here. It's a hot dog business being as I had all these other ingredients just sitting around. Like, I already have open source Terraform scripts I manage, and I used to run my entire infrastructure and some of my client stuff. So I already have that. Next time, like, oh, I could spin up a domain that we can knock down. Actually, I bought explodeme.com yesterday, which is gonna be a re attack. So, I I can spin that up.

Matt Lee [00:47:35]:
I didn't mean it's not quite ready yet, but I I can have it up in a few hours. But all I gotta do is take some of these scenarios that I've got written down, make them more formal, you know, figure out, okay, this is what I'm gonna do phase one, phase two, and I will put some hints in there and stuff. So I already just had these pieces of the puzzle ready to go, and that they were just gathering dust. I'm like, alright. It's it's kinda like hot dogs. Bunch of other pieces of meat that you got, the little leavings on it, you put them together, and now you got hot dogs and bologna or whatever else. So so that so that was uniquely that I was lucky enough to have that, fortunate enough to have that. So

Will Button [00:48:10]:
Right on. How long have you been in the mastermind groups?

Warren Parad [00:48:13]:
How long

Will Button [00:48:13]:
have you been doing that?

Matt Lee [00:48:17]:
Yeah, on and off throughout the years, but I over the last two years, I really went heads down. I I I hired a business coach, and I think it was 2022. They didn't quite work out, but then I found this solo expertise one in 2023. The beginning of that, that was great. I actually ended up engaging the founder of that for a one on one coaching. And he said we met yesterday's phenomenal. Then I threw, a parallel I found a parallel group that was more hands off, smallbets.com, that, then I found another person over another mastermind group that was more hands on, and I jumped in with them. And I I think I almost mentor them as much as they mentor me luckily at this point.

Matt Lee [00:49:00]:
But, yeah, that's what a mastermind group is. So that's it's been, that was about nine months ago there, and I'm looking for more. And I'm probably gonna start another one next

Will Button [00:49:08]:
few weeks.

Matt Lee [00:49:09]:
Well, maybe months. I should say next few months. I'm gonna

Will Button [00:49:16]:
finish. Right on. Cool. And then your your, cartoons that you release, where do you release those at for anyone who wants to see those?

Matt Lee [00:49:28]:
I I have my mailing lists, my socials, and then, I push them. If they're relative to Terraform, I push them in those Terraform groups here and there, r slash Terraform. If they're relevant r slash programmer humor, that's where I launch a bunch of them. Those are the ones where I get the most views. Yeah. So yeah. And there was one week with, there was yeah. There was never mind.

Matt Lee [00:49:54]:
There was one that was relevant to something we said earlier that that just came out last week, but I don't think I can connect the threads anymore. So

Will Button [00:50:03]:
The one you sent that cracked me up was the, the y two k one because I, like, I know people that in 1999, they went into bunkers, and I've never seen them since. And so pretty frequently, I wonder, like, wow. Did they because some of them were gonna stay in their bunkers for, like, ten or twenty years. I was like, did they stay? Like, are they still alive? What happened when they came out? And I don't know where any of them are.

Matt Lee [00:50:32]:
Yeah. I got to thinking. I was this could be an annual one because this actually I released on the December 31. But I got to thinking, what's the dumbest thing that you could go back into a bunker and live your life in hiding for? JavaScript everywhere or what?

Speaker D [00:50:47]:
And they had a little bit of

Matt Lee [00:50:48]:
a controversy to it because, you know, of course, there's, like, half the Internet hates JavaScript and half the Internet's like, JavaScript. Let's put it in everything. Right. That that helped it. That definitely helped it go up in numbers.

Warren Parad [00:51:00]:
For sure. I mean, if people I I know about this controversy, and I just I wonder if the people who are against JavaScript, will be more in favor or against whatever LM generated programming language the whole world ends up utilizing, in the future. I mean, we'll be forced by our robot overlords to use the one true programming language, whatever that is. And, I wonder if they'll be okay with that or whether or not they'll have wished to go back to the days of JavaScript and everything.

Matt Lee [00:51:30]:
You just gave me an idea for a camera. This is we're arguing about JavaScript versus not JavaScript. But what about we can all agree that people that use no code solutions are fools?

Warren Parad [00:51:41]:
Right.

Will Button [00:51:45]:
I I think I probably get pinned in the anti JavaScript crowd a lot, and I'm not I'm not anti JavaScript. Like, the point I try to drill in there is that JavaScript's not always the right answer. You know? Like, you don't hire a carpenter based on the type of hammer that they use. You hire them based on their ability to build, a garage or a house or a fence or whatever you are. And I think JavaScript to me is a tool, and so JavaScript's not always the right answer. So it's not that I'm against JavaScript. I'm against saying JavaScript is always the answer.

Warren Parad [00:52:29]:
Yeah. I mean, I was definitely in, in my life on the like, they're just all the program language are tools. But, you know, I've definitely come more around to the fact that every single one of these tools has some specific quirks, which change the utility in the solution space that we're in. And so it actually is more important that we evaluate whether or not this tool, this language, what it implies, how it works, who the maintainers are, what their mentality for the long term is for the type of solution that we're building. And I I feel like we're still very much stuck in the fact of whether or not people sort of like the tool itself. Like, I like hammers. I don't like hammers rather than the the company that manufactures the hammers is who we should be evaluating.

Matt Lee [00:53:15]:
Yeah. Yeah. From a from a my thoughts on JavaScript from a business standpoint are you know, this is probably a little dated now, but back in, let's just say, mid two thousand tens, I can high if you're if you're gonna if you're gonna write PHP and have a website, you know, you're gonna need to have JavaScript in the browser. Right? That's just you can't run PHP in the browser. Same thing with Java. Same thing with a lot of things. So a lot of these developers that I was finding had some amount of JavaScript background. And I found that you can repurpose, you know, that little bit of JavaScript in the in the browser to, run server side if you wanted to, much easier.

Matt Lee [00:53:57]:
So it was it was more of a cost efficient thing. I could take a PHP dev that's full stack and make them a a Node. Js dev without breaking the bank on that one. With that said, now AI being a big thing. It would be silly to try and use neural networks in JavaScript. I know because I've tried it.

Speaker D [00:54:14]:
Probably not a good idea.

Warren Parad [00:54:16]:
You know?

Matt Lee [00:54:16]:
So, but no. Yeah. You you can you you so you're right. You gotta pick the right tool for the job, but it was strictly on, like, a who could I you know, we can catch a wider net and hire people that have JavaScript experience and and, you know, so we can get a wider range of people coming in, better applicants, and ideally, better outcomes. So that was that's that business, you know, numbers. You gotta kinda do that calculus as well. But, again, you're right. Not the right tool for AI, not the right tool for a lot of things.

Will Button [00:54:42]:
Yeah. But it might be the right tool for your business because you just need a versatile programmer who can get shit done.

Matt Lee [00:54:50]:
Yeah. Yeah. If I'm hiring for the top Python devs, a lot harder to figure out how to hire those guys than just, you know,

Will Button [00:55:02]:
For sure. So for people who wanna go check out, Cloud War Games, CloudWarGames.com?

Matt Lee [00:55:10]:
Cloud War Games Com? Yeah. Yep. That's, yep, that's where you find it. You can find me. Is this the time where I'm supposed to kinda tell you all the different things?

Will Button [00:55:17]:
I I was trying to I was trying to tee it up to let you plug the website, but then I just plugged the website for you and stole the answer out from under you. I butchered it.

Warren Parad [00:55:27]:
For for the release of the episode on on the website, ventureindevops.com, there'll be a a bunch of blurbs along with whatever links you want us to post there.

Will Button [00:55:36]:
So

Warren Parad [00:55:36]:
if you wanna just respond to us after the podcast, you know, all of the things that we've talked about in this episode, for anyone who's interested, well, I'll be listed there.

Matt Lee [00:55:44]:
Sure. Okay. Well, yes. Cloudwargames.com is where you could find that. Smedical.com, that's the word smedicalal at the end, is where you could find most of my writings, Smedical.com. And soon, I'm gonna announce this publicly. This is the first time announcing it. I actually just spent the last three months of my life building a new video course for O'Reilly's, the publishing company.

Matt Lee [00:56:04]:
It was an on demand video course that were very wanted me to say on demand. On demand video course for O'Reilly publishing. And so that will be live hopefully by the next two weeks or so. Maybe by the time this comes out. So if you wanna learn zero to hero on AWS security, an animated guide to security in the cloud, then that'll be there. So if I'm overplugging this, feel free to cut it. But, now

Will Button [00:56:28]:
I'm super interested in this. So now this is an an animated course on AWS security?

Matt Lee [00:56:36]:
Yes. Yes. It is. Actually, if you if you want, I've got a diagram here that I queued up. Okay. We can cut this if you want, but, I if you guys want, I'm actually we can play a micro round of cloud war games right now. Let's do it. It's not real infrastructure.

Matt Lee [00:56:49]:
It's not real infrastructure.

Will Button [00:56:50]:
Let's do it. Alright.

Matt Lee [00:56:51]:
Can I present here? Share screen. Share screen. This right here, can you guys tell me if you could see it?

Will Button [00:57:04]:
We can. Not anyone listening to the podcast can't, so I'm gonna narrate it for them.

Matt Lee [00:57:08]:
Okay. Here we go. Alright. We have our infrastructure here. This is a network map. Alright? And we've got a user right here. So this is I'm gonna go as fast as I can because I know we're I wanna I wanna be respectful with time, but a request come in, hits the application load balancer, hits the app hits the application server, then, your shopping cart so I should set the tone. Ecommerce store.

Matt Lee [00:57:30]:
You run an ecommerce store here. The shopping cart store in Redis. So they say, hey. I want to, buy this thing. It goes into Redis, goes back to the application layer, and now they've added it to their cart. And let's just say that they actually go to make a purchase. It's the application server. We're doing event driven architecture.

Matt Lee [00:57:52]:
Nope. Nope. Nope. Nope. Maybe it's not the right one. Alright. We're gonna feedback the animation for now. Event driven architecture, pump it to Kinesis.

Matt Lee [00:57:58]:
That updates the database saying a purchase was made. It updates the inventory management software. It updates the, product worker which charges them and updates the mail service and all that stuff there. Alright. I'm getting to you in the details here. Here's the scenario. K? Customer service comes to you and says, we are getting reports that people are missing stuff in their cart. They they're they're adding stuff to their cart, and all of a sudden when they go to checkout, it's not there.

Matt Lee [00:58:24]:
They go to another page, it's not there. Alright. What do we do?

Will Button [00:58:27]:
So I'm gonna highlight this real quick. We're looking at an animated diagram with an AWS environment. Right? And is this part of the Cloud Wars games interface?

Matt Lee [00:58:38]:
It will be. It's not actually this is running locally right now because I've never bothered to put it on a website, but it'll be up eventually. I'll use it locally if I have to for now.

Will Button [00:58:48]:
Right on. Okay. So there are stuffs missing from their cart. I'm assuming that the cart is persisted somewhere in some data store somewhere?

Matt Lee [00:59:00]:
That is correct. It is in Redis.

Will Button [00:59:04]:
Right on. So do we have a way to check logs on Redis or check the whatever's writing logs to Redis?

Matt Lee [00:59:11]:
It's running on ElastiCache. ElastiCache does, have events. I do not I don't know if you can get directly to the logs there, but, you can log in to Redis and run some stuff. You've got metrics too. Don't forget you got you do have CloudWatch metrics.

Will Button [00:59:30]:
Good work.

Warren Parad [00:59:30]:
I guess the first question is, like, is the data in the database where it's supposed to be? If we're using Redis, like like Redis Commander or something to tell us whether or not the items are actually in the shopping carts, the store.

Matt Lee [00:59:41]:
Yeah. So you you go in there. You've got, let's call it a hundred thousand keys. And you look at it. There is data in there. You can I've got I wouldn't do this step for you normally, but I'm gonna do it just because I'm gonna speed things along. You go in there and you try and add things to your cart. Of course.

Matt Lee [00:59:58]:
You add things to your cart. You check your session key against it, and your stuff is in the cart there.

Warren Parad [01:00:03]:
Does the cart for the customer who's complaining, are you able to have a session for that that you can track to check to see if there are actual items for that session in the cart?

Matt Lee [01:00:13]:
So this is this is where I wanna get, like, a five sided diet. Do you have a technical minded customer or not? K. You just rolled. Do you have a technical minded customer? They can open up their browser and copy the cookie for the session and send it to her. And so, yes, you have their session.

Warren Parad [01:00:26]:
I mean, I always assume they're not a technical customer, and so you try to get them to purchase some fake item in the cart that only only their session will will have access to.

Matt Lee [01:00:37]:
I see.

Warren Parad [01:00:37]:
Give them a magic URL, right, that allows

Matt Lee [01:00:40]:
them to

Warren Parad [01:00:40]:
be attracted.

Matt Lee [01:00:41]:
Yeah. Yeah. I love that. And if yeah. I was just being nice because we don't wanna be for the sake of time. But, yes, I love that solution. See, this is the type of solution, but I didn't I wouldn't have thought of that. I would have thought this is the type of stuff you can get with these type of games.

Warren Parad [01:00:53]:
Well, we we gotta we gotta debug this stuff for real all the time at our company. I mean, we're not ecommerce, but, you know, someone we we do login and access control, and so it's always a point of contention. Like, why didn't this user have access to this resource at this particular moment? And so figuring that out, you know, requires a bunch of things. So we have ways to give our customers information that they can inject into their environment, when they're performing the sort of investigation that we can see on our side so that we can highlight exactly which traces make sense. Otherwise, you know, you're looking at all of the data for every customer, in order to narrow that down. Yeah. So, I mean, you said technical customer, or some magic solution. And, are we getting their items in the cart, or are they session busted?

Matt Lee [01:01:36]:
You check the session against the the, Redis database. You got the session key there. Items are gone. Not there. There's not there's nothing there. There's nothing with that key. Key's gone.

Will Button [01:01:49]:
So you mentioned ElastiCache. I'm assuming that wasn't just randomly throwing out words. Where's that at in this pipeline?

Matt Lee [01:01:57]:
Redis running on AWS is ElastiCache. I just don't have ElastiCache is a service that runs value key stores. Okay. I got it. Couple different ones, but I picked Redis. It's my go to.

Will Button [01:02:08]:
Gotcha. Understood. So they put something through the browser in the cart, never made it up never made it into Redis. Who's responsible for sending it to Redis?

Matt Lee [01:02:20]:
To clarify so just again, I'm giving you way too much here. I never said it didn't make it into Redis in the first place, just to be clear.

Warren Parad [01:02:27]:
Just got deleted instantaneously. Like, you know, by the time it gets to the cart by the time you look at it, it's no longer there. So there could be some sort of TTL on the items that are going into the cart or some other process that's eliminating them from Redis or, you know, of course, not getting there in the first place.

Matt Lee [01:02:44]:
Mhmm. Yeah. So, yeah, I so just to speed this along, I'm gonna drop what I would give as a hint, that you do have CloudWatch logs for, for Redis. So you could ask me what what they are saying.

Will Button [01:02:57]:
Well, what are the logs saying?

Matt Lee [01:02:59]:
So number of items is pretty oh, no. Number of items went up, but flatlined. K. CPU usage is fine. Number of evictions is up. Quite a bit. Evictions.

Will Button [01:03:12]:
Oh, so they got they got evicted.

Warren Parad [01:03:19]:
Yep.

Matt Lee [01:03:20]:
And then, I guess, the usual memory dropped. So it's again, I'm I I'm gonna simulate this.

Will Button [01:03:25]:
Yeah.

Matt Lee [01:03:25]:
Yeah. This is the I I wasted one of my real juicy ones, but I guess it's probably good here. So, yeah, evictions are up. So what what's the, what's the solution there?

Will Button [01:03:37]:
So I don't use Redis a whole lot. So, obviously or it seems like Redis evicted for some reason because it either has, like, a max number of key values that it can store or it ran out of space?

Matt Lee [01:03:50]:
Yep. That is correct. It just runs out of data. It's you can set eviction policy, but in this case, I think the default is to just start evicting the older ones. And so that's what happens is they they it filled up. Maybe it was a big shopping day, you know, and you didn't expect this much, this many people to put stuff in the cart. And so it filled up. Or I mean, it's again, I'm I'm gonna just kinda speed through this, but, possibly, you could look at it and some change, made the payload of each key too big.

Matt Lee [01:04:20]:
You know? So it's like, oh, we we added something. Now we're caching, like, this whole structure. Instead of just the product IDs and quantities, we're still caching the metadata on the products. For some reason, somebody actually messed up and put that in there. So and now it was just bloated. So the fix would I mean, I'll get you I'll let you guys present the fix if you guys have anything there and that'll tell you the problem.

Will Button [01:04:43]:
I think the emergency fix has increased the size of Redis so that we That

Matt Lee [01:04:48]:
that's that's probably what I would do. And then the long term fix would be to fix the bug. Adam cache all that stuff. Actually happened once. Actually, it happened. They had to cache a bunch of HTML in there that was like I'm like, what? This is a JSON object with HTML in it. Like, what what what are you doing here? Stop caching that stuff in this thing, but it happened. So yeah.

Matt Lee [01:05:08]:
Alright. Congratulations. After action report and so on and so forth. So Yeah.

Warren Parad [01:05:14]:
I mean, this is great. I I'm actually sort of curious on the scenario. Is is using Redis as a temporary store for the, cart, common pattern in ecommerce stores?

Matt Lee [01:05:25]:
When at early stage, yeah. So a lot of my clients, you know, I love them to death. Clients understand. They, you know, they they start off with something simple like WordPress, and they start slapping more things in there. Not the way I would do it. K? K? Not the way I would do it because, I mean, there's there's a lot of data you can extract from carts. It's not ephemeral data. You'd wanna know who's adding what, when, where.

Matt Lee [01:05:44]:
You can send a reminder emails and all that stuff, but that that a lot of times with with devs, a lot of times, it's just stuff everything in the session and then, pray.

Warren Parad [01:05:53]:
Yeah. K. So I mean, I I I get I get the the perspective of knowing that it's temporary data. It's not the order hasn't been created yet, so storing it in a data store that may not be persistent. I also get the idea that people believe that Redis is this, nonephemeral data store and, you know, go ahead and use it as your source of truth database. But, you know, in a in the mature business, as you said, like, you you actually don't even wanna throw away that session data. As soon as you have it, you might as well be persisting it for for perm permanency so that you can evaluate, you know, why didn't that convert to an actual order? You know, why didn't that person come back? And then once they do actually click buy, and payment gets processed, you don't need to convert it from, your temporary data store into into a permanent one. You already have it there.

Warren Parad [01:06:39]:
The payment is sort of separate.

Matt Lee [01:06:40]:
So I'm trying I'm actually trying to get you to exactly what you're describing for my, the video I did on Amazon Blue and, you know, putting those that data in a long term cold storage. Basically telling people the difference between source of truth, ephemeral, and long term term, you know, redshift or big data analytics tools, data warehouse, data lake type of situation. So

Warren Parad [01:07:04]:
This is one of your videos that's going on on demand, on O'Reilly? Or

Matt Lee [01:07:09]:
No. That's my YouTube videos. Yeah. The O'Reilly videos are specific to the security course. So you can see I have a bunch of that in here. But

Warren Parad [01:07:16]:
I see.

Matt Lee [01:07:16]:
Yeah. Alright. I'm gonna stop sharing some I'm I Yeah. Yeah. No. No. I didn't prepare that one.

Warren Parad [01:07:23]:
Will's gone Will's gone dark for me. He's got no camera on. He's he's hiding from the from the public. Oh,

Will Button [01:07:29]:
I went anonymous. Yeah.

Warren Parad [01:07:31]:
I I mean, I think, you know, I think it's a good point point to, maybe wrap up the episode, unless there's anything last word you wanna share about, Cloud War Games, or your consultancy.

Matt Lee [01:07:43]:
No. I wanna thank you both for being the first participants and, I mean, somewhat of an unofficial sorry. It's somewhat of an unofficial capacity there, but thank you both for, being the first participants. I know we kinda played a micro around there, but I invite you. Once, I get this thing moving, I'd invite you guys to to come and participate. And if you guys regain, maybe I come back and run a full length game with you guys or some of that nature. So it's throwing that out there if you're interested.

Will Button [01:08:12]:
That'd be a blast, and would love to have you back on at any time. For that and and there's a bunch of other topics we we, like, touched on here that I think would be cool to dig deeper on. Yeah. For sure. I'm always

Matt Lee [01:08:28]:
be my pleasure. You let me know when, and I'll I'll, you know, if another guest drops out, you can come find me. I will

Warren Parad [01:08:34]:
do. With that, should we move on to picks well?

Will Button [01:08:38]:
We should loop we ah, man.

Warren Parad [01:08:41]:
Words are hard.

Will Button [01:08:42]:
Yeah. Right? Just like to take this opportunity to point out to everyone that English is my native language, and I still screw it up.

Matt Lee [01:08:51]:
You should see me editing my my videos while I'm swearing.

Will Button [01:08:56]:
Well, for sure. I I used to do quite a few, YouTube videos and did some video courses and stuff, and my wife would always give me a hard time. She would say, I can always tell how it's going by how many cuss words I can hear coming out of your office. Yeah. Then at one point, I made, like, a blooper reel where I just took all the cuss words that I cut out of all my videos, and it was like an eight minute stream of just some of the most offensive language that has ever been recorded.

Matt Lee [01:09:28]:
Yep. Yep.

Will Button [01:09:31]:
Alright. Anyway, on to picks. Warren, you brought it up, so I'll go first. I'm picking a couple of weeks ago, I picked Kunk on Earth, the Netflix special, and I think that was when we had AJ Funk on. And he said that there was a new one called kunk on life, and so I watched that. And I thought kunk on earth was so great that there would be no way a a follow-up to it could be as entertaining. And I'm happy to tell you that I was wrong on so many levels. So kunk on life, go check that out on Netflix.

Will Button [01:10:11]:
The just the sense of humor there and the dryness of that sense of humor had me just rolling on the floor laughing. Just absolutely amazing. And they accidentally made it educational as well. So go check that out.

Warren Parad [01:10:30]:
Well, I I definitely will because I liked all of the other series, by Filomena. Yeah. I guess that's her her stage name and not her, not the actress's name. Okay. Mine is, I guess I also have a show. So I, I read the books three body problem, by, Leo, Cixin. And, it's actually one of the scenarios where I actually like the series better. I think the series is absolutely fantastic.

Warren Parad [01:10:57]:
I think some of the books are good, but the series, like, absolutely really well done. I really look forward to the follow-up seasons. If you haven't seen it or read the books, I highly recommend.

Will Button [01:11:09]:
There's a

Warren Parad [01:11:10]:
lot of good philosophical ideas that are brought up.

Will Button [01:11:13]:
Okay. Give me the high level of what it's about because I'm just assuming from the title three body problem, this is how to get away with murdering someone. I may be wrong.

Warren Parad [01:11:23]:
No. It actually has to do with the unsolvability of finding out without the initial conditions what the path and orientation of a solar system with three masses in it is.

Will Button [01:11:37]:
Okay. That's not gonna be as useful to me as I thought it was at first. The

Warren Parad [01:11:41]:
Yeah. Without without without spoiling without without spoiling, really that much. There is, one of the very close solar systems in our galaxy, to the mil in the Milky Way, to our solar system is one with three stars in it. And so there is a relation it's science fiction, really good, philosophical dilemmas, absolutely fantastic. Well done.

Will Button [01:12:07]:
Is that the Alpha Centauri system

Warren Parad [01:12:10]:
Yeah. Yeah.

Will Button [01:12:10]:
That you're referring to? Yeah.

Warren Parad [01:12:12]:
Yeah.

Will Button [01:12:12]:
Because they originally thought it was one star and then found out that it was, two binary stars in orbit with another star close by, I think.

Warren Parad [01:12:20]:
Yeah. Yeah. Yeah. It's, four light years away. Right?

Will Button [01:12:26]:
Just across the street.

Warren Parad [01:12:27]:
Yeah. So, Matt, what do you got for us?

Matt Lee [01:12:31]:
So you guys you got all this Einstein stuff.

Speaker D [01:12:33]:
I I did not come prepared for that. I've

Matt Lee [01:12:37]:
got the comic book Scott Pilgrim versus the world. Nice.

Speaker D [01:12:42]:
They do a lot of yeah. They they do

Matt Lee [01:12:44]:
a lot of pixel art in there, and I've been doing and I've been propping them on my desk. And I've been doing a lot of, comic book stuff. So I'm like, okay. This would be a good thing to pick up. And I'm trying to look at screens less before bed, so I've been, just reading through this lately. Yeah, not not quite as not quite as big of as interesting as as the three body problem, but, you know, helps me go to sleep.

Warren Parad [01:13:10]:
What was what was that last thing that you held up?

Matt Lee [01:13:12]:
Scott Pilgrim versus the world is just, that's volume one, volume

Warren Parad [01:13:16]:
I see. Oh, it's I see. I didn't know it was a written work of art.

Matt Lee [01:13:21]:
Yeah. That's I believe that's how it started, and then I think they made the video game and then the movie. So

Warren Parad [01:13:26]:
Interesting.

Matt Lee [01:13:27]:
But the video game, it was the guy in the dark for that is a guy named Paul Robinson, I think, or whatever. And he's just a phenomenal pixel artist, and I just I my work doesn't come posted. I got a style that I could do in two seconds, but he spends, like, years building these amazing animations. It's phenomenal. So yeah.

Warren Parad [01:13:49]:
Right on.

Will Button [01:13:49]:
Cool. Well, Matt, thanks again for coming on the show. This has been a blast, and we will definitely have you back on. For everyone else, check out cloudwargames.com. And anywhere else that you hang out, you hang out on, like, X, LinkedIn, any of that kind of stuff?

Matt Lee [01:14:06]:
What you You can find me yeah. You can find me most of those places, Schematical. Okay. I try and grab it anywhere I can. So

Will Button [01:14:12]:
Okay. Yeah. So check it out on Schematical, and, thanks for listening, and we'll see everyone next week.