Sascha_Wolf:

Hey, everybody, and welcome to another episode of Elixir Mix this week. On the panel we have at Iron Gar,

 

Hello, Hello,

 

Allen Wymer

 

Hello,

 

and me, such a wolf, and we have a special guest this week which is Michael Lobes, And Michael wants to tell us a little bit about a little thing he's been working on which is called Paraxil, I, O, But Michael, why don't you tell our audience why we invited you? what all of this is about?

 

Hey, thank you for having me on the show. I appreciate it. Yeah, so my name is Michael Lebas, I'm the founder of Praccolio, which is an Elixir, Phoenix Focus security company, So we make you know a platform that's focused on Elixer, So my background is mostly in computer security, Web applications and bought detection. Yeah, it's just great to be sitting down and talking with you all today.

 

So maybe first things first. Like how did you end up founding Peracallike? What's the story there right? Don't assume you woke up one day and I was like, Yeah, I'm goin to do this. So how did you end up to being there where you are now?

 

Yeah, it's interesting. So my professional background, I was like a penetration tester, which is doing security assessments for web application. Most people who listen to this show of, probably had to do that in a professional environment where you're probably working on just some software. You show it to a company that does this, and then you get a report back. Same here, the security issues and things like that. So I had a series of jobs as well. A few years ago I learned Elixer, Because I started working at Frame Io, which is video collaboration platform. They make software for kind of the media industry. It was acquired by Adobe. Recently it was a huge acquisition Was something like a billion dollars. But yeah, their whole back end is is Elixer, and I was hired for the security team there, and naturally I said Okay, I'm the security engineer. The whole back end is an elixir. I'm going to learn this language. That's how I learned about it and it was just fantastic. I really love just working in the environment, learning about kind of the underlying Er. laying and beam. And that was how I kind of came to elixir. So for starting practclio, what I also saw when I was working was there's a lot of companies that you know need sort of enterprise security software such as you know. Bought Detection is a common one where if you like a recapture for Ample when you're browsing the web, nobody really likes those. You have to solve a puzzle of like a fire hydrant before you log in, but the reason

 

Yeah,

 

for that is

 

I don't

 

Ah,

 

like those yet.

 

of course, A. Yeah, they're terrible, but that's just an example where the reason companies have that is because of bought attacks, because they're dealing with malicious pots that are doing credit card fraud or trying to log into people's accounts. It's costing the business money, or businesses need to manage owner Bilities and their dependencies, or they need to do static analysis and also insure those findings are being tracked correctly and try aged. So this is a category of enterprise software. There's a lot of providers. None of them care about Elixer, So I saw the opportunity to start a company that was going to serve this market, But you know when you're dealing with the security company, you now have the opportunity to really talk to someone who knows Elixer and Understan, That's what you're working on. Um. most cybersecurity companies are are venture funded, Meaning for you as a customer, they're not going to be focused on Elixer. They're taking the V C money to grow as fast as possible, so they're going to focus on Java Python Java script. The largest market segment possible. If they have Elixer support. It's probably because one person there really like Lixer and somehow convinced their manager to let them code the integration. It's never going to be the focus of the company, So that's really how practical Got started with this goal of serving the Elixer community.

 

Yeah,

 

So I speracial, not V. C. funded like boot strapped. How? how are Yeah? you to explain a bit that Ud be great.

 

Exactly? Yeah, we have not taken any mantra capital money. I don't have any plans to like being boot strapped. I think it's much better for our customers as well.

 

That's great. So I mean, and I hadn't heard of you guys. Sounds like you have enough revenue to sustain, which is great like you have. like. Can you name some of your clients like that's okay.

 

Yeah, absolutely, the ones I can talk about publicly are on our website right now. I have the logo cloud Umbatafe, For example, they make this really cool software product for user research, So to understand how users are you know interacting with your product, they actually blocked a bought attack on launch day with practice, which I was very.

 

Nice.

 

Yeah. so they had to deal

 

Wow,

 

with this person who was probably you know what we know. For a fact, they were sending these mill The sign up events and they could have been trying to do credit card fraud. I can talk to Yu through that attack. Actually, if you'd like give some background on what I mean.

 

Yeah, you go ahead. I mean like. I think it's a good example. Like also, for our listeners to kind of growtpracal like Offer what kind of things you might want to be on the lookout for us. An engineer, because I assume as you said earlier, this is not something a lot of us had to do in our day to day, but it's still in the back of our heads. We probably should do something about that, so real world example. Just go ahead.

 

Exactly. Yeah, So if you let's say you, you've started a company, say you're just making a project management site, So you log n you create an account you pay. you know, nine dollars a month or something. You enter your credit card and that's how it works, And maybe you go. Oh, here, practical is cool, but I don't need it because all of my users just to even use our platform, we need your credit card. So that's an authentication. You know who's going to sign up with you know, Fee, credit card or something. Unfortunately, a lot of people will. So what will happen once you launch M? Basically, there's groups of people who do credit card fraud for a living, and they have you know, a thousand stolen credit cards and they need to test which one of them work, And you're not going to go to your local store, you know with a thousand credit cards and ask the guy to run it, because you get the cops called on you pretty quickly, so you go to online shops, Um, all of the big online retailers that you've heard of are aware of this problem And have defenses in place, so unfortunately they actually really like to target newer companies, So a common sentiment is I'm new or two small. Our company isn't big enough to be targeted, but unfortunately your, it's actually a little bit more likely because they want to target people who don't have defenses in place. so what you'll probably wake up to one morning, either in your logging or your application performance monitoring is this huge spike of new accounts that work Created, and that seems like great news. Like for the business, we have all these new accounts until you investigate the transactions, and you see you now. something like seventy or eighty per cent of them have been declined, And the reason for that is those credit cards are stolen and all of those accounts are bought. So now you have a really difficult day at work where you have to explain. Not only are the metric are wrong marketings excited, but you have to tell them why they shouldn't be now. You're you're dealing with your transaction, your pay Process, or who might be trying to ban your account over this, Um, Because they want you to implement bought detection on your side. The kind of try to transfer that Um. So for example, a really come and control that I've actually seen payment processors recommend is to use Recapture M, which I disagree with just from a business perspective, Because think about when you purchase something on Amazon Dot com. Do you have? Do you have to select photos of stop signs? It's Ridiculo Is because they know people will spend less money on their site, But then you have these smaller retailers where it's really important that your conversion rate is high. musing, recapture, and there's other problems. its recapture to. It's pretty easy to buy passing things, so I'm very happy that practicing is able to provide the service especially to new companies. You know that are not big tech. That are you know, serving a small business or serving in a smaller community. You just giving, Need to you know another tech giant. So that's something I really like about practical as a business as well.

 

Okay. that makes a lot of sense. I honestly never thought about like that. smaller companies are more likely to be talking with. It makes so much sense when you say it out loud. but yeah, it never crossed my mind like never.

 

Every new product I have launched without exception has had a first attack like first increase in traffic. I think my theory is some of these people have like crawlers on these company registration websites, and

 

M.

 

just like you know, Oh, let's try to find domain names that match the company names and just keep attacking those like it. Just it's crazy

 

There was a great paper out of us, Nick, recently, where they talked about how with H to P, s and certificate transparency logs. when you register your certificate for issue to P. S. like you've probably used Surf bot or something. That information gets published. People scrape that and they just plug it into their motonfostructure. That's a Yeah.

 

That

 

there's

 

makes

 

just

 

sense.

 

leaks.

 

This is why we can't have nice things. Boy. Okay if you don't, If this is not like sharing, I don't know trade secrets. but like how in general would you even go about like detecting a butt attack Like I'm sitting here, as like a soft engineer was built a bunch of different systems, but like Okay, Of course we captured. maybe something like that, but you deliberately just sat like this is not a very Us friendly way forwards In general, like high level, How does a product like practial does the things it does you know. How does this work?

 

Yeah, it's a great question, so I think I've I've done the recapture comparison, so I'll also compare it to cloud flayers bought detection offering. So most people on this, you know, you're probably familiar with Cloud Flare if you're listening to this show. But a very basic summary of how their bought detection works is you have your website. Let's go. Let's use that project management example, and you point the dens of your project management site to a cloud floor server, and there's a little bit of down time. and now all of that Raficisyou, know your user types in your domain name and their browser connects to a Cloud Flare server. And then that cloud flare so over talks to your real, will call it the Origin server. Okay, So Cloud Flare does you know? rate limiting but detection, Um, similar functionality to Peractcial, but the reason I'm giving you this comparison is because practical runs in your elixer application code compared to cloud flare, and a common you know mark And positive to cloud flare is because we're doing both attection at the edge. Your performance is better because you're off loading that computation to the middle server. Um. But from

 

Hm,

 

a security point of view, that is not ideal for two reasons. The first is there's a very common problem with sites that are protected. Quote, unquote protected by cloud flare is when an attacker needs to run a boat attack on the site. we'll see. Okay. Here's the site. It's protected by cloud flare. That's very easy to see publicly. but let's search on the Internet for the origin I P address, like the real P address of that server. And

 

Hm,

 

there are a million ways you can leak that it's very difficult to get right once they figure that out. the Um. you know, you can literally go to get hub and type in like a cloud flare. I p, by pass, and twenty projects will pop up

 

Hm,

 

Once the attacker figures out the real I P address. They just send

 

India,

 

their bout traffic directly and they completely by pass the

 

Yeah,

 

For server that's in the middle

 

you kind of get off your pants down at that point right,

 

Exactly, And it's sad because they're offering is pretty expensive. It's enterprise software. so you're paying all of this money for the service, But you know they're not going to really tell you. Okay, your origin. I p. leaked. Um, so it's like you're paying for this product, but they're not really solving your problem. Um, so that's something I want to do with Practical was have the ability to really engage with customers and ensure that they're actually being protected.

 

I guess like one possible approach there could be to only allow traffic from cloud, where cloud far to your infrastructure. but again at that point you need to be aware of it right like you need to jump through these tubes. and

 

And and sadly, it doesn't even end there because I've read a recent research article about somebody who proceed. There's a cloud flare product called cloud flare workers, so he just proxied his mot traffic

 

Uh,

 

through.

 

uh,

 

Yeah, like it doesn't end. you know

 

Oh, my god, yea, I was. I was like. I mean, maybe maybe that could work. Okay, Okay, so that makes sense to me. So having it inside of your own system gives you some guarantees Like an offering like cloud, Like, can't do as easily. Okay, Fair enough.

 

The other, before this episode started, It was funny. We were talking about robot vacuums and how they leak information. They'll send your floor plan to the cloud, so kind of relevant to that. The other thing with cloud flare is because they're kind of a man in the middle between your user and your Origin server. So the kind of man in the middling that incription, they can see your users passwords, authentication tokens, Cookies. And the reason that we know this for a fact, is a few years ago There was this incident that's kind of been dubbed Cloud Bleed where cloud flare servers were dumping like raw memory on the Internet, researcher from Google noticed that and was kind of tryashing the incident because it affected such a big part of the Internet And there were just you know, cloud flares, customers passwords in that data. Um. And the reason I mentioned this is because the design of practicing Lo. the agent has opened source. Your passwords never Touch our infrastructure. Um, you choose what data you want to send to us, but we don't just you know. cover like cover up everything for collection. Ou have the ability to be careful about what data because you know you can go on all day about how secure your code is, but really just the ability to not have that information is the most secure thing possible.

 

Okay, but now now we are like in our application and I say like we've used a product like like Paractal. What exactly is like Paractal doing to like the detect bot tax, Like of the eristic at work. Like what is like in general, like the underlying model which is used to figure out this is. This is like a. Since you're user right, likeccessing website right now, this is probably a bot. Like how does this work? Because this is not something and I would assume a lot of people ever had to do

 

Yeah, it's It's not fun when you also have to implement it yourself, so I'll just talk about three technical measures. There's there's quite a few, but I

 

Apple

 

think

 

to

 

these

 

see.

 

are. Yeah, just they're easy to talk about, Kind of a podcast for Mat. So the first one that I really like to spend a lot of time on is rate limiting, which seems very simple. like one I p address limit. The number of requests like logging attempts it can send in a thirty second period seems like Simple description. I have seen this like screwed up in every way possible in real companies. it is so easy to mess up the cloud flare example I gave you right now was one people say, Oh, we have capture in place, so we don't need this, And then there's an error in the capture implementation and then you know there's a hundred thousand. Logan attempts. another common sort of technique for doing rate limiting on logging attempts. There's a tool called Fail to Ban, but the word, like the term fail to band is just kind of come to describe this technique where you look at your log events like you have your web server that's running and you look at the logs that are being printed, and if too many logging attempts from N i p are written in a period of time, you ban. The problem is there can be like that window, or let's say it's a thirty second window and you want to do a max of like ten logging attempts. Somebody can shove like five hundred log in it Into that you know, period per I p. address. So it's not really the rule you expect with practice I, o, you know, the listeners of this show are elixir minded, so we're using e t s to essentially track. you know, aevents like that, and then on your six log in attempt. If the rule is like five log in attempts in a period, the sixth one immediately gets banned and that's just because of you. O. T. S is a fantastic tool that you get with on Lixer Code. based. The second example I'll give you is actually a way to buy path I pbasraghtlimiting. So if you're let's say you have your rate limiting set up. There's also open source libraries you can use for this. I wrote a blog posts about Plug attack, which is really fantastic library. I do recommend using it. You know, if you, maybe you dont have a budget for practceal or something, it's great, but you should just kind of be aware that there is a way to buy path. I raight limiting. So what Hackers do now is they're aware of, As has this service called a Pi gateway. Essentially what you can do is an attacker is proxy all of the traffic from your computer through a w. S. And what that does is every T p request now as a different I, P. So all of the requests are coming from the w, s. I P range, but each one has a different Y P. So your Ip rate limiting just no longer works In the logs. It's like Oh, we got a hundred thousand Logan attempts, but they were from a hundred thousand different P. So they're probably legit. but you just you know, suffered an attack. So the way pracselblocks this, we use eroducts to try, which is just an efficient data structure for storing Ip addresses. It's fully implemented in Elixer. That's a really nice library called I, p. try, Um, The code for the agent is open source, so you can inspect it yourself, but I also wrote a block post on this, So this is like kind of public information, but essentially the way it works is you have a plug. that's just block. you know cloud I, P. address you. you. you do want bought traffic in certain parts of your application, for example, up time monitoring on your home page. You probably expect Awsips there, but for your logging page, you know if you get a log in attempt from a rented Aw server, just deny that M. it's malicious nine, nine In percent of the time, And then you know maybe you have a big customer that has like a hosted Vpamonaws. You can just add that I p address and practice I to be allowed through, So you kind of see the benefit of having the web. Um, kind of like the two components, So those are the two examples So far, Just rate limiting and checking for cloud Ipse, I'll pause. Do you at any questions about that

 

I'm like laughing, because those are the exact two problems that I had in my started a little over a year ago that I solved a little over a year ago. I ended up because I tried to for the Padres matching, because there's so many of those in the range and an t. S. but yeah, this is his awesome and plug attacked. Totally. I'm glad you made that call because I feel like it doesn't get talked about as often. It's something that. So I have this like Phenix Temple that I used for every new project that I create. Even if I'm creating a side project. Plug at a guess is a part of that like it's always important to rate. like, like add limit to you know, Have a have a maximum rate right like it affects, doesn't not only affect security, it also affects the user experience of other users interacting with you software. But I guess one attack that I could not really figure out how to deal with properly. I'm curious what your thoughts are is like, just like bounce rate, not necessarily coming through. You know sub set of I P. addresses. That was another attic that we had which I did not know how to deal with. Really Well,

 

Interesting?

 

I would be curious about your thoughts on that.

 

Sorry you said bounce rate. Could you describe what?

 

So so so I guess boundstrit is like called again. I'm using it wrong. I'm not very familiar with like all these terminologies, but it's like how often one specific page of the website is hit compared to the rest of the page that's called bounce right right? I think so

 

Oh, yeah, yeah,

 

right so, but if it's not coming from like a set of ip addresses that is known because You know for a fact that you know if you look at logs manually, there's something wrong Like if someone is trying to hit. If someone is is hitting a public. You know page that has a list of prices, you price shop, and a competitor is trying to like. You know. Watch that like, how would you deal with stuff like that?

 

Okay, so it seems like you kind of want to do analysis on the behavior of people on the site and see Okay, like this is probably malicious.

 

Yeah,

 

Yeah, so that's a very. That's a very difficult problem. Because you kind. It's really defining what malicious means and that's different for every customer. So example practical I owe right now were focused mostly on sort of attacks against web applications. things like the credit card example, Scribe, or credential stuffing on the log in form. However, there's also this whole other area and bought detection relating to advertising fraud, where let's say you purchased an ad campaign and you're expecting you know a hundred real people to view your website, but you look at the traffic and something like ninety five percent of that is bought. You know we're not currently focused on that, but that is kind of like another area of research. It's like very

 

Gotcha

 

act, And in that in the security side of it.

 

Gotcha.

 

So I've been talking a lot about the botdetection side. We also recently launched vulnerability management for elixer applications. I can talk about that if you like, or do you hae any more questions?

 

Go ahead. Could you define Alto of vulnerability management In this particular context means

 

Yeah, very simple. It just means keeping track of all of the vulnerabilities for your elixer application. Um,

 

Hm,

 

I guess I should. I should define the terms a little bit, because it's confusing. so the term like vulnerability in your application. Let's say you have like a sequel injection problem where somebody can send you a string. Um. your application is interpreting that as like an s. Q query. An, then they can dump your data base or modify other users data. That's very bad Security

 

The classic

 

problem.

 

one

 

Yeah, exactly so, you'd refer to that as a sequel, injection vulnerability. Not very common, an elixir, because Ecod does a really fantastic job of preventing it. That's what we mean by vulnerability. Um, but that's that would be an example of vulnerability in the source code of Like the Phoenix app that you're writing, for example, M. But then you also have situations where you are dependencies that you're pulled into your project. They might be vulnerable, so it's not The fact that you wrote the vulnerable code, but maybe this library for pagenation or something in to or in sequel, that's vulnerable. so now you need to monitor all of the dependencies that are being pulled into your elixir, project. Um, or maybe there's a malicious package, or maybe a package is no longer being maintained, so you don't want to use it, Because if there is a security problem that's uncovered, you know there's not goin t be an update to fix it. So the term vulnerability Management, in my definition, it's really just being able to keep track of all of these things. Um. another component to it is, Um, insuring that like the scans or the detection was actually done, So for Peraciaio, there's currently three open source tools that we do want, or building management for Bthe. First is so blow, which I hope is familiar to the listeners of the show. but if you're not familiar with it, it is static analysis security tool for So it's for finding vulnerabilities in the source code of the Phenix application that you're writing. It can also detect vulnerable dependencies, but I don't recommend using it for that purpose. Use it for the stack analysis. It's the best security stack analysis tool for elixir, But the reason you don't want to do the vulnerable dependencies is because that is just in, coded in the source of So blow, it's not dynamically updated. There's another tool Called Mix audit, which scans your project, but it use. it uses the vulnerability data base that Get Hub maintains, So if there's a new vulnerability tomorrow, it will pull that information in. It's dynamically updated. Um. And then the third one is actually built into Hec itself, which is Mix Hex audit, And that's for detecting packages that are marked as retired, so they're no longer receiving updatesso. Recommend using all three of these. If you. If you're you know, you have a company where security is important. You're handling customer data. you know, financial or health care. put them in your C. i, C. D. One thing though is, there's a lot of regulations around this where you need to for audits. Be able to produce a report that says okay When as the last time the scan was run Like, Was it run two months ago? What were the results two months ago? And you don't want to be that security engineer who's now you know, groping through your Ci And trying to convert like the playing texts to like a Excel sheet for the sock to audit or whatever that's going on. It's not a fun job, so prate. Just simplify that process. Gives you a chart showing that the scans are running. Here's the findings and just in a nice display. this is a pretty standard type of product. But as you can imagine, none of the vendors that do this, you know, Like Sneak supports vulnerable dependencies for Alixe. they don't support. So blow, There's other vendors that support Blow. So is like. It's like kind of spread across all of these different vendors. Um, So that's where I saw kind of the business opportunity to just centralize everything for for liker developers.

 

That makes a lot of sense to me. I mean, at the very least you could, You could probably set it up and lock your see system to run it nightly and then report some. But then again you need to do it. You need to set it up to need to manage it to make sure it notifies someone somewhere to actually look at this. So

 

That's hundreds of engineering Ours like it's It's yeah. I mean, if if I had known, I don't know if parexsaleio existed like a little over a year ago. If I hadn't known that you had existed or you had existed, I would have definitely reached out because we spent were a start up. We had three engineers at that time, and I spent like like close to two hundred hours paring solve these problems and that you guys probably felt better than us anyway.

 

Thank

 

So

 

you. I

 

yeah,

 

appreciate that.

 

Um, it kind of kind of paints again, this this old battle, which, just like it's, a mental image of often seen used in the context of security. S that like it's an uphill battle right like that's, this only one thing you need to miss and then you. In the worst case, you're here getting getting kind of caught with your hands down. An ask

 

Yep,

 

your objection if it's like only one one instance

 

Uh,

 

you missed and someone is going to find it, And your way. Have fun with that.

 

I think the thing about security is like a lot of companies they like being reactive Right,

 

M.

 

They see something happening

 

hm,

 

and then you put in the solution to that, having some like, kind of offloading some of that responsibility to security experts, especially who experts within your domain like makes it pro active a lot of times, like with health care and fenttecdata, it's already too late. If you're reactive, you know, they already have a liability which could potentially lead to a law suit and your company. You know now if you're a start up, so Yeah, it's so cool that something that does exist, especially like something targeted towards elixir. at application layer. It's awesome.

 

Yeah, thank you. I appreciate it. and I think what you said about how a lot of security S reactive. That's true of the security industry. In general, you know anti virus. There's all these products you can purchase, or you know, even pracialiois. You can think of it as you know. If you get a scan result and there's a vulnerability. Well, it exists and now you're reacting to it. I suppose I should plug my training out of lixerconfdncy. Like this is just a nice time to do it. There's kind of this culture Shift your scene in security, where they want to train software developers like insecurity. So, rather than having to pay an auditor to find all of these problems, you know six months after the things already in production. Having your engineers trained in security and aware of these problems, it is an investment on the businesses part, because you have to spend the time and money to train your team, but it pays off a lot down the road because you don't have these Curity problems shifting left. If you, If you've heard the term shifting left in security, that's I think, part of part of what it means, But Yes, for my self promotion at Elixercofou this year there's I'm giving a remote training on Phenix Security. So if you're developer and you're interested in learning more about these topics, you don't have to go to Europe. I'll be giving it remotely. I would love to have you know in my class.

 

Yea and night We can definitely include the show. Not for people. check it out.

 

I have a quick question. I, I fig. I'm the only one Have not been talking entire. Just been listening. I'm just thinking you know, I never put this stuff into my, as I don't know. Maybe I'm kind of playing with fire or something. Make me feel like a little bit nervous. Um, maybe do you mind to kind of give like a quick, Small list that people should do when they start off the Phoenix apart. You've already named a couple of really good libraries so below, I don't feel Ike. I'm saying that one wrong. I'm not sure how to have to say that one

 

Yeah, so blow.

 

so you say so blow, but I say so below. So

 

Oh, it's

 

whatever

 

fine.

 

is, um, you know,

 

It's

 

mix

 

so below Alan.

 

M.

 

so

 

hm.

 

below so below me, Uh, yeah, I mean, you named a couple of packages right, maybe to somebody who's been in this industry for some When you're giving this training. Do you mind to kind of give what you think would be some stuff that we should you know, keep in our tool box, or even check blocks. check lix, that we should do every time we create a Phoenix, Uhi'mwonna, say Phoenix Appriphenx is not your P. Every time we create an Ap that uses Phoenix,

 

Yeah, now it's a great call out two because that's a common question. I get. I actually wrote an article on this topic. It's called securing Elixer Phoenix applications. Five tips to get started. It's up on the practice Lio blog. We can go through it right now if you like. If you want to know if you guys can pull it up.

 

What's the name of the article? Exactly?

 

I'll paste it in shot. Is this show edited?

 

Yeah,

 

Okay?

 

Yeah,

 

okay?

 

we can

 

there.

 

ask to cut this out.

 

Yeah,

 

okay,

 

I just

 

exactly?

 

based it on the chart.

 

And then by the way, since we're already cutting something out miccurrenty, you're not muted. So like I think you, kind of.

 

Oh,

 

I don't know.

 

sorry

 

Like what

 

about

 

happened?

 

that.

 

I could hear a scroll. Okay, Yeah, this would be cut out somebody. Hello. Okay,

 

All right. I think I'm back. We're good. So yeah, this article I wrote just five tips to kind of get started with Phoenix Security. The first one is just to be aware that the Arline Eco System Foundation, the security working group publishes guide lines on secure coding and hardening in Lixeranerlang And there's some really great tips in there. For example that talks about Adam exhaustion, which is this denial of service problem that's unique, sort of to Elixer. Erling Adams, aren't garbage collected. But just being aware that this document exists in reading it, I think is probably the best first step if you're interested in this topic.

 

Yeah. I'm surprised that the Adam exhaustion thing is not more people's minds. I've seen. I think we talked about this a couple of weeks ago. I think

 

Hm,

 

audio was not here. There was a guy who tried to slide this in in terms of convenience and I was like, No, no, no, that's that's not going to happen.

 

I mean, scale is a big factor that. rightly if you woke in something that like you know, I just joined the score there. Like so Anti Adams. Like, Yeah, it, just it. Just it could kill your memory very very very quickly.

 

Yeah, exactly, and it's nice because it's these guide lines were written. I was not on the security working group at the time they were written. I'm a member now, but the people who wrote them just are really focused on this topic and it's just fantastic guidelines. Um. the second suggestion on the articles just to simply use. So blow in your application. Um, if your Phoenix application is like a production, know piece of software that customers or users entered their data, It just use. so blow. It's very easy to get set up and it will catch all of those security problems who were talking about earlier, such as sequel injection and cross site scripting.

 

I was just looking and I didn't notice that there's a list. to Adam. It's very weird, but is to a char list. So makes sense As saw that before or not. I've always seen string that to to Adam. Never seen a list that to Adam.

 

Yeah, that's

 

Yeah,

 

why I like about the secure goading guides. Because you think of Okay, this is may be the correct way to do it, but then there's also an incorrect way to do it, which might be lurking,

 

Yeah,

 

and especially large code bases, so it's good to just be familiar with the right and wrong way to do things securely.

 

you know. I also fee a little bit weird To why is it that some of the things are still in? Like still available. Like, Why are we able to create at run time? all these atoms? Is that even a good idea sounds like usually a bad idea, but I guess here must be a good reason else we would be be ripped up by now. right?

 

So there was an interesting discussion on the Arling Melling list. Or it was the Ear Lang for Em about Like why this is possible and there actually have been proposals I believe to do garbage collection of Adams. Um, I think this part of it is that there is good tooling. At least to catch this sort of thing. It is something you have to learn when you're coming to Elixer and Erling, But if you're dealing with this problem, kind of in your application, There's ways to detect it and kind of steer away from it. And then there's also the difference of you know you have your needs as someone who's writing like a web application and elixir. But then Erling has this whole history, and the people who work on Erling and the D P platform. they've put a lot of care into this, So if there isn't, I feel like there's probably a good reason. just based on the interactions I've had with them. They do seem to think these things through very well.

 

Ye think there's a good block post. I don't see plug attack being mentioned here, though, but I mean, didn't mention it over in the beginning, but that's

 

It should be in the blog post. Oh, it's the Tyler Young article from Felt. He used a plug attack. I don't think you can't. You can't search for plug attack in the article, But that link that he walks through using a plug attack

 

Gotcha

 

in production at Felt, it's a great article.

 

Gotcha. Yeah, I guess one more thing I generally add if I'm doing authentication in the application itself is like, like Have I been pawned or something like that is also super useful like make sure the passwords are good, enforcing good passos for you, users like it's oftentimes ignored. like like user level security. Because again, if you're dealing with data that's like you know, health care. I explained earlier Like that's like increase your liability. That's you need to enforce better security measures. Speaking of which, I would love to hear what you think about this, Michael, but like I feel like it's always good to offload things like authentication to external services as much as possible, buying overbuilding. These, especially like crazy, security level, crazy things that that need high level of security like authentication. Um, instead of building it on your own and potential doing something wrong, like always buying something odd. Seriously, One that I generally buy because it's like sometimes free and mostly like very very inexpensive. Uh, but I know people differ, but I on't know what you think about that, Michael.

 

Yeah, in my experience, security is something that is very important for companies that are using elixir. However, they usually are not in a business position where they can have a fully staff security team. There are businesses. I worked for a business that had a staff security team and was using Elixir, And I think that is really fantastic that we have more security. People like doing work in Elixer. However, you do need these things in place like you do need bouton Action. That is sort of a basic business requirement, Or you do need vulnerability management and for teams that are kind of staring down this project of Okay, Are we going to build this thing for the next you know, few months and budget and maintain it. Um, When there's you know, kind of business requirements, features that have to be built. Um, you know you face that trade. off. The other thing too is the security teams of companies they're usually dealing with security problems At are most relevant to the business, which are unique. So what I, what I see is there's large security teams. They're focussing on these problems that are just most relevant to the company they're working on. But then there's also this generic thing of you know, You need vulnerability management. You need bought detection and elixir, So that's where I kind of see the benefit of praccial, where it's common to all of these teams, and by everyone you know, buying it. not only does the ecosystem benefit because now now the company gets better, but kind of everyone benefits Well where the centralization is beneficial compared to everyone trying to build this system on their own. So there's eight vulnerability management. You know, tools or whatever, and that's not always an ideal situation.

 

So it's like someone who's living Europe, and like Germany, Right? Like all of those kind of things, Like what is you said? That you can decide what kind of information you re sending to Peracul? Can you give a little bit of like insight into what data privacy expect that contains? Because I would assume that, especially people from Europe and company from Europe. that is something which is very much relevant for them.

 

Yeah, exactly. I've talked to a lot of businesses to that had concerns about using Google or using cloud Flirt. Because of g, d, p. R concerns, You know, they didn't want their data going through. You know this Ad tech. you know, Big Tech company, or their data being decrypted, So practical, the data we collect is strictly for you know, security purposes, and that collection is treated a bit differently under current law as compared to marketing data. For example, So collecting Ip addresses to do rate limiting or to do alerting on. you know this, Ipded too many logging attempts. We'd like to bandit and collect metrics on that M. that is treated differently, for example for using the information to market or track users. Um, so pracalio. it's strictly security software. You know, we don't sell data or trade data with any third parties. It's collected by us, strictly used just for security purposes, and then deleted when you know you On using it. Um. That's something we really like. We don't profit off of like data resale or anything. Um, and it is very common concerned with businesses in Europe, especially they're very interested in. You know, where's their data going? Do I even have control over what data I'm giving to my vendors? You know.

 

Yeah, hank, you, I, if you're not, don't mind, Michael. I would like to ask some bit harder technical questions because I'm curious. and basically earlier you said, for example of rate limiting that you're using T, S and I, I'm also very much fun of S. But the point then it's like What if you have multiple notes running right? Like what if you have multiple version of? Maybe they're even clustered right at that. Some people are doing clustering. I don't think it's a very common thing happening with every application out there, but I, for example have built cleric That application. So is the something there where perhaps it also may solve some of these problems or offers like alternative, plugable back ends like reds. I mean the dates, like the the non clustering approach. Let's say that right, like you have, maybe readisponding somewhere and then you can keep track on that for rate limiting purposes. So on so forth.

 

Yeah, so it seems like you have a clustered application. You have requests coming in and then this problem you've described. The requests are being routed to different modes and it

 

Yeah,

 

seems

 

exactly

 

like you, you want to introduce reds so that there's kind of this event or like consistency where they like Okay, we compared across all of these modes and we've come to the determination this Ip did like five logans on this note, three logans on this node. And then that's communicated. As you can imagine. That does introduce quite a lot of overhead for

 

Yeah.

 

users,

 

I'm aware.

 

So that is not something that's currently in practceal. For example, we do do monitor on the back end as well, so it's like the two parts we do t s locally just because it's such an. It's just so beneficial to do it. It's like built in, but then we also have the back end component. Um, Also, in my experience, maybe it's different for you, but when traffic comes in, it seems to get routed to the to the same note. At least for a burst of it. you know enough

 

Yeah,

 

to

 

usually

 

trigger

 

it is

 

rate limits. Yeah, yeah, so there. This is like huge benefit to doing it in T. S. because it's like

 

M.

 

you can kind of ship it today. You know you can get rid of set up and everything, but it is a lot more work. I'll say.

 

If ever men kind of play thought, I will play a little bit of devil avocat here, because I would assume that some of your listeners also have some of these questions. But so you do you? I wanted to say something.

 

I was going to say like, I mean, it will also slow down each request right, like if you have like a central shared rites instance that every lust, every note has to talk to you before. like, Oh, I'll allow the iser to go forward, right E. S being in the same vicinity as the application as that advantage. If I would be solving this problem, which I'm not quite sure isn't needed to be solved yet because of what Michael said Like, Mostly you expect it to be in the same note, and even if it's not in the same note, the cost That might not be as bad. you know like because you know if I'm making five million request, and if I make five million first request, and if it goes to a new note, Five lion second request might go to the same note to write like again, only for one request. It will add the advantage, but say, let's say it adds. Let's say it solves a problem. I would still use T. s. and you know after the t. S verification happens, I would write that to you know, shared ready or something, And do like a post response sink. With other tis, you know, not let it slow down the initial request. That's

 

Yeah,

 

coming.

 

far enough

 

Yeah, and have like a lag between S. That might be a good way of doing that, but I actually, I might have misunderstood what Michael was saying about the thought he was using T. s for the to store the try, which is static right. Pretty

 

Oh,

 

much.

 

oh, it's a very funny story about that. if you, if you like to

 

Okay.

 

indulge.

 

Yeah, go for it. Yeah,

 

So when I was researching how to solve this, Um, it's the subject of the talk. I gave it a lixercof. It's okay. you have. let's say a million I P prefixes So you put them in the try. That's like part one of the problem, but in elixer, shared access to a large chunk of memory is actually a pretty difficult problem Because of you know how I like Work, so your first stop might be to use a Gen server. I just put in a Gen server message comes in. You're shaking your head because you know you've just introduced a bottle neck. This is actually an

 

Exactly.

 

example from the elixer. Like getting started Guide. You're like Hey, you have a bottle neck now through like one process Gen Server Kind of just made your single threaded a bit. use t. S. instead. So naturally you, you reached for T. S. Second. The problem Run into. If you're using the library, which is what I did is it's the I. R is represented as just a map and elixir, and T, S does not have like a native data type for maps, so let's say it's like twenty megabites every process that calls in to T. S. That's a twenty megabite memory copy out to the calling process. So you get enough request, Then you just blew up your memory. Consume. And there's a feature though in earlaying called persistent term, and it's created for the purpose of kind of shared access to a big chunk of memory on the down side. S. you can't updateid as frequently. So for this problem of the Ip addresses, let's just remove the requirement for updates. Let's just stay on start. because these things don't really in. They change like once a month or something. Maximum, Um, just remove that requirement. And then Now, when you call into it you know it gets created on your application. Start up. there's no longer a memory copy. Um, there's also ways to do it. Actually, I'm playing around with this idea of like, kind of a railway system where like you keep two copies of it and then you can actually change it by like switching over. Somebody talked to me about this code beam, but esentually that that's if you've never used persistent turn before. Here's kind of like a real world example of why you would want to use it instead of T. S.

 

That's such a cool idea to persistent term. That's really cool. I had the same problem with. I was going to ask that question to you. How to rest Of tiring to try, I had to create my own structure, And it s that so I'm not accesinask. much amount of memory at once, But that made my implementation complex, but persistent term. I don't even think about it. That's such a good idea.

 

I feel. persistent Term is one of these little unknown tools in the Elixer and the early Developers tool belt, which it is rarely you could use case for it, but sometimes it comes around me like, Yes, thank you.

 

Exactly

 

exactly this.

 

Yeah, I guess it's good to summarize it if you have a lot of data that you want to stare in memory and you're like let's use t. s for that, But if it's not getting up dated as often. consider pesistent term like that's, that's a good. summarize. it.

 

Exactly. Yeah, Because I was reading the documentation. I didn't really click for me until I was like, kind of working through this problem. It's Ike. Oh yeah, that's becase. There's a warning label and the Arlie documentation. like, Do not use this as a replacement for T. S.

 

Yeah, we. also. we also used persistent term once in an application where we kind of red configuration from from a file on this and that occasionally changed. But like, maybe I don't know. Like every few days because somebody changed the configuration thing there right like it was. I think it was mounted conflict map from like Cubanetis, And then we actually had like one process which read that thing like every half an hour something, and then just up to Videnpersien. I'm not exactly sure what the timing there was, but it was also a good use case. We got. It's like one of conflagration things, which need to be excess, but basically every process on the particular application, so it can be really useful, but it's one of those things. Yeah, I'm not even sure how we came across it like. I think somebody was like. Wait. I think there's persistent term. I heard it before and it's not. I don't feel. It's like that much public knowledge. So to speak,

 

They heard about it on your podcast.

 

I don't think so, because only after that afterwards I spoke about other podcast. At some points, maybe maybe previous piso, I wasn't here. So yeah, I actually ave another question because I would assume you probably managed that pretty well with poactil, but I think some other people have the same question, and I don't presume that for every request which comes in, you talk with like your infrastructure, proecture, right, because they will just introduced one A relatency. But what I'm getting at is basically how gracefully does the integration into your application degrade? If there is a down time on your side right? Is there even any kind of degration happening? Is there even some kind of phoning home to a certain degree? And what? Basically What happens if I have Frank integrated by application and your service?

 

So this is this is. actually, it's interesting because engineers really like the answer I'm about to give, which is, let's say, practceals. you know, having an issue, your application will continue to function as normal, which is exactly what you want. You know. No down time

 

Yes, Yes,

 

from a business point of view, Though this is horrible, because what you really want is to make your customers as dependent on you as possible. So like if cloud flare goes down like your site goes down, and you don't want to change your dens setting, So it's like very sticky. So that is something that's nice for developers. Is practseal. It's very easy to install. It's very. You know. it's not going to get in the way of your application. If there's an issue, it's sort of. it would fail into a position where you stay up, which is like, kind of in the spirit of Erling and Elixir, I think M. But it's also a little bit of downside from a business point of view, because it's now easier to remove or something. Um, where it's like these big enterprises. My conspiracy theory is they make them so difficult to install. And now you'll never remove them because it's like so integrated into everything.

 

So

 

I think

 

if you

 

it's more than just a conspiracy theory. Some truth to that.

 

just wouwould, you be willing to give a litte bit of insight Like what kind of communication do you usually do with? Like you like. What is something which like when, when I have praccil set up in our application? Like what kind of communication does happen there? Like what kind of formation does we potentially fetch? And like If you're down for like I would assume, like, one day. Probably not big of an issue. You don't for longer than that. Any of different problems anyway, right, but like just to give. Like a bit of an inside there.

 

Yeah, exactly so that's a really common question to I get is when the request comes in. You know it does. Now the practcial agent have to make a network request and wait for reply before it goes through. That's not the case, so I'll give the example for the two we've talked about for rate limiting. So if you have a rate limit, that's like you know, five Logan attempts in a five second period that's evaluated in e, T, S. like locally on the machine, So there's no latency. Really delay at all the micro seconds for T. S evaluation. but there's no round trip at all, and then the evaluation is also done on the back end as well, for let's say a longer rule where the memory consumption would blow up because you have too much data in T. S. that's done on the back end as well. So there's a network communication there. For example, your the allow list or the block list. There's kind of like a regular at P request. It's very similar in architecture to like the application performance monitoring you're probably Using like Ap, Signal Century, New Relic. All of these agents kind of have a similar set up with the C. P. requests. Prac, sort of similar set up as well. The other thing would be the cloud I P. address example I gave you, Because that's the roducts. Try that's compressed on the back end and just retrieved it. Start up. I had actually the idea to release like a library that would pull all that data down as like an open source tool. The problem that I ran into the reason I shifted it to the back end is downloading all of that data and then passing the different for mats from the providers and compressing it blows up your memory consumption. You count, even deploy on like the fly Iofreeter anymore. That's why I kind of abandoned that project. but essentially for evaluating if a request is coming from the cloud, I have to be provider. It takes the bench mark in my talk, but it's like micro seconds you know, So there's There's no network communication at all for all of those because it. It's a very important performance wise too, because everything is coming through your plug. So you really

 

M

 

want

 

hm.

 

that code to be fast?

 

Ah, That makes a lot of sense. thank you. thank you, Michael, and also so so that that I guess a lot of questions people might be having you. You took that into consideration. right like a ever. Try to get to a point where this thing helps you and get out of your way.

 

That is really my goal. I really don't want practcials customers to have to worry about security any more. I want them to feel like

 

M.

 

they have this protection and they also have a company that understand what problems they're having and how to solve them effectively.

 

I guess I have a very quick question to say, the I P addresses and one of the cloud providers gets updated. Right What happens then? Like you said, the persistent term is loaded at the start time right. Like what's the process of likeupdating the list of ipaddresses and recreating that. Try

 

Oh, you would just re deploy your application and I would get updated pretty straightforward.

 

Awesome,

 

Yeah, exactly

 

And whenever you start, the agent pulls latest version of

 

Yeah.

 

the story

 

exactly.

 

part.

 

You know. it's great to feel these questions to, because I get them all the time on calls and stuff, too, so I feel like your audience is ideal because they're probably asking the same question. So and being

 

Yeah,

 

able to talk about it it's fantastic.

 

I will assume the next time you get these calls you can just point at this podcast episode like here. Listen to this,

 

I don't know if that would not my. now.

 

Have you seen? Is it the it crowd that the answer the phone like that? Have you got seen episode? Now there think it's called it crowd right, But it's a show in U. K. where these

 

Yeah,

 

guys are.

 

yeah,

 

I don't know. There was no. It's every episode. just one. I've seen where like they just pick up the phone and put it next to a recording and they pressed play And it's just the same questions for every guy idea. Did

 

Uh,

 

you try it were

 

uh,

 

okay, we started Okay. Okay, Yeah, it works great like they just recorded all their own answers and Evrything just matches up every time

 

Nice,

 

You on'tkindofyou don't kind of feel like that that like, sometimes you really just answered the same question over and over again. Does it drive you crazy

 

No, I really like it because I think it's a competitive advantage for the business that you get to talk to a real person who actually is listening. Um, I mean, that's true. like you do. That is like a business optimization thing where somebody at a v C back, Um, Pyper security provider said, Oh, here's like a copy pace thing. So you talk to them and it's almost like a chap might literally be a chap bought these days. Um, but I view that as an advantage of track seal where you know somebody is going to list, And to you like a person is going to really listen to your problem and come up with something that's bit more thoughtful than that and more respectful of your time as well,

 

If you're like founder of a company that passionate about you want to keep talking about that right, because that that's what excites you.

 

Exactly. I love it. I don't feel any you know problem. It's great just being able to talk to. I think that was probably a big motivation you said. Did you wake up and just start the company one day and I realized I want to talk to like people using a lixor. so this seemed like a good avenue to do it.

 

I guess I have one more question. Im going to circle back to that, the persistent term discussion we were having, So the the deploy time the application start is dependent on Peraxilios up time. then right,

 

Yeah, it's just a network request. It's pretty straight forward. You can. also, if you're in a development environment, you know, disabled the agent naturally, Um,

 

Gotcha

 

so it's just you know the development experience is better. that sort of

 

Yeah,

 

thing.

 

got it awesome.

 

Now? are you as I called Dog Dog footing? Are you also using plaxouiom on your own

 

Yes,

 

website?

 

I absolutely love the fact that we are. It gets a little bit confusing. I don't know if you've ever read this book. Go to Esher block, and in the book there's like these recurs ive stories where there's like two people telling a story about two people telling a story, and then those two people and it's um. it kind of goes down, but the praccalio itself is defended by peracclio. But then I need to set it up where there's an instance of praccil. Oh, for the Praciolio back end, and I have these diagrams which are very fun to draw, But that is something that I think is cool about the company. as well as that we are. We are dog food. In that way,

 

Okay folks. Unless is there anything else you would like to add Michael before we kind of transition to the the pick part of a podcast.

 

Um, I, I suppose I should do my plugs and things. If you want to talk

 

Yeah,

 

to me My email. it's Michael at practice. Io, also pretty active on linked in. I'm sure you'll you'll put it in the show. Notes were on Twitter. linked in as a company, so out to Gen server, Dot Social were active on there as well. It's the petty verse instance. Um, yeah, I suppose I should also mention the. I don't know when this this episode will come out, but you know, presumably before the training that we Talked about. If you're interested in, you know, learning about Elixorandphenix security. I'll be giving the training fully remote at Elixorcof in April, so I hope to see you there as well. Yeah, thank you guys.

 

Okay, then let us let us talk about picks. So what ave you brought today?

 

Yeah, I have a cool one. Sou a game came out yesterday today. whatever, hogwislike right, been excited about that for years, But what happened at midnight? Because you know my wife and I was excited. We trying to sign up to games dot com and they had a ball attack, so I've already mentioned that it's super important, super important at launch to have some kind of some kind of throttling, some kind of attack prevention. But that's that's my only pick today. I think. Oh, I think. also quick celebration. I learned that three people. well, Yeah, okay, three people got a job through our shout out on this podcast in the last year, so super excited, super excited about the fact that the shoots are working. To some extent.

 

No, Ellen. What are your pick for this week?

 

Yeah, I just got one pick. I've been playing this crazy game called Vertigo. You ever heard that one before? So it's Alfred Hitchcock Vertigo. It's a very famous film called Vertigo, but the game is got like. It's like nothing to do with the movie. Necessarily even tough. I, says Alfred Hitchcock's Vertigo on it. The story is just outrageous. It's basically a very simple point on click kind of game. Uh, yeah, like you just look at something and you press button, and that's it any. It's kind of the screen tells you what to do, But the story for me is just like Drawn Me and it's just so outrageous. I'm almost near the end and I just every time I get closer to finishing the story because I want to know the ending of it. something always pops up. I have to do something somebody calls me or bugs me or whatever. So I kind of regret having a wife sometimes, but anyways, anyways, it's really fantastic game. If you like interesting stories, I think it's really great. I'm surprised on steam. It only has mixed Recent reviews. Uh, but yeah, I think it's about like a twelve hour long game or nine or twelve hours. It's not too bad. The only negative part is like there's no like how I say The whole game is like a movie Where you're just there's like no stopping so you don't rally. know he, you'eactually playing the game or watching something just random. Tell you like to press a button. If you guys remember that game from a long time ago were like you're like a movie cartoon character and you got to press a button at the right time or else you die. Something like that

 

No, Ellen. I

 

forgot

 

actually

 

that

 

have

 

name.

 

no freak ing clue what you're talking about,

 

Anyway, It's a really cool, really cool game. I got it on Humble Bundle a while ago, and I think it's pretty cool, so that's my pick.

 

Michael. Do you have any picks for us?

 

Um, before this episode started we were talking about about picks, and I was going to do three blue one Brown, which is the Mathtyoutube channel. Um, I feel like viewers probably seen it, but he makes like animations for math concepts like like how Bit coin works from a like a technical point of view, or like linear algebra. But I suppose I should do a video game to, because everyone else did Um,

 

You don't have to don't feel obliged.

 

Because the last one I played was the. It was the original Daasac Um, which came out like over two decades ago. I think, but I love that game. It is so much fun. Um, and the plot is crazy. It's like there's a pandemic and this conspiracy. You're like an agent for this. It's called you Nackco, in the game, But it's it's like. Kind of. it's kind of got this cult status so I feel lie. There's a little bit of like renewed interest in it, But I love it is and I think it's like ten bucks on good old games or something. I don't own a gaming pie Thing right now, so a lot of the games I play are old. Maybe I'm just getting older, but yeah, that's my pick for this week.

 

Nice, thank you. I also have to join the ranks of picking video games. I sorry, but my pick this week is a game I've had on my back lock for the longest time. Like I did start play it a few months back. And then I kind of lost interest. And the game is this Co, Elysium, and I picked it up again

 

Oh, it's

 

recently.

 

great one. Yeah,

 

Yeah, because they

 

That's

 

actually

 

a great game.

 

a while ago they added voice acting and that actually changed the whole De. For me. I was kind of surprised for how much that changed my my experience with it, because before that it was actually purely text, excellently written text. I must say, like the dialogues are amazing, but it's still. It's just text. I enjoyed it and then I put it down, planning to play it again and I never did How summer times these things go. you know, but lately I've been starting another play through and like voice acting is amazing. So So good. And what this Clusius, basically about? It's a role playing game without any combat Has no combat. In this there's just a bunch of mental and it's also physical attributes, but they're all like weird, and it's about this cup, which is at a low point in his life. I mean, the game basically starts off him waking up from been drinking and been drinking so much that he lost his memory like he doesn't remember who he is, But he's assigned to solve a case of his death guy hanging hanging hanging on a tree. And but he kind of has to puzzle together who he is like murdered this person and it's very entertaining and pretty hilarious, so I can really recommend a disgilesium and another pick would be. Since we talked about rate limiting, There's actually one plug in one library. You can be using there, maybe for personal project, or whatever, It's called, Hammer, And it's a plug, which is also has a plug able storage back. And so in theory you could be using it. I think they even have back. Also the defopacdi's. just so If you maybe have like a personal project, and for whatever reason, what we talked about today with Parapara, I knew this would happen. Paraxialforaxi is not something

 

It's all

 

you

 

good.

 

can use. then I am at least a little Tool you can be using to set up a rat limiting. so yeah, that is something I use in the past. I was pretty straight forward. I'm pretty happy with it so I can recommend it. Okay any,

 

Hey, I got. I got one more pick. I forgot about this,

 

I go ahead.

 

so I played a very cool board game last week, Um, first board game that I've played. That's like not turn based like real time Is it's called Captain Sonar. It is epic. I was so surprised by how much fun it was. It can't get tricky. I think it's also minimum six players. But can I think it's like eight or eight or ten players, Ma, And which is like there are are many boat games you can play with tent players which are this much fun. Um, Yeah, definitely given a try If you guys aren't about games, Captain Sonar.

 

Okay, then what's a pleasure talking to you, Michael. Thank you for coming into the show

 

Yeah than you for having me. This was a fantastic. appreciate it.

 

And thank you all for sticking sticking to us and for listening to me screwing up the name. for I knew this would happen. Was so happy I got here without doing that. so yeah, thanks for listening to in next time when we have another episode of exhibits by bye.