Will_Button:
Cool, well welcome everyone to another episode of Adventures in DevOps. I'm super excited today because back from her worldwide grand tour, extravagant lifestyle, it's Jillian Rowe. Hi Jillian.
Jillian_Rowe:
Hello.
Will_Button:
I'm sorry I missed last week, but I am glad to have you back.
Jillian_Rowe:
I'm glad to be back. It's nice to be back and be on a more normal schedule again and I don't know, be out in the world talking to people or at the computer when talking to people. I don't know.
Will_Button:
Yeah, it is. It's a challenge anymore to have like real face to face conversations. I think so many of them happen over Slack or Discord or email and you get the occasional hangout or zoom conference. where you can actually have a real conversation. And even a lot of those, you've got like 15 or 20 people in the conversation and you're like, yeah, I'm just not even gonna try to say anything.
Jillian_Rowe:
Yeah, you know, in line with what we were saying earlier, and like the kids need to get off my lawn, I cannot keep track of these like streaming conversation apps where like there are just conversations happening in real time. Like I can kind of keep track of it's like one of the old school forums where like people start a thread and then it just lives there for three years. But these things where like people are just talking and it goes, I'm supposed to keep up with that in real time, like. I can't do that. That is, first of all, a lot of peopling for me, and second of all, like, just, I don't know, I just can't do it. I don't have, like, the working memory or something like that for it.
Will_Button:
I agree. To me, it's a lot like standing in a subway station in New York City, taking part in every conversation that everyone else in the station is having.
Jillian_Rowe:
That is, that's a very good way of putting it. I like that. I'm going to steal that for the next time I have to describe like just how uncomfortable I find all of these real time streaming apps. The only one I can deal with is Slack sometimes, but that's only because it's work related and it's like very specific conversations. It's not, uh, I don't know. Yeah. It's just, it's just very like specific guided conversations about a specific topic. usually when something is broken and somebody wants to tell me about it, which I think is a good segue into our show's topic today.
Will_Button:
Absolutely. So today's topic, I've been thinking about this for quite a while and had some random conversations about it. So today we're going to formalize it. The conversation is, what do you do for protecting yourself from those third party SaaS apps that are so cool and so valuable? And I'm specifically talking about not to call these guys out as like, guilty of this, but just using them because they're common examples that I think the listeners can relate to. Like Terraform Cloud, Terraform Cloud has access to your infrastructure environment. And another example is a data analytics tool like Snowflake, where you're sending them all of your data that you want to analyze. So what happens, not if, but when those third party providers get hacked? If we take a look at the Terraform example, if they get hacked, it could potentially mean that some unauthorized party has full access or has access to my AWS account. Or in the case of Snowflake, if they're able to access the data I've sent to Snowflake, now someone has information on my customers that I've sent to Snowflake. So either way, it's my business. So I'm on the hook for making this right. with my customers, but I have this feeling that the third party SaaS providers are not really on the hook. They're like, oh, hey, yeah, we got hacked. Really sorry about that. And I think as far as remediation, as far as I know, the best you can hope for is getting an at mention in the tweet from the CTO where they say how sorry they are and what they're going to do to keep it from happening again. But my customers are. not really going to be okay with that as the remediation so that's our topic for today.
Jillian_Rowe:
I think that's a really, really good topic, mostly because my answer is I don't know. And I like talking about things where the immediate answer is I don't know. I work with a lot of healthcare data and those people are like deeply, deeply paranoid. So I would never get the okay to send any data whatsoever to any kind of third party application. But if you've been thinking about this, I mean, you must have some experience with it. I don't know if you have any horrors kind of stories you can actually share, or maybe anything you can talk about in general terms, but what- What do you do? I mean, because, so I know for myself, I have contracts and there's like very specific clauses on protecting data and protecting any kind of intellectual property. And that includes like data, data like data that's collected in a lab, as well as access to that data as well through, you know, like if we're on AWS, something like IAM credentials or secret keys or whatever. So what do you do in that kind of scenario? Do you just not tell your clients about it and hope they don't know?
Will_Button:
Right? And then when they come up, go, oh, now that was probably one of any other thousand of data breaches that happened this year. Wasn't from my stuff.
Jillian_Rowe:
I don't think that would help your story though.
Will_Button:
Hahaha!
Jillian_Rowe:
I don't know. I'm not sure that's the best approach that you could take to inspire confidence, but maybe it is.
Will_Button:
No, I've worked with quite a few medical companies as well in my background and had the same experience as you where they're just like, no, no third parties, end of conversation. And this is a large part of why they take that stance. And it's understandable, you know, because I think medical industry is one of those few there's actually financial implications to data breaches. And let's be realistic. The people who were affected by that data breach never see that money because of attorneys. But that doesn't mean that the organizations that did experience the breach didn't get hit with massive fines that are then. levied against them and paid and I don't actually even know where that money does go but I know it rarely makes it to the customers. So I understand their motivation there. So yeah I think like there's some things you can do right there's some level of responsibility if you're going to use a third party provider there's some level of responsibility that you can take. And the first one is The same with everything we do on the internet. Understand that you are going to get hacked. It's guaranteed to happen. The best case scenario that you can hope for is to just know that it happened. And then minimize the blast radius. And we do that through granting the. you know, the least amount of privileges necessary to do the job so that whenever this particular thing does get hacked, they only have access to a fixed set of resources. So I think that's the first thing is just knowing that it's going to happen and minimize the blast radius. When you're talking about third party providers, using the examples that we've talked about here, you know, someone like Terraform and your Terraform cloud settings, you know, make sure that Terraform only has access to the AWS resources that it's responsible for deploying, which still can be quite a bit. And then this is new to me. I didn't know about this until recently. GitHub has implemented OIDC, the Open ID Connect Protocol, where you can integrate with providers like AWS and Terraform. and generate a secure JWT that then gets a set of one-time use credentials to do its job. So you kick off your build in GitHub, and your GitHub action says, oh, I've got to deploy this on AWS. So then it goes to AWS with a secure token that AWS verifies that token and returns a set of credentials. And then GitHub uses those credentials for that one job. and then those credentials immediately expire. So that's another way that you can minimize the blast radius.
Jillian_Rowe:
I do like that approach too, even though all my data is always hosted internally, it's never going anywhere. I do always try to make sure that using keys or secrets or all that kind of thing that expire you can set up, in AWS at least, you can set up auto rotating secrets where essentially like after a period of time they get expired and then they're automatically regenerated. So I mean, if I did have to talk to a client, I would at least try to take the approach of... like, hey, listen, these things happen. We prepare for these kind of events happening, and this is how we prepare, and your keys are changed so often, or that approach that you just talked about on GitHub, that's a great one. These credentials can only be used once, and then they expire. I would hope that you would have some kind of logging if you have very private data. You can set up logging in terms of who's accessing it, like from what IPs, how many times is it being accessed? And then you can get, you know, like if things are kind of looking fishy from there. So then you can always do sort of a post-mortem approach. Well, you know, where we breached, did we lose data? Did anybody gain access to any data? And then follow the protocols for that afterwards.
Will_Button:
Yeah, and I think those same rules apply to the third party SaaS providers that you use. Those are all great practices and a lot of them on their websites will say, yeah, we do this. But then how do you know that they do that? Have you ever called up a third party provider and said, hey, can you send me a log of everyone who's accessed? my stuff that's living in your network for the last 12 months? I don't know that that would, I mean it seems like a legitimate question. I've never done it so I'd be interested to try it. I'm actually going to try it and see what happens.
Jillian_Rowe:
I don't know either. And I mean, my sort of distrust of the third party applications as well is what's not to say there's not some piece of code in there that's mining your data. Even if it's just like metadata about your data kind of deal, how am I supposed to protect against that kind of, that kind of breach as well? Which I suppose, I mean, for that matter, it could be there in open source software. It's just less likely to be there in open source software because we're all broke. We're not paying for those kinds of bills to have like.
Will_Button:
Really?
Jillian_Rowe:
I'm going to write
Will_Button:
Right!
Jillian_Rowe:
some Python library that has some secret piece of code that's logging a data dog. No, I don't have that kind of money. What's wrong with you people? So I'm not sure if that's a good reason to have more trust in the open source community, but that is why I do. That's it. That's it right there.
Will_Button:
Yeah, it's like, have you seen my backlog? Does it look like I have the kind of time to go do that?
Jillian_Rowe:
look at my quickbooks profile! no i do not! I have daughters that like to shop people, I have better things to do.
Will_Button:
Right? Cool, yeah, so minimizing the blast radius, I think, is the first step. And that's the one that we can own. Having this conversation with your provider, I think it's a good exercise to do. I haven't done it. I'm going to implement it. And happy to share what that looks like
Jillian_Rowe:
That does sound
Will_Button:
here
Jillian_Rowe:
like an interesting
Will_Button:
on the podcast.
Jillian_Rowe:
experiment. Let me know how that goes. Like, do they just, you know, do they even have it? Do they immediately cough it up? Do you have to, like, how much escalation are you gonna have to do before you actually even get, like, a yes or a no, we will give that to you? Because I'm sure it's not in that, like, first tier support when you email the help desk. That's
Will_Button:
Alright.
Jillian_Rowe:
gonna have to go up a couple levels there. You're gonna have to get, like, one of the back-end engineers who's just very disgruntled with their life. to be able to get that kind of data.
Will_Button:
Yeah, just going to open up the web chat on their website and go, hey, can you send me all the data for who's accessed my account? And the first level responder is going to be like, are you kidding me? Yeah. Chat
Jillian_Rowe:
No.
Will_Button:
was disconnected.
Jillian_Rowe:
We are having technical difficulties right now.
Will_Button:
Right?
Jillian_Rowe:
We will be with you in a moment.
Will_Button:
Yes, I think the other part of it too is, and this is unique to, actually, so I'm working for Polygon now and we have a legal team and this is the first time, I've had a pretty long career, this is the first time I've ever encountered an organization that handled it this way. So before we can use any third party provider. We negotiate the service that we're going to do, and the pricing, and all that kind of stuff. But then before we can sign that, we send it over to our legal team. And they actually review it. And then supposedly, they go head to head and battle with this company and hammer out these details, which I thought was really cool because I think it's one of the places where attorneys really do add value. And making sure that either you are protected or if not you have some recourse available to you. And so I thought that was a really good approach. Not every company has a legal team and even those that do
Jillian_Rowe:
Yeah.
Will_Button:
have legal teams they may not have people on that team who are familiar with data breaches and data storage and that kind of stuff so they may not know the right questions to ask. But I thought that was a really cool approach.
Jillian_Rowe:
be like a very interesting breed of lawyer. They must have some very interesting stories that I want to hear very, very much because, I don't know, they must have seen some things, right? Like, that would be cool.
Will_Button:
Yeah, there's, um.
Jillian_Rowe:
Try to get them on the show sometime.
Will_Button:
There was actually a Twitter account that I stumbled across a few times that was an attorney who was a former programmer. So they were very knowledgeable about writing code and data storage and that type of stuff and then became an attorney. And that was the whole context of their Twitter account was like. like just legal fails. It was like a. I'm completely drawn a blank on the word but it was like just a sitcom TV show done on Twitter of things that they saw and you just read it and you're like oh my god. It's like I want to laugh but I'm probably who they're talking about.
Jillian_Rowe:
You It's like, oh, just moving along, sweeping this under the rug. Like, well,
Will_Button:
Right?
Jillian_Rowe:
we'll be going now.
Will_Button:
Um, yeah, so. Minimize the blast radius, have your legal team review it. And then we touched on this a little bit, I think having the conversation about... you know, show me that you're following the process that you're doing. And this falls a lot into like PCI, SOX compliance, HIPAA, GDPR, where you have these compliance, regulatory compliance frameworks that you have to follow. And it almost feels like there's room for that here. But. I really don't want to see the government get involved in this because then it'll just never turn out to be anything actually usable. But it feels like maybe there's room for like some community like as a as the software building community like we can agree on these guidelines and just like self police ourselves on that.
Jillian_Rowe:
I mean, yes and no, because the software isn't built in a bubble, typically. It's built for a reason. So if you take like a lot of, okay, maybe not so much health care, but let's just take like the biotech space
Will_Button:
Hahaha
Jillian_Rowe:
in general. There's a lot of money put into research that's publicly funded research, right? It's funded through government money, through grants, this and, you know, all these kind of things. So the results are supposed to be accessible to the government. And for that reason, you have a lot of these kind of rules around not only do you have to... to check the data and make sure the data is like anonymous if it comes from people and all this kind of stuff. But you also have to make sure that you're not losing data which is another really big aspect of like research. And then also it's the same thing for HIPAA compliance too. You can't lose data. So I don't know, are we, I mean, yeah, I get it because I kind of like the sense of autonomy that I have being in tech and that a lot of time I'm just left to my own devices. And I think that's probably what attracted me to this kind of. in the first place, but then on the other hand, there are like actual real world consequences to data breaches happening or losing data. My favorite story to sort of bring all this together is during COVID, there was a company in the UK that was using an Excel spreadsheet as a database. And first
Will_Button:
Yeah
Jillian_Rowe:
of all, I'm sure that somebody on this team was like, I'm sure at some point it came up, this was a terrible idea. I'm not ragging on the team. You know, they were like sleeping under their desk and like somebody was like, get this, get results out now, but they actually wound up losing a lot of data because the Excel spreadsheet could only handle, you know, I don't know how many rows, however many rows it was. And then it just stopped depending data. And that was, that was government funds that was funding like this contact trace study to do that. So it's, I don't know. I can kind of see both, both sides of it. I don't want my tax dollars going towards crappy code, but then it's all crappy code, so what are you gonna do? And if it's not crappy now, it'll be crappy in five minutes, like just wait.
Will_Button:
Yeah, for sure. Yeah, I don't definitely not calling anyone out for writing crappy code because I would be the poster child for that. And
Jillian_Rowe:
I would be first on the chopping block there.
Will_Button:
that's, you know, I think that's a fact of life that we readily acknowledge that good code today is going to be crappy code tomorrow. It just doesn't age well, despite the fact that it's, you know, just ones and zeros. They still have a half life. Um... But I think there's still room that you could say, you could not publish the data, but share with the people, share with your customers. Because from a SaaS perspective, I'm their customer. And so I think there could be a generally agreed upon set of principles that say, hey, as our customer, we'll share with you. these details to show that we're actually upholding the security things that the landing page on our website said we were doing. You know, this week, here's some kind of indicator that we rotated our security keys, or we reviewed our access logs. And we accessed your account this many times or something like that.
Jillian_Rowe:
Yeah, I mean, I do definitely know companies that do go through that process with all of their SaaS providers, even, you know, it's not exactly a SaaS because it's self-hosted. So, for example, if you go to the AWS marketplace, you can find quite a few kinds of software that like they're a SaaS and that you can get the hosted version. And then you can also pay the fee to get the self-hosted version. And then you get like some kind of support or AWS CloudFormation template or something like that to be able to host it internally. And I do know companies that they not only go for the self-hosted version, but they do also go through this kind of process with the, with the SaaS provider. I think a lot of that is to make sure that there's like nothing in the code that can be sort of reading or mining their data or anything like that. But I'm pretty sure there's a lot more to it than that. I'm not, I'm not entirely sure though, because I've never really deeply been a part of any of these processes before.
Will_Button:
Yeah, me neither. Other than just signing up for SAS applications or working in organizations that have third party SAS applications that we're sending data to and just kind of going, huh, wonder what they really do with that.
Jillian_Rowe:
I don't know, I just, I really have trouble believing that anybody could be that kind of altruistic because, you know, data is the new oil or so I'm told. That there could just
Will_Button:
Hahaha.
Jillian_Rowe:
be all this data and they're not doing anything with it? They're not doing anything with it. I don't, I don't know, I think it's like, you know, when we all found out Facebook was mining our data, was anybody surprised by that? Anybody? Like, you know, just, they were getting the data, of course they were doing things with it. I just, I don't know. I don't know. Also to be kind of selfish about it, a lot of this kind of stuff is what keeps me in a job because a lot of places are like, we want this, but we can't send our data to this third party provider because, you know, HIPAA or legalese or privacy issues or whatever. So we need to build it in house. Come on in and build it. And that's where I get a lot of my business from.
Will_Button:
So you're just not motivated to address this problem at all.
Jillian_Rowe:
no no i am i'm very self-motivated i don't know like when this when this bubble is gonna burst but i'm sure it will someday everybody's out here trying to pull
Will_Button:
I'm gonna
Jillian_Rowe:
me
Will_Button:
eat.
Jillian_Rowe:
out of a job i need to have a nice little nest egg set aside before that
Will_Button:
It's like someday they're going to figure me out. So I just
Jillian_Rowe:
I'm
Will_Button:
want
Jillian_Rowe:
sorry.
Will_Button:
enough cash parked when that happens. So I can go buy my trailer house in Montana.
Jillian_Rowe:
Wow, that's already the plan. That sounds like such a great plan though. I would be all over that.
Will_Button:
Yeah, and you can still get internet access too. I saw yesterday that Starlink is taking orders for the flat-topped mobile satellite internet. So you can just bolt this flat satellite dish on top of your RV and get internet access as you're driving down the road.
Jillian_Rowe:
See? That's, I'm telling you, I have another 10 years before, you know, I'll be, like, my kids will have launched, and that's the plan. Once they're all
Will_Button:
I'm just...
Jillian_Rowe:
grown, that's gonna be it.
Will_Button:
I'm just going to get a great big cowboy hat and mount it on top of that.
Jillian_Rowe:
There you go, that'll be fun.
Will_Button:
Yeah, yeah. But good luck on being free after the kids are gone. That was my plan too. Like, yeah, we're going to travel. We're going to go and do all this stuff. And then turned around, and there's both dogs staring at us like, do we get to go too? It's like, ah, damn
Jillian_Rowe:
Oh,
Will_Button:
it.
Jillian_Rowe:
yeah, then you can't have like the puppy eyes staring at you and
Will_Button:
Yeah,
Jillian_Rowe:
being sad or anything like that.
Will_Button:
yeah, so that crushed those plans because they were older at the time and yeah, just not really suitable for travel.
Jillian_Rowe:
Couldn't do van life with dogs? All the YouTubers who do the van life thing have dogs.
Will_Button:
Yeah, man, I don't know that I'm into that. Because vans are pretty small. And you get a wife and two dogs in there. And that's a little cramped for every single day.
Jillian_Rowe:
I would be okay with that. My first apartment was like not much bigger than one of those, you know, from like one of the bands that they show on band life and that thing was great. I could clean that from top to bottom, like deep clean, like you know, in like the grout and stuff like that in like two hours. Whole place, done. Done. could live out of his suitcase, which is fantastic. Now, no, there's like a whole house here. There's a lot of floors. It's a lot of floors.
Will_Button:
So one of the other things I thought of on this, we kind of touched on it as far as minimizing the blast radius. But I think it's worth elaborating on when we talked about minimizing the blast radius, minimizing what access they had. But also, for analytics type providers, you know, analyzing what you're actually sending them. You know, how much data are you sending them, and do they actually need every piece of data that you're sending them, or could you send them a few less fields that would make it potentially less identifiable, or at least out of context for someone who were to gain access to that.
Jillian_Rowe:
That's a good idea. Yeah, if you can like do anything to, I don't know, not exactly randomize, but make it look more random on their side anyway, so that there's less of a pattern to the data in case they are doing any kind of data mining, that would be a very good choice.
Will_Button:
Yeah. And then there's always the default stuff of encryption. Encrypt everything everywhere, in transit, at rest.
Jillian_Rowe:
But everybody has the keys to unencrypt it. Like, I mean, I get it. It's better than not encrypting it. But to me, it's still the same thing. At some point, you have to have the key to unencrypt the data so that it can look at it. And what are you doing about those keys? And I like the idea of the sort of one-time credentials that expire. That's a good one.
Will_Button:
Yeah, no, that's a fine line, right? And a really good point, like if it's encrypted, everyone who needs access to that data is going to have to have some way of reading the unencrypted version. So that means you've got keys to manage. Where are those keys at?
Jillian_Rowe:
Yeah,
Will_Button:
And then
Jillian_Rowe:
and
Will_Button:
that
Jillian_Rowe:
then if
Will_Button:
falls
Jillian_Rowe:
you're dealing
Will_Button:
into
Jillian_Rowe:
with
Will_Button:
a bigger.
Jillian_Rowe:
data scientists, you just know that somebody's just got that completely unencrypted. There's like multiple Excel copies just sitting on people's desktops. It's there somewhere.
Will_Button:
Right? right next to the sticky note with the password on the keyboard.
Jillian_Rowe:
Exactly. Exactly.
Will_Button:
Yeah.
Jillian_Rowe:
You know, this is how to unencrypt all the data and just get, you know, like a database dump of it immediately without having to do anything else.
Will_Button:
Yeah, I don't know. So those are my initial thoughts on it. That's some of the ways you can address it. I'm sure there's more ways to address it that I haven't thought of. But it just seems like a topic that I feel like we should be talking about more that we're not actually talking about and just generally having the conversation of, OK, you have a SAS, and that's great. And I'm just as excited as you are when things are working well, but when they're not, we need an agreed upon plan of how we move forward. And the current plan is you say you're really, really sorry and move on.
Jillian_Rowe:
And also, I mean, discussing whose liability is it too, because, you know, so let's, let's just say that you're working with a company and you're an employee or a consultant. Um, you know, is it, are you a part of the liability chain there, or is it just the, the kind of upper company? I've always kind of gotten around that because if there's any sort of software licensing. I just always make clients license it themselves. Like I won't do any of that on their behalf. And that's part of why is because if something like that does happen, I didn't have anything to do with any of that process. And so I'm hoping they can't come after me. I don't know like legally, just how, this is not legal advice, you guys. I don't know how much legally that would hold up. Like, you know, get your own lawyers, do your own contracts. But that's always what I've kind of hoped for is that that would work out. So I would say, you know, definitely have think on that. I do think it's really important, even career wise, to kind of keep an ear out, keep an eye out, and see what sort of things are happening. I did have, I don't know how much I want to get into it, but I have been in like a couple situations where I was just kind of looking around and I was like, this is illegal, this is definitely illegal, I should find another job because maybe it's not going to come up today, maybe it's not going to come up tomorrow, but it is a matter of time. So yeah, if you're in like a job situation or anything where these things are happening, my vote is find a new job, especially if you're not in a situation where you can do anything to change it, but there's still a very good chance that you would be given the responsibility or the liability, I suppose, for said decisions. So definitely, yeah, I mean, definitely keep that in mind because I do know there are, I mean, I have sort of either been a part of on the on the periphery or heard of cases where, you know, there's just somebody just just doing their job, right, like just coding things doing their job. Here's this, you know, this other software, this third party. And then it turned out that that other software was like actually mining data or doing this or doing that. And where is the liability in that situation, especially if you haven't already defined it beforehand, you know, beforehand very clearly. And yeah, every- when something like that does happen, I do- I know for a fact people always want to pass off the blame, the person, you know, if you're in research, the principal investigator is almost never going to get that blame for that scenario, it's gonna- it's going to be staff that are going to get the axe, so just, you know, always watch out for that sort of thing.
Will_Button:
Oh, absolutely. I remember one company I worked for quite a few years ago. I'll be vague on that so it can't be pinpointed. But there were two things that just served as big red flags. One was doing the data analysis to show the net impact our company was having in this particular environment. we had a very liberal use of the term outlier in our data. Where basically an outlier was any piece of data that didn't match up with the thing that we said we were doing, just got tossed as an outlier. I was like, well, that's kind of weird. And then the other part of that was I worked for him for about, well, I worked for him for a short period of time and went through three CFOs during that time. And when the third one came in, I said, you know, I'm just going to go see how this dude's doing. And got to know him. And after a short period of time, it became clear that they were going to be moving on as well. I'm like, dude, you got to level with me. What's going on here? And that individual revealed some things that were. They were not quite what I would have expected them to be too. So yeah, it was the case that you were just illustrating. There was nothing I could do about it in the role that I was in. So I just took that as strong leading indicators that I should probably move on as well.
Jillian_Rowe:
That would be another big one. If like the C-suite, there's an awful lot of turnover, something is happening. Something is happening and you probably also want to brush up your resume and get on out there. I had one contract, cause I know sitting here spilling all the dark secrets of my industry. But so like I said, for HIPAA compliant data, for research data, if you lose data, it is a really big deal. Like it's a very big deal. Your funding is very likely to be cut off. even if you don't exactly get fired, you don't get funding, which kind of amounts to the same thing, because like, academia and research is very weird about what exactly they can do to fire you. So they don't fire people, they just sort of let the money run out and or, you know, kind of make everybody so miserable that they leave themselves.
Will_Button:
Yeah
Jillian_Rowe:
And I remember kind of early on in my consulting career, getting a pretty good offer for, you know, for a contract. And it just went so like weird, so quick where... I was getting all of these very strongly worded emails about, we expect the results for this, this, and this. And I'm like, what results? You haven't given me the data yet. I don't have the data. What are you guys talking about? And it became very, very apparent early on that they had lost the data. And they were trying to find somebody to pin it on because that does occasionally
Will_Button:
Hahaha
Jillian_Rowe:
happen. But luckily, it was still early enough that I hadn't actually gotten access to any of their internal systems. So I very quickly cut bait. And I just ran screaming. my husband
Will_Button:
Hehehehe
Jillian_Rowe:
was like, why are we so paranoid? I was like, I've seen some things. I've seen too many things.
Will_Button:
I hate that!
Jillian_Rowe:
I'm outta here, guys.
Will_Button:
It's, yeah, it's like borderline on like the old hippie from the 60s, like, I've seen things, man. You're not going
Jillian_Rowe:
I'm
Will_Button:
to believe
Jillian_Rowe:
sorry.
Will_Button:
what I've seen. Like, no, I'm serious, dude.
Jillian_Rowe:
Research is wild. There's some very, very out there things that happen.
Will_Button:
Right? Cool. So I think those are my thoughts on SaaS providers. For our listeners, if you all have different ideas, I would be super curious to hear what you do to address this. Or if you think I'm just like often conspiracy theory land and that this is not an issue at all, I'd be really curious to hear that as well.
Jillian_Rowe:
Yeah, so I think to summarize, I mean, we said try to minimize the blast radius, rotate your keys, rotate your secrets if there's, you know, some kind of auto expiring credential sort of scenario with tokens or whatever is being used these days, go with that. If you can self-host it, I would always go for self-hosting overdoing the SaaS provider. Some, I mean, a good amount of SaaS providers do allow you to self-host. It does tend to cost more. But that's probably why it costs more. So just keep that in mind. And then the other one is, you know, always, always look out for you. And just if you think that something very strange is going on, it probably is, and you should run. You should run screaming like I do. There you go. Yep.
Will_Button:
Yeah, and if you have a legal team, know that they are looking out for you too as well. But
Jillian_Rowe:
No
Will_Button:
it may
Jillian_Rowe:
they're
Will_Button:
be an
Jillian_Rowe:
not.
Will_Button:
education
Jillian_Rowe:
That's like
Will_Button:
process.
Jillian_Rowe:
saying HR is looking out for you. No they're not.
Will_Button:
Well, you meaning you the company you work for not you personally.
Jillian_Rowe:
Yeah, those are two different views to me,
Will_Button:
Yeah,
Jillian_Rowe:
but alright,
Will_Button:
yeah, for
Jillian_Rowe:
okay.
Will_Button:
sure. For sure, one was a very liberal use of the word you. But my point being that they are there to protect the assets of the company. But there may be an education process required for them to effectively do so. So always good to have a chat with those guys.
Jillian_Rowe:
Definitely. Yeah, I always go talk to people like in other departments and other, you know, other like walks of life. I suppose you will definitely pick up on some very interesting little tidbits that you might not have known otherwise that will give you a fuller picture of exactly what is happening.
Will_Button:
Yeah, that was one of the things I liked about doing IT support in an office is you could wander into any office anywhere and either had a reason to be there or pretend to have a reason to be there and could just get to know different people and get different perspectives on the company and, you know, find out like what the rest of the story is.
Jillian_Rowe:
Yeah, I do kind of miss that about being in an office, like in an office environment, is getting to go talk to different kinds of people. Cause now for the most part, I talk to, well, I talk to researchers, I guess, and startup founders and other tech people like me. And I kind of miss talking to all the other people. I miss talking to like the students and just going and hanging out with the people in the lab and they show me like their lab equipment, which is another thing then you get like a really good idea of. Oh, this data is not anywhere near as pristine as I might have thought, because I am looking at this lab equipment and somebody definitely ate lunch in here yesterday.
Will_Button:
Right?
Jillian_Rowe:
So there's like all these kind of just different insights into data and what's happening and sort of like things that happen outside of the computer, which I think is always very important to get.
Will_Button:
Yeah, for sure. It's definitely a trip to Oz and pulling back the green curtain.
Jillian_Rowe:
It is, it is.
Will_Button:
Cool. Well, let's do some picks. Do you have a pick for this week?
Jillian_Rowe:
No, I think I need a minute. Can you go first? I did not come prepared. Do you have anything?
Will_Button:
I can yeah I've got one so I'm picking the Lubani dog paw pad protector it's Lubani L-O-O-B-A-N-I and it's basically a little sticker that's got traction on one side and I got them for my dog's feet because she's getting really really old and we have tile floors everywhere in the house. And her little paws are slippery. So as she's getting older, having hip problems, it's harder for her to get up and lay down. And when we saw the vet a few months back, she suggested these things. And so it's just a sticker that looks like your dog's paw pad. And you peel it off and stick it on the bottom of their foot. And it's got little grippy things on the bottom. And so now she can go walking up and down a tile. And she can. lay down and get up and she still struggles to get up but at least her little feet aren't sliding out from underneath her while she's doing it. So if you don't have an older dog or if you have a younger dog just keep these in mind because there'll be a time when they come in really really handy just to help them be a little more comfortable in their old age. So that's my pick for the week.
Jillian_Rowe:
Oh, that's a nice pick. All right, well, I guess I might've picked it before, actually, but I went like all in on having a sort of ergonomic working station set up because I work from home, which is for the most part really, really good, but like Will and I were just talking about, I don't like organically get up and talk to people, which means that I could potentially be sitting for a very long time, especially if it's quiet and I get like really kind of into a problem that I'm trying to solve. So recently I got a standing desk, just like one from IKEA. It's not especially fancy, but it's really, really great. And I got one of those anti-fatigue mats and I got a gamer chair, which I think is gonna be my anti-pick because I hate it. I thought that it would be great. I thought
Will_Button:
Hey.
Jillian_Rowe:
that it would be like, the gamers are sitting all day. They must know what's up, but I hate it. Anyways, but the standing desk is great.
Will_Button:
Yeah
Jillian_Rowe:
It's electric. It has this like up and down button. So that's been really good for me in terms of being able to like. I don't know, like stand up, sit down, move around, do like TikTok dances while things are building. That's been my new hobby. I've been like all over Bollywood TikTok. I don't know why it's just, it's where
Will_Button:
Hahaha
Jillian_Rowe:
I wound up. And so that's been great. But if you're a tech person and you're like over the age of 30 and you're like, oh geez, I can't just sit down at a desk anymore. This is all very bad scenes all around. I think getting a standing desk has really, really helped me. So I don't know. That's my pick for the week.
Will_Button:
Right on. Yeah, I agree. The standing desk is definitely worth the money for it. And you can get one for any budget.
Jillian_Rowe:
Yeah,
Will_Button:
But it's
Jillian_Rowe:
mine
Will_Button:
definitely
Jillian_Rowe:
was like 200
Will_Button:
a good upgrade.
Jillian_Rowe:
bucks or something. It's, and it's, it is the electric one. It's the one that goes up and down.
Will_Button:
Yeah, yeah, I think whenever this one starts to wear out, I'm just going to scrap the pieces and make like a cool skateboard or like a low rider skateboard or something out of the motors. Just because. Just because.
Jillian_Rowe:
That does sound like a fun project.
Will_Button:
Yeah that's what I need I need a project out in the garage especially now because the weather is turning nicer here we made it through the summer and it's actually you can actually go out in the garage and it's not a hundred and twenty degrees out there so yeah.
Jillian_Rowe:
That's right, you sympathize with my extreme heat situation. I always forget that, but Arizona's almost the same temperature as Doha's. Yeah?
Will_Button:
Yeah, yeah, a month or so ago, I was really close to you. I was over in Dubai.
Jillian_Rowe:
Oh really? What were you
Will_Button:
Yeah.
Jillian_Rowe:
doing in Dubai?
Will_Button:
We had a conference there, an offsite engineering conference, where the engineering team got together, and they chose to buy.
Jillian_Rowe:
I worked in Abu Dhabi for a couple of years and I used to, so I was like only an hour from Dubai and we used to go there a lot on the weekends and go to the different parks. Did you go to
Will_Button:
Yeah.
Jillian_Rowe:
Legoland?
Will_Button:
I didn't make it to Legoland, no.
Jillian_Rowe:
No, Legoland is the most fun part of Dubai as far as I'm concerned.
Will_Button:
There were so many things to choose from there to do. And I was there for a week. And yeah, like if you're into that kind of stuff, the number of options were just mind blowing.
Jillian_Rowe:
They do have too many things than you could possibly do in a week.
Will_Button:
Yeah, like scuba diving in the aquarium that's inside the shopping mall with the sharks. I never thought all those words would go into one sentence.
Jillian_Rowe:
Yeah, I don't know, they're good at thinking up that kind of stuff. You gotta hand it to them. They like all those things, and they like them all to be in shopping malls.
Will_Button:
Yeah. Well, they don't have many other options, you know, like if they don't come up and build something creative, aside from that, it's just a desert. There's really no other reason to be there.
Jillian_Rowe:
No, it is kind of cool going out to the dunes, or at least it was for me because I grew up in sort of New England which is very forested and whatnot, so going out and seeing like the completely, you know, white, or if you would go up towards Oman, they're more like red sand dunes, and then in some places you can just see like sand dunes out to the sea and it looks like some kind of crazy alien landscape, which I think is kind of fun.
Will_Button:
Yeah, for sure. Cool. All right. Well, it's good to see you. Thanks for listening, everyone. We'll see you all next week.
Jillian_Rowe:
Bye.