Secrets Management Best Practices and Tools - RUBY 655
Charles and Valentino are joined by special guest Brian Vallelunga, CEO and co-founder of Doppler, a leading secrets management platform often described as "GitHub for secrets."
Special Guests:
Brian Vallelunga
Show Notes
Charles and Valentino are joined by special guest Brian Vallelunga, CEO and co-founder of Doppler, a leading secrets management platform often described as "GitHub for secrets."
Dive into an engaging conversation about best practices for managing sensitive information, such as API keys and encryption keys, and treating all environment-configured settings as secrets. Brian shares insights on using tools like AWS Secrets Manager, Docker, and Doppler’s seamless integration with popular development workflows, ensuring robust access control and audit logging.
They discuss about the severe consequences of data breaches, compelling real-world scams, and the human cost of leaked data. Learn how Doppler helps developers avoid these risks while enhancing productivity. They also explore the intersection between developer efficiency and security, and emerging trends in secrets management like passwordless authentication.
Tune in for valuable tips, personal stories, and a peek at the future of secrets management and developer security. Let’s get started!
Socials
Transcript
Charles Max Wood [00:00:04]:
Hey, folks. Welcome back to another episode of the Ruby Rogues podcast. This week on our panel, we have Valentino Stoll.
Valentino Stoll [00:00:11]:
Hey now.
Charles Max Wood [00:00:12]:
I'm Charles Max Wood from Top End Devs. And this week, we're here with Brian Valalunga. Now you are the CEO and cofounder of Doppler, and, that's an app that helps manage secrets. You probably have a better pitch than that, so I'll let you explain what you do.
Brian Valalunga [00:00:33]:
Hey, y'all. It's great to be here. Yeah. Doppler is a secrets manager designed to be your single source of truth for secrets across all your projects, environments, team members, and infrastructure. So you can kind of think about it like as get up for secrets.
Charles Max Wood [00:00:48]:
Right. And I I think I don't know. In in the Ruby community, at least where I deal with people and, especially in Rails, Right? We we kind of understand what the secrets are. It's your your passwords, maybe your, you know, you have your Rails master key, which gives you access to the secrets in Rails, but you also have, like, your what is that? It's a token for your sessions. I can't remember what it's called. Anyway so you you need that secret key and, yeah, people can compromise your security if you don't have that, secured. Are are there things besides, like, passwords and API keys that you consider secrets?
Brian Valalunga [00:01:28]:
Yeah. Outside of, like, API keys, like a Stripe, token Yeah. Like, database URLs, encryption keys are typically the ones that we see the most. Okay. I also just, like, have, like, a general thought process around this of anything that's configured by the environment should be treated by, as a secret, so it should be treated like the most sensitive thing. So, we kinda tell all of or we recommend to, our audience that, you should kinda treat your environment variables and your secrets all the same. Right? Your port and feature flags should be treated as securely as your secrets. That way developers don't need to make the choice of is this secure or is this not.
Brian Valalunga [00:02:05]:
They're always just doing the secure thing by default. And then the trick is just make that that whole workflow and tooling as developer friendly as possible so they wanted to go down that path.
Charles Max Wood [00:02:15]:
Right. So I'm just gonna kinda cover the basics of how I do this, and then maybe you can tell us if there are other ways that you're seeing people do it. But, typically, these kinds of things where I see them, and I'm sure Valentino's in kind of the same boat, is they either show up in environment variables, or they show up in, in the rails secrets or and and, effectively, what it is is it's a, it's an encrypted file. And so then the encryption key is usually put into an environment variable or, you know, passed into the system that you're running it on on the other end in some way. And lately, I've been doing my deployments with Kamal, which uses Docker. And so, it kinda sets that all up when it sets up the container. But I've always kinda wondered because you can ask the system if you can get into it what the environment variables are, so I don't even know if that's the best practice anymore.
Brian Valalunga [00:03:18]:
Yeah. It's a, that's pretty much the the the two paths that we see, either some encrypted file somewhere or it's just directly, available in the environments. I'm a pretty big fan of, if you're going to do the environment path, or inject it through the environment, immediately do a cleanup of the environment before all your other dependencies get get loaded in. So, like, basically, load those, secrets out of the environment, then clean them up from the environment, then load everything else. That way, you can't have, like, a rogue dependency go and capture all your secrets and then send it somewhere. The other path of, like, using a file, that's encrypted is also a really good path too. So I think both are pretty good.
Charles Max Wood [00:04:00]:
What are you doing, Valentino? Is it kind of the same? Or
Valentino Stoll [00:04:05]:
we use a lot of AWS stuff, to handle a lot of the
Brian Valalunga [00:04:11]:
lot lot
Charles Max Wood [00:04:12]:
of the secrets to management.
Valentino Stoll [00:04:13]:
We have a secrets to management. In there. Yeah. And I think that it may even be isolated, from the main environments too and, handled that way in a isolated environment. We we use Docker as well, so it could be, like, an isolated container that then feeds back into the other container.
Brian Valalunga [00:04:32]:
So are you, I'm guessing you have your secret stored on, like, AWS secrets manager, and then those secrets are then pushed into the container as environment.
Valentino Stoll [00:04:40]:
I think so. Something like that. It may even be one step further where, it's in AWS and then at, like, run run time, it's like the secrets are loaded into the app's memory, for the particular use case. I don't really know all of the details, to be honest. I'm not on the infrastructure team.
Charles Max Wood [00:05:00]:
Every once
Valentino Stoll [00:05:01]:
in a while, we have to rotate keys, and, that's definitely, you know, an easy process for us. But, yeah, we have some command line that does it. It's not just like changing the environment variable.
Brian Valalunga [00:05:16]:
Sounds like you guys are pretty well set up. I mean, that's that's, like, the dream is to to make secret expansion super easy and simple, but also very secure at the same time. Yep.
Valentino Stoll [00:05:26]:
Yeah. That that makes me curious because, like, I think at one time we were using, forget what HashRockets product is. We ended up moving away from that. Yeah. I don't know. We were we were using some vendor for doing the secrets management. But, yeah, having a command line or some kind of central place, a command line doesn't even need to be your, you know, your UX of choice, but just having the centralized place where all of that access points happen, it definitely helps, like, isolate and track, to be honest, who's touching what and changing it. I I don't know if Doppler has those kind of features as far as, like, tracking changes and, getting the full logs and things like that, but, that's definitely super valuable.
Brian Valalunga [00:06:20]:
Yeah. So I think, DOP where kinda comes from a first principle standpoint about, like, what is a good secret manager look like? And for us, we have a couple of things that we deeply care about. First is that all your secrets are centralized in one place, across, your projects and environments. And, once they're all in one place, we can set up a pretty robust access controls either by users or by groups that are policy driven. And then from there, you get a complete audit story. So you know every time someone has read a secret, written a secret, changed some settings, on every device with their IP address, you get a a whole, a picture about everything that's going around or going on around your secrets. And then the other key part is that it's deeply connected to their workflows and their infrastructure. So on the workflow side and, like, local developments, we want you to be able to edit your secrets side by side with your code.
Brian Valalunga [00:07:08]:
So you have a Versus code extension. And so, those secrets are actually kinda look like a a YAML file, but they're actually not. It's like a we use something called a virtual, file system. So it behaves like a file, but it's actually not. It's connecting with on the back end. So it's super secure. And then, on the orchestration side for production is we will write into AWS secrets manager and Kubernetes or wherever it may be so that your infrastructure immediately gets those secrets when they change, and restart your, your deployment so you don't have any downtime.
Charles Max Wood [00:07:38]:
Yeah. I remember using a system like this way back in the day. I think you had to log in to their interface or use their command line interface in order to edit your secrets. And I can't for the life of me remember what it was called, but, I played with it and it was cool. It was just there there were steps to setting it up, and I'm lazy, and so I just went back to what I was doing.
Brian Valalunga [00:08:00]:
I am for that matter. You you gotta make it at like, we have this theme at Doppler, and it's it's quite simple. It's make vegetables taste like candy. Vegetables being the security, candy being the developer productivity. If you do not make it if the tool does not make the developers' lives better like, you can't just say, hey. Use this tool because it's more secure. Like, that doesn't fly because developers have been they care about moving fast and shipping product. And so you have to aid them in that.
Brian Valalunga [00:08:22]:
You have like, in our case, we try to get them 2 hours a day, or sorry, 2 hours a week back in productivity. And so if you if you do that, if you give them good candy, then you'll get the vegetables too.
Charles Max Wood [00:08:33]:
Yeah. That makes sense. I I kinda wanna dive in a little bit more on just why we protect the secrets, because I I think we all kind of intellectually know.
Valentino Stoll [00:08:46]:
Mhmm.
Charles Max Wood [00:08:47]:
But I don't know. When I get into stuff and I start, you know, dealing with the secrets, it's like, oh, somebody got access to my, like, my Stripe tokens. I mean, that that could be really, really bad. Yeah.
Brian Valalunga [00:08:58]:
It's really bad. So I like to kinda bucket the risk in in or or sorry. Yeah. Bucket the risk into 2 buckets. The first is, like, the risk of the company, and the second is the risk to people. And I think the company side is, like, pretty well understood. Right? You're gonna have a pretty if you have a data breach, you're gonna have a pretty big brand hits and trust hits. You're gonna probably have some unexpected spend because now you're investing, like, legal PR insurance.
Brian Valalunga [00:09:24]:
You may get sued as well because if you're not managing your secrets and the secrets are things that get leaked that leads to a data breach, that's just straight of negligence, and you your and the customers are right are in their rights to sue. But I think the thing that's, most of the time not talked about is, like, the cost to real people. At the end day, all these companies that we use, we're trusting them to store our data and keep our private data private. Imagine if, like, every YouTube video that you ever saw got out or, every ride you ever took on an Uber, every transaction you ever made on a credit card, all got out. And I actually have a real personal story that of, like, how this has impacted me. Sadly, I've been part of a couple data breaches now from some services I've used, and, that came back and really bit me in a in an interesting way. I was so I just moved to Austin, Texas couple months ago and, from California, and I convinced my mom to fly out with me and, see Austin for itself when we were at this barbecue joint. And while we're eating food, I get a call from, the Texas Customs and Borders, and they say, hey.
Brian Valalunga [00:10:24]:
We found a package in your name that has, illegal money and drugs in it, and we're, federally investigating you now. And this is, like, the phone call for me where I was like, oh, shit. My life has just ended. Like, from the highest
Valentino Stoll [00:10:36]:
to the
Brian Valalunga [00:10:36]:
most really, really quickly. And I was really scared. And but, like, also kinda like this is odd. Like, never bought anything from Mexico before. So, like, okay. And the back of my head, I was like, this is a scam probably. But then they started rattling off all this information about me that they knew, all the places I've lived in the past, locations I had been to, like, just so like, a wealth of of information about me that I was like, no one could possibly know this but the government. And so, like, I trusted them.
Brian Valalunga [00:11:09]:
I was like, okay. I'm actually being investigated here. And, obviously, we get lawyers on the call as well. Like, this becomes a really scary ordeal for all of us. And it wasn't until about, like, an hour in that our lawyers picked up that it wasn't that and but during that hour, we also gave them a whole lot more information about me, right, because we thought we'd be investigating. They were asking questions. And so, like, this is and, like, again, I'm a CEO of a of a cybersecurity company. I'm trained on this.
Brian Valalunga [00:11:37]:
Our employees are trained on this. Our legal team is trained on this, and they still got us for an hour. Right? I can imagine everyday Joe who's not trained, could really get, attacked here. And, like, I was lucky that all they did was get a little bit more information about us, But they were able to get that information because of other information that got breached. Right? That that bought them the credibility, to get new information. But imagine, like, my mom, my sister, or anyone else like that who's not gone through any of that training does not have legal counsel immediately, readily available to them that can jump on the call for this. Oh my gosh. I cannot imagine all the information that have been given up and all of a sudden their bank accounts are drained or there's or or their credit score is completely ruined or whatever it may be.
Brian Valalunga [00:12:15]:
It can get really terrifying really fast. And that's why I come back to, like, whenever I see, like, a data breach and it's, like, x millions of people's data was just got out. It's like, wow. Like, scammers are now, like, parading in the streets going like, oh my gosh. I just got a ton of data that's gonna get me so much more data that can eventually get me to, like, the jackpot of all this money or or or whatever they're trying to go after. And so there's real people's lives, including mine, that have been impacted by this, and that is something that's, like, often not talked about at all. And so when you're a developer and you're like, oh, should I put this in a new default or not, or should I secure my secrets? Just one token. It's it's a string.
Brian Valalunga [00:12:48]:
It's like, well, that string may impact a 1000000 people's lives someday, so to to treat right.
Charles Max Wood [00:12:54]:
Right. Well, it it reminds me your story a little bit of, have you read Ghost in the Wires? I I can't is it Kevin Mitnick?
Brian Valalunga [00:13:02]:
Yeah. Kevin Mitnick. Yeah. I I've watched the movie about it, but I haven't read the book.
Charles Max Wood [00:13:06]:
And see, I didn't know there was a movie, but I read the book. Right. But, yeah, he talks about how effectively yeah. He he gets just a little bit of information about the next person in the chain. Right? And so he doesn't start with the the analyst or whoever. Right? He starts with, you know, the the lowdown person on the totem pole and then, you know, gets enough information to know who the next person is and then works his way up. And so it's yeah. It was it was a fascinating read just from the the sense of yeah.
Charles Max Wood [00:13:44]:
You know? Just just anyway. So yeah. So I can definitely see that. I I think the other thing that I worry about is, like, if I have a key to, I don't know, like, chat GPT or something like that, somebody could run up my bill Yep. Or, you know, somebody gets in my hosting account and, you know, all of a sudden I've got you know, they they breach my other stuff that way. And, anyway
Brian Valalunga [00:14:13]:
No. I think you're totally right. I mean, there's 2 common things that we see all the time. One is, like, some source code goes public or some email gets leaked and that has a secret in it to, like, an AWS account, and you'll see 1 of 2 things happen immediately. The first is their build just, like, skyrockets in seconds because now they're using, AWS for some other bot attack or for mining crypto or whatever it may be. That happens very, very commonly. And it's surprisingly, like, how fast that that, like, that those secrets get found. Like, I think there was, like, an average stat that it's, like, within 3 to 5 seconds of, like, a repo going public.
Brian Valalunga [00:14:45]:
If it has a secret in it, even in the history, it will get found and and and start getting exploited. So it's, like, very quick. Like, it's it's it's insanely quick. And then the other side of it is now that they have access to AWS, they can start scanning all your resources. Okay. What, what execution environments are set up? What databases are set up? Can I get access to those database? How privileged is this credential? And you start having this ripple effect where it's like, oh my gosh. Now that especially if it has, like, like, strong super admin permissions, it can start creating other credentials. And now you have this, like, cat and mouse game of, like, how do I get out of the system? Could it lock me out of the system? Did I, are there backdoors it's placing in? So even if we remove it, it's still actually there.
Brian Valalunga [00:15:25]:
Like, it gets scared really quickly. Once they get into the system, it's very, very hard to get them out.
Valentino Stoll [00:15:29]:
Yeah. That's always makes me wonder, like because there's two parts of this. Right? Like, we have the secrets and then there's, like, the whole encryption aspect of it, which is its own separate piece of the discussion.
Charles Max Wood [00:15:42]:
Wait. When you say the encryption aspect of it, is that, like, how you're securing your secrets, or is that, like, encryption keys that
Valentino Stoll [00:15:47]:
you're I mean, ultimately, what you're using a lot of the secrets for, I would imagine, is to encrypt something, or or even to, yeah, I mean, to encrypt stuff.
Charles Max Wood [00:15:58]:
Yeah. You're either hiding stuff or accessing stuff is usually what you're doing.
Brian Valalunga [00:16:02]:
Based on the data that we have from Doppler, we can't see customer secrets at all, but we can, get an auto report of the names of the secrets they use, if they give us that permission. And, from the data we have, it's, like, 90% of the of the sensitive secrets that we put in Doppler and customer's inevitability say, this is sensitive or this is not, is actually, like, access token. So it's not encryption tokens. So it's, like, if you get that Stripe key, you have it all. Like, you have every customer's credit card and bank account in and there. There's no extra layer of protection.
Valentino Stoll [00:16:37]:
Yeah. That's all I I always wanna one of the the biggest things that, I always hope for when I'm, like, getting a new vendor, like, access token or or whatever it may be, is that they, like, have some kind of, like, you know, white list stability with, like, either their domain, like, a domain key or some kind of identifier that can be, like, you know, securely matched when the handshake is made, using their service from whatever server that you end up using. Right. And it's in my experience, like, that it's very, honestly, very rare Yes. That that actually happens. And so I I under I that makes a lot of sense, you saying that your customers, like, mostly store, like, access tokens because most vendors don't give you that, like, white list ability and, like, handshake process, which is, like, very important. Right? I think causes a lot of the breaches when it's not there.
Brian Valalunga [00:17:38]:
Agreed. I I do think that the world is changing. It's just gonna be very, very slow. So, like, you can see this in passwords today where we're going from passwords to pass keys. So anyone, familiar with pass keys, it's basically like, you can sign into a website with, like, your face or touch ID, and you never had to put in a password. It's a password passwordless world. And the way that's working is through cryptography. You have, like, a fingerprint that leads to a cryptographic signature that can be used to authenticate yourself.
Brian Valalunga [00:18:04]:
And that that thing that's happening in passwords is now starting to happen in secrets with identity based authentication. And so, like, the kind of picture that we can start to see happening very, very slowly in, like, Doppler's job is to, like, help move this along, help move the industry along is, to have, like, this bridge of connection. So, like, on one side, you have, like, your AWS, GCP, execution environment, and that has an identity already today. And then you can pair that with Doppler, and now you're authentic by the fact that you're in that e c two instance, sir, you are now automatically authenticated to Doppler. And then eventually, Doppler will have the connection of, now that you're authenticated with Doppler, you're automatically authenticated to Stripe. There's no more API key needed. We are all cryptographically, authorizing each other across this chain or this highway of, authentication. And that's kinda like the dream state that we're gonna be getting to over the next couple years.
Brian Valalunga [00:18:55]:
It's gonna take a lot of work of working with partners, but now we have to convince Stripe, Twilio, and so many other providers to enable this. And we're gonna try to make this easy as possible, but it's gonna be it's gonna be an uphill battle. But if we can do that, we're in a secretless world.
Valentino Stoll [00:19:07]:
Is there an open standard, that is following this process or no?
Brian Valalunga [00:19:12]:
There is a little bit of an open standards mostly between, like, AWS and x, like Doppler or whatever it may be, but there's no open standard on or even a standard altogether, on the, like, secrets manager to, third party service like Stripe or Twilio. So I think our goal is we're gonna probably try to create an open standard that both sides adhere to so that if you're anywhere in this chain, you can directly, you can just plug in very quickly. And I think we'll probably have to build SDKs for, like, every primary language, and Kubernetes and a bunch of other, infrastructure tooling as well, so just, like, works out of the box.
Charles Max Wood [00:19:49]:
So, I I mean, I I I find this all really fascinating. But I I guess my question is, what kinds of mistakes might I be making now? Or, you know, what what kinds of things should I be considering doing now to protect these secrets? Because, yeah, you know, what you're talking about sounds brilliant, but it it doesn't exist yet.
Brian Valalunga [00:20:11]:
Yeah. And I'm
Charles Max Wood [00:20:12]:
just I'm a little worried that, okay, we're gonna talk the rest of this time about futuristic stuff, and then I'm gonna be yeah. I'm gonna miss something.
Brian Valalunga [00:20:20]:
Alright. So I I have 4 questions that I think every person listening, should ask themselves, and it's, like, basically, a very simple sniff test to see if I have a problem. So, to highlight what the problem is, it's called secret sprawl, and that is basically all your secrets are scattered across a bunch of different places. And because of that, you have no way of controlling it or governing it, which is, like, one of the most important parts about protecting your secrets is being able to govern those secrets. And so the 4 questions are, can you confidently answer where are all my secrets? Right? Not just the ones that you know about in, in a secrets manager somewhere, but all the ones on developer's laptops, all the ones in Slack and email and, your code somewhere. Like, you gotta have a complete picture where all your secrets are, and you gotta be able to make that guarantee that you that you're not gonna miss 1 by not knowing about it. And I'll I'll talk about how to solve that in a second. The second is, do you know who has access to all of those secrets? And this gets really scary, especially if you don't have, like, a centralized system because, like, if I have a file, I can just share it to another person, and no one on the security team is gonna know about that.
Brian Valalunga [00:21:22]:
Mhmm. Can I control who has accessed those secrets when they were accessed, and by who? Right? So do I have access controls, and do I have an auto log associated with those access events? And then most importantly, when data breach does happen, can I stop it very, very quickly? When I say quickly, I mean, in the orders of seconds and minutes, not in the the order of days and and weeks or months. And so if you cannot answer those 4 questions confidently, like, to a point where your job would be on the line for it because it essentially is on the line for it, then you have a problem. And I can talk about in a sec what it looks like to have a solution there. I don't have to go into, like, Doppler's features, but I can talk about, like, the fundamental properties you'd wanna look for in any secrets manager, Doppler or, someone else.
Charles Max Wood [00:22:12]:
Yeah. I think that's important. It's funny because so I kinda have two levels of this, one where I'm working for a client and the other one where I'm just working on my own stuff. And on my own stuff, I'm fairly confident that I can answer these questions because I'm basically the only person accessing or using them. But then the question for me becomes, okay. Well, how do I how do I you know, let's say I bring in another developer. How do I make sure that I know the the answers to those questions as I bring them on board?
Brian Valalunga [00:22:46]:
Yep. And I think once your surface area grows of team members, infrastructure, devices, it gets pretty hard to to to answer those questions confidently, unless you have a dedicated system for it.
Charles Max Wood [00:23:01]:
Right. So why
Brian Valalunga [00:23:02]:
don't we be able to go into, like, what actually does a dedicated solution look like? The first thing is you can centralize all your secrets in it. So once you know where all your secrets are, you can confidently say, this is the place we're doing. We're not gonna store our secrets in code. We're not gonna send it over email or Slack or Microsoft Teams or whatever it may be. This is the one place you can get all your secrets. Now you have a complete landscape of your secrets. The next thing you should be able to do in this system is you should be able to, can set up access controls and auto logs around them. So you can say, you have this develop, full stack developers have access to the development environment for these sets of projects for the teams they're on, but the DevOps team has access to production.
Brian Valalunga [00:23:38]:
So you can start gaining access and you have a full audit picture of it, and, obviously, that audit picture then gets pushed into your auditing tools like Datadog, Sumo Logic, Amazon, whatever maybe. Mhmm. You should be able to have this infrastructure natively integrated with their developer workflows for local developments. That way developers aren't working around the system and then recreating that risk you're trying to avoid. And it should be natively integrated with your with your production and and staging infrastructure so that when it is time to roll out a secret, you don't have this race condition of the code arriving before the secrets do. The secrets have to arrive first if the code is gonna rely on the those secrets. And so then this kinda gives you this whole picture of I know that if I can see the secret in this project, in this environment, on the solution like the secrets manager, then it's guaranteed in my production infrastructure or it's guaranteed on my developers' laptops in a secure way. And that basically gives you the ability to remediate secrets pretty quickly.
Brian Valalunga [00:24:33]:
And so the last part of this puzzle is being able to rotate secrets. Rotation is the fancy way of saying, I'm gonna swap out the locks on the door. And you gotta do it in a in a way because if you just, like, immediately destroy the lock, in this case, you'll just bring yourself down in production. So you have to do it in a way that does not create any downtime, and it's usually requiring a 2 k system. So, like, in in a database, for example, you don't ever wanna not be able to connect to the database. So you'd have, 2 credentials to the database, and you swap them, and then you rotate 1, and then you swap again. And so you want a system that does all that. That should not be done by humans.
Brian Valalunga [00:25:08]:
That should be done by a a system that's running on a schedule where you have an immediate, like, I'm fucked button. I need to click this thing to get me out of a a data breach. And then, obviously, it orchestrates to the rest of your infrastructure. And so those are the things that you should be looking for in a in a in a developer for secrets manager.
Valentino Stoll [00:25:23]:
So I love this. You have this great, secrets for all page on Doppler, I linked to. And, it's really great. Kind of helps you quickly, assess your risk exposure based on your team size. You you go through kind of, like, a quick calculation. Oh, we have, like, this number of projects, estimate number of, secrets per project, then you have certain number of environments. Right? So it's a multiplier, and then you have, you know, a multiplier over the number of team members too. And so those quickly add up, and you make a great point.
Valentino Stoll [00:25:55]:
Like, you know, the malicious actor, they only needs they need 1, right, of the 1,000 that that adds up to. And so I'm I'm curious, like, you know, from the from the like, the whole point kind of is, like, risk assessment and remediation. Right? And, like, do you find that, focusing on the that risk aspect is just as important as, like, kind of the secret storage management mechanisms.
Brian Valalunga [00:26:25]:
Yeah. They go they go in hand in hand for sure. You need something that developers are gonna wanna use that is secure when they do use it, and you wanna think about that risk. Like, a good a good way that I kinda, like, talk about it when I'm talking internally with our employees is the game of offense and defense are very different games. The game of defense is build walls around your perimeter as high as possible, as strong as possible with no, no cracks in the foundation. Right? And then the goal of offense of an of an attacker of an hack of a hacker is to find one crack in the foundation and hit it as hard as freaking possible that bring brings down the entire wall. Right? They only need one weak link. And if seekers happen to be that weak link, well, then you're really giving them the keys to your digital kingdom.
Brian Valalunga [00:27:08]:
That's extra bad. I'd much rather be the weak link be something far outer in the perimeter like, I don't know, some authentication bug or something like that that that impacts, like, 1 or 2 users, not your entire users, user base. And so, yeah, managing risk is incredibly more near. That's that's the end goal that you're trying to accomplish is, like, make sure you can stay up and orchestrate, like, the get get all the DevOps and platform tooling you're looking for, and then also manage the risk associated with that.
Valentino Stoll [00:27:38]:
So I'm curious, like, if you have any tools or, like, what would you do to kind of, like, visualize the, like, you know, organizational structure in combination? Like, it's more than just, like, secrets management, it seems. Right? Like, how do you how do you personally visualize, like, the security aspects surrounding the secrets and who accesses them and and things like that, to understand kind of what is happening and where the access points are?
Brian Valalunga [00:28:08]:
Yeah. Great question. Truly great question. We've tried really hard to build a model that maps very closely to what you'd have, with code, so with GitHub, or GitLab. So we have projects just like you have repos in GitHub and GitLab, and we have environments which are very similar to branches. And then we have user groups that map in, with access control policies. And then we have integrations that connect to those environments. And so you can have these, like, this bare bones hierarchical structure of projects and environments.
Brian Valalunga [00:28:41]:
From there, you can assign users to them and you can assign infrastructure that connects with them. And it's very easy within Doppler to kinda see all that connect together in an extremely visual way. We even have, to to show kinda how far we've taken it, when you go into a project, you actually have a kanban board, style, you want user experience, that shows all your different environments from development to staging to production, everything in between.
Valentino Stoll [00:29:05]:
So what inspired you to start Dropletr? Like, where where does the story start? Like, I'm curious, like, where your background is and, you know, how how you're like, oh, we need a better, you know, secrets manager.
Brian Valalunga [00:29:17]:
Yeah. It had nothing to do with secrets from the started. I was working on, a crypto machine learning marketplace, all kinda all the buzzwords in one. One on the side. It was literally every buzzword you can find jammed together. And, man, that was a hard project. I worked on it for about 8 months straight, and it felt like pushing a massive boulder up a hill. Every foot forward, I'd slip 5 5, 6, 5 to 6 feet back from exhaustion.
Brian Valalunga [00:29:44]:
It was just hard. And at one point, I, like, really got to my breaking point. I was like, I don't know if I can keep working on this thing. And so I decided to take a trip to to Cancun, Mexico, and the whole goal was, like, don't think about this project. Actively do not. And, obviously, I failed at that miserably. I was thinking about it the entire time I was there. And what I realized is that this pro I was never gonna be able to get this project off the ground, but that doesn't mean that I had to stop there.
Brian Valalunga [00:30:10]:
I think, like, there's a lot of people that, like like, follow, like, sports players, and, like, they know all the stats of all these players. I'm very much the same way with founders. I'll I, like, I love learning about founder stories, and I think Stuart Butterfield from Slack is, like, the best feeling upwards I've seen. Creates a video game. Out of that video game the video game fails, born out of that was Flickr. Comes back a second time, so, like, I'm gonna create a video game again. Born out of that was, Slack, which we all use today. And so I was like, what can I learn from, from my experience with this crypto machine learning marketplace? And managing secrets was just one of them that was, like, super painful in all these weird interesting ways.
Brian Valalunga [00:30:49]:
It was weird from a developer productivity side because, like, I always felt like I was copying and pasting things in, But sometimes the code would get into production faster than the secret shorts I had a have an outage. I had a couple of developers that I was working with on this project, and it was, like, really painful to sync all these secrets with them. Like, I literally Slack them and email them the secrets, and it I like my heart would cringe every time I do that. Like, oh, that didn't feel right, but that's the only thing I have at my disposal to do. And there's actually one time where we we dropped the crypto part of the marketplace because it wasn't working. We just did straight up machine learning marketplace using Stripe. And I had the Stripe production key in staging and the staging 1 in prod, and it took us a month to figure out why we weren't doing any transactions. And it was because of that, because the Stripe, the Stripe testing key, rejects all live credit cards.
Brian Valalunga [00:31:33]:
It only uses test credit cards. And we didn't realize that for a long time. And so I was like, okay. This is a real problem. And so I come back from Cancun, and I go to this dinner that Stripe was hosting that had a bunch of founders and developers added. And I we were all going around the room talking about what we were thinking about, and I just asked her openly, am I a shitty developer? Is the world broken? I just can't tell anymore. Are you guys having this problem too? And, like, 60% of them were like, yes. This is a very big problem for us.
Brian Valalunga [00:32:01]:
And one woman in particular, like, literally comes running up to me right afterwards. I thought she was gonna trample me. And she says, I've had 3 outages this week. Hand me a solution by Sunday. And I was like, no. No. No. No.
Brian Valalunga [00:32:11]:
No. Like, I'm not even started yet, and this is a Wednesday. There's no way this that's happening. She goes, I don't give a fuck. Have it by Sunday. And I was like, okay. There are some interest here if people want it. But we didn't obviously have it by Sunday.
Brian Valalunga [00:32:23]:
That that never happened. But, what we did is we started doing a lot of research, and we found that individual developers are struggling because there wasn't tools made for developers. Medium sized companies are struggling building their own things, and large companies were buying Hashcore Vault, which really isn't meant to solve this problem and still struggling. So, like, everyone's struggling. And I was like, okay. We can build something that's developer first instead of instead of just purely security first. And we got our first version working in 3 weeks, and we got a set of customers right after that by just dragging them to Chipotle and saying, hey. I'll get you anything on the menu as long as I get a rant at you for about 2 hours.
Brian Valalunga [00:32:57]:
And that got us, our first set of customers and off to the races we were.
Charles Max Wood [00:33:02]:
Yeah. It's funny how, yeah, you wind up scratching an itch that's not how do I put it? It's not like the core thing you were trying to build Yeah. Anyways. Off.
Brian Valalunga [00:33:11]:
But turned out to be a good business.
Charles Max Wood [00:33:13]:
Yeah. It's it's interesting because I've been kind of in that same space where I've been looking for a solution that does a whole bunch of things that I want for the podcast, and I've just you know? So I've built little pieces of it here and there, and so I finally just started bolting it together so that I have the full solution. And, yeah, it's it's interesting how many how many times you have, yeah, things come out of that.
Brian Valalunga [00:33:37]:
Yeah. I would probably index and be like, okay. Are other people that I know that also do podcast recording asking for this tool for this tool? Do does it continuously get better and better the more I I mount on to this, like, hodgepodge of a product? And is that value going up, exponentially not linearly? And if the both those answers are are yes and you still feel like there's nothing in the market that does anything close, you have a product. The question is, is that products product 10 x better than whatever's in the market? Because if it isn't 10 x better or 10 x different, people there won't be enough of a differentiator to create a buying event, I found. Right.
Valentino Stoll [00:34:13]:
So I'm curious, like, what what are some common, like, preventable breaches that you've encountered, and, like, how can developers avoid them? Right?
Brian Valalunga [00:34:22]:
Yeah. There's a couple that really come to mind. I mean, let's look at
Charles Max Wood [00:34:27]:
He left like he's got some doozies of some stories.
Brian Valalunga [00:34:32]:
They're all public.
Valentino Stoll [00:34:33]:
Okay.
Brian Valalunga [00:34:34]:
I'll share one that, like, isn't public, but I won't say the name of the company, and then I'll share a couple that are public. So Twitch, Toyota, and Twitter all kinda had the exact same breach happen. And they had secrets in code. The code got leaked somehow either by them leaking their, them opening up the repos themselves or by hackers being able to get access to the GitHub account from, an employee that got compromised. So, like, someone, got called and said, hey. I'm your boss, and, give me access to GitHub, and then the hacker got access to GitHub. And then from get from GitHub, they saw the code, and the code had the secrets in it, and boom. Now the the hackers are really off to the races.
Brian Valalunga [00:35:12]:
And that's, like that happened multiple times. It keeps happening because, there wasn't a system in place to make sure secrets never entered code. They should never be in code. So yeah.
Charles Max Wood [00:35:22]:
So I I just wanna just jump in here, because, essentially, what you're telling us is just because your GitHub repo is private does not mean that you can check your secrets into it and expect them to stay safe.
Brian Valalunga [00:35:34]:
Absolutely. I mean, that is, like, the number one thing most attackers are gonna be going after is, like, GitHub because they're they're assuming you're gonna be making this exact mistake. And, it's very easy to impersonate a boss, especially in the days of AI. So, these attacks are gonna get more and more sophisticated and feel more and more real, like you're actually talking with your boss. And all of a sudden, boom, they you have access or they have access. And then I'll tell a story. I can't say the name of the company. I can promise you this.
Brian Valalunga [00:36:02]:
You have absolutely heard of the name of the company. You probably have used their product at some point in time, and, they're very big. And this is before they were a customer of ours. They had a data breach, where they hired a malicious act from the company. So there was a rogue agent that they had hired. That rogue agent stole all their credentials, or all their secrets, about 2,000 of them. They detected a bit a bit late in the game, obviously, exited that employee. But then the security team was tasked with rotating these secrets.
Brian Valalunga [00:36:34]:
So, again, swapping out the locks, making sure that all those secrets they stole aren't valid keys to the the locks anymore. Right. And it took their security team, 6 months across 3 engineers, to rotate all of these credentials. So 6 months. Think about that. That's not just, like, 6 months of only doing this and not doing any other projects and a major distraction, but also 6 months that the attackers got to steal data. Well, if data moves at gigabytes per second through AWS, they have an enormous amount of time to drain all the data. Right? Like, they had all the time in the world, basically.
Brian Valalunga [00:37:05]:
Like, when you're dealing in data breaches, you're dealing in seconds minutes, not months. So months is in a in an absolute insane amount of time that the attackers had to, basically use all those stolen credentials to get anything they wanted out of the system or compromise the system in any way. And it was from what I what we heard, it was quite a bit of a battle between the hackers and, like, just because you wrote the key, well, if they use that key to to create new keys, now you're like you have this cat and mouse game. Right? And so it was quite a big challenge for them. Post Doppler, they, have a secret station set up, a whole level of orchestration with their infrastructure, and that same thing would have taken about, like, 30 seconds to a minute. They'd clicked a button and had been done. Wow. But, yeah, it was quite painful for them, and you can imagine all the users that were deeply impacted by that along the way.
Brian Valalunga [00:37:57]:
It's not fun. I think that the the the surprising thing is, like, that that I hear a lot is, oh, this won't happen to me because my company is really big and we had the best security team. Or the I also hear the quite the the the another inversal chapters all the time of, like, I'm a small startup. Hackers won't care about my company. Right? Right. And, like, you hear this from both from the opposite sides. Like, the startups are saying, one thing that the big companies are saying the other. And the truth is hackers are just looking for any company that they can exploit.
Brian Valalunga [00:38:27]:
Does not matter big and small. They're looking for prizes. Sometimes they hit a jackpot, $1,000,000 payout. Sometimes they get a 10 k payout. Right? And, like, or small a small attack or a big attack, but they're just looking to to make an exploit. And so really every and that's why the number of data breaches are just rising exponentially every single year. Like, if you look at the graph, it's terrifying. And it's getting and it's getting cheaper and cheaper for them to do that because the the techniques are getting more and more understood.
Brian Valalunga [00:38:54]:
The AI is getting better and better at doing this. So it's it's a really big growing problem. So, like, everyone is vulnerable. It's not just if you're a startup, you're insulated because of your scale, or if you're a large company you have you're invulnerable because of your scale. Like, Microsoft got hacked. NVIDIA got hacked. Samsung got hacked. Like, these are really, really large companies that you'd think would have their stuff together, and still they got hacked.
Brian Valalunga [00:39:14]:
So it is, it's on all of us. Every sing every all of us are responsible for keeping these credentials safe, so we keep our users safe.
Valentino Stoll [00:39:22]:
Yeah. I think you make a great point. I mean, being able to respond quickly is definitely, like, high up on that list, and being able to remediate that as fast as possible because it will happen. Like, you kind of just, like, have to go with, eventually, something will happen, and you have to, like, just stop it as soon as possible. And in my experience, just having that ability to move quickly, to rotate things around, has been, like, the best defense in the long run, at least from minimizing, damage. Right?
Brian Valalunga [00:39:58]:
Agreed. Like, when you're in a firefight, you shouldn't be strategizing how to fight the firefight. You should just be doing the strategizing should have happened before the firefighting happened. Right. You just wanna be in execution mode when that happens, and you want the execution mode to be as quick as possible. Like, you should click a button and it should be done 30 seconds later, and that's it.
Valentino Stoll [00:40:14]:
Yeah. This it just reminds me too, of a story I I forget what the company was. But some some, like, you know, software engineer at a big company had a home Plex server, and the Plex server was outdated. And it gave, like, you know, access to their home network. And on their home network was a work laptop that they then were able to gain access to because it was on a shared network. And then from there, they then hopped through remote systems. And so it's like, you know, it the you're right. It's getting much more, like, evolved.
Valentino Stoll [00:40:54]:
Oh, yeah. Like, I mean,
Brian Valalunga [00:40:56]:
there's some interesting ones that I've heard of recently of, like, printers, like, old printers that are connected to Wi Fi getting exploited or, like, everyone has all these IoT devices in their home that are connected to, like, the home hub system, for Apple or Alexa or whatever, and those are Internet connected too. So, like, a very small device can then basically get access to the Wi Fi network, which then gets access to every device on that network, and thus, the attack starts. So there's a lot of exploits outside that that don't have to start with just your computer or just AWS.
Charles Max Wood [00:41:27]:
So let's say that I don't know. I'm cheap, and I don't want to use something like Doppler. I don't even know what you cost. Maybe it's, like, ridiculously inexpensive. But let's just say that, you know, I I decide for whatever reason I wanna do this by myself. What what does that look like?
Valentino Stoll [00:41:44]:
Yeah.
Brian Valalunga [00:41:46]:
Well, first off, Doppler is super cheap or free, actually, to start. So it doesn't get cheaper than that. Yeah. It's free for the first three years users, and then it's basically GitHub priced from there. So if you if you're comfortable paying for GitHub, you should be comfortable paying for Doppler. But let's say you don't wanna do that for for whatever reason. I think the most important thing you could do if you're not gonna use any tooling at all is or bare bare minimum tooling is use, like, a cloud secrets provider, like AWS secrets manager, GCP secrets manager, and so on, and then build some tooling around it so that, you can fetch secrets locally in development without them touching disk. So what you need to do is have some wrapper, around your application that starts your application.
Brian Valalunga [00:42:32]:
So I'd imagine the workflow would be, like, fetch secrets from AWS secrets manager, inject them into the environment or write them to an encrypted file, and then have the decryption key in the environment, either path works. And then once the application starts, you clean up the environment, and then load all your other dependencies. And if you did that, plus having your a your secrets in AWS secrets manager and, that being, the same process happening for production workloads, at least you solve most of the risk in my mind. Also, when you're doing this, and this where things get a little bit more of a burden, because AWS, or managing privileges like I'm roles is quite complicated, in my opinion, is you're gonna wanna make sure that every individual developer and every set of infrastructure has their own identity. And that way, you have robust audit logs of, like, okay. The secret was taken by x developer at this time or read by this time, and we we can we have this auditor of what happened. So you don't wanna basically create one master I'm credential and just share it with your entire team because then you lose a lot of your auditing story along with it. And you also probably wanna set up, if you have a little bit of extra time to spend, an auditing story around reads.
Brian Valalunga [00:43:46]:
So, every time a secret is downloaded from this endpoint, you're gonna wanna make sure that you have the right logging and auditing set up in AWS for that, and that you're actually looking at those logs and have analysis of over those logs.
Charles Max Wood [00:43:59]:
Gotcha.
Brian Valalunga [00:44:00]:
Absolute bare minimum.
Charles Max Wood [00:44:02]:
One other thing that I've seen, and I think that I saw this on the Kamal, which is a deployment system that was written by, David Heinemeier Henson. Mhmm. And, anyway, it pull it can pull secrets from, like, 1 password.
Brian Valalunga [00:44:19]:
Yeah.
Charles Max Wood [00:44:19]:
You've seen people do using 1 password for their secrets management?
Brian Valalunga [00:44:24]:
Less and less. So they they, added something called 1 password secrets automation. Everything that I've heard from my community is that they are not actively investing in that product anymore, and it's kinda like shelfware now. So it's She's unaware
Charles Max Wood [00:44:40]:
of that.
Brian Valalunga [00:44:41]:
It's there. It's available. You can use it, but it's not actively being invested in. Right. It's on the shelf. But the amount of people every year that I hear using is less and less because, a, I think there's, like, 1st class products now that that really work. And the amount of things that you want out of secrets manager is growing far faster than what they're adding to, which is at this pace, from my understanding, it's 0. But you can't do it.
Brian Valalunga [00:45:06]:
I mean, like, Secrets Manage or One Password is end to end encrypted. So Mhmm. Like, you're getting a lot of security benefits from that. So I'm not gonna say, hey. Don't ever use it. If if it push came to shove, use it. I just think there's, like, far better things that are also easier to use and far better priced out there that are readily available.
Charles Max Wood [00:45:25]:
Valentino looks deep in thought.
Valentino Stoll [00:45:27]:
Is it is Doppler part of the GitHub, secret scanning part of the program? Yep. That's awesome.
Brian Valalunga [00:45:35]:
If you've if there is a Doppler token a Doppler issue token, found on GitHub, GitHub will contact us immediately. We'll immediately revoke the token, notify you, and work with you to generate a new token. And we're gonna be investing next year pretty aggressively in the secret scanning space as well of, like, really having a full auto picture there. Like, I'd love to be able to say, hey. We found some secrets in your code. We automatically revoked them, issue new ones, and those new secrets are now in Doppler, not in your code anymore. Like, some really powerful workflows like that can just make sure that even though the tooling is set up so that you'd wanna use it, if you actually don't use it and you fall back to bad habits, the the system catches you.
Valentino Stoll [00:46:15]:
Yeah. I'm curious on, like, as far as the observability goes, like, what are the latest, like, trends happening now for, like, observing, like, secret use, and, like, maybe detecting bad actors, you know, ahead of the breach?
Brian Valalunga [00:46:32]:
Detection ahead of the breach, at least in the secret space, is almost 0 from my understanding right now. It's there there isn't a class of tool yet for that. But on the scanning side, there there's TruffleHog, there's GitGuardian. GitHub is adding some stuff too. There is a space developing right now where, basically, you can scan the goal is to scan the perimeter of your infrastructure. Right? So, like, Slack, Microsoft Teams, email, your code base, all of that infrastructure, basically, will be scanned in real time for for secrets. That's basically the extent of what we have today. The problem is, like, you have to connect those dots.
Brian Valalunga [00:47:12]:
Right? So, like, you found a secret. What do you do next? Right? Especially if you're, like, an individual developer, you don't control the secrets manager that's being used. You don't control or if they're not using a secrets manager, what the flow is. And so, like, if you're an if you like, a classic example that I hear all the time is, oh, we added a secret scanner, and the secret scanner found all of our EMV files. But the company has decided we use EMV files. So, like, now it's a moot point because we know seek we've always known secrets were in code because they're in EMV files, but there's nothing to get us to the next step of, like, getting that of code. So now all you're doing is surfacing a problem that we can't solve. So I think that's really where, like, we're trying to help move, the industry is connecting the dots.
Brian Valalunga [00:47:50]:
You found a secret, and you can immediately take an action or an action is taken on your behalf based on a policy, to to clean up the mess.
Charles Max Wood [00:47:57]:
I mean, if you go back and listen to this and then go go through all my stuff.
Valentino Stoll [00:48:01]:
Run your checklist? Yeah. Do you have a checklist, Brian, of, like, you know, at least run through this for your projects or your team, or, do you publish anything around that?
Brian Valalunga [00:48:14]:
Yeah. We have a couple blog posts that that highlight, like, basic secrets management practices. Outside of the stuff I already shared today around, working with, like, as cloud, cloud provider, I would strongly recommend having a secret scanner. So truffle hog straight. Use it. It's open source. Or Get Guardian. Both are partners of ours.
Brian Valalunga [00:48:38]:
The other big thing is outside of secrets, I would have a vulnerability scanner, really important, on all your major dependency streams. So, like, Docker images, Ruby Gems, if you're using Node, NPM. But, basically, any dependency you have, you should have vulnerability scanning on it. Super freaking important. Because it's so easy these days for, like, one dependency that nobody's cared about for 10 years to just get bought for, like, couple $100, and then someone pushes an update for it. And all the other dependencies haven't frozen on that older version. And so now all these dependencies get that update, which means you get the update, and that update has malicious code in it. And that's, like, a very common, like, paradigm for attack these days.
Brian Valalunga [00:49:19]:
So, a, freeze your, your, your dependencies, meaning, like, I'm fixed at this version, and no amount of them pushing a new update will will change that I'm on this version. And this version has been, like, guaranteed good. No, no vulnerabilities in it. And that at least can help prevent any issues where, like, of, vulnerable code gets pushed up and you guys immediately get attacked from it. What you really want is you want all your dependencies frozen, and then, when a new exploit happens, you go, oh, that was kinda cool. We are using that that package, but our version doesn't have that problem, so we're good. And then you just wait for the fix to come out, and then you can do another refresh or another update.
Charles Max Wood [00:50:02]:
Cool. Anything else that you guys wanna dive into?
Brian Valalunga [00:50:05]:
So
Charles Max Wood [00:50:07]:
give us kind of the the, what, 2 minute pitch on on Doppler just just to let people know what it does and how it makes your life better.
Brian Valalunga [00:50:17]:
Yeah. So, at the end of the day, Doppler makes, your life better in a couple of ways. 1, it's gonna add 2 hours of productivity, per developer per week. So, in a way, it kinda pays for itself. On top of that, it's gonna make you far more secure from an organization standpoint, managing your secrets. You can get, you can sign up today, by creating accounts, for free. And then if you're a much, much larger org, you can talk to our solutions engineering team who can help, figure out how this integrates into your complex infrastructure. We have a whole host of great companies that are using us.
Brian Valalunga [00:50:49]:
So you're not alone from Accenture, Crumble Cookies, Puma, Children's Cancer Institute, StockX, travel, TripAdvisor. So there's a number of great companies. You won't be alone in this, and we manage about 30,000,000,000 secrets a month right now. So, like, that's quite large scale. And I I think the key thing here, if if you decide not to use Doppler, is at least manage your secrets. Like, at the end of the day, I think we are all tired of data breaches happening because they keep impacting us, and they impact your customers. And so if you're not gonna use Doppler, that's totally fine by me. You can even use a competitor.
Brian Valalunga [00:51:24]:
Just use a secrets manager. Protect those secrets.
Charles Max Wood [00:51:27]:
I'm a little gratified you mentioned Crumble Cookies there, local company. Anyway,
Valentino Stoll [00:51:32]:
let's go ahead
Charles Max Wood [00:51:33]:
and do our picks, and then we'll wrap up. Now I don't know if you had a chance to listen to other episodes of the show, but we just shout out about stuff that we're, playing, doing, or that we like. Valentino, do you wanna start us off?
Valentino Stoll [00:51:47]:
Sure. Yeah. I mean, the first thing that comes to mind is the Darknet Diaries podcast, which is just, like, an incredible resource of deep diving, exploits. And just like people are just so creative out there. And, nefarious or not, it's just, like, wild to see what, what kind of attack vectors there are and how people are using and exploding your behavior, which can be patterned. Right? So I'd recommend if you're interested in any of that kind of stuff, it's it's a awesome show. Next to that, I I've been following a lot of the rails world, fallout, which is just like, I definitely missed out.
Brian Valalunga [00:52:31]:
I don't wanted to go to
Valentino Stoll [00:52:32]:
speech conference. Yeah. There's so much, so much great stuff that I'm just watching from afar, come out of that, and, it's really exciting. So I guess just watch the YouTube channel for updates of, new talks that are coming out of there. And then the the last pick I have is Landon Gray. He's in in the Ruby AI space, and, he basically made Streamlit, for Ruby, which I'm really excited to, mess around with. And if you're not familiar with Streamlit, it's kind of like the, you know I don't know if it's JavaScript or it's some other language, but it's what a lot of the, you know, machine learning enthusiasts will use to just quickly get up a chat and, you know, fine tune or or use LLMs in creative ways. And Langchain has kind of, like, promoted it for a long time for just quickly iterating through, AI products development, and it's been really incredible, just, like, quickly get up and running if you on something new.
Valentino Stoll [00:53:39]:
And so I'm excited to see that come to Ruby, in a very, like, Ruby way. So it looks very much like a, you know, a rack app or or something like that where you can just run it, and it has a a chat that you can use. And, really excited to see that, develop more.
Charles Max Wood [00:53:57]:
Cool. I'm gonna throw in my picks. So the first one, I always do a board game pick. I'm gonna pick keep the heroes out. So the kind of the design of the game and stuff is kinda like a dungeon for, like, D and D except simplified, and, you're not playing the heroes. You're playing the monsters. And so, each player gets to play a different set of monsters, and they have different abilities. And so the way that you play is you take your turn by, you know, playing your cards.
Charles Max Wood [00:54:34]:
You have a deck of cards, and so it's a little bit of a deck building game. And, you can do all kinds of stuff. You can move. You can create artifacts. You can do stuff with the artifacts depending on which room you're in and what actions are available to you. And, anyway, you can also spawn new monsters, you know, again, depending on what actions you can take. And so then after you do all your stuff, then what happens is then you spawn heroes. And so, you draw the card, and it tells you what rooms to put what kinds of heroes in.
Charles Max Wood [00:55:10]:
The heroes do their special abilities, and then they'll either attack monsters or try to open the chest in the room. Or if if they can't do either of those two things, then they'll move one, step closer to the the vault. And so and the chests are numbered. So if it's a 1, then it takes 1 activated hero to open it. If it's a 2, it takes 2. You kinda get the idea. So, anyway, it that's essentially the game. There are a couple of other, you know, nuances to it, and, of course, it has a series of dungeons that you can play through.
Charles Max Wood [00:55:48]:
And so, you know, there are different rules for the different dungeons as far as how they work. The way you win is you exhaust the hero pile twice. And, yeah, the way you lose is if the heroes open the chest in the vault, which is the 4, chest. So, anyway, really fun game. Takes about, what, 45 minutes ish to play. The artwork on it is awesome. The different types of monsters are fun. And so, yeah, I've played it it plays 1 to 4 players.
Charles Max Wood [00:56:28]:
I've played it with 3 and 4 players. And, anyway, it's it's a fun game. Yeah. So I'm gonna pick that. And then one movie I've been wanting to watch for a while, mostly because I just heard it was really good, is hillbilly Elegy. Now I recognize that, this is the memoir by JD Vance, who's currently, Donald Trump's running mate. So politics aside, right, and I try not to get into it too much in on this show. Longtime listeners know where I stand on all this stuff, but I I don't wanna make the show about politics.
Charles Max Wood [00:57:08]:
But the the story is amazing. And, you know, the fact that, you know, this kid can come from, basically the kind of place and lifestyle that he did and then become you know, wind up going to Yale Law School and then, you know, become a US senator. I mean, I just find that incredibly inspiring. And, you know, at the end of the day then, you know, how you feel about him now, whatever, it it's really well done. It has Glenn Close and Amy Adams in it. I haven't read the book, so I I don't know how closely they stuck to it. But, anyway, there were just several, I think, life lessons that anybody can benefit from, from the standpoint of, hey. Look.
Charles Max Wood [00:57:56]:
If if you really want these things out of your life, then you can go have them if you're willing to do what it takes. And so, you know, from that standpoint, yeah, I I really liked the movie. There was something else I was gonna pick, and I just oh, yeah. It was the micro, film. And this one is on the Angel Studios app. I think it's called some something to stand for, and this is more it's almost a documentary. So if you've listened to the rest of the story by Paul Harvey, it's kind of based on that. That's also the premise of Mike's podcast, at least when he started it.
Charles Max Wood [00:58:39]:
Now he just interviews people that are interesting to him. But what he would do is he would tell the tell the story without telling you who it was about. Right? And so at the end, then he you know, there's the big reveal. And and so the the this person was, you know, Neil Armstrong or whatever. Right? And, you know, and so that's the rest of the story is kind of the idea. And so he tells a bunch of those from, US history, from moments that in some way shaped the direction of the country. And, anyway, I really, really enjoyed that too. So I'll put a link to that in the comments as well.
Charles Max Wood [00:59:15]:
But yeah. And then the last pick I have so Valentino mentioned Rails World, and the stuff that came out of it. I haven't watched any of the talks. I haven't really dived into what came out of it, but I've been using Kamal to deploy my stuff. And I've gotten really tired of fighting the traffic, reverse proxy load balancer. And so I wanted to upgrade to COMAL 2 so I could get, the COMAL proxy put in, and make my life a little simpler because it looked like it may it would make my life a little simpler. And so I did the upgrade, and I kinda screwed it up and then fixed it. So it took a little longer and a little more work than I thought it would.
Charles Max Wood [00:59:59]:
Essentially, what I messed up was, the directions are a little bit the documentation on Kamal is just not awesome. I'll just put it that way. But, anyway, what what it does is it, it tells you to upgrade by upgrading to Kamal 1 9, 1.9. There's 1.9.1 now, but I think it tells you to do 1.9.0. Anyway, because you wanna make sure that you can deploy with 1.9, and then you can upgrade to 2.1 and do the Kamal upgrade. And then it goes in and it stops traffic and it starts the Kamal proxy and it, you know, does the let's encrypt and all of that stuff for you, without having to know all of the arcane magic that you need to know in order to get traffic to do all that for you. And so I'm anyway, I'm pretty excited about it. It's it's been pretty awesome so far.
Charles Max Wood [01:00:56]:
So I I'm gonna pick that. I got a couple of things I still need to figure out on it because I wanna do multi tenancy and allow people to pick their own domains. And so then I want it to do the let's encrypt stuff. And so I'm trying to figure out if I can make Kamal do that for me or, Kamal proxy do that for me without having to have it in my configuration. But for now, it's it's deploying great, so I've been pretty happy with that. Brian, what are your picks?
Brian Valalunga [01:01:27]:
Yeah. Those are some good picks. Hard to compete with that. I think I a huge Star Wars fan, so I'll just shout out Star Wars Clone Wars. I feel like not a lot of people know about it, but it's between episode 23, and it's so freaking good. It's like I'm a guy that usually doesn't, like like, animated stuff, and it's just blown me away. So if you're a big Star Wars fan, there's a lot of seasons, and it's insanely good, and it's basically Dave Filonia at at his best. Outside of that, really big fan of the Perplexity app.
Brian Valalunga [01:01:58]:
It's, like, basically replaced any Google searches I do now. For anyone who doesn't know what Perplexity is, it's Google Google Search plus chat gbt. It's like so ask it a question, and it just gives you the knowledge back. It doesn't just give you a bunch of links. And it uses, and just like a Google search, it's updated in real time. So if something comes out yesterday or earlier today, they'll be informed on that. And it's just an incredible job at answering, like, really in-depth questions. Like, you can ask it very, very complicated questions that have a lot of nuance, that have a lot of requirements on real time data.
Brian Valalunga [01:02:32]:
Like, I don't know. What's the compare the market cap of NVIDIA to its revenue and then compare that to Microsoft's and figure out which one is a better stock to invest in? Like, you can ask that question. I'll give you a pretty concrete answer with, like, charts and graphs, included. So very impressed with, like I think that's a, like, a pretty big step up and the ability to, like, get knowledge very, very quickly. And then the other one is John Coogan. I think he does a great it's a YouTube channel that he that's out there, and he just does an incredible job of, like, talking about, I think, tricky topics. So, like, what's happening with Taiwan and chips or and or old, Prime Defense. Like, he just does a good job of telling the story of some event or some company really, really well, and I feel like it's extremely fact based and not, like, bias based.
Brian Valalunga [01:03:25]:
So I I generally go to him when I'm looking for, like, really high impact news, breakdowns. And then if you're also a Star Wars fan, I'll just end with this again because, again, huge Star Wars nerd here. There's a board game called, Star Wars rebellion. It's played with 2 players. 1 guy gets to play as the empire, the other as the rebellion, and you guys battle it off, and it's it's quite fun. It's a very asymmetric game, but I think that's what makes it fun because, like, Star Wars is very asymmetric. The Empire had a lot more resources than the rebels. But if you like Star Wars, both Clone Wars and that board game will do you a lot of good.
Charles Max Wood [01:04:04]:
Awesome. I don't think I've played this one. It's, it's pretty highly ranked on BoardGameGeek. It has a weight of 3.74, which means it's a pretty complicated game.
Brian Valalunga [01:04:15]:
Oh, yeah. It took us, like, 8 hours to figure out.
Charles Max Wood [01:04:19]:
Yeah. It says, 2 to 4 players, and the playtime is, like, 3 to 4 hours. And I'm assuming that's if you know how to play.
Brian Valalunga [01:04:28]:
Yeah. I'd say me and my brother, we play it took us 8 hours the first game to to get it. The second game, we actually did it across 2 days just because, like, we're we're kinda, like, very slow with every decision to be really methodical, and now we'll play, like, in an hour and a half. So, like, it it once you get comfortable, it ramps up pretty quickly.
Charles Max Wood [01:04:46]:
Yep. But yeah. So the the rank on it overall is 9. So it's the 9th, most popular game on BoardGameGeek.
Brian Valalunga [01:04:57]:
Hey. There we go.
Charles Max Wood [01:04:58]:
Yeah. And it's, it's the 5th thematic, which means that it has some, you know, some theme to it like Star Wars. So yeah. So, next time I'm at a board game convention, maybe I'll see if they have it and if I can play it. Anyway, I'm I'm a board game, not I really enjoy that stuff. So
Brian Valalunga [01:05:21]:
It's, a lot of fun.
Charles Max Wood [01:05:23]:
Yeah. So, Brian, if people wanna connect with you online, how do they find you?
Brian Valalunga [01:05:28]:
Brian@dopper.com for email. Vowel and good brian for LinkedIn and, Twitter. Cool.
Charles Max Wood [01:05:36]:
Alright. Well, thanks for coming.
Brian Valalunga [01:05:38]:
Yeah. Thank you.
Charles Max Wood [01:05:39]:
Alright. Till next time, folks. Max out.
Hey, folks. Welcome back to another episode of the Ruby Rogues podcast. This week on our panel, we have Valentino Stoll.
Valentino Stoll [00:00:11]:
Hey now.
Charles Max Wood [00:00:12]:
I'm Charles Max Wood from Top End Devs. And this week, we're here with Brian Valalunga. Now you are the CEO and cofounder of Doppler, and, that's an app that helps manage secrets. You probably have a better pitch than that, so I'll let you explain what you do.
Brian Valalunga [00:00:33]:
Hey, y'all. It's great to be here. Yeah. Doppler is a secrets manager designed to be your single source of truth for secrets across all your projects, environments, team members, and infrastructure. So you can kind of think about it like as get up for secrets.
Charles Max Wood [00:00:48]:
Right. And I I think I don't know. In in the Ruby community, at least where I deal with people and, especially in Rails, Right? We we kind of understand what the secrets are. It's your your passwords, maybe your, you know, you have your Rails master key, which gives you access to the secrets in Rails, but you also have, like, your what is that? It's a token for your sessions. I can't remember what it's called. Anyway so you you need that secret key and, yeah, people can compromise your security if you don't have that, secured. Are are there things besides, like, passwords and API keys that you consider secrets?
Brian Valalunga [00:01:28]:
Yeah. Outside of, like, API keys, like a Stripe, token Yeah. Like, database URLs, encryption keys are typically the ones that we see the most. Okay. I also just, like, have, like, a general thought process around this of anything that's configured by the environment should be treated by, as a secret, so it should be treated like the most sensitive thing. So, we kinda tell all of or we recommend to, our audience that, you should kinda treat your environment variables and your secrets all the same. Right? Your port and feature flags should be treated as securely as your secrets. That way developers don't need to make the choice of is this secure or is this not.
Brian Valalunga [00:02:05]:
They're always just doing the secure thing by default. And then the trick is just make that that whole workflow and tooling as developer friendly as possible so they wanted to go down that path.
Charles Max Wood [00:02:15]:
Right. So I'm just gonna kinda cover the basics of how I do this, and then maybe you can tell us if there are other ways that you're seeing people do it. But, typically, these kinds of things where I see them, and I'm sure Valentino's in kind of the same boat, is they either show up in environment variables, or they show up in, in the rails secrets or and and, effectively, what it is is it's a, it's an encrypted file. And so then the encryption key is usually put into an environment variable or, you know, passed into the system that you're running it on on the other end in some way. And lately, I've been doing my deployments with Kamal, which uses Docker. And so, it kinda sets that all up when it sets up the container. But I've always kinda wondered because you can ask the system if you can get into it what the environment variables are, so I don't even know if that's the best practice anymore.
Brian Valalunga [00:03:18]:
Yeah. It's a, that's pretty much the the the two paths that we see, either some encrypted file somewhere or it's just directly, available in the environments. I'm a pretty big fan of, if you're going to do the environment path, or inject it through the environment, immediately do a cleanup of the environment before all your other dependencies get get loaded in. So, like, basically, load those, secrets out of the environment, then clean them up from the environment, then load everything else. That way, you can't have, like, a rogue dependency go and capture all your secrets and then send it somewhere. The other path of, like, using a file, that's encrypted is also a really good path too. So I think both are pretty good.
Charles Max Wood [00:04:00]:
What are you doing, Valentino? Is it kind of the same? Or
Valentino Stoll [00:04:05]:
we use a lot of AWS stuff, to handle a lot of the
Brian Valalunga [00:04:11]:
lot lot
Charles Max Wood [00:04:12]:
of the secrets to management.
Valentino Stoll [00:04:13]:
We have a secrets to management. In there. Yeah. And I think that it may even be isolated, from the main environments too and, handled that way in a isolated environment. We we use Docker as well, so it could be, like, an isolated container that then feeds back into the other container.
Brian Valalunga [00:04:32]:
So are you, I'm guessing you have your secret stored on, like, AWS secrets manager, and then those secrets are then pushed into the container as environment.
Valentino Stoll [00:04:40]:
I think so. Something like that. It may even be one step further where, it's in AWS and then at, like, run run time, it's like the secrets are loaded into the app's memory, for the particular use case. I don't really know all of the details, to be honest. I'm not on the infrastructure team.
Charles Max Wood [00:05:00]:
Every once
Valentino Stoll [00:05:01]:
in a while, we have to rotate keys, and, that's definitely, you know, an easy process for us. But, yeah, we have some command line that does it. It's not just like changing the environment variable.
Brian Valalunga [00:05:16]:
Sounds like you guys are pretty well set up. I mean, that's that's, like, the dream is to to make secret expansion super easy and simple, but also very secure at the same time. Yep.
Valentino Stoll [00:05:26]:
Yeah. That that makes me curious because, like, I think at one time we were using, forget what HashRockets product is. We ended up moving away from that. Yeah. I don't know. We were we were using some vendor for doing the secrets management. But, yeah, having a command line or some kind of central place, a command line doesn't even need to be your, you know, your UX of choice, but just having the centralized place where all of that access points happen, it definitely helps, like, isolate and track, to be honest, who's touching what and changing it. I I don't know if Doppler has those kind of features as far as, like, tracking changes and, getting the full logs and things like that, but, that's definitely super valuable.
Brian Valalunga [00:06:20]:
Yeah. So I think, DOP where kinda comes from a first principle standpoint about, like, what is a good secret manager look like? And for us, we have a couple of things that we deeply care about. First is that all your secrets are centralized in one place, across, your projects and environments. And, once they're all in one place, we can set up a pretty robust access controls either by users or by groups that are policy driven. And then from there, you get a complete audit story. So you know every time someone has read a secret, written a secret, changed some settings, on every device with their IP address, you get a a whole, a picture about everything that's going around or going on around your secrets. And then the other key part is that it's deeply connected to their workflows and their infrastructure. So on the workflow side and, like, local developments, we want you to be able to edit your secrets side by side with your code.
Brian Valalunga [00:07:08]:
So you have a Versus code extension. And so, those secrets are actually kinda look like a a YAML file, but they're actually not. It's like a we use something called a virtual, file system. So it behaves like a file, but it's actually not. It's connecting with on the back end. So it's super secure. And then, on the orchestration side for production is we will write into AWS secrets manager and Kubernetes or wherever it may be so that your infrastructure immediately gets those secrets when they change, and restart your, your deployment so you don't have any downtime.
Charles Max Wood [00:07:38]:
Yeah. I remember using a system like this way back in the day. I think you had to log in to their interface or use their command line interface in order to edit your secrets. And I can't for the life of me remember what it was called, but, I played with it and it was cool. It was just there there were steps to setting it up, and I'm lazy, and so I just went back to what I was doing.
Brian Valalunga [00:08:00]:
I am for that matter. You you gotta make it at like, we have this theme at Doppler, and it's it's quite simple. It's make vegetables taste like candy. Vegetables being the security, candy being the developer productivity. If you do not make it if the tool does not make the developers' lives better like, you can't just say, hey. Use this tool because it's more secure. Like, that doesn't fly because developers have been they care about moving fast and shipping product. And so you have to aid them in that.
Brian Valalunga [00:08:22]:
You have like, in our case, we try to get them 2 hours a day, or sorry, 2 hours a week back in productivity. And so if you if you do that, if you give them good candy, then you'll get the vegetables too.
Charles Max Wood [00:08:33]:
Yeah. That makes sense. I I kinda wanna dive in a little bit more on just why we protect the secrets, because I I think we all kind of intellectually know.
Valentino Stoll [00:08:46]:
Mhmm.
Charles Max Wood [00:08:47]:
But I don't know. When I get into stuff and I start, you know, dealing with the secrets, it's like, oh, somebody got access to my, like, my Stripe tokens. I mean, that that could be really, really bad. Yeah.
Brian Valalunga [00:08:58]:
It's really bad. So I like to kinda bucket the risk in in or or sorry. Yeah. Bucket the risk into 2 buckets. The first is, like, the risk of the company, and the second is the risk to people. And I think the company side is, like, pretty well understood. Right? You're gonna have a pretty if you have a data breach, you're gonna have a pretty big brand hits and trust hits. You're gonna probably have some unexpected spend because now you're investing, like, legal PR insurance.
Brian Valalunga [00:09:24]:
You may get sued as well because if you're not managing your secrets and the secrets are things that get leaked that leads to a data breach, that's just straight of negligence, and you your and the customers are right are in their rights to sue. But I think the thing that's, most of the time not talked about is, like, the cost to real people. At the end day, all these companies that we use, we're trusting them to store our data and keep our private data private. Imagine if, like, every YouTube video that you ever saw got out or, every ride you ever took on an Uber, every transaction you ever made on a credit card, all got out. And I actually have a real personal story that of, like, how this has impacted me. Sadly, I've been part of a couple data breaches now from some services I've used, and, that came back and really bit me in a in an interesting way. I was so I just moved to Austin, Texas couple months ago and, from California, and I convinced my mom to fly out with me and, see Austin for itself when we were at this barbecue joint. And while we're eating food, I get a call from, the Texas Customs and Borders, and they say, hey.
Brian Valalunga [00:10:24]:
We found a package in your name that has, illegal money and drugs in it, and we're, federally investigating you now. And this is, like, the phone call for me where I was like, oh, shit. My life has just ended. Like, from the highest
Valentino Stoll [00:10:36]:
to the
Brian Valalunga [00:10:36]:
most really, really quickly. And I was really scared. And but, like, also kinda like this is odd. Like, never bought anything from Mexico before. So, like, okay. And the back of my head, I was like, this is a scam probably. But then they started rattling off all this information about me that they knew, all the places I've lived in the past, locations I had been to, like, just so like, a wealth of of information about me that I was like, no one could possibly know this but the government. And so, like, I trusted them.
Brian Valalunga [00:11:09]:
I was like, okay. I'm actually being investigated here. And, obviously, we get lawyers on the call as well. Like, this becomes a really scary ordeal for all of us. And it wasn't until about, like, an hour in that our lawyers picked up that it wasn't that and but during that hour, we also gave them a whole lot more information about me, right, because we thought we'd be investigating. They were asking questions. And so, like, this is and, like, again, I'm a CEO of a of a cybersecurity company. I'm trained on this.
Brian Valalunga [00:11:37]:
Our employees are trained on this. Our legal team is trained on this, and they still got us for an hour. Right? I can imagine everyday Joe who's not trained, could really get, attacked here. And, like, I was lucky that all they did was get a little bit more information about us, But they were able to get that information because of other information that got breached. Right? That that bought them the credibility, to get new information. But imagine, like, my mom, my sister, or anyone else like that who's not gone through any of that training does not have legal counsel immediately, readily available to them that can jump on the call for this. Oh my gosh. I cannot imagine all the information that have been given up and all of a sudden their bank accounts are drained or there's or or their credit score is completely ruined or whatever it may be.
Brian Valalunga [00:12:15]:
It can get really terrifying really fast. And that's why I come back to, like, whenever I see, like, a data breach and it's, like, x millions of people's data was just got out. It's like, wow. Like, scammers are now, like, parading in the streets going like, oh my gosh. I just got a ton of data that's gonna get me so much more data that can eventually get me to, like, the jackpot of all this money or or or whatever they're trying to go after. And so there's real people's lives, including mine, that have been impacted by this, and that is something that's, like, often not talked about at all. And so when you're a developer and you're like, oh, should I put this in a new default or not, or should I secure my secrets? Just one token. It's it's a string.
Brian Valalunga [00:12:48]:
It's like, well, that string may impact a 1000000 people's lives someday, so to to treat right.
Charles Max Wood [00:12:54]:
Right. Well, it it reminds me your story a little bit of, have you read Ghost in the Wires? I I can't is it Kevin Mitnick?
Brian Valalunga [00:13:02]:
Yeah. Kevin Mitnick. Yeah. I I've watched the movie about it, but I haven't read the book.
Charles Max Wood [00:13:06]:
And see, I didn't know there was a movie, but I read the book. Right. But, yeah, he talks about how effectively yeah. He he gets just a little bit of information about the next person in the chain. Right? And so he doesn't start with the the analyst or whoever. Right? He starts with, you know, the the lowdown person on the totem pole and then, you know, gets enough information to know who the next person is and then works his way up. And so it's yeah. It was it was a fascinating read just from the the sense of yeah.
Charles Max Wood [00:13:44]:
You know? Just just anyway. So yeah. So I can definitely see that. I I think the other thing that I worry about is, like, if I have a key to, I don't know, like, chat GPT or something like that, somebody could run up my bill Yep. Or, you know, somebody gets in my hosting account and, you know, all of a sudden I've got you know, they they breach my other stuff that way. And, anyway
Brian Valalunga [00:14:13]:
No. I think you're totally right. I mean, there's 2 common things that we see all the time. One is, like, some source code goes public or some email gets leaked and that has a secret in it to, like, an AWS account, and you'll see 1 of 2 things happen immediately. The first is their build just, like, skyrockets in seconds because now they're using, AWS for some other bot attack or for mining crypto or whatever it may be. That happens very, very commonly. And it's surprisingly, like, how fast that that, like, that those secrets get found. Like, I think there was, like, an average stat that it's, like, within 3 to 5 seconds of, like, a repo going public.
Brian Valalunga [00:14:45]:
If it has a secret in it, even in the history, it will get found and and and start getting exploited. So it's, like, very quick. Like, it's it's it's insanely quick. And then the other side of it is now that they have access to AWS, they can start scanning all your resources. Okay. What, what execution environments are set up? What databases are set up? Can I get access to those database? How privileged is this credential? And you start having this ripple effect where it's like, oh my gosh. Now that especially if it has, like, like, strong super admin permissions, it can start creating other credentials. And now you have this, like, cat and mouse game of, like, how do I get out of the system? Could it lock me out of the system? Did I, are there backdoors it's placing in? So even if we remove it, it's still actually there.
Brian Valalunga [00:15:25]:
Like, it gets scared really quickly. Once they get into the system, it's very, very hard to get them out.
Valentino Stoll [00:15:29]:
Yeah. That's always makes me wonder, like because there's two parts of this. Right? Like, we have the secrets and then there's, like, the whole encryption aspect of it, which is its own separate piece of the discussion.
Charles Max Wood [00:15:42]:
Wait. When you say the encryption aspect of it, is that, like, how you're securing your secrets, or is that, like, encryption keys that
Valentino Stoll [00:15:47]:
you're I mean, ultimately, what you're using a lot of the secrets for, I would imagine, is to encrypt something, or or even to, yeah, I mean, to encrypt stuff.
Charles Max Wood [00:15:58]:
Yeah. You're either hiding stuff or accessing stuff is usually what you're doing.
Brian Valalunga [00:16:02]:
Based on the data that we have from Doppler, we can't see customer secrets at all, but we can, get an auto report of the names of the secrets they use, if they give us that permission. And, from the data we have, it's, like, 90% of the of the sensitive secrets that we put in Doppler and customer's inevitability say, this is sensitive or this is not, is actually, like, access token. So it's not encryption tokens. So it's, like, if you get that Stripe key, you have it all. Like, you have every customer's credit card and bank account in and there. There's no extra layer of protection.
Valentino Stoll [00:16:37]:
Yeah. That's all I I always wanna one of the the biggest things that, I always hope for when I'm, like, getting a new vendor, like, access token or or whatever it may be, is that they, like, have some kind of, like, you know, white list stability with, like, either their domain, like, a domain key or some kind of identifier that can be, like, you know, securely matched when the handshake is made, using their service from whatever server that you end up using. Right. And it's in my experience, like, that it's very, honestly, very rare Yes. That that actually happens. And so I I under I that makes a lot of sense, you saying that your customers, like, mostly store, like, access tokens because most vendors don't give you that, like, white list ability and, like, handshake process, which is, like, very important. Right? I think causes a lot of the breaches when it's not there.
Brian Valalunga [00:17:38]:
Agreed. I I do think that the world is changing. It's just gonna be very, very slow. So, like, you can see this in passwords today where we're going from passwords to pass keys. So anyone, familiar with pass keys, it's basically like, you can sign into a website with, like, your face or touch ID, and you never had to put in a password. It's a password passwordless world. And the way that's working is through cryptography. You have, like, a fingerprint that leads to a cryptographic signature that can be used to authenticate yourself.
Brian Valalunga [00:18:04]:
And that that thing that's happening in passwords is now starting to happen in secrets with identity based authentication. And so, like, the kind of picture that we can start to see happening very, very slowly in, like, Doppler's job is to, like, help move this along, help move the industry along is, to have, like, this bridge of connection. So, like, on one side, you have, like, your AWS, GCP, execution environment, and that has an identity already today. And then you can pair that with Doppler, and now you're authentic by the fact that you're in that e c two instance, sir, you are now automatically authenticated to Doppler. And then eventually, Doppler will have the connection of, now that you're authenticated with Doppler, you're automatically authenticated to Stripe. There's no more API key needed. We are all cryptographically, authorizing each other across this chain or this highway of, authentication. And that's kinda like the dream state that we're gonna be getting to over the next couple years.
Brian Valalunga [00:18:55]:
It's gonna take a lot of work of working with partners, but now we have to convince Stripe, Twilio, and so many other providers to enable this. And we're gonna try to make this easy as possible, but it's gonna be it's gonna be an uphill battle. But if we can do that, we're in a secretless world.
Valentino Stoll [00:19:07]:
Is there an open standard, that is following this process or no?
Brian Valalunga [00:19:12]:
There is a little bit of an open standards mostly between, like, AWS and x, like Doppler or whatever it may be, but there's no open standard on or even a standard altogether, on the, like, secrets manager to, third party service like Stripe or Twilio. So I think our goal is we're gonna probably try to create an open standard that both sides adhere to so that if you're anywhere in this chain, you can directly, you can just plug in very quickly. And I think we'll probably have to build SDKs for, like, every primary language, and Kubernetes and a bunch of other, infrastructure tooling as well, so just, like, works out of the box.
Charles Max Wood [00:19:49]:
So, I I mean, I I I find this all really fascinating. But I I guess my question is, what kinds of mistakes might I be making now? Or, you know, what what kinds of things should I be considering doing now to protect these secrets? Because, yeah, you know, what you're talking about sounds brilliant, but it it doesn't exist yet.
Brian Valalunga [00:20:11]:
Yeah. And I'm
Charles Max Wood [00:20:12]:
just I'm a little worried that, okay, we're gonna talk the rest of this time about futuristic stuff, and then I'm gonna be yeah. I'm gonna miss something.
Brian Valalunga [00:20:20]:
Alright. So I I have 4 questions that I think every person listening, should ask themselves, and it's, like, basically, a very simple sniff test to see if I have a problem. So, to highlight what the problem is, it's called secret sprawl, and that is basically all your secrets are scattered across a bunch of different places. And because of that, you have no way of controlling it or governing it, which is, like, one of the most important parts about protecting your secrets is being able to govern those secrets. And so the 4 questions are, can you confidently answer where are all my secrets? Right? Not just the ones that you know about in, in a secrets manager somewhere, but all the ones on developer's laptops, all the ones in Slack and email and, your code somewhere. Like, you gotta have a complete picture where all your secrets are, and you gotta be able to make that guarantee that you that you're not gonna miss 1 by not knowing about it. And I'll I'll talk about how to solve that in a second. The second is, do you know who has access to all of those secrets? And this gets really scary, especially if you don't have, like, a centralized system because, like, if I have a file, I can just share it to another person, and no one on the security team is gonna know about that.
Brian Valalunga [00:21:22]:
Mhmm. Can I control who has accessed those secrets when they were accessed, and by who? Right? So do I have access controls, and do I have an auto log associated with those access events? And then most importantly, when data breach does happen, can I stop it very, very quickly? When I say quickly, I mean, in the orders of seconds and minutes, not in the the order of days and and weeks or months. And so if you cannot answer those 4 questions confidently, like, to a point where your job would be on the line for it because it essentially is on the line for it, then you have a problem. And I can talk about in a sec what it looks like to have a solution there. I don't have to go into, like, Doppler's features, but I can talk about, like, the fundamental properties you'd wanna look for in any secrets manager, Doppler or, someone else.
Charles Max Wood [00:22:12]:
Yeah. I think that's important. It's funny because so I kinda have two levels of this, one where I'm working for a client and the other one where I'm just working on my own stuff. And on my own stuff, I'm fairly confident that I can answer these questions because I'm basically the only person accessing or using them. But then the question for me becomes, okay. Well, how do I how do I you know, let's say I bring in another developer. How do I make sure that I know the the answers to those questions as I bring them on board?
Brian Valalunga [00:22:46]:
Yep. And I think once your surface area grows of team members, infrastructure, devices, it gets pretty hard to to to answer those questions confidently, unless you have a dedicated system for it.
Charles Max Wood [00:23:01]:
Right. So why
Brian Valalunga [00:23:02]:
don't we be able to go into, like, what actually does a dedicated solution look like? The first thing is you can centralize all your secrets in it. So once you know where all your secrets are, you can confidently say, this is the place we're doing. We're not gonna store our secrets in code. We're not gonna send it over email or Slack or Microsoft Teams or whatever it may be. This is the one place you can get all your secrets. Now you have a complete landscape of your secrets. The next thing you should be able to do in this system is you should be able to, can set up access controls and auto logs around them. So you can say, you have this develop, full stack developers have access to the development environment for these sets of projects for the teams they're on, but the DevOps team has access to production.
Brian Valalunga [00:23:38]:
So you can start gaining access and you have a full audit picture of it, and, obviously, that audit picture then gets pushed into your auditing tools like Datadog, Sumo Logic, Amazon, whatever maybe. Mhmm. You should be able to have this infrastructure natively integrated with their developer workflows for local developments. That way developers aren't working around the system and then recreating that risk you're trying to avoid. And it should be natively integrated with your with your production and and staging infrastructure so that when it is time to roll out a secret, you don't have this race condition of the code arriving before the secrets do. The secrets have to arrive first if the code is gonna rely on the those secrets. And so then this kinda gives you this whole picture of I know that if I can see the secret in this project, in this environment, on the solution like the secrets manager, then it's guaranteed in my production infrastructure or it's guaranteed on my developers' laptops in a secure way. And that basically gives you the ability to remediate secrets pretty quickly.
Brian Valalunga [00:24:33]:
And so the last part of this puzzle is being able to rotate secrets. Rotation is the fancy way of saying, I'm gonna swap out the locks on the door. And you gotta do it in a in a way because if you just, like, immediately destroy the lock, in this case, you'll just bring yourself down in production. So you have to do it in a way that does not create any downtime, and it's usually requiring a 2 k system. So, like, in in a database, for example, you don't ever wanna not be able to connect to the database. So you'd have, 2 credentials to the database, and you swap them, and then you rotate 1, and then you swap again. And so you want a system that does all that. That should not be done by humans.
Brian Valalunga [00:25:08]:
That should be done by a a system that's running on a schedule where you have an immediate, like, I'm fucked button. I need to click this thing to get me out of a a data breach. And then, obviously, it orchestrates to the rest of your infrastructure. And so those are the things that you should be looking for in a in a in a developer for secrets manager.
Valentino Stoll [00:25:23]:
So I love this. You have this great, secrets for all page on Doppler, I linked to. And, it's really great. Kind of helps you quickly, assess your risk exposure based on your team size. You you go through kind of, like, a quick calculation. Oh, we have, like, this number of projects, estimate number of, secrets per project, then you have certain number of environments. Right? So it's a multiplier, and then you have, you know, a multiplier over the number of team members too. And so those quickly add up, and you make a great point.
Valentino Stoll [00:25:55]:
Like, you know, the malicious actor, they only needs they need 1, right, of the 1,000 that that adds up to. And so I'm I'm curious, like, you know, from the from the like, the whole point kind of is, like, risk assessment and remediation. Right? And, like, do you find that, focusing on the that risk aspect is just as important as, like, kind of the secret storage management mechanisms.
Brian Valalunga [00:26:25]:
Yeah. They go they go in hand in hand for sure. You need something that developers are gonna wanna use that is secure when they do use it, and you wanna think about that risk. Like, a good a good way that I kinda, like, talk about it when I'm talking internally with our employees is the game of offense and defense are very different games. The game of defense is build walls around your perimeter as high as possible, as strong as possible with no, no cracks in the foundation. Right? And then the goal of offense of an of an attacker of an hack of a hacker is to find one crack in the foundation and hit it as hard as freaking possible that bring brings down the entire wall. Right? They only need one weak link. And if seekers happen to be that weak link, well, then you're really giving them the keys to your digital kingdom.
Brian Valalunga [00:27:08]:
That's extra bad. I'd much rather be the weak link be something far outer in the perimeter like, I don't know, some authentication bug or something like that that that impacts, like, 1 or 2 users, not your entire users, user base. And so, yeah, managing risk is incredibly more near. That's that's the end goal that you're trying to accomplish is, like, make sure you can stay up and orchestrate, like, the get get all the DevOps and platform tooling you're looking for, and then also manage the risk associated with that.
Valentino Stoll [00:27:38]:
So I'm curious, like, if you have any tools or, like, what would you do to kind of, like, visualize the, like, you know, organizational structure in combination? Like, it's more than just, like, secrets management, it seems. Right? Like, how do you how do you personally visualize, like, the security aspects surrounding the secrets and who accesses them and and things like that, to understand kind of what is happening and where the access points are?
Brian Valalunga [00:28:08]:
Yeah. Great question. Truly great question. We've tried really hard to build a model that maps very closely to what you'd have, with code, so with GitHub, or GitLab. So we have projects just like you have repos in GitHub and GitLab, and we have environments which are very similar to branches. And then we have user groups that map in, with access control policies. And then we have integrations that connect to those environments. And so you can have these, like, this bare bones hierarchical structure of projects and environments.
Brian Valalunga [00:28:41]:
From there, you can assign users to them and you can assign infrastructure that connects with them. And it's very easy within Doppler to kinda see all that connect together in an extremely visual way. We even have, to to show kinda how far we've taken it, when you go into a project, you actually have a kanban board, style, you want user experience, that shows all your different environments from development to staging to production, everything in between.
Valentino Stoll [00:29:05]:
So what inspired you to start Dropletr? Like, where where does the story start? Like, I'm curious, like, where your background is and, you know, how how you're like, oh, we need a better, you know, secrets manager.
Brian Valalunga [00:29:17]:
Yeah. It had nothing to do with secrets from the started. I was working on, a crypto machine learning marketplace, all kinda all the buzzwords in one. One on the side. It was literally every buzzword you can find jammed together. And, man, that was a hard project. I worked on it for about 8 months straight, and it felt like pushing a massive boulder up a hill. Every foot forward, I'd slip 5 5, 6, 5 to 6 feet back from exhaustion.
Brian Valalunga [00:29:44]:
It was just hard. And at one point, I, like, really got to my breaking point. I was like, I don't know if I can keep working on this thing. And so I decided to take a trip to to Cancun, Mexico, and the whole goal was, like, don't think about this project. Actively do not. And, obviously, I failed at that miserably. I was thinking about it the entire time I was there. And what I realized is that this pro I was never gonna be able to get this project off the ground, but that doesn't mean that I had to stop there.
Brian Valalunga [00:30:10]:
I think, like, there's a lot of people that, like like, follow, like, sports players, and, like, they know all the stats of all these players. I'm very much the same way with founders. I'll I, like, I love learning about founder stories, and I think Stuart Butterfield from Slack is, like, the best feeling upwards I've seen. Creates a video game. Out of that video game the video game fails, born out of that was Flickr. Comes back a second time, so, like, I'm gonna create a video game again. Born out of that was, Slack, which we all use today. And so I was like, what can I learn from, from my experience with this crypto machine learning marketplace? And managing secrets was just one of them that was, like, super painful in all these weird interesting ways.
Brian Valalunga [00:30:49]:
It was weird from a developer productivity side because, like, I always felt like I was copying and pasting things in, But sometimes the code would get into production faster than the secret shorts I had a have an outage. I had a couple of developers that I was working with on this project, and it was, like, really painful to sync all these secrets with them. Like, I literally Slack them and email them the secrets, and it I like my heart would cringe every time I do that. Like, oh, that didn't feel right, but that's the only thing I have at my disposal to do. And there's actually one time where we we dropped the crypto part of the marketplace because it wasn't working. We just did straight up machine learning marketplace using Stripe. And I had the Stripe production key in staging and the staging 1 in prod, and it took us a month to figure out why we weren't doing any transactions. And it was because of that, because the Stripe, the Stripe testing key, rejects all live credit cards.
Brian Valalunga [00:31:33]:
It only uses test credit cards. And we didn't realize that for a long time. And so I was like, okay. This is a real problem. And so I come back from Cancun, and I go to this dinner that Stripe was hosting that had a bunch of founders and developers added. And I we were all going around the room talking about what we were thinking about, and I just asked her openly, am I a shitty developer? Is the world broken? I just can't tell anymore. Are you guys having this problem too? And, like, 60% of them were like, yes. This is a very big problem for us.
Brian Valalunga [00:32:01]:
And one woman in particular, like, literally comes running up to me right afterwards. I thought she was gonna trample me. And she says, I've had 3 outages this week. Hand me a solution by Sunday. And I was like, no. No. No. No.
Brian Valalunga [00:32:11]:
No. Like, I'm not even started yet, and this is a Wednesday. There's no way this that's happening. She goes, I don't give a fuck. Have it by Sunday. And I was like, okay. There are some interest here if people want it. But we didn't obviously have it by Sunday.
Brian Valalunga [00:32:23]:
That that never happened. But, what we did is we started doing a lot of research, and we found that individual developers are struggling because there wasn't tools made for developers. Medium sized companies are struggling building their own things, and large companies were buying Hashcore Vault, which really isn't meant to solve this problem and still struggling. So, like, everyone's struggling. And I was like, okay. We can build something that's developer first instead of instead of just purely security first. And we got our first version working in 3 weeks, and we got a set of customers right after that by just dragging them to Chipotle and saying, hey. I'll get you anything on the menu as long as I get a rant at you for about 2 hours.
Brian Valalunga [00:32:57]:
And that got us, our first set of customers and off to the races we were.
Charles Max Wood [00:33:02]:
Yeah. It's funny how, yeah, you wind up scratching an itch that's not how do I put it? It's not like the core thing you were trying to build Yeah. Anyways. Off.
Brian Valalunga [00:33:11]:
But turned out to be a good business.
Charles Max Wood [00:33:13]:
Yeah. It's it's interesting because I've been kind of in that same space where I've been looking for a solution that does a whole bunch of things that I want for the podcast, and I've just you know? So I've built little pieces of it here and there, and so I finally just started bolting it together so that I have the full solution. And, yeah, it's it's interesting how many how many times you have, yeah, things come out of that.
Brian Valalunga [00:33:37]:
Yeah. I would probably index and be like, okay. Are other people that I know that also do podcast recording asking for this tool for this tool? Do does it continuously get better and better the more I I mount on to this, like, hodgepodge of a product? And is that value going up, exponentially not linearly? And if the both those answers are are yes and you still feel like there's nothing in the market that does anything close, you have a product. The question is, is that products product 10 x better than whatever's in the market? Because if it isn't 10 x better or 10 x different, people there won't be enough of a differentiator to create a buying event, I found. Right.
Valentino Stoll [00:34:13]:
So I'm curious, like, what what are some common, like, preventable breaches that you've encountered, and, like, how can developers avoid them? Right?
Brian Valalunga [00:34:22]:
Yeah. There's a couple that really come to mind. I mean, let's look at
Charles Max Wood [00:34:27]:
He left like he's got some doozies of some stories.
Brian Valalunga [00:34:32]:
They're all public.
Valentino Stoll [00:34:33]:
Okay.
Brian Valalunga [00:34:34]:
I'll share one that, like, isn't public, but I won't say the name of the company, and then I'll share a couple that are public. So Twitch, Toyota, and Twitter all kinda had the exact same breach happen. And they had secrets in code. The code got leaked somehow either by them leaking their, them opening up the repos themselves or by hackers being able to get access to the GitHub account from, an employee that got compromised. So, like, someone, got called and said, hey. I'm your boss, and, give me access to GitHub, and then the hacker got access to GitHub. And then from get from GitHub, they saw the code, and the code had the secrets in it, and boom. Now the the hackers are really off to the races.
Brian Valalunga [00:35:12]:
And that's, like that happened multiple times. It keeps happening because, there wasn't a system in place to make sure secrets never entered code. They should never be in code. So yeah.
Charles Max Wood [00:35:22]:
So I I just wanna just jump in here, because, essentially, what you're telling us is just because your GitHub repo is private does not mean that you can check your secrets into it and expect them to stay safe.
Brian Valalunga [00:35:34]:
Absolutely. I mean, that is, like, the number one thing most attackers are gonna be going after is, like, GitHub because they're they're assuming you're gonna be making this exact mistake. And, it's very easy to impersonate a boss, especially in the days of AI. So, these attacks are gonna get more and more sophisticated and feel more and more real, like you're actually talking with your boss. And all of a sudden, boom, they you have access or they have access. And then I'll tell a story. I can't say the name of the company. I can promise you this.
Brian Valalunga [00:36:02]:
You have absolutely heard of the name of the company. You probably have used their product at some point in time, and, they're very big. And this is before they were a customer of ours. They had a data breach, where they hired a malicious act from the company. So there was a rogue agent that they had hired. That rogue agent stole all their credentials, or all their secrets, about 2,000 of them. They detected a bit a bit late in the game, obviously, exited that employee. But then the security team was tasked with rotating these secrets.
Brian Valalunga [00:36:34]:
So, again, swapping out the locks, making sure that all those secrets they stole aren't valid keys to the the locks anymore. Right. And it took their security team, 6 months across 3 engineers, to rotate all of these credentials. So 6 months. Think about that. That's not just, like, 6 months of only doing this and not doing any other projects and a major distraction, but also 6 months that the attackers got to steal data. Well, if data moves at gigabytes per second through AWS, they have an enormous amount of time to drain all the data. Right? Like, they had all the time in the world, basically.
Brian Valalunga [00:37:05]:
Like, when you're dealing in data breaches, you're dealing in seconds minutes, not months. So months is in a in an absolute insane amount of time that the attackers had to, basically use all those stolen credentials to get anything they wanted out of the system or compromise the system in any way. And it was from what I what we heard, it was quite a bit of a battle between the hackers and, like, just because you wrote the key, well, if they use that key to to create new keys, now you're like you have this cat and mouse game. Right? And so it was quite a big challenge for them. Post Doppler, they, have a secret station set up, a whole level of orchestration with their infrastructure, and that same thing would have taken about, like, 30 seconds to a minute. They'd clicked a button and had been done. Wow. But, yeah, it was quite painful for them, and you can imagine all the users that were deeply impacted by that along the way.
Brian Valalunga [00:37:57]:
It's not fun. I think that the the the surprising thing is, like, that that I hear a lot is, oh, this won't happen to me because my company is really big and we had the best security team. Or the I also hear the quite the the the another inversal chapters all the time of, like, I'm a small startup. Hackers won't care about my company. Right? Right. And, like, you hear this from both from the opposite sides. Like, the startups are saying, one thing that the big companies are saying the other. And the truth is hackers are just looking for any company that they can exploit.
Brian Valalunga [00:38:27]:
Does not matter big and small. They're looking for prizes. Sometimes they hit a jackpot, $1,000,000 payout. Sometimes they get a 10 k payout. Right? And, like, or small a small attack or a big attack, but they're just looking to to make an exploit. And so really every and that's why the number of data breaches are just rising exponentially every single year. Like, if you look at the graph, it's terrifying. And it's getting and it's getting cheaper and cheaper for them to do that because the the techniques are getting more and more understood.
Brian Valalunga [00:38:54]:
The AI is getting better and better at doing this. So it's it's a really big growing problem. So, like, everyone is vulnerable. It's not just if you're a startup, you're insulated because of your scale, or if you're a large company you have you're invulnerable because of your scale. Like, Microsoft got hacked. NVIDIA got hacked. Samsung got hacked. Like, these are really, really large companies that you'd think would have their stuff together, and still they got hacked.
Brian Valalunga [00:39:14]:
So it is, it's on all of us. Every sing every all of us are responsible for keeping these credentials safe, so we keep our users safe.
Valentino Stoll [00:39:22]:
Yeah. I think you make a great point. I mean, being able to respond quickly is definitely, like, high up on that list, and being able to remediate that as fast as possible because it will happen. Like, you kind of just, like, have to go with, eventually, something will happen, and you have to, like, just stop it as soon as possible. And in my experience, just having that ability to move quickly, to rotate things around, has been, like, the best defense in the long run, at least from minimizing, damage. Right?
Brian Valalunga [00:39:58]:
Agreed. Like, when you're in a firefight, you shouldn't be strategizing how to fight the firefight. You should just be doing the strategizing should have happened before the firefighting happened. Right. You just wanna be in execution mode when that happens, and you want the execution mode to be as quick as possible. Like, you should click a button and it should be done 30 seconds later, and that's it.
Valentino Stoll [00:40:14]:
Yeah. This it just reminds me too, of a story I I forget what the company was. But some some, like, you know, software engineer at a big company had a home Plex server, and the Plex server was outdated. And it gave, like, you know, access to their home network. And on their home network was a work laptop that they then were able to gain access to because it was on a shared network. And then from there, they then hopped through remote systems. And so it's like, you know, it the you're right. It's getting much more, like, evolved.
Valentino Stoll [00:40:54]:
Oh, yeah. Like, I mean,
Brian Valalunga [00:40:56]:
there's some interesting ones that I've heard of recently of, like, printers, like, old printers that are connected to Wi Fi getting exploited or, like, everyone has all these IoT devices in their home that are connected to, like, the home hub system, for Apple or Alexa or whatever, and those are Internet connected too. So, like, a very small device can then basically get access to the Wi Fi network, which then gets access to every device on that network, and thus, the attack starts. So there's a lot of exploits outside that that don't have to start with just your computer or just AWS.
Charles Max Wood [00:41:27]:
So let's say that I don't know. I'm cheap, and I don't want to use something like Doppler. I don't even know what you cost. Maybe it's, like, ridiculously inexpensive. But let's just say that, you know, I I decide for whatever reason I wanna do this by myself. What what does that look like?
Valentino Stoll [00:41:44]:
Yeah.
Brian Valalunga [00:41:46]:
Well, first off, Doppler is super cheap or free, actually, to start. So it doesn't get cheaper than that. Yeah. It's free for the first three years users, and then it's basically GitHub priced from there. So if you if you're comfortable paying for GitHub, you should be comfortable paying for Doppler. But let's say you don't wanna do that for for whatever reason. I think the most important thing you could do if you're not gonna use any tooling at all is or bare bare minimum tooling is use, like, a cloud secrets provider, like AWS secrets manager, GCP secrets manager, and so on, and then build some tooling around it so that, you can fetch secrets locally in development without them touching disk. So what you need to do is have some wrapper, around your application that starts your application.
Brian Valalunga [00:42:32]:
So I'd imagine the workflow would be, like, fetch secrets from AWS secrets manager, inject them into the environment or write them to an encrypted file, and then have the decryption key in the environment, either path works. And then once the application starts, you clean up the environment, and then load all your other dependencies. And if you did that, plus having your a your secrets in AWS secrets manager and, that being, the same process happening for production workloads, at least you solve most of the risk in my mind. Also, when you're doing this, and this where things get a little bit more of a burden, because AWS, or managing privileges like I'm roles is quite complicated, in my opinion, is you're gonna wanna make sure that every individual developer and every set of infrastructure has their own identity. And that way, you have robust audit logs of, like, okay. The secret was taken by x developer at this time or read by this time, and we we can we have this auditor of what happened. So you don't wanna basically create one master I'm credential and just share it with your entire team because then you lose a lot of your auditing story along with it. And you also probably wanna set up, if you have a little bit of extra time to spend, an auditing story around reads.
Brian Valalunga [00:43:46]:
So, every time a secret is downloaded from this endpoint, you're gonna wanna make sure that you have the right logging and auditing set up in AWS for that, and that you're actually looking at those logs and have analysis of over those logs.
Charles Max Wood [00:43:59]:
Gotcha.
Brian Valalunga [00:44:00]:
Absolute bare minimum.
Charles Max Wood [00:44:02]:
One other thing that I've seen, and I think that I saw this on the Kamal, which is a deployment system that was written by, David Heinemeier Henson. Mhmm. And, anyway, it pull it can pull secrets from, like, 1 password.
Brian Valalunga [00:44:19]:
Yeah.
Charles Max Wood [00:44:19]:
You've seen people do using 1 password for their secrets management?
Brian Valalunga [00:44:24]:
Less and less. So they they, added something called 1 password secrets automation. Everything that I've heard from my community is that they are not actively investing in that product anymore, and it's kinda like shelfware now. So it's She's unaware
Charles Max Wood [00:44:40]:
of that.
Brian Valalunga [00:44:41]:
It's there. It's available. You can use it, but it's not actively being invested in. Right. It's on the shelf. But the amount of people every year that I hear using is less and less because, a, I think there's, like, 1st class products now that that really work. And the amount of things that you want out of secrets manager is growing far faster than what they're adding to, which is at this pace, from my understanding, it's 0. But you can't do it.
Brian Valalunga [00:45:06]:
I mean, like, Secrets Manage or One Password is end to end encrypted. So Mhmm. Like, you're getting a lot of security benefits from that. So I'm not gonna say, hey. Don't ever use it. If if it push came to shove, use it. I just think there's, like, far better things that are also easier to use and far better priced out there that are readily available.
Charles Max Wood [00:45:25]:
Valentino looks deep in thought.
Valentino Stoll [00:45:27]:
Is it is Doppler part of the GitHub, secret scanning part of the program? Yep. That's awesome.
Brian Valalunga [00:45:35]:
If you've if there is a Doppler token a Doppler issue token, found on GitHub, GitHub will contact us immediately. We'll immediately revoke the token, notify you, and work with you to generate a new token. And we're gonna be investing next year pretty aggressively in the secret scanning space as well of, like, really having a full auto picture there. Like, I'd love to be able to say, hey. We found some secrets in your code. We automatically revoked them, issue new ones, and those new secrets are now in Doppler, not in your code anymore. Like, some really powerful workflows like that can just make sure that even though the tooling is set up so that you'd wanna use it, if you actually don't use it and you fall back to bad habits, the the system catches you.
Valentino Stoll [00:46:15]:
Yeah. I'm curious on, like, as far as the observability goes, like, what are the latest, like, trends happening now for, like, observing, like, secret use, and, like, maybe detecting bad actors, you know, ahead of the breach?
Brian Valalunga [00:46:32]:
Detection ahead of the breach, at least in the secret space, is almost 0 from my understanding right now. It's there there isn't a class of tool yet for that. But on the scanning side, there there's TruffleHog, there's GitGuardian. GitHub is adding some stuff too. There is a space developing right now where, basically, you can scan the goal is to scan the perimeter of your infrastructure. Right? So, like, Slack, Microsoft Teams, email, your code base, all of that infrastructure, basically, will be scanned in real time for for secrets. That's basically the extent of what we have today. The problem is, like, you have to connect those dots.
Brian Valalunga [00:47:12]:
Right? So, like, you found a secret. What do you do next? Right? Especially if you're, like, an individual developer, you don't control the secrets manager that's being used. You don't control or if they're not using a secrets manager, what the flow is. And so, like, if you're an if you like, a classic example that I hear all the time is, oh, we added a secret scanner, and the secret scanner found all of our EMV files. But the company has decided we use EMV files. So, like, now it's a moot point because we know seek we've always known secrets were in code because they're in EMV files, but there's nothing to get us to the next step of, like, getting that of code. So now all you're doing is surfacing a problem that we can't solve. So I think that's really where, like, we're trying to help move, the industry is connecting the dots.
Brian Valalunga [00:47:50]:
You found a secret, and you can immediately take an action or an action is taken on your behalf based on a policy, to to clean up the mess.
Charles Max Wood [00:47:57]:
I mean, if you go back and listen to this and then go go through all my stuff.
Valentino Stoll [00:48:01]:
Run your checklist? Yeah. Do you have a checklist, Brian, of, like, you know, at least run through this for your projects or your team, or, do you publish anything around that?
Brian Valalunga [00:48:14]:
Yeah. We have a couple blog posts that that highlight, like, basic secrets management practices. Outside of the stuff I already shared today around, working with, like, as cloud, cloud provider, I would strongly recommend having a secret scanner. So truffle hog straight. Use it. It's open source. Or Get Guardian. Both are partners of ours.
Brian Valalunga [00:48:38]:
The other big thing is outside of secrets, I would have a vulnerability scanner, really important, on all your major dependency streams. So, like, Docker images, Ruby Gems, if you're using Node, NPM. But, basically, any dependency you have, you should have vulnerability scanning on it. Super freaking important. Because it's so easy these days for, like, one dependency that nobody's cared about for 10 years to just get bought for, like, couple $100, and then someone pushes an update for it. And all the other dependencies haven't frozen on that older version. And so now all these dependencies get that update, which means you get the update, and that update has malicious code in it. And that's, like, a very common, like, paradigm for attack these days.
Brian Valalunga [00:49:19]:
So, a, freeze your, your, your dependencies, meaning, like, I'm fixed at this version, and no amount of them pushing a new update will will change that I'm on this version. And this version has been, like, guaranteed good. No, no vulnerabilities in it. And that at least can help prevent any issues where, like, of, vulnerable code gets pushed up and you guys immediately get attacked from it. What you really want is you want all your dependencies frozen, and then, when a new exploit happens, you go, oh, that was kinda cool. We are using that that package, but our version doesn't have that problem, so we're good. And then you just wait for the fix to come out, and then you can do another refresh or another update.
Charles Max Wood [00:50:02]:
Cool. Anything else that you guys wanna dive into?
Brian Valalunga [00:50:05]:
So
Charles Max Wood [00:50:07]:
give us kind of the the, what, 2 minute pitch on on Doppler just just to let people know what it does and how it makes your life better.
Brian Valalunga [00:50:17]:
Yeah. So, at the end of the day, Doppler makes, your life better in a couple of ways. 1, it's gonna add 2 hours of productivity, per developer per week. So, in a way, it kinda pays for itself. On top of that, it's gonna make you far more secure from an organization standpoint, managing your secrets. You can get, you can sign up today, by creating accounts, for free. And then if you're a much, much larger org, you can talk to our solutions engineering team who can help, figure out how this integrates into your complex infrastructure. We have a whole host of great companies that are using us.
Brian Valalunga [00:50:49]:
So you're not alone from Accenture, Crumble Cookies, Puma, Children's Cancer Institute, StockX, travel, TripAdvisor. So there's a number of great companies. You won't be alone in this, and we manage about 30,000,000,000 secrets a month right now. So, like, that's quite large scale. And I I think the key thing here, if if you decide not to use Doppler, is at least manage your secrets. Like, at the end of the day, I think we are all tired of data breaches happening because they keep impacting us, and they impact your customers. And so if you're not gonna use Doppler, that's totally fine by me. You can even use a competitor.
Brian Valalunga [00:51:24]:
Just use a secrets manager. Protect those secrets.
Charles Max Wood [00:51:27]:
I'm a little gratified you mentioned Crumble Cookies there, local company. Anyway,
Valentino Stoll [00:51:32]:
let's go ahead
Charles Max Wood [00:51:33]:
and do our picks, and then we'll wrap up. Now I don't know if you had a chance to listen to other episodes of the show, but we just shout out about stuff that we're, playing, doing, or that we like. Valentino, do you wanna start us off?
Valentino Stoll [00:51:47]:
Sure. Yeah. I mean, the first thing that comes to mind is the Darknet Diaries podcast, which is just, like, an incredible resource of deep diving, exploits. And just like people are just so creative out there. And, nefarious or not, it's just, like, wild to see what, what kind of attack vectors there are and how people are using and exploding your behavior, which can be patterned. Right? So I'd recommend if you're interested in any of that kind of stuff, it's it's a awesome show. Next to that, I I've been following a lot of the rails world, fallout, which is just like, I definitely missed out.
Brian Valalunga [00:52:31]:
I don't wanted to go to
Valentino Stoll [00:52:32]:
speech conference. Yeah. There's so much, so much great stuff that I'm just watching from afar, come out of that, and, it's really exciting. So I guess just watch the YouTube channel for updates of, new talks that are coming out of there. And then the the last pick I have is Landon Gray. He's in in the Ruby AI space, and, he basically made Streamlit, for Ruby, which I'm really excited to, mess around with. And if you're not familiar with Streamlit, it's kind of like the, you know I don't know if it's JavaScript or it's some other language, but it's what a lot of the, you know, machine learning enthusiasts will use to just quickly get up a chat and, you know, fine tune or or use LLMs in creative ways. And Langchain has kind of, like, promoted it for a long time for just quickly iterating through, AI products development, and it's been really incredible, just, like, quickly get up and running if you on something new.
Valentino Stoll [00:53:39]:
And so I'm excited to see that come to Ruby, in a very, like, Ruby way. So it looks very much like a, you know, a rack app or or something like that where you can just run it, and it has a a chat that you can use. And, really excited to see that, develop more.
Charles Max Wood [00:53:57]:
Cool. I'm gonna throw in my picks. So the first one, I always do a board game pick. I'm gonna pick keep the heroes out. So the kind of the design of the game and stuff is kinda like a dungeon for, like, D and D except simplified, and, you're not playing the heroes. You're playing the monsters. And so, each player gets to play a different set of monsters, and they have different abilities. And so the way that you play is you take your turn by, you know, playing your cards.
Charles Max Wood [00:54:34]:
You have a deck of cards, and so it's a little bit of a deck building game. And, you can do all kinds of stuff. You can move. You can create artifacts. You can do stuff with the artifacts depending on which room you're in and what actions are available to you. And, anyway, you can also spawn new monsters, you know, again, depending on what actions you can take. And so then after you do all your stuff, then what happens is then you spawn heroes. And so, you draw the card, and it tells you what rooms to put what kinds of heroes in.
Charles Max Wood [00:55:10]:
The heroes do their special abilities, and then they'll either attack monsters or try to open the chest in the room. Or if if they can't do either of those two things, then they'll move one, step closer to the the vault. And so and the chests are numbered. So if it's a 1, then it takes 1 activated hero to open it. If it's a 2, it takes 2. You kinda get the idea. So, anyway, it that's essentially the game. There are a couple of other, you know, nuances to it, and, of course, it has a series of dungeons that you can play through.
Charles Max Wood [00:55:48]:
And so, you know, there are different rules for the different dungeons as far as how they work. The way you win is you exhaust the hero pile twice. And, yeah, the way you lose is if the heroes open the chest in the vault, which is the 4, chest. So, anyway, really fun game. Takes about, what, 45 minutes ish to play. The artwork on it is awesome. The different types of monsters are fun. And so, yeah, I've played it it plays 1 to 4 players.
Charles Max Wood [00:56:28]:
I've played it with 3 and 4 players. And, anyway, it's it's a fun game. Yeah. So I'm gonna pick that. And then one movie I've been wanting to watch for a while, mostly because I just heard it was really good, is hillbilly Elegy. Now I recognize that, this is the memoir by JD Vance, who's currently, Donald Trump's running mate. So politics aside, right, and I try not to get into it too much in on this show. Longtime listeners know where I stand on all this stuff, but I I don't wanna make the show about politics.
Charles Max Wood [00:57:08]:
But the the story is amazing. And, you know, the fact that, you know, this kid can come from, basically the kind of place and lifestyle that he did and then become you know, wind up going to Yale Law School and then, you know, become a US senator. I mean, I just find that incredibly inspiring. And, you know, at the end of the day then, you know, how you feel about him now, whatever, it it's really well done. It has Glenn Close and Amy Adams in it. I haven't read the book, so I I don't know how closely they stuck to it. But, anyway, there were just several, I think, life lessons that anybody can benefit from, from the standpoint of, hey. Look.
Charles Max Wood [00:57:56]:
If if you really want these things out of your life, then you can go have them if you're willing to do what it takes. And so, you know, from that standpoint, yeah, I I really liked the movie. There was something else I was gonna pick, and I just oh, yeah. It was the micro, film. And this one is on the Angel Studios app. I think it's called some something to stand for, and this is more it's almost a documentary. So if you've listened to the rest of the story by Paul Harvey, it's kind of based on that. That's also the premise of Mike's podcast, at least when he started it.
Charles Max Wood [00:58:39]:
Now he just interviews people that are interesting to him. But what he would do is he would tell the tell the story without telling you who it was about. Right? And so at the end, then he you know, there's the big reveal. And and so the the this person was, you know, Neil Armstrong or whatever. Right? And, you know, and so that's the rest of the story is kind of the idea. And so he tells a bunch of those from, US history, from moments that in some way shaped the direction of the country. And, anyway, I really, really enjoyed that too. So I'll put a link to that in the comments as well.
Charles Max Wood [00:59:15]:
But yeah. And then the last pick I have so Valentino mentioned Rails World, and the stuff that came out of it. I haven't watched any of the talks. I haven't really dived into what came out of it, but I've been using Kamal to deploy my stuff. And I've gotten really tired of fighting the traffic, reverse proxy load balancer. And so I wanted to upgrade to COMAL 2 so I could get, the COMAL proxy put in, and make my life a little simpler because it looked like it may it would make my life a little simpler. And so I did the upgrade, and I kinda screwed it up and then fixed it. So it took a little longer and a little more work than I thought it would.
Charles Max Wood [00:59:59]:
Essentially, what I messed up was, the directions are a little bit the documentation on Kamal is just not awesome. I'll just put it that way. But, anyway, what what it does is it, it tells you to upgrade by upgrading to Kamal 1 9, 1.9. There's 1.9.1 now, but I think it tells you to do 1.9.0. Anyway, because you wanna make sure that you can deploy with 1.9, and then you can upgrade to 2.1 and do the Kamal upgrade. And then it goes in and it stops traffic and it starts the Kamal proxy and it, you know, does the let's encrypt and all of that stuff for you, without having to know all of the arcane magic that you need to know in order to get traffic to do all that for you. And so I'm anyway, I'm pretty excited about it. It's it's been pretty awesome so far.
Charles Max Wood [01:00:56]:
So I I'm gonna pick that. I got a couple of things I still need to figure out on it because I wanna do multi tenancy and allow people to pick their own domains. And so then I want it to do the let's encrypt stuff. And so I'm trying to figure out if I can make Kamal do that for me or, Kamal proxy do that for me without having to have it in my configuration. But for now, it's it's deploying great, so I've been pretty happy with that. Brian, what are your picks?
Brian Valalunga [01:01:27]:
Yeah. Those are some good picks. Hard to compete with that. I think I a huge Star Wars fan, so I'll just shout out Star Wars Clone Wars. I feel like not a lot of people know about it, but it's between episode 23, and it's so freaking good. It's like I'm a guy that usually doesn't, like like, animated stuff, and it's just blown me away. So if you're a big Star Wars fan, there's a lot of seasons, and it's insanely good, and it's basically Dave Filonia at at his best. Outside of that, really big fan of the Perplexity app.
Brian Valalunga [01:01:58]:
It's, like, basically replaced any Google searches I do now. For anyone who doesn't know what Perplexity is, it's Google Google Search plus chat gbt. It's like so ask it a question, and it just gives you the knowledge back. It doesn't just give you a bunch of links. And it uses, and just like a Google search, it's updated in real time. So if something comes out yesterday or earlier today, they'll be informed on that. And it's just an incredible job at answering, like, really in-depth questions. Like, you can ask it very, very complicated questions that have a lot of nuance, that have a lot of requirements on real time data.
Brian Valalunga [01:02:32]:
Like, I don't know. What's the compare the market cap of NVIDIA to its revenue and then compare that to Microsoft's and figure out which one is a better stock to invest in? Like, you can ask that question. I'll give you a pretty concrete answer with, like, charts and graphs, included. So very impressed with, like I think that's a, like, a pretty big step up and the ability to, like, get knowledge very, very quickly. And then the other one is John Coogan. I think he does a great it's a YouTube channel that he that's out there, and he just does an incredible job of, like, talking about, I think, tricky topics. So, like, what's happening with Taiwan and chips or and or old, Prime Defense. Like, he just does a good job of telling the story of some event or some company really, really well, and I feel like it's extremely fact based and not, like, bias based.
Brian Valalunga [01:03:25]:
So I I generally go to him when I'm looking for, like, really high impact news, breakdowns. And then if you're also a Star Wars fan, I'll just end with this again because, again, huge Star Wars nerd here. There's a board game called, Star Wars rebellion. It's played with 2 players. 1 guy gets to play as the empire, the other as the rebellion, and you guys battle it off, and it's it's quite fun. It's a very asymmetric game, but I think that's what makes it fun because, like, Star Wars is very asymmetric. The Empire had a lot more resources than the rebels. But if you like Star Wars, both Clone Wars and that board game will do you a lot of good.
Charles Max Wood [01:04:04]:
Awesome. I don't think I've played this one. It's, it's pretty highly ranked on BoardGameGeek. It has a weight of 3.74, which means it's a pretty complicated game.
Brian Valalunga [01:04:15]:
Oh, yeah. It took us, like, 8 hours to figure out.
Charles Max Wood [01:04:19]:
Yeah. It says, 2 to 4 players, and the playtime is, like, 3 to 4 hours. And I'm assuming that's if you know how to play.
Brian Valalunga [01:04:28]:
Yeah. I'd say me and my brother, we play it took us 8 hours the first game to to get it. The second game, we actually did it across 2 days just because, like, we're we're kinda, like, very slow with every decision to be really methodical, and now we'll play, like, in an hour and a half. So, like, it it once you get comfortable, it ramps up pretty quickly.
Charles Max Wood [01:04:46]:
Yep. But yeah. So the the rank on it overall is 9. So it's the 9th, most popular game on BoardGameGeek.
Brian Valalunga [01:04:57]:
Hey. There we go.
Charles Max Wood [01:04:58]:
Yeah. And it's, it's the 5th thematic, which means that it has some, you know, some theme to it like Star Wars. So yeah. So, next time I'm at a board game convention, maybe I'll see if they have it and if I can play it. Anyway, I'm I'm a board game, not I really enjoy that stuff. So
Brian Valalunga [01:05:21]:
It's, a lot of fun.
Charles Max Wood [01:05:23]:
Yeah. So, Brian, if people wanna connect with you online, how do they find you?
Brian Valalunga [01:05:28]:
Brian@dopper.com for email. Vowel and good brian for LinkedIn and, Twitter. Cool.
Charles Max Wood [01:05:36]:
Alright. Well, thanks for coming.
Brian Valalunga [01:05:38]:
Yeah. Thank you.
Charles Max Wood [01:05:39]:
Alright. Till next time, folks. Max out.
Secrets Management Best Practices and Tools - RUBY 655
0:00
Playback Speed: