Embracing Human Factors and Rapid Development in Ruby on Rails Systems - RUBY 661
In today's episode, they dive deep into the fascinating intersections of system security, rapid development, and the human factors that influence them, with our esteemed guest Emil Kampp. Emil, a seasoned expert in critical infrastructure and fintech, shares his experiences and strategies for using Ruby on Rails to achieve unparalleled speed and robust testing in development.
Show Notes
In today's episode, they dive deep into the fascinating intersections of system security, rapid development, and the human factors that influence them, with our esteemed guest Emil Kampp. Emil, a seasoned expert in critical infrastructure and fintech, shares his experiences and strategies for using Ruby on Rails to achieve unparalleled speed and robust testing in development.
They explore the nuances of dynamic typing versus static programming, why Ruby on Rails is often the go-to for swift feature deployment, and the significance of stability in critical systems. Emil also sheds light on the complexities of ensuring robust voting processes and the challenges of maintaining security in banking systems. Additionally, we'll touch upon the importance of documentation, compliance, and visual tools in system design.
Join our hosts Charles, Ayush ,and Valentino as they navigate through Emil's diverse projects, from online voting to aerospace applications, and discuss how tools, testing practices, and redundancy can shape the future of secure and efficient development. Whether you're a seasoned developer or just starting, this episode promises valuable insights and thought-provoking discussions. Stay tuned!
Sponsors
Socials
Transcript
Charles Max Wood [00:00:04]:
Hey, folks. Welcome back to another Ruby Roads. This week on our panel, we have Ayush Nwatiya.
Charles Max Wood [00:00:11]:
Hello. Hello.
Charles Max Wood [00:00:13]:
We also have Valentino Stohl.
Valentino Stoll [00:00:15]:
Hey now.
Charles Max Wood [00:00:17]:
I'm Charles Max Wood from Topten Devs. And this week, we have a special guest, and that's Emil Komp.
Emil Kampp [00:00:24]:
Hey, everybody.
Charles Max Wood [00:00:25]:
So, Emil, I'm just gonna clear my throat because wow. Okay. Let me try that again. So, Emile, I found you. I've been, corresponding with people some on the Discord for Kamal, which is the deployment technology that, DHH, put out. I think he made kind of a deal about it at, rails world last year and then a whole bunch of exciting stuff this year with, Kamal proxy and stuff like that. I mean, I switched over to and I switched over from Kamal to Kamal too with a whole bunch of help from you and others on there. And, I have to say, switching over was a little rough.
Charles Max Wood [00:01:10]:
Having switched over, it's a lot nicer. So, anyway, so beyond kinda what I know about you from that, what what should people know about you?
Emil Kampp [00:01:23]:
Well, so I think, first off, I am a mountaineer. And in order to facilitate that, I live in a very flat country, so I do a lot of running. And then I need to pay for my travels, so I do also engineering. And I've made a career of, doing sort of critical infrastructure or things that have real world impact and real world con consequences if something goes wrong. Right? So both in terms of security and robustness and things like that. And that's everything from the Fintech or the payment industry to civil engineering, to, online voting.
Charles Max Wood [00:02:15]:
Wow. Yeah. It sounds like kinda what our topic is here, which is building critical infrastructure. So, you've done some stuff that I kinda need to work,
Emil Kampp [00:02:26]:
I guess. But, like, what are we putting in? I mean, so banking obviously needs to work. Right? People get sort of pissed if they don't get their money. Yeah. And then, for civil engineering, I worked on, tools that the engineers would use to build the airplanes, which and and run tests on them. Right? So both test benches and design benches for airplanes. And, I mean, that has a real world consequence because they fall out of the sky, right, if you mess it up. Yeah.
Emil Kampp [00:03:02]:
That's good. And then, yeah, online voting, that's a whole thing. And, you guys in the US just went through one, one of these messy events, and they're messy everywhere. Right? Oh, yeah. And recently, I moved to to compliance and sort of making trying to make it trying to make that easier and a better experience to become compliant and to obtain, certifications.
Charles Max Wood [00:03:36]:
Right. Makes sense. So, let's just dive in. I mean, I'm not I'm trying to think if I've ever worked on anything that was, you know, kind of at the level of, hey. If this breaks, then, you know, really bad things happen. I mean, if it breaks, I've I've I've had them come to me and be upset like really bad thing happened, but right. It it doesn't affect society at large. What's the difference between working on those apps and working on just kinda your, you know, you mentioned not ecommerce or not, you know, hey.
Charles Max Wood [00:04:13]:
I've gotta manage my social media or, you know, get podcasts out on time.
Emil Kampp [00:04:18]:
Yeah. I mean, there's nothing just to be clear, there's absolutely nothing wrong with that. Right? That's just I I I started out in in that area as many developers do. Right? Building, like, some app system when I was very young and then moved to some, you know, sort of offer platform, and then I sort of realized that it was it was just inconsequential in a way. Right? I looked at people. You know, I had people around me. My my father was a is a civil engineer. Right? And so I have people in my, sort of, my family, my near family that was building things in the real world.
Emil Kampp [00:04:58]:
And I was like, I wanna do that, but I do like this computer stuff. So can we sort of maybe find a way where I can build things in the real world just using my computer? And so I slowly started to shift over into something that was founded more in in civil engineering and, sort of industry rather than commerce. Mhmm. And then I started in a Fintech company, where we were where I was part of building building out and continuing to build on a on a system where you can basically it's like direct debits, bank transfers, but worldwide. Okay. So and and there, it it doesn't necessarily mean a lot to you and me, and it's the same with there's a bunch of them now. Right? Like, these send money home, types things. It doesn't mean a lot to you and me, but it does mean a ton to those who is on the receiving end of that.
Emil Kampp [00:06:06]:
Right? Right. Because that's, like, half a year's pay or something that comes in. Right? So it does matter to those people. So so, yeah, that was sort of where I ended up. And we are moving a lot of money. Like, it it it seems it's one of those where little like, it's it's a lot of tiny things that add up. Right? There's, like, 1,000,000 and 1,000,000 of dollars being moved around for that purpose. And you don't wanna mishandle those money.
Emil Kampp [00:06:38]:
Right? So you are hooking into the to the banking infrastructure, into Swift, and into the IBAN network and the whole sort of, PC compliance and everything with that. Right? So you need to make sure that the software is robust, well tested. Like, testing has been a main thing of of mine for a long time because of that, and also documented well. Right? So that's where I sort of started out with being required to document what I was doing and how it was supposed to work, rather than just shipping fast and breaking things. I was required to test and document, then test, then stage it, then test some more, then ship and not break things. So that was a different paradigm.
Valentino Stoll [00:07:33]:
Mhmm.
Emil Kampp [00:07:35]:
Yeah. So that's sort of where I I first met that. Right? And that was a a pretty great experience. And so I was like, oh, can I have more of this? Can I maybe even get even closer? Right? Yeah. So that's, that's where all that started. The and then I was hooked on the whole, can we test it? Can we build something that runs Ruby on Rails but makes a difference to, to people's lives, like, in real world. Right? So this was basically a a payment, situation just building Ruby on Rails. And back then, many of the banks didn't have a proper API, so we ended up scraping a lot, and that's, you know, against terms of policy, in terms of service, and all of that, and they were really mad.
Emil Kampp [00:08:24]:
And we went through that whole shebang of getting, you know, sort of, scolded, for for that for a while. Yeah. So they it was a bit the Wild West back then. But, it's definitely gotten better, and I think there's a lot of, again, sort of compliance work and legal work that went into, the EU, did a lot to make sure that banks would have not standardized APIs, but a minimum level of information that you could retrieve with an API, which made all of that a lot easier. Right? It also means that I can technically build my own banking app for my own bank account because I have API access to it. It may not be obvious. It may be poorly documented, but it is there. So I think that's pretty cool.
Charles Max Wood [00:09:19]:
Yeah. I'm curious if Valentino or Ayush have experience doing this stuff.
Charles Max Wood [00:09:28]:
Nothing that critical. Now the closest I kinda came to anywhere near that world was when I worked for TransferWise, but I was an iOS developer, so I didn't really have to do any of the hard stuff. I just made API calls, but I know they were also doing quite a lot of the scraping and stuff. There's a lot of selenium going on behind the head to get into different bank accounts. It's, yeah, it's, it's all a bit, scary when you think about it.
Valentino Stoll [00:09:56]:
Mhmm.
Emil Kampp [00:09:57]:
Yeah. Yeah. It really is because, like, your whole banging experience is tied up on some scraper working correctly and not messing around with the, the 1,000 delimiters and the, the sent commas are different on different regions. Right? So you don't wanna mess around with that. I did transfer some money incorrectly. Luckily, the the backbone banking system is is pretty robust as well, so we were able to basically not move the money anyways. Right? But it, I did move far too many money, the wrong place.
Charles Max Wood [00:10:33]:
I won't be in there.
Emil Kampp [00:10:36]:
Yeah. That was a that
Charles Max Wood [00:10:37]:
was Oh, never mind.
Charles Max Wood [00:10:38]:
That
Emil Kampp [00:10:39]:
was a backbreaking day.
Charles Max Wood [00:10:41]:
I'll bet. So I'm a little curious as we get into this. Like, you mentioned testing. What what is your what is your testing setup look like? Because I mean, I I'm I'm sorry. When I write my test, it's like, okay. It's good. Like, I'm I'm reasonably confident. Right? But I'm not, like, ironclad confident, like, banking level confident that my code works right.
Emil Kampp [00:11:07]:
Yeah. Yeah. That's a good question. I tend to not be too dogmatic. I know there's a lot of, like, do this or do that paradigm or do test first or whatever. I I tend to just play around with it and make myself certain first that this is gonna work, whatever I'm sitting with. And then when I know that it you know, I'm sort of I can see, like, okay. It works.
Emil Kampp [00:11:37]:
It it does what I need it to do, then it is worth testing. Right? And then it's I try and test on the lowest possible level first, which means I will start with something like unit tests. And for that, I I wanna permute everything. Right? You wanna make sure that every argument so you basically start with the least amount of arguments for a method, and then you add an argument slowly in I use Aspect, so it's in every context block or in every describe block. Right? So you're you're certain that every version of every argument is, is tested and and has been permuted. Yeah. So that everything like, every variable has been tested on its own. So you know how it behaves both with and without that particular variable.
Emil Kampp [00:12:36]:
And you also know how it behaves if the variable has, like, the wrong type. Right? So because Ruby isn't quite where we have type safety yet, there's a lot of, work going on with that, but we aren't there yet. So I you know, you wanna make sure that we can handle strings, numbers, not strings, trues, booleans, whatever. There's like I have a little thing that basically says, take this and just, you know, cycle it through all of the different types it can be and just put in garbage so that I know it cannot it doesn't break when a number is presented rather than a string or something stupid like that. Right? And then you sort of you slowly move up. So you're confident that this method works, and then you move up to have recorded, and then you I what I do is I then I go through and test that. But I think opposed to what the s DHH, preaches or or at least sort of, he he has this idea that you go through the entire stack, and I believe that they their tests do that. I tend to not I tend to sort of stop testing at each level.
Emil Kampp [00:14:02]:
Right? So model tests are their own thing. And then when you test the controller, you basically verify that the model is being called with the expected parameters because you know you've already dealt with the parameters on a lower level. So it's not mocking per se. It's just that I am I I only need to test that the model was created or was called. I don't really need to test that it was the correct arguments. Right? Because I'm certain about the correctness of the arguments on a lower level than that. And then you saw slowly move up, which means that I I find that that means that the individual testing levels, it gives you a little bit more freedom to sort of move a model around because it's it's coupled somewhat on a test level, but not tremendously. Right? I only coupled the controller to the model in the sense that I checked that the model was called, not that it was called with a particular set of arguments.
Emil Kampp [00:15:06]:
So it makes it a little bit easier to also restructure your tests when your code change. But, yeah, it it's just a lot of work on lower level things. I'm a big fan of Sorbet, but, it's not it's not quite there yet. I think it it's it works reasonably well, but it's very hard for developers to use. So I think we need to do there there needs to be a lot of work done in in terms of making it nice and sexy to use so that we can get people to use it. Right?
Valentino Stoll [00:15:40]:
I'd I'd love to dig in a little bit more on the testability aspect. Because one of the biggest things I see, miss is maybe, like, skipping over, like, the maybe some lower level implementations and then, like, a system update breaks something, and it just wasn't covered. Right? Like, how do you go about, like, you know, weighing whether or not that is worth it to, like, actually run, like, a system command or an external API call or something like that?
Emil Kampp [00:16:13]:
Yeah. So, system commands, I usually check just that the signature of the call is correct, and probably that it returned the right thing. Right? So that if I hit the OS, I'm I'm basically testing that I gave it these parameters, and I expected something back, like a string or whatever it is. Right? Because it's so hard to know what it was supposed to be because sometimes the system commands will change their their signature, but not necessarily return value or the return value, but not the signature. So that's that's very hard. I try to not go go there. I try really hard not to go there because it is it's out of my control. Right? On the other end, when I call other APIs, I definitely test that.
Emil Kampp [00:17:11]:
I use something like VCR or something. I capture all the network traffic, and I make sure to do the same kinds of, sort of making sure that if I call it with these arguments that I have control over, I get this particular response. And if I do something else, then then it breaks, right, or it gives me the incorrect response. And I also found that, for example, with, with banking, that one of the cool things we did, was that we basically took all of the historical records, right, that we had for all the bank accounts, and we basically used it as a test bed because then you can train on that. You can basically say, if I feed it all of this information that we know happened and it happened correctly because it it happened, you know, like, a year ago or 2 or whatever, it should result in the in in a particular set of of of, like, it should result in a particular value on the account. Right? So you can actually use historical data as a test bed. And that may that means that you have a factual way to check if it's correct or not. So, yeah, I I tend to use that as well.
Emil Kampp [00:18:35]:
I also use that when when I did software for the for the for the sort of civil engineering and building flight and testing that. Right? We also used a lot of existing information to make sure that we had input output. So we understood that given these inputs, this is the output that we would expect. Right? We could basically take old designs and try and run them through, and you can see if does does it break it down and you'd like you would expect? Does it run the does it sort of fit to the engineering standard design standard that we were implementing against? They also have test value that you can use, right, or test data. So it's also about going out and finding big test datasets that you can use to also make a statistical sort of approach. Right? I'm not making any statistical claims, but I am running a lot of tests on a lot of data that, you know, you can be reasonably certain that, that if if there was errors in your system, it would have been caught by this because it's so much effective effective production data. Right?
Valentino Stoll [00:19:54]:
Yeah. I imagine that working with, like, the public sector that you would have a lot of access to that data. Right? I guess depending where you live.
Emil Kampp [00:20:05]:
Yeah. I mean I mean, interestingly enough, none of this is for the public, sector, which is a little bit weird. You would think that it was, but it actually isn't like app the regulation of of the aerospace is public sector. Right? But the actual building of airplanes is a private commerce. Right? Private industry. And likewise, with banking, the the regulation of it is public, but the actual banks are private entities. And likewise with
Valentino Stoll [00:20:36]:
find it difficult for, like, the data access? Like, is that data proprietary then for, like, doing large swaths of tests or finding it?
Emil Kampp [00:20:46]:
Yes. But most of the times, the companies are interested in providing it. Right? Like, if you if you go on to big, to big airplane or an aerospace company and and you're you need to design a test system for them or, something to help them build. Right? They are interested in giving you that kinds of data. So it wasn't that it was hard. It's just that it is also quite, narrow in scope. Like, it's it's more more or less only for that one. Right? So in in banking, it would be a lot of, records, but only for one bank.
Emil Kampp [00:21:23]:
So you would only know how it looked in one type of account system. Right? And then you would run all your tests on that. And now 2 seconds. Alright. Yeah. There we go. Hello.
Charles Max Wood [00:21:48]:
Hi.
Emil Kampp [00:21:50]:
So so it it's not that it's hard to get. It's that it is it's quite narrow, in terms of, sort of, like, what yeah. It doesn't have a lot of breadth. You get it for, like, one bank or one airplane company or, you get it for one company who runs elections. Right? Public elections is a bit different. By definition, you don't have any test data because all of the records are sort of you don't know who who voted, so you can't know Mhmm. How to test that. Right? So you have to do a different testing paradigm there.
Charles Max Wood [00:22:30]:
Yeah. That makes sense. I mean, here in the US, we have a a anonymous ballot as well. Right? Yeah. And I I don't know how they do it in other states. I mean, I'm I'm involved in the process here in Utah. And, yeah, the you send in your envelope and they with your ballot in it, and they validate that your, like, your signature matches and stuff off the envelope. But then they pull your ballot out of the envelope and put it in the pile to be counted.
Charles Max Wood [00:22:58]:
And Yeah. Right? They they don't they track who you voted for in that case or at least they're not supposed to.
Emil Kampp [00:23:04]:
Right? Correct. And and that is also the case for the digital solutions.
Valentino Stoll [00:23:08]:
Right.
Emil Kampp [00:23:09]:
A friend of, a colleague of mine, Stefan, who's a cryptographer, he, a previous colleague of mine, he did a lot of work of sort of, gathering together a lot of the academic work so that we could exactly do that in a cryptographically safe way, but online.
Valentino Stoll [00:23:30]:
Mhmm.
Emil Kampp [00:23:31]:
And you could also do it in a way that it was you know, you could track it, and you could ensure that this was in fact the case. Like, you could give the the log data and the trace data to mathematicians, and they could verify that mathematically, it could only ever have worked if these rules were followed, which was like, you know, you anonymize and you did you did mixing and you did a lot of these things. Right?
Charles Max Wood [00:23:57]:
Right. So you protect the identity of the voter, but you also ensure that only people who are legally permitted to vote vote and that they only cast one ballot and things like that.
Emil Kampp [00:24:09]:
Yeah. Yeah. Exactly. And the test the test, sort of the the build idea for that and the testing regime for that is is significantly different. Right? Because this is in fact the public space to to your point earlier, Valentino. This is a public, domain. Right? So you you are a bit more scrutinized. You're also publicly scrutinized by everybody.
Emil Kampp [00:24:31]:
Right? Because we all know how it went with the domain Dominion Voting Machine case. Right? That went real fast, real downhill. So Yeah.
Charles Max Wood [00:24:45]:
Well and you bring up the Dominion, voting machine case. And whether you believe that they worked or didn't, there's if there's enough population that doesn't trust it, then it causes problems. Right?
Emil Kampp [00:24:59]:
Yep. And that's that's the same everywhere. Right? Yep. And that's I found that sort of on a on a completely different but somehow related topic, I really found that the aerospace or the airplane industry was was really good because the, what's the American one, the TSB, Transportation Safety Board. Is that NTSB. NTSB. NTSB. Right? Yeah.
Emil Kampp [00:25:27]:
Those are the ones that that do crash audit that sort of, like, go through
Charles Max Wood [00:25:32]:
Oh, yeah. The NTSB. Yeah.
Emil Kampp [00:25:35]:
Both them and the European counterparts have a really, really cool way of going about it. They don't ever go and say, oh, it was that person's fault. Right? If if the person hadn't slept enough, if the person couldn't read the dials, it is the fault of their plan that that made him not have enough hours to sleep, and it is the fault of the dial for not being visible enough. Okay. Right? So it it's not they don't ever cast personal blame or they very rarely cast personal blame. And I really thought that was a super cool way of going about it. Also, just from a sort of, like, evaluating, if something works well. Right? Just, like, from a human perspective and when you do leadership in tech or if you, guide or lead others that I found that really valuable, sort of that whole way of thinking about it.
Emil Kampp [00:26:37]:
And they also they collect a ton of evidence. Right? They have a very structured way of collecting that before they then conclude that it works or didn't work or that this was the fault or that was the fault. But they also I I think I learned a lot also in terms of, like, having redundancies. Like, what happens if this thing breaks? What happens if your job runner doesn't run? How are you alerted? What happens if your email doesn't send? How are you alerted? How do you have alternative options? Do you have an alternative email provider that you can switch to easily? Do you have alternate job runners that you can switch to easily? Do you have any of this? Right? So that redundancy ID ideologies was pretty cool. And I'm for, like, large scale systems, it it pays off to have extras of these even though, you know, you you use somebody like SendGrid or you use somebody like Mailchimp or something, and, you know, they say that they scale and they do. Right? But then ever so often, it's down, and it's pretty nice to have a backup. Even if it's a tiny backup and even if it's just, you know, up and coming fairly cheap one, it will do the job for, like, a day. Right? And you will actually still send stuff out or you will still process jobs or you will still whatever it is.
Emil Kampp [00:27:56]:
Right? Also running, like, on the big scale, running something like multi cloud or running something like an open stack where you can actually move to somewhere else pretty fast. There was that case with that the Australian case where the where the bank got taken down, because the someone at Google decided to delete their account, right, for, like, a 160,000,000 US worth of servers, just got nuked. So for those cases, it's pretty nice to have a second provider backup, and some way of getting there. Right? That is more a DevOps problem. It's not really a problem that I've ever solved. It's a problem that I have, you know, sort of, like, assisted with and talked about and verbalized a lot in companies. But it is to high degree a DevOps solution that needs to go in there and not one that I am gonna build.
Valentino Stoll [00:29:01]:
So you're talking a lot about, like, people's lives and, you know, their livelihood, their money, like, things that sustain life. Right? And so, like, obviously, these infrastructures are critical. Like, what tools you're using are critical. So I'm curious, like, what about, like, Ruby and Rails, you know, specifically feature wise? Like, do you see as being like, oh, this is, like, part of that critical process?
Emil Kampp [00:29:30]:
I think the the main part for me is that it can be built, you can write it so that it is absolutely readable and understandable what's going on. So you can reason about the what's going to happen even without necessarily understanding the language fully or even without understanding what this active record method is going to do underneath because stuff is named well and because it's ordered and organized well. You can reason about what's gonna happen, and you can have an expectation, and it makes it a lot easier to build complicated things. Like, you can go and test it afterwards. Right? But just getting something that's complicated up, like, just getting it up on its legs, it's really nice to have something that that is structured well and that you can reason about and that you talk about, sort of what's the consequence of this method or that method. Right?
Valentino Stoll [00:30:34]:
Yeah.
Emil Kampp [00:30:37]:
There you go. So that's that's cohost. Yeah. Exactly. But I think and, also, I find that it's I mean, I've I've done a few frameworks throughout my career. Right? It I've always come back to to Rails and Ruby, and I've I also started there. So maybe I'm biased, but I I I haven't found any of the other frameworks that provides the same sort of consistent experience through every part of the application. Whether or not you're doing low level active record stuff or you're doing high level system tests or whatever you're doing, it's consistently the same experience everywhere, which means that it's fairly easy for a developer to switch gears and do something else, right, in in the stack, I mean.
Emil Kampp [00:31:34]:
So, like, you can have a person who is sitting and doing the swift transfer, 1 week. Right? And then he can go and he can look at the web scraper the next because it's consistently the same experience, and you can write it in a way. If you have a good sort of if you have a good style guide and you have good code review, you can get it to a point where it's the same experience every time. And you can get abstraction levels and abstraction layers that feels the same. And that makes it easy to build something that where you can also easily plug out something and put something else in. Right? Like, you can switch out one database provider for the other without much,
Charles Max Wood [00:32:20]:
blocking.
Emil Kampp [00:32:21]:
Right? Or you can do whatever whatever it is that you feel like you need to change is can be quite easy and quite straightforward if you've set it up, to be and I'm not necessarily talking about modularity in the site work, level of, aggressiveness that, that Shopify has. Right? But you can you can definitely do something along those lines with just talking about it, organizing the organizing the code well, and, and just sort of thinking about how how is this gonna be tested? I I find that that's also part of it. Right? It's like, how are we going to test this? Because if we are going to if we're going to test it in this particular way or that particular way, then that that informs the the API and the sort of the interface for it. And a database, you tend to test the same way, You know, if it's file uploading or downloading, you tend to to test that the same way, which makes it easy to switch out one component for the other if your test paradigm is the same, right, for for that component. So I feel that's that's some of the reasons that I've chosen this, again and again. Right? And, also, like, DJ said, on his keynote, This is a one like, it can be a one man army. Right? You can be pretty fast and pretty, dangerous with it, which means you actually need fairly low head count to get somewhere really fast. And you can actually get something that's really secure, solid, safe, robust, and easy on the eyes and nice to use with only a handful of people.
Emil Kampp [00:34:23]:
Right? No. You can get somewhere with just one person, but you can definitely get a long way with a handful of people.
Charles Max Wood [00:34:32]:
You are so speaking my language, man. This is why I'm here. Yeah. So one thing that I'm wondering about with the all of this is, you know, it sounds like depending on what the concerns are in the app. Right? So whether it's banking or voting or right? You you may have different concerns about, uptime or stability or, you know, resumability, you know, if if Google deletes all 1800 of your servers or, you know, whatever. Right? So how do you begin to start looking at your application and saying, well, you know, for example, in banking. Right? We have to make sure that we don't have any of these kinds of issues, bugs show up. Right? You know, make sure our accounting is super robust.
Charles Max Wood [00:35:20]:
But then you're talking about the voting and it's, hey. We gotta protect user data and make sure that right. So the the the areas of concern are different. So how do you start to address that? And then from there, where do you start to dive in and make sure that you've got a sound strategy for making sure that you're meeting those requirements?
Emil Kampp [00:35:38]:
Yeah. That's a good question. So for let let's just take voting because it's, up and hot and a topic these days. Right? So let's just start there. Always. So one of the one of the things there is that you definitely want so, like, use some privacy. There's a few right there. It's, like, the the whole use of privacy, and then there's these Vogue secrecy, and then there is the use use it needs to be receipt free and, like, there's a bunch of these, academic properties of a vote and an election.
Emil Kampp [00:36:16]:
Right? And you go through each of those and you you effect you just sort of sit down and you start at 1 and then you try and figure out how to do that. Right? This is really the job of the cryptographer. Like, how do we get this organized so that we match all of these? And then once we have a viable, protocol that will do this, then the developers come in and we look at the protocol and we go like, okay. This thing is about encrypting the vote. That needs to happen as early as we can possibly get it to happen, which means it needs to happen in the voter's device. Right? So we need to generate, we need to pick a pick a a curve, a crypt a cryptography curve, and then we need to basically generate keypads in the device, and we need to encrypt it as early as possible. Right? And so you go you go through sort of, like, rigorously and slowly and just sort of reason about when will this part of the process happen and where does it need to happen, sort of more on a timeline. Then we need to transmit the vote.
Emil Kampp [00:37:30]:
We transmit the vote to the server, and then the server needs to basically allow if you need if you need to allow tracking or if you need to allow receipts or whatever is that property you need to have, then you need to figure out a way to provide that at this point. And then you go on to something like anonymization, which is effectively where you take one envelope out of the other envelope, right, to do the the analogy. That needs to happen in a way that you can prove that it happened. So you need to have a good trace here. You need to have good logs and traces that you can publicly show. And then you know? So you sort of move through the chain sort of like the the event of an of a thing. Right? It's the same. And so you you sort of need to reason about where is it gonna happen.
Emil Kampp [00:38:30]:
That gives you some ideas of should we need do we need a web app? Do we need a mobile device, mobile app? Do we need do we need to run cryptography? Do we need to run something advanced on the user's device, or is the user's device just a portal? And and that gives you sort of ideas on or some insight into how this should be organized, how this should be tested, how this should be planned out. Right? But most importantly, for for voting is is basically making sure that everything is tracked, and, we grabbed part of the concept of blockchain, which is hash chain, right, to basically hash everything together so that you can't move stuff in the middle without everybody being able to know. Right? Things like that. For banking, right, you wanna you wanna make sure that see, the the point is to get money from a to b. So you wanna reason about well, first off, how am I gonna get the number that they intend to send? Right? Like, where does that come from? Where does the where does the sort of outgoing amount of money come from? I know. This is the second one.
Charles Max Wood [00:40:15]:
How many
Emil Kampp [00:40:17]:
It's it's hard to be home alone, with, with kids. But so you wanna basically say, okay. So where does the money come from? And then you wanna reason them out. Well, how can I prove that the money that came from there that that I got it right? Like, how can I make sure that the intent was to send it to me? How can I make sure that the amount was correct? Right? Because, like, like we talked about earlier with TransferWise, for example, there is a lot of web scraping. There is a lot of this, sort of craziness going around. Less so now, more so then. Right? And you wanna make sure that you have either done it a few times that you're quite confident that you've run it on big datasets. Like, you need to be certain that you can pass these things correctly.
Emil Kampp [00:41:10]:
And then you wanna sort of grab that information, and then we get into the whole banking of how do you then move them. There's, like, Swift, and that's pretty taken care of. The only problem here was that it was slow. Right? So what we would do is that we would basically assume that if we could read the intended money here, we could effectively pay it out here immediately and then just do the in between transfer later. So we would basically have big stacks of money around the world, and we could immediately pay out locally, or take money in locally, and then we would, transfer later. So in that case, it was more a question of, can you verify the amount here, and can you verify that it's the sort of the correct amount and the correct intended amount? And can you then make that payout over here? Okay. It's common. So then can you do the payout here? Is there sufficient funds? And then after that, you sort of say, okay.
Emil Kampp [00:42:19]:
Now we've we've effectively solved the user journey. Right? We've taken money in. We've paid money out. We're effectively, the user is happy or both users are happy. Right? Nobody knows that we didn't actually move the money. And then we can take, you know, take it slow. Just do it properly. Make sure that we interact with the swift network.
Emil Kampp [00:42:39]:
That's a bit slower. So you need to have a a buffer in, in both ends to do it. Right? And the English is more simple. English. And the the most the important part here is the ability to scrape. Right? So one of the critical points here was that you have to be able to always scrape these websites, in some capacity, and you have to actually write your your code. Back then, it was, you know, scraping. You have to write it quite robustly and sort of reason about what happens if the layout changes because banks change the layout surprisingly more than you would think.
Emil Kampp [00:43:25]:
Mhmm. Because they don't care about your ability to scrape it. That's not their business. Right? Which is fair. It just means that you have to be you have to think about the robustness of of a scraper like that because what happens if the if the item is not classified the same if the dom element doesn't have the same class or the same ideas it used to have. Can we find it by reasoning with words? Like, can we figure out well, if it's like, if it says some, can we maybe then figure out what's over here next to it? Because that might be the number I'm looking for. Right? So there's sort of, like, reasoning like that that that goes on. That's yeah.
Emil Kampp [00:44:07]:
And then there's accounting. Like, a lot of accounting, a lot of keeping tracks just for legal and compliance reasons on where is the money going, who's it going to, KYC, all of these. There's a bunch of tooling after that, right, Which is which is sort of where I was introduced to this whole compliance thing. Like, how do you make sure that the people you're operating with are in fact the people that they say they are and are in fact the right people that you're supposed to be operating with. Right? So that was also a journey. Like, how do you even know that? How do you make tooling for them to identify themselves at the latest possible time so it's the least amount of sort of burden or the least amount of friction, and make it as easy as possible. Right? There's a bunch of providers out there that that can help you do that. You can also do it yourself.
Emil Kampp [00:45:02]:
It's not super complicated, but you are now dealing with people's private information because now you maybe have a copy of their passport. How do you deal with a copy of a passport? You have to be you have to sort of be like, okay. Where do I store this? How do I make sure that it's encrypted? How do I make sure that none of the none of the developers can access this? Right? Because you don't want developers to just SSH onto the box and just be like, oh, let me just download all the attachments. So you need to be you need to make reasons, you know, sort of need to think about how do you then protect this data from yourself in some way. Right? Yeah. So that it it goes deep and it goes far. Right? You can talk about this for a long time, and about all the reasonings you have to make. But at some point, you also have to protect your systems from other developers, from other, also high level people in the organization needs to be kept out.
Emil Kampp [00:46:01]:
Because the higher up you are, the more the spearfishing you are or whaling you are effectively the subject of, and the greater the risk. Right? So you want people to not have access. Maybe you even want the CTO to not have access. Like, there's a bunch of considerations like this. In in the banking one, I was a senior developer. I had access. Then we had the one of the compliance officers, she had access to to the swift network. And then, but but the CTO and the head of sort of the the COO type person didn't.
Emil Kampp [00:46:46]:
Right? Because that that would it post too much of a risk. So there's also, like, these human factors you have to think about, when you reason about this the security of a system that that actually impacts, and it has something to do for real world people. That was a very long spiel. Sorry.
Charles Max Wood [00:47:08]:
Yeah. It's fascinating when there's so much on the line. Like, most of the apps I've worked on, like, if it went down, it's, like, yeah, fine. Okay. It's not ideal, but it's not the end of the world. I kinda wanna pick up on something, you mentioned quite a while ago, actually, on on about type checking and Sorby and stuff like that. Like, I'm very much a fan of the dynamic typing. I I I do have a background in, static programming.
Charles Max Wood [00:47:38]:
Like, because I I started as a mobile developer. So I spent a lot of time with Objective C, with Java, and with Swift. And so I do have that background, but ever since I kinda switched to Ruby, I just love that dynamic style, and I just find it easier to work with. But I also get that when it's critical infrastructure, the type checking is just, almost essential. So I'm curious about why you went with Ruby and Rails when, the the infrastructure is so critical and type checking is clearly something that's important to you, and it's kinda bolted onto the language. So what what motivated you to go with Ruby on Rails instead of something that has it natively?
Emil Kampp [00:48:22]:
Yeah. So it is it is, to a high degree about the speed and the very well built testing frameworks that is around it. Right? So even if it's not statically typed, you can test a lot of things, and you can test it in a lot of ways. Typing is just or types is just one way of ensuring that the data is correct. That doesn't necessarily guarantee that the actual data you are trying to read in is of the correct format or whatever. It just means that once you read it in, it will fail because it's not the right type. So it depends a little on what you're trying to do. You can also because it's dynamic, you can then sort of do things like, okay.
Emil Kampp [00:49:08]:
You gave you told me it was a CSV, and it's obviously not a CSV. And it obviously doesn't have a string where you told me a string was supposed to be. But I can see how this can look like a string. Maybe I can just make it into a string for you and move on. Right? Like, you have these options of making things go a little bit fast and a little bit smoother, because you can sort of just fiddle a little around with it. What tends to happen in these systems as they grow older and as they mature is that parts of the core of the system gets replaced with something like Java or Rust or something like that. So that once you have sort of you have a really solid part, let's say, the transfer engine, right, that transfers from one account to the other that that hooks onto the swift network. Maybe you wanna move that into Java or something that's more static, more slow, but more type robust, and more maybe even performance optimized for IO or whatever it is that you need.
Emil Kampp [00:50:14]:
Right? But then if what tends to happen is that you still want the speed and ease of using Ruby on Rails and having only a few developers do a lot of features so you can figure out what's good or bad or what works or doesn't, or do a lot of development around trying out this new standard to verify with the engineers that it will in fact test correctly with this airplane or, you know, sort of having that that capabilities to do feature spikes and to do tests or, like, mini projects. I it seems to me that that is to some degree lost when you use a more as, well, some other language. Right? But but you are right. Like, once you have it down, once you have it locked in, once you don't have any changes there, what tends to happen is it gets replaced with something that's faster or better for suited for the purpose. Yeah. That I've seen that happen. So the reasoning is that you can go really fast with quite few people, and you can actually get something that's meaningful that will affect people's lives, that will better somebody's life, you can get it done pretty quickly. Right? Like, you can get a a design system for to to basically do, design some we're not talking, like, free sketches of of airplanes.
Emil Kampp [00:51:49]:
That's that's not necessarily how you design all of it. That's just the building of it, but you also need to reason about the logic of who does who does what, what are the processes. And you can get something like that built in 7, 8 months. Right? That's as far as I know, that's and and what I could see in the market, right, and what what happened was that we basically went a lot faster than everybody else, right, with the same amount of errors. Like, it it seemed that there was no real drawback to using Ruby in that case. Right? And so it it was a pure benefit to use the the language and the framework.
Charles Max Wood [00:52:32]:
Nice. You know, there's there's really nothing that can compare to the speed of development of Rails. It's, I think we we kinda take it for advantage, like, for granted rather, as Rails developers. It's when you go and look at the outside world is when you realize, bloody hell, this is a good tool.
Emil Kampp [00:52:49]:
Yeah. Yeah. It is quite insane.
Charles Max Wood [00:52:51]:
Mhmm.
Emil Kampp [00:52:52]:
Like, I had I had a colleague who did a, you know, sort of a an AI based, feature spike. You know? Like, oh, can we just timebox this to, like, 3 days? And in 3 days, we had a tool that seems to be working, did integrate with AI, had UI, had tests, were behind the feature flag, and was effectively shippable. Right? It was not, you know, it was not the best or the most robust or whatever, but, like, in 3 days, a single person made a full feature spike that could technically be deployed and could work. Right? That's pretty amazing.
Valentino Stoll [00:53:27]:
I'm curious real quick. Like, I'm in in my experience working with, like, critical systems like this, like, once it's built, there's, like, it's hard to change. Right? Like, and mostly because, like, 2, the requirements probably don't change very frequently either. But, like, how how do you, like, approach in those cases like that change? Like, is is it, like, build something new and and cut one off and switch to the other? Or, like, what what are, like, what kind of systematic approach do you take to that?
Emil Kampp [00:54:06]:
I think so for me, it's certainly about you have to reason about the protocol and the process somewhere other than the application. So you have to diagram it or put it into dance or whatever you like to do, but we need to have a different place to talk about it than in code. Because we need to reason about when somebody has a change, we need to be able to reason about it before we spend time in code, and we need to figure out if the the sort of the signature of that particular change, let's say you want to change one, hashing algorithm with another hashing algorithm. Right? You need to be able to reason if that will break the system just in principle before you start to even try and implement it. Right? So my I find that I it really benefits to have it in a big diagram or have it written out in long form on in a document or, like, somewhere that you can reason with people who aren't developers on whether or not it will work or not before you even try to implement. Because if you can reason that it will in fact work, it will plug in, it will replace. If you can reason about that before you actually see the code, when you come to the developers, it's a much smoother experience to say, okay. I need this piece replaced with that piece.
Emil Kampp [00:55:38]:
And, yes, the hashing algorithm is not the same, but it is whatever based on the same something something, so it will plug in. Right? And then we can either have both. If they plug well, we can replace one with the other and do and you can also then you need to trends or migrate the data. Right? But now that you're migrating the data, it it's really nice to also have that conversation with beforehand and with others who are not developer. Well, we have existing data, and you want me to pluck this thing into this place in this big diagram? What happens to all the old data that went through that point beforehand? Like, what do we what do we feel we need to do about that? Do we wanna, revision it and say that version less than something can never be opened again or can never be edited again, but can only be viewed? Like, what's our reasoning, on that? Right? So having somewhere else to talk about it than in code really helps to figure out if you should replace, plug in, or, you know, make it as a different service that you call sometimes, you know, it whatever it is. I find that that having something other than code really helps. I'm still. From there.
Emil Kampp [00:57:00]:
So again, so it really needs somewhere else to, to have that conversation. Right?
Valentino Stoll [00:57:07]:
Yeah. You know, it it makes me think about the, the old days of Yahoo pipes, like, where you could visualize and, like, in real time update the data flow of of any particular
Emil Kampp [00:57:18]:
Yeah.
Valentino Stoll [00:57:19]:
Aggregate, thing you were trying to collect. Are there any tools like that that you use, to, like, maybe automate this process, or is it just a matter of getting it documented?
Emil Kampp [00:57:31]:
That's, I mean, I've tried a bunch of them. I haven't found one that I really like. I haven't found one that really does it for me. I tend to try and make them diagrams as code because then they can at least get checked into Git, and we can have this version control on top of it. But I I haven't really found anything that really does it for me. I've done I tend to either do whimsical or or something like that. And then it is just a lot of work to keep it maintained, and that's a that sucks a bit, but, you know, it's it's hard to it's hard to do, without. Right? Without somewhere else to reason.
Emil Kampp [00:58:16]:
And especially when you need to talk to because it's not only a a technical problem. Right? It is also a business problem, and it is a regulatory problem. If I'm gonna change this piece of the testing bet for an airplane or this piece of, code for the design software for an airplane, I need to be able to have a conversation with somebody who's not a software engineer about the consequences of that. Right? Because I need to talk to the civil engineer. I need to talk to the test manager, you know, and I need to talk to a lot of people about the consequence of doing it and the way to do it. And are there any consequences outside of my system that we need to have a conversation about? So it's it it is just sort of, like, trying to get as many people, joining that conversation as possible. And for that, it it seems to work better if it's not quite code because then more people can participate in that conversation. Right?
Valentino Stoll [00:59:17]:
Yeah. I think that makes a lot of sense. I I'm still personally trying to find the, silver bullet. And the closest I've gotten is Mermaid.
Emil Kampp [00:59:26]:
Yeah. And that also works quite well, actually. It's just it it works pretty I love it. It works pretty reasonable, and it it is definitely diagram as text. Right? So I've also used that a ton, and it's it's decent enough. I find that it has some problems when you get up to really large size stuff. Like, it's it's hard to get enough in there and still have it be viewable and and understandable as a human. But, I mean, a lot of, like, building these systems and a lot of participating in building them is surprisingly not about code.
Emil Kampp [01:00:09]:
It's about talking to a bunch of people who have nothing to do with software development and figuring out if you're gonna break their lives. Right? Like, are you gonna break something completely insane that you had no idea that you ever were ever gonna touch? Because, you know, in in the case of this, well, I'm gonna change this thing in my, my design software. And then suddenly, somebody in the factory somewhere is totally confused about how to build an airplane because I did something that you know? So it's it's it's really not that much of a software challenge. I mean, it is, but it is to high degree a communication challenge and sort of, like, how do we even communicate what this software does to people so that they can use it and so we can reason about the safety of it, together. Right?
Charles Max Wood [01:01:01]:
Yeah. It it makes sense in the I I mean, I've worked on things at this level of complexity that you have to get right. So I worked at Morgan Stanley and I mean, we weren't doing, like, banking level stuff, but we were doing, like, investment level and, business information stuff. Right? And so, yeah, the world didn't end if we got it wrong, but we kinda needed it to get right. And a lot of it was just a matter of, like you said, making sure that all the people who understand all of the concerns and all the things that it touch are able to weigh in and and be part of the conversation. And that way, yeah, you don't do things without understanding the implications of of how it's gonna ripple. I'm working a contract now, and, you know, I'm still trying to figure out the ins and outs of the system. And they're they're managing funds and stuff like that.
Charles Max Wood [01:02:01]:
Right? And so, yeah, I'm definitely feeling it here where I don't wanna make any changes that ripple through the system and right? And so this is where you when you're talking about, like, testing and documentation and things like that, where the documentation really isn't around, and there are a lot of other things I wind up having to ask a lot of questions, and I'm still not sure that all of my assumptions are correct. So, yeah, a lot of this really helps in the sense of, oh, okay. If if we're gonna nail down robust, you know, reliable things, then then, yeah, you know, all of these things are important. And I think part of my issue with some of it is just that I like to just get in and solve problems. Right? I'm my mentality I was talking to some other friends in the the other day, and my mentality is go go go go go go go. And so I yeah. Having to slow down and, you know, dot all my I joke and say I dot all my t's across all my i's. It it just it's, you know, it it it's it's not my natural state, and I think a lot of developers yeah.
Charles Max Wood [01:03:16]:
They just they just wanna write great code that solves problems, and so the meetings are tedious. The conversations are boring, you know, and, you know, why can't I just go in and just make it work? And the reality is is, you know, yeah, what what we're what we're talking about here. And so I think I think it's not just we have these specific concerns whether the regulatory or something else. I think it's also that there's enough complexity to it to where you really do have to get in and understand the nuance of what you're doing.
Emil Kampp [01:03:49]:
Yeah. And and I think also the the breadth of what you're doing. Right? Like
Charles Max Wood [01:03:54]:
Mhmm.
Emil Kampp [01:03:54]:
I I know this for myself, right, when I was younger and and started developing code. Right? It was like, well, there's only one truth. Right? Like, either it's 1 plus 1 is 2 or it's wrong. Right? But that's just not the case when you have these really big connected systems. Right? Where, like, well, what happens if I change this protocol? Well, now it's right because it's a different protocol, but now it's wrong because it's a different protocol. So it can be both at the same time. Right? Because because now, you know, all the other people who used it for one protocol can't use the other, and, like, it just gets really messy. I think at some point, I realized that I liked doing documentation, and that really helped to sort of, like, make me be confident that this could be some way you can do a career.
Emil Kampp [01:04:49]:
Like, you can do a career in something that's not obviously developer ish territory. Right? Like, because I actually enjoyed doing, like, architecture diagrams and reasoning outside of the code. And all of these things I found was pretty interesting, and that makes it easier for to be me. Right? Like, it it's more fun to have that conversation with everybody to figure out, well, if we do this thing over there in wherever we are operating in Australia, that's fine. But now we've done this change. So what happens to our operations in Austria? Like, is that the same tax system? Can we do both? Like, it just it it can become really complicated when you have when you're operating under multiple jurisdictions and multiple governances. Right? Then that that can become complicated as well. And, you know, you're doing currency conversions or you're doing unit conversions or you're doing all of these things.
Emil Kampp [01:05:52]:
Like, it's so hard that I know that even, the space agencies are getting it wrong, and sometimes their stuff just won't work when you get it into space. Right? Because somebody miss, converted inches to centimeters. Right? So it is just and that should, like, to if you go to a a fresh out of college developer and says to him that I need a unit unit conversion system, a unit conversion method that can go from inches to centimeters. That's pretty straightforward. That's a that's a pretty straightforward ask. Right? And you could do that method pretty quickly. It's not really super hard. What is really super hard is getting people to use it.
Emil Kampp [01:06:41]:
Right? So you need to build a unit conversion method that is easy to use, fun to use at hand when you need to use it, so that you will in fact use it. Because otherwise, you're gonna do it in your head, and then it doesn't really matter that you build a piece of software. Right? So, like, it it there's a lot of other components to to this than just the software. Like, there's a lot of things, like, you know, there's a lot of backup, a lot of everything, but it is also about making sure that the people who use it use it correctly. And the way to do that, like we talked about before, like, with the TSB and everything, it's not to blame the person for not using the thing. It's to blame the thing for not being available when it was needed. Right? Mhmm. So you need to be like if if I have a unit conversion thing and it's hidden behind, you know, a multiple 3 dot options that you need to figure out 5 levels in, that's the wrong place to have it then.
Emil Kampp [01:07:38]:
Right? You you don't blame the person for not using it. You blame the system for not putting it in his face when he needed to convert the the measurement.
Charles Max Wood [01:07:49]:
Yep. I I like I like that. Just from the standpoint of I mean, I guess I guess where my brain goes is is everybody has the responsibility to understand the system and how it works. Right? And so everybody has the responsibility then to, you know, to to be aware that that this method exists. But at the same time, if I create it, then it's also my responsibility to make sure everybody knows it's there and knows how to use it. And I also agree though that if you make it easy or natural to do the right thing, then people will generally do the right thing.
Emil Kampp [01:08:32]:
Yeah.
Charles Max Wood [01:08:32]:
And so yeah, I I I think I agree with you, but I don't wanna absolve anybody of the responsibility of learning to do the right thing either.
Emil Kampp [01:08:42]:
No. No. I mean, that's not what it I mean, you can you can absolutely end up being blamed for crashing a plane. Like, it's not entirely out of the realms of like, the pilot can be blamed. Right? But it is also about looking at, did we actually provide the correct tuning at the correct time with the correct intensity and all of this. Right? And it's the same here, like, well, the system didn't work. Did we provide enough? Did we scale enough? Did we have enough compute resources? Did we have enough database storage? Was it just so slow that people couldn't use it and it crashed and somebody couldn't get their vote in? Because if somebody couldn't get their vote in, then, you know, we sort of we get into the territory of, somebody prevented me from voting situation. Right? Mhmm.
Emil Kampp [01:09:34]:
So it like, yeah, they could have done it again. Like, they could have just waited 5 minutes till the peak was down and voted again. But that's, like, it's also not really their fault. Right? Right. So it's there's a lot of there's a lot of sort of, like, ancillary nondeveloper stuff to talk about when when designing. I'm pretty sure there is on all systems. Right? But the the fun thing here is just that this happens to include other engineers, which is very fun. And it happens to include things like cryptographers and, people who actually like politicians or whatever it is.
Emil Kampp [01:10:16]:
It's just I like the types of people that I get to talk and interact with, right, in in this rather than, I get to talk less with marketing and social media and more with, like, people who build spaceships and airplanes. And I find that that that's basically what drew me here. Right? Is this idea that I get to talk to other people who also have this engineering spirit and this idea to build something in the same mindset that I do.
Charles Max Wood [01:10:48]:
Awesome. Alright. Well, I think we're gonna go to picks because we've we've been talking for an hour and 15 minutes. This is cool though. If people want to connect with you or if they have questions about what you're doing, you know, maybe you wanna pick your brain about some of this stuff. Yeah. How do they find you on the Internet?
Emil Kampp [01:11:08]:
I think go to LinkedIn. That's, I I have more or less tried to ex absolve myself of all social media. So LinkedIn is probably your best bet. It's my name. It's right there on the screen y thing. There it is. And then also, go go on Discord. You can ping me there.
Emil Kampp [01:11:28]:
PM me, like you did. I'm both on the Kamal, the Rails, and the Ruby servers, so you can just go find me there. It's e k a m p p. So just my initial letter and my last name.
Charles Max Wood [01:11:46]:
Nice. I didn't know that there were Ruby and Rails Discords. I'll have to go find those.
Emil Kampp [01:11:52]:
I can, I can send you invite links?
Charles Max Wood [01:11:54]:
That that would be perfect. Cool. Alright.
Emil Kampp [01:11:57]:
There is there's one for, like, Rails development. That's the Rails one. And then there is one for Ruby, which is more like application support and, like, oh, I don't understand what this method does or I need to change this. Whereas the rails one is more for, like, I need approvals or reviews of PRs for, the actual, call. Right? So that's more about supporting of that. So it's less interesting if you're trying to solve a business problem, but it is kind of cool if you wanna just peek at what the others are doing.
Charles Max Wood [01:12:29]:
Nice. Okay. Well, let's go ahead and do some pics. Valentino, do you wanna start us with
Emil Kampp [01:12:35]:
pics? Sure.
Valentino Stoll [01:12:39]:
I've got nothing but AI picks today. There's been a lot of activity in the Ruby world. Andrew Cain has released, yet another AI gem. Bless his heart. It's called Informeris, and it basically lets you run, inference in Ruby on, Onyx models. So lots of Hugging Face models are Onyx supported, and you can now just run inference with those. And he also has transformers RB if you are interested in running inference on non Onyx models. So now we have, like, a complete picture.
Valentino Stoll [01:13:15]:
Hopefully, those, there are some some missing pieces still for larger models, but, it has a ton of pipelines already built in. So, I would suggest checking that out. The the next pick, somebody built this awesome chat app where, they've aggregated all of the Rails and Ruby, like, articles and blogs and guides and everything like that. So you can basically rag the entire Rails community ecosystem,
Charles Max Wood [01:13:49]:
in
Valentino Stoll [01:13:49]:
a in a chat GPT style interface, and it's it's remarkable. And it does, like, citations and everything like that. So you can see the original article that reference whatever it was you're interested in. It's really awesome. It's called Ship on Rails, and it's built with this, really cool tool called Nausea, which provides basically, like, a a Rails app ready to go so that you can ingest your own documents to rag against, and do a very similar thing. And it's really cool. So I'd recommend checking all those out.
Charles Max Wood [01:14:25]:
Nice. How about you, Ayush?
Charles Max Wood [01:14:29]:
Not sure what to take for picks this this week. I haven't really thought of anything. It's a it's a it's a it's a cheap one, but I'm gonna go with Blue Sky because I'm quite enjoying Blue Sky in the last week as I finally signed up for it. And a lot of Ruby folks have, have signed up recently. It's quite a lot of activity on there. I'm quite quite enjoying it on the social network. Hopefully, it can stay the way it is. I'm active on, both Blue Sky and Mastodon for now.
Charles Max Wood [01:15:00]:
But, yeah, we we'll see what happens because Blue Sky is still a VC funded, privately held company. So, we'll have to see how long if things remain good. But for now, I'm quite quite happy with it. What else can I pick? I haven't really watched anything or anything new recently, because I think I I had picked the TV show called shrinking, which I did the last time, so I don't wanna pick that again. I'll go with the music pick because I always go with the music pick when I'm when I'm struggling. What's in front of me? The new tears for fierce album. Check this one out. It's really good.
Charles Max Wood [01:15:36]:
It's a live album with a few new songs, from eighties pop band Tears For Fears. So, yeah, I think I'll I'll go with those.
Charles Max Wood [01:15:48]:
Awesome. I've had a number of people tell me I need to join Blue Sky. So, anyway, let's see. I'm gonna throw in some pics. So I played I played a new game. I actually played a couple of new games this week because the game convention that I usually teach games at is, is gonna be on Saturday, and so I need to be able to teach these games. So I'm gonna pick one of those. It's called MLEM, l m l e m.
Charles Max Wood [01:16:31]:
And, essentially, what it is is it's how do you describe it? It's it's Cat Astronauts. And the way that it works, it's it's a relatively simple game. It was almost too simple for me, to be honest. But if you're kind of a casual board gamer, right, it's it's definitely approachable. So, let me pull it up on BoardGameGeek. But yeah. So what the way that it works is you have astronaut cats that are going going into space, and they're basically occupying planets and moons. And so you pick one of your your cats, put it on the rocket, and somebody's the captain, and they roll the dice.
Charles Max Wood [01:17:28]:
And if they're if the dice match what's, on the space on the board that you're on, then, you can choose which dice to use in order to move up the board. Right? And so if you if you have boosters, boosters move you up for free. The other ones, you have to give up the dice in order to move. And so if you ever get to the point where you roll the dice and you, you can't use any of the numbers, then you can't push the ship. And at any point, people can get off the ships onto moons or, planets, and you get points for having the most cats on planets. You get points for every moon you occupy. If you go all the way to deep space, then you get bonuses for that. And then the different cats have different abilities.
Charles Max Wood [01:18:24]:
Right? So, one of the cats lets you start further up the board. 1 of the cats lets you, you know, lets you get off the ship after it crashes. Right? So you can go to the planet it's next to or whatever when it crashes. One of them gives you a bonus, doubles your points for the planet it's on, one of them doubles the points for the moon it's gone it's on. Anyway, you get the idea. One of them doubles the points if you're in deep space, but deep space is hard. I've played it once, and we never made it to deep space. We got close a couple times.
Charles Max Wood [01:18:57]:
But then I was talking to some other people who had played it a couple of other times, and one of the times they played, they made it to deep space, like, 3 or 4 times. So, anyway, so if you're the captain, you're rolling, and then you just rotate who the captain is. If the captain gets off the ship, then the next person in line just finishes out that round. Anyway, that that's basically the game. BoardGameGeek has it as a at a 1.68 weight. It says ages 8 plus. And, yeah, I I have an 8 year old. She could definitely play it.
Charles Max Wood [01:19:31]:
It says 30 to 60 minutes. That's probably about accurate. Plays 2 to 5 players. I played it with 4 players, and that felt pretty good. So, it's Mlem, m l e m, Space Agency. And, yeah. Anyway, check it out. And then yeah.
Charles Max Wood [01:19:51]:
I'll just put a link in the comments here. And then, let's see. I'm not sure if I really have any other picks. I know I've kinda been doing other things and getting involved in other things, but, I guess the the big thing that that I went through this lately is I went up to a retreat up in Sundance, which is actually, like, 45 minutes from here. You heard of the Sundance, film festival? They do actually do that in Park City, not in Sundance. But from here, it's about 2 thirds of the way to Park City. And, anyway, it's it's up Provo Canyon, and these guys were running essentially a mastermind for 3 days for Christian entrepreneurs. And so we we did a lot of faith based stuff.
Charles Max Wood [01:20:53]:
We did a lot of business based stuff. And it was amazing because they just kinda helped me get clarity on what I wanna do and who I wanna be and what kind of difference I wanna make. So just keep an eye out because I I have stuff in the works. But it it was terrific, and I I really want to just put forward, I guess, that just connecting with other people who are doing what you're doing or who are, learning what you're learning or who are maybe a little ahead of you on on the path you wanna go down. It's so helpful. So, so helpful. You know, kinda like what we were talking about with, like, the Discord servers. Right? So much help helpful stuff there.
Charles Max Wood [01:21:37]:
So, anyway, those are my picks. Emil, what are your picks?
Emil Kampp [01:21:43]:
So recently, I'm I'm gonna pick a book. I recently read, a book called How Big Things Gets Done, and it's by, by a Danish guy called Ben Flubia. So what he did was that he went I don't know if you guys know it, but he went through statistical data on overruns on big projects, like how far behind is an IT project in general, how much of an overrun on cost do you get, how much of an overrun on time do you get. And he goes through sort of, like, these different industries and different, factors that play into to why why big things fail and why some big things gets done. Right? Like, what are the ways to make sure that big things gets done, and what are the pitfalls and everything? And he goes through everything from the Sydney house, Sydney Opera House to, like, big government IT projects that fail. It's just super cool because it's very sort of data driven, and he has spent his career collecting data on why big projects fail, why all kinds of projects fail. So it's a really interesting, sort of window into do this if you really wanna mess it up. So by inversion, if you don't wanna mess it up, don't do all of this.
Emil Kampp [01:23:07]:
Do do the other thing. Right? So that's kind of an interesting, statistics to have. And so, yeah, how big things gets done. I found that was, really cool.
Charles Max Wood [01:23:19]:
Awesome. Alright. Well, thanks for coming, Emil. We're gonna go ahead and wrap it
Emil Kampp [01:23:24]:
here. Yep. Thank you for having me. It was a pleasure.
Charles Max Wood [01:23:27]:
And until next time, Max
Emil Kampp [01:23:32]:
out.
Hey, folks. Welcome back to another Ruby Roads. This week on our panel, we have Ayush Nwatiya.
Charles Max Wood [00:00:11]:
Hello. Hello.
Charles Max Wood [00:00:13]:
We also have Valentino Stohl.
Valentino Stoll [00:00:15]:
Hey now.
Charles Max Wood [00:00:17]:
I'm Charles Max Wood from Topten Devs. And this week, we have a special guest, and that's Emil Komp.
Emil Kampp [00:00:24]:
Hey, everybody.
Charles Max Wood [00:00:25]:
So, Emil, I'm just gonna clear my throat because wow. Okay. Let me try that again. So, Emile, I found you. I've been, corresponding with people some on the Discord for Kamal, which is the deployment technology that, DHH, put out. I think he made kind of a deal about it at, rails world last year and then a whole bunch of exciting stuff this year with, Kamal proxy and stuff like that. I mean, I switched over to and I switched over from Kamal to Kamal too with a whole bunch of help from you and others on there. And, I have to say, switching over was a little rough.
Charles Max Wood [00:01:10]:
Having switched over, it's a lot nicer. So, anyway, so beyond kinda what I know about you from that, what what should people know about you?
Emil Kampp [00:01:23]:
Well, so I think, first off, I am a mountaineer. And in order to facilitate that, I live in a very flat country, so I do a lot of running. And then I need to pay for my travels, so I do also engineering. And I've made a career of, doing sort of critical infrastructure or things that have real world impact and real world con consequences if something goes wrong. Right? So both in terms of security and robustness and things like that. And that's everything from the Fintech or the payment industry to civil engineering, to, online voting.
Charles Max Wood [00:02:15]:
Wow. Yeah. It sounds like kinda what our topic is here, which is building critical infrastructure. So, you've done some stuff that I kinda need to work,
Emil Kampp [00:02:26]:
I guess. But, like, what are we putting in? I mean, so banking obviously needs to work. Right? People get sort of pissed if they don't get their money. Yeah. And then, for civil engineering, I worked on, tools that the engineers would use to build the airplanes, which and and run tests on them. Right? So both test benches and design benches for airplanes. And, I mean, that has a real world consequence because they fall out of the sky, right, if you mess it up. Yeah.
Emil Kampp [00:03:02]:
That's good. And then, yeah, online voting, that's a whole thing. And, you guys in the US just went through one, one of these messy events, and they're messy everywhere. Right? Oh, yeah. And recently, I moved to to compliance and sort of making trying to make it trying to make that easier and a better experience to become compliant and to obtain, certifications.
Charles Max Wood [00:03:36]:
Right. Makes sense. So, let's just dive in. I mean, I'm not I'm trying to think if I've ever worked on anything that was, you know, kind of at the level of, hey. If this breaks, then, you know, really bad things happen. I mean, if it breaks, I've I've I've had them come to me and be upset like really bad thing happened, but right. It it doesn't affect society at large. What's the difference between working on those apps and working on just kinda your, you know, you mentioned not ecommerce or not, you know, hey.
Charles Max Wood [00:04:13]:
I've gotta manage my social media or, you know, get podcasts out on time.
Emil Kampp [00:04:18]:
Yeah. I mean, there's nothing just to be clear, there's absolutely nothing wrong with that. Right? That's just I I I started out in in that area as many developers do. Right? Building, like, some app system when I was very young and then moved to some, you know, sort of offer platform, and then I sort of realized that it was it was just inconsequential in a way. Right? I looked at people. You know, I had people around me. My my father was a is a civil engineer. Right? And so I have people in my, sort of, my family, my near family that was building things in the real world.
Emil Kampp [00:04:58]:
And I was like, I wanna do that, but I do like this computer stuff. So can we sort of maybe find a way where I can build things in the real world just using my computer? And so I slowly started to shift over into something that was founded more in in civil engineering and, sort of industry rather than commerce. Mhmm. And then I started in a Fintech company, where we were where I was part of building building out and continuing to build on a on a system where you can basically it's like direct debits, bank transfers, but worldwide. Okay. So and and there, it it doesn't necessarily mean a lot to you and me, and it's the same with there's a bunch of them now. Right? Like, these send money home, types things. It doesn't mean a lot to you and me, but it does mean a ton to those who is on the receiving end of that.
Emil Kampp [00:06:06]:
Right? Right. Because that's, like, half a year's pay or something that comes in. Right? So it does matter to those people. So so, yeah, that was sort of where I ended up. And we are moving a lot of money. Like, it it it seems it's one of those where little like, it's it's a lot of tiny things that add up. Right? There's, like, 1,000,000 and 1,000,000 of dollars being moved around for that purpose. And you don't wanna mishandle those money.
Emil Kampp [00:06:38]:
Right? So you are hooking into the to the banking infrastructure, into Swift, and into the IBAN network and the whole sort of, PC compliance and everything with that. Right? So you need to make sure that the software is robust, well tested. Like, testing has been a main thing of of mine for a long time because of that, and also documented well. Right? So that's where I sort of started out with being required to document what I was doing and how it was supposed to work, rather than just shipping fast and breaking things. I was required to test and document, then test, then stage it, then test some more, then ship and not break things. So that was a different paradigm.
Valentino Stoll [00:07:33]:
Mhmm.
Emil Kampp [00:07:35]:
Yeah. So that's sort of where I I first met that. Right? And that was a a pretty great experience. And so I was like, oh, can I have more of this? Can I maybe even get even closer? Right? Yeah. So that's, that's where all that started. The and then I was hooked on the whole, can we test it? Can we build something that runs Ruby on Rails but makes a difference to, to people's lives, like, in real world. Right? So this was basically a a payment, situation just building Ruby on Rails. And back then, many of the banks didn't have a proper API, so we ended up scraping a lot, and that's, you know, against terms of policy, in terms of service, and all of that, and they were really mad.
Emil Kampp [00:08:24]:
And we went through that whole shebang of getting, you know, sort of, scolded, for for that for a while. Yeah. So they it was a bit the Wild West back then. But, it's definitely gotten better, and I think there's a lot of, again, sort of compliance work and legal work that went into, the EU, did a lot to make sure that banks would have not standardized APIs, but a minimum level of information that you could retrieve with an API, which made all of that a lot easier. Right? It also means that I can technically build my own banking app for my own bank account because I have API access to it. It may not be obvious. It may be poorly documented, but it is there. So I think that's pretty cool.
Charles Max Wood [00:09:19]:
Yeah. I'm curious if Valentino or Ayush have experience doing this stuff.
Charles Max Wood [00:09:28]:
Nothing that critical. Now the closest I kinda came to anywhere near that world was when I worked for TransferWise, but I was an iOS developer, so I didn't really have to do any of the hard stuff. I just made API calls, but I know they were also doing quite a lot of the scraping and stuff. There's a lot of selenium going on behind the head to get into different bank accounts. It's, yeah, it's, it's all a bit, scary when you think about it.
Valentino Stoll [00:09:56]:
Mhmm.
Emil Kampp [00:09:57]:
Yeah. Yeah. It really is because, like, your whole banging experience is tied up on some scraper working correctly and not messing around with the, the 1,000 delimiters and the, the sent commas are different on different regions. Right? So you don't wanna mess around with that. I did transfer some money incorrectly. Luckily, the the backbone banking system is is pretty robust as well, so we were able to basically not move the money anyways. Right? But it, I did move far too many money, the wrong place.
Charles Max Wood [00:10:33]:
I won't be in there.
Emil Kampp [00:10:36]:
Yeah. That was a that
Charles Max Wood [00:10:37]:
was Oh, never mind.
Charles Max Wood [00:10:38]:
That
Emil Kampp [00:10:39]:
was a backbreaking day.
Charles Max Wood [00:10:41]:
I'll bet. So I'm a little curious as we get into this. Like, you mentioned testing. What what is your what is your testing setup look like? Because I mean, I I'm I'm sorry. When I write my test, it's like, okay. It's good. Like, I'm I'm reasonably confident. Right? But I'm not, like, ironclad confident, like, banking level confident that my code works right.
Emil Kampp [00:11:07]:
Yeah. Yeah. That's a good question. I tend to not be too dogmatic. I know there's a lot of, like, do this or do that paradigm or do test first or whatever. I I tend to just play around with it and make myself certain first that this is gonna work, whatever I'm sitting with. And then when I know that it you know, I'm sort of I can see, like, okay. It works.
Emil Kampp [00:11:37]:
It it does what I need it to do, then it is worth testing. Right? And then it's I try and test on the lowest possible level first, which means I will start with something like unit tests. And for that, I I wanna permute everything. Right? You wanna make sure that every argument so you basically start with the least amount of arguments for a method, and then you add an argument slowly in I use Aspect, so it's in every context block or in every describe block. Right? So you're you're certain that every version of every argument is, is tested and and has been permuted. Yeah. So that everything like, every variable has been tested on its own. So you know how it behaves both with and without that particular variable.
Emil Kampp [00:12:36]:
And you also know how it behaves if the variable has, like, the wrong type. Right? So because Ruby isn't quite where we have type safety yet, there's a lot of, work going on with that, but we aren't there yet. So I you know, you wanna make sure that we can handle strings, numbers, not strings, trues, booleans, whatever. There's like I have a little thing that basically says, take this and just, you know, cycle it through all of the different types it can be and just put in garbage so that I know it cannot it doesn't break when a number is presented rather than a string or something stupid like that. Right? And then you sort of you slowly move up. So you're confident that this method works, and then you move up to have recorded, and then you I what I do is I then I go through and test that. But I think opposed to what the s DHH, preaches or or at least sort of, he he has this idea that you go through the entire stack, and I believe that they their tests do that. I tend to not I tend to sort of stop testing at each level.
Emil Kampp [00:14:02]:
Right? So model tests are their own thing. And then when you test the controller, you basically verify that the model is being called with the expected parameters because you know you've already dealt with the parameters on a lower level. So it's not mocking per se. It's just that I am I I only need to test that the model was created or was called. I don't really need to test that it was the correct arguments. Right? Because I'm certain about the correctness of the arguments on a lower level than that. And then you saw slowly move up, which means that I I find that that means that the individual testing levels, it gives you a little bit more freedom to sort of move a model around because it's it's coupled somewhat on a test level, but not tremendously. Right? I only coupled the controller to the model in the sense that I checked that the model was called, not that it was called with a particular set of arguments.
Emil Kampp [00:15:06]:
So it makes it a little bit easier to also restructure your tests when your code change. But, yeah, it it's just a lot of work on lower level things. I'm a big fan of Sorbet, but, it's not it's not quite there yet. I think it it's it works reasonably well, but it's very hard for developers to use. So I think we need to do there there needs to be a lot of work done in in terms of making it nice and sexy to use so that we can get people to use it. Right?
Valentino Stoll [00:15:40]:
I'd I'd love to dig in a little bit more on the testability aspect. Because one of the biggest things I see, miss is maybe, like, skipping over, like, the maybe some lower level implementations and then, like, a system update breaks something, and it just wasn't covered. Right? Like, how do you go about, like, you know, weighing whether or not that is worth it to, like, actually run, like, a system command or an external API call or something like that?
Emil Kampp [00:16:13]:
Yeah. So, system commands, I usually check just that the signature of the call is correct, and probably that it returned the right thing. Right? So that if I hit the OS, I'm I'm basically testing that I gave it these parameters, and I expected something back, like a string or whatever it is. Right? Because it's so hard to know what it was supposed to be because sometimes the system commands will change their their signature, but not necessarily return value or the return value, but not the signature. So that's that's very hard. I try to not go go there. I try really hard not to go there because it is it's out of my control. Right? On the other end, when I call other APIs, I definitely test that.
Emil Kampp [00:17:11]:
I use something like VCR or something. I capture all the network traffic, and I make sure to do the same kinds of, sort of making sure that if I call it with these arguments that I have control over, I get this particular response. And if I do something else, then then it breaks, right, or it gives me the incorrect response. And I also found that, for example, with, with banking, that one of the cool things we did, was that we basically took all of the historical records, right, that we had for all the bank accounts, and we basically used it as a test bed because then you can train on that. You can basically say, if I feed it all of this information that we know happened and it happened correctly because it it happened, you know, like, a year ago or 2 or whatever, it should result in the in in a particular set of of of, like, it should result in a particular value on the account. Right? So you can actually use historical data as a test bed. And that may that means that you have a factual way to check if it's correct or not. So, yeah, I I tend to use that as well.
Emil Kampp [00:18:35]:
I also use that when when I did software for the for the for the sort of civil engineering and building flight and testing that. Right? We also used a lot of existing information to make sure that we had input output. So we understood that given these inputs, this is the output that we would expect. Right? We could basically take old designs and try and run them through, and you can see if does does it break it down and you'd like you would expect? Does it run the does it sort of fit to the engineering standard design standard that we were implementing against? They also have test value that you can use, right, or test data. So it's also about going out and finding big test datasets that you can use to also make a statistical sort of approach. Right? I'm not making any statistical claims, but I am running a lot of tests on a lot of data that, you know, you can be reasonably certain that, that if if there was errors in your system, it would have been caught by this because it's so much effective effective production data. Right?
Valentino Stoll [00:19:54]:
Yeah. I imagine that working with, like, the public sector that you would have a lot of access to that data. Right? I guess depending where you live.
Emil Kampp [00:20:05]:
Yeah. I mean I mean, interestingly enough, none of this is for the public, sector, which is a little bit weird. You would think that it was, but it actually isn't like app the regulation of of the aerospace is public sector. Right? But the actual building of airplanes is a private commerce. Right? Private industry. And likewise, with banking, the the regulation of it is public, but the actual banks are private entities. And likewise with
Valentino Stoll [00:20:36]:
find it difficult for, like, the data access? Like, is that data proprietary then for, like, doing large swaths of tests or finding it?
Emil Kampp [00:20:46]:
Yes. But most of the times, the companies are interested in providing it. Right? Like, if you if you go on to big, to big airplane or an aerospace company and and you're you need to design a test system for them or, something to help them build. Right? They are interested in giving you that kinds of data. So it wasn't that it was hard. It's just that it is also quite, narrow in scope. Like, it's it's more more or less only for that one. Right? So in in banking, it would be a lot of, records, but only for one bank.
Emil Kampp [00:21:23]:
So you would only know how it looked in one type of account system. Right? And then you would run all your tests on that. And now 2 seconds. Alright. Yeah. There we go. Hello.
Charles Max Wood [00:21:48]:
Hi.
Emil Kampp [00:21:50]:
So so it it's not that it's hard to get. It's that it is it's quite narrow, in terms of, sort of, like, what yeah. It doesn't have a lot of breadth. You get it for, like, one bank or one airplane company or, you get it for one company who runs elections. Right? Public elections is a bit different. By definition, you don't have any test data because all of the records are sort of you don't know who who voted, so you can't know Mhmm. How to test that. Right? So you have to do a different testing paradigm there.
Charles Max Wood [00:22:30]:
Yeah. That makes sense. I mean, here in the US, we have a a anonymous ballot as well. Right? Yeah. And I I don't know how they do it in other states. I mean, I'm I'm involved in the process here in Utah. And, yeah, the you send in your envelope and they with your ballot in it, and they validate that your, like, your signature matches and stuff off the envelope. But then they pull your ballot out of the envelope and put it in the pile to be counted.
Charles Max Wood [00:22:58]:
And Yeah. Right? They they don't they track who you voted for in that case or at least they're not supposed to.
Emil Kampp [00:23:04]:
Right? Correct. And and that is also the case for the digital solutions.
Valentino Stoll [00:23:08]:
Right.
Emil Kampp [00:23:09]:
A friend of, a colleague of mine, Stefan, who's a cryptographer, he, a previous colleague of mine, he did a lot of work of sort of, gathering together a lot of the academic work so that we could exactly do that in a cryptographically safe way, but online.
Valentino Stoll [00:23:30]:
Mhmm.
Emil Kampp [00:23:31]:
And you could also do it in a way that it was you know, you could track it, and you could ensure that this was in fact the case. Like, you could give the the log data and the trace data to mathematicians, and they could verify that mathematically, it could only ever have worked if these rules were followed, which was like, you know, you anonymize and you did you did mixing and you did a lot of these things. Right?
Charles Max Wood [00:23:57]:
Right. So you protect the identity of the voter, but you also ensure that only people who are legally permitted to vote vote and that they only cast one ballot and things like that.
Emil Kampp [00:24:09]:
Yeah. Yeah. Exactly. And the test the test, sort of the the build idea for that and the testing regime for that is is significantly different. Right? Because this is in fact the public space to to your point earlier, Valentino. This is a public, domain. Right? So you you are a bit more scrutinized. You're also publicly scrutinized by everybody.
Emil Kampp [00:24:31]:
Right? Because we all know how it went with the domain Dominion Voting Machine case. Right? That went real fast, real downhill. So Yeah.
Charles Max Wood [00:24:45]:
Well and you bring up the Dominion, voting machine case. And whether you believe that they worked or didn't, there's if there's enough population that doesn't trust it, then it causes problems. Right?
Emil Kampp [00:24:59]:
Yep. And that's that's the same everywhere. Right? Yep. And that's I found that sort of on a on a completely different but somehow related topic, I really found that the aerospace or the airplane industry was was really good because the, what's the American one, the TSB, Transportation Safety Board. Is that NTSB. NTSB. NTSB. Right? Yeah.
Emil Kampp [00:25:27]:
Those are the ones that that do crash audit that sort of, like, go through
Charles Max Wood [00:25:32]:
Oh, yeah. The NTSB. Yeah.
Emil Kampp [00:25:35]:
Both them and the European counterparts have a really, really cool way of going about it. They don't ever go and say, oh, it was that person's fault. Right? If if the person hadn't slept enough, if the person couldn't read the dials, it is the fault of their plan that that made him not have enough hours to sleep, and it is the fault of the dial for not being visible enough. Okay. Right? So it it's not they don't ever cast personal blame or they very rarely cast personal blame. And I really thought that was a super cool way of going about it. Also, just from a sort of, like, evaluating, if something works well. Right? Just, like, from a human perspective and when you do leadership in tech or if you, guide or lead others that I found that really valuable, sort of that whole way of thinking about it.
Emil Kampp [00:26:37]:
And they also they collect a ton of evidence. Right? They have a very structured way of collecting that before they then conclude that it works or didn't work or that this was the fault or that was the fault. But they also I I think I learned a lot also in terms of, like, having redundancies. Like, what happens if this thing breaks? What happens if your job runner doesn't run? How are you alerted? What happens if your email doesn't send? How are you alerted? How do you have alternative options? Do you have an alternative email provider that you can switch to easily? Do you have alternate job runners that you can switch to easily? Do you have any of this? Right? So that redundancy ID ideologies was pretty cool. And I'm for, like, large scale systems, it it pays off to have extras of these even though, you know, you you use somebody like SendGrid or you use somebody like Mailchimp or something, and, you know, they say that they scale and they do. Right? But then ever so often, it's down, and it's pretty nice to have a backup. Even if it's a tiny backup and even if it's just, you know, up and coming fairly cheap one, it will do the job for, like, a day. Right? And you will actually still send stuff out or you will still process jobs or you will still whatever it is.
Emil Kampp [00:27:56]:
Right? Also running, like, on the big scale, running something like multi cloud or running something like an open stack where you can actually move to somewhere else pretty fast. There was that case with that the Australian case where the where the bank got taken down, because the someone at Google decided to delete their account, right, for, like, a 160,000,000 US worth of servers, just got nuked. So for those cases, it's pretty nice to have a second provider backup, and some way of getting there. Right? That is more a DevOps problem. It's not really a problem that I've ever solved. It's a problem that I have, you know, sort of, like, assisted with and talked about and verbalized a lot in companies. But it is to high degree a DevOps solution that needs to go in there and not one that I am gonna build.
Valentino Stoll [00:29:01]:
So you're talking a lot about, like, people's lives and, you know, their livelihood, their money, like, things that sustain life. Right? And so, like, obviously, these infrastructures are critical. Like, what tools you're using are critical. So I'm curious, like, what about, like, Ruby and Rails, you know, specifically feature wise? Like, do you see as being like, oh, this is, like, part of that critical process?
Emil Kampp [00:29:30]:
I think the the main part for me is that it can be built, you can write it so that it is absolutely readable and understandable what's going on. So you can reason about the what's going to happen even without necessarily understanding the language fully or even without understanding what this active record method is going to do underneath because stuff is named well and because it's ordered and organized well. You can reason about what's gonna happen, and you can have an expectation, and it makes it a lot easier to build complicated things. Like, you can go and test it afterwards. Right? But just getting something that's complicated up, like, just getting it up on its legs, it's really nice to have something that that is structured well and that you can reason about and that you talk about, sort of what's the consequence of this method or that method. Right?
Valentino Stoll [00:30:34]:
Yeah.
Emil Kampp [00:30:37]:
There you go. So that's that's cohost. Yeah. Exactly. But I think and, also, I find that it's I mean, I've I've done a few frameworks throughout my career. Right? It I've always come back to to Rails and Ruby, and I've I also started there. So maybe I'm biased, but I I I haven't found any of the other frameworks that provides the same sort of consistent experience through every part of the application. Whether or not you're doing low level active record stuff or you're doing high level system tests or whatever you're doing, it's consistently the same experience everywhere, which means that it's fairly easy for a developer to switch gears and do something else, right, in in the stack, I mean.
Emil Kampp [00:31:34]:
So, like, you can have a person who is sitting and doing the swift transfer, 1 week. Right? And then he can go and he can look at the web scraper the next because it's consistently the same experience, and you can write it in a way. If you have a good sort of if you have a good style guide and you have good code review, you can get it to a point where it's the same experience every time. And you can get abstraction levels and abstraction layers that feels the same. And that makes it easy to build something that where you can also easily plug out something and put something else in. Right? Like, you can switch out one database provider for the other without much,
Charles Max Wood [00:32:20]:
blocking.
Emil Kampp [00:32:21]:
Right? Or you can do whatever whatever it is that you feel like you need to change is can be quite easy and quite straightforward if you've set it up, to be and I'm not necessarily talking about modularity in the site work, level of, aggressiveness that, that Shopify has. Right? But you can you can definitely do something along those lines with just talking about it, organizing the organizing the code well, and, and just sort of thinking about how how is this gonna be tested? I I find that that's also part of it. Right? It's like, how are we going to test this? Because if we are going to if we're going to test it in this particular way or that particular way, then that that informs the the API and the sort of the interface for it. And a database, you tend to test the same way, You know, if it's file uploading or downloading, you tend to to test that the same way, which makes it easy to switch out one component for the other if your test paradigm is the same, right, for for that component. So I feel that's that's some of the reasons that I've chosen this, again and again. Right? And, also, like, DJ said, on his keynote, This is a one like, it can be a one man army. Right? You can be pretty fast and pretty, dangerous with it, which means you actually need fairly low head count to get somewhere really fast. And you can actually get something that's really secure, solid, safe, robust, and easy on the eyes and nice to use with only a handful of people.
Emil Kampp [00:34:23]:
Right? No. You can get somewhere with just one person, but you can definitely get a long way with a handful of people.
Charles Max Wood [00:34:32]:
You are so speaking my language, man. This is why I'm here. Yeah. So one thing that I'm wondering about with the all of this is, you know, it sounds like depending on what the concerns are in the app. Right? So whether it's banking or voting or right? You you may have different concerns about, uptime or stability or, you know, resumability, you know, if if Google deletes all 1800 of your servers or, you know, whatever. Right? So how do you begin to start looking at your application and saying, well, you know, for example, in banking. Right? We have to make sure that we don't have any of these kinds of issues, bugs show up. Right? You know, make sure our accounting is super robust.
Charles Max Wood [00:35:20]:
But then you're talking about the voting and it's, hey. We gotta protect user data and make sure that right. So the the the areas of concern are different. So how do you start to address that? And then from there, where do you start to dive in and make sure that you've got a sound strategy for making sure that you're meeting those requirements?
Emil Kampp [00:35:38]:
Yeah. That's a good question. So for let let's just take voting because it's, up and hot and a topic these days. Right? So let's just start there. Always. So one of the one of the things there is that you definitely want so, like, use some privacy. There's a few right there. It's, like, the the whole use of privacy, and then there's these Vogue secrecy, and then there is the use use it needs to be receipt free and, like, there's a bunch of these, academic properties of a vote and an election.
Emil Kampp [00:36:16]:
Right? And you go through each of those and you you effect you just sort of sit down and you start at 1 and then you try and figure out how to do that. Right? This is really the job of the cryptographer. Like, how do we get this organized so that we match all of these? And then once we have a viable, protocol that will do this, then the developers come in and we look at the protocol and we go like, okay. This thing is about encrypting the vote. That needs to happen as early as we can possibly get it to happen, which means it needs to happen in the voter's device. Right? So we need to generate, we need to pick a pick a a curve, a crypt a cryptography curve, and then we need to basically generate keypads in the device, and we need to encrypt it as early as possible. Right? And so you go you go through sort of, like, rigorously and slowly and just sort of reason about when will this part of the process happen and where does it need to happen, sort of more on a timeline. Then we need to transmit the vote.
Emil Kampp [00:37:30]:
We transmit the vote to the server, and then the server needs to basically allow if you need if you need to allow tracking or if you need to allow receipts or whatever is that property you need to have, then you need to figure out a way to provide that at this point. And then you go on to something like anonymization, which is effectively where you take one envelope out of the other envelope, right, to do the the analogy. That needs to happen in a way that you can prove that it happened. So you need to have a good trace here. You need to have good logs and traces that you can publicly show. And then you know? So you sort of move through the chain sort of like the the event of an of a thing. Right? It's the same. And so you you sort of need to reason about where is it gonna happen.
Emil Kampp [00:38:30]:
That gives you some ideas of should we need do we need a web app? Do we need a mobile device, mobile app? Do we need do we need to run cryptography? Do we need to run something advanced on the user's device, or is the user's device just a portal? And and that gives you sort of ideas on or some insight into how this should be organized, how this should be tested, how this should be planned out. Right? But most importantly, for for voting is is basically making sure that everything is tracked, and, we grabbed part of the concept of blockchain, which is hash chain, right, to basically hash everything together so that you can't move stuff in the middle without everybody being able to know. Right? Things like that. For banking, right, you wanna you wanna make sure that see, the the point is to get money from a to b. So you wanna reason about well, first off, how am I gonna get the number that they intend to send? Right? Like, where does that come from? Where does the where does the sort of outgoing amount of money come from? I know. This is the second one.
Charles Max Wood [00:40:15]:
How many
Emil Kampp [00:40:17]:
It's it's hard to be home alone, with, with kids. But so you wanna basically say, okay. So where does the money come from? And then you wanna reason them out. Well, how can I prove that the money that came from there that that I got it right? Like, how can I make sure that the intent was to send it to me? How can I make sure that the amount was correct? Right? Because, like, like we talked about earlier with TransferWise, for example, there is a lot of web scraping. There is a lot of this, sort of craziness going around. Less so now, more so then. Right? And you wanna make sure that you have either done it a few times that you're quite confident that you've run it on big datasets. Like, you need to be certain that you can pass these things correctly.
Emil Kampp [00:41:10]:
And then you wanna sort of grab that information, and then we get into the whole banking of how do you then move them. There's, like, Swift, and that's pretty taken care of. The only problem here was that it was slow. Right? So what we would do is that we would basically assume that if we could read the intended money here, we could effectively pay it out here immediately and then just do the in between transfer later. So we would basically have big stacks of money around the world, and we could immediately pay out locally, or take money in locally, and then we would, transfer later. So in that case, it was more a question of, can you verify the amount here, and can you verify that it's the sort of the correct amount and the correct intended amount? And can you then make that payout over here? Okay. It's common. So then can you do the payout here? Is there sufficient funds? And then after that, you sort of say, okay.
Emil Kampp [00:42:19]:
Now we've we've effectively solved the user journey. Right? We've taken money in. We've paid money out. We're effectively, the user is happy or both users are happy. Right? Nobody knows that we didn't actually move the money. And then we can take, you know, take it slow. Just do it properly. Make sure that we interact with the swift network.
Emil Kampp [00:42:39]:
That's a bit slower. So you need to have a a buffer in, in both ends to do it. Right? And the English is more simple. English. And the the most the important part here is the ability to scrape. Right? So one of the critical points here was that you have to be able to always scrape these websites, in some capacity, and you have to actually write your your code. Back then, it was, you know, scraping. You have to write it quite robustly and sort of reason about what happens if the layout changes because banks change the layout surprisingly more than you would think.
Emil Kampp [00:43:25]:
Mhmm. Because they don't care about your ability to scrape it. That's not their business. Right? Which is fair. It just means that you have to be you have to think about the robustness of of a scraper like that because what happens if the if the item is not classified the same if the dom element doesn't have the same class or the same ideas it used to have. Can we find it by reasoning with words? Like, can we figure out well, if it's like, if it says some, can we maybe then figure out what's over here next to it? Because that might be the number I'm looking for. Right? So there's sort of, like, reasoning like that that that goes on. That's yeah.
Emil Kampp [00:44:07]:
And then there's accounting. Like, a lot of accounting, a lot of keeping tracks just for legal and compliance reasons on where is the money going, who's it going to, KYC, all of these. There's a bunch of tooling after that, right, Which is which is sort of where I was introduced to this whole compliance thing. Like, how do you make sure that the people you're operating with are in fact the people that they say they are and are in fact the right people that you're supposed to be operating with. Right? So that was also a journey. Like, how do you even know that? How do you make tooling for them to identify themselves at the latest possible time so it's the least amount of sort of burden or the least amount of friction, and make it as easy as possible. Right? There's a bunch of providers out there that that can help you do that. You can also do it yourself.
Emil Kampp [00:45:02]:
It's not super complicated, but you are now dealing with people's private information because now you maybe have a copy of their passport. How do you deal with a copy of a passport? You have to be you have to sort of be like, okay. Where do I store this? How do I make sure that it's encrypted? How do I make sure that none of the none of the developers can access this? Right? Because you don't want developers to just SSH onto the box and just be like, oh, let me just download all the attachments. So you need to be you need to make reasons, you know, sort of need to think about how do you then protect this data from yourself in some way. Right? Yeah. So that it it goes deep and it goes far. Right? You can talk about this for a long time, and about all the reasonings you have to make. But at some point, you also have to protect your systems from other developers, from other, also high level people in the organization needs to be kept out.
Emil Kampp [00:46:01]:
Because the higher up you are, the more the spearfishing you are or whaling you are effectively the subject of, and the greater the risk. Right? So you want people to not have access. Maybe you even want the CTO to not have access. Like, there's a bunch of considerations like this. In in the banking one, I was a senior developer. I had access. Then we had the one of the compliance officers, she had access to to the swift network. And then, but but the CTO and the head of sort of the the COO type person didn't.
Emil Kampp [00:46:46]:
Right? Because that that would it post too much of a risk. So there's also, like, these human factors you have to think about, when you reason about this the security of a system that that actually impacts, and it has something to do for real world people. That was a very long spiel. Sorry.
Charles Max Wood [00:47:08]:
Yeah. It's fascinating when there's so much on the line. Like, most of the apps I've worked on, like, if it went down, it's, like, yeah, fine. Okay. It's not ideal, but it's not the end of the world. I kinda wanna pick up on something, you mentioned quite a while ago, actually, on on about type checking and Sorby and stuff like that. Like, I'm very much a fan of the dynamic typing. I I I do have a background in, static programming.
Charles Max Wood [00:47:38]:
Like, because I I started as a mobile developer. So I spent a lot of time with Objective C, with Java, and with Swift. And so I do have that background, but ever since I kinda switched to Ruby, I just love that dynamic style, and I just find it easier to work with. But I also get that when it's critical infrastructure, the type checking is just, almost essential. So I'm curious about why you went with Ruby and Rails when, the the infrastructure is so critical and type checking is clearly something that's important to you, and it's kinda bolted onto the language. So what what motivated you to go with Ruby on Rails instead of something that has it natively?
Emil Kampp [00:48:22]:
Yeah. So it is it is, to a high degree about the speed and the very well built testing frameworks that is around it. Right? So even if it's not statically typed, you can test a lot of things, and you can test it in a lot of ways. Typing is just or types is just one way of ensuring that the data is correct. That doesn't necessarily guarantee that the actual data you are trying to read in is of the correct format or whatever. It just means that once you read it in, it will fail because it's not the right type. So it depends a little on what you're trying to do. You can also because it's dynamic, you can then sort of do things like, okay.
Emil Kampp [00:49:08]:
You gave you told me it was a CSV, and it's obviously not a CSV. And it obviously doesn't have a string where you told me a string was supposed to be. But I can see how this can look like a string. Maybe I can just make it into a string for you and move on. Right? Like, you have these options of making things go a little bit fast and a little bit smoother, because you can sort of just fiddle a little around with it. What tends to happen in these systems as they grow older and as they mature is that parts of the core of the system gets replaced with something like Java or Rust or something like that. So that once you have sort of you have a really solid part, let's say, the transfer engine, right, that transfers from one account to the other that that hooks onto the swift network. Maybe you wanna move that into Java or something that's more static, more slow, but more type robust, and more maybe even performance optimized for IO or whatever it is that you need.
Emil Kampp [00:50:14]:
Right? But then if what tends to happen is that you still want the speed and ease of using Ruby on Rails and having only a few developers do a lot of features so you can figure out what's good or bad or what works or doesn't, or do a lot of development around trying out this new standard to verify with the engineers that it will in fact test correctly with this airplane or, you know, sort of having that that capabilities to do feature spikes and to do tests or, like, mini projects. I it seems to me that that is to some degree lost when you use a more as, well, some other language. Right? But but you are right. Like, once you have it down, once you have it locked in, once you don't have any changes there, what tends to happen is it gets replaced with something that's faster or better for suited for the purpose. Yeah. That I've seen that happen. So the reasoning is that you can go really fast with quite few people, and you can actually get something that's meaningful that will affect people's lives, that will better somebody's life, you can get it done pretty quickly. Right? Like, you can get a a design system for to to basically do, design some we're not talking, like, free sketches of of airplanes.
Emil Kampp [00:51:49]:
That's that's not necessarily how you design all of it. That's just the building of it, but you also need to reason about the logic of who does who does what, what are the processes. And you can get something like that built in 7, 8 months. Right? That's as far as I know, that's and and what I could see in the market, right, and what what happened was that we basically went a lot faster than everybody else, right, with the same amount of errors. Like, it it seemed that there was no real drawback to using Ruby in that case. Right? And so it it was a pure benefit to use the the language and the framework.
Charles Max Wood [00:52:32]:
Nice. You know, there's there's really nothing that can compare to the speed of development of Rails. It's, I think we we kinda take it for advantage, like, for granted rather, as Rails developers. It's when you go and look at the outside world is when you realize, bloody hell, this is a good tool.
Emil Kampp [00:52:49]:
Yeah. Yeah. It is quite insane.
Charles Max Wood [00:52:51]:
Mhmm.
Emil Kampp [00:52:52]:
Like, I had I had a colleague who did a, you know, sort of a an AI based, feature spike. You know? Like, oh, can we just timebox this to, like, 3 days? And in 3 days, we had a tool that seems to be working, did integrate with AI, had UI, had tests, were behind the feature flag, and was effectively shippable. Right? It was not, you know, it was not the best or the most robust or whatever, but, like, in 3 days, a single person made a full feature spike that could technically be deployed and could work. Right? That's pretty amazing.
Valentino Stoll [00:53:27]:
I'm curious real quick. Like, I'm in in my experience working with, like, critical systems like this, like, once it's built, there's, like, it's hard to change. Right? Like, and mostly because, like, 2, the requirements probably don't change very frequently either. But, like, how how do you, like, approach in those cases like that change? Like, is is it, like, build something new and and cut one off and switch to the other? Or, like, what what are, like, what kind of systematic approach do you take to that?
Emil Kampp [00:54:06]:
I think so for me, it's certainly about you have to reason about the protocol and the process somewhere other than the application. So you have to diagram it or put it into dance or whatever you like to do, but we need to have a different place to talk about it than in code. Because we need to reason about when somebody has a change, we need to be able to reason about it before we spend time in code, and we need to figure out if the the sort of the signature of that particular change, let's say you want to change one, hashing algorithm with another hashing algorithm. Right? You need to be able to reason if that will break the system just in principle before you start to even try and implement it. Right? So my I find that I it really benefits to have it in a big diagram or have it written out in long form on in a document or, like, somewhere that you can reason with people who aren't developers on whether or not it will work or not before you even try to implement. Because if you can reason that it will in fact work, it will plug in, it will replace. If you can reason about that before you actually see the code, when you come to the developers, it's a much smoother experience to say, okay. I need this piece replaced with that piece.
Emil Kampp [00:55:38]:
And, yes, the hashing algorithm is not the same, but it is whatever based on the same something something, so it will plug in. Right? And then we can either have both. If they plug well, we can replace one with the other and do and you can also then you need to trends or migrate the data. Right? But now that you're migrating the data, it it's really nice to also have that conversation with beforehand and with others who are not developer. Well, we have existing data, and you want me to pluck this thing into this place in this big diagram? What happens to all the old data that went through that point beforehand? Like, what do we what do we feel we need to do about that? Do we wanna, revision it and say that version less than something can never be opened again or can never be edited again, but can only be viewed? Like, what's our reasoning, on that? Right? So having somewhere else to talk about it than in code really helps to figure out if you should replace, plug in, or, you know, make it as a different service that you call sometimes, you know, it whatever it is. I find that that having something other than code really helps. I'm still. From there.
Emil Kampp [00:57:00]:
So again, so it really needs somewhere else to, to have that conversation. Right?
Valentino Stoll [00:57:07]:
Yeah. You know, it it makes me think about the, the old days of Yahoo pipes, like, where you could visualize and, like, in real time update the data flow of of any particular
Emil Kampp [00:57:18]:
Yeah.
Valentino Stoll [00:57:19]:
Aggregate, thing you were trying to collect. Are there any tools like that that you use, to, like, maybe automate this process, or is it just a matter of getting it documented?
Emil Kampp [00:57:31]:
That's, I mean, I've tried a bunch of them. I haven't found one that I really like. I haven't found one that really does it for me. I tend to try and make them diagrams as code because then they can at least get checked into Git, and we can have this version control on top of it. But I I haven't really found anything that really does it for me. I've done I tend to either do whimsical or or something like that. And then it is just a lot of work to keep it maintained, and that's a that sucks a bit, but, you know, it's it's hard to it's hard to do, without. Right? Without somewhere else to reason.
Emil Kampp [00:58:16]:
And especially when you need to talk to because it's not only a a technical problem. Right? It is also a business problem, and it is a regulatory problem. If I'm gonna change this piece of the testing bet for an airplane or this piece of, code for the design software for an airplane, I need to be able to have a conversation with somebody who's not a software engineer about the consequences of that. Right? Because I need to talk to the civil engineer. I need to talk to the test manager, you know, and I need to talk to a lot of people about the consequence of doing it and the way to do it. And are there any consequences outside of my system that we need to have a conversation about? So it's it it is just sort of, like, trying to get as many people, joining that conversation as possible. And for that, it it seems to work better if it's not quite code because then more people can participate in that conversation. Right?
Valentino Stoll [00:59:17]:
Yeah. I think that makes a lot of sense. I I'm still personally trying to find the, silver bullet. And the closest I've gotten is Mermaid.
Emil Kampp [00:59:26]:
Yeah. And that also works quite well, actually. It's just it it works pretty I love it. It works pretty reasonable, and it it is definitely diagram as text. Right? So I've also used that a ton, and it's it's decent enough. I find that it has some problems when you get up to really large size stuff. Like, it's it's hard to get enough in there and still have it be viewable and and understandable as a human. But, I mean, a lot of, like, building these systems and a lot of participating in building them is surprisingly not about code.
Emil Kampp [01:00:09]:
It's about talking to a bunch of people who have nothing to do with software development and figuring out if you're gonna break their lives. Right? Like, are you gonna break something completely insane that you had no idea that you ever were ever gonna touch? Because, you know, in in the case of this, well, I'm gonna change this thing in my, my design software. And then suddenly, somebody in the factory somewhere is totally confused about how to build an airplane because I did something that you know? So it's it's it's really not that much of a software challenge. I mean, it is, but it is to high degree a communication challenge and sort of, like, how do we even communicate what this software does to people so that they can use it and so we can reason about the safety of it, together. Right?
Charles Max Wood [01:01:01]:
Yeah. It it makes sense in the I I mean, I've worked on things at this level of complexity that you have to get right. So I worked at Morgan Stanley and I mean, we weren't doing, like, banking level stuff, but we were doing, like, investment level and, business information stuff. Right? And so, yeah, the world didn't end if we got it wrong, but we kinda needed it to get right. And a lot of it was just a matter of, like you said, making sure that all the people who understand all of the concerns and all the things that it touch are able to weigh in and and be part of the conversation. And that way, yeah, you don't do things without understanding the implications of of how it's gonna ripple. I'm working a contract now, and, you know, I'm still trying to figure out the ins and outs of the system. And they're they're managing funds and stuff like that.
Charles Max Wood [01:02:01]:
Right? And so, yeah, I'm definitely feeling it here where I don't wanna make any changes that ripple through the system and right? And so this is where you when you're talking about, like, testing and documentation and things like that, where the documentation really isn't around, and there are a lot of other things I wind up having to ask a lot of questions, and I'm still not sure that all of my assumptions are correct. So, yeah, a lot of this really helps in the sense of, oh, okay. If if we're gonna nail down robust, you know, reliable things, then then, yeah, you know, all of these things are important. And I think part of my issue with some of it is just that I like to just get in and solve problems. Right? I'm my mentality I was talking to some other friends in the the other day, and my mentality is go go go go go go go. And so I yeah. Having to slow down and, you know, dot all my I joke and say I dot all my t's across all my i's. It it just it's, you know, it it it's it's not my natural state, and I think a lot of developers yeah.
Charles Max Wood [01:03:16]:
They just they just wanna write great code that solves problems, and so the meetings are tedious. The conversations are boring, you know, and, you know, why can't I just go in and just make it work? And the reality is is, you know, yeah, what what we're what we're talking about here. And so I think I think it's not just we have these specific concerns whether the regulatory or something else. I think it's also that there's enough complexity to it to where you really do have to get in and understand the nuance of what you're doing.
Emil Kampp [01:03:49]:
Yeah. And and I think also the the breadth of what you're doing. Right? Like
Charles Max Wood [01:03:54]:
Mhmm.
Emil Kampp [01:03:54]:
I I know this for myself, right, when I was younger and and started developing code. Right? It was like, well, there's only one truth. Right? Like, either it's 1 plus 1 is 2 or it's wrong. Right? But that's just not the case when you have these really big connected systems. Right? Where, like, well, what happens if I change this protocol? Well, now it's right because it's a different protocol, but now it's wrong because it's a different protocol. So it can be both at the same time. Right? Because because now, you know, all the other people who used it for one protocol can't use the other, and, like, it just gets really messy. I think at some point, I realized that I liked doing documentation, and that really helped to sort of, like, make me be confident that this could be some way you can do a career.
Emil Kampp [01:04:49]:
Like, you can do a career in something that's not obviously developer ish territory. Right? Like, because I actually enjoyed doing, like, architecture diagrams and reasoning outside of the code. And all of these things I found was pretty interesting, and that makes it easier for to be me. Right? Like, it it's more fun to have that conversation with everybody to figure out, well, if we do this thing over there in wherever we are operating in Australia, that's fine. But now we've done this change. So what happens to our operations in Austria? Like, is that the same tax system? Can we do both? Like, it just it it can become really complicated when you have when you're operating under multiple jurisdictions and multiple governances. Right? Then that that can become complicated as well. And, you know, you're doing currency conversions or you're doing unit conversions or you're doing all of these things.
Emil Kampp [01:05:52]:
Like, it's so hard that I know that even, the space agencies are getting it wrong, and sometimes their stuff just won't work when you get it into space. Right? Because somebody miss, converted inches to centimeters. Right? So it is just and that should, like, to if you go to a a fresh out of college developer and says to him that I need a unit unit conversion system, a unit conversion method that can go from inches to centimeters. That's pretty straightforward. That's a that's a pretty straightforward ask. Right? And you could do that method pretty quickly. It's not really super hard. What is really super hard is getting people to use it.
Emil Kampp [01:06:41]:
Right? So you need to build a unit conversion method that is easy to use, fun to use at hand when you need to use it, so that you will in fact use it. Because otherwise, you're gonna do it in your head, and then it doesn't really matter that you build a piece of software. Right? So, like, it it there's a lot of other components to to this than just the software. Like, there's a lot of things, like, you know, there's a lot of backup, a lot of everything, but it is also about making sure that the people who use it use it correctly. And the way to do that, like we talked about before, like, with the TSB and everything, it's not to blame the person for not using the thing. It's to blame the thing for not being available when it was needed. Right? Mhmm. So you need to be like if if I have a unit conversion thing and it's hidden behind, you know, a multiple 3 dot options that you need to figure out 5 levels in, that's the wrong place to have it then.
Emil Kampp [01:07:38]:
Right? You you don't blame the person for not using it. You blame the system for not putting it in his face when he needed to convert the the measurement.
Charles Max Wood [01:07:49]:
Yep. I I like I like that. Just from the standpoint of I mean, I guess I guess where my brain goes is is everybody has the responsibility to understand the system and how it works. Right? And so everybody has the responsibility then to, you know, to to be aware that that this method exists. But at the same time, if I create it, then it's also my responsibility to make sure everybody knows it's there and knows how to use it. And I also agree though that if you make it easy or natural to do the right thing, then people will generally do the right thing.
Emil Kampp [01:08:32]:
Yeah.
Charles Max Wood [01:08:32]:
And so yeah, I I I think I agree with you, but I don't wanna absolve anybody of the responsibility of learning to do the right thing either.
Emil Kampp [01:08:42]:
No. No. I mean, that's not what it I mean, you can you can absolutely end up being blamed for crashing a plane. Like, it's not entirely out of the realms of like, the pilot can be blamed. Right? But it is also about looking at, did we actually provide the correct tuning at the correct time with the correct intensity and all of this. Right? And it's the same here, like, well, the system didn't work. Did we provide enough? Did we scale enough? Did we have enough compute resources? Did we have enough database storage? Was it just so slow that people couldn't use it and it crashed and somebody couldn't get their vote in? Because if somebody couldn't get their vote in, then, you know, we sort of we get into the territory of, somebody prevented me from voting situation. Right? Mhmm.
Emil Kampp [01:09:34]:
So it like, yeah, they could have done it again. Like, they could have just waited 5 minutes till the peak was down and voted again. But that's, like, it's also not really their fault. Right? Right. So it's there's a lot of there's a lot of sort of, like, ancillary nondeveloper stuff to talk about when when designing. I'm pretty sure there is on all systems. Right? But the the fun thing here is just that this happens to include other engineers, which is very fun. And it happens to include things like cryptographers and, people who actually like politicians or whatever it is.
Emil Kampp [01:10:16]:
It's just I like the types of people that I get to talk and interact with, right, in in this rather than, I get to talk less with marketing and social media and more with, like, people who build spaceships and airplanes. And I find that that that's basically what drew me here. Right? Is this idea that I get to talk to other people who also have this engineering spirit and this idea to build something in the same mindset that I do.
Charles Max Wood [01:10:48]:
Awesome. Alright. Well, I think we're gonna go to picks because we've we've been talking for an hour and 15 minutes. This is cool though. If people want to connect with you or if they have questions about what you're doing, you know, maybe you wanna pick your brain about some of this stuff. Yeah. How do they find you on the Internet?
Emil Kampp [01:11:08]:
I think go to LinkedIn. That's, I I have more or less tried to ex absolve myself of all social media. So LinkedIn is probably your best bet. It's my name. It's right there on the screen y thing. There it is. And then also, go go on Discord. You can ping me there.
Emil Kampp [01:11:28]:
PM me, like you did. I'm both on the Kamal, the Rails, and the Ruby servers, so you can just go find me there. It's e k a m p p. So just my initial letter and my last name.
Charles Max Wood [01:11:46]:
Nice. I didn't know that there were Ruby and Rails Discords. I'll have to go find those.
Emil Kampp [01:11:52]:
I can, I can send you invite links?
Charles Max Wood [01:11:54]:
That that would be perfect. Cool. Alright.
Emil Kampp [01:11:57]:
There is there's one for, like, Rails development. That's the Rails one. And then there is one for Ruby, which is more like application support and, like, oh, I don't understand what this method does or I need to change this. Whereas the rails one is more for, like, I need approvals or reviews of PRs for, the actual, call. Right? So that's more about supporting of that. So it's less interesting if you're trying to solve a business problem, but it is kind of cool if you wanna just peek at what the others are doing.
Charles Max Wood [01:12:29]:
Nice. Okay. Well, let's go ahead and do some pics. Valentino, do you wanna start us with
Emil Kampp [01:12:35]:
pics? Sure.
Valentino Stoll [01:12:39]:
I've got nothing but AI picks today. There's been a lot of activity in the Ruby world. Andrew Cain has released, yet another AI gem. Bless his heart. It's called Informeris, and it basically lets you run, inference in Ruby on, Onyx models. So lots of Hugging Face models are Onyx supported, and you can now just run inference with those. And he also has transformers RB if you are interested in running inference on non Onyx models. So now we have, like, a complete picture.
Valentino Stoll [01:13:15]:
Hopefully, those, there are some some missing pieces still for larger models, but, it has a ton of pipelines already built in. So, I would suggest checking that out. The the next pick, somebody built this awesome chat app where, they've aggregated all of the Rails and Ruby, like, articles and blogs and guides and everything like that. So you can basically rag the entire Rails community ecosystem,
Charles Max Wood [01:13:49]:
in
Valentino Stoll [01:13:49]:
a in a chat GPT style interface, and it's it's remarkable. And it does, like, citations and everything like that. So you can see the original article that reference whatever it was you're interested in. It's really awesome. It's called Ship on Rails, and it's built with this, really cool tool called Nausea, which provides basically, like, a a Rails app ready to go so that you can ingest your own documents to rag against, and do a very similar thing. And it's really cool. So I'd recommend checking all those out.
Charles Max Wood [01:14:25]:
Nice. How about you, Ayush?
Charles Max Wood [01:14:29]:
Not sure what to take for picks this this week. I haven't really thought of anything. It's a it's a it's a it's a cheap one, but I'm gonna go with Blue Sky because I'm quite enjoying Blue Sky in the last week as I finally signed up for it. And a lot of Ruby folks have, have signed up recently. It's quite a lot of activity on there. I'm quite quite enjoying it on the social network. Hopefully, it can stay the way it is. I'm active on, both Blue Sky and Mastodon for now.
Charles Max Wood [01:15:00]:
But, yeah, we we'll see what happens because Blue Sky is still a VC funded, privately held company. So, we'll have to see how long if things remain good. But for now, I'm quite quite happy with it. What else can I pick? I haven't really watched anything or anything new recently, because I think I I had picked the TV show called shrinking, which I did the last time, so I don't wanna pick that again. I'll go with the music pick because I always go with the music pick when I'm when I'm struggling. What's in front of me? The new tears for fierce album. Check this one out. It's really good.
Charles Max Wood [01:15:36]:
It's a live album with a few new songs, from eighties pop band Tears For Fears. So, yeah, I think I'll I'll go with those.
Charles Max Wood [01:15:48]:
Awesome. I've had a number of people tell me I need to join Blue Sky. So, anyway, let's see. I'm gonna throw in some pics. So I played I played a new game. I actually played a couple of new games this week because the game convention that I usually teach games at is, is gonna be on Saturday, and so I need to be able to teach these games. So I'm gonna pick one of those. It's called MLEM, l m l e m.
Charles Max Wood [01:16:31]:
And, essentially, what it is is it's how do you describe it? It's it's Cat Astronauts. And the way that it works, it's it's a relatively simple game. It was almost too simple for me, to be honest. But if you're kind of a casual board gamer, right, it's it's definitely approachable. So, let me pull it up on BoardGameGeek. But yeah. So what the way that it works is you have astronaut cats that are going going into space, and they're basically occupying planets and moons. And so you pick one of your your cats, put it on the rocket, and somebody's the captain, and they roll the dice.
Charles Max Wood [01:17:28]:
And if they're if the dice match what's, on the space on the board that you're on, then, you can choose which dice to use in order to move up the board. Right? And so if you if you have boosters, boosters move you up for free. The other ones, you have to give up the dice in order to move. And so if you ever get to the point where you roll the dice and you, you can't use any of the numbers, then you can't push the ship. And at any point, people can get off the ships onto moons or, planets, and you get points for having the most cats on planets. You get points for every moon you occupy. If you go all the way to deep space, then you get bonuses for that. And then the different cats have different abilities.
Charles Max Wood [01:18:24]:
Right? So, one of the cats lets you start further up the board. 1 of the cats lets you, you know, lets you get off the ship after it crashes. Right? So you can go to the planet it's next to or whatever when it crashes. One of them gives you a bonus, doubles your points for the planet it's on, one of them doubles the points for the moon it's gone it's on. Anyway, you get the idea. One of them doubles the points if you're in deep space, but deep space is hard. I've played it once, and we never made it to deep space. We got close a couple times.
Charles Max Wood [01:18:57]:
But then I was talking to some other people who had played it a couple of other times, and one of the times they played, they made it to deep space, like, 3 or 4 times. So, anyway, so if you're the captain, you're rolling, and then you just rotate who the captain is. If the captain gets off the ship, then the next person in line just finishes out that round. Anyway, that that's basically the game. BoardGameGeek has it as a at a 1.68 weight. It says ages 8 plus. And, yeah, I I have an 8 year old. She could definitely play it.
Charles Max Wood [01:19:31]:
It says 30 to 60 minutes. That's probably about accurate. Plays 2 to 5 players. I played it with 4 players, and that felt pretty good. So, it's Mlem, m l e m, Space Agency. And, yeah. Anyway, check it out. And then yeah.
Charles Max Wood [01:19:51]:
I'll just put a link in the comments here. And then, let's see. I'm not sure if I really have any other picks. I know I've kinda been doing other things and getting involved in other things, but, I guess the the big thing that that I went through this lately is I went up to a retreat up in Sundance, which is actually, like, 45 minutes from here. You heard of the Sundance, film festival? They do actually do that in Park City, not in Sundance. But from here, it's about 2 thirds of the way to Park City. And, anyway, it's it's up Provo Canyon, and these guys were running essentially a mastermind for 3 days for Christian entrepreneurs. And so we we did a lot of faith based stuff.
Charles Max Wood [01:20:53]:
We did a lot of business based stuff. And it was amazing because they just kinda helped me get clarity on what I wanna do and who I wanna be and what kind of difference I wanna make. So just keep an eye out because I I have stuff in the works. But it it was terrific, and I I really want to just put forward, I guess, that just connecting with other people who are doing what you're doing or who are, learning what you're learning or who are maybe a little ahead of you on on the path you wanna go down. It's so helpful. So, so helpful. You know, kinda like what we were talking about with, like, the Discord servers. Right? So much help helpful stuff there.
Charles Max Wood [01:21:37]:
So, anyway, those are my picks. Emil, what are your picks?
Emil Kampp [01:21:43]:
So recently, I'm I'm gonna pick a book. I recently read, a book called How Big Things Gets Done, and it's by, by a Danish guy called Ben Flubia. So what he did was that he went I don't know if you guys know it, but he went through statistical data on overruns on big projects, like how far behind is an IT project in general, how much of an overrun on cost do you get, how much of an overrun on time do you get. And he goes through sort of, like, these different industries and different, factors that play into to why why big things fail and why some big things gets done. Right? Like, what are the ways to make sure that big things gets done, and what are the pitfalls and everything? And he goes through everything from the Sydney house, Sydney Opera House to, like, big government IT projects that fail. It's just super cool because it's very sort of data driven, and he has spent his career collecting data on why big projects fail, why all kinds of projects fail. So it's a really interesting, sort of window into do this if you really wanna mess it up. So by inversion, if you don't wanna mess it up, don't do all of this.
Emil Kampp [01:23:07]:
Do do the other thing. Right? So that's kind of an interesting, statistics to have. And so, yeah, how big things gets done. I found that was, really cool.
Charles Max Wood [01:23:19]:
Awesome. Alright. Well, thanks for coming, Emil. We're gonna go ahead and wrap it
Emil Kampp [01:23:24]:
here. Yep. Thank you for having me. It was a pleasure.
Charles Max Wood [01:23:27]:
And until next time, Max
Emil Kampp [01:23:32]:
out.
Embracing Human Factors and Rapid Development in Ruby on Rails Systems - RUBY 661
0:00
Playback Speed: