Today on the show, Jonathan and Will talk with Vince Reed, Director of DevOps at Polygon Technology, a decentralized Ethereum scaling platform that enables developers to build scalable user-friendly dApps with low transaction fees without ever sacrificing security. They discuss how to build and scale a company that has grown from 40 to over 600 employees in 36 countries in the last year, implementing frameworks, communication, automation, security, and much more.

- Addressing the unknown challenges of scaling and securing blockchain networks
- What skills are the same in blockchain as other industries
- What skills are unique to blockchain compared to other industries
- Building open source software
- The future of Blockchain and DevOps
- Working at Polygon- Building a remote-first company

Will

Jonathan

Vince

Welcome, everyone, to another exciting, thrilling episode of Adventures in DevOps. I'm your host today, Will Button, joining me in the studio, my co-host, Jonathan Hall.

 

Hello everyone!

 

and our special guest this week, Vince Reed. Welcome, Vince.

 

Thank you. Greetings, folks.

 

So, Vince, you want to give us a little bit about your background?

 

Sure, I am the director of DevOps at Polygon Technologies, a blockchain company that has grown explosively over the last few years. Two and a half years ago, we were 40 people. Now we are about 600 people in officially 36 countries with a blockchain that has a token that is in the top 15. tokens worldwide and has an evaluation of depending on the day and market conditions somewhere between five and twenty billion dollars. And we are busy growing and scaling and finding opportunities to allow people to use our blockchain to provide various tools and services. And before being the director of DevOps at Polygon, I worked for an AI for radiology startup, which was considered one of the top 50 smartest companies by MIT's Technology Review. And the goal there was to eliminate 90% of the data screened by radiologists, because it mostly what they technically deem unremarkable, and just gets in the way of them screening the things that actually matter that can save people's lives. And the big challenge there was just the sheer volume of image data that needed to be processed and managed.

 

It's funny because I worked in the same field as well. And it's amazing how much time and effort and work is put into just boilerplate language about reading a radiology image just to keep them legally out of trouble.

 

Yes, that and the fact that every specific medical entity involved has their own lexicon for describing all the terminology and for describing the process used for evaluating those images. So there's actually business out there whose job is just to map terms from one provider to the next so that they can actually have a conversation about the actual content. of the data.

 

Yeah, good times. So I wanted to ask, when you started with Polygon, there's two huge things that stand out to me. Number one, just the sheer growth in terms of number of people

 

Yes.

 

has got to be tough to take on. But then blockchain itself is a new technology that doesn't really have defined standards and like. known answers to the questions of how do we scale this, how do we secure this. So what was your thought process and approach whenever you started this and how do you break that down and prioritize, like, where do you start?

 

Sure. So having been around the technology block a few times, the first thing I noticed when going in the door is that there wasn't very much in the way of process or documentation or procedure across the organization for any particular task, role or purpose. I was aghast to find that a development team member could say, hey, this is done, let's ship it. And then the minute they type that in a Slack channel, members of the DevOps team would start deploying it. It's like, wait, what version number is that? What's the release plan? What's the rollout plan? What's the recovery plan? Who are the stakeholders? Have they been notified? Where are

 

Yeah.

 

we watching for outages? and 20 other questions, and they're like, oh, we should be doing that?

 

I believe that's

 

And

 

called Yolo Ops. Ha ha ha ha ha ha ha.

 

yes.

 

the

 

So job number one was to sort of define all the processes and procedures that should be in place, start creating documentation for those, and then walking the team through training and scenarios to. some in some cases gradually for specific details but overall to rapidly start implementing those controls so we had cleaner release processes and cleaner troubleshooting efforts as a result. That was number one but then on the organizational side there were similar questions and concerns and when a company grows in hyper growth mode It's hard to keep up with the current state anyway. But when the basic state is that very little is documented and things grow that quickly, you end up with a game of telephone just to try to get anything done. Hey, who does what? You ask person one who says talk to person two who says talk to person three and on and on until you get a name. And that name may be the actual implementer of said tool process or service. or they may be the owner of the team who does it, or some other such answer. And I found that we were spending a large amount of time just talking about work rather than actually working because of all that missing documentation, organization and structure. So then I set out to start addressing some of those types of structural issues.

 

and

 

And

 

the service

 

so,

 

of

 

The basic

 

the business.

 

answer is

 

Amazing

 

frameworks,

 

answers.

 

frameworks,

 

Great

 

frameworks,

 

work, great

 

frameworks

 

work. Great work.

 

for following processes, frameworks for documenting what is to happen and how it's to happen, and frameworks for how the team members communicate with each other and communicate status so that everyone could agree on where the conversations would happen, which helped way more than you would expect or believe. that was the most important set of things to work on first, is just agreeing how to talk, where to talk, and went.

 

Yeah, because a lot of those people were in 36 countries. So there's some time zone issues to deal with there, which means a question could go a full day before you get

 

Mm-hmm.

 

an answer. And then the answer might be, oh, yeah, that's not me. Sorry.

 

Yep.

 

And so now you're back to another day trying to get another

 

Right?

 

answer.

 

Time zone issues, definitely. But then also issues with the fact that it's web 3. So working locations matter. And people have various levels of experience ranging from, this is the first time I ever had a real job before, to, Actually, this isn't even my real job yet. I'm just an intern to, hey, I've been doing this for a couple of decades, and I know some stuff, and everywhere in between. And trying to get them all on the same page and following the same plan and methods to get things done is kind of a big deal.

 

Yeah.

 

So it was a lot of work about how to work and how to communicate about working. in order to free up the bandwidth to actually do the work and track how the work is being. And then with all of those things done, then we could actually get to hey here are some operational best practices and procedures we should be following and Hey, I've noticed you you do this thing. That's that's a small thing, but you do it every day and Cumulatively you're spending a couple days a month doing this you ever thought about automating it

 

Yeah.

 

Which is devops right and sometimes the answer is that it isn't automated because there's not the bandwidth to do it. Sometimes it's because there's some logic involved that hasn't been able to be captured. Sometimes it's not automated just because it never occurred to them how much time is actually being spent on it. Then the next order of business is basically identify what things to automate and in what order for the biggest payoff. That's kind of what we're working through today, even as we still scale up and support more platforms and tools and services

 

Thanks for watching!

 

and team members using those and products being delivered using them that need to be exposed to.

 

There you go. Piece of cake. Ha ha ha.

 

Fun times for sure. And then there's the whole bit with incident response. When you're playing telephone and you don't have good communications, it's kind of hard to communicate well when doing issue management and crisis management. So then it was about defining tools and processes for how do we deal with things are not behaving as expected, both in terms of how we follow a process to diagnose mediate the problem and how we communicate about that process. Discovering the problem, discovering the root cause of the problem, discovering

 

Thanks.

 

the resolution for the root cause of the problem, and discovering the ETA to finally resolving the problem by applying whatever the resolution is to that root cause. And standardizing the communication about all of that. So that… we don't have to communicate in 12 different channels where in some channels, executive leadership is communicated with high value partners and other channels, some partners are directly pinging engineers saying, hey, what about this? What about this? What about this? While they're actively trying to solve the problem. So lots of things that needed to be standardized and codified so that they could be followed. and smooth out those processes, but we're getting a lot done. So basically it's a lot of metal work in order to actually get to the actual.

 

Yeah, and it sounds like there's a lot of, I think this might surprise some people, given that it's blockchain and Web3, there's still a lot of skills and tools and processes that we've refined over the last few decades that just translate directly over to that and still apply.

 

Oh, absolutely. People look at blockchain as this new mystical, magical technologies. But if you boil it down to its barest essence, it's a collection of globally distributed nodes of distributed databases that are collaboratively run and interact instead of being owned by a single enterprise. Basically, you have an entire community that's running a shared distributed database. if we boil it down to its basic technologies. And there's precedent and documentation and process for managing distributed systems and all of it applies. Yeah,

 

Right on.

 

what's different is, hey, we're doing it in Go, we're doing it in Rust, excuse me, and we're doing it with new APIs and applications and tool sets on top of that distributed database. But it's lots of fun, for sure.

 

Jonathan, what do you think?

 

I'm not really sure where to start. You know that I'm kind of a novice with the blockchain

 

Mm-hmm.

 

world. The

 

Hehehe

 

universe is kind of nebulous to me, so I don't really have a lot of hooks to hang questions on.

 

Okay, well,

 

Yeah.

 

to that I respond, well, back in October, so was I.

 

Ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha ha

 

Hahaha

 

My background is pretty long, as is the case for many of us, and my first exposure to crypto in general was because of the information security and data privacy aspects and, you know, I'm one of those people who probably has some substantial amount of Bitcoin in a closet somewhere that I've forgotten about because I was

 

Yeah

 

involved early when it wasn't worth anything and it was easy and said, ah, this isn't going anywhere. I'm going to go do some other stuff. And then when it started to get really interesting and be widespread and usable, was actively paying attention and then in the 2015 to 2017 time frame all the well-known big name exchanges and several who no longer exist were coming online and the surest way to divest yourself of your crypto assets was to give them to an exchange because they're actual malice. They were going to lose your assets. Some of them, the founders said, hey, 700

 

Thanks for

 

million

 

watching!

 

in assets in the bank. I think those belong to me now and I'm going to move to a non- extradition country and ha-ha, too bad for you guys. That happened. Some exchanges just had operational issues that led to data loss, which led to asset loss. And then... Some exchanges were badly run and had security breaches and people ran off with assets. And then of course, there's

 

Thanks for watching!

 

all the phishing. So the easiest way to lose your crypto was to host it in an exchange. So in 2017, I actually walked away from crypto completely even though I was actually at the time working for a Forex startup that did currency trading and was adding crypto to their platform. I kind of stayed down the hall from that team and every time they would make decisions about implementation practices and policies and I thought they were going to do something that didn't make sense or was inappropriately scaled, I'd point that out, but declined to join those teams and then later left for the AI for Radiology startup and wasn't really paying attention to crypto at all. Polygon reached out to me and said hey you want to come do this stuff and my first thought was I Don't think so

 

Hehehehe

 

Hahaha!

 

But honestly after taking a closer look at the tools the technologies and the team I got really excited and Back in October. I had never been to a crypto conference before and sure I could name tokens and I can explain what a consensus mechanism is and I could tell you about some of the major tokens and their networks. But other than that, I really wasn't actually paying attention to crypto at all. And I joined Polygon October 18th last year. And by November 1st, I was at my first crypto conference in New York City. And now not quite a year later, I've been to over half a dozen around. world and could have gone to more but hey I do actually have a team to run and real work to do and attending the conferences was good both to help me get my head in space and understand the players and tools and technologies and to find out what the community needs from Polygon in order to help drive adoption and to foster successful use of the blockchain. So I didn't have any background in any of this stuff really

 

Hehehehehe

 

either before last October and just rely on the fact that everything old is new again and that you can always take any technology and break it down to its basic elements and understand it in order to be able to ride herd over this rapidly evolving set of tools, technologies and services. Thanks for watching!

 

What are the main differences that you see now that you've been doing this for a year? How would you describe working in blockchain as different than working in any other technical field?

 

Ah, so.

 

With regard to DevOps, of course, I mean, not

 

Sure.

 

in the wildest interpretation.

 

So first thing is it's distributed, it's public, it's global, so your products always have an audience that you're not entirely aware of all of its members for one. And for two, in some businesses early on, you make the decision cloud or no cloud, or you start in one and eventually migrate towards the other, but by the nature, of what we're doing since it's public and global, it kind of presupposes cloud in the first place and all the operation models are basically factoring cloud cost into deployment. So if you're building a blockchain, if you're running a blockchain, it presupposes cloud, whether that's something like AWS or GCP or Azure, or it is some... private cloud with public endpoints is a separate question, but it's definitely a cloud first environment and that's not

 

Thanks.

 

always the case, even though many people associate DevOps with cloud, exclusively, that's not always the case. But it's definitely so here, that's the first thing. And then the next thing is that it's very much open source. So. we're building products and services and technologies, but those are based on core tools, technologies, and services that are public and open source. So in some cases, we fork them completely and go on our own divergent path, but in other cases, it means we're coupled, in some cases very tightly, in other case loosely, with work produced by open public foundations that we then build on to. create the things we create. And that is interesting because of dependencies and visibility and external collaboration needed in order for that to work well. If there is an underlying technology that's in 60% of the things you deliver and it has issues, it's probably a good idea to have some regular and open communication with the foundation that manages the building of that. And hey, maybe you even actually have conversations within GitHub about the content of the repos and the timing and pacing of releases and things like that. And then another more technical aspect is blockchains. By definition, have a history that goes from inception of those networks, as they call them to the current transactions happening right now. And there are nodes on the network that have every transaction that's ever existed. They tend to call those archive nodes. And in the case of ours, the size of an archive node today is 20 terabytes. So there are some data challenges associated with those having good performance, keeping those up to date and doing remediation work. on them or spinning up new instances of those nodes and having them get current with the transactions that are happening now and absorbing all of what has happened before. So that's a pretty interesting set of challenges to work on. How do you take backups and make snapshots available of the whole history? in a public way for people to consume, to spend up nodes that need that much data, and how much does it cost to run a node that needs that much data, and so on and so on. So there are definitely things that are very interesting about blockchain, both in terms of the working model and in terms of the actual features and requirements of the different nodes being operated.

 

Mm-hmm.

 

Yeah, I think from a DevOps perspective, one of the interesting things to consider there is given that the blockchain nodes can be run by anyone, anywhere at any time. Like in my experience with DevOps, we've always built and deployed the infrastructure for our company. But with blockchain, that infrastructure can actually be anyone from other enterprise organizations. to

 

Okay.

 

some guy living in his mom's basement.

 

Absolutely. It's completely true. We actually have two sets of deployment instructions as a result. Well, really it's eight because for each release, they actually get released four different ways. There's the source for those who want to compile and configure on their own. There are binaries that are just a package in the tarball that you can pick up and deploy and wrap your own configs around. There's actual distribution packages for various flavors of Linux, and then there's Docker images. And there are both small and large ecosystem participants that use all of the above. And so we have documentation on our public website that explains how to use some or multiple of those. Delivery methods to deploy nodes and then we also have conversations with partners and collaborators about Deployment issues they run into trying to deploy at one scale or the other Using one of those for deployment methods and yeah, that's definitely an interesting case as we tune for scale and Automatability and auditability We're producing one set of instructions and tools and workflows purely for internal use and then yet another set for use externally. And we have to keep both up to date and in sync with the supported documentation. This is another

 

Hahaha!

 

big challenge. It's so much of a challenge that we're facing the question internally of whether we actually put documentation team members within the DevOps group or if we get dedicated technical writers assigned to us by the writing teams within the company. And that's a question we're actively. working towards an answer for right now. Because there's so much DevOps work to do within the company that at one point it was suggested that every team acquire their own DevOps people. And we were able to explain why that's a bad idea and argue that we should own all the DevOps people within the DevOps team and maybe in some cases, loan some out to teams to have a more tightly looped where as they march through their various development stages towards production, they have DevOps people along the way to provide design expertise, to provide operational practice and procedure expertise, and to do that as things progress so that DevOps is not a checkbox, and at the end of their development process, they don't hand over something that they've designed in a vacuum. can expect us to support and deploy it, but maybe we don't even use the technologies they've selected. And then arguing that the DevOps team owns all the DevOps members and that teams don't select their own DevOps people and embed them, we basically made the argument that we should not get our own technical writers within the DevOps team because we would be doing the same thing that we're teaching the other teams. It's a bad idea for them to do so. But

 

Right?

 

there's definitely a need for. dedicated writing resources because there's so much to do and so much unique automation and process being

 

What are you? Can you talk a little bit about what your DevOps team and DevOps people do specifically? Because, you know, I mean, we joke about it all the time on the show that DevOps is such a vague term that nobody knows what it means.

 

Absolutely.

 

And you know, the idea of a DevOps team is widely considered an anti pattern. So what do you

 

Mm-hmm.

 

what does yours actually mean? What does that? What does it do?

 

Okay. So, actually, I've discussed this with one of my senior teammates who also happens to be a member of this podcast

 

Uh huh.

 

and

 

Wait,

 

Yeah.

 

what?

 

and is a highly respected and very valued recent member of the team. That's Will here. Thank you for inviting me to the podcast. Excited to be here. And this podcast is one of many reasons we chose to add him. because he has a great voice and is good for explaining complex technologies

 

You

 

and

 

hear

 

processes

 

that, Will? They hired you for

 

to

 

your voice.

 

people.

 

I always said I have the face for radio.

 

Yes, and because he's good at explaining things, technical things to less technical folks. So what do we do? We have five categories of work. In category one is incident response. There are things we've built and made available to the public and anything that's available can have issues. And those issues need resolution. Incidence response covers that. But given the number of things we have deployed, the scale at which they're deployed, and the reach, and the fact that incidents come in three flavors, vendor problems, environment problems, and code problems, we are. making changes to our processes and to our observability tool so that

 

Thanks for watching!

 

when alerts arrive about service issues that we can clearly tell which category they fall into and if it's a vendor problem what we get on the phone with the vendor and we just communicate about it and hope that the resolution is quick and if it's if it's a previously unknown issue, then we add design patterns for working around it to future deployment plans. But if it is an environment issue, that's the preview of the DevOps team. That's things like we need more disk space or we need new firewall rules or those types of issues. We need more network bandwidth. But if it's a code problem, like the reason we need more disk space is because a node client got a bad opcode and started writing 100 times more log data than it normally does, well, that's an application problem and it belongs to the team. So part of incident response is building observability tooling that can differentiate those problem classes and route accordingly. So that's category one. Category two is service request. which is we now have a ticketing system which we didn't have before. And the ticketing system is great because it is a means for all the teams to communicate their requests and for those requests to be populated with the appropriate level of technical detail to be actionable. And then for ETA to be communicated without a bunch of talking about working rather than actually working. So... It reduces chatter and allows more time for teams to request work and more time for us to actually get the work done. And more importantly, it also is a means for me to justify how many people are on the team because I can point to the ticket queue and say, look, every week we run through this many tickets. And even though we ran through that many tickets, there's still this many tickets pending. We need more team members. So category two is service requests. And those are making technical decisions about deployments to come, about operating technologies to be used in deployments. It's about building automations for future deployments and setting up CICD processes so that when teams make new releases, those things can be automatically deployed after some appropriate checks and verifications are done. It's all about taking the things we build and making them available outside and facilitating that availability. So those are service requests. That's category two. Category,

 

Good.

 

and there's a three, four and five. Basically, there is design work for advancing the state of our environment's tool and processes. That's all internally driven. Sometimes it comes from watching what happens with the service request or incident response. Sometimes it's just there are new best practices. Sometimes tools

 

Thank

 

evolve.

 

you.

 

But category three, it's evolving the state of the art in our environment. And then category four is loaning team members to the various teams producing things that are going to get exposed outside and letting them work in a tight loop produce new documentation to have the DevOps team members become subject matter experts in new tools and technologies to help that team. And then category five is security. There's so much stuff to secure, both in terms of operational practices and policies, and in terms of configuring all the tools and environments we use that it now needs its own category. And... So those are all the things that DevOps means for us in R5.

 

Alright,

 

Yeah,

 

sounds like

 

it's

 

a

 

a

 

pretty

 

lot.

 

good list.

 

Yeah.

 

I think one of the cool things that, because I've only been with Polygon for just over a month now, but one of the coolest things I've taken away in that month is how we've approached security. And I hadn't seen this model before, so I think it's worth pointing out and getting your feedback on it. But we have a dedicated security team who obviously has a lot on their plate. But one of the unique things about it is they only have read-only access to everything. So they can scan and monitor and audit everything, but they don't have any permissions to take action on that. So if they identify something, their only response is to log a ticket and then the appropriate team acknowledges and works through and resolves that issue. And then when they close the ticket, the security team can... perform their checks again and decide if they feel the issue is resolved or not. So I thought that was super cool and a really unique approach that I hadn't seen before. What was the thought process behind setting it up that way?

 

So we set it up that way for two reasons. Reason number one is teams have, all the teams have scalability issues and separating, identifying issues and reporting them from actually implementing them makes having variable availability of auditing resources and varying availability of resolution resources more manageable, but even more importantly than those things, what it does is separate identifying problems and correcting them. And it means that the teams that actually use the systems and understand the systems and do all the works on the systems are educated about security practices and policies by being guided through. directives about correcting exposed issues. And that's the best part. We end up with through our ticketing system and through other documents that will produce a body of documentation about how to secure the environment, what has been secured and what remains to be secured. And if the security team just went off on their own and discovered and resolved things, they'd be a black box and the organization would not any security expertise amongst the team members. And in addition to all of the directives and guidance they provide about securing our tools and endpoints and the code bases we work with, they're also educating the team members about being savvy against social engineering tasks. So there's anti-phishing training, there's... and anti-spoofing training. And we're also putting the tools and technologies in place to monitor our brand assets and all of our communications to help secure the work we do and by proxy then secure the assets that people are managing using our technologies. And yeah, so if we didn't separate security, the team, from security to process as we've talked about separately, then the visibility into the security processes and policies being implemented and growing the security knowledge would not happen. But it's a direct result of separating things the way we have. And that also means that the DevOps team members get to be security people, but don't have to own all the security because in an org this size, it definitely needs its own team. There's way too much to do.

 

So which of these five categories is so unimportant you're willing to put Will on it? Ha ha ha

 

hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah

 

ha.

 

Yeah Well, it's definitely not incident response.

 

Ha ha ha!

 

Incident response is just one of those in the moment, follow the process, document what's happening and get it done kinds of things. And it's over and done. And sure, you can make incremental improvements to how that happens. But once you've got a good framework for communicating about how incidents. are uncovered, diagnosed, and then resolved, there isn't a whole lot to do there, whereas all those other categories, there is much, much, much to improve. And so we very much need him in all of those categories. And actually the way we most use him is as a best practices technical reference.

 

they take a look at my work and say, see, this is what you don't do.

 

Right.

 

Hahaha!

 

But there's a joke on there. There's a joke there somewhere. Some people, there's all the conversations about what's your purpose in life, and there's a joke.

 

Mm-hmm.

 

I'm going to butcher it, but it's something like, sometimes your purpose in life may just be to serve as a warning to others.

 

Hehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehehe

 

Yes. Yeah. Sometimes your story is not the story. You're the footnote in someone else's story.

 

Right?

 

Or the object lesson.

 

Hahaha

 

Yeah.

 

So I'm curious what you think the, we talked a little bit about how blockchain is in its early stages, and there's still lots of room to grow. What is the future of blockchain and DevOps look like in the years to come, do you think?

 

Well, people talk about how amazing blockchain is and how performant some of the tools and technologies are, but Visa processes 28,000 transactions per second worldwide. Blockchains are still in the single digit integer transactions per second. usable transactions, although at the database layer, we're talking hundreds or thousands of transactions per second. But there's the difference between database transactions for the sake of blockchain consistency and data distribution versus storing the transactions that are the actual token movements and balance changes and those sorts of things. So. Technology wise, I think there's an evolution that needs to happen to reduce the disparity there so that the performance can increase. And then once that happens, it'll be much easier for blockchains

 

Thanks for watching!

 

to take over some of the functionality that's traditionally associated with other systems like banking systems for credit cards and other stored value systems or doing things like. medical records management, for example, that to pick a problem that everybody hates dealing with. You know, if you grow up in one specific medical system, at least here in the US, maybe it's so other places as well, but they kind of own your data, even though it's data about you and

 

you

 

really is your data. And then when you move to a different system, you want to take that history with you. But there's a lot of process and procedure required to do that and it doesn't scale and it's non-standard, but if your medical records were stored immutably in a blockchain, getting access to them would be pretty simple and pretty easy and you could standardize the formatting and it doesn't matter who your healthcare professional is or where they are, whether they're within the US or somewhere else in the world. Data is there, it's distributed, it's available, and it's public in the sense that the raw record information is there. But if it's encrypted, it still would be safe. There's lots that can be done, but the technologies have to improve. And then where does DevOps fit into all of that? It has to be scaled, it has to be tuned, it has to be configured for reliability and disaster recovery and all of those things associated with any sort of production enterprises of any system. And today blockchains do that just by virtue of the fact that they have lots of nodes and if any particular node goes down another node has the full copy of the data so there it is. But even with that being the case there are still issues with usability, latency and DevOps things can help with all of those once some of the more basic usability and design issues are addressed and not for nothing maybe DevOps is the place where some of those things get worked out too. In our case we know that some of our underlying technologies use public standard database technologies that work well for some use cases and not so well for others and it's an exercise to find better tools that, database tools that are more suited to purpose and graph them into some tool sets and that's a process that DevOps can work with as well. So there's lots for DevOps to do just as there's lots to do to improve the technologies to make them more global and useful.

 

Do you see any opportunity for blockchain to merge with some of the other big buzzwords around serverless, for example? Or does blockchain inherently require servers?

 

So yes, there is opportunity for some of those things, and it depends on scale. So as

 

All

 

I

 

right.

 

was saying, there are, as a fully, as a full network, yes, you need some servers because of the scale, because the amount of data managed, but there are design considerations that could be made to change that. That data would still have to live somewhere, and that somewhere would be a server. can't really get away from servers. But then in other cases, some of the transactions and processing that needs to happen are small and light. So they definitely could be serverless. And there are tools and technologies in use today that do deploy some of those types of use cases and operate in those ways. So yes, definitely possible with the right selection of scenarios.

 

Cool.

 

It's funny because after the last few weeks, in a previous three or four episodes, we've had Matt and Connor and Taylor, all who are leading WebAssembly on the server initiatives. And after chatting with each one of those, and then my work at Polygon, I just keep thinking, man, it's only a matter of time before we run blockchain nodes as WebAssembly.

 

Oh

 

And

 

and Kubernetes

 

I know you,

 

in the browser.

 

right, with Kubernetes in the browser, absolutely.

 

Ha ha ha.

 

Wow. Those are going to be some expensive laptops.

 

hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah

 

The M3 will be out soon, it'll be fine.

 

Right?

 

Hahaha!

 

I'm holding Apple stock. We're good for it.

 

Ha ha

 

Yeah. Nice. That'll do it.

 

I'm sorry.

 

Another interesting thing about blockchains is that since it all runs on Linux is any processor flavor is fine. And we find that in many cases running things on ARM is a much better value proposition. So when we release those release artifacts, I was talking about earlier, there are x86 versions, of course, but there are also ARM. and they actually get used. So unlike the world at large where ARM is still kind of an exception or used in very narrow and specific use cases, blockchain uses whatever CPU you want to throw at it, which is an interesting operational vehicle and is definitely more exciting for DevOps team. We get to use Graviton in some cases. when talking about acceleration, we're considering talking about deploying things on GPUs and so on and so on. So that's another interesting aspect of DevOps blockchain technologies.

 

Yeah, there's definitely tons of opportunity to leverage skills that you've spent a number of years building to run traditional applications. But then I think just the inner nerd in all of us wants to play with new stuff as well. And there's just tons of opportunity for that here, combined with the things that we've learned in the decades past of. Okay, it's cool to go play with it, but I've got to capture a little data here to see if this is worth pursuing or not, or if it's actually making things better.

 

Absolutely.

 

Cool. Anything else we should talk about?

 

Yes, absolutely. We are hiring. The DevOps team is not big enough yet. And if you visit polygon.technology.careers, you will see DevOps team member roles available in six different time zones. Because we follow the sun and adding more members and more time zones when incident response is needed, no one has to get out of bed to do it. And it means that we have more bandwidth to deal with those five different work streams we have to deal with all in parallel, while still allowing time for folks to go to training, take PTO, and deal with various other personal downtime required items. So come join us. Check us out.

 

Nice.

 

Awesome. A lot of people are working for work, so hopefully they're listening to this show.

 

Yeah, and we get cool swag too.

 

Hahaha

 

See you mate. See you mate. Ready?

 

Nice. Yeah, we definitely get loaded up with the swag and quality equipment to work from. Yeah,

 

Do you

 

definitely.

 

get to work with any cool sort of public personalities, you know, maybe people on podcasts or anything like that?

 

Yeah, nobody worth mentioning.

 

Ha ha ha.

 

I'm sorry.

 

We also have training allowance and home office allowance and a few other interesting perks. So yeah, it's a good place to work. We don't care where you are or where you are. We have a spot for you if you're talented and capable and motivated.

 

Awesome.

 

And you don't even have to be out there.

 

Right?

 

Hahaha

 

Oh, yeah, that's another thing worth pointing out, too, is all positions are obviously remote,

 

Yes, every

 

which I

 

single

 

know

 

one

 

a

 

of

 

lot

 

them.

 

of people are looking for.

 

36 countries and we actually do not have any physical officers. We all meet on Slack or Zooms or Google Maps. which is pretty fun actually. And then occasionally we show up in one place just to terrify each other.

 

Right. to see how tall someone actually is versus how tall they look on camera.

 

Or in some cases, how tall they actually are, voices how tall their voice makes people believe they are.

 

Ha ha ha ha.

 

Hahaha!

 

So Will, how tall are you? Cause you know, seeing you on camera, I'm assuming you're about four or five inches.

 

Yeah, sounds about right.

 

Yeah, okay.

 

Yeah.

 

Mm-hmm.

 

Cool. Ha ha.

 

Yeah.

 

I don't think you have any legs either.

 

No, no, no,

 

Yeah.

 

just chest

 

Just,

 

up, that's all

 

yeah,

 

there is.

 

yeah,

 

Like,

 

yeah.

 

y'all are both old enough to remember this, Mac's headroom.

 

Yeah.

 

Yes,

 

Yeah. Yeah.

 

absolutely. Definitely.

 

Cool,

 

So yeah.

 

should we do some picks?

 

Is that your pick for the week Max Headroom?

 

Yeah, right? Yeah, so if you don't know who Max Headroom is, surely he's on YouTube.

 

Yes.

 

There's probably some old video clips that somebody recorded with their VHS camcorder and uploaded to YouTube, because I think that predates any other technology that would have been used.

 

Yeah,

 

Right.

 

So definitely.

 

yeah. And just for context, it's sort of a, like a 1980s vision of what the future of computers would be like,

 

Yes.

 

sort of. Computer is

 

Yeah.

 

an AI, so it's a comedy, dark comedy, actually. It's, he's a, anyway. Go look it up, it's interesting.

 

Yep.

 

Right?

 

Definitely.

 

So, Pix, who's going first?

 

Pix, you said you were ready. Fire it up.

 

I am ready.

 

Let's do it.

 

So I picked a book a couple weeks ago. I'm going to repick it. I still haven't finished it because I have too much email.

 

Hmm

 

The book is by Cal Newport. called A World Without Email, reimagining

 

Oh wow.

 

hahahaha

 

work in an age of communication overload. And I really need to finish this book because it promises to tell me how to organize my time better and my work so I can be more productive and not interrupted by Slack and email and all that crap all the time so I'm not getting work done. But it's a catch-22. I need to find the time to read the book to learn how to find the time to read the book.

 

Hahaha

 

Anyway, I recommend the book. So far, it's good. And I've read a couple of the books by Cal Newport. They've all been gold. So that's my first repick. My second pick is Cal Newport has a podcast. I didn't realize. And it's gold too. I mean, I've only listened to a few episodes, but it's pretty cool. So he, it's called Deep Questions. And which I guess is a play on his earlier book called Deep Work. So it's mostly listener questions being answered. on the air, so to speak. So I recommend both, especially if you struggle to get work done or to find the time to concentrate, which I know isn't a problem for most people in IT,

 

No.

 

but it

 

Hehehehe

 

is for me. So I recommend the book

 

Outlier.

 

and the podcast.

 

Mm-hmm.

 

right on. I bet you Vince, get any pics for us.

 

Definitely, I got two of them. Number one is Do the Hard Things First by Scott Allen. I'm still reading it. How to Win Over Procrastination and Master the Habit of Doing Difficult Work. And in our case, it's a good way when dealing with complex projects and deadlines to rapidly address all the points of concern so that we can. quickly build confidence in actually delivering said complex technology or more rapidly discovered. Yeah, you know, this seemed like a great idea. We knew there were some challenges in it and this is a no-go because look at this big red flag. We are not going to be able to get around because of time, because of tools, because of coordination challenge. So we can put this and go spend our time where it's better spent right now rather than... six months from now when it's really gonna hurt. So instead of just tackling things in what seems the standard reasonable order, it's go after the hardest challenges first to either build confidence and momentum or fail the project early and move on to the next stages. Why are we talking about that? Because as an org, as we're defining processes and technologies and working on ways to integrate all the technologies. There's some discovery and design of new workflows, tools, and environments, and this is a good way

 

Thanks

 

to...

 

for watching!

 

get to success or failure pretty quickly. And then the other one is a fun one. Randall Monroe, the ex-KCD guy, has a series of books called What If? And the subtitle for that is Serious Scientific Answers to Absurd Hypothetical Questions. And if you do the type of work we do, you have a bet for... the technical and the interesting, and you have a need to definitely goof off sometimes. And this is a good way to do that and learn some interesting, useless, fun stuff in the class.

 

Nice. Right on. Cool.

 

There's a whole series of those.

 

definitely sounds worth checking out.

 

So what you're telling me is I need to do the hard thing first, which is read this book about how to-

 

Right? Absolutely.

 

Hahaha!

 

You got it!

 

He got

 

That's

 

kicked

 

all.

 

off completely. He's just going to go read the book right now.

 

Yeah, seems that way.

 

We thought

 

That's,

 

you

 

yeah.

 

just dropped the podcast

 

Yeah,

 

right now to

 

I

 

go

 

was

 

finish

 

so

 

reading

 

upset

 

your book.

 

about having to do the hard thing first. I was like, I'm out of

 

Yeah,

 

here.

 

right. That's

 

My browser

 

one way

 

just

 

to get

 

crashed.

 

things done.

 

Never

 

And

 

happens.

 

speaking of remote and global and distributed, where are we all? I'm in California. How about you, Will? And you, Jonathan.

 

I'm in Arizona.

 

And I'm

 

any.

 

near Amsterdam, Holland, in Europe.

 

Wow, how awesome is that? Global

 

I don't

 

That's

 

know.

 

remote

 

pretty cool.

 

and distributed right here.

 

Yeah. All right, cool. I've got two picks. And it's funny that one of yours was a repick, Jonathan, because one of mine is a repick as well. I'm

 

Wow.

 

going to start with the other one first. warp.dev, W-A-R-P dot dev

 

Is this for OS

 

is.

 

2 development? I'm so excited

 

It

 

already.

 

is.

 

Ha ha ha.

 

Yes, absolutely.

 

Hehehe

 

No, it's a replacement for the terminal, because we've all been using bash for decades now. So. the guys and the people over at warp reimagined the terminal and it's actually pretty cool. They wrote a new terminal using Rust and it does everything that you would expect the terminal to do but it has like modern features in it where you can copy and paste text and you can just capture the command output easily and my favorite feature is they've got an Because the reason I've never done anything to customize my bash terminal is because I spend a lot of time SSH'd into remote servers and

 

Mm-hmm.

 

as soon as you do that all your settings are gone. But with warp they actually wrap the SSH commands or wrap the SSH session so that it comes back into the customized settings in your terminal and it works pretty near flawlessly. So it's actually been kind of cool playing with that. Um, my second pick is my repick. I've picked it. It's been a while since I've picked it. So I think it's a good time to bring it back up. It's the network state from Balaji Srinivasan. And one of these days I'm going to have to go learn how to pronounce his last name because I'm positive I'm butchering it. But the network state is his, um, it's his book that just talks about cryptocurrency and blockchain and the current state of economics and government

 

than me.

 

and like how all of this plays out into current society. It's an insightful read. I'm on my second read of it because it's just that good. The first time you read it and you're like, wow, and I wanted to go through it again and continue to pick up more stuff and just write down more notes from it. It's I think it's required reading for everyone on the planet. That's how strongly I feel about it. And there's stuff in there that you won't agree with, but I think it's good to think through those problems and have the internal dialogue of why you don't agree with it. So Balaji, The Network State, those are my picks for the week.

 

Nice.

 

Yeah, and I believe we've got ourselves a podcast.

 

I

 

Yes,

 

think so.

 

I think so. Good talking

 

All right,

 

with

 

Thanks

 

you

 

Vince,

 

guys

 

for coming

 

today.

 

on, Vince.

 

thanks for taking the time to hang out with us.

 

Hey, that was fun. Good talking with

 

right

 

you both.

 

on. Cool. All right. Thanks, everyone, and we will see you all next week.

 

Catch you later.