Monorepos, Linting, and CI, Oh My! - AiA 354
Today on the show, Charles and Subrat interview Miroslav Jonas to discuss various approaches related to monorepos, linting and CI. Enjoy this broad conversation as the panel shares their industry insights on these various topics and strategies you can start to implement today.
Special Guests:
Miroslav Jonaš
Show Notes
Today on the show, Charles and Subrat interview Miroslav Jonas to discuss various approaches related to monorepos, linting and CI. Enjoy this broad conversation as the panel shares their industry insights on these various topics and strategies you can start to implement today.
In this episode…
- What are monorepos?
- Migrating and linting engines
- Nx and monorepos
- Utilizing plug ins
- Integration libraries and linting
- Benefits of Nx
- Front end vs back end communications
Sponsors
Links
- Imgur
- Taming Code Organization with Module Boundaries in Nx | by Miroslav Jonaš | Nrwl
- Smart, Fast and Extensible Build System
- Github: meeroslav
- Twitter: @meeroslav
Picks
Transcript
Charles Max_Wood:
Hey everybody and welcome back to another episode of Adventures in Angular. This week on our panel we have Subrat Mishra.
Subrat_Mishra:
Hello.
Charles Max_Wood:
I'm Charles Max Wood from Topendevs, and this week we're talking to Miroslav Yonash. I hope I got somewhere close to that.
Miroslav_Jonaš:
Yeah, actually it was perfect. Hi everyone.
Charles Max_Wood:
Awesome. So Miro, do you want to just remind people who you are and why you're important, all that good stuff?
Miroslav_Jonaš:
Yeah, I don't know why I'm important. As Charles mentioned, I'm Miroslav Janas, but everyone knows me as Miro. I am a software developer originally from Serbia, currently living in Austria. I work for Narwal where I'm one of the core members of the NX. And I'm mostly focusing
Charles Max_Wood:
Nice.
Miroslav_Jonaš:
on Linter and our CI.
Charles Max_Wood:
Very cool. Yeah, we talked to Lars Brink a couple of weeks ago. I guess he has a book on NX. So
Miroslav_Jonaš:
Yep.
Charles Max_Wood:
good stuff, good stuff. But yeah, let's dive in and just kind of start breaking this down because I'm really curious. NX does all kinds of stuff and I'm always like, oh, it does that too. So do you want to just give us kind of the high level overview on NX and Monorepos and then we can kind of go down into the linting NCI?
Miroslav_Jonaš:
Yeah, so I guess most people by now already have some understanding of what monorepos are, but generally in a nutshell, it means that we have multiple projects that are leaving collocated in a joint repository, but they don't just sit together in the same folder, they can actually communicate to each other. So we have a toolkit that can then help them communicate to each other. So we can run the processes that then spawn between different projects. We can see, we can orchestrate them. We can see how they impact each other. So if you change something in project A and project B depends on project A, you can immediately get informed that this change happened. So when you later on run the tests or builds, we can nicely link them together. And by knowing how things are connected to each other, you can also see what parts are not affected. So for example, in most monorepos, including annex, we have commands affected, which then tells you whether your change has affected certain project or not. So by making some change in your repository and calling affected, you can make sure that you're only running. the given command on the projects that might have been affected. It doesn't mean that they were necessarily affected, but there's a chance that they were affected and we can tell which projects for sure were not affected by this. And when you have the repository that has, I don't know, hundreds of projects, this can tremendously improve the performance and your developer experience. Because now instead of running build on, on a hundred projects, you can only run them on two or three projects that might have been affected by the change. And then this also, of course, then saves money on your cloud because your, your processes are then running just a subset of commands. And this is just one part. And of course there's caching, so we can cache certain commands or all of your commands, which means that if you already run the build for a certain project, you don't have to run it again for just imagine standard serve standard serve command first runs the build and then serves the on the VAPX server. Now, if you didn't change anything in the project, you don't have to rebuild it again. You can just use the build artifact and serve those. So this of course saves a lot of time. And the bigger the project is, the more time you save.
Charles Max_Wood:
Yep. It's funny because I remember when Narwhal came out with NX. Of course, I remember also when a bunch of folks left Google and went and put Narwhal together. Then they built NX because Google works off of a monorepo. This is a lot of the same kind of tooling that they would use at Google for their approach. I don't know how different it is from Blaze or Bazel or whatever
Miroslav_Jonaš:
Basil, yeah.
Charles Max_Wood:
But,
Miroslav_Jonaš:
The
Charles Max_Wood:
um...
Miroslav_Jonaš:
concept is similar, yeah, actually. It was, it was, of
Charles Max_Wood:
Yeah,
Miroslav_Jonaš:
course...
Charles Max_Wood:
but...
Miroslav_Jonaš:
Yeah, please go ahead.
Charles Max_Wood:
Yeah, go ahead.
Miroslav_Jonaš:
Yeah, I just wanted to mention that it,
Charles Max_Wood:
No,
Miroslav_Jonaš:
of course...
Charles Max_Wood:
I was just...
Miroslav_Jonaš:
Yeah, Bazel was one of the projects that we looked up to and from which we borrowed
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
multiple ideas. So as you might know, Google is using Bazel. Similarly, Facebook was using Buck and both of these tools back in the days were not open for the community. So you couldn't really try those tools. Once they open it, we become aware of all the features. But the problem was that... Both of those tools are quite complicated to master and it takes a lot of time to configure it properly. Once configured properly, they are blazingly fast and amazing, but it takes a lot of constant taming to make it work to your liking. And NX and all the other modern monoripot tools approach the same problem by giving the same features, but on the other hand, automating a lot of the tasks for you so you don't have to spend time configuring, we provide already nice defaults for you.
Charles Max_Wood:
I was going to say that. I'm just
Miroslav_Jonaš:
Ha ha.
Charles Max_Wood:
kidding. Anyway, so yeah, so as far as, you know, kind of going down to the next level, we've been talking a bit about the linting capabilities in NX. Now does it use tools like ESLint and stuff like that, or do you have your own linting engine?
Miroslav_Jonaš:
No, no, no, we use the SLINT. We used to have support. We still have support for TSLINT as well, but this is being now un-maintained. So we are slowly switching it off already. When you generate new workspace and then when you run migrations, we'll try to push you to SLINT. You can no longer generate a solution with TSLINT and for all the projects that still use TSLINT, we try to remove it. And probably in the next version, it will be kicked out altogether.
Charles Max_Wood:
Gotcha. So, yeah, I'm a little curious what that means for the people who are using ts-lit. I mean, how long do they have before it starts saying, sorry guys, we just don't do this anymore.
Miroslav_Jonaš:
Um, yeah, we, we, we have luckily migrations that automate most of the stuff for you. So you can run migration that will migrate your repository from TSLint to ESLint. There might be a couple of things that you have to do on your own, but like most of the rules will be migrated. Most of the comments will be migrated. There will be a couple of things, especially if you did some custom stuff with TSLint that you would have to migrate yourself, but I would say that most of the work is done automatically by NX. So, it shouldn't be scary for folks. And I really advise people to do it as soon as possible because you're missing all the great features that are coming out in ESLint. And also you're using tool that is not maintained and it's slower and yeah, has more limitations than ESLint. So why use something that... That's half of the work that ESLint could be doing.
Subrat_Mishra:
I think slowly they are transiting all the projects in our organization as well. If you see slowly everyone is transiting from TSLint to ASLint. And I think it's a pretty good approach as well. To get rid of as soon as possible but somehow some project if it is bigger it's hard to change anything.
Miroslav_Jonaš:
Yeah, I guess that's the painful reality that when you have some old legacy
Subrat_Mishra:
Yeah.
Miroslav_Jonaš:
projects and especially if they are large monoliths and it's hard to migrate to anything, but yeah, with automated migrations, it's, it's easier because most of the heavy lifting is done for you. And then you just, once you run the linter, it will complain of all the things that are broken. So usually what we do in this situations is that we migrate and then you run the lint and then stuff that that is breaking that maybe was commented for you or some rules were changed. You just use ESLint ignore to ignore all of those so that you can continue with your feature development or whatever you're doing
Subrat_Mishra:
Hmm.
Miroslav_Jonaš:
and then just kind of set yourself a reminder to come back to those and kind of switch off one of the one by one of those ESLint ignores and fix them.
Charles Max_Wood:
Right. So how does this tie into NX then? And how do you use it in a Monorepo? Well, I have more questions about how to use it in Monorepo, but why don't we just start there.
Miroslav_Jonaš:
Yeah, so, so NX has kind of two approaches. One is we could call it core or learner like approach where we just provide a base NX that gives you all these nice things like effected orchestration, caching, et cetera, but it doesn't give you plugins. So you don't have a plugin for, for Angler, for Cypress, for nest, for ESLint, for Jest, et cetera. So you just get. the core functionality of Monorepo and you can simply ignore install annex to any repository and start using it. You can even install it into some other Monorepo toolkit and it will still work. But then if you want to have a bit more battery included, you can install also plugins and each plugin is targeting specific. framework or library and provides additional information. So for example, we have plugin for Angular, which is in a way similar to Angular CLI. So it provides similar functionality, gives you opportunity to generate libraries, applications to generate components, like all of the things that you would normally do with Angular CLI. And it gives you also, powerful migrations that we have and some additional stuff like generating single component single file components or or creating secondary entry points or module federal setting up module federation. So all these nice things and we have similarly plugins for for react. For example, we have also for just and for for ESLint. So when you install Angular. Normally you get also along with it, just Cypress and ESLint as your
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
unit test library, your integration library and your linting library. And this then gives you the additional executors that you can, which are pretty much just a wrapper around a standard call. So for example, our linter plugin, Executor, it's just a simple wrapper around standard ESLint call, but it gives you this standardization between different plugins. So regardless, whether you're
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
running command for Angular or for Cypress or for Linter, you are always providing the same parameters with the same name. So you don't have to know which library requires what. It's always standardized. And this is what our plugins are doing. So they are... creating nice wrappers around native or not native, but like external libraries, so that it's easier for you as a developer to navigate with them.
Charles Max_Wood:
That's nice, because I always have to create like a command line alias or something,
Miroslav_Jonaš:
Hmm.
Charles Max_Wood:
or you know, an NPM task so that I can remember all the flags and switches I want to stick on the end of that command. That makes it easy.
Miroslav_Jonaš:
Yeah, and one of the most powerful things, of course, is the migrations, because whenever a new version comes out, Angular already has nice migrations. So that works most of the times, but they provide
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
migrations only for Angular. So if you're using, for example, also NGRX or some additional tools that are not necessarily part of Angular ecosystem, you will have to migrate those on your own. And NX by using the plugin provides migrations for all of them. So you run migrations, which will under the hood run some of the Angular internal migrations. It will also run some of the NX migrations and it will migrate not just Angular, but also your ESLint, your Cypress and whatever you have there set up. And it will not just migrate version as it bumped the version, it will also check your code. And most of those migrations have some sort of rules, like whenever version changes, there are some API changes. So we find your code that is using this API and we change it to the newest format so you don't have to do it on your own.
Charles Max_Wood:
Yeah, makes sense. So what
Subrat_Mishra:
If
Charles Max_Wood:
if
Subrat_Mishra:
I...if
Charles Max_Wood:
I want
Subrat_Mishra:
I...
Charles Max_Wood:
to use different... Oh, go ahead.
Subrat_Mishra:
No, no, sorry, sorry. Yeah, I was...
Charles Max_Wood:
I was just going to say, you know, what if you want to use different flags or a slightly different setup than what NX gives you out of the box?
Miroslav_Jonaš:
You can pass your own flags and you don't necessarily have to use anything that plugin provides. You can create your own plugin or not use plugin at all. It's up to you. And you can always specify whether you would like to pass some flags that NX doesn't know about. And normally what for most of the plugins, we have this setup so that you can simply just pass. All the flags that are not known by, by NX will simply be just propagated down to whatever you're trying to call.
Charles Max_Wood:
Go ahead, Subra, what are we gonna say or
Subrat_Mishra:
Yeah,
Charles Max_Wood:
ask?
Subrat_Mishra:
what I was saying is the NX acting most powerful way I can think of is use. I have used each with angular and nest club together. And even you have bunch of project, you create different environment. And now you can say a lot of things between front end and back end. And the, our coding will become a lot easier that way to maintain some variables, maintain some, uh, like some. way some variables can be used in both the places instead of maintaining two separate depository maintaining and like the maintenance will be a headache if you maintain two different one instead of creating a single annex project
Miroslav_Jonaš:
Yeah, definitely. Like when you have backend and frontend, you can create like a library of your domain models that then both of them can consume.
Subrat_Mishra:
Hmm.
Miroslav_Jonaš:
And that is somehow doesn't care about whether you use it for frontend or for backend. So that once you change the contracts, both of them are affected immediately. then you know that
Subrat_Mishra:
Yes.
Miroslav_Jonaš:
you are always in sync that whatever backend is sending that frontend is expecting the exact format.
Charles Max_Wood:
Yeah,
Subrat_Mishra:
Mm-hmm.
Charles Max_Wood:
I guess. Oh, so you're talking about actually testing, say that the front end of the backend are compatible, that they're speaking the same, I guess, language, or they're using the same data structures kind of thing.
Subrat_Mishra:
model also it's fine
Charles Max_Wood:
What was that?
Subrat_Mishra:
I think like the most of the scenario like for example for the for the listener I'm saying suppose you are calling API and the backend is returning some kind of some data some object and are receiving the same object here as well. Most of the time what happened is backend change something you need to update in the front end as well according to that but if you are if you're pointing to the same object and same reference you are getting that If you're changing something, it will be affected in both the places.
Charles Max_Wood:
Right.
Miroslav_Jonaš:
Yeah, when you
Subrat_Mishra:
Yeah.
Miroslav_Jonaš:
have.
Charles Max_Wood:
So you're
Subrat_Mishra:
Am
Charles Max_Wood:
specifying
Subrat_Mishra:
I correct?
Charles Max_Wood:
that contract.
Subrat_Mishra:
Am I correct? Miru?
Miroslav_Jonaš:
Yeah, when you have two separate repositories, what usually happens is that occasionally someone changes something in the backend and they just forget to inform you or they don't think you should
Subrat_Mishra:
Yeah.
Miroslav_Jonaš:
know about this change. And
Subrat_Mishra:
This always
Miroslav_Jonaš:
since
Subrat_Mishra:
happens.
Miroslav_Jonaš:
nothing
Charles Max_Wood:
Hey,
Miroslav_Jonaš:
changed
Charles Max_Wood:
I'm a backend
Miroslav_Jonaš:
in your
Charles Max_Wood:
guy, I have never done that. Ever.
Miroslav_Jonaš:
Right. The worst thing is that you don't find
Subrat_Mishra:
Thanks for watching!
Miroslav_Jonaš:
this usually on your development environment. You find this in production when someone in a couple of days complained that something
Subrat_Mishra:
Thanks for watching!
Miroslav_Jonaš:
is broken and then you have to figure out when did it change, how it was changed. Why is it breaking first of all?
Charles Max_Wood:
Because the project manager told me to.
Subrat_Mishra:
Hehehe
Miroslav_Jonaš:
No, I mean,
Charles Max_Wood:
It's never
Miroslav_Jonaš:
whenever
Charles Max_Wood:
my
Miroslav_Jonaš:
you,
Charles Max_Wood:
fault.
Miroslav_Jonaš:
whenever you depend on human to handle something,
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
eventually it will break. It's not that we are lazy or don't care, it's just it's in our nature to occasionally forget something important. And when you have things automated, like
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
in monorepo that it's just working without your intervention, that gives you a security that things are safe. things know each other. Of course you can break in Monorepo just as well, but you have to work a bit harder to break stuff in Monorepo in that sense that you have contracts that don't match.
Charles Max_Wood:
Yeah, that makes sense. One other thing that I'm curious about, because Subrat mentioned that maybe you have an Angular front end app and you have a Nest back end app. Maybe you have some other stuff in there. And yeah, for the, hey, I'm going to call Jest against this and run the tests, I want to use the same flags because I want the output to look the same. And I generally want it to follow the same rules. But what about for the stuff that isn't the same for like a between Angular and Nest or, you know, Angular and Express or whatever, right? So maybe I have some special thing that I just needed to be aware of on the backend or, you know, maybe Angular, you know, I want to use the Angular CLI or some feature of that for the front end. Is there a way to tell NX, hey, this is different, do this instead? Or add this part of this thing to your... standard stuff. I don't know.
Miroslav_Jonaš:
Um, so normally all of the commands that you're setting up are per project. So you can change per
Charles Max_Wood:
Okay.
Miroslav_Jonaš:
project, whatever you want. Each project has its own configuration, so you can set whatever you need for this project. Additionally, if you're having some external libraries that are, or libraries that are then being consumed by either frontend or backend, again, you can have a dedicated, for example, feature libraries for Angular. that have some parts of UI and then you can have some sort of service libraries for your backend that then provide pieces of logic for the backend. And one cool thing about Linter is that we created the enforce module boundaries rule for Linter with which you can configure using tags which libraries can be consumed where. So you can make sure that... your nest backend will not consume, for example, Angular feature library, because it shouldn't. Or that library cannot import an application. So stuff like this, some of the things are done automatically, like checking whether application is not being imported into library or checking whether there are circular dependencies in your code, this is done automatically by the rule. but some things
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
like whether a certain library should be imported by other library or other project, this has to be set up by using tags. And then you can specify nicely, for example, you can mark all your front end stuff with front end tag and all your back end stuff with back end tag. And then you can say that front end libraries can only be consumed by other front end libraries, not by back end or utility or end to end or whatever other you have set up. And this actually helps with large repositories because it helps your architecture from descending into chaos. You know, the situation where people just keep inferring stuff all over the place and keep importing here and there. And like you find something useful and you just import it in your project. And suddenly you have weird circular dependencies and stuff that depends on each other that normally shouldn't even know about each other. And with this tags, you can nicely. keep your architecture under control so you don't have some weird net of intertwined libraries and projects. And yep, this is.
Charles Max_Wood:
So how does this all, oh go ahead.
Miroslav_Jonaš:
Yeah, I just want to mention that this is the part that I'm mostly working on. So. It's very, very interesting because most people don't even know about this or are not aware. And it helps tremendously because when you have larger teams, you just can't control every commit, every pull request you can. And occasionally in large companies, you have juniors or just people who are new to projects and they don't know what, what belongs where. And usually, even if you have some architecture diagram, most likely it's sitting somewhere in some network drive where people don't. see it all the time and it's most likely outdated. And with NX, just by running craft, you can immediately see up-to-date craft of your entire repository. You can see how things are connected up-to-date. And it also delinter tells you then if you make something, some import where it shouldn't belong.
Charles Max_Wood:
Gotcha. So it keeps you from having NX in the multiversive madness.
Miroslav_Jonaš:
Yeah.
Subrat_Mishra:
I think NX is nowadays also becoming little popular for the micro front end for the state sharing if I am not wrong.
Miroslav_Jonaš:
Yeah,
Subrat_Mishra:
So can
Miroslav_Jonaš:
yeah,
Subrat_Mishra:
you
Miroslav_Jonaš:
we.
Subrat_Mishra:
please just have a please.
Miroslav_Jonaš:
Yeah, we were one of the first to jump on the whole micro frontends ship and my colleague Colm did a lot of work there and created executors and generators for for Angular. And now we also have one for React. And people are
Subrat_Mishra:
Mm-hmm.
Miroslav_Jonaš:
using it more and more and kind of becoming a standard. If you're in Angular world and you want to create micro frontends, most likely with You will do it inside NX.
Subrat_Mishra:
Yeah.
Charles Max_Wood:
Yeah, that makes sense given that most of the micro front ends are independent of each
Subrat_Mishra:
Hmm.
Charles Max_Wood:
other and so you're going to have them in separate repos or you put them in a mono repo. You manage them all together because for the most part you're using the same, I guess, framework within the framework in order to put them together using the same setup. And so at the end of the day, yeah, having something like NX and just running it all through NX makes a ton of sense.
Miroslav_Jonaš:
And since those things eventually get stitched together online, it makes sense to have them together in the same repository so that you know if something
Charles Max_Wood:
Yeah.
Miroslav_Jonaš:
changes, even if they can
Charles Max_Wood:
Yeah,
Miroslav_Jonaš:
live
Charles Max_Wood:
you can
Miroslav_Jonaš:
completely
Charles Max_Wood:
you
Subrat_Mishra:
Mm-hmm.
Charles Max_Wood:
can blend
Miroslav_Jonaš:
separately.
Charles Max_Wood:
the ones that have to blend.
Miroslav_Jonaš:
Yeah.
Charles Max_Wood:
Yep.
Subrat_Mishra:
So
Charles Max_Wood:
So.
Subrat_Mishra:
is it possible we can, sorry,
Charles Max_Wood:
Nope. Good.
Subrat_Mishra:
what I was asking is, is it possible we can create a project with Angular and React as a different project inside NX project? For the microphone purpose I'm asking.
Miroslav_Jonaš:
To be honest, I'm not sure. I think, yeah, I generally you should be able. So NX can host Angular and React together.
Subrat_Mishra:
Hmm.
Miroslav_Jonaš:
When it comes to microfrontends, I guess you would just have to choose which one will be driving. So whether the Angular will be providing shell or the React will be providing shell. But then each project separately, I think it can be built with whatever you want.
Subrat_Mishra:
Hmm. It just came to my mind while Charles was asking. I have also never tried, tried only with Angular. So
Miroslav_Jonaš:
Yeah, I think
Subrat_Mishra:
thought
Miroslav_Jonaš:
that
Subrat_Mishra:
to ask you.
Miroslav_Jonaš:
people usually think about Microfrontend as that, yeah, cool. Now I'm going to build a website where one page will be in Angular. The other one will be in React. The third one will be in Vue. Fourth one will be in Svelte. And it was like, why would you do that? I mean, yes, you can, but
Subrat_Mishra:
Yeah.
Miroslav_Jonaš:
why would you do that? Just doesn't make sense. And then
Charles Max_Wood:
I'm going
Miroslav_Jonaš:
Microfrontend
Charles Max_Wood:
to write a book.
Miroslav_Jonaš:
is...
Charles Max_Wood:
The first page is going to be in English, the second page is going to be
Subrat_Mishra:
I'm
Charles Max_Wood:
in French,
Subrat_Mishra:
sorry.
Charles Max_Wood:
the third page is going to be in Japanese.
Miroslav_Jonaš:
Yeah, I mean, in some cases it makes sense for something like, I don't know, there are some large and old companies that have built stuff in, for example, jQuery and they don't want to rewrite all that stuff. So they would just partially rewrite certain pages to different framework. And they still didn't like to stitch them together. But when you're starting a project, you usually don't have this problem. So you can just stick to single technology
Subrat_Mishra:
Yeah.
Miroslav_Jonaš:
and Microfront is just like any other. methodology, it's nice and it solves certain problems, but it doesn't solve all the problems and people usually just grab the latest buzzword and think that this will then solve all their problems.
Charles Max_Wood:
Yeah, that's true. I mean, all of my micro friend ends back up on the blockchain. I'm sorry. Anyway.
Miroslav_Jonaš:
Yeah, with artificial intelligence
Charles Max_Wood:
So
Miroslav_Jonaš:
in
Charles Max_Wood:
I want
Miroslav_Jonaš:
the background.
Charles Max_Wood:
it. There. Yeah, there we go. An AI that runs on blockchain.
Miroslav_Jonaš:
Yeah.
Charles Max_Wood:
Anyway, so one thing that I wanted to kind of get into here that we haven't talked as much about with all the monorepos and linting and stuff is the CI end of things. So when you're running the CI against all the different projects within your monorepo, do you have like different CI jobs set up for each project or do you just run CI against the whole thing? Does it only run it against the ones that have changed? Like how does that all work?
Miroslav_Jonaš:
Yeah, so one of the things that NX or Narble actually provides is NX Cloud. And NX Cloud is our cloud infrastructure for the running NX on your CI. And what it
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
gives you is it gives you remote caching or distributed caching, which means that... Normally with, with NX, you can cache stuff locally so that you don't have to run build twice, but you can also link your NX cloud account and then you can also cache this on your cloud. Meaning that if you run something in your build, your colleague that if they try to run the same build, they will get the immediate response from the cloud. They don't have to run it on their machine. And this naturally helps tremendously on your cloud or on your CI, because if you already run it locally, it will be stored. Then the result will be stored in a cloud so that when you run CI later on, CI will don't have to spend time running this. It will just take it from the cache. And apart from that, NxCloud also provides distributed task execution, meaning that we can take our jobs, split it into tasks, and then distribute tasks between different machines. So for example, Let's imagine you have hundreds projects and you want to build all of them. Normally you would run 100 projects one by one, one after another, sequentially, and this would take
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
huge amount of time. With distributed task execution, you can create a, first of all, graph of dependencies, because normally if you would run it on your own, you would have to know which project depends on what, so you would have to organize in which order they will be. built on your own. Annex does this for you because we know how things are connected. So we can create orchestration for you. And then we can take those tasks and distribute between different agents. So, Annex Cloud allows you to use multiple agents. You can use from two agents up to hundreds of agents, depending how much you paid for your CR infrastructure. But you can then take, for example, Let's say if you have a hundred tasks that you have need to run and you have 10 agents, you can split those tasks between 10 agents. And normally people would think about this as a binning. So like simply taking hundred projects and splitting into 10 different agents. That would mean like each agent would get 10 tasks. But in reality, it's not so simple because first of all, they depend on each other. So they can't just be split randomly. And additionally, some tasks are larger and take more time and some tasks are super fast. So you need to know how to split them optimally so that you don't have too much idle time on your agents. And this is also something that NX provides. And up to my knowledge, I think only NX and Bazel from all the modern solutions provide this functionality. code size and based on some history and the things you're trying to do, so the operation you're trying to run, we can guesstimate how long it will take, relatively
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
between each other, so we know how to distribute them in the best possible way so that you end up getting the result from all of those builds in the least time possible. nice things about DT is that once everything is finished, you get one result as a unified document. So you don't have to go through all of these agents and collect the data and then stitch them together, but you get it in the main thread or the main agent. You get as a single report, regardless whether it was run on single agent or it was spread between hundreds of agents.
Charles Max_Wood:
Gotcha.
Miroslav_Jonaš:
Yeah, and this is...
Charles Max_Wood:
Now do you run linting as part of your CI? Because we're kind of tying a lot of these ideas together.
Miroslav_Jonaš:
Yeah, we run, for example, for standard setup would be to run lint test and build.
Charles Max_Wood:
Uh huh.
Miroslav_Jonaš:
What we do
Charles Max_Wood:
Okay.
Miroslav_Jonaš:
for, for annex, for example, we run Lintest and build, of course, we run a subset of end to end tests. Uh, and we are also running some, some additional checks, like we check our documentation, whether it's. It has been changed. We checked the formatting and some, some additional minor stuff. That's kind of important for the health of our repository. And then additionally, we have our nightly builds, which then run. a bit more end-to-end tests and also we run them in different environments. So for example, we run them in yarn and PM and PM PM. And we also run them on, on Mac machine on Linux and windows so that we get kind of all different environments combined for normal CI. So for normal PRS and main branch, we just run the PM PM and we just run it on Linux because that's fastest. Of course there are plugins that only work on on Mac like the React Native and Detox so those then we run on on Mac but everything else runs on Linux.
Charles Max_Wood:
And that's all done in NX Cloud.
Miroslav_Jonaš:
Yeah. So, so all of this runs on NX cloud or NX cloud actually
Charles Max_Wood:
Nice.
Miroslav_Jonaš:
organizes it's run on, on circle CI and the nightly
Charles Max_Wood:
Hmm,
Miroslav_Jonaš:
runs on, on GitHub.
Charles Max_Wood:
okay.
Miroslav_Jonaš:
But, but NX cloud organizes the tasks and handles the caching.
Charles Max_Wood:
I gotcha.
Miroslav_Jonaš:
So you still use your own CI infrastructure, whatever you choose in your own provider, but NX Cloud is handling the distribution of the tasks.
Charles Max_Wood:
I gotcha. That's cool.
Subrat_Mishra:
So according to the environment, you guys are deciding which system to give like you mentioned about the React Native and that it will run on Mac or everything else it will run on Linux. So according to the code structure you are giving or you have some configuration that handles those things.
Miroslav_Jonaš:
No, so it's a matter of simple configuration. So we know that project that implements React Native and Detox plugin needs Mac to run the tests. So
Subrat_Mishra:
Yeah.
Miroslav_Jonaš:
we run those two on Mac. And the same way we know that everything else doesn't then needs to be run on Mac, because it's not really relevant, at least for the CI. But yeah,
Subrat_Mishra:
Hmm.
Miroslav_Jonaš:
I'm not sure if you would be able to somehow split this in the package, like halfway through. Like, for example, our test, linked and built runs on Linux because that's the fastest. But then our end to end tests run on specific machines. But yeah, we know which machine certain end to end package requires. So we know how to distribute them. So we simply configure them. what to run where.
Charles Max_Wood:
Make sense.
Miroslav_Jonaš:
And it's very easy, for example, on, on GitHub to set up this matrix configurations where it can then specify what operating systems you want, what NPM versions you want, what package managers you want and like what projects to run where and which doing exclude. And then you just kind of run this matrix and it handles everything.
Charles Max_Wood:
Nice. So what else should we know about NX?
Miroslav_Jonaš:
Um, one thing that that's very cool that happened actually this week is that we managed to surpass Angular CLI by weekly downloads, which is very important milestone
Charles Max_Wood:
Nice.
Miroslav_Jonaš:
for us. So we had a couple of milestones that we reached recently just before the new year's eve, we crossed 1 million downloads weekly. That was very important. And then we surpassed Lerna, which was kind of the tool to use. for Mono repo and as some of you might know that we actually took over the stewardship over Lerna. So Lerna was being unmaintained for a very long time and the original maintainer just didn't have time to continue working on it and we offered to take it under the was that we started now learning using Lerna in the NX. So for example, for our local versioning and publishing of the package, now we use Lerna. And at the same time, Lerna now has use NX flag, which then replaces the outdated orchestration and task management implementations that were there. And you can use
Charles Max_Wood:
Mm-hmm.
Miroslav_Jonaš:
then NX. within your learner without any change. You just set the flag use annex to true. And you get all the nice features from the annex. You don't get plugins because this is part of the full annex solution, but you get the local caching effected graph, the visual graph, the task orchestration, all of these cool things that speed up your development.
Charles Max_Wood:
Very cool.
Miroslav_Jonaš:
So yeah, now
Charles Max_Wood:
So
Miroslav_Jonaš:
we,
Charles Max_Wood:
what
Miroslav_Jonaš:
yep. So,
Charles Max_Wood:
kind
Miroslav_Jonaš:
now we,
Charles Max_Wood:
of benefit or, oh
Miroslav_Jonaš:
yep.
Charles Max_Wood:
go ahead. I was just
Miroslav_Jonaš:
Just.
Charles Max_Wood:
going to ask what kind of benefit are you seeing from people using NX? Like do you measure how much time it saves them or how much simpler it makes it for them or how much money it saves them?
Miroslav_Jonaš:
Yeah, on, on, on annex cloud, for example, you can see how much time it saved you, how much cloud time it saved you. So you can calculate them based on your monthly CI bill, how much money was, was it saved for, for developers, what this means. So that that's important for company, of course, to, to reduce,
Charles Max_Wood:
Right.
Miroslav_Jonaš:
to reduce costs for developer. The most important thing is that, for example, when you create a PR, you get the result from your CI way faster, so you don't have to wait a long time. The same thing, of course, applies on your machine. And when you're running build or whatever other processes, you get the results more quicker. So your developer experience improves tremendously. Also, as I mentioned, you don't have to... You just have to learn one set of commands and then you can use it between different projects, different frameworks and libraries. You don't have to think every time was it NPM server or NPM start or NPM. Whatever develop because it each framework implements it slightly differently, but we try to have this unified and we are constantly working on improving things. So making things easier for developers and especially. Our generators help you when you're starting project or when you need to quickly scaffold something you can nicely set it up so that you can quickly generate whatever you need to have in your project. At the end of the day it saves a lot of time for developers, it makes things easier and with our linting rule it also helps you from catching this architectural errors quicker. before they escalate and bring your entire architecture to the point where it's just better to throw everything out of the window and rewrite it from scratch.
Charles Max_Wood:
Before your users realize that you broke the contract between the front end and the back end.
Miroslav_Jonaš:
Exactly.
Charles Max_Wood:
Cool. Anything else you wanted to bring up or ask, Sue brought?
Subrat_Mishra:
Yeah, what I was saying is that the most initial thing, I think the most important thing is that messing up with the request response. Basically, in the starting phase, if a two independent team are working, then it's always happened. Maybe in the later stage, you may have built some processes or do something. But NS solved that from the first day itself. So it's a pretty good use case for me.
Miroslav_Jonaš:
One of the nice side effects
Charles Max_Wood:
Nice.
Miroslav_Jonaš:
of NX or actually working in monorepos in general is that people start to collaborate more. In most of the companies I worked in the past, you had kind of clear separation between back-end guys and front-end guys, and then you had some full-stack guys sitting on two chairs, but there wasn't a lot of communication between different teams, especially if the project is large and they are really... like a couple of teams working on the front end, couple of things working on backend. And then you have like teams working on backend as a service and then teams working on databases, a separate layer. And it usually required some complex meetings to get up to speed and to know what's happening where. And Monorepo somehow slightly teaches you how to communicate often and how to collaborate more. So when you change something that breaks front end, you see it immediately. And then you need to talk to the people who are responsible for that layer. And often you try to solve these things together. So it kind of brings people closer back, despite they might be working on different teams and even across different continents.
Subrat_Mishra:
And I think this monorep is also a very good tool for full stack developers. So they did not need not go and choose multiple repository to back and forth. They can just go ahead and check the same repository who are maintaining both front and back end database.
Miroslav_Jonaš:
Yes, exactly. And despite people might be thinking that monorepos only makes sense if you have more than one project, it actually using NX as a developer toolkit makes sense even for a single project. Because of all these things like caching and linting and just in general, nice developer experience that helps you work on your single project even. So you shouldn't shy away from using NX if you have just a single project, because even there it can help you tremendously.
Charles Max_Wood:
Cool. Well, let's, we're kind of getting toward that time where we need to do picks. Yeah, let's jump in and do picks. Subrat, do you want to start us off with picks?
Subrat_Mishra:
Yeah, so this week my pick will be a programming book. I think long time I haven't given a programming book. So it's a pragmatic programmer. So it's by David Thomas and Andrew Hunt. So it says that I haven't finished the book. I just started reading the book like say Journey to Master. So it's kind of interesting kind of I'm getting some new concepts. How to think while programming. I will re-update you guys when I finish the book and how it was but for this week, this will be the pragmatic programmer.
Charles Max_Wood:
Yeah, that's a terrific book. Dave and Andy are terrific guys too, so
Subrat_Mishra:
Yeah.
Charles Max_Wood:
go check them out. They kind of parlayed that into Pragmatic Programmers where they publish a bunch of other books and courses and stuff like that. I think it's at PragProg.com. Anyway, good stuff. It's a terrific book. That's stuff that you should definitely be looking at and practicing the stuff that they put into that book
Subrat_Mishra:
Hmm.
Charles Max_Wood:
on a regular basis. Um, awesome. And anything else you brought before I throw out my picks?
Subrat_Mishra:
No, that's all for me.
Charles Max_Wood:
All right, I'm going to throw out a few picks then. I usually do a board game pick first. I'm going to pick a card game this week. It's called Quiddler and it's a word game. And I like it because I'm good at word games and I usually win. But anyway, it's also relatively simple. So you deal out the cards, you put, you know, you discard and draw, and then you put down word or words, you know, using all your cards and then everybody else gets a chance to put down their cards. Then you get scored, you get extra points for having the longest word or the most words. And yeah, anyway, it's a fun game. So on Board Game Geek, I'm looking it up right now. It comes in at 1.43. on its weight or complexity. So it's pretty easy game for people to pick up. Definitely a game my kids could play with us. You play through a number of rounds. And I think we played a full game in like 20 minutes or a half hour. So anyway, it was awesome. So I'm going to pick that. And then I just want to let folks know we are putting together the Angular Remotes Conference. That's going to be in November. You can go to the website and find it. It's under conferences. I'll put a link in the show notes as well, but yeah, it's going to be awesome. We're going to have all kinds of awesome people come and speak. And the CFP is also open. So if you want to speak, you can submit a talk. I'm trying to think what else I want to pick. Lately my wife and I have been watching The Lost Symbol on Peacock. Peacock's just another streaming service. But it's based on Dan Brown's book. And so they're following clues and being shot at and all that stuff. It's pretty good so far. So yeah. So I'm going to pick The Lost Symbol. And I think that's all I got. So, uh, Nero, what are your picks?
Miroslav_Jonaš:
Just to say I read Lost Symbol and I really like the book. I like most of these books, so
Charles Max_Wood:
Yeah.
Miroslav_Jonaš:
I'm really looking forward to that
Charles Max_Wood:
Yeah.
Miroslav_Jonaš:
one. On the series note, I noticed that recently Sandman came out on Netflix, which is this amazing comic, and I haven't started watching it yet, but I'm looking forward to that. I think it will be amazing. If it's half as amazing as this comic is, it will be just great. So I... suggest people to just try it out and see how it goes. Currently it has 7.9 on IMDB, so not amazing, but still high enough to give it a try. And on another note, I also want to mention one conference. Narvel is organizing an XConf in October. It will be in Dempe, Arizona. It's on October 17th. So everyone who is interested, please in annex should come. We also have a remote version, so you can either be there in person or or watch it remote. It will be one day conference and there will be workshop day before for beginners and then day after as an advanced one. So it's all together three days from 16 to 18. That's all from my
Charles Max_Wood:
Awesome,
Miroslav_Jonaš:
side.
Charles Max_Wood:
very cool. All right. I don't think I have any other picks or anything else to announce, so we'll go ahead and wrap up here. Thanks for coming, Nero. This was awesome.
Miroslav_Jonaš:
Thank you for having me.
Charles Max_Wood:
All right, I'm going to go ahead
Subrat_Mishra:
Thanks, Nidul.
Charles Max_Wood:
and wrap this up. Until next time, folks, Max out.
Subrat_Mishra:
Bye bye.
Monorepos, Linting, and CI, Oh My! - AiA 354
0:00
Playback Speed: