Check out RailsClips on Kickstarter!!

 

02:01 - Richard Kennard Introduction

02:04 - Geraint Luff Introduction

02:07 - David Luecke Introduction

02:57 - Object-relational Mapping (ORM)

10:57 - Online Interface Mapper (OIM)

12:53 - How OIMs Work

16:02 - Editing a UI That’s Automatically Generated

23:01 - Why Do We Need These?

25:24 - Protocol? 

27:56 - Plugging Into Frameworks

33:48 - Making Judgement Calls

49:27 - Example OIMs

52:08 - Testing

Picks

The Legend of Zelda: Majora's Mask 3D (AJ)
80/20 Sales and Marketing: The Definitive Guide to Working Less and Making More by Perry Marshall (Chuck)
A Wizard of Earthsea by Ursula K. Le Guin (Chuck)
Conform: Exposing the Truth About Common Core and Public Education by Glenn Beck (Chuck)
Miracles and Massacres: True and Untold Stories of the Making of America by Glenn Beck (Chuck)
3D Modeling (Richard)
Blender (Richard)
Me3D (Richard)
Bandcamp (David)
Zones of Thought Series by Vernor Vinge (David)
Citizenfour (Geraint)
Solar Fields (Geraint)
OpenPGP.js (Geraint)
forge (Geraint)

Special Guests: David Luecke, Geraint Luff, and Richard Kennard.

 

I’m a hardcore professional something!

Hey everybody and welcome to episode 150 of the JavaScript Jabber Show. This week on our panel, we have Richard Kennard.

Good day.

Geraint Luff.

Geraint actually, with a hard G. Hello.

David Luecke.

That sounds about right. It’s… [Chuckles] DAVID:

The German is always confusing people. Hey.

Awesome. So, those are our guests. We also have AJ O’Neal.

And under some advisement, I was asked to try something different out this week. So, instead of my normal catchphrase, I’m going to go with, what’s up, y’all? Coming at you live from Provo. [Chuckles]

Alright, and I’m Charles Max Wood from DevChat.TV. I just want to throw a quick reminder out there that I just launched the Kickstarter campaign for the Ruby on Rails videos. But the rewards are things that I think all of the people who listen to all the shows are going to be interested in. And you can go check that out at DevChat.tv/Kickstarter. And really, what it’s about is about supporting the shows. So, if you want to give some financial support to the shows, then that’s a good way to do it and get something back. So, that’s all I’ve got going on.

This week we’re going to be talking about what it says on here is form generators. Before the show, we got an idea that it had a little bit more to do with general UI and things like that. Do you guys want to go into a little bit more about what this is about?

Sure. I can take that one. Probably I’d like to explain what it is by talking about something else first, which is always a strange way to go. But we’ll go around and hopefully get back to where we need to be. So, there’s a technology called an object-relational mapper or an ORM that is used quite commonly in a lot of web developments and enterprise developments. And most platforms offer some kind of ORM. So, on .NET you’d have something like the Entity Framework. In Rails you’ve got Active Record and Java EE has things like Hibernate. And there are quite a few ORM technologies out there that are pretty popular, I think.

And to understand I suppose why they’re popular, you need to take a step back to life before ORM. So, before anyone had any kind of object-relational mapper, if you wanted to write an application that stored something into a database, a relational database in this case but just general persistence, you have this world where first you would write your domain object, your business object code in C# or Java or whatever language you were using. And then you would switch over to your database environment and you would start creating your SQL schemas.

And so, say if you had a person class that you were trying to model and the person had a first name, well you’d create a class called Person with a field called first_name in Java or .NET or whatever language you were using. And then you’d flip over to your database and you’d create a table schema called person with a field called first_name. Then you’d normally have to write a

bunch of SQL statements like an insert, and update, and delete, on that person table, each again using update first name, delete this. You might even go further and wrap it in a bunch of stored procedures which are called insert_person_stored_procedure and then took a bunch of arguments.

And so, you’re in this world where you’re basically repeating this same definition about a person and a first name in multiple places. And that sucked for numerous reasons. But one of course is just a lot of work and a lot of time. But the other is it’s a real potential maintenance headache because you’re defining the same thing several times. You’re defining this. There’s a person and he’s got a first name in both your business code and also in your SQL schema and also in your SQL statements and also in your stored procedures, and so on and so forth.

And so, I think one of the things that when this side… one of the good things about an objectrelational mapper or some way of addressing this is to try an remove that level of duplication and that level of error-prone duplication in particular. So, when you have an object-relational mapper, you have this idea where you just define person and first name in one place. And then it takes care of mapping that down to the database. And there are just a whole lot of subtle errors that go away, subtle bugs that just go away, when you don’t have that level of duplication. There’s crazy stuff.

Like say for example, I haven’t written a SQL insert statement in a long time now, but the SQL insert statement, you got INSERT INTO and then first you list off all the fields and then you do VALUE and you list off all the values. And you’ve got make sure that your values come in the same order as your fields. So, it’s rife with possibilities there for you accidentally transposing a couple of things. And nothing will blow up. You won’t get any hard errors off that. You’ll just start subtly finding data is going into the wrong column in your database. So, ORMs were designed to solve that class of problem. And I think they’ve done very well. And certainly nowadays if you’re doing a web app or any kind of thing, you’ll often use an ORM.

I know also you might adopt a NoSQL environment as well. And in many ways, something like NoSQL is actually solving the same problem in a different way. One of the reasons I think people love NoSQL is that you can just take your objects and shove them in your database. And you don’t have to worry about mapping them from one type of a non-relational world into a relational world. There’s no, you don’t have to double declare everything. You can just put the object in and get the object back out again. So, in terms of solving that same problem which is removing the margin for error, then it’s worked out very well.

So, ORMs have been very popular I think for a couple of reasons. One is they get rid of all the duplication. But the other thing they do that’s important that’s often overlooked I think is they don’t dictate your whole app. So, all the ORM is trying to do is get your business objects from your .NET/Java/Ruby code or whatever into your database. It’s not saying to you, “Okay, you have to use Oracle,” or you have to design your app in a certain way. It’s not trying to take over your whole application stack. It’s just this little piece that fits between the business object and the database and solves that particular problem. And I think that’s why they’ve been very popular and why nowadays, you can still write SQL statements by hand when you’re doing a web app but these things are very useful.

So, we live in this nice world where when you’re trying to develop an application, when you’re going from the… if you think about your application stack, say you’ve got your UI at the top and you’ve got your business stuff in the middle and then you’ve got your database towards the bottom. Going from your business layer down to your database is a pretty good story. You have a tool in place that will not force your hand, not dictate how you got to do things, but will also remove all that duplication. And so, I think we’re all pretty happy with that. And there’s not a lot of error-prone stuff in that world anymore. It’s very DRY, very Don’t Repeat Yourself.

But then the problem is that you’re in that world and that’s good. But then you pivot and you look up the stack instead of looking down the stack. So, you look up from your business objects, up to your UI. And suddenly, you’re back in this world where there’s an enormous amount of duplication. So, if you’ve got a person class in your business layer and it’s got a first name, then yes you can map that down to your database. But when you come to start doing UI, you’re literally writing, if it’s a web app that you’re doing and you happen to use something like Bootstrap for example, then if you’re using the standard Bootstrap approach to how you do a form, then you’ve got to do a label and you’ve got to call it first name. And it’s going to have a ‘for’ attribute that points to a first name control. And then you have to have a control that has got a name, a first name, and also an id, a

first name. And you’ve got to make sure that it’s… the textbox or whatever it is. And you’ve got to put your length on there.

And so, suddenly all this duplication is straight back in your face. You’ve got a lot of places where you’re typing the same thing again. And if you accidentally type it wrong, so if you have a field and on the screen it’s labeled first name but because you copied and pasted it from somewhere else, you forgot the change the codename of that control. And it’s actually, the name of the control is surname. Then suddenly, the user thinks they’re typing it to one field and it’s going to end up in another field. And it’s not going to give you a hard error. There’s not going to be, nothing’s going to blow up. And the only way you’re going to find that is either through integration testing, or worst case, your uses pointing out that data’s going missing. And so, there are some things like that.

There are some things like mapping data into the wrong field. But there are far more subtle things. Like for example, almost all databases will have a maximum length on a field. But a lot of other technologies won’t necessarily have that restriction. So, ultimately that field is going, that data is going to end up in a database somewhere that’s got say a 30-character limit. But on your front, so on your frontend really, you should be putting

<input type=”text” maxlength=30> on every single field that you do.

Generally speaking, people don’t do that. And the reason they don’t do that is because they know that six months down the track, if you do decide to change the length of that field to 50 in the database, then you’re going to have to remember to update all your maxlength attributes on all your fields throughout your UI. And the same with minimum and maximum values and all these things. There are all these things that have to stay in sync. And oftentimes, people don’t… they either get out of sync over time or people just don’t bother putting them in because they’re worried that something will break. And they just hope that their database will catch that error down the track.

But there’s all these class of problems that come in, mainly to do with the fact that there’s just a lot of duplication between the business objects and the UI. So, the idea that we’re sort of discussing is this idea of an OIM, an object-interface mapper. So, if an ORM goes from the business object down the stack to the database, then an OIM would go from the business object up to the UI. And it would try and solve a similar problem in that it’s got to try and remove all that duplication. But also, just like an ORM you don’t want the OIM to try and take over your entire application stack.

So, the reason I like to use a term like OIM is I don’t like to use a term like a UI generator. Because there are things out there called UI generators that try and, yes they generate your UI but they basically try and generate your whole app. You give them some business objects like person and address. And they basically try and generate you an entire UI. And oftentimes, because they try and be very generic about that, then you basically end up with a CRUD, a create, retrieve, update, delete type UI, which is not very usable and not very nice, particularly nowadays where the UIs have become… so much care an attention’s gone into them. So, they’re not really effective for a large class of problems, because they’re trying to do so much. Whereas something like an OIM which is just trying to solve this little problem of you’ve got a piece of your UI, not your whole UI.

Now, if you were developing a game then maybe your main UI is some kind of man running through a world or something. But at some point in that game, you will pop up some kind of little form or little input which has a few fields. And you want to collect data there. And that’s the bit that you want to be able to sync tightly with the backend, because it has to by synced tightly with the backend. There’s no benefit to it being out of sync with the validation that it needs to have. So, you have to get that sync in. So, that’s where you need an OIM. But you don’t want the OIM or the UI generator to try and take over the whole of your application stack. Because then, it just becomes less useful.

So basically, if I understand this, so we have an ORM. And this is something that I think most people with a backend framework are familiar with. I use Rails and so I’m usually using Active Record or something. So, the OIM sits on the frontend of a web app and maps back to some API that I have on my web application.

That’s correct. So, in similar ways to the… so, the ORM, at least some of them, actually will generate your SQL schemas and insert, select, update statements for you.

Right.

So, what it needs to do is lock down the mechanics but be flexible enough with the rest of it. So, if you want to use Bootstrap or you want to use a certain type of, you want to lay out using divs or tables or ul elements or whatever, whatever layout you want to do, whatever different look you want to do, that has to be flexible enough. But you would want something that can generate you those parts of the form that have to stay in sync.

So, I’ve got a question about this generation, what you mean by that. Because I see, there’s dynamic generation and there’s static generation. So, when I used to use Doctrine, that was static generation.

Yeah.

But Rails seems to be a little bit more dynamic. You can have a method that doesn’t exist, like find user by id, and it dynamically generates it. So, what are you suggesting? Or you don’t care between the two?

No, look, I personally care. I think OIMs can be implemented either way and they would be useful. Personally, I’m not a big fan of static generation. I think the problem with static generation is it looks great at first. You click a button and you suddenly get an enormous amount of code out. And you think, “Oh, I’ve done a good day’s work here.” But then you actually look at the code. And oftentimes because it’s synthetically generated code, it’s not the best quality code. It’s very hard to understand. But the worst part of it, and the thing that I think the real nub of the problem here, is duplication of definitions.

And so, if you statically generate something, although you have solved that initial generation, you’ve still got a duplicate definition in your code. And so, in six months’ time, if you decide to change the type or the length or the order or anything of a particular field, then unless your static generator offers a really nice round-tripping capability, which some of them do, but if you don’t, if you have to manually go in and start editing that statically generated code, you’re probably worse off than just doing it by hand. So, personally I’m a big fan of runtime generation.

Okay. So, I definitely understand that point of view. The thing that I’ve seen that has frustrated me about the dynamic ones is that when you’re talking about UI, well you just said we don’t want to focus on the UI too much with the OIM, right?

Yeah.

But how am I going to edit the UI that’s dynamically generated? The first time I want to make a combo box, the missing element from html that everyone implements every time, how hard is that going to be to do with an OIM?

Yeah, so I just want to chip in and say that Richard, you’re coming from a very, the point of view of the programmer. And I tend to come from this abstract, fairly user’s perspective, having a math background. And for me, it’s mostly about not having to give instructions about anything that you haven’t made a decision on. So, with something like Ruby on Rails, it makes a lot of decisions about, for example what the URL structure or something should be like. Because there’s an obvious way to structure URLs, given a certain base structure. And with ORMs, there’s an obvious way to structure the database, for a given type of data. And an obvious way to program the class. And it’s, given that it should be a no-brainer plugging those two together, the general problem we’re solving is just creating an automatic bridge between those two.

So, the question of static and dynamic or anything else like that, the principle that we’re coming from is that you specify the information you need, or the shape the information is, and then make the obvious choice. So, with the UI, if the data that you need is exactly one of the following, then one of these multiple choice selectors might fall out or whatever, or whatever it is that you need, because the idea is that you specify the shape of your data. And then something appears that lets you input that data. And with the UI that fits in the website, then the exact styling will be tweakable to look like everything else and maybe even hooked into your Bootstrap or possibly other frameworks you’re using. But at the end of the day, the thing that we’re looking at, from my perspective, is to generate something on the screen that matches the underlying shape of the information we’re talking about.

So, how would you know? Like for example, if my list is three items long, a radio box or checkbox makes sense. If it’s 10 items longs, a combo box starts to make sense. If it’s a hundred items long, a tag cloud starts to make sense.

Well, okay yeah, that’s a really good point. And so, for me actually that kind of question is why discussing UI generation in this kind of sense is really exciting. Because the answer is not going to be the same for all environments. So, it could be that… and if I’m looking at a laptop or a desktop, I’ve got everything. I’ve got a mouse, I’ve got a keyboard. It could be that the controls and the way that that data/information needs to be entered is different to… the Apple watch that just came out, it’s got basically no UI at all. But they could both demand the same shape of information. And then the appropriate controls would be different for both.

And so, my interest is coming from what information do you need and then how do you present that? And that being generated means that you don’t have to think about the exact issues of inputting various sizes, how [many is] going to be this, how [many is] going to be that. It’s that someone somewhere who knows better than you or at least better than me makes these calls. And you can tweak them or whatever. But at the end of the day, you say what you want and then it gives you something appropriate. The personal shopper of UIs.

I think it’s an excellent point, AJ. It is a definite tension between… on one hand you don’t want to have to specify this stuff because if you do, you are potentially introducing duplication and that’s where a lot of these maintenance headaches come from. But on the other hand, if you’re not specifying it explicitly, then you risk, losing control. And if there’s one area where you don’t want to lose control, it’s in the appearance of the UI. It’s such a personal and human connected type thing. You don’t want to risk having a substandard UI just for the sake of…

Yeah, absolutely, yeah.

[inaudible] Yeah, yeah. So, that is a definite tension. And that’s probably why this area is so hard to get right.

Yeah, I think there’s always room for overrides and tweaks and that kind of thing where you say, “Oh, if this absolutely has to be a textbox instead on an input, single-line input, then you need to tell it that because that’s an actual UI constraint.”

And then that’s another good analogy back to an ORM, is that ORMs, they do map the data for you and they do generate the SQL statements and schemas for you. And they work great a lot of the time. But almost all of them still offer you a way to drop straight back down to native SQL when you want to do something specific and you want to tweak. So that 20% of the time you still have that full control. But the 80% of the time it’s taken away the maintenance headaches for you. So, an OIM in exactly the same way would need to offer that same path.

So, that makes me think, one pattern that I like is almost a hybrid between the static and the dynamic, which is where… I get confused when there’s too much convention over configuration because I don’t know how to change something. But maybe when you boot it up or you generate it, it generates your JSON config file with all the metadata in it. And then you know where to go to change something and you know what the defaults are, maybe.

Well, I agree with Richard’s concerns about that kind of model, is that if it’s generated a file and then you edit that file, then you can only generate it once by definition. So, what you really need is an override configuration instead of modifying something that was generated once. Because…

Well, I was thinking that it would generate the defaults. So, it would print out what the defaults it wants to use are. And you’d be editing that. Not that it would be code-like but it would be, oh what’s the, I forget the word, imperative declaration-like.

Right, okay.

Right, yeah. And I think this is why it’s a potentially very active research area. There are probably two things here. One is that we need to recognize this problem of duplication between the UI and the business layer is an issue in the same way as it was an issue for ORMs and that something needs to be done. And at the moment, not a lot is being done.

Most new frameworks that come out, even things like Angular, Backbone, if you look at the coding examples they offer of how, our best practice, when it comes to the UI, yes things like Angular and some of the newer technologies, they’ll do some kind of data binding for you. So, Angular’s probably a bit better than jQuery in that you don’t have to write so much data-binding code. But you still are generating, you still are writing a lot of input controls and labels. And you’ve got to get all these types the same.

So, the one thing to do is acknowledge this is a problem. But how you actually go about solving that, I think is an active area of… and certainly things like how do you generate it well enough that it’s useful without also imposing itself too much on the final product? Do you do static generation? Do you do runtime generation? These are things that different OIM vendors for example can try different approaches with.

So, now I think it’s also worth asking a bit more in detail about when we need these as why we need these. So, when I created one of these, I was motivated by the idea that I didn’t like writing UI code. I wanted to write stuff that did stuff and I wanted to make it API and then click it. And so, my initial requirements are based around being able to prototype quickly. And then that’s actually a different set of requirements from people who want a final product that’s really highly customized and melts very much into the rest of the UI. So, I think we’ve got, when we’re talking about UI generators and that kind of thing, there’s a whole spectrum from debugging tools to productivity tools for the more designer end and that kind of thing, to write things fast.

Mm. I think certainly you’re right there. And for particularly the prototyping tools, something like static generation is pretty good because as you say AJ, you can then get in and modify it if you need to. You do get an end result. And it’s nice to have that initial thing. But it breaks down a little bit when you start going beyond the prototype because the static generation is no longer going to hold your hand past that initial generation. And then you end up with the same problem where you decide to add a new field to an object somewhere and now you have to run around every screen in your app that might be displaying that particular type of data and make sure that extra field is added in, or that field is changed.

And like you said Geraint, there’s a spectrum here. If you’re working in an enterprise app where you’ve got 300 forms, then these things are extremely useful. If you’re working on an iPhone game where you’ve just got a little popup input, then I think they’re still quite useful in the same way that an ORM can be quite useful. It’s probably not as essential. But there’s this mindset where wherever you’re typing the same thing twice, particularly in two different types of world, you’re typing one thing in html and one thing in JavaScript, and any time you’re trying to express the same concept in two different places, it’s very likely that over time you’re going to accidentally cut and paste something wrong.

From what I was looking at, what you said that you have different types of places where you need to change it right, because you need your JavaScript and you need the html. You need an inbetween format that’s easier to generate than html and JavaScript and easier to change on both ends. I think that was the goal from what I started.

So, I’m wondering though. It seems like unlike an ORM that’s going to back onto a specific database, those databases define their own protocols. And I… let me back up for a second, because I’m totally behind what you’re saying about OIM/ORM, with keeping your code DRY and encapsulating logic and stuff like that. It makes total sense. But your backends are going to vary.

So, do these OIMs specify the type of, basically the protocol? So, they’re… are they specific types of REST or JSON APIs or XML or SOAP or something else?

Right, so now I suppose you’re getting into a particular implementation. So, I can certainly speak to my…

Yeah.

My own implementation, which is a project called Metawidget. And so, that’s an implementation that takes a particular approach. And the approach that I’ve taken is, so again, with an ORM for example, you’ll often get plugins for different database dialects. So, you might get your standard ORM and then you might get a plugin or a driver for Oracle or for MySQL or some other environment that you’re trying to interface with. And so, Metawidget takes a similar approach in that you have this fairly simple well-structured pipeline. And then there are plugin points along that pipeline, the way you plug in different plugins for different use cases.

So for example, the first part of the pipeline is, where do I get my data from? For want of a better term, it’s called the inspector. So, that part of the pipeline is all about, where do I get my data from? And you can either plug in an inspector that will read it straight from your JavaScript code, just like a little JSON object that you’ve got. Or you can plug in there something that will pull it in over REST, or XML, or whatever. So, the idea is that you have plugin points for these different use cases. And people can swap in different data on it just as they need to.

So again, you don’t try and dictate that, because I think this is one of the dangers that people get into with a grand… because we’re all programmers. And one thing about programmers is you go two steps and then you think, “Oh, I could go two steps more.” And so, if you give someone a definition of an object, it’s very tempting just to try and generate the entire app from that, because you could, right? You know roughly where the database is going to be and you know roughly where the UI is going to be. It will look awful. But you can do it. And so, you have to consciously with an OIM not do that and say, “Okay. I’m going to build the little bit that maps the business to the UI, or the domain objects to the UI.” But then leave everything else pluggable, open-ended. If they want to use REST, they can use REST. If they want to use JSON Schema, they can use JSON Schema, that kind of thing.

So, the other issue that I see is that sometimes, at least in Rails, you wind up… or outside of Rails, you wind up using Active Record. And so, I’m wondering. Do these plug nicely into frameworks like Ember or Angular? I’m assuming Backbone’s not that terrible, because you’re essentially just replacing the Backbone model setup.

There is a Backbone, what I would call as Backbone OIM project called Backbone Forms already, which tries to do a similar kind of thing. Because even in Backbone, you still have to out of the box write the HTML code and get it right. So, they do have an equivalent idea. And this is part of the field, is that a lot of people have reinvented this same idea over and over again on a lot of different platforms over the years. A lot of people have looked at building their UI and gone, “Gee, I’m a programmer. I should be able to use a bit of reflection here and do this stuff programmatically without having to do it all by hand.”

It’s very common. So, there are OIM-type tools out there for a lot of different platforms. None of them, or there’s not a big groundswell around this, I don’t think, in terms of being standardized. But a lot of people have written this kind of idea many times over.

I think a side-effect of that so far is if you try to make it right for everybody, you end up making it so complicated that you might as well type it all out yourself. Like, it’s so much configuration that you have to end up configuring it instead of actually writing out the HTML form. And then you look at it and you’re like, at least I had that problem, like, “Gee, this is more code than if I actually wrote out that HTML form myself.”

Yeah, you end up [inaudible]...

So, it’s a tough balance.

Look, it is a tough balance.

You end up with [inaudible].

But I would probably argue that even in that worst case where the configuration code you have is bordering on the same amount of code you would have written by hand, the fact that you haven’t written it by hand means you lose that duplication problem. So, even if you have a lot of code, at least it’s not code that will subtly break without any warning down the track. At least you’ve removed that issue of duplicating definitions.

Another thing that’s interesting for me is, well as I say, I came to this from the angle that Richard pointed out and I mentioned earlier, which is that I wrote an API and then decided to do something. And you also mentioned JSON Schema already, because that’s definitely where I’m coming from, being one of the authors of the JSON Schema format. And so, we ended up with this idea that actually when things were in flux, that you didn’t want a UI at all. And so, it wasn’t even a question of, do we tweak this? Do we configure this? It’s more that when you’ve got so many things changing in your alpha, alpha, alpha release, then the dynamism is super important.

And I think that’s why I’m excited about this form generation or UI generation, whatever name you put on it, because you mentioned this idea of 300 forms or something in an enterprise application. And I think that that’s the way that things are going, because you’re going to end up with so many different interfaces and new things that people invent. And you might want to be able to systematically interface with all of them. So, you end up with this constant stream of new things to plug into. And creating a UI for each of them is infeasible. And as you say, you’re going to mess up one of them. The more code you write, the more mistakes you’re going to make, the more typing you do. So yeah, I think that’s why this is so exciting.

You’re right. It would be a terrific world if next time you build your app and then you’ve got a web client and you’ve got an iPhone app and an Android app and an Apple Watch and whatever interfaces they have, and then someone says to you, “Hey, we need to add a new field into our app.” Nowadays, that could be a massive engineering challenge, because you’ve got to go around potentially three or four different clients and make sure everything’s updated correctly. If all of those clients were using some kind of OIIM technology, then all of them would update automatically for you in sync with whatever change you just made on the backend. And you wouldn’t have this class of problems coming in.

Absolutely. So, that’s been one of the dreams of various meta-formats and so on, is that if you can sufficiently describe the data… so many people have taken swings at this. You think, “Okay. Well [chuckles] obviously this is one of those traps for programmers.” But the dream is still there. [Chuckles]

That if you’ve got enough information about the data then you can present the bits of the data you don’t understand in a way that humans can still interpret meaningfully and that doesn’t look horribly jarring. So, I’ve got a couple of different perspectives on this, because I came at it first from a web design point of view. And then I found that I ended up working in the Internet of Things for ARM, the chip designer. And the message is that everything’s going to be connected. Everything’s going to have an API. And then you want to have these UIs for the things around you.

And one of the solutions is this physical web kind of solution, which is Google’s Physical Web thing where you just, you give them a URL and then they make their own interface in HTML or JavaScript or whatever it is. And then you leave it alone. But the other thing, if you wanted to tie into an interface, then you’d instead of describing the UI, you describe what you need. And then they assemble a UI that is appropriate for them. And that’s very similar to this kind of thing that I think we want to see on websites where you have, say I don’t know, some WordPress plugin or something that you want to link into a Google Doc, Google Form, or I don’t know, some new API you’d never seen before. And if it’s sufficiently described itself, then it can just drop in. You can be like, “Give me one of these,” and it will give you something nice at the end of it. And so, I think interoperability as well as ease of use is one of the strengths, and one the pieces of promise [that it has].

So, I’m looking at this solution. And I know you talked a little bit about this already. But I’m looking at this idea. And it seems like there are some apps where you just need a couple of little JavaScript [inaudible] here or there, some widgets or this or that, that are really simple, and this might even be overkill for. And then on the other end you’ve got the fully-integrated app that you probably want to pull in Ember or Angular or something on, because it’s going to give you some full-stack manipulation stuff and you can totally manage your entire experience with it. And this comes in somewhere in between. So, how do you make the judgment call to say, “You know what? I don’t need a framework, or I’m going to use this in conjunction with a framework,” as opposed to going full Angular or full Ember or whatever?

Right. So, I guess from my perspective, I don’t think this is a halfway thing. I don’t think something like Angular does solve this problem at all. That is to say, when you’re developing an Angular app, if you look at the examples of how to develop an Angular app, they’re still very heavy on you have to write out your UI code by hand. They’ll give you a bit of an ng-model to help you with a bit of binding. But that’s only binding you to one field. It’s not binding you to an actual object and all the fields therein.

And so, something like Metawidget which is just one implementation of an OIM comes with plugins for many, many different frameworks. So, there’s an Angular version of it that appears and an Angular tag. And so, instead of saying

<input type=”text” ng-model=”person.first_name”> to map it to that particular field, there’s a metawidget tag and you do <metawidget ngmodel=”person”>

And so, that’s the whole tag. And then that tag expands at runtime into being the little form that contains whichever fields in whatever order with the correct types and the correct validations and restrictions on them. And then that, you can drop in, in your Angular app any way you like. And so, it’s as important for Angular as it is. But we also have plugins for jQuery mobile and a lot of the [Java] frameworks out there.

So, it’s still something that I think… I haven’t seen a UI framework yet that has actually tried to handle this problem natively. Most of them walk you up close to it and then just stop. They get to the data binding stage or maybe they’ll have some validation stuff in there. But they’ll stop short of actually saying, you know, a large part of this problem is that you’re having to write div, label, input, div, label, input over and over again in your code. They’re not really trying to address that part of this issue.

Well, WebComponent seems kind of similar. And you see things like React that do a lot of the… you spell out a lot of the UI but you’re talking more about generating the kind of bindings and plugging into…

I’m saying that bindings should go beyond a single field, because…

Right, yeah, yeah, yeah.

Yes, you’re binding the single field. But clearly we know more about this. You having to do person.first_name, person.surname, well, why not just do person? And so, WebComponent is a good point. So, Metawidget 4 recently added WebComponent support. So now yes, it comes with an HTML tag just called metawidget. So, instead of writing out all your form, all your divs, all your layout and everything by hand… and some of these layouts, Bootstrap and things, there’s quite a lot involved in getting all the divs and all the CSS classes and everything correct.

And generally speaking, you’re cutting and pasting. But there’s a lot of stuff you’re cutting and pasting. If all of that got collapsed into a single tag, which is highly configurable and doesn’t restrict you and doesn’t get in the way, but does generate based on the stuff that has to be in sync, there’s no benefit to flexibly allowing you to have the label different from the field it’s pointing to. Having to do that by hand is not offering you anything. All it’s offering you is a margin for error.

You said a good thing. I like the idea of WebComponents because I’m pretty sure that in the future it doesn’t really matter what framework, what it’s using. You can throw whatever you want on your page and then you can use it. So, we’d have a form for and then you just give it your JSON Schema endpoint or your API endpoint that describes how its data look in some form. And then it’ll just put a form on the page for you, right? And then you can start and modify it with CSS and the overrides that WebComponents allow. But you get your standard form already. And then you just give your WebComponent a callback that says, “Okay, when the user submits the form, give me all the data that we put in there,” right?

Absolutely, yeah.

So, it doesn’t really matter what framework we’re using. We’re using WebComponents and we’ll get a form for our API endpoint. And if it changes, we’ll have it update it automatically on our page. It might not look perfect right away, but that’s kind of… WebComponents work really well with that idea, I think.

Yeah.

And that’s why I think it’s super important that the OIM, whatever, it doesn’t try and tie itself to a particular technology. If you had an ORM that only worked with Oracle’s database, it wouldn’t be very useful to a lot of people. And so, as an OIM, you really need to be open to working with whatever UI technology that people want to adopt. If they want to use Angular, if they want to use jQuery Mobile, if they want to use WebComponents, that shouldn’t impact their decision to just remove this problem of duplication between their UI and their domain objects.

There’s also, I guess there’s the danger of if you tie yourself into a particular… or if you’re using an OIM then yeah, you can change the other frameworks underneath it. But you’re then, you’re tied to that OIM as well. You’ve got this…

Absolutely.

You have to make that decision fairly on. And so, I think… so, if I could try and go back and answer Chuck’s answer from a while ago, just how would you choose when to use different bits and how do you transition between them? Is that…?

Yeah, I think that’s definitely another angle on what I was asking.

Yeah. So, I think my answer to that would be the whole point of this UI generation, form generation, OIM, whatever, is that [chuckles] I don’t know how to say this. It shouldn’t be your problem unless it’s your problem. So that, that’s a judgment call made for, up to the people who make the UI. And so, it’s decoupling it.

Someone should in theory be able to create this UI. And as David says, maybe it’s a JSON Schema and then some styling. Or maybe it’s a framework. Maybe it’s Metawidget. Maybe it’s whatever. But the point is that you have the different people working on different things. One person’s busy vaguely saying what it should look like. And then someone else should be able to swap out how it works and say, “Oh no. We’re going with Angular.” And then you plug in your different, your OIM backend or whatever it is. And so, that kind of progressive replacement of bits of your UI is I think the essential bit.

The thing that I ended up making which is a project called Jsonary was the coolest use that I found is actually a Chrome plugin where you, whenever you view a JSON API which has been annotated with JSON Schema, suddenly it gets buttons all over it and forms and stuff. So, you point at a JSON API in your browser and it suddenly inter-actable with. And I think that’s pretty exciting. But that’s not really usable. But the point is you, as a backend person should be able to use this API and not care about the UI. And then there’s someone at a different area if you’re a group who’s making forms and making other stuff and making judgment calls about it. Like, “Hey, we’ve mocked up this prototype. But half the buttons don’t work. I’ve decided we’re going to go with React, or we’re going to go with WebComponents,” or whatever it is. And then they can make that choice.

And if you’ve got this system where the UI is in general generated by the information requirements rather than specific hooks from any given point, then you end up with this flexibility. And so, as a backend person, I shouldn’t have to care what the judgment call is or what we’re using at the front. Even as a midlevel person, someone writing Schemas or config files for display. And then the frontend person can then liaise between the models that have been described and the designer’s vision, using whatever they like and whatever OIM backend, as Richard’s calling them.

And so, I think that’s my answer, is that it’s undefined. It’s a judgment call. But that’s the point, is that it’s not a judgment call that you have to care about unless you’re the person making it.

So, I have a hard time seeing that we’ll realistically get to a point where we could have something that abstracts between frontend frameworks. Coz when I look at the backend, it’s like, well I’ve got Oracle and I’ve got MySQL and Postgres and SQLite and a couple of different things. And although every single one of them escapes differently, one uses backticks, one uses single quotes, one uses double quotes, there are some really annoying differences. But the SQL syntax and the way it works is the same.

When we’re looking at things like React and Angular, both React and Angular claim to have built a better browser. They don’t claim to be working as a layer on top of the browser. They’re saying, “We are a browser that abstracts your browser away.” So, it’s like it’s going the reverse direction from browser DOM implementations being like SQL where they’re kind of the same but kind of a lot different, kind of annoying, to completely different things that aren’t similar at all.

Absolutely, yeah. Look, it is a different problem. There’s a lot more variables I think in going up to the UI than going down to the database. But at the same time, the feeling that you have in your gut is while you’re typing out whether you’re using Angular or React, whatever framework you’re using, while you’re busy typing out your UI, you know in your gut that a lot of what you’re typing is basically boilerplate code, and has to be correct.

And so, although it’s not clear how you automate that, whether runtime generated or statically generated (and there’s obviously problems and challenges in doing that) you know in your gut it is something that could really do with being better. Because at the moment, you are typing stuff that you know, if you mistype this then either you’re going to get a subtle error or you’re going to get no error at all. And one day, something will blow up.

Yeah. So, I totally agree that it should be better. It’s just, just trying to get people to agree on how we should do Angular, just if we were trying to get people to agree we should do Angular this way, and the API for UIs for Angular should be this way, that alone I see as just a huge challenge. Look at Yeoman-generator versus Angular-seed versus these other patterns.

Yes. And I think it’s important to really restrict what you’re trying to do here. And that’s what I was sort of getting at before, is if you try and generate too much, you do lose feasibility I suppose. If you… I think it’s possible to generate just those pieces of HTML that we know are pretty rigidly defined. But if I could just go to what Geraint was saying, it made an excellent point. One of the things that you get from the decoupling of this UI generation type layer is, yes, you lose a lot of the duplication. But one of the things that we’ve seen with Metawidget is the choice of control now gets left up to this decoupled place.

So for example, if you have in your app somewhere where someone has to enter in a color, and initially with Angular out of the box or whatever framework you’re using out of the box, the best you can do there is a textbox with a type and an RGB number or code. And so, that’s how you develop your app initially. And that’s probably fine for the first version. But then down the track someone comes along with a color picker component. Now, if you’ve manually gone and put all your screens together, you might think that color component’s great. But you have to then go around all your screens and swap out the textbox for the color component on everywhere that you’ve accepted that input. Whereas if you’ve…

Yes, and I…

If you decoupled that choice, then clearly it’s a one-line change in the OIM to say, “Hey, anytime you’re trying to render something that needs to take a color, instead of spitting out a textbox, why don’t you spit out this fancy new color picker control that we’ve just got.” And suddenly, everything in your app’s updated.

So, I think that that’s an abstraction of the… well sorry, the concern raised out is kind of an abstraction of that. But it’s not just that when you want to switch out the UI component. Like if you make a major choice like, “Oh, we’re going to use React instead of Angular” or even have a big tiff about how you use Angular, you have this, oh, do you go through and replace all your different… can you just swap out a backend that seamlessly works all of these? And as far as I can see, maybe there’s going to be some grand form generator that wins out over everything else. And maybe there’s not. Maybe we’re going to end up with, there’s going to be a form generator for, or several form generators for Angular. And one eventually wins.

And there’s going to be a form generator for React or something, or various plugins. But we’re still pursuing this idea that none of that should be the concern of the person who’s writing the data models. That you should have this faith that you can add a field and then add something to a model somewhere. And then whatever framework that the person above you in the abstraction chain is, or the UI chain, is using, that it will be able to accommodate this stuff.

And so, you don’t have this starting point where you start out from this horrible hardcoded stuff and then you go, “Oh, okay, let’s write a whole lot of schemas for this particular ORM.” That you end up writing, you write these models and maybe you have to batch convert the models to something else, something horrible. Because we’re never going to do away with all these problems. But maybe, if you make a huge change from Angular to React, then the pain should be fairly minimal if both systems have a form generator that work off some abstracted model of the shape of the data.

Because you have, the more you’ve abstracted about the nature of the data and the nature of the requirements that the form has, then the less you have to rewrite when you change stuff. Similar to the [LESS app], you have to rewrite when you change what a color picker looks like. So, even at the large scale, when you’re changing OIMs or changing something fundamental, which means that you have to treat your OIM slightly different [or you’ll call it], or I don’t know what the model is. Maybe you get incremental updates or you get progress change events as opposed to a submit button, whatever it is. If there is that sort of ubiquitous, if everyone has this abstraction and this is a common way to approach this kind of information display and input, then you waste less effort even when making quite big changes. Does that answer anyone’s? [Chuckles]

I think it definitely eases the pain. I know for example going from Bootstrap 2 to Bootstrap 3 for a lot of people was a lot of rework. They’ve changed a lot of the CSS. They’ve changed a lot of how your forms and stuff were structured. Not a great deal, but subtly, but enough that you had to go and revisit all your code and update everything. And in some ways, that was Bootstrap’s problem, not yours. So, if Bootstrap had a plugin for some kind of OIM that generated their particular type of CSS and div structure based on how they liked it, then when they went from 2 to 3, rather than having everyone else go and tweak and update their code, they could have just updated their little OIM plugin. So, instead of spitting out Bootstrap 2 CSS and divs it’s spitting out Bootstrap 3. And so, the amount of rework that everyone else has to, that gets inflicted on everyone else is that much less pain.

So, I guess the other question I have is what examples are there out there? You guys have talked about a few of the examples of OIMs out there. Are there certain ones that you should consider?

Or should you write your own?

Well, okay. So, I think a few of us at least are going to be fairly biased on the basis that we’d like to plug them. But I think on the basis of my conclusion that I don’t know whether there is necessarily going to be one that works equally well on React and Angular and WebComponents or whatever, because you’re going to end up with this, the fundamental different paradigms of how they do updates or whatever it is. Maybe you can do that. And that’s a difficult problem. But I think actually, the fact, just evangelizing this idea that it’s a good idea that if you’ve got your dynamic information requirements that you should have a dynamically assembled UI is a useful model to be thinking about.

In terms of different options, there’s the JSON Schema site, which is the project I’m involved in, has a set of implementations. And so, there are a few different examples for UI generation. One of them is Metawidget, which is Richard’s. One of them is Jsonary, which is mine. There’s one called JSON Editor. And I’m absolutely sure that there are several which are not on the list.

And so, from my point of view, I think actually keeping innovating is pretty important. Because I don’t think we’ve necessarily… I don’t feel this is a field which has got a clear winner yet. There is no clear winner for web frameworks. We’re still arguing about React. And then someone releases a new one that’s like, “Oh, this one’s only four kilobytes.” We haven’t solved that and we might never solve this other one as well. But pick an existing one, write a new one if it sounds like fun and you don’t need it to work. I think just the same as all of the coding, but as a pattern, I think form generation, UI generation based on dynamic requirements is a cool thing to be pursuing.

Yes, that’s absolutely correct. I think, I would speak with David here as well as myself in that when you build these tools, you’re building them to solve a particular problem that you’ve got. If someone’s already written that tool to solve that particular problem, you’d be more than happy to use that. But as it was, at the time, so an ongoing basis, it doesn’t really bother me which tool becomes a successful OIM or whatever. But as long as someone’s got that tool. What I see at the moment though is oftentimes when you… frontend frameworks and stuff came out, there’s just nothing in this area. And so, immediately you’re back to writing all these duplicate definitions out. And the pain all comes back again. And you wish that this was a more commonly accepted pattern that everyone was trying to get behind.

Very nice. Are there any other aspects of this that we should talk about? One other that comes to mind just as I’m asking is testing.

Sure, I can briefly talk to that. I think the testing story is excellent when you come to an OIM because of the fact that it can generate you some really nice UI code. So, oftentimes when you’re doing testing, particularly integration testing, the main thing you need are sensible id’s on things so that your jQuery selectors or your Angular integration tests can select these various fields in ways that are predictable.

And so, oftentimes when you’re writing your tests, you’re finding yourself going, “Oh, hang on. I forgot to name this field,” or, “I gave it an id that, I got my uppercase and lowercase mixed up,” or, “It’s not sensible the way I want it. So now, I’m switching between writing my tests and going back and fixing my app a little bit so that it’s more testable,” and that kind of thing. Because the UI code that you write by hand is not necessarily pristine, whereas when it’s generated, then you can do a much nicer job on all the id’s and all the naming of everything. So that when you come to do your integration testing, the thing you’re testing against is already pristine.

And you are taking out that [inaudible] error. If the generator’s good enough, right, you could almost be sure, well not be sure but you don’t have to test every single field all the time just because you know it’ll be tied to your backend. What Richard is saying, you’d end up having to test every single field and making sure that end-to-end actually it shows up in the database as well. And with that, we can actually make sure that in the normal case it will work, versus in the normal case it’ll break because you made a typo or something. So, I think that makes it a little more safe to actually work with.

Very nice. Alright, well I don’t think I have any other questions. Do you, AJ?

Not at this time.

Alright. Well, let’s go ahead and do some picks then. AJ, do you have some picks for us?

Yes, I do. So, as many of you are well aware, Majora’s Mask 3DS came out. And it is good. And you should buy it while you can, because the same thing that happened to Ocarina of Time or worse is going to happen to Majora’s Mask. Right now, if you want to get Ocarina of Time, you got to pay $80, $90 to get a copy, used. So, if you’re on the fence about it, just go ahead and get one. It’s a fun game. And I’m glad to be able to play it. And there’s a hint system in it so that it’s not impossibly difficult this time.

Alright. I’m going to pick a couple of books. The first one is ‘80/20 Marketing’ by Perry Marshall. I listened to it on Audible. It’s about six hours long on Audible. I was listening at double speed or something, so I listened to it in three hours or so. It was so good. Definitely something I’m going to go back and re-read.

The other book that I’ve listened to was ‘The Wizard of Earthsea’. And this is an older fantasy book, just a terrific fun book to read. So, I’m going to pick that.

And then the last book that I’ve been reading or that I’ve read lately is called ‘Conform’. If you’re in the US and you’re concerned about the education system, regardless… So, it’s written by Glenn Beck. And I know that he’s kind of controversial in some circles. [Chuckles]

We’ll just put it that way. But regardless of which side of the aisle you’re on, I’ve been checking some of the stuff that he points out in the book and finding that a lot more of it is true than I really want to be. I actually tend to agree probably 70% of the time with Glenn. So, it’s just really frustrating to see where things are at. So, if you’re interested in that, you can go check it out. It’s called ‘Conform’.

And finally, I’m reading another book by Glenn now. But this one’s just stories from US history. And it’s really interesting to read. And it’s called ‘Miracles and Massacres’. And the stories that I read so far are from the revolutionary war. There’s no political slant on it. It’s just stuff that happened. So anyway, history’s really interesting to me. The situation politically is also very interesting to me.

Anyway, so those are my picks.

Richard, what are your picks?

I do have one. I went to one of these hackathon type weekends recently where they lock you in and you sleep on the floor. And it was a room full of probably 200 people, all technology people. And I’m a technology person and I guess I’m used to being… if I was in a room full of cricket fans, I wouldn’t think that I would have much to add to that conversation. But if I’m in a room full of technology people, I kind of expect to be able to offer something of value. And it was a real eyeopener to me that probably, maybe a third of the people in that room, so dozens of people in that room, were doing something which I… was completely opaque to me, I had no value in at all, which was 3D modeling.

So, some people were doing it because they were developing games. And some people were doing it because they were doing VR. There were a lot of 3D printing people. There were some CGI type people. But it occurs to me that 3D modeling as a skill is one of these real gateway, or is becoming a real gateway kind of thing that I wonder for my kids, it’s going to be important for them to have that core skill which will then open up a lot of different careers for them. So, I’m now trying to get into that.

So, I’m using a software called Blender, which is a 3D modeling package. And I’ve also got a, I’ll do a shout-out to a 3D printer company called Me3D, so Me3D.com.au. And I got a 3D printer off them. So, I’m trying to skill up in an area that is completely new to me.

Alright. David, what are your picks?

I have two. One, I like to listen to music when I work. And top 40’s not my thing. And I can only recommend to everybody I found my best music lately on Bandcamp. I’m just going to post a link in there. There’s a lot of underground music in there and new things that… it’s really awesome to discover new things. And their share model is actually pretty fantastic as well. I ended up, even if the music is also on iTunes, ended going with Bandcamp because they only charge 15% instead of 30 that iTunes does. So, it helps the artists a lot. And people are very grateful. And they probably have the best pricing model as well when it comes to publishing music, as a band that’s [inaudible], it’s something.

And the other thing, I’m going to share some books, too. I’m currently reading the ‘Zones of Thought’ series by Vernor Vinge. He’s a professor for mathematics, computer science, and also one of my favorite sci-fi authors now. He wrote this series in a universe that has AI in it an alien lifeforms and was probably the best sci-fi I’ve ever read. And I did all of the ones before and this was part of it. So, I can only recommend that series. It’s pretty fantastic.

Awesome. Geraint. Did I say that right this time?

It’s close enough.

Close enough?

So, I think, well I have to give the kind of obligatory plug to Citizenfour. I know everyone’s heard of it by now because it won everything. But yeah, it’s amazing. Everyone should go and see it at some point.

In terms of music, I’ve been quite into Solar Fields. It’s amazing trance-like stuff, very relaxing and hypnotic.

I also came across some interesting JavaScript stuff which is again kind of recreational for me because it’s not really to do with what I do day to day. But someone wrote an implementation of PGP in JavaScript. So, OpenPGP.js.

And Forge which is an implementation of TLS in JavaScript. So, you can do an entire TLS stack browser-side, which is just one of those things where I’m like, “Why did you do that? But it’s great that you did.” And I think that’s it for recommendations for me.

Alright. Well, thank you guys for coming and talking about this. It’s really interesting to see where it fits into the development ecosystem and just how it can be used to simplify our code.

Pleasure.

Thanks.

Alright. Well, let’s wrap up this show. We’ll catch you all next week.