CMOS #3: Joel Berger on Mojolicious
Mojolicious is a modern, non-blocking Web framework in Perl. Joel Berger member of the development team talks about the project and on how to get started with it.
But I had learned some Perl. And so, I started making big models for these electrons and learned quite a bit of programming that way. And eventually once I got my PhD, it happened that the US had its a sequester, if you have heard this term.
Basically where the government decided that in order to cut the budgets they were going to do some sort of a deal where they would all cut the budgets at the same time but they all excluded their favorite things. So, science got cut more than the rest and it was very hard for scientists. So, luckily I had quite a bit of Perl programming by this point because of all that modeling I was doing and I asked around at my local Perl mongers, Chicago Perl mongers where I am still quite active and they found me a job in the programming world and I have been programming ever since, mostly in Perl and now I am lucky enough to actually be working in Mojolicious shop here in Chicago.
It is called Server Central, we do server hosting and whatever but we are doing a lot of our back-end work now in Mojolicious as well as some other languages. The other groups are using some other languages but of course the group that I am in hired me because they were doing Mojolicious already. It is been a lot of fun working there.
So, we are writing software and an API and all this to actually spin up real physical servers in that sort of same convenient way and that will help us respond more when a new client will come to us and say "Hey, we need 50 new servers." We will not need to take all the humans out to the racks and get them installing operating systems and all that. We are going to have API calls and all that other fancy stuff too so that they can provision operating systems and set up networks and such all through the web frontend and that will all run through Mojolicious and we have actually got a fair amount of working ready. So, it is a very exciting project.
I think as many people do, you sort of pick your first web framework because you have heard of it and the thing that really kept me were some of the features that I am sure we will get to describe later but specifically the asynchronous stuff, really played with my brain in the right way. I really enjoyed working on non-blocking code.
So, all of the incoming calls, outgoing calls are all event based, but that also lets you have the freedom to do things like asynchronous database queries or responding by web sockets or all kinds of other things you cannot do when you have to do a procedural request response platform.
And because Perl did not have a framework built on a non-blocking event loop like that, it was rather difficult to write a web application in that way. It was not impossible. You could use POE or you could use AnyEvent to write this but you had to be a little closer to the event loop than most people would like to be in. So, when you build a web framework on top of it, it isolates you from that. You can do a lot more. It feels more like regular programming for the most part as much as non-blocking ever can do that.
Of course if you start doing things like making 12 calls, three different services and you can do them in batches of whatever, it gets complex of course but you do not need to start at that level.
And actually I should say here a lot of people come to us and say "Oh, it is most Mojolicious but I am afraid of it because it is non-blocking. That scares me." You do not need to do anything non-blocking. It works just fine as a blocking framework. Still there is parts that are reacting in the background but you do not have to think about that at all if you do not want to.
The only thing that becomes a little bit difficult is if you have got very long running processes that you have started. Because you have now blocked the loop, the next iterations of the loop also were blocked.
It is a tricky question, right? In a normal web app you will have no problem going to sort of not-blocking incrementally. The bigger problem is you have got sort of gigantic database queries that take minutes to complete or other crazy things but usually anything where you have not had to extend the client's timeout, you probably would be fine anyway because only the client will timeout first anyway.
First of all you can do sort of, well, that is the easiest way and particularly if you are using one of the.. So, Mojolicious project has spawned a couple of sort of spin-off projects that wrap databases in a more asynchronous way. The primary one is Postgres.
Our project founder Sebastian Riedel has really taken on to Postgres lately and actually one of other core members Abhijit Menon-Sen, I hope I am saying that right, is Crab on IRC, that is a lot easier to say, is also on the Postgres development team. So, that is a really cool crossover we have there.
When we have questions about PostgreS, we can just call over Crab and say "Hey, this thing is confusing us" and he can help us out but we have a wrapper. It is still just using DBI and DBD::PG but it provides you a non-blocking interface and when you do that, you have got this query that is going to take 5 minutes to run and assuming you have set up all your timeouts correctly, you can keep serving other clients while that query is running.
And when the query is completed and it starts sending you data back, then you can respond to that client. That is the easiest way to do it. If you cannot do that for some reason and if you are not using a database that has a reactive or a non-blocking client, then you can do several different sorts of forking where you fork after getting the request, do your computation in some other process and then you pipe the results back to your original server.
And Jan Henning Thorsen, another of the Mojo core developers has one called Mojo::IOLoop::ReadWriteFork. You can actually interact with your forked process. You can send it data over a pipe and it thinks that it is standard input and you can send data back over it. So, that is where we are doing really long running things where you want to monitor some process or whatever. So, there are plenty of options for what we call unblocking code and they all work fairly well with a slight asterisk around Windows of course because forking and piping and things do not work so well on Windows.
Actually with the new Windows subsystem for Linux which is the real name, people have been calling Bash on windows, it is really called WSL or Windows Subsystem for Linux is we have actually had some fairly good reports of Mojolicious. The corner cases that Windows did not work as well are working quite nicely on Windows Subsystem for Linux. So, we are very excited about that.
We are very used to in the Perl world taking pieces from all over different parts of CPAN and each one presents a slightly different interface and we are very used to that and let us not think that that is a bad thing, but Mojolicious was designed really to work together with itself quite well.
You can of course use whichever modules you want but if you use the tools that come with it and it comes with a lot of built-in tools like a DOM parser and JSON parser and. I am sure I am going to forget everything because I am on the spot here, but it comes with lots of batteries included and the really nice thing as you get to use them more and more is that they were all designed with the same API in mind and consistency is always a key for us.
I already watched your first video with Jason Chrome from the Dancer group and I think I probably ought to comment that for a lot of the people in the Perl world we value stability over everything else and I think it would be wrong to not say that Mojolicious takes a slightly different approach to this. We have a very defined breakage policy where we can at specific times break backwards compatibility and we have done that more in the past then we have lately but we have done quite a bit and the point has been it allows us to keep making the framework more and more consistent and if we need to change a thing that would change that consistency, we can change at other places too and the code as a result has now gotten to be very lean and very easily read while still providing really consistent API. And as it turns out, after a while of iterating like that we have not actually broken much lately. So, Mojo got this reputation for "Oh, you know, you have to be very careful about your dependencies" and we will probably never say that it is never going to change again but it is gotten quite stable lately and quite beautiful as a result of our willingness to iterate on it.
We also have then a porting document that we try to keep up-to-date whenever we make a breaking change. It says "This thing has changed. Use this one instead." It is in the wiki and for the most part, especially since say about 5.0, we are now into the 7.0 series, the changes have been mostly semantic. You change name here or there, a few behaviors have gone away but for the most part it is been fairly easy. Oh, you used to use this. Use that instead. Not too many things have really just plain out gone away.
It feels a lot like Slack. Its IRC in the browser. You can pull it up on your phone or you can pull it up on your desktop or your laptop and it is all the same connection because you are hosting it in one place and it behaves then as a web server that you can connect to and type into. The web is convos.by if you want to look at that and they have a little demo there and actually it is kind of fun to mention now that they have just done a huge rewrite and it is about to be released for the 1.0 version which should be a lot easier to install. That was a little bit hard to install which was a big problem.
Also, I know Tobi Oetiker, I hope I am it right. Tools are starting to be written into Mojolicious which is quite fun for us to see people doing server administration using Mojolicious based tools. Kind of interesting thing, I did not expect to see but it was fun to see happen.
As far as websites, first of all, we have lists of people who are using the Mojolicious on our wiki and even some user quotes about it. I know for some reason Norway has a huge user base of Mojolicious. So, I know Telenor is using it for some things and there is a travel service out there that Jan Henning is actually moving to. I know they are using it. I wish I knew the name. I do not off the top of my head.
We have some bigger sites. I do not know if I am supposed to say. I think I am allowed to say that we know that mail.ru which is in the Alexa Top 50 actually uses Mojolicious, not for all of their sites but I think as they go, they are coding more and more to it. We were only just allowed to share that recently. So, that is kind of fun for us to say.
I have a nod and a wink that told me that there is some Mojo being used at Craigslist. I hope I am right in saying that but that is kind of fun for us too. And plenty other smaller sites that you can see if you go to our Wiki and see the listings.
We have an IRC channel, IRC.perl.org #mojo and we are quite friendly. So, we try to be at least. If it is not, come tell me and I will help you out. I am your guy.
So, we do most of our communication through IRC. A lot of development is through IRC. We are very proud it. If you come into the IRC and find a bug, we often have it fixed within minutes and released. Because we have kept the code base very clean over this time, we can react to issues very quickly.
I think you will remember back when there is that bug was found in Bugzilla about the code interpolation, the privilege escalation, the same thing that everyone has did you get a list of parameters thing.
So, we are very proud of our ability to respond very quickly especially to security bugs which have been few.
I should probably help him link that into the website. Unfortunately, Glenn has spent more of his time in iOS development lately. So, although he is still around if we ever need to call me in for a for a group chat, he will be there but he has not had as quite as much time to spend on the Perl side but his is podcasts are amazing.
I believe it is Mojocasts.com but let me just quickly Yes, Mojocasts.com. I was worried if it was .org or something but Mojocasts.com. It is really good. And there are a few things that have changed since then we have noted that on the wiki as well.
No, it is not a CGI script although it can actually run under a CGI server. If you have like a shared hosting or some old servers, you can still run Mojolicious under a CGI environment and it will work correctly. You cannot use web sockets or anything because of course CGI has no way to know how to do that. And I also should say that if you are using a PSGI server which is fine, you can do that as well. Mojolicious will run as a PSGI server but again does not have the web sockets because you need to have the event loop in the server level in order for that to work.
So, to say that then we do bundle our own web servers in with distribution. We have a few of them. We have a basic Damon that is very capable and we have a script called Morbo which is nice because as you develop, it will notice that you have saved files and made changes and it will restart the server for you which is quite nice. And then we have a much more capable production server called Hypnotoad and Hypnotoad is basically equivalent to the PSGI server Starman if you are aware Starman, same sort of pre-forking architecture.
It has one extra little feature of hot deployment. So, if you say I have made all these changes I would like to release a new server or on post production, however you want to call this, and you do not want to have to bring your site down, you can actually tell Hypnotoad to start the new server and as the old clients keep being served by that old server, new requests will go to the new server and they never miss a beat.
You can upgrade Mojolicious. You can upgrade your site. You can even upgrade your Perl while doing that. So, it is really quite interesting.
And then I should also say for all that capability, Mojolicious is still quite small. It something about 8,400 lines of code and as its kind of famously known, it has no non-core dependencies but can use CPAN modules for extra behavior like IO::Socket::SSL for SSL connections and all kinds of other things if you want to use CPAN modules but it does not need them. So, it is very easy to install. And it is got 11,000 or so tests. So, it has more tests than lines of code. So, you can be reasonably sure that we have been careful about what we are doing. And all that installs in less than a minute. So, if you are coming from my platform where it was hard to install, Mojolicious might be for you too.
Actually, I think I could say one of the things I particularly like about Mojolicious is there is a test framework called Test::Mojo. It comes in the distribution but I really like how it works and I would encourage people that if you had a hard time testing your web application before, even if it is not a Mojolicious application, you can use Test::Mojo with a plug-in that I have written and I am sure if we give some links here, I can give a link to it but using that Test Mojo and the plug-in that I have written, you can use it to test any other PSGI application as well and it can be very nice to have the expressive nature of Test Mojo for any website you have developed. So, I would give that a look too.