How to Handle a Bunch of Requests Using JS Promises
Yesterday I procrastinated the stuff I wanted (and should) do and spent a lot of time browsing Github and checking what is going on in the JS world. I discoverd a discussion held by the grunt guys about how Grunt can be promoted better. That was quite a good read and it was really nice to see, that these people try to push Grunt forward to make tooling much better for everyone.
A lot of new issues were created at Github to push the project to the next level. It turns out that the Gruntjs.com website is a seperate repository whose code is available on Github (man, I really love that Open Source approach).
What else can I do than checking out the source? I mean the website of Grunt itself must include a lot of best practices and stuff to discover. I forked it and opened in my editor and there they were – a lot of JS promises…
Basically I know how promises work, but I have to admit that they still irritate me a bit. It is just another way of thinking and sometimes it takes me a while to understand what is going on when the code in front of me is promises based.
The Grunt website makes usage of the framework Q. There are a lot of promises frameworks out there, but Q is with over 10000 downloads a day according to NPM stats not a bad choice. ;)
Handling of requests at /plugins
There is not much functionality in the site (most of it is static content), so the most interessting part of it is probably the
plugins site. It fetches all available Grunt plugins from “somewhere” – I will explain where it comes from, so keep reading ;) – and displays them. Let us check, what is going on there.
The fun part starts inside of a file called
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
What we see here is a route definition for the Express Framework. If you are not familiar with Express it describes itself as follows:
Express is a minimal and flexible node.js web application framework, providing a robust set of features for building single and multi-page, and hybrid web applications.
It is a quite handy framework and when you are into node.js it is definitely worth a try.
But anyway… The shown function will be called when a user enters the url
/plugin-list. And there it is – the first promise
pluginListEntity. You see that it is a promise, because an included function
then gets called with an anonymous callback function. This is the basic pattern when dealing with promises.
Comparing callback and promise way
Particular functions return a promise object instead of the wished data. This becomes quite handy when you have to deal with asynchronous operations, because the “normal” way would be to implement a callback function that deals with wished data, when the operation succeeded.
1 2 3 4 5 6 7 8 9 10 11 12
So far so good – let us check where this variable
pluginListEntity seen in the first code block comes from. It is basically the result of a function called
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
What we see here is the actual “kick off” functionality for the plugin fetch process. It gets called at the initial start of the Express application. The most important thing is, that the variable
pluginListEntity is set with a promise at initial start by calling
getPluginListEntity does two things when called. First of all it creates a new promise and returns it. Additionally it refreshes
pluginListEntity with the created promise. That is actually really smart, because this way it is possible to update it automatically using
setInterval constantly which is absolutely necessary, because gruntjs.com should be up to date all the time.
But wait… you may have noticed that there is more promises stuff going on inside of
getPluginListEntity. This line
gruntPlugins.fetchPluginList.then( ... ) hints to another object, that does the actual fetching job for the application and obviously return another promise.
Let us check it out.
Fetching data from NPM
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
BAM!!! That is it – much more promises –, but let us break it into little pieces. :)
To wrap up what is needed to do:
- Fetch all plugins from database ‘reqistry’ that include keyword ‘gruntplugin’ – one call
- Fetch plugin details of all fetched grunt plugins from database ‘registry’ – * calls
- Fetch download statistics of all fetched plugin – * calls
Just to make sure you understand to advantage of that, here is the callback way of doing it:
1 2 3 4 5 6 7
Doing it like that is less readable and it is a perfect example of the so called ‘callback hell’. Additionally dealing with a lot asynchronous requests can be really tricky and keeping it in sync to call the next callback is not so easy like it sounds.
So what is really going on? The function
fetchPluginList makes usage of
Q.fcall. This gives us the possibility to return a new promise by this function. Following is a simplified version of this approach to make it clearer.
Example for Q.fcall
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
This way we are able to stick a lot of functions together and avoid creating callback trees. The end result of a deep nested callback is then easily accessible using the
then function of the particular promise.
Example of Q.defer with one request
Now starts the tricky part – in each step ( remember
fetchDownloads? ;) ) will be made an asynchronous request and this request is handled by …? Yeah, you are right – another promise.
Here is the complete first step to fetch all grunt plugins including one call to the CouchDB:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
A new promise is created by using
Q.defer acts as an interface when you have to deal with callback based functions ( like
request in this case ) and you want to do it the promise way. All you have to do is getting a new defer object by calling
Q.defer() and then resolving/rejecting it inside of the asynchronous callback function. The response of the request is then easily accessible be calling
1 2 3 4 5
Example of Q.defer with multiple request
If the first step is clear to you ( if not feel free to comment or ping me on Twitter ) let us check the second and third, because there is a bit more magic going on in it.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
This step does basically the same as the first step, but makes a lot of more requests. For each plugin a seperate request has to be made to fetch the plugin details. The function will be executed with an Array containing all information that was fetched in first call to get all grunt plugins.
This Array (
list ) will be transformed using the
map function of lo-dash. Each item is replaced by a new deffered object which will be resolved, when the request for plugin information succeeded. Q provides the really nice function
Q.all which gives us a lot of power to handle multiple requests.
Q.all will return a new promise which will be resolved when all promises inside of the handed in Array will be resolved and enriched with detail information.
1 2 3 4 5 6 7 8 9 10 11 12 13 14
Sticking promises together
Now, we have created a lot of promises, so let us have another look where we started with a more detailed look and lots of comments. ;)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98
fetchPluginList function looks first really heavy, but when you understood the principle ( looking at you @TeixeiraPedro ;) ) the code has much more structure and is much easier to read. Especially lots of asynchronous operations that has to be in sync at some point loose complexity by using
Q.all, which absolutely blew my mind.
For me it is clear, that I will structure next projects with lots of requests definitely promises based like the Grunt guys do to make my life easier.
And that is it. I hope you enjoyed it and thanks. :)