Home > JavaScript > Node.js is no toy

Node.js is no toy

 

Node.js is not a toy. Checkout the new large scale web application I helped build. signforms.com

SignForms

SignForms

 

TLDR:

We produced a very complex, solid, secure, fast and scalable server solution using node.js in a very short timeframe with amazing results. We used a number of different modules and server side tools to achieve our goal.

 

What got me excited about this project was the fact that we were tasked with building it in 2 weeks, and had complete technical control over the project.

After looking into what was going to be required for the project, including security, scalability and complex document processing, we decided to use node.js for the server side and a very powerful client side rendering and binding system named gaffa-js on the client side. My colleague has written a blog post about the front end development if you are interested in reading more about that Deploying a web app in 14 days no html

 

Node.js as a server

I am a big fan of JavaScript and have used it as my primary programing language for some time now. The reason this is even possible is the introduction of node.js.

Node is a JavaScript runtime which allows server-side applications to be written in JavaScript. It excels at delivering fast, efficient data processing with a minimal CPU and Memory footprint. This makes it perfect for what we were trying to achieve with SignForms.

 

Modules

I am not a fan of the “web application frameworks” such as express.js being widely used without thinking about what is really required. I find them bloated, and unnecessary in most situations (kinda like jQuery…) and personally I much prefer to use a number of smaller modules to build my own solution.

Beeline – A laughably simplistic router for node.js

I have been using Beeline for a lot of my projects lately. As it says on the box, it is laughably simple router which takes seconds to setup. It supports everything from simple catch all routes to very powerful regex matching.

Cookies and Keygrip

Cookies and Keygrip are 2 modules from Jed Schmidt that again I have been using in many, if not all of my projects of late. Cookies provides a simple wrapper around the getting and setting HTTP(S) cookie headers, and can be used in conjunction with Keygrip to provide additional security by signing the cookie to prevent tampering.

Bcrypt – The only way to store passwords

If you are doing anything with passwords or any other sensitive data, and you are not using bcrypt, please, please, please get educated. Here is a good place to start. How to Safely Store A Password

Cryptr

Cryptr is a simple encrypt and decrypt module for node.js that I have written myself for doing simple obfuscation / aes256 encryption on values that need to be decrypted at a later time

I currently use this for a number of items which are encrypted for obscurity rather than any real security.

Formidable

Formidable is a module for parsing form data, but comes into its own when processing file uploads.

I had not used Formidable before this project but after doing a lot of research and doing a lot of testing, it came out on top as the simplest, fastest and most secure way of handling multiple file uploads.

 

Database Selection

A lot of time was spent to determine the correct database platform to use for SignForms. In house there was already precedence and knowledge in OrientDB, CouchDB and MongoDB so the decision was around what platform would best suit the type of data we are going to have to store.

While there was a lot of research done and a lot of points considered, a lot of comparisons such as Visual Guide to NoSQL Systems were used as references and eventually we decided on MongoDB. The key decision points for use were:

  • Dynamic Queries
  • Most likely gain benefit from indexes vs map reduce (although we do have some map reduce requirements)
  • Database size will most likely grow large thus distributed partitions maybe be required in the future.
  • Performance of large datasets.
  • Storing a large amount of volatile data, at least until documents are finalised.

Additionally, Mongoose the node.js wrapper for MongoDB was a breeze to setup and work with making database access simple and fast from our chosen server platform.

 

Document Processing

Here in lies the challenge. When looking into the requirements for this project the only thing that really scared me was how the hell was going to process and manipulate the pdf files. I spent days working with different tools and approaches to the problem. I hit many a brick wall but also came across a number of insightful discoveries of things I didn’t even know existed.

This was a challenging journey and probably deserves its own post about how these problems were solved. I will aim to have that post up in a few days.

 

I am very proud of what we have achieved in a short amount of time and owe it all to the speed and power of JavaScript! So go check it out. signforms.com

 

Advertisements
  1. gp
    May 21, 2013 at 1:52 pm

    Hi, What solution did you use for the cpu intensive task? Most task queues for node are either abandoned or not mature enough and forking processes seems to be wasteful.

    • May 21, 2013 at 3:19 pm

      I will go into it a bit more once I have finished the document processing post, but the most CPU intensive process we have so far is producing an image of each page in a PDF.

      We tested a number of approaches to this problem and found that the fastest, least CPU and memory intensive method that still produced good results was a call to Ghostscript with some tweaked parameters via a ‘child_process’ exec command.

      This call is wrapped up in a helper method that handles the stdout and stderr streams and returns them in a callback but it is very simple.

      Using this approach we can process hundreds of pages a second without causing much CPU load and without blocking the original application thread.

  2. Leo
    November 11, 2013 at 8:15 pm

    I’m very very curious about your experiences regarding the PDF generation. Is that post available anywhere or will it ever be?

  3. November 12, 2013 at 10:30 am

    Hi Leo,

    I have actually been swamped lately and haven’t had a chance to write the post. I do intend to though.

    If there is something specific you would like to know or like some help with hit me up on twitter (@butlermaurice) and I am more than happy to let you know what I can.

    Maurice.

  4. July 1, 2014 at 5:25 pm

    Hey, can you please expand on the pdf generation module, I happen to have a similar problem but, I could not find the right tools to convert a doc/ppt file to pdf ( since it might include data other than text like images, and other graphic content )

  5. Zod
    February 18, 2015 at 2:18 am

    Hello Maurice , i want to know how i can add signature digital with NodeJS and Edit PDF ??????

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: