Node.js is not a toy. Checkout the new large scale web application I helped build. signforms.com

SignForms

TLDR:

We produced a very complex, solid, secure, fast and scalable server solution using node.js in a very short timeframe with amazing results. We used a number of different modules and server side tools to achieve our goal.

What got me excited about this project was the fact that we were tasked with building it in 2 weeks, and had complete technical control over the project.

After looking into what was going to be required for the project, including security, scalability and complex document processing, we decided to use node.js for the server side and a very powerful client side rendering and binding system named gaffa-js on the client side. My colleague has written a blog post about the front end development if you are interested in reading more about that Deploying a web app in 14 days no html

Node.js as a server

I am a big fan of JavaScript and have used it as my primary programing language for some time now. The reason this is even possible is the introduction of node.js.

Node is a JavaScript runtime which allows server-side applications to be written in JavaScript. It excels at delivering fast, efficient data processing with a minimal CPU and Memory footprint. This makes it perfect for what we were trying to achieve with SignForms.

Modules

I am not a fan of the “web application frameworks” such as express.js being widely used without thinking about what is really required. I find them bloated, and unnecessary in most situations (kinda like jQuery…) and personally I much prefer to use a number of smaller modules to build my own solution.

Beeline – A laughably simplistic router for node.js

I have been using Beeline for a lot of my projects lately. As it says on the box, it is laughably simple router which takes seconds to setup. It supports everything from simple catch all routes to very powerful regex matching.

Cookies and Keygrip

Cookies and Keygrip are 2 modules from Jed Schmidt that again I have been using in many, if not all of my projects of late. Cookies provides a simple wrapper around the getting and setting HTTP(S) cookie headers, and can be used in conjunction with Keygrip to provide additional security by signing the cookie to prevent tampering.

Bcrypt – The only way to store passwords

If you are doing anything with passwords or any other sensitive data, and you are not using bcrypt, please, please, please get educated. Here is a good place to start. How to Safely Store A Password

Cryptr

Cryptr is a simple encrypt and decrypt module for node.js that I have written myself for doing simple obfuscation / aes256 encryption on values that need to be decrypted at a later time

I currently use this for a number of items which are encrypted for obscurity rather than any real security.

Formidable

Formidable is a module for parsing form data, but comes into its own when processing file uploads.

I had not used Formidable before this project but after doing a lot of research and doing a lot of testing, it came out on top as the simplest, fastest and most secure way of handling multiple file uploads.

Database Selection

A lot of time was spent to determine the correct database platform to use for SignForms. In house there was already precedence and knowledge in OrientDB, CouchDB and MongoDB so the decision was around what platform would best suit the type of data we are going to have to store.

While there was a lot of research done and a lot of points considered, a lot of comparisons such as Visual Guide to NoSQL Systems were used as references and eventually we decided on MongoDB. The key decision points for use were:

• Dynamic Queries
• Most likely gain benefit from indexes vs map reduce (although we do have some map reduce requirements)
• Database size will most likely grow large thus distributed partitions maybe be required in the future.
• Performance of large datasets.
• Storing a large amount of volatile data, at least until documents are finalised.

Additionally, Mongoose the node.js wrapper for MongoDB was a breeze to setup and work with making database access simple and fast from our chosen server platform.

Document Processing

Here in lies the challenge. When looking into the requirements for this project the only thing that really scared me was how the hell was going to process and manipulate the pdf files. I spent days working with different tools and approaches to the problem. I hit many a brick wall but also came across a number of insightful discoveries of things I didn’t even know existed.

This was a challenging journey and probably deserves its own post about how these problems were solved. I will aim to have that post up in a few days.

I am very proud of what we have achieved in a short amount of time and owe it all to the speed and power of JavaScript! So go check it out. signforms.com

I hope for those that attended it will help you to realise that it is not so scary and is quite readable.

The slides can be found here.

Again, credit to Jared Wyles (@rioter) whos talk at DDDBrisbane last year is where the idea was blatantly stolen ”borrowed”.

… if it walks like a duck and talks like a duck, it’s a duck, right? So if this duck is not giving you the noise that you want, you’ve got to just punch that duck until it returns what you expect.

Last night at BrisJS I gave an impromptu lightning talk on the concept of Duck Punching.

And no, I am not going around beating up poor little duckies…

I’ll show here the brief example I spoke about last night, as I did a very quick overview without explaining it much, but for some better examples and some more info check out this post from Paul Irish.

Duck punching also know as Monkey Patching. is an approach to extend or modify the runtime code of dynamic languages (such as JavaScript) without altering the original source code.

So for example I am sure everyone is aware of the jQuery function “toDuck()” and the current bug where the duck says the wrong thing…

jQuery.fn.toDuck= function(){
        this.text("The duck says: Moo!")
    };

On our site we want to use this function to show some ducks.

<div>Hello World</div>
<button>Make it a duck!</button>​

$('button').on('click', function() {
    
    $('div').toDuck();
    
});

Result:

 The duck says: Moo! 

OK so we want to change the functionality / fix this bug but we don’t want to go edit the original source as its minified, will make it harder to take upgrades, etc. To achieve this then we can use duck punching to “enhance” this functionality.

(function($) {

    // store original reference to the method
    var orig = $.fn.toDuck;

    $.fn.toDuck = function() {
   
        // Do original function
        orig.apply(this,arguments);
            
        if (this.text() === 'The duck says: Moo!') {
            this.append("  *Punch*  Woof!  *Punch* *Punch*  Quack?... *Gives Ducky bread*");
        }
        
    };
}(jQuery));

And our new result:

 The duck says: Moo! *Punch* Woof! *Punch* *Punch* Quack?... *Gives Ducky bread* 

As you can see the original functionality remains untouched but we get our desired quack.

Should the original method be fixed down the track our condition will fail and the additional functionally will not be run.

This very strange and contrived example can be on jsFiddle here.

Documentation of the JavaScript API sucks… Well it did.

I would assume everyone has as some stage come across the scourge that is w3schools. <- see not linked. Also please don't Google them…

These guys are bad, crap shocking and in many cases just plain old wrong. Unfortunately they take out the top few results in a Google search for most JavaScript methods.

This has been a problem for some time and even blocking them from your search results is not easy.

Fortunately someone has done something about this.

Enter the MDN JavaScript Reference. This has been around for a while and I know a number of informed people go here directly and use it.

Unfortunately it is not high in the search rankings however thus non informed / beginners are not seeing the gold mine of information.

This is something I thought I don't have much control over and just needed to do my best to tell people not to use w3schools.

Last night I attended and spoke at the first BrisJS Meetup for 2012

One of the other speakers Tim Oxely pointed out something which I was not aware of but am very excited about. promotejs.com

This is great! I get a simple, easy way to promote a far superior reference site and also assist in the SEO for MDN to hopefully one day knock w3schools off the top.

It might not be time to spam F5 at arewefirstyet.com yet but the day will come!

So I implore you. Every chance you get. Add a promotejs.com link on your site, page, blog or anywhere you can get away with it…

Last night I spoke at the first BrisJS Meetup for 2012

The title of the presentation was “Confusing & completely unreadable JavaScript performance tricks that you will most probably never use.”

For those interested the slides for the presentation can be found here.

(Please feel free to send me a pull request to fix my atrocious spelling…)

As this is an impress.js presentation a decent browser is required to view the correctly but the fall back should work if you are still forced to use a trojan…

Also as was pointed out by Tim Oxely every chance you get link to these guys… promotejs.com

In this post I will try to explain some of the basic concepts and approaches used in JavaScript development that may not seam as obvious to someone first starting out.

For a quick easy way to follow the examples in this post I recommend jsfiddle.net.

1. Scope

I can say without a doubt, the most commonly misunderstood concept of JavaScript by new developers is Scope. Scope in JavaScript works very different than scope in C#. Unfortunately to explain how JavaScript Scope works I first need to cover a few other items. We will revisit Scope a little later.

2. Types
JavaScript there are only 5 primitive types: boolean, string, number, null and undefined. Anything else we come across is an object. (including functions but we will get to that later…)

The JavaScript boolean and string types behave very similarly to their C# counter parts. There are some minor differences, but I wont go into those here.

The other types however are a little bit more interesting.

The JavaScript number type for instance represents a 64-bit floating-point double. JavaScript has no float, int, double or any other types to represent numbers. The other interesting point is the 2 valid number values NaN and Infinity. The value NaN (stands for Not a Number) occurs when a value is returned as a number type, but the value is not parseable as a number. This is usually when trying to parse a non numeric string value (such as "abc") to a number. The value infinity occurs when a number exceeds the upper limit of the floating point numbers, which is 1.7976931348623157E+10308.

The JavaScript null type is similar to the C# Null however it does have one gotcha. When applying the typeof operator to a null value, it will return "object". This is actually completely wrong, and is a mistake that was made very early in the language’s standardisation.

The JavaScript undefined type is similar to null, however while null means a variable or property has no value, undefined means that the variable or property does not exist. For a variable it can also mean that the variable was declared but never assigned a value. This concept may be strange when first starting to work with JavaScript as usually a compiler would complain if a variable or property used when it didn’t exist. Because JavaScript is a dynamic language and the differences between different browsers, these cases are common and a defensive approach needs to be taken when you are not sure a variable or property is going to exist at runtime. (see the Truthy and Falsey section for how to check for the existence of a variable or property)

var foo;
console.log(typeof foo); // undefined

foo = 123;
console.log(typeof foo); // number

foo = "abc";
console.log(typeof foo); // string

foo = true;
console.log(typeof foo); // boolean

foo = null;
console.log(typeof foo); // object

foo = {};
console.log(typeof foo); // object

As you can see the variable foo was declared without assigning a value. At this point in time the type of foo is undefined. The same variable foo is then reused and assigned multiple values. As each value is assigned, the type of the variable changed to the type of its value.

Update:
For completeness I will also add the function type. While not a primitive type, performing a typeof on a function object will return the sting ‘function’. Function objects are explained further in the Functions are objects section.

foo = function() {};
console.log(typeof foo); // function

3. Truthy and Falsey

While JavaScript has a bool type, due to the magic or coercion we just saw in the types section, these are not the only types that can be evaluated in a true or false manner. Enter Truthy and Falsey.

In JavaScript any object can be automatically coerced into a boolean representation (i.e. true or false). I had a number of goes at simplifying my explanation of this and eventual got it down to one sentence:

Anything that exists and has a value will evaluate as true unless the value is false, null, undefined, 0, NaN or an empty string.

var foo;

if(foo) // exists but is undefined -> evaluate as false
{
    console.log(foo); // will not execute
}

foo = "Hello World";

if(foo) // exists and has a value -> evaluate as true
{
    console.log(foo); // will execute    
}

In this example, If the variable is undefined, the condition evaluates as false. If the variable exists, but has a value of 0, the condition again evaluates as false. It’s only if the variable has been declared and assigned a truthy value that the if block would be entered.

We can now use these concepts to write defensive JavaScript, by ensuring variables or properties exist before using them. This is also a common approach for feature detection.

if (someObject && !someObject.foo){
    someObject.foo = 'foo';
}

console.log(someObject.foo); 

The above code checks if someObject is truthy (not null etc) and if someObject.foo has been defined. If someObject.foo has not been defined it will set it to the string value ‘foo’ before continuing to log it out.

4. Functions are objects

Now this is the tricky one. A function in JavaScript is an object just like any other object. While this concept might not be as huge a deal for someone familiar with Generics and Lambda expressions, it is still a big difference between C# and JavaScript.

You can create anonymous functions, assign a function to a variable, pass that variable to another function, change the function object to do other things, parse it as a string and many more weird and wonderful things.

This should look familiar. Define a function called foo and then call the named function.

function foo(){ 
    console.log('bar');
}

foo();  // bar

But how about this?

var foo = function (){ 
    console.log('bar');
}

foo();  // bar

In the second example, we defined an anonymous function and assigned to to the variable foo.

We now have a variable foo which we can do what we please with.

console.log(foo); // function (){ 
                  //     console.log('bar');
                  // }

foo.randomProperty = "We just added a property to an object";

console.log(foo.randomProperty); // We just added a property to an object

These few things are but a scratch on the surface of what can be done, so you can see a function object is very flexible.

5. Hoisting

When JavaScript is executed, the interpreter moves or “hoists” all variable declarations to the top of their containing function / scope boundary, regardless of where they occur. Take the below code for example.

var foo = 'global foo';

function myFunction() {
    console.log(foo);
}

myFunction();

This code defines a global variable foo and sets its value to ‘global foo’. We then call my function that logs the value of foo. Now lets change this function so that it logs the global variable and then logs a local version of foo;

var foo = 'global foo';

function myFunction() {
    console.log(foo); // undefined
    var foo = 'local foo';
    console.log(foo); // local foo
}

myFunction();

Why does the first console.log return undefined? What has happened is the redefinition of foo has been hoisted to the sounding scope boundary. The code that is actually being executed after hoisting has occurred looks like this.

var foo = 'global foo';

function myFunction() {
    var foo;
    console.log(foo); // undefined
    foo = 'local foo';
    console.log(foo); // local foo
}

myFunction();

As you can see, at runtime the re-declaration of foo has been defined but not assigned, thus the first console.log is correctly logging undefined.

6. Scope Revisited

OK. Time to talk about Scope.

In C#, scope is introduced by braces or are "Block Scoped". A variable declared inside a class, function, loop, condition block, etc are available to all members within the blocks braces. That being the case we should all agree that the below C# code will write 0 1 2 3 4 5

 public void Method1()
 {
  var elements = new int[] { 0, 1, 2, 3, 4, 5 }; // available everywhere in the function

  for (var i = 0; i < elements.Length; i++)
     {
        var element = elements[i]; // available only within this for loop
          Console.Write(element);
     }
 }

In JavaScript however, functions define scope. A variable declared outside of a function is in the global namespace. As in any language, you should try to avoid polluting the global namespace. A variable declared in a function is visible anywhere inside that function, and because of hoisting this is the case regardless of where it is defined. Variables defined within a function are not visible outside the function. This sounds a lot simpler than it is and the best way to explain is with examples.

Here is the JavaScript version of the above code.

function Method1(){
var elements = [0, 1, 2, 3, 4, 5]; // available everywhere in the function
         

for(var i = 0; i < elements.length; i++) {
  var element = elements[i]; // remember Hoisting? will be available everywhere, even though it was originally declared inside the for loop
  console.log(element);
}
}

Again executing the code will produce 0 1 2 3 4 5.

As we covered in the Functions section, a function can be treated like any other object, including being added to an array. We will now take the the array of values and create an array of functions that return the original values.

function Method1(){
var elements = [0, 1, 2, 3, 4, 5],
    functions= [];

for(var i = 0; i < elements.length; i++) {
  var element = elements[i]; // remember Hoisting?
  functions.push(function() {
      console.log(element);
    });
}

functions[3]();
}

The above code again iterates through the elements array, but this time adds an element to the functions array which is an anonymous function to log the corresponding value from the elements array. The last line of the above snippet addresses the object in the 3rd position of the functions array (functions[3]) and then executes it as a function (();).

Still with me?

If you have got your mind around that, now take a guess at what value would be logged by the execution of this function. The C# developer in you will be thinking it should log 3. If that is the case, the C# developer in you would be wrong. Its actually 5…

As discussed before, JavaScript does not have "Block Scope" thus the for loop has not introduced new scope. This means that each time the element variable is accessed, the same memory location is updated, rather than a loop iteration specific variable as one would expect in C#. So when the loop has finished, the variable ‘element’ has the value of 5.

To correct this we need to introduce a function to create scope around our variable reference. For ease of explanation we will first just wrap our for block in an anonymous, auto executing function, (often referred to a closure) and pass the element parameter into this function.

function Method1(){
var elements = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
    i,
    functions= [];

for (i = 0; i < elements.length; i++) {

    (function (element) {                    // start of closure

        functions.push(
            function() {
                console.log(element);
            });
        
    }(elements[i]));                         // end of closure
}

functions[3]();
}

We now log 3 as originally expected.The code is now similar to the C# example as the element variable is declared, assigned used and destroyed within the for loop. This is because Hoisting moves variables to the top of a function, the closure stops it bubbling up to the same location as the other variable declarations.

7. Namespacing Patterns

In C# we rely heavily on namespacing to group together like functionality and avoid collisions with other objects or variables in the global namespace.

As JavaScript has no concept of a class, there is also no concept of namespacing in the form you are used to. We can however use what we have just learned by using objects, functions and closure scope to achieve a similar effect.

I am not going to go into details on the different approaches to namespacing within JavaScript, because I stumbled across this post by Addy Osmani (@addyosmani) that covers it much better and in more detail than I ever could. http://addyosmani.com/blog/essential-js-namespacing/

8. Conclusion

There is a whole lot more to the language than what I have described here but I hope this post will help some people making the transition to JavaScript avoid some of the common misunderstands that can cause frustrations in the early days.

I was working on a remote server, trying to determine why our build system was failing trying to sign our executables.

The command being run was:

signtool.exe sign /SHA1 [thumbprint] /t http://timestamp.verisign.com/scripts/timstamp.dll /d [Executable Name] [Executable Path] 

And the error I was receiving was

SignTimeStamp (-2147467259/0x80004005)

As far as I could tell (-2147467259/0×80004005) appeared to be a very generic COM exception and was not very helpful.

The problem actual turned out to be very simple.

While I did know this, I had forgotten that the server in question had no access to the internet. This meant the call to the time server via the -t switch was failing and producing this lovely, and informative error.

The short term workaround for me was to completely remove the -t switch until this group of machines is given internet access. This allows the executable to be signed, and the build and deploy succeed, while we continue to setup this environment. This is however a temporary environment.

Be aware that by not using a time stamp when you sign your executable means the signature is only valid until the certificate expires and thus is most likely not a valid work around in a production system..

With any luck this will stop someone else wasting their time trying to track down the cause of this lovely error message.

That being said, while I know this problem was my fault for being lazy, I have come across my first batch of strange code generation errors. While simple to solve I assume I am probably not the only person to make this mistake so thought I would share my findings.

As part of learning about LightSwitch, I have been going through and following a number of tutorials on the various aspects.

Currently I am looking at using RIA services as a data source and have been following the Creating a Simple LightSwitch RIA Service (using POCO) tutorial from LightSwitch Help Website.

Rather than download the code sample, I was rewriting the code step by step. However once I added my data source to the LightSwitch application and tried to compile I received these errors in DataServiceImplementation.cs within the ServerGenerated project.

Error	1	) expected
Error	2	Invalid expression term ')'
Error	3	; expected
Error	4	; expected
Error	5	A new expression requires (), [], or {} after type
Error	6	; expected

Of course I thought generation errors probably mean I have done something wrong. I looked over the code I had written and it all roughly matched what was provided in the tutorial. I then downloaded the tutorial source and saw that it compiled and worked correctly.

Puzzled I took a closer look at my code and decided to clean it up as I had been lazy and just dumped everything in the same file while I was working. Upon doing so I noticed that when I created the CustomerRecord model class, I accidentaly created it as a nested class of the domain service class.

i.e.

public class DomainService
{
    public class CustomerRecord
    {           
    }
}

instead of

public class DomainService
{
}

public class CustomerRecord
{
}

After refactoring this class into its own file as I should have done in the first place, and updating the data source the solution copiled correctly.

With any luck this will help someone else that makes the same stupid mistake as me.

One of the sessions that took my fancey was the Introduction of Visual Studio LightSwitch by Andrew Coates (@coatsy).

As a result, over the past week or so I have been playing with LightSwitch and have been very impressed with what I have found.

For those of you who have not seen or heard of it yet, LightSwitch is a self-service development tool that enables non developers to build business applications quickly and easily. While it lends itself well to forms over data type systems, with a SilverLight client it is not limited to such applications.

In my eyes it is a replacement tool for building the equivalent of all of the old archaic MS Access and Excel “programs” out there that have been built by non developers out of necessity for something simple right then and there.

It is also very powerful for prototyping and initial requirements gathering for a bespoke system, as real working software can be shown rather than just wire frames and mock ups.

What I see as the added bonus however is that LightSwitch is also a very extensible, customisable and flexible, opening it up to mid and high level developers as well. This along with the built in 3 tier architecture and simple deployment scenarios, I see this filling a lot of holes, and increase speed of delivery in the SME (Small and medium enterprise) market.

This also means that these “necessity” applications can later be taken over by a professional development team without the usual screams of pain…

If you are getting started I would suggest you run through the Lightswitch How Do I? Videos and download the slides from @coatsy’s Tech Ed sessions.

The other resource I have found very helpful is the LightSwitch Help Website and another large list is available from Glenn Wilson’s (@Mykre) blog post here.

I am still learning this stuff myself but if you have any questions leave me a comment and I will see what I can do.

For those of you not really interested in my explanations of the changes I made and just want the software download links are at the bottom of the post.

I currently have one of these DreamCheeky USB Rocket Launchers setup on my desk at work. At the time I purchased this device, the best .NET open source project for controlling the DreamCheeky rocket launcher was a project called Sharp Launcher. http://code.google.com/p/sharplauncher/

However, once my work machine was upgraded to Windows 7 64 bit, I had not been able to enjoy the device as the latest version of Sharp Launcher (1.3), does not work under a 64 bit operating system.

After some Googling and finding no solution, only a number of people asking the same questions, I decided to jump in and fix it.

Now I want to point out that I have tried to get in contact with the original developer to supply him with the changes and have got no response. I have also looked at contributing to the code.google project but it is not actually setup as a source repository, just has a zip of the code for download.

My alternative is to post both the binaries and the source here and hopefully someone stumbles across it who knows how to contact this guy.

I take no credit for the original code just the few minor tweaks I have made to make it work in a 64 bit environment. I have also not made any attempt to clean up the code, remove any warnings, test it on more than 1 or 2 systems, or anything else as this was purely an effort to get this existing software to work under 64 bit.

After downloading the source code for version 1.3 from here and converting the solution to VS 2010, I found the straight away the code did not compile.

Error number 3 is the issue so I change the code from

public bool UnregisterHandle()
{
    if (this.handle != null)
    {
        return Win32Usb.UnregisterForUsbEvents(this.handle);
    }
}

to

public bool UnregisterHandle()
{
    return this.handle != null && Win32Usb.UnregisterForUsbEvents(this.handle);
}

This allows us to compile and run the software, however as expected my device is not found.

By debugging in I found that the call to SetupDiEnumDeviceInterfaces on line 266 of LibHid/HIDDevice.cs was returning false. Now this next bit took me a hell of a long time to work out. Looking back it should have been pretty obvious but I began my search in completely the wrong direction. Long story short, I worked out this was due the type of the Reserved property of the DeviceInterfaceData struct in LibHid/Win32Usb.cs.
This needs to be changed from int

public int Reserved;

to ulong

public ulong Reserved;

This is due to the fact that the returned SP_DEVICE_INTERFACE_DATA Structure returns the reserved property as a ULONG_PTR.
This of course works in a 32 bit environment but the 64 bit data does not fit into a standard 32 bit int.

Once this change was made I continued to debug the process and came across this gem at line 234 in LibHid/HIDDevice.cs.

oDetail.Size = 5;	// hardcoded to 5! Sorry, but this works and trying more future proof versions by setting the size to 
                           the struct sizeof failed miserably. If you manage to sort it, mail me! Thx

To rectify this we need to declare a dll import and define the IsWow64Process

[DllImport("kernel32.dll", SetLastError = true, CallingConvention = CallingConvention.Winapi)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool IsWow64Process([In] IntPtr hProcess, [Out] out bool lpSystemInfo);

And then change that offending line to

 bool is64BitProcess;
 IsWow64Process(Process.GetCurrentProcess().Handle, out is64BitProcess);
 oDetail.Size = (IntPtr.Size == 8 || (IntPtr.Size == 4 && is64BitProcess)) ? 8 : 5;

I know at least 1 reader of this will say the IsWow64Process method wont exist in earlier version of kernel32 such as under Windows 2000 or XP < SP1 but for what I'm trying to achieve here I am assuming if you are on those older versions you will be using the already released version of Sharp launcher and wont need this Windows 7 x64 version.

So now we can compile and run the application and this time we will see the device has been detected correctly and we now have control of our cube defense system.

I hope this has provided some insight to a few people and also provided an opportunity to enjoy your USB rocket launcher again.

Now to hook this up to the build failure notifications… Muhahaha…

UPDATE: I have uploaded the source to GitHub

Updated source can be found here: SharpLauncher 1.3.0.1 on GitHub

Binaries can be found here: SharpLauncher 1.3.0.1 Binaries