Basic approaches to using multi-core processors with Node.js and load balancing

Node JS services are uniquely positioned to take advantage of multiple CPU cores and multiple servers for easy horizontal scaling. Theres are a few different configurations you may which to consider to take advantage of this and each are suited different situations depending on levels of performance and redundancy required, and also the number of individual services you want to make available on your server. In this post: Service = An actual service your server is providing, E.g. a Rest PI or an instant chat service. Process = An actual Node JS process, there could be more than one used for a single service. Load balancing a single service In this approach you create multiple instances of the same node process and load balance requests to your service between them. Each process instance has it’s own CPU core but shares the rest of the servers hardware. Characteristics of this approach:
  1. Potentially very slightly slower response times under low-load due to load balancer over-head
  2. Faster response times for your service under high load – especially if there are processor intensive elements to your service
  3. Added fault tolerance and redundancy – If one of your processed exits, your load balancer can ensure requests are routed to a process which is still available
Using this approach you have one service which needs to be fast and reliable. Process per service This approach is suitable if you have multiple node services on the same server which should act independently from one-another. An example maybe a configuration in which you have a RESTapi service and a separate asset management and instant chat services residing on the same hardware. Characteristics of this approach:
  1. Your complete product is split into multiple services handled by individual processes
  2. Leads to managing and maintain your server code in separate chunks – easier to manage
  3. Services don’t slow or take each other down
  4. Services can share the same storage if residing on the same hardware
Hybrid A combination of the above approaches, use two or more cores for services which require redundancy, use other cores for additional services. Extending to Multi-server You can achieve even greater scalability by extending your processing onto additional servers, however there are a couple of considerations you need to bare in mind. Firstly remember that a session information associated with your Node process will not cross servers. This means that either your load balancer will need to ensure it routes all requests from one client to the same server all the time, or (the better option) you implement a Memcache or Redis store for ‘session’ information. The other consideration is that again, your database configuration will need to reflect where your database resides in relation to your process. Implementation I’m not going to cover much implementation in this post as I just wanted to note down the core approaches, but just as a primer: A basic implementation of either approaches can be helped along with the handy http_proxy module for node by nodeitsu. You could set-up a single entry route for your services which the http_proxy will then ship off to it’s respective process. A simple software load balancer using http_proxy may look something akin to this: var http_proxy = require('http-proxy') var servers = [ { host: "localhost", port: 8000 }, { host: "localhost", port: 8001 } ] var load_balancer = httpProxy.createServer(function(request, response, proxy) { var node = server.shift() proxy.proxyRequest(request, response, node) servers.push(node) }) The above (un-tested) example takes a list of servers and cycles through it choosing the next process node to handle each request. It’s not checking for process availability first though. I will cover more specifics and implementation information in a latter load-balancing post. For more advanced process management and load balancing checkout PM2 node module by Unitech