Horizontally scalable Socket.io on AWS ELB ALB (with HTTPS)

Comment

Setting up a horizontally scalable socket.io stack to work with AWS ELB can be extremely frustrating as there are several moving parts. There’s also a lot of outdated documentation about AWS ELB that doesn’t include the new Application Load Balancer setup. Here’s a simple setup for getting started with using socket.io behind an ELB using HTTPS.

Create a socket.io server

This is a basic socket.io setup with the server listening on port 5000.

Install Socket.io
This step is standalone, but I include it in the automated script below for AWS.

npm install --save-dev socket.io

Create your server script (index.js)

var io = require('socket.io').listen(5000);
io.on('connection', function (socket) {
    
    socket.on('join', function(e) {
      console.log(socket.id);
    });

    socket.on('disconnect', function(e) {
      console.log(socket.id);
    });
});

I like to run my socket server with Forever JS. It will make sure your socket server stays running even if it hits an exception.

Create a socket.io client

Setup your client side application to communicate with your socket server. We’re going to connect to our ELB using port 3000. Your socket server is listening to port 5000, and that’s okay because we’re going to use nginx to handle the proxy in one of the next steps. This script should be used in your web application.

socket = io.connect('https://yourdomain.com:3000);
socket.on('connect', function () {
	console.log("We're connected!");
});

Setup AWS Launch Configuration

Before you can create an autoscaling group, you have to create a launch configuration. This is simply a way for AWS to know exactly what EC2 resource to spin up on its own. The most important part of this step us the ‘Advanced’ settings where you can enter user data. This can be base64 encoded or plaintext bash scripts. The user data allows you to run all the commands you would normally run manually, automagically. Here’s sample user data you can use to bootstrap your instances quickly. I’m going to break this down in chunks. The entire user data script is here.

Install nginx which will act as our reverse proxy

sudo yum -y install nginx
#generate a self-signed cert for nginx ssl handling
sudo mkdir /etc/nginx/ssl
sudo openssl req -nodes -x509 -days 1024 -newkey rsa:2048 -keyout /etc/nginx/ssl/server.key -out /etc/nginx/ssl/server.crt -subj "/C=US/ST=YOURSTATE/L=YOURCITY/O=YOURORG/OU=Web/CN=YOURWEBSITE"

Install NodeJS and ForeverJS

sudo yum -y update
sudo yum install -y gcc-c++ make
sudo curl -sL https://rpm.nodesource.com/setup_6.x | sudo -E bash -
sudo yum install -y nodejs
sudo npm install forever -g --save

Create your nginx configuration
Notice we create an upstream that points to our socket server listening on port 5000. We listen for SSL on port 3000 to handle the handshake.

sudo echo "map \$http_upgrade \$connection_upgrade {
    default upgrade;
    '' close;
}

# Create a socket upstream server for the proxy
upstream websocket {
    server 127.0.0.1:5000;
    keepalive 8;
}

add_header Strict-Transport-Security "max-age=15768000" always;
server {
    # Listen for HTTP to force SSL
    listen 80;
    
    # Listen for any traffic from ELB on 3000
    listen 3000 ssl default_server ssl;

    # Change this to whatever domain you've attached your ELB SSL to
    server_name www.yourserver.com;

    ssl on;
    ssl_certificate /etc/nginx/ssl/server.crt;
    ssl_certificate_key /etc/nginx/ssl/server.key;
    ssl_session_timeout 5m;

    # Force SSL
    if ($ssl_protocol = '') {
      rewrite ^ https://\$host\$request_uri? permanent;
    }
    
    # This is where the magic happens
    location ~ ^/(chat|socket\.io)(.*)$ {
        # Point our proxy to the socket upstream server
        proxy_pass http://websocket; 

        # Forward headers from ELB
        proxy_http_version 1.1;
        proxy_set_header Upgrade \$http_upgrade;
        proxy_set_header Connection \$connection_upgrade;
        proxy_redirect off;
        proxy_buffers 8 32k;
        proxy_buffer_size 64k;
        proxy_set_header X-Real-IP \$remote_addr;
        proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
        proxy_set_header Host \$host;
        proxy_set_header X-NginX-Proxy true;
        proxy_set_header X-Forwarded-Proto \$scheme;
    }

    # Pass on any 443 traffic (namely for ELB health checks) to a static site
    root /var/www/html;
    index index.html index.htm;
    location ~ ^/(chat|socket\.io) {
      try_files \$uri \$uri/ /index.html;
    }
}" > /etc/nginx/conf.d/virtual.conf

Add your socket server script

sudo npm install --save-dev socket.io
sudo echo "var io = require('socket.io').listen(5000);
io.on('connection', function (socket) {
     
    socket.on('join', function(e) {
      console.log(socket.id);
    });
 
    socket.on('disconnect', function(e) {
      console.log(socket.id);
    });
});" > /home/ec2-user/socket.js

Start nginx

sudo service nginx start

Start the socket server

sudo forever start /home/ec2-user/socket.js -O OUTPUT.log -e ERROR.log

For the full script, please see this gist.

Setup AWS Target Group

Create AWS Auto Scaling Group

This is pretty standard. You will need to launch your targets into the desired VPC and subnet. Once you have created the Auto Scaling Group, you will need to assign your Target Group by editing the ASG in the lower pane of the UI. This is a very important step, otherwise you will have no group to launch your instances into.

Create a AWS ELB



SSL Certificate
Next, you will need to choose your certificate since you are listening for SSL connections. I recommend using Amazon Certificate Manager. It’s easy and they take care of the six month renewals automagically.

Target Group
The ELB will want to direct traffic to a target group. Select the group you created earlier from the dropdown.

Security Group
You will need to configure inbound and outbound traffic via security group for the ELB. The outbound traffic goes to your Target Groups. Make sure port 3000 is allowed to pass through to your socket security group. Port 80 and 443 should be allowed to go to your web server group.

Sticky Sessions
Since this setup does not necessitate a cluster of socket servers within the EC2 cluster, you can simply enable sticky sessions on your ELB for users to maintain their session on the EC2 instance.

DNS

I used Route53 to point my domain name to my ELB. They have a great UI that will format the DNS record for you. It should look something like this: ALIAS dualstack.my-awesome-application-xxxxxxxxx.us-east-1.elb.amazonaws.com. (xxxxxxxxxxxxxx)
Once your DNS has propagated, you should be able to point your socket.io client to https://www.yourdomain.com:3000 and get a connection!

Troubleshooting

  • Take your time to go through all of these steps as each one is necessary. Double check if you missed any step.
  • Watch your client console to see what the connection status is. If there’s an error, it will flood your console with logs
  • Use tail -f /home/ec2-user/OUTPUT.log or tail -f /home/ec2-user/ERROR.log to see if your server script is error’ing out. CTRL-C to exit the tail.
  • If you’re not getting healthy targets in your group, spot check instances via SSH to make sure you’re getting traffic in.
  • A really useful log to look at to see if your bootstrap worked properly is to check /var/log/cloud-init.log. This is where you will see the stdout from your user data script.
  • Test your websocket connection using this online tool. It helps to sanity check!
  • If you find an error in my tutorial, please contact me so I can fix it for others. 🙂

Leave a Reply

Your email address will not be published.