Horizontally scalable Socket.io on AWS ELB ALB (with HTTPS)
Setting up a horizontally scalable socket.io stack to work with AWS ELB can be extremely frustrating as there are several moving parts. There’s also a lot of outdated documentation about AWS ELB that doesn’t include the new Application Load Balancer setup. Here’s a simple setup for getting started with using socket.io behind an ELB using HTTPS.
Create a socket.io server
This is a basic socket.io setup with the server listening on port 5000.
Install Socket.io
This step is standalone, but I include it in the automated script below for AWS.
npm install --save-dev socket.io
Create your server script (index.js)
var io = require('socket.io').listen(5000); io.on('connection', function (socket) { socket.on('join', function(e) { console.log(socket.id); }); socket.on('disconnect', function(e) { console.log(socket.id); }); });
I like to run my socket server with Forever JS. It will make sure your socket server stays running even if it hits an exception.
Create a socket.io client
Setup your client side application to communicate with your socket server. We’re going to connect to our ELB using port 3000. Your socket server is listening to port 5000, and that’s okay because we’re going to use nginx to handle the proxy in one of the next steps. This script should be used in your web application.
socket = io.connect('https://yourdomain.com:3000); socket.on('connect', function () { console.log("We're connected!"); });
Setup AWS Launch Configuration
Before you can create an autoscaling group, you have to create a launch configuration. This is simply a way for AWS to know exactly what EC2 resource to spin up on its own. The most important part of this step us the ‘Advanced’ settings where you can enter user data. This can be base64 encoded or plaintext bash scripts. The user data allows you to run all the commands you would normally run manually, automagically. Here’s sample user data you can use to bootstrap your instances quickly. I’m going to break this down in chunks. The entire user data script is here.
Install nginx which will act as our reverse proxy
sudo yum -y install nginx #generate a self-signed cert for nginx ssl handling sudo mkdir /etc/nginx/ssl sudo openssl req -nodes -x509 -days 1024 -newkey rsa:2048 -keyout /etc/nginx/ssl/server.key -out /etc/nginx/ssl/server.crt -subj "/C=US/ST=YOURSTATE/L=YOURCITY/O=YOURORG/OU=Web/CN=YOURWEBSITE"
Install NodeJS and ForeverJS
sudo yum -y update sudo yum install -y gcc-c++ make sudo curl -sL https://rpm.nodesource.com/setup_6.x | sudo -E bash - sudo yum install -y nodejs sudo npm install forever -g --save
Create your nginx configuration
Notice we create an upstream that points to our socket server listening on port 5000. We listen for SSL on port 3000 to handle the handshake.
sudo echo "map \$http_upgrade \$connection_upgrade { default upgrade; '' close; } # Create a socket upstream server for the proxy upstream websocket { server 127.0.0.1:5000; keepalive 8; } add_header Strict-Transport-Security "max-age=15768000" always; server { # Listen for HTTP to force SSL listen 80; # Listen for any traffic from ELB on 3000 listen 3000 ssl default_server ssl; # Change this to whatever domain you've attached your ELB SSL to server_name www.yourserver.com; ssl on; ssl_certificate /etc/nginx/ssl/server.crt; ssl_certificate_key /etc/nginx/ssl/server.key; ssl_session_timeout 5m; # Force SSL if ($ssl_protocol = '') { rewrite ^ https://\$host\$request_uri? permanent; } # This is where the magic happens location ~ ^/(chat|socket\.io)(.*)$ { # Point our proxy to the socket upstream server proxy_pass http://websocket; # Forward headers from ELB proxy_http_version 1.1; proxy_set_header Upgrade \$http_upgrade; proxy_set_header Connection \$connection_upgrade; proxy_redirect off; proxy_buffers 8 32k; proxy_buffer_size 64k; proxy_set_header X-Real-IP \$remote_addr; proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for; proxy_set_header Host \$host; proxy_set_header X-NginX-Proxy true; proxy_set_header X-Forwarded-Proto \$scheme; } # Pass on any 443 traffic (namely for ELB health checks) to a static site root /var/www/html; index index.html index.htm; location ~ ^/(chat|socket\.io) { try_files \$uri \$uri/ /index.html; } }" > /etc/nginx/conf.d/virtual.conf
Add your socket server script
sudo npm install --save-dev socket.io sudo echo "var io = require('socket.io').listen(5000); io.on('connection', function (socket) { socket.on('join', function(e) { console.log(socket.id); }); socket.on('disconnect', function(e) { console.log(socket.id); }); });" > /home/ec2-user/socket.js
Start nginx
sudo service nginx start
Start the socket server
sudo forever start /home/ec2-user/socket.js -O OUTPUT.log -e ERROR.log
For the full script, please see this gist.
Setup AWS Target Group
Create AWS Auto Scaling Group
This is pretty standard. You will need to launch your targets into the desired VPC and subnet. Once you have created the Auto Scaling Group, you will need to assign your Target Group by editing the ASG in the lower pane of the UI. This is a very important step, otherwise you will have no group to launch your instances into.
Create a AWS ELB
SSL Certificate
Next, you will need to choose your certificate since you are listening for SSL connections. I recommend using Amazon Certificate Manager. It’s easy and they take care of the six month renewals automagically.
Target Group
The ELB will want to direct traffic to a target group. Select the group you created earlier from the dropdown.
Security Group
You will need to configure inbound and outbound traffic via security group for the ELB. The outbound traffic goes to your Target Groups. Make sure port 3000 is allowed to pass through to your socket security group. Port 80 and 443 should be allowed to go to your web server group.
Sticky Sessions
Since this setup does not necessitate a cluster of socket servers within the EC2 cluster, you can simply enable sticky sessions on your ELB for users to maintain their session on the EC2 instance.
DNS
I used Route53 to point my domain name to my ELB. They have a great UI that will format the DNS record for you. It should look something like this: ALIAS dualstack.my-awesome-application-xxxxxxxxx.us-east-1.elb.amazonaws.com. (xxxxxxxxxxxxxx)
Once your DNS has propagated, you should be able to point your socket.io client to https://www.yourdomain.com:3000 and get a connection!
Troubleshooting
- Take your time to go through all of these steps as each one is necessary. Double check if you missed any step.
- Watch your client console to see what the connection status is. If there’s an error, it will flood your console with logs
- Use tail -f /home/ec2-user/OUTPUT.log or tail -f /home/ec2-user/ERROR.log to see if your server script is error’ing out. CTRL-C to exit the tail.
- If you’re not getting healthy targets in your group, spot check instances via SSH to make sure you’re getting traffic in.
- A really useful log to look at to see if your bootstrap worked properly is to check /var/log/cloud-init.log. This is where you will see the stdout from your user data script.
- Test your websocket connection using this online tool. It helps to sanity check!
- If you find an error in my tutorial, please contact me so I can fix it for others. 🙂
Leave a Reply