AWS-CLI – IAM Role Fails – “Unable to locate credentials”

If you run a bootstrap script that uses AWS CLI, you’ve probably run into an intermittent fatal error that says, “Unable to locate credentials”. IAM Roles should just work on the instance, but the credentials are not reliable in my experience on Linux systems. Before I attempt to run any bootstrap script, I use the following code to ensure the IAM credentials are where they should be.

config_exists=$(aws configure list | grep access_key | grep iam-role);
until [ ! -z "$config_exists" ]; do echo 'Amazon is playing games here. Trying to get IAM creds that should already be here ...'; sleep 2; config_exists=$(aws configure list | grep access_key | grep iam-role); done;

Get Latest Project Build from CodeBuild on Single EC2 Instance

If you are bootstrapping a new instance into your cluster, you probably don’t want to kick off an entire deployment pipeline to get the latest build onto just that one instance. I grab the latest build from CodeBuild S3 artifact storage and untar it right into the pertinent path. It’s a much quicker bootstrap and doesn’t take any other instance out of commission for a deployment. Note: This requires the jq package for parsing the response JSON.

LOCATION=$(aws codebuild batch-get-builds --ids $(aws codebuild list-builds-for-project --project-name $PROJECT | jq -r '.ids[0]') | jq -r '.builds[].artifacts.location' | sed 's/arn\:aws\:s3\:\:\:/s3:\/\//g')
tar -xvf $LOCATION

Why does RDS MySQL use so much storage?

We recently migrated 23GB of data from an EC2 MySQL database server to RDS. During the migration, we noticed that the free storage on the 100GB instance was being eaten up quickly. It took some digging to find out that AWS does some configuration trickery here to make more money from unsuspecting customers, in my opinion. They set innodb_file_per_table to true by default.This creates a file per table in the database instead of one global tablespace. There’s really no benefit that I’ve found for performance by doing this. It’s a clever way for users who have file access to migrate tables individually instead of entire databases. The key here is that there is no SSH access for RDS, so, why do we need a file per table? It doesn’t make sense if we can’t even use the feature for its original purpose.

Redirect www to non-www via DNS

Apps rise and fall on single characters. One misplaced character can tank ten thousand lines of code. The same is true with URL redirection. Most DNS providers have a GUI for redirecting URLs. I’ve learned the hard way that the forward slash character is of utmost importance when using these features. For example, if I want to forward all www traffic to (sans slash), all of my traffic for www will end up only on my homepage which is no good for SEO, analytics, user experience, etc…. If I want the DNS provider to forward the entire request, I have to have a trailing slash (

Find all sites on server by IP address

I don’t advocate using the following to hack other sites. I found it interesting that my host was serving pornography from the same box I was using, so I switched to my own cloud server. There are two ways to go about finding other sites on your server. If you don’t know your server IP, you can search


and there are plenty of sites that will expose the server domain.

Bing IP Search

This seems to be pretty reliable and up-to-date. Bing has an IP operator that allows you to specify an IP address when searching.

Example Search:


Hackers will use this operator to find sites running WordPress using the images tab. Once they identify several domains, they can easily ascertain your WP version with the generator meta tag. A hacker can know all of the WordPress installations and versions of those sites without leaving a probing footprint. If there is a known vulnerability with one of your versions, it’s a cinch to attack it. Keep your software up-to-date and your back-doors closed friends. Bing is not your friend with this horrid search operator.

Example Search:

ip: wp-content


Recursive FTP get for command line

If you’re like me, you hate using a FTP GUI for doing simple gets. Sometimes, GUIs get in the way and bog the transfer down. I prefer to take out the middle man and use command line. Here’s a nifty snippet I picked up along the way that lets you recursively download entire directories. The files are stored in their identical structure in your working directory.

wget -r --user %user% --password %password% ftp://%server%/full/path

If you want to stick with pure FTP commands, you can use the following:

#Turn off confirmation for getting each file
ftp>mget *

Arrrrrrr, clean up your whitespace matey

I get thoroughly annoyed at code files littered with EOL and BOL tabs and spaces. Here’s a simple regex I use to garbage collect rogue whitespace. Just use find/replace and set your search to regex within your IDE. Replace with null.

[ \t]+$

Accept only specified arguments in a function or class method

There are times that I’ve needed to limit input into a class method or function while using the extract() function. Problems arise at times from rogue argument keys that aren’t necessarily needed. Here’s a simple way to use the awesomeness of extract(), but dictate its output to the parameters of your allowed vars.

$allowed_vars('foo', 'bar', 'foobar'); 

# Set allowed vars as $ variables

# Extract only allowed vars from user-submitted arguments
# EXTR_IF_EXISTS will only convert array keys of foo, bar, and foobar
extract($args, EXTR_IF_EXISTS);