Part 3: Going to Production, Configuration & Security

Off to production

Before getting into CI/CD pipelines and build steps, the first decision was where the source code would be stored. The project was originally on GitHub, but recurring service instability, paired with RAVN’s direction as a proprietary, commercial application, convinced me to go with AWS CodeCommit instead. Choosing AWS CodeCommit reinforces the core philosophy of this build: sticking natively to AWS and reducing external dependencies.

I had the code living in AWS CodeCommit, but pulling raw source code onto a production VM just to build and run it there is a terrible approach. Doing this meant that releases on my PC and production won't be equivalent, and fetching dependencies for the app each time also means that the build isn't the same as the one tested locally.

This is where container registries come in, specifically Amazon Elastic Container Registry (ECR). But because my local dev rig is an x86 machine and my target EC2 instance was an AWS Graviton (ARM64) instance to save on compute costs, a simple docker build wouldn't cut it. I set up docker buildx to build and push multi-architecture images (linux/amd64 and linux/arm64) straight to ECR. That way, the exact same image tag runs natively on my local setup for debugging or on Graviton in production with zero Docker emulation overhead.

I also had to make a decision about whether I wanted to use CodePipeline or not. For the time being, with the scale of the app, I decided it wasn't worth it even though CodePipeline and CodeBuild are relatively inexpensive. I made this decision because my PC has a Ryzen 5600X, which is a relatively fast processor and with the assistance of Go build caching, build times for the server locally were very quick. I decided to use my PC to push prebuilt ARM64 and x86 container images straight to ECR instead.

With my app containerised and now accessible on AWS, I then got to work setting up the EC2 instance mentioned earlier, using an Ubuntu AMI, and installing Docker CLI onto the machine. I was able to get my app running on the EC2 machine and publicly accessible. But this was the start of a slippery slope that I didn't see coming.

The issues

The most deal-breaking issue I knew I was inevitably going to face was the separation of environments. Having my DEV testing logins enabled in Cognito and also available on prod made no sense. The standard approach was to strictly isolate the DEV and PROD environments, where each had its own dedicated AWS infrastructure and separate environment variables. At the time, I had all the environment variables stored in a .env file; however, this wasn't going to work in the long run, as this file wasn't git-tracked and was vulnerable to data loss.

The answer was to use AWS SSM Parameter Store. With its generous 10,000 free parameters and its support for KMS-encrypted SecureString types, it allowed me to store secrets securely. The standard practice would have been to use AWS Secrets Manager; however, its cost didn't justify its use case for the time being.

Using AWS SSM Parameter Store meant that my environment variables were saved securely and could be used anywhere, provided you have the IAM credentials. Stemming from my past experience with mission-critical systems, I decided to also implement AWS AppConfig and its sidecar too. I did this because protecting all changes was second nature for me, and it meant that reverting changes in production or having group-specific features was easy to implement. This sidecar was lightweight with a small memory footprint and even implemented caching on the feature flags.

Now that I had my DEV and PROD environments set up, I had to now pipe PROD to AWS CloudFront to utilise its caching and delivery capabilities. I hated the idea of leaving the server exposed to the public internet—you can’t attack what you can’t reach. Cutting off direct access shields the origin from direct DDoS attacks and forces all inbound traffic through CloudFront's cache, keeping the system lean by preventing unwanted direct requests. Something to note is that AWS CloudFront doesn't avoid DDOS attacks but rather absorbs them, stopping them before they can reach our origin.

As a final cut-off, I implemented a security group directly on the EC2 instance that only permits ingress from CloudFront IP ranges, dropping any direct internet requests before they ever reach the server

Next up in Part 4: Battling Hidden AWS Costs, IP Automation & Lessons Learned →