An ALBatross Around the Neck

Overcoming Integration Challenges with AWS ALB and WAF to secure NIST 800–53 Compliant Environments

Back in September 2021, AWS announced a new feature for the Network Load Balancer — the ability to add an Application Load Balancer as a direct target of Network Load Balancer. At the surface, this was a big win for customers: NLBs and ALBs have two completely different use cases and feature sets and this allowed an application to leverage both simultaneously. I won’t go thru an exhaustive list comparing them as there are hundreds of articles on the internet that already do that (link to AWS comparison). The primary reason for this new feature is to accommodate for the fact that ALBs do not have static IP addresses — they change randomly; while NLBs have static IPs.

This caused a lot of integration issues with security services — mainly firewalls and Network Address Translation. A common pattern would be:

Sometimes there would be an Application Load Balancer fronting the Firewalls, oddly called an ALB sandwich — with a public ALB and an internal ALB (this would help limit the # of interfaces and IPs required on the firewall since you could forward to different ports on the firewall based on Layer 7 routing and PAT the traffic to its final destination). I never really understood why that architecture pattern was named ALB sandwich though. A turkey sandwich does not have turkey on the outside...

1. The traffic to app1.domain.com would be forwarded to the public interface (sometimes called “Untrusted” interface in firewall parlance) on one of the firewalls based on DNS round-robin.
2. The firewall would inspect the packet based on its rules.
3. The firewall would Destination NAT — i.e. change the destination IP address from the firewall’s Public Interface to the IP address of the internal Application Load Balancer in the Application VPC and then route it accordingly.

However, as we’ve established already — the ALB IPs will change at some point. This will always break NAT and by consequence your application— firewalls are NOT reverse proxies — you cannot NAT to an FQDN, you can only NAT to an IP address.

This left customers with three options:

1. AWS released a solution (I wouldn’t be shocked if AWS was using something like this under the hood of this new ALB target type).

https://aws.amazon.com/blogs/networking-and-content-delivery/using-aws-lambda-to-enable-static-ip-addresses-for-application-load-balancers/

1. Query DNS for IP addresses in use by the ALB. Upload the results (NEW IP LIST) to the S3 bucket.
2. Call the describe-target-health API action to get a list of the IP addresses that are currently registered to the NLB (REGISTERED LIST).
3. Download previous IP address list (OLD LIST). If it is the first invocation of the Lambda function, this IP address list is empty.
4. Publish the NEW LIST to the Lambda function’s CloudWatch Logs log stream. This can be used later to search for IP addresses that were used by the ALB.
5. Update the CloudWatch metric that tracks the number of the internal ALB IP addresses (created on first invocation). This metric shows how many IP addresses changed since the last run. This is useful if you want to track how many IP addresses your load balancer had over time. You can disable it by setting CW_METRIC_FLAG_IP_COUNT to “false”. Here is an example of the CloudWatch metric, showing that the number of IP addresses of the ALB changed from 20 IP addresses to 24 then to 28.
6. Register IP addresses to the NLB that are in NEW LIST but missing from the OLD LIST or REGISTERED LIST.
7. Deregister IP addresses in the OLD LIST that are missing from the NEW LIST.

2. A lot of Firewall Vendors learned of this ALB “feature” and built a similar functionality into their firewall. You would provide the FQDN of your ALB, the firewall would query DNS on a schedule (e.g. every 30 seconds) and then store the IP addresses returned from the DNS query as the actual values to use in the NAT. We’ve implemented this in dozens of AWS environments seamlessly.

3. Use a custom script running a cron to do the above — only used if your firewall did not support “FQDN NAT” or you did not want to introduce an NLB and the Lambda cron job that AWS offered for whatever reason.

Here’s my gripe:

Why do I have to pay full-price for this additional NLB if its sole purpose is to provide a static IP to my ALB? I’m not asking for it to be free — I understand completely that the NLB is using resources and providing functionality to the customer. I, myself, was grateful to AWS to be able to recommend a customer use this new NLB-ALB target integration when implementing a similar architecture outlined in Figure 11 of this link. Our plan was to use a central Ingress ALB with AWS WAF applied and using IP target mode to forward traffic to resources behind an internal NLB in another VPC. (Shameless plug, we also leveraged my cdk-nwfirewall CDK construct I blogged about to streamline AWS Network Firewall as well).

Here’s a copy of Figure 11 so you don’t have to click the link — though in my architecture I combine the Inspection VPC and Central Egress VPC.

But I digress. The cost of a network load balancer can be non-trivial for large workloads — we estimated it would cost our customer an extra $300–500 / per year to have this NLB in front of their ALB just so we could get the static IP. Not a huge prohibitive cost, but it’s never fun to explain to a customer that their cost just went up.

What I would like to see:

I would like the NLB to be offered at a discounted rate if it is ONLY being used to front an ALB — say 50% off. Obviously, if the NLB also has standard EC2 targets I wouldn’t expect to have a discount. However, it just seems punitive to customers to have to pay full price to get around a design limitation that AWS introduced and introduces additional complexity into an environment. Even in the architecture I referenced above, using only AWS Native Services this additional layer would be required if you needed layer 7 routing in your application and wanted to host it behind a central WAF. I get there are other architectures that eliminate the need for the NLB but they are not always an option— for example, if you deploy WAF thru Firewall Manager, to centrally manage Web ACLs applied to all Application Load Balancers in an Organization — that account hosting the Application Load Balancer must be managed by the Organization.

In this architecture, we don’t need to centrally control traffic through a single ALB and forward to static IP targets, but all Application Accounts must be in the AWS Organization.

This wasn’t the case for us — we were essentially providing a Firewall as a Service to our customers — they would still own their application accounts, they just wanted someone to configure and manage the NIST compliant boundary protection.

I will add that this feature does have some overlap with Global Accelerator as they both provide static IPs to ALBs — though Global Accelerator would only work for public IPs. I can reconcile the cost of Global Accelerator due to the routing magic that is happening under the hood. The resources required to support that certainly merit the additional cost.

Warning: Please do not use a public NLB to front a public ALB in lieu of using Global Accelerator. Global accelerator uses edge locations to get your application traffic on to AWS’s backbone network (the 8th Wonder of the World??) more quickly as opposed to having that traffic traverse the entire internet to get to your public IP.

Matthew is Senior Solutions Director at stackArmor, a leading AWS cyber security partner that designs custom solutions for customers looking to meet security requirements of compliance frameworks: FedRAMP, NIST 800 series, PCI-DSS, DoD SRG, HIPAA, FISMA, FIPS 140–2 (and 3) and more. StackArmor offers an AWS vetted solution to accelerate and decrease the cost of your FedRAMP ATO by over 40%.