The Hunt for ALBeast: A Technical Walkthrough

When Miggo onboards customers, we gain visibility into application behaviors from within. This unique perch allows Miggo Research to discover and address new vulnerabilities impacting thousands of organizations. That’s exactly what happened with ALBeast. This blog details the technical aspects of that discovery, including Miggo’s recommendations for mitigation. For a broader overview, we invite you to check out our initial blog about ALBeast.
Skip to our first encounter with ALBeast

Discovering ALBeast: An Unexpected Security Flaw

We identified a critical configuration-based vulnerability that enables authentication and authorization bypass in applications using the AWS ALB authentication feature, provided they do not comply with the updated AWS documentation issued after Miggo’s disclosure. This vulnerability allows attackers to directly access affected applications, particularly if they are exposed to the internet.

Here are the primary issues:

• Misconfiguration: Applications misconfigured as ALB target groups and accessible directly, bypassing the ALB, can be exploited. Attackers can use a shared public key server for all AWS accounts in the region (e.g., https://public-keys.auth.elb.region.amazonaws.com/key-id) to set an arbitrary key ID (kid). This allows the attacker to supply a public key that the application uses to validate the forged ALB JWT token. Following our report, AWS updated their documentation to refine best practices for configuring Security Group restrictions.
  
  
• Misimplementation: Until recently, the AWS ALB user authentication docs did not include guidance on validating a token’s signer—a crucial field for ensuring that the token was signed by the trusted ALB. Without this validation, applications might trust an attacker-crafted token. The absence of this best practice leaves applications vulnerable to ALBeast attacks. Note that ALB tokens do not contain an aud field, which complicates validation.
  
  
• Issuer Forgery: An attacker can forge an authentic ALB-signed token with arbitrary identities, claims, and issuers (IdP) using its controlled ALB. This means that misconfigured applications that verify the identity issuer are also vulnerable to ALBeast.

We reported these issues to the AWS security team on April 6th, and have since been closely collaborating with AWS throughout the disclosure and remediation process. 
‍

Systems Affected by ALBeast

ALBeast can impact applications hosted in any environment–AWS, other public cloud providers, or on-prem. Since discovering ALBeast, Miggo has identified over 15,000 (out of 371,000*) potentially vulnerable applications using AWS ALB’s authentication feature. 

We’ve done our best to contact each affected organization with our findings and provide support where needed.

Out of multiple implementations and open-source projects we encountered, the vast majority (> 95%) lack the signer validation implementation. A significant share does not restrict access according to the latest recommendations, allowing applications to be accessed from the internet and other footholds in the VPC.

There are two AWS ALB authentication mechanisms, both of which make applications vulnerable:

1. OIDC using IdP
   
2. AWS Cognito

Miggo Recommendations

For AWS Customers

1. Validate the signer of the ALB JWT token is the expected ALB

On May 1st, 2024, AWS updated their ALB user authentication docs with the following:

“To ensure security, you must verify the signature before doing any authorization based on the claims and validate that the signer field in the JWT header contains the expected Application Load Balancer ARN.”

The signer is a field that AWS inserts into the JWT header to mention which instance of ALB signed this token. 

AWS added this piece of code to validate the signer, which is the ALB instance that signs the token:

import jwt
import requests
import base64
import json

# Step 1: Validate the signer
expected_alb_arn = 'arn:aws:elasticloadbalancing:region-code: \
account-id:loadbalancer/app/load-balancer-name/load-balancer-id'

encoded_jwt = headers.dict['x-amzn-oidc-data']
jwt_headers = encoded_jwt.split('.')[0]
decoded_jwt_headers = base64.b64decode(jwt_headers)
decoded_jwt_headers = decoded_jwt_headers.decode("utf-8")
decoded_json = json.loads(decoded_jwt_headers)
received_alb_arn = decoded_json['signer']

assert expected_alb_arn == received_alb_arn, "Invalid Signer"

# Step 2: Get the key id from JWT headers (the kid field)
kid = decoded_json['kid']

# Step 3: Get the public key from regional endpoint
url = 'https://public-keys.auth.elb.' + region + '.amazonaws.com/' + kid
req = requests.get(url)
pub_key = req.text

# Step 4: Get the payload
payload = jwt.decode(encoded_jwt, pub_key, algorithms=['ES256'])

2. Ensure that applications receive traffic exclusively from the trusted ALB

On July 19th, 2024, AWS updated the authentication feature documentation to clarify best practices for configuring Security Groups:

“Also, as a security best practice we recommend you restrict your targets to only receive traffic from your Application Load Balancer. You can achieve this by configuring your targets' Security Group to reference the load balancer's Security Group ID.”

For more details, see the Target Groups reference and the ALB Security Groups reference.

‍

A Note by Miggo Research

Ultimately, AWS does not view issuer forging as a vulnerability. They state that the service operates as intended and that the shared responsibility model applies to this issue, implying that applications should follow the latest documentation by updating their code and Security Groups configurations.

The root problem of the ALBeast vulnerability is addressed through the aforementioned customer-required changes. AWS has opted to track the number of customers in problematic configurations and communicate this information to them, rather than refactor the ALB component itself. As a result, only applications that adhere to the updated documentation on signer validation by the application and the refined Security Groups recommendations are considered safe.

‍

For AWS 

ALB verifies the original issuer and stores it in the encrypted cookie. However, on follow-up requests, ALB uses the new issuer from the ALB configuration without validating it against the original issuer. We recommend that AWS modify ALB to retrieve and validate the original issuer from the encrypted cookie.

Not all ALB applications validate the issuer. Even if they do, one issuer could be shared among multiple organizations. For example, in the Google SSO use case, the issuer is https://accounts.google.com could be used across multiple organizations. Further improvements to strengthen ALB security could include:

1. Adding an aud validation mechanism. AWS ALB could implement an aud mechanism to prevent this issue. This could be made in addition to the docs' recommendations to validate the signer field.
   

2. Notifying users regarding the exposure of ALB-connected application instances (EC2, EKS, etc.) in target groups that receive traffic from sources other than the ALB (Docs were updated on July 19th).
   ‍
   
3. Updating AWS Docs to mention verifying the signer field in the X-Amzn-Oidc-Data, which verifies that requests came from a trusted ALB (Docs were updated on May 1st).

Story Time: Our First Encounter With ALBeast

We first learned of an issue when onboarding a customer using AWS Application Load Balancer (AWS ALB) for user authentication. Interested in learning more about this mechanism, we researched how it works and how Miggo’s ADR (Application Detection and Response) might be able to detect malicious behavior in applications that use it.

This is when we uncovered two widespread customer misconfigurations and one AWS implementation decision within the AWS ALB itself.

‍

Key Definitions

What is an ALB?

An Application Load Balancer (ALB) is a service that automatically distributes incoming application traffic across multiple targets, such as EC2 instances, containers, and IP addresses, within one or more Availability Zones. It operates at Layer 7, allowing routing decisions based on content, such as URL path or host field in HTTP headers, authentication enforcement, or firewall integration. ALBs increase web applications' scalability, reliability, and fault tolerance by load-balancing network traffic across different servers, ensuring no single server bears too much demand.

What are JWT and JWK?

JWT and JWKs facilitate secure authentication and data integrity in web applications.

JWT (JSON Web Tokens) are compact, URL-safe tokens that securely transmit information between parties. They consist of a header, a payload (which contains claims about a user), and a signature to prevent tampering.

JWK (JSON Web Key) is a JSON format representing cryptographic keys. These keys are used to encrypt or verify JWT signatures. JWKs are often grouped in a JWK Set (JWKS), serving as a public key directory for verifying JWT authenticity.

This animation helps illustrate the resulting authentication flow:

To understand the authentication and request-response flow of AWS ALB, please see the diagram below, which we pulled directly from AWS Docs.

The Initial Red Flag

Now, let’s note how the ALB authentication docs previously recommended applications verify an ALB token (steps 9-10 in the figure): 

import jwt
import requests
import base64
import json

# Step 1: Get the key id from JWT headers (the kid field)
encoded_jwt = headers.dict['x-amzn-oidc-data']
jwt_headers = encoded_jwt.split('.')[0]
decoded_jwt_headers = base64.b64decode(jwt_headers)
decoded_jwt_headers = decoded_jwt_headers.decode("utf-8")
decoded_json = json.loads(decoded_jwt_headers)
kid = decoded_json['kid']

# Step 2: Get the public key from regional endpoint
url = 'https://public-keys.auth.elb.' + region + '.amazonaws.com/' + kid
req = requests.get(url)
pub_key = req.text
# Step 3: Get the payload
payload = jwt.decode(encoded_jwt, pub_key, algorithms=['ES256'])

See the original in full here.

We were immediately struck by the beginning of step 2: In a given region, a single JWK-like service provides cryptographic keys for applications operating in that area.

This means that even though these applications may have different requirements, use different systems for verifying identities (known as IdPs), and belong to different customers, they both trust the keys provided by this single service, differentiated only by the kid parameter. Regardless of their differences, they can potentially use the key of another application.
‍

Attacking Applications With Forged Tokens

Assuming an application has not yet followed the latest AWS update and validated the signer, if we wanted to target an application in the us-east-1 region, what would stop us from:

1. Configuring an us-east-1 ALB with a personal IdP
2. Signing a malicious token with the ALB
3. Passing the malicious token to the application inside x-amzn-oidc-data

…Well, nothing!

So, that’s exactly what we did. We configured an ALB, minted a token, and passed it along to a dummy application we created in the same region. 

…but this first attempt was thwarted.

The ALB would have none of it. It detected the x-amzn-oidc-data, discarded it, redirected the user to sign in using the configured IdP, and continued with its regular flow. Our malicious header, though potentially valid, didn’t even reach the application! 

It’s ok, we’re stubborn. Now, on a mission, we understood we needed a way to bypass the ALB. This proved tricky. We tried several attack techniques, most of them variations of request smuggling, but mitigations in place prevented us from smuggling our request through.

We ended up writing two simple Python Flask applications behind ALBs. The first one prints the headers back to the user, which is nicer when playing with requests using BurpSuite:

from flask import Flask, request, jsonify
app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def home():
    return jsonify({
        "Received Headers": dict(request.headers),
        "Body": request.get_data(as_text=True) if request.data else "No Body"
    })

if __name__ == '__main__':
    context = ('/home/ubuntu/test/idp/pyop/example/https.crt', 
'/home/ubuntu/test/idp/pyop/example/https.key')
    app.run(debug=True, host="0.0.0.0", port=443, ssl_context=context)

The second one verifies the JWT header using AWS recommended code and from the documentation and issuer verification that we added as needed:

from flask import Flask, request, jsonify
import jwt
import base64
import json
import requests
region = "eu-central-1"
expected_iss = "https://cognito-idp.eu-central-1.amazonaws.com/eu-central-1_QKBTDh9s5"

def validate_jwt(token):
  jwt_headers = token.split('.')[0]
  decoded_jwt_headers = base64.b64decode(jwt_headers)
  decoded_jwt_headers = decoded_jwt_headers.decode("utf-8")
  decoded_json = json.loads(decoded_jwt_headers)
  kid = decoded_json['kid']
  url = 'https://public-keys.auth.elb.' + region + '.amazonaws.com/' + kid
  req = requests.get(url)
  pub_key = req.text
  payload = jwt.decode(token, pub_key, algorithms=['ES256'])
  if payload['iss'] == expected_iss:
    return payload
  raise Exception("Bad issuer")

app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def home():
    headers = dict(request.headers)
    result = ''
    if 'X-Amzn-Oidc-Data' in headers:
        try:
            result = validate_jwt(headers['X-Amzn-Oidc-Data'])
        except BaseException as ex:
            return str(ex)
        return f"Welcome {result['email']}"
    return "unauthenticated"

if __name__ == '__main__':
    context = ('/home/ubuntu/test/idp/https.crt', '/home/ubuntu/test/idp/https.key')
    app.run(debug=True, host="0.0.0.0", port=443, ssl_context=context)

At this point, we were able to forge a JWT token signed by AWS for any app exposed to the internet that doesn’t verify the issuer (iss) field as part of the identity validation process and the signer field during the JWT validation. We didn’t have any means to pierce an ALB, but we were creative and found ones that were directly exposed instead.

Having gotten this far, we had to ask: Is this capability interesting? Sure, we could forge a token, but would applications verify the JWT’s issuer beyond the user identity supplied in the sub field?

It never hurts to check.

To our surprise, we looked through open-source projects with an ALB integration and quickly found applications that didn’t verify the issuer and were vulnerable to attack! While most of the open-source projects we audited were vulnerable to ALBeast by not verifying the signer, some had failed to include the issuer altogether from the start.

Broadening The Scope Of Our Attack

It turned out that we had discovered a way to forge a token with an arbitrary identity for applications that implement the ALB auth mechanism, even when deployed on another cloud provider! 

This vulnerability could be exploited in the following scenarios:

1. The application could be accessed directly, bypassing the ALB.
2. The application did not verify the signer field as part of the JWT verification (a real issue for almost every application at the time of our research).
3. The application did not verify the issuer IdP (iss) field as part of the user identity validation (though most applications do validate the Issuer IdP).

We should have been satisfied, but we couldn’t help but wonder… How far could we take this?

We continued our research to find out if we could also forge the issuer. We first tried changing the issuer in the OIDC configuration section of the ALB, but the authentication failed with status code 401. The ALB verifies the issuer during the authentication process. Our malicious configuration said the issuer is google.com, but the token is signed with a key that doesn’t exist on google.com. The mismatch caused the ALB to return a 401 status code.

‍

While brainstorming how to bypass their validation, we noted two important connections:

1. The session between the user and the ALB
2. The headers that are derived from the session and the configuration sent from the ALB to the application

This begged the question: What if some parameters in the user’s session (from the first connection) were out of sync with the configuration or what the application receives (in the second connection)?

We found the answer by reviewing AWS’s documentation on the
AWSELBAuthSessionCookie cookie:

“… the load balancer shards a cookie that is greater than 4K in size into multiple cookies. If the total size of the user claims and access token received from the IdP is greater than 11K bytes in size …”

This is a major indicator that what’s inside the AWSELBAuthSessionCookie is an encrypted form of the user claims and the token from the IdP. To prove the cookie is encrypted and the claims are inside, we configured the required scopes to be as minimal as possible:

We then authenticated and got this cookie:

As you can see in the top right corner, it’s 1452 bytes long.
‍
Then, we added a few scopes to get more claims (email, profile, address):

We can also control the length of the claims data by extending properties in our controlled IdP:

This resulted in a much larger cookie, 2,540 bytes long.

Bingo. We discovered at least one place where the ALB stores data - it creates a session cookie and stores the token. At this point, all we needed was to find one more place where the ALB stores our data to see if we could force a desync between them.

We looked at the X-Amzn-Oidc-Data header, which contained the JWT signed by the ALB. It had a one-minute expiration time. 

Hurrah, we’ve found another place where data is stored! 
‍

Putting Our Theories To The Test

What would happen if we authenticated, got a session with a token in AWSELBAuthSessionCookie, changed the ALB configuration, and the token expired? We knew that the ALB would have to mint a new token. Would the ALB take the issuer from the encrypted token? Or…would it take it from the configuration as-is?

Remember, the user could be set to anything. The ALB only determines if the issuer we provided is valid after it communicates with the IdP. In the session, the issuer is validated after the IdP check, whereas in the configuration, it is specified before the validation occurs.
‍
To our surprise, the ALB indeed took the issuer from the configuration! In other words, we would be able to use our own controlled ALB to sign a valid token in the same region as any application we felt like attacking, and we could even control the issuer. This means we could also forge a completely valid JWT token for any target application that could be accessed directly and not through the ALB!

F2F With ALBeast

We had discovered how to forge a token that would be accepted by any target application in a few simple steps:

• Create an ALB in the target region pointing to our IdP.
• Mint a token with desired claims.
  • This token has a valid issuer: our IdP.
  • Its public key is stored in the regional JWKs server, trusted by the target application.
• Reconfigure the ALB to target the target issuer.
• Wait for the token to expire, then refresh it.
• The refreshed token now contains the target issuer.
• Send this token directly to the target application, bypassing the ALB.
• The target application validates the issuer and confirms that all is in order.
• We’ve now provided the application with a forged token!
  

This is HUGE. 

Any ALB that uses this feature exposes the /oauth2/idpresponse endpoint. Combined with a technique to search instances online serving the same TLS certificate, finding potentially vulnerable instances became a turkey shoot.
‍

Responsible Disclosure

In the course of our research, we found and reported vulnerable configurations in open-source applications and organizations with responsible disclosure programs. Publication of these findings will be handled separately and aligned with each policy.

Among the open-source projects, we found two vulnerable AWS repositories:

1. AWS Configuration for ALB Identity library for .NET ‍

2. AWS ALB library for Istio‍

After we reported the vulnerable implementations to AWS, the repositories were archived and claimed to be in a “retirement” process.

‍

Disclosure Timeline

March 21, 2024 - Miggo discovers a customer’s service that uses ALB authentication. An application suspected to be vulnerable to auth bypass ignites the generic research.

April 06, 2024 - Miggo reports the vulnerability to AWS.

April 07, 2024 - AWS starts an investigation.

May 01, 2024 - AWS updates the documentation of the ALB authentication feature to verify the signer field.

June 05, 2024 - AWS confirms they view the case closed.

June 15, 2024 - Miggo reports to AWS about incomplete documentation regarding preventive actions that should be defined by Security Groups.
July 11, 2024 - AWS confirms the issue is affecting customers. They are actively monitoring and reaching out to those who are impacted.
July 19, 2024 - AWS updates the documentation of the ALB authentication feature regarding Security Group best practices.
August 20, 2024 - Public disclosure.
August 20, 2024 - Post public disclosure. Since Miggo publicly disclosed ALBeast on August 20, AWS has asserted that it is incorrect to call ALBeast an authentication and authorization bypass of ALB or any other AWS service because the technique relies on a bad actor having access to a misconfigured customer application that does not authenticate requests. 

We agree! That’s why we call it a configuration-based vulnerability. The problem remains that even with the suggested configuration changes that AWS added to the documentation, customers still need to change their code implementation to be protected. 

This exemplifies the cracks in the shared responsibility model, which is the “lightning in the cloud” that no one wants to talk about. As part of the shared responsibility model, CSPs must proactively inform their customers of these issues and the required modifications in the application implementation but ultimately rely on customers to act on them. If possible, CSPs should amend their product to minimize the required customer modifications. Just updating the documentation is not enough. 

Simply put, if an application uses the ALB authentication feature and does not follow the two new best practices added to the documentation, then it remains vulnerable.

We arrived at the 15,000 instance conservative estimate by scanning all IPv4 addresses in the AWS public range that responded with ALB headers and with an indication that the authentication feature of the ALB was enabled. This scan revealed 15,000 potentially vulnerable unique IP addresses; we assume there are more.

“Without Miggo Security, I can’t help but wonder how long ALBeast would have remained undetected. Given the source of this vulnerability and the fact that ALBeast impacted thousands of customers is an industry wake-up call while further highlighting the danger of supply chain vulnerabilities.

We need to be vigilant in understanding and applying good security practices ourselves, but also be realistic that companies like large cloud providers can have flaws that have devastating effects on their customers.  Adopting security by design, implementing guardrails against risky configurations, and continuously learning and adapting remain key tenets of defense strategy. 

In a sea of application security specialists, Miggo has proven capable of supporting these principles while helping to uncover many oversights.” - Han Chae, Head of Security at HyperScience

We would like to express our appreciation to AWS for their prompt attention to this issue and their swift updates to the documentation following our disclosure. The responsiveness of their security team has been crucial in addressing these vulnerabilities and safeguarding their customers who are using the AWS ALB authentication feature.

Want to learn more? Find more materials about ALBeast, the shared responsibility model and how Miggo can help:

1. ALBeast Security Advisory by Miggo Research
2. How Miggo Can Help

Let's keep in touch!

Feel free to contact on Twitter: Liad Eliyahu (@liadeliyahu)