Unable to decrypt AWS Secret Key

Problem

You encounter single sign-on (SSO) failures with "Unable to decrypt AWS Secret Key"  error messages.

Cause

These errors indicate that Identity Federation for AWS is unable to decrypt the AWS Secret Key from your long term credentials. The most common cause is the absence of the file holding the encryption key used for the AWS Secret key in question.

Background

1. The root cause is a 'referential integrity' violation regarding the encryption scheme for persisting long-term AWS security credentials, where the AWS secret access keys are stored encrypted in the database (DB), and the installation specific private secret key used for the DB encryption is stored in the file system (FS). This ensures that you can loose either one without being compromised right away.

2. The error message will read Failed to retrieve temporary AWS credentials: Unable to read encryption key file going forward, which usually indicates that the encryption key file cannot be found, e.g. due to an incomplete backup/restore process.

3. Our app stores the private secret key in files within the Bamboo home directory, more specifically in BAMBOO_HOME/data/net.utoolity.atlassian.bamboo.identity-federation-for-aws-bamboo/keys/, with file names like 23f497278b00a846f6e219040e08b5027b90c70b.txt (hex encoded hash based on the secret)
   Depending on how your Bamboo instances has been restored/upgraded/migrated in the past, there may be more than one of these files.

4. While our app's DB and the FS storage should be part of the recommended automated Bamboo backup operations, esp. the encryption key files can easily be missed when restoring/migrating Bamboo only partially, or with a different process, e.g. for development scenarios.

Solution

The specific solution depends on what caused the encryption key file(s) to be missing in your scenario, but copying the file(s) from the backup (or if absent there, from the origin, if still available) to the new Bamboo product installation should address the issue.

Workaround

Alternatively, you can always provide new long-term AWS credentials by rotating the access keys. If a encryption key file has been lost, all AWS secret access keys encrypted with the old key cannot be decrypted anymore and will need to be updated so that they can be encrypted with a new key that will be generated automatically:

1. Find and copy the connector ID in the failing task's log output
   (it's a GUID like 888f971a-d720-44c7-92ca-afa745edc752)

2. Click 'Manage AWS Connectors' to navigate to the connectors table

Related articles