Using the Amazon S3 Object task in Bamboo

Created by Steffen Opel [Utoolity]

You can use the Amazon S3 Object task to upload, download, delete or copy Amazon Simple Storage Service (Amazon S3) (files); in particular, you can upload your Bamboo job's build artifacts or select local files and directories (optionally via Ant Patterns) - when addressing S3 objects (files), it matches those by key prefix, which allows to operate on multiple objects at once (virtual directories resp. folder hierarchies).

 

Configuration

To configure an Amazon S3 Object task:

1. Navigate to the Tasks configuration tab for the job (this will be the default job if creating a new plan).

2. Click the name of an existing Amazon S3 Object task, or click Add Task and then Amazon S3 Object Task to create a new task.

3. Complete the following settings:

a. Common to all tasks

b. Actions supported by this task:

S3 Features

Most features offered by S3 are available by means of specifying additional Metadata fragments.

 

Upload File(s)

 

Upload File(s)

 

Don't fail if nothing to upload

Check to not fail the build, if there is nothing to upload. Clear to trigger a build failure otherwise.

Artifact

Select the artifact you want to upload.

Source Local Path

Specify the local path (relative to the Bamboo working directory) to the files you want to upload. Use commas to separate files and directories. You can also use:

See How to use Ant regular expressions to rename files on S3 uploads for common renaming patterns, for example:

Use default excludes when selecting files

Check to apply Ant default excludes when selecting files. Uncheck to ignore default excludes.

  •  Due to the way file selection is implemented in Bamboo, this setting also applies when the Use Ant patterns to select filesoption is not checked!

  •  As of Ant 1.8.2 the default excludes also contain common version control meta files/directories (refer to the related upstream issueSCP Task to disable Ant default excludes (BAM-17438) for details):

    Default excludes contains common version control meta files/directories

    1 2 3 4 5 6 7 8 9 10 11 12 13 14 **/.git **/.git/** **/.gitattributes **/.gitignore **/.gitmodules **/.hg **/.hg/** **/.hgignore **/.hgsub **/.hgsubstate **/.hgtags **/.bzr **/.bzr/** **/.bzrignore

Upload as ZIP archive

Check to compress the selected file(s) into a single ZIP archive before uploading. Uncheck to upload separately.

Target Bucket Name

Specify the name of the S3 bucket where you want the files to be uploaded to.

Target Object Key Prefix (Virtual Directory)

(Optional) Specify the S3 object key prefix you want the uploaded files to gain in the target bucket.

Metadata Configuration

(Optional) Specify additional metadata in JSON format. Insert fragments from the inline Examples dialog to get started.

  • Refer to PUT Object and PUT Object - Copy for available metadata options and values covering various use cases, for example:

  • Content type and encoding – Declares file(s) to be compressed css.

  • Encryption and storage class – Activate server-side encryption and reduced redundancy storage.

  • Website redirect location – Redirect request to another location, if the bucket is configured as a website. 

  • Cache control – Sets a one hour maximum age.

  • Access control – Grant public read access.

  • (Copy only) Replace metadata – Replace source metadata with new values.
    Note that this has somewhat tricky implications, refer to PUT Object - Copy for details.

  • Tagging – Use object tagging to categorize storage, refer to Object Tagging for details.

  • Object lock retention and legal hold – Store objects using a "Write Once Read Many" (WORM) model, refer to Object Lock for details.

  • ...

Tags

(Optional) Specify tags to apply to the resulting object(s) in JSON format. Insert fragments from the inline Examples dialog to get started.

Download Object(s)

 

Don't fail if nothing to download

Check to not fail the build, if there is nothing to download. Clear to trigger a build failure otherwise.

Source Bucket Name

Specify the name of the S3 bucket where you want the objects to be downloaded from.

Source Object Key Prefix

Specify the key prefix of the S3 objects you want to download from the source bucket.

Target Local Path

(Optional) Specify the local path (relative to the working directory) where the objects will be downloaded to.

Delete Object(s)

 

Don't fail if nothing to delete

Check to not fail the build, if there is nothing to delete. Clear to trigger a build failure otherwise.

Source Bucket Name

see Download above

Source Object Key Prefix

see Download above

Copy Object(s)

 

Don't fail if nothing to copy

Check to not fail the build, if there is nothing to copy. Clear to trigger a build failure otherwise.

Source Bucket Name

see Download above

Source Object Key Prefix

see Download above

Target Bucket Name

see Upload above

Target Object Key Prefix (Virtual Directory)

see Upload above

Metadata Configuration

see Upload above

Tags

(Optional) Specify tags to apply to the resulting object(s) in JSON format. Insert fragments from the inline Examples dialog to get started.

  •  If you do not specify any tags, the existing tags will be copied – you can prevent the copying of existing tags by providing an empty JSON array.

  •  Refer to Object Tagging for details.

Generate Pre-signed URL

Bucket Name

Specify the name of the S3 bucket the URL should target.

Object Key

Specify the key of the S3 object the URL should target.

Method

Select the HTTP method the URL is supposed to be used with.

Expiration

Specify for how long the URL should be valid (seconds).

 

Variables

Upload File(s)

1 2 3 4 5 Creating resource variables for uploaded object 'prefix/taws-tst-object-4B.txt': ... bamboo.custom.aws.s3.object.first.BucketName: taws-tst-target-us-east-1 ... bamboo.custom.aws.s3.object.first.ETag: 1dafad37f6d9e169248bacb8485fd9cc ... bamboo.custom.aws.s3.object.first.ObjectKey: prefix/taws-tst-object-4B.txt ... bamboo.custom.aws.s3.object.first.VersionId: null

Download Object(s)

N/A

Delete Object(s)

N/A

Copy Object(s)

1 2 3 4 5 Creating resource variables for copied object 'taws-tst-object-4B.a.txt': ... bamboo.custom.aws.s3.object.first.BucketName: taws-tst-target-us-east-1 ... bamboo.custom.aws.s3.object.first.ETag: 1dafad37f6d9e169248bacb8485fd9cc ... bamboo.custom.aws.s3.object.first.ObjectKey: taws-tst-object-4B.a.txt ... bamboo.custom.aws.s3.object.first.VersionId: null

Generate Pre-signed URL

1 2 3 4 5 6 Creating resource variables for pre-signed URL 'https://taws-tst-source-us-east-1.s3.amazonaws.com/taws-tst-object-4B.txt': ... bamboo.custom.aws.s3.object.first.PresignedUrl.password: ****** ... bamboo.custom.aws.s3.object.first.BucketName: taws-tst-source-us-east-1 ... bamboo.custom.aws.s3.object.first.ObjectKey: taws-tst-object-4B.txt ... bamboo.custom.aws.s3.object.first.Method: GET ... bamboo.custom.aws.s3.object.first.Expiration: 20170511T103911Z

How-to Articles

Frequently Asked Questions (FAQ)