Skip to main content

s3 (Sink)

S3 sink publishes events as Amazon AWS S3 buckets.

Syntax

CREATE SINK <NAME> WITH (type="s3", map.type="<STRING>", credential.provider.class="<STRING>", aws.access.key="<STRING>", aws.secret.key="<STRING>", bucket.name="<STRING>", aws.region="<STRING>", versioning.enabled="<BOOL>", object.path="<STRING>", storage.class="<STRING>", content.type="<STRING>", bucket.acl="<STRING>", node.id="<STRING>")

Query Parameters

NameDescriptionDefault ValuePossible Data TypesOptionalDynamic
credential.provider.classAWS credential provider class to be used. If blank along with the username and the password, default credential provider will be used.EMPTY_STRINGSTRINGYesNo
aws.access.keyAWS access key. This cannot be used along with the credential.provider.classEMPTY_STRINGSTRINGYesNo
aws.secret.keyAWS secret key. This cannot be used along with the credential.provider.classEMPTY_STRINGSTRINGYesNo
bucket.nameName of the S3 bucketSTRINGNoNo
aws.regionThe region to be used to create the bucketEMPTY_STRINGSTRINGYesNo
versioning.enabledFlag to enable versioning support in the bucketfalseBOOLYesNo
object.pathPath for each S3 objectSTRINGNoYes
storage.classAWS storage classstandardSTRINGYesNo
content.typeContent type of the eventapplication/octet-streamSTRINGYesYes
bucket.aclAccess control list for the bucketEMPTY_STRINGSTRINGYesNo
node.idThe node ID of the current publisher. This needs to be unique for each publisher instance as it may cause object overwrites while uploading the objects to same S3 bucket from different publishers.EMPTY_STRINGSTRINGYesNo

Example 1

CREATE SINK UserStream WITH (type='s3', bucket.name='user-stream-bucket',object.path='bar/users', credential.provider='software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider', flush.size='3', map.type='json', enclosing.element='$.user', map.payload=""""{"name": "{{name}}", "age": {{age}}}"""") (name string, age int);  

This creates a S3 bucket named user-stream-bucket. Then this will collect 3 events together and create a JSON object and save that in S3.