Amazon S3 bucket (Beta)

Suggest Edits

Lightup user setup

Lightup needs an IAM user than can access the data you want to include in the datasource. The following procedure helps you create the account and a policy to enable read access, and attach the policy to the new account.

  1. Create a new IAM user, and leave Provide user access to the AWS Management Console unchecked. You'll receive access keys (Access Key ID and Secret Access Key) when you create this user:

    • If you're running Lightup Self-hosted or Hybrid and plan to use IAM to fully manage the datasource connection, you won't need these keys.
    • Otherwise, you'll need to save these keys for when you configure the datasource connector.

  2. At Step 2 - Set permissions, select Attach policies directly, then Create policy.

  1. For Policy editor, select JSON.
  1. Use one of the following templates to create the new policy.
  • Grant access to all buckets and files:
{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Effect": "Allow",
			"Action": [
				"s3:GetBucketLocation",
				"s3:ListAllMyBuckets"
				"s3:ListBucket",
				"s3:GetObject"
			],
			"Resource": "arn:aws:s3:::*"
		}
	]
}
  • Grant access to specific buckets (once for each bucket, replacing {bucket_name} with the actual bucket name):
{
	"Version": "2012-10-17",
	"Statement": [
        // Necessary to list buckets
		{
			"Effect": "Allow",
			"Action": [
				"s3:GetBucketLocation",
				"s3:ListAllMyBuckets"
			],
			"Resource": "arn:aws:s3:::*"
		},
		{
            // Necessary to list and retrieve files from the allowed buckets
			"Effect": "Allow",
			"Action": [
				"s3:ListBucket",
				"s3:GetObject"
			],
			"Resource": [
				"arn:aws:s3:::{bucket-name}",
				"arn:aws:s3:::{bucket-name}/*",
			]
		}
	]
}
  1. Name the policy s3-read-only, then select Create policy.
  2. Back on the Create User page, refresh the policy list.
  3. Select the s3-read-only policy, and then select Next.
  4. Finish the Create User dialog.

Configure connector

There are two options for configuring access via the connector:

  • Managed by IAM Role — If you're running Lightup Self-hosted or Hybrid, Amazon recommends that you use an IAM role to grant permissions to applications running on Amazon EC2 instances, such as S3 datasources. To do so for this datasource, follow Amazon's instructions (preceding link) to grant permissions with an IAM role, and then at the top of the Configure connector section slide the Managed by IAM Role toggle to the right.
  • Access Keys — If you can't or prefer not to use the Managed by IAM Role option, provide the following inputs:
    • Region - Specify the AWS Region where your data is hosted, e.g. "us-west-2".
    • Access Key ID - You'll receive the Access Key ID when you create the new user.
    • Secret Access Key - You'll receive the Secret Access Key when you create the new user.

Note that S3 datasources use virtual tables and not tables.

Advanced/Schema scan frequency

You can adjust how often scans run for a datasource.

  • In section 3 - Advanced, select a value for Schema scan frequency: Hourly, Daily, or Weekly.

Query Governance

S3 datasources support all query governance settings. For steps, see Set query governance settings for a datasource.

Metadata metrics

S3 datasources currently do not support metadata metrics.

Date/time data types

These S3 date/time data types are supported:

  • DATE
  • TIMESTAMP

Object types

These S3 object types are supported:

Partitions

S3 datasources support partitions.

Deep metrics

S3 datasources support all deep metrics except for row by row and SQL metrics. However, the following metric features are not supported:

Updated 13 days ago