You can let Lightup connect to an AWS account and access the datasources it contains by setting up AWS PrivateLink.
Before you set up PrivateLink, you'll need a VPC where you can create an endpoint service and a network load balancer to route traffic to it. When you set up PrivateLink, you must explicitly add the IAM Principals that can see it (in this case, Lightup), accept their connections, and handle routing within the source VPC.
For each datasource in your VPC, you need an EC2 network load balancer (NLB) to route inbound connections from Lightup. For help creating your NLB, see What is a Network Load Balancer?.
Within each NLB you set up, you also need to create a VPC endpoint service to enable access to the datasource.
When you create the endpoint service:
- You are the service provider and Lightup is the service consumer. This is because the connection is one-way, with you providing access and Lightup consuming data via your endpoint service.
- The endpoint service is assigned a Service name, such as
com.amazonaws.vpce.us-east-1.vpce-svc-0e123abc123198abc. Lightup will need this Service name— you can find it in the endpoint service's service properties.
- To make connecting easier, also provide a friendly name you want Lightup to use for the endpoint, such as prod-databricks-1. If you are exposing several datasources (i.e. creating several endpoint services), make sure the friendly names will help you tell them apart.
- You'll need to specify Lightup as an IAM Principal, using our AWS ARN identifier:
For details and more options, see Share your services through AWS PrivateLink.
Using the Private DNS option is not currently supported by Lightup.
Lightup needs information about the PrivateLink to create a connection. Email your Lightup SE the Service name and friendly name for each PrivateLink. Lightup will then create a connection.
After Lightup creates a connection, you must accept it before the connection can be used.
You can use the VPC console or the AWS CLI (whichever you prefer) to accept Lightup's connection. For steps, see the AWS page Accept or reject connection requests, under the heading To accept or reject a connection request using the console.
Now that you've set up AWS PrivateLink and accepted Lightup's connection, you can add datasources from your VPC. Each datasource will be reachable from
<friendly-name>.<your-organization>.lightup.ai where <friendly-name> is the friendly name you provided for the VPC endpoint service and <your-organization> is the tla identifying your Lightup cluster. The URL is only accessible from within the Lightup VPC— it is not a public url.
Updated 3 months ago