AWS Deployment Guide

Overview

This Quick Start guide provides instructions for deploying a self-hosted CloudQuery Platform in Amazon Web Services (AWS). CloudQuery is an open source high-performance data integration platform that helps you transform your cloud infrastructure into queryable datasets.

This deployment is for customers who want to self-host a complete CloudQuery Cloud Governance solution. During deployment, customers can configure the AWS CloudFormation templates to define the desired cluster configuration and settings.

Architecture

Deploying this Quick Start for a new virtual private cloud (VPC) with default parameters builds the following CloudQuery environment in AWS Cloud using a combination of CloudFormation (for infrastructure deployment) and Helm (for application deployment in the k8s clusters).

Reference Architecture for CloudQuery Platform on AWS

Infrastructure Components

As shown in the diagram, the Quick Start sets up the following:

• A highly available architecture that spans two Availability Zones.

• A VPC configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.

• An internet gateway to allow internet access for load balancers.

In the public subnets:

• Managed NAT gateways to allow outbound internet access for resources in the private subnets.

• Public load balancers for ingress traffic to the platform.

In the private subnets:

• An Amazon EKS cluster for running the CloudQuery platform components

• Worker nodes in an Auto Scaling group

• Private subnets for the database tier

In the database subnets:

• Amazon RDS PostgreSQL instance for platform state storage

• Isolated network tier with no direct internet access

Additional Components:

• AWS Secrets Manager for secure credentials storage

• IAM roles and security groups for least-privilege access

Platform Components

The NGINX ingress controller provides L7 load balancing:

• Terminates TLS at the edge

• Provides URL-based routing

• Integrates with cert-manager for automatic TLS

External Secrets Operator (ESO) provides a vendor-agnostic approach to secrets management in Kubernetes. It allows CloudQuery to:

• Securely fetch secrets from AWS Secrets Manager

• Automatically sync secrets to Kubernetes

• Rotate credentials without pod restarts

• Support multiple secret backends if needed

Cert-Manager automates certificate management within Kubernetes:

• Automatically provisions TLS certificates from Let's Encrypt

• Handles certificate renewal

• Integrates with ingress resources

• Provides vendor-agnostic certificate management

Installation

The installation guide provides two installation options using a CloudFormation template:

• a complete infrastructure deployment (including VPC, EKS, and PostgreSQL), and

• a minimal deployment that works with your existing EKS installation.

Prerequisites

• AWS CLI v2.x

• Kubectl v1.30 or later

• Helm v3.x

• A CloudQuery activation key. This is usually an API key from https://cloud.cloudquery.io/team-api-keys. For more information, see Platform Activation.
• ClickHouse Database and a ClickHouse DSN string (See Deploying ClickHouse for help).

The expected format for the ClickHouse DSN is: clickhouse://<user>:<password>@<host>:9440/<db>?secure

The minimal deployment requires additional prerequisites:

• An existing EKS cluster

• A Postgres Database and a connection string

The expected format for the Postgres DSN is: postgres://postgres:[email protected]:5432/postgres?sslmode=disable

• A VPC

AWS Permissions

The AWS user/role executing the installation needs permissions for:

• CloudFormation full access

• EKS admin access

• IAM create/modify permission

• VPC full access

• RDS full access

• Secrets Manager write access

Option 1: Complete Infrastructure Deployment

This option will install everything you need as part of one CloudFormation stack, including a VPC, EKS, and PostgreSQL.

1. Generate a private key:

openssl genrsa -out platform_private.pem

1. Configure the following environment variables (see prerequisites above for how to obtain these):

export ACTIVATION_KEY=<your-cloudquery-activation-key>
export CLICKHOUSE_DSN=<your-clickhouse-dsn>
export PRIVATE_KEY=$(cat platform_private.pem)

1. Download the CloudFormation template from: https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml (Example: wget https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml)
2. Deploy the CloudFormation stack

    aws cloudformation create-stack \
    --stack-name cq-platform \
    --template-body file://template.yml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters \
        ParameterKey=CreateInfrastructure,ParameterValue=true \
        ParameterKey=ActivationKey,ParameterValue=$ACTIVATION_KEY \
        ParameterKey=ClickHouseDSN,ParameterValue=$CLICKHOUSE_DSN \
        ParameterKey=PrivateKey,ParameterValue=$PRIVATE_KEY
      

3. Wait for stack deployment to complete (this can take around 20 minutes).

    aws cloudformation wait stack-create-complete --stack-name cq-platform

4. Continue to Installing Dependencies.

Option 2: Minimal Deployment

This option assumes you already have a VPC, Postgres database and EKS cluster configured.

1. Generate a private key:

openssl genrsa -out platform_private.pem

1. Configure the following environment variables

export CLUSTER_NAME=<your-cluster-name>
export ACTIVATION_KEY=<your-cloudquery-activation-key>
export POSTGRES_DSN=<your-postgres-dsn>
export CLICKHOUSE_DSN=<your-clickhouse-dsn>
export EKSOIDCProviderURL=<your-eks-oidc-provider-url>
export PRIVATE_KEY=$(cat platform_private.pem)

The EKS OIDC provider URL can be found on the EKS cluster overview page:

Where to find EKS OIDC provider URL on the EKS cluster overview page

1. Download the CloudFormation template from: https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml (Example: wget https://cq-cloud-cloudformation-templates.s3.us-east-1.amazonaws.com/cq-platform/template.yml )
2. Deploy the CloudFormation stack

    aws cloudformation create-stack \
    --stack-name cq-platform \
    --template-body file://template.yml \
    --capabilities CAPABILITY_NAMED_IAM \
    --parameters \
        ParameterKey=CreateInfrastructure,ParameterValue=false \
        ParameterKey=ClusterName,ParameterValue=$CLUSTER_NAME \
        ParameterKey=ActivationKey,ParameterValue=$ACTIVATION_KEY \
        ParameterKey=ClickHouseDSN,ParameterValue=$CLICKHOUSE_DSN \
        ParameterKey=PostgresDSN,ParameterValue=$POSTGRES_DSN \
        ParameterKey=EKSOIDCProviderURL,ParameterValue=$EKSOIDCProviderURL

3. Wait for stack deployment to complete (this can take around 20 minutes).

   aws cloudformation wait stack-create-complete --stack-name cq-platform

4. Continue to Installing Dependencies.

Installing Dependencies

1. Once the stack creation is complete, configure access to the cluster

aws eks update-kubeconfig --name <stack-name>-cloudquery-platform

1. Add and update the required Helm dependencies

helm repo add jetstack https://charts.jetstack.io
helm repo add external-secrets https://charts.external-secrets.io
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo add cloudquery https://cloudquery.github.io/helm-charts/
helm repo update

1. Install the dependant operators

kubectl create namespace platform-system

helm upgrade --install external-secrets external-secrets/external-secrets \
 -n platform-system --wait

helm upgrade --install cert-manager jetstack/cert-manager \
 -n platform-system \
 --set crds.enabled=true --wait

helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx \
 -n platform-system \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-scheme"="internet-facing" \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/aws-load-balancer-attributes"="load_balancing.cross_zone.enabled=true" \
 --wait

Install CloudQuery Platform

1. Create a helm values yaml file (e.g. platform-values.yml) filling in the relevant variables. See table below for explanation.

platform-values.yml

persistence:
  enabled: true
otelCollector:
  database: "<clickhouse_db_name_from_clickhouse_dsn>" # Update this
externalSecrets:
 enabled: true
 region: "<secrets-region>" # Update this
 cloudquerySecretsKey: "<secrets-key>" # Update this
 externalSecretsRoleARN: "<secret-role-ARN>" # Update this
letsEncrypt:
 enabled: true
 email: "<email-address>"  # Update this
ingress:
 enabled: true
 className: "nginx"
 annotations:
   cert-manager.io/cluster-issuer: letsencrypt-production
 hosts:
   - host: "<domain>"  # Update this
     paths:
       - path: /
         pathType: ImplementationSpecific
 tls:
     - secretName: cloudquery-platform-tls
       hosts:
         - "<domain>"  # Update this

Variable	Description
secrets-region	The AWS region where the cluster secrets are stored, which should be the same as the cluster
secrets-key	The full ARN of the secrets key in AWS. This can be found in the CloudFormation stack output variable CloudQueryPlatformSecrets
secret-role-ARN	The full ARN of the AWS role used by the secrets operator to fetch the secrets. This can be found in the CloudFormation stack output variable ExternalSecretsRole
email-address	The email address to use with Lets Encrypt for certificate registration
domain	The domain for which a certificate will be created. This needs to be a domain you own and are able to create DNS records for.

CloudFormation outputs containing the secret and secret role ARNs

1. Install the CloudQuery platform

helm upgrade --install cloudquery cloudquery/platform \
  -n cloudquery \
  --create-namespace \
  --values platform-values.yml \
  --wait

1. Install the CloudQuery operator

helm upgrade scheduler cloudquery/operator \
  -n cloudquery --install --atomic

Set up a domain

1. Retrieve the hostname of the NLB

kubectl get svc -n platform-system ingress-nginx-controller \
 -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

1. Add a DNS CNAME record to your DNS provider pointing the domain you selected in the previous section, which will direct traffic to your ingress URL and allow the certificate verification to complete.

Verification

1. Verify all components are running

kubectl get pods -n cloudquery
kubectl get pods -n platform-system

1. Check ingress status

kubectl get ingress -n cloudquery

1. Check certificates status

kubectl get certificates -n cloudquery

Note: It may take from a few minutes to a few hours for the certificates to be installed. This mostly depends on DNS and its caching mechanism.

Log in

Once installed, log into the platform at https://<your-domain> using a username of admin@localhost and password pass - this will be changed on first login. Follow the setup instructions in the platform UI.

Next steps

See Platform Activation for how to activate your new CloudQuery installation.

See Adding S3-to-SQS notifications for information on how to set up an SQS queue which will receive `s3:ObjectCreated:*` messages when items are added to an existing bucket.