Integrations

One of CloudQuery's key strengths is its ability to bring together data from hundreds of different sources. It does this through a plugin-based architecture, with source, transformer and destination plugins acting as independent components.

Source Integrations

The core responsibilities of a source integration are:

• Define the schema (tables).

• Authenticate with the supported API, SaaS service and/or cloud provider.

• Extracting data from the supported APIs and transform them into the defined schema.

• Send the data for further processing by the rest of the pipeline.

Destinations

The core responsibilities of a destination integration are:

• Authenticate with the destination (such as database, message queue, storage).

• Auto-migrate the schemas defined by the source integrations.

• Save each incoming object in the appropriate table.

Transformers

Transformers can only be used with CloudQuery CLI at the moment

Transformers are optional components that sit in between the source and destination. They allow you to make changes to the data before they are written to the database, such as modifying the table name, obfuscating secrets, or adding additional columns.

Transformers are only supported with CloudQuery CLI syncs at the moment.

Example

For example, consider setting up an AWS integration (Source) to fetch resources from AWS, such as EC2 instances, S3 buckets, network topology, and more. This source integration delivers the data to the next step in the pipeline, which could be a transformer or multiple destinations.

By default, the CloudQuery platform stores data synced by source integrations in ClickHouse as the default destination. You can also sync the data to additional destinations simultaneously, without triggering extra API requests from the source integration.