Integrate Tableflow with the AWS Glue Catalog in Confluent Cloud

Tableflow enables integrating with the AWS Glue Data Catalog as an external catalog, allowing the metadata of Apache Iceberg™ tables materialized by Tableflow to be published to AWS Glue. This makes the Iceberg tables accessible to any Iceberg-compatible query or compute engine that leverages the AWS Glue Data Catalog.

These tables must be consumed as read-only tables.

Diagram showing Tableflow integration with AWS Glue Data Catalog

Integrating with the AWS Glue Data Catalog ensures that external catalogs maintain up-to-date and consistent metadata, while the Tableflow catalog remains the single source of truth.

The AWS Glue Data Catalog integrates with Tableflow at the cluster level, enabling the automatic publication of all Tableflow-enabled topics as tables within Glue. As shown in the following diagram, the database and table names map directly to their corresponding cluster ID and topic name, establishing a clear relationship between these elements.

Diagram showing how Tableflow syncs Kafka clusters with AWS Glue Data Catalog

Configure AWS Glue Catalog integration

The following steps show how to enable AWS Glue Catalog integration at the cluster level.

Important

Topics must be materialized in order for catalog synchronization to complete. Enable Tableflow on a topic before enabling your external catalog provider. Catalog sync remains in the pending state until at least one topic is enabled with Tableflow.

  1. Open Confluent Cloud Console and navigate to the cluster that has the topics you want to sync with tables in Glue.

  2. In the navigation menu, click Tableflow and scroll to the External Catalog Integrations section.

  3. Click Add Integration.

  4. Select AWS Glue as the catalog, provide a name to identify your catalog integration, and click Continue.

  5. Click the link in Go to the Provider integration page, and in the Create new IAM Role in AWS dialog, click Confirm.

    The Configure role in AWS provider integration page opens.

  6. Select New Role and click Continue.

    The Create permission policy in AWS page opens.

  7. In the Select Confluent resource dropdown, select Tableflow Glue Catalog sync, copy the IAM Policy template, and click Continue.

    The Create role in AWS and map to Confluent page opens.

  8. Click the Create Role in AWS link to open the AWS IAM console.

  9. Click Policies under Access Management and click Create Policy.

  10. Using the IAM Policy template you copied previously, update the region and Glue account ID with your own, and create a new AWS IAM policy. Name the new policy, for example, tableflow-policy-glue-access.

  11. Navigate to AWS IAM Roles and click Create Role.

  12. Select Custom trust policy as the Trusted entity type.

  13. Attach the permission policy you created previously, for example, tableflow-policy-glue-access, and save your new IAM role. Name the new role, for example, tableflow-catalog-role.

  14. Copy the role ARN, which resembles arn:aws:iam::<xxx>:role/<your-role-name>.

  15. Return to Cloud Console, and in the Map the role in Confluent section, paste the role ARN you copied previously in the AWS ARN textbox.

  16. Enter a name for the provider integration, for example, tableflow-glue-pi.

  17. Click Continue, and update the trust policy of the AWS IAM role using the trust policy presented in the UI.

  18. Click Finish.

    After you create the provider integration, Confluent Cloud can access your AWS Glue Catalog and publish Iceberg table metadata pointers to it.

  19. In the catalog integration wizard, select the provider integration and proceed to the next step.

  20. Review the configuration and launch the catalog integration.

Configure read-only access

The Iceberg tables materialized by Tableflow should be read-only tables. The following steps show how to set the required permissions to AWS Glue and S3 buckets.

  1. Open the AWS Glue Catalog console.

  2. Find the Iceberg table that was published in the previous steps as an AWS Glue table.

    • The cluster ID maps to the AWS Glue database name.
    • The Kafka topic name maps to the AWS Glue table name.
    Screenshot of AWS Glue Console showing Tableflow integration

  3. You can query these tables from any analytics or compute engine that supports AWS Glue Data Catalog. For more information, see Query Data.

    You must consume Tableflow Iceberg tables as read-only, ensuring that downstream analytics engines have read-only access to them.

Related content

Note

This website includes content developed at the Apache Software Foundation under the terms of the Apache License v2.