DevOps4UA

Deploying Amazon MSK Serverless Across Multiple Environments with Terraform

Jenkins

Source: Dev.to | Jan 30, 2026 (UTC)

👋 Hey there! This is Pratik, a Senior DevOps Consultant with a strong background in automating and optimizing cloud infrastructure, particularly on AWS. Over the years, I have designed and implemented scalable solutions for enterprises, focusing on infrastructure as code, CI/CD pipelines, cloud security, and resilience. My expertise lies in translating complex cloud requirements into efficient, reliable, and cost-effective architectures.
Through this article, I aim to share practical insights into building and managing AWS MSK Serverless using Terraform, helping fellow engineers and teams design scalable, secure, and resilient streaming architectures on AWS.

⚡Amazon MSK: Overview and Types

AWS MSK is a fully managed service that makes it easy to run Apache Kafka on AWS without managing the infrastructure yourself. Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps.

AWS MSK offers two deployment types:

1. Amazon MSK Provisioned

You manage the cluster capacity by choosing instance types and the number of brokers. It offers more control over performance, scaling, and configuration, making it suitable for predictable workloads and production environments that need fine-tuning.

2. Amazon MSK Serverless

AWS automatically manages capacity, scaling, and broker infrastructure. You don’t need to choose instance types or manage brokers. It is ideal for variable or unpredictable workloads and for teams that want minimal operational overhead.

⚙️Core Components and Their Functionality

  • Broker Nodes: When you create an Amazon MSK cluster, you define the number of broker nodes per Availability Zone (minimum one per AZ). In MSK Provisioned, you can choose between Standard and Express broker types. In MSK Serverless, broker management is handled automatically, and you only configure cluster-level capacity.

  • ZooKeeper Nodes: Amazon MSK automatically provisions Apache ZooKeeper nodes to support reliable cluster coordination.

  • KRaft Controllers: KRaft is Kafka’s modern metadata management mode that replaces ZooKeeper. Metadata is managed internally by Kafka controllers, with no additional setup or cost required.

  • Producers, Consumers, and Topics: You can use standard Kafka operations to create topics and publish or consume data.

  • Cluster Operations: You manage clusters using the AWS Console, AWS CLI, or SDKs to perform actions such as creating, updating, viewing, or deleting clusters.

🎯Learning Objectives and Hands-On Walkthrough

This article demonstrates how to design and implement reusable Terraform modules for core infrastructure components such as the VPC and Amazon MSK, while supporting multiple environments (for example, Dev and UAT) through dedicated environment configurations using terraform.tfvars and variable definition files. It also covers configuring a remote Terraform backend using Amazon S3 to securely store and manage the Terraform state.
The solution provisions a complete Amazon MSK infrastructure, including the VPC, subnets, security groups, IAM roles, the MSK cluster, and a client EC2 instance. With a small set of Terraform commands, you can reliably create, update, and decommission the entire environment in a consistent and repeatable manner across environments.

Resources Covered in This Guide:

  • A VPC with public and private subnets across three Availability Zones.
  • Full networking setup, including Internet Gateway and route tables.
  • A secure Amazon MSK Serverless cluster.
  • An EC2 instance configured with Kafka tools and authentication.
  • IAM roles and security groups to enable secure communication between the EC2 instance and the MSK cluster.

🚀Let’s begin!

📋1. Prerequisites

  • AWS Account: Ensure you have an AWS account with programmatic access and that your AWS credentials are configured locally using the AWS CLI.

aws configure

Enter fullscreen mode

Exit fullscreen mode

Installing Terraform

Terraform is straightforward to set up. The following sections provide installation instructions for the most common operating systems.

On macOS (using Homebrew):

brew tap hashicorp/tap brew install hashicorp/tap/terraform

Enter fullscreen mode

Exit fullscreen mode

On Windows (using Chocolatey):

choco install terraform

Enter fullscreen mode

Exit fullscreen mode

On Linux (Debian/Ubuntu):

wget -O- [https://apt.releases.hashicorp.com/gpg](https://apt.releases.hashicorp.com/gpg) | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] [https://apt.releases.hashicorp.com](https://apt.releases.hashicorp.com) $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list sudo apt update && sudo apt install terraform

Enter fullscreen mode

Exit fullscreen mode

After installation, verify it’s working by running:

terraform --version

Enter fullscreen mode

Exit fullscreen mode

💻2. Deep Dive into the Terraform Code

The following directory structure shows the Terraform modules for provisioning the VPC and Amazon MSK Serverless.

This section establishes the core network infrastructure for our environment. We begin by creating a new VPC, followed by provisioning three public and three private subnets one in each AWS Availability Zone to ensure high availability. The Internet Gateway, NAT Gateway, and route tables are configured to provide secure internet connectivity for the EC2 instance.

Instead of embedding AWS access keys, we use an IAM role to grant permissions securely. This configuration creates an aws_iam_role that the EC2 instance can assume. An attached aws_iam_policy provides the required permissions to connect to the MSK cluster, describe resources, and read from or write to Kafka topics. This approach follows AWS security best practices and is the recommended way to manage service access.

This resource creates an Amazon MSK Serverless cluster with a configurable name. It deploys the cluster within the specified subnets and associates it with the MSK security group for secure network access. IAM-based SASL authentication is enabled to allow secure client access using IAM roles. Resource tags are also applied for easier management and identification.

This instance acts as the Kafka client for our setup. We provision a t2.micro EC2 instance and use a user_data script that runs automatically during the first boot to configure the environment.
The script performs the following tasks:

This ensures the EC2 instance is fully prepared to connect to and interact with the MSK cluster.

☁️3. Provisioning the Infrastructure

Note: The full Terraform scripts are available here⬇️:

This repo contains terraform module code to provision AWS VPC and MSK Serverless cluster

Залишити відповідь

Ваша e-mail адреса не оприлюднюватиметься. Обов’язкові поля позначені *

Читайте також