July 16th - Migrating live infrastructure from flat Terraform to modules

Let's do a walkthrough of migrating the blog subscription service resources covered in previous posts from flat terraform to leveraging terraform modules!

I highly recommend you open the example code here to reference as you read if you are just starting out.

What are terraform modules?

No point in reinventing the wheel, so here is the definition directly from HashiCorp's website

A module is a collection of resources that Terraform manages together. This page provides an overview of module concepts and phases for adopting modules. For information about creating and distributing modules, refer to Develop modules. Reference

Where do they live

At a high level, modules can be handled in two ways, local or remote.

Local modules - These live in a directory inside the project that utilizes them. This is the approach that I will show in this blog post as there is little need to use remote modules for this simple project. Although I may do that in the future purely to show how it's done to help others learn.

my-project/
└── terraform/
    ├── main.tf
    ├── variables.tf
    └── modules/
        ├── lambda/
        │   ├── main.tf
        │   ├── variables.tf
        │   └── outputs.tf
        └── dynamodb/
            ├── main.tf
            ├── variables.tf
            └── outputs.tf

Remote Modules - These live in a repository separate from the repository for the project that utilizes the modules. These could be private modules that you or your organization have written, public modules created by vendors, or public modules created by a passionate person or group. Another worthy note is that modules can be sourced from repositories specific to the module or a mono-repository that contains multiple modules.

# Separate repository: gitlab.com/your-org/terraform-modules
terraform-modules/
├── lambda/
│   ├── main.tf
│   ├── variables.tf
│   └── outputs.tf
└── dynamodb/
    ├── main.tf
    ├── variables.tf
    └── outputs.tf

# Your project repository
my-project/
└── terraform/
    └── main.tf  # references the remote module repo

The source attribute in a module call is what tells Terraform where to find the module. The value looks different depending on whether the module is local or remote.

# Local module — relative path to the module directory
module "lambda" {
  source = "./modules/lambda"
}

# Remote module — Terraform Registry
module "lambda" {
  source  = "terraform-aws-modules/lambda/aws"
  version = "~> 7.0"
}

# Remote module — GitLab repository
module "lambda" {
  source = "git::https://gitlab.com/your-org/terraform-modules.git//lambda?ref=v1.2.0"
}

Versioning

When creating Terraform modules that will be consumed by other engineers or other projects it is highly advisable to establish a versioning standard for your modules. This will allow you to pin your module calls to a specific version of the module. I will configure this in the future and write a post about it to show how it can be done. In the meantime I highly recommend giving this a read Semantic Versioning 2.0.0.

Why use terraform modules?

Learning/Teaching

I could realistically go without using modules for this project and be completely fine. I remember it being a tricky concept to understand when I was early in my career, so if this helps one person learn, that's a win to me!

Reusability

This is not about going crazy with DRY, let us clear that up from the get go. With that out of the way, if you are building out a standardized path to deploy a set of infrastructure then a module can be extremely valuable so you can have consistent results deploy after deploy.

Defined Standards

I hinted at this above, but let's elaborate further with an example.

I'm part of an organization that wants to provide a single or few known good configurations of resources that development teams can use. I can write a module that locks down aspects of the configuration that should not be allowed to change and provide a way for consumers to customize aspects of the configuration that is okay to change such as URL's, tag values, or component names.

Opinion

Related to the Defined Standards section above, in most situations a module should not be created where every single aspect of the configuration is configurable via the module call. At that point why put the effort into writing and maintaining a module; abstraction for the sake of abstraction annoys me.

The Migration

Design considerations

Before writing a single module, there are a few decisions worth making upfront that will save you from a tedious refactor later.

Single source of truth for the project name. Every resource in this stack shares a name prefix. Rather than repeating the string "readme" in a dozen places, I introduced a local.project in locals.tf and built all resource names, IAM ARNs, and log group ARN strings from it. If the project ever gets renamed, one change covers everything.

File layout with a clear purpose per file. Before the refactor, main.tf was doing too many jobs: provider config, data lookups, locals, and resource definitions all lived there. I split it out so each file has an obvious job: backend.tf owns the Terraform block and provider config, data.tf owns data source lookups, locals.tf owns derived values, and main.tf is nothing but module calls. This is not strictly required but more a personal preference. In professional environments the team(s) should define a standard to follow and stick to it. Another important note is that you can technically make these files named what you like, for example if you have a file that does all of the module calls related to a web service you could put those in a webservices.tf rather than main.tf, the important point is to be consistent with what you use.

Breaking the circular dependency between IAM and Lambda. The IAM policy for each Lambda needs the CloudWatch log group ARN, but the Lambda module is what creates the log group. If you wire those up as resource references you get a cycle that Terraform can't resolve. The fix is to pre-compute the log group ARNs as plain strings in locals.tf using values you already know (var.aws_region, local.account_id, the function name).

Keep the IAM module generic. Rather than writing one IAM module per Lambda with hardcoded policy statements, the module accepts a policy_statements variable as a list of objects and renders them via dynamic blocks. Any Lambda can use the same module; the caller decides what permissions it needs.

Example code: locals.tf

Creating the modules

Six modules cover all the resources in this stack: iam, lambda, dynamodb, ses, api_gateway, and eventbridge. Each follows the same three-file structure: main.tf, variables.tf, outputs.tf.

To make the before/after concrete, here is what a Lambda function looks like as a flat resource block versus a module call.

# Before — flat resource block in main.tf
resource "aws_lambda_function" "subscribe" {
  function_name = "readme-subscribe"
  filename      = ".build/subscribe.zip"
  handler       = "index.handler"
  runtime       = "python3.13"
  role          = aws_iam_role.subscribe.arn

  environment {
    variables = {
      DYNAMODB_TABLE = aws_dynamodb_table.subscribers.name
    }
  }
}

resource "aws_cloudwatch_log_group" "subscribe" {
  name              = "/aws/lambda/readme-subscribe"
  retention_in_days = 14
}

# After — module call in main.tf
module "subscribe" {
  source        = "./modules/lambda"
  function_name = "${local.project}-subscribe"
  source_dir    = "${path.module}/../lambda/subscribe"
  handler       = "index.handler"
  runtime       = "python3.13"
  role_arn      = module.subscribe_iam.role_arn
  environment = {
    DYNAMODB_TABLE = module.dynamodb.table_name
  }
}

The module call is shorter, the log group is handled automatically inside the module, and every other Lambda in the stack follows the exact same pattern.

A few things worth calling out:

The lambda module handles zipping the source directory via archive_file, creating the Lambda function, and creating the CloudWatch log group. The archive output path uses path.root so build artifacts always land in terraform/.build/ regardless of which module is doing the zipping. The Lambda source code itself lives outside the Terraform directory and is referenced via source_dir.

Lambda source code: lambda/subscribe | lambda/poller

The IAM module accepts a policy_statements list and renders them with a dynamic "statement" block inside the policy document. The service principal is configurable but defaults to lambda.amazonaws.com since that is the only consumer so far.

The DynamoDB module has no default for table_name. It is required, passed explicitly from root using local.project. Hardcoding a project-specific default in a module defeats the reusability point entirely.

The api_gateway and eventbridge modules both accept a name_prefix variable that threads local.project into the resource names inside the module. Same pattern, same reason.

Variable defaults - not all variables should have defaults. If it is a variable that the consumer can modify but doesn't have to modify, it's fine to set a default, but if it's a variable that should be set unique to a project, omitting a default value will force the consumer to set it.

Outputs are the module's public interface. Once a resource lives inside a module you can no longer reference it directly from outside the module. If root/main.tf needs the Lambda function ARN, the module must explicitly expose it in outputs.tf. The caller then references it as module.subscribe.function_arn instead of aws_lambda_function.subscribe.arn. Anything not in outputs.tf is private to the module.

# modules/lambda/outputs.tf
output "function_arn" {
  value = aws_lambda_function.this.arn
}

output "invoke_arn" {
  value = aws_lambda_function.this.invoke_arn
}

# root/main.tf — consuming the output
module "api_gateway" {
  source             = "./modules/api_gateway"
  subscribe_invoke_arn = module.subscribe.invoke_arn
}

terraform init is required when adding a new module. Terraform needs to initialize any new module source before it can plan against it. If you are running plans locally, run terraform init after adding a module call. If you have CI/CD configured with terraform init as part of the pipeline, pushing the branch is enough — the pipeline handles it.

Example code: modules/iam | modules/lambda | modules/dynamodb | modules/ses | modules/api_gateway | modules/eventbridge

Updating existing code to use modules

With the modules written, the existing flat files get replaced by module calls in main.tf. The six resource files (lambda.tf, iam.tf, dynamodb.tf, ses.tf, api_gateway.tf, eventbridge.tf) are deleted. main.tf goes from a wall of resource blocks to a clean set of module calls, each one passing the values the module needs.

A few other files get updated along the way:

• outputs.tf references are updated to point at module outputs instead of direct resources
• backend.tf absorbs the required_version, required_providers, and provider "aws" blocks that were previously sitting in main.tf
• Every variable in variables.tf gets a description field - future you will thank you.

At this point the code is correct, but Terraform does not know any of that yet. As far as the state file is concerned, the resources it knows about no longer exist in the config. Running a plan right now would show everything being destroyed and recreated. That is where the next step comes in.

Example code: main.tf | backend.tf | variables.tf | outputs.tf

Migrating the live resources

Terraform tracks every resource by its address in state. Moving resources into modules changes their address, so without some additional guidance Terraform will plan a destroy and recreate of everything.

There are two ways to handle this. The old way is terraform state mv, which is a series of imperative commands you run one at a time, are not version controlled, and leave no trace of what was done. The better way, available since Terraform 1.1, is moved blocks.

A moved block tells Terraform that a resource has relocated. You write them in a moved.tf file, they get reviewed in a merge request/pull request like any other change, and they are applied automatically on the next terraform apply. No manual commands, no undocumented side effects.

moved {
  from = aws_lambda_function.subscribe
  to   = module.subscribe.aws_lambda_function.this
}

One block per resource. After writing a block for every resource that changed address, you push the branch and let the pipeline run a plan.

Example code: moved.tf

Since there is already CI/CD configured to automatically run terraform init terraform validate terraform plan on development branches, that is where test plans should be run. You could run plans locally, but this introduces potential variance between your local development environment and your CI/CD workflows as defined in code. I would rather test with the same mechanism that is going to deploy the infrastructure.

My plan output showed zero destroys. The only changes were three in-place updates to Lambda zip paths, going from ./.build/subscribe.zip to ./.build/readme-subscribe.zip because the module generates the archive path from the function name. Non-destructive so perfectly fine to allow.

After verifying the plan, merge and apply. Once the apply succeeds the moved blocks have done their job. You can remove them, or leave them commented out if you want a record of the migration. I removed them. The git history is the record.

Example code: moved.tf

Closing Thoughts

The refactor itself is not complicated once you become comfortable with using modules, hopefully this post helps with that. The moved blocks make the migration safe and reviewable, and the whole thing lands in a single merge request rather than a series of imperative commands run locally by whoever happened to be around that day.

The bigger payoff is what comes after. Adding a new Lambda to this stack now means calling the module with a few inputs rather than copy-pasting 50 lines of resource config and hoping you caught every place the function name was hardcoded. That compounds fast as the project grows.

In the future I will cover remote modules and versioning, which I mentioned briefly above. That is where modules really shine for teams managing muliple distinct projects.

If there are any parts of this post they need clarification feel free to reachout to me via one of the contact methods listed on my website!