Terraform Tips: Scalable and Reliable Infrastructure-as-Code – Part 2

It is critical to plan ahead for infrastructure needs and have a long-term vision for scaling services. Both individual use cases and potential limitations and challenges of Terraform should be considered to ensure scalable and maintainable infrastructure management.

Key questions to ask in this regard include:

• How complex is my service?
• How many resources and providers are needed?
• How often will the infrastructure change (monthly, weekly, daily, on every commit)?
• How can resources be best and sustainably grouped (by environment, region, or project)?

However, caution is warranted, as attempting to future-proof the code too early and "over-engineering" can lead to convoluted and hard-to-understand code. This principle applies to both traditional code development and Infrastructure-as-Code. It is important to keep the Terraform structure simple and comprehensible, while creating a flexible foundation for future changes.

As described in part 1 of the Terraform series, cloud API calls are made for the resources during the "terraform plan" and "terraform apply" phases. Instead of defining the entire Infrastructure-as-Code in a single large "monolithic" state file, it is better to keep complexity low and implement manageable state files. State files should be kept small and compact to facilitate dependency management and speed up deployment. Another advantage of smaller state files is that they make testing and reviewing easier.

Since Terraform configurations evolve over time, it is important to implement appropriate structuring measures early on to allow for flexible changes and to maintain the long-term maintainability and scalability of resources. Below, I present an example of a Terraform structure with a modular approach, where the main branch serves as the "source of truth" for all environments. However, it is important to note that this is only an example, and many other approaches and solutions are possible.

In the example shown, the configuration files use a shared directory in which reusable components are defined as "modules". This directory could also be implemented as a standalone repository, allowing it to be used and reused by different applications and teams. 

Additionally, the example includes an infrastructure directory, which serves for bootstrapping (e.g. for disaster recovery) and for overarching cloud environments that are not environment specific (e.g. logging, secret manager).

The organization of the Terraform code in the example is divided by environment. At first glance, the code may seem somewhat redundant. However, there are many advantages to this setup, as it ensures loose coupling. Resources can be managed independently and configured in a more granular, environment-specific manner. For instance, if a service needs to be deleted while the database instance remains, this can be easily accomplished through separate state files and folder structures. Similarly, if the live environment requires more processing units than the development environment, this can be easily handled. In addition to separation by development environment, it may also be beneficial to group these components by infrastructure layers (such as networking, database, etc.).

Here’s a quick look at the Terraform files used in the example and their contents:

• main.tf – Calls modules, locals, and data sources to create all resources
• backend.tf – Defines the cloud bucket where the remote state is stored
• variables.tf – Contains declarations of the variables used in main.tf
• outputs.tf – Contains outputs of the resources created in main.tf
• versions.tf – Contains version requirements for Terraform and providers

Another tip is to utilize "terraform graph" and visualization tools for current Terraform configurations to quickly identify areas for improvement in the setup.

In summary, Infrastructure-as-Code should be written and maintained with the same rigor as application code, adhering to best practices and standards. A clear directory structure leads to improved quality, maintainability, and collaboration within the development team.

To maintain a long-term manageable setup, standardization, consistency, and reusability should also be prioritized for input and output values. There are many useful tips and best practices available from the Terraform community, and cloud providers often follow certain conventions that point us in the right direction:

• When naming resources, it’s better to use descriptive terms such as “private,” “public,” and “database”, while avoiding redundant information (e.g., do not repeat the resource type in the resource name).
  
  
• Use singular, lowercase, and no spelled-out numbers (“2” instead of “two/second”).
  
  
• Place arguments such as “count” or “for_each” at the top of a resource or data source block, while placing “tags”, “depends_on” and “lifecycle” at the bottom.
  
  
• When using “count” or “for_each,” prefer boolean values instead of other expressions.
  
  
• Use underscores (_) instead of hyphens (-) for resource, data source, variable, and output names.
  
  
• Use hyphens in all instances that are publicly exposed (DNS names, instances).
  
  
• Always describe numerical values (RAM size or disk sizes) with their corresponding units (“ram_size_gb”).

Input variables in Terraform are a powerful tool for making configurations flexible and reusable. However, they can also lead to increased complexity in code, so they should only be used when truly necessary. It’s helpful to ask yourself whether the value of the potential variable really needs to be changeable before implementation. Additionally, consider whether there’s a specific use case, or whether using local values might be a more sensible approach.

• Store variables in a variables.tf file.
  
  
• Separate required variables from optional variables within a file for better code readability.
  
  
• Use the “description” argument for a clear, contextual description of the variable (e.g. Why does this variable exist? What value is expected?).
  
  
• Utilize a validation block to define rules and error messages for rule violations (even allows regex in the “conditions” field to ensure naming conventions/prefixes).
  
  
• Provide a “default” for optional variables.
  
  
• Specify the “type”.
  
  
• Prefer simple types (string, list(...), map(...), any) over specific types (objects).
  
  
• Specific types can be useful if they have uniform elements (a map where all elements are of type “String” can be converted more easily). Use “tomap” instead of “map” to avoid creating an object.
  
  
• Use the type "any" if you want to disable type checking beyond a certain depth or support multiple types.
  
  
• For boolean values, ensure positive names like “enable_external_access” are used.

Using output values can have many benefits. Child modules can use outputs to pass a subset of resources to the parent module. The root module can utilize outputs to display values, for example, via the CLI. To clarify, the root or parent module is the module that calls other modules (child modules) to include their resources in the infrastructure configuration. 

Additionally, outputs can be used by other infrastructure configurations after deployment (in the case of a remote state). To facilitate the use of a module for the user, it is helpful to design the outputs in such a way that clearly indicates which value types and attributes they provide.

• Store outputs in an outputs.tf file.
  
  
• Use the following naming convention: <name><type><attribute> (e.g., for a resource “aws_vpc_endpoint” “test” -> test_vpc_endpoint_id).
  
  
• Use plural in names when the return value is a list.
  
  
• Provide “description” and “type” fields.
  
  
• Avoid the “sensitive” field to enhance security and prevent unintended data leaks.
  
  
• Use “try” instead of “element(concat(...))” for better readability, more robust configuration, and error prevention.

1. Long-term Planning and Vision: Regularly review the infrastructure and adapt it to future requirements. Consider how resources are interconnected and what dependencies exist. Keep the complexity of API calls to a minimum.
   
   
2. Modular Design and Reusability: The infrastructure should be divided into smaller, reusable modules. Compact state files help improve maintainability and scalability.
   
   
3. Directory Structure and Organization: Establishing a consistent and logical structure for Terraform files makes navigation and management easier. Important questions include: How complex is my service? How many resources and providers are needed? Where can good cuts be made (project, region, environment)?
   
   
4. Input and Output Values: Using variables and outputs is beneficial for making configurations flexible and adaptable. The community best practices by Terraform provide valuable support.

In conclusion, it is crucial to consider the specific requirements and constraints of each project. It may also be appropriate to deviate from recommendations and develop custom solutions.