How to Provision AWS Cloud Infrastructure Using Terraform (IaC) — A More Real-World & Practical Approach
In this article I am going to explain how I used Terraform (TF) to setup infrastructure on AWS cloud on a real-world project.
Terraform Code on GITHUB: https://github.com/haarsh85/terraform-with-AWS.git
Scope of this article
Included:
This article will focus on structuring TF in a way to support microservices based application consisting multiple environments. I will not dive deep into the technical details here but will explain more practically how and what details we have to be aware of while using TF as IaC tool.
Excluded:
- Detailed configuration guide / details on AWS resources
- Details on Application code
Background
Firstly, I thought to write this and share my experience with using TF on multi environment development since I didn’t find many resources on how to actually do this. So I’ll try to share everything that I did to help someone else in the same situation.
This application has been created following the microservice approach. 03 teams were developing different aspects (i.e. Frontend, API1 and API2) of the application. Also, due to the nature of such applications different technology stacks were used during the development.
Technologies used
Frontend — Javascript (Framework7)
API1 — JAVA Springboot, PostgreSQL RDS instance
API2 — Node.js, mongoDB
Also, another requirement was to have 03 separate environments to facilitate development. In this case environments are named as dev, staging and prod.
AWS Infrastructure Overview
The following diagram depicts an overview of the AWS resources that provisioned through TF.
Okay, now it’s time to go ahead and provision these resources with TF. Can we do it just yet, I guess no. First there are some key points that we should consider to avoid the pain later on.
Workspaces or Environment-Specific Directories?
Now, since we have to setup infra for multiple environments we need to figure out how we are going to do this first.
There can be so many ways to achieve this and everything depends on your particular use-case and preference. I selected environment-specific directories approach because of the following;
- To keep things simple
- Reduce accidental configuration changes into wrong environment
- Better isolation of resources and variables
- Separate state files and backends for each environment
Great, now we can think on how we are going to setup the directory structure. The basic starting hierarchy looks like follows;
I have a parent directory called “terraform” and created 02 sub-directories as “app” (to store application related config), “init” (to store initial infrastructure related config). Finally, “modules” directory is created to store re-usable app and infra configs. This is to reduce code duplication.
Then created 03 directories to denote the specific environment under “app” and “init” (to store environment specific configs). Further, created directories to reflect the repository names (i.e. frontend, providerservice, etc.) of the microservices to segment the related configs. I haven’t created dirs like frontend, providerservice, etc under environment dirs in “init” because I preferred to keep all infra related to single environment without further division.
How to Manage TF State files
okay, the basic folder structure looks good. Now it’s time to think how we are going to store the state files. Terraform state file contains all the sensitive details regarding your configuration resources and setup. When it comes to state files
You should store your state files remotely, not on your local machine. Also, It is not a good idea to store the state file in source control.
Since, we are using AWS cloud I used secure S3 bucket to remotely store TF state files.
The above configures TF to work with AWS. I have configured S3 bucket named “terraform-state-config-bucket” to store the state. Moreover, I have created respective path to save the respective state files for different application modules as shown in “key” variable. S3 bucket needs to be already created for this.
More details on providers can be found here.
Modules Saves the Day!
Modules are a good way to maintain re-usable code where you only define resource blocks once and refer them using module blocks throughout the code. Below screenshot shows the modules I have used. TF allows to use multiple sub-modules as well.
I’ll just show how I used module approach to create VPC resource.
Inside “vpc” module directory I have main.tf, outputs.tf and variables.tf files.
main.tf — Contains all the resource blocks required and related to VPC
outputs.tf — The values to be printed on the command line. Also, it makes easier to refer and use these values in other modules.
variables.tf — Here the variables that we used in our main.tf file are declared.
Okay, now our VPC module is ready and it’s time to create the resource.
Inside my terraform > init > dev > main.tf, I added the following code.
As per the above code, the “source” is the path to the VPC module that we created earlier. The environment specific variables are passed here which will get applied when the resource is actually created.
Okay, now you might be thinking from where we get the environment specific variables. It’s a good question indeed.
Let’s see what is the answer.
Setting up Environment Specific Values?
Simple answer is using terraform.tfvars file.
The following sample snapshot shows the content of the file.
Once these values are defined here, it can be used inside the module “dev_vpc” like in the previous screenshot.
Before winding up I would like to explain a few additional things.
Using Existing Resources with Terraform?
We might face some situations were it is required to use existing resources — created outside of TF — within Terraform. In my situation, AWS IAM roles were created by Admins since I didn’t have permissions to do so. So, we can use the data feature in TF to integrate and use these resources within TF.
More details can be found here.
Getting the values of resources created through TF?
If we need to get values of the resources which have been created through Terraform we can simply use terraform_remote_state Data Source.
Example:
Creating the data block using “terraform_remote_state”
Referring the required values during resource creation.
I hope this article will give you a better understanding and idea on how to structure Terraform in order to use it without any issues. Also the complete terraform code is available at https://github.com/haarsh85/terraform-with-AWS.git
Thank you for reading.
Have a nice day!
