Figuring out how to structure your Terraform deployments is a common issue across the community.
From an administrator’s perspective... Do you set standards? Do you enforce policy? Do you set a defined way of creating repositories? Do you create a list of approved modules? How do you keep up with what the teams are doing?
From an engineer’s perspective... Should I put all resources in a single monolithic deployment? Should I break the deployment up into small components or modules? How do I ensure one deployment can pull data from another deployment? What if variables need to be shared between deployments, do I use a common file?
To be honest, there isn’t a right or wrong way to do this, it’s all situational depending on your organization and use case, and there are a ton of different ways to structure the Terraform deployments. The key is a common framework that can accommodate various workflows depending on your likely ever changing requirements.
To make things easier, we’ve built a flexible hierarchical model that meets the needs of various perspectives and business requirements. There are two scopes in the hierarchy:
If you were to compare this to the AWS landing zone concept, the account would be an organization and the environment would be organizational units.
The hierarchy was designed with two types of personas in mind, the administrators and engineers.
The administrator, who creates and maintains environments, VCS providers, provider credentials, module registries, OPA policies, IAM, and agents would mainly operate at the account scope. All of the objects can be managed from the account scope and assigned/shared with the environments as required (note: the Scalr provider can be used to manage these objects).
The administrator can add as many VCS providers, cloud credentials, or any of the objects from a single location and not have to be concerned about flipping back and forth between contexts.
On top of managing objects from a single location, the administrator also has operational views across all environments in a single dashboard, which is invaluable for large scale deployments as seen in the OPA dashboard below:
In the example, you can view OPA checks across all environments and workspaces from a single dashboard saving an administrator a ton of time going into each individual workspace.
This control plane concept gives administrators peace of mind when enabling engineers to use a self service model.
A Scalr environment is a logical grouping of workspaces that has a common set of policies, provider credentials, and users assigned to it… Commonly we see environments aligned to applications, functions, or sub organizations. A workspace, the child of an environment, is the result of a Terraform deployment, in the simplest terms, a workspace holds the deployments state, variables, and history.
As an engineer, you mainly operate at the environment level where workspaces are created and Terraform runs are executed through their own workflows, whether they are CLI or VCS driven.
Administrators have their controls and standards in place, engineers have freedom to deploy with whatever workflow they choose...we’re getting close to an efficient Terraform structure, but now how do you actually structure your code?
While there are different opinions on structure, we think it is fair to say that everyone agrees that keeping code dry is the easiest way to standardize on modules and scale Terraform. A hierarchy helps with this as the private module registry follows the inheritance model we described earlier. A module can be placed at the account or environment scope. If the module is placed at the account scope, it is automatically inherited by all environments and can be used by any workspace. If a module is placed at the environment scope, only workspaces within that environment can use the module. This model allows engineers to easily view and consume approved modules within their workspace, but how does this actually keep the code dry?
The module registry is the first step because you are using genericized code for your Terraform deployment, and the workspace is the second step to keeping the code dry. A common use case we see is to have an environment per application team and workspaces for dev, stage, and prod with the only thing differing between workplaces being the variable file, thus keeping the code dry.
In this case, the modules and OPA policies remain the same across all environments and are created/assigned from the account scope. The reason for separating out the environments per application is that they do not share the same provider credentials or team membership.
Another use case is to have Scalr environments as dev, stage, and prod with each application having a workspace within them. This type of separation generally happens because the OPA policies or provider credentials might differ between those environments, but the modules are shared across the environments. In this scenario, the variables are differing per workspace again.
The biggest difference from use case A is that you now have multiple teams in each environment and you can use the workspace level RBAC segmentation to ensure the teams don’t accidentally impact another teams workspace.
As we mentioned earlier, there are a ton of different ways that Terraform deployments can be automated, standardized, and structured, but the key is making sure you have modules set up in a way that makes them easy to consume and helps keep the code dry. The hierarchy we implemented helps do exactly that for any size organization due to the inheritance model as well as the administrative control plane.