I'm seeking advice on the best approach to orchestrate multiple Terraform stacks actually store as one big stack on a GitLab repository.
We're facing currently a few problems :
- Execution Speed : The execution of terraform plan/apply is notably slow due to the sheer number of resources contained within the stack.
- Team Collaboration : Another significant challenge is the handling of this stack by the team as only one person at a time can modify the stack. (Current workflow is pulling the repository, making modifications, applying the plan/deploying locally, pushing the updated stack and inventory)
The objective is to efficiently manage separate stacks for different tenants within the organization, with a shared stack containing mutual resources to which the tenant-specific stacks will attach.
Here is a view we have of the repo structure:
terraform-projects :
Despite the isolated nature of the tenants (with dependencies on the "mutu" branch), they are all intended for the same infrastructure. While each tenant operates within its own scope, they ultimately target the same underlying equipment.
We're looking for the most effective way to manage these different environments. Initially, we considered having a dedicated Terraform inventory (tfstate) for each tenant. However, due to the dependencies with shared resources (contained in the "mutu" stack), relying solely on data sources for each resource might pose scalability issues, especially if there are many dependencies between tenant resources and mutual resources.
Regarding the deployment management, we initially thought of using GitLab CI pipelines with Downstream Pipelines generated by "trigger" jobs using the only:changes function. This setup triggers a plan/apply pipeline whenever a commit occurs on the main branch, targeting specific tenant or shared stack modifications. It utilizes the tenant/mutu folder to feed the TF_ROOT and TF_STATE_NAME variables.
While this setup generally works, we encounter occasional issues such as version mismatch errors ("Saved plan does not match the given state") and challenges with saving the inventory after deployment. We're unsure if this issue provide from GitLab itself (currently version 13.12.10ee) or from other factors.
As we're at an impasse, we've started exploring other solutions such as Terraform Cloud. However, aside from the need to expose the stack and store backend externally (which is not preferable due to company restrictions), We dont see how this would address the dependency between stacks.
We've also looked into Terragrunt, but most examples we found seem more suited for similar environments (e.g., stage/dev/prod) rather than completely distinct branches. Additionally, many examples involve using modules, which we haven't implemented yet.
We're open to alternative solutions, preferably open-source and self-hostable, considering the constraints imposed by my company's access restrictions to external solutions.
Any insights or suggestions would be greatly appreciated.
Thank you!
What we tried : Gitlab repo with CI/CD and Downstream pipeline
Looking for : solution to manage multiple different stacks while mainting dependencies between them.