How to Organize Your Git Repositories
How do you organize your Git repositories to manage multiple microservices across multiple environments while using GitOps to automate continuous deployment? Learn more
GitOps starts with Git. You are probably already using Git to manage the source code of the microservices that make up your applications. Hopefully you also already use Git to manage the Kubernetes manifests that configure and deploy those microservices. Finally there is the additional dimension of environment e.g. development, staging and production. How do you organize your Git repositories to manage multiple microservices across multiple environments while using GitOps to automate continuous deployment? I’ll attempt to provide some guidance in the next few chapters.
Source and Manifests
The decision to have source code and manifests in the same or separate repositories is probably the most contentious. There are arguments for both do and don’t.
Separate
Using separate repositories for source code and deployment configuration can be used in more traditional organizations. Developers write the functional code and are responsible for the source. Operators take the releases from the developers and run them in the different environments and are responsible for the configuration. Operators work with security teams to ensure that the configurations meet compliance requirements. This arrangement can have disadvantages, for example, code may depend on the setting of certain environment variables. A breakdown in communications between development and operations can result in an incorrect setting and will cause runtime problems.
Together
With a DevOps team, they are responsible for both the source code and the deployment configuration. A platform team is separately responsible for running the Kubernetes clusters. With the DevOps team having the responsibility of both writing the code and the deployment configuration, it makes sense to keep both file sets in the same Git repository. However, this makes the security team nervous as they believe that many DevOps engineers do not have the security experience to correctly configure services to run in a secure and robust way. Weave GitOps provides Trusted Delivery via Policy as Code which validates configurations for security, resilience and coding standards at commit, build, deploy and run time. Therefore, significantly reducing the chance of misconfigured services.
Development, Staging and Production
Typically new features and fixes are promoted through environments as they continue to pass tests. Initially, deployment to the development environment will include automated testing and functional testing by the developer. Staging deployments are subject to more rigorous tests usually by a testing team. Finally, the ultimate test, real users in the production environment. Development, staging, and production is just the entry level of multiple environments, many organizations add geographic regions and multiple tenancy which increases the complexity exponentially.
Separate
One option is to use a separate Git repository for each environment, however, that could easily spiral out of control as the permutations of development, staging and production are multiplied by geographic region. Keeping common configuration items in synchronisation across multiple repositories soon becomes a burdensome and error prone task.
Together
Having a common repository for all configuration files whether included with the source code or not is a much better option. However, managing configuration that is common across all environments with the configuration that is unique to each environment is a challenge.
Branches
Using Git’s capability of tracking different changes in separate branches is one option. Managing merges and resolving conflicts once again soon becomes a burdensome and error prone task as complexity increases.
Kustomize
Built into the Kubernetes command line interface and into Weave GitOps, Kustomize has the concept of a base configuration and overlays. The base is where the bulk of the configuration is defined, different environments then specify overlays to that base and specify patches and additional configuration to be included. For example, the base configuration has minimum replicas and no ingress defined, developers will use port forwarding for testing. The production environment patches on that base configuration to add Horizontal Pod Autoscalers and ingress definition.
~/myApp
├── base
│ ├── deployment.yaml
│ ├── kustomization.yaml
│ └── service.yaml
└── overlays
├── development
│ ├── cpu_count.yaml
│ ├── kustomization.yaml
│ └── replica_count.yaml
└── production
├── cpu_count.yaml
├── hpa.yaml
├── ingress.yaml
└── kustomization.yaml
Source and Management
GitOps has its own configuration for its entities that are managed in Git. Are these part of the application and included within the Git repository for the microservice?
Together
A subdirectory within the repository containing the microservice source code and its configuration could be used to configure GitOps. This is an usual use as it only really works if you’re only deploying the one service to a Kubernetes cluster; a “Hello World” example. However, most applications are a collection of microservices.
Separate
A typical application will consist of numerous microservices, each in their own Git repository. Therefore, it makes sense to have a totally separate repository for the GitOps configuration which will reference the individual microservice repositories. Weave GitOps supports both subdirectories within the management repository and Kustomize as well. This allows for different clusters in subdirectories and patching for different clusters and/or environments.
Helm
Just when you were getting your head around Git repositories and their directory structures, what about Helm charts? Helm is a package manager for Kubernetes, it’s an ideal tool for deploying microservice applications because a Helm chart can include multiple components; Helm charts can also include other Helm charts. A Helm chart has various values that can be configured at deploy time, this mechanism can be used for deployment into different environments e.g. development, staging, production, geographic regions.
Together
One approach is to have one Helm chart that deploys all the microservices in your application. The configuration values for each environment can be managed in separate values files. The Helm chart definition and the value files are all under the control of Git in their own repository. Weave GitOps will collect the chart and the values from this repository and deploy it to the cluster. For an application made up of many microservices, using just one repository can create a bit of a bottleneck and the values files can become extremely large, complex and difficult to manage. Version control is also a challenge and tends to lead to a waterfall release strategy.
Separate
Because a Helm can reference or call in other Helm charts, each microservice in the application can have its own Helm chart as part of the source repository with the top level chart in a separate repository. Each microservice DevOps team is able to manage the chart and its values files for their services, maintaining agility.
apiVersion: v2
name: observability
version: 1.0.1
description: Prometheus, Loki, Grafana, dashboards
dependencies:
- name: kube-prometheus-stack
version: 34.0.0
repository: https://prometheus-community.github.io/helm-charts
- name: loki
version: 2.11.0
repository: https://grafana.github.io/helm-charts
- name: grafana
version: 6.26.0
repository: https://grafana.github.io/helm-charts
- name: app-dashboards
version: 1.2.5
repository: https://artifact.acme.net/internal-helm-charts
One Size Fits All
One word of caution before you start building complex Helm charts with many dependencies, Helm does not provide a way to specify the order the dependencies get deployed. Sometimes this will cause problems if, for example, a chart relies on a custom resource definition from another chart. Because of this limitation Weaveworks chose to enhance the Helm with Profiles.
Of course when it comes to organizing your Git repositories or Helm chart hierarchy, there is not one size that fits all. You should go with what is going to work best for your applications. There is nothing wrong with using different organizational strategies for different applications and environments. Hopefully this article has given you some insight into the positives and negatives of the different approaches. For further reading, Flux also offers some guidance on structuring your repositories; it’s well worth a read.
