Magic Modules #
Magic Modules is a code generator and CI system that’s used to develop the Terraform providers
for Google Cloud, google (or TPG) and
google-beta (or TPGB).
Magic Modules allows contributors to make changes against a single codebase and develop both
provider versions simultaneously. After sending a pull request against this repository, the
modular-magician robot user will manage (most of) the heavy lifting from generating a
complete output, running presubmit tests, and updating the providers following your
change.
How Magic Modules works #
Magic Modules can be thought of as a source of truth for how to map a Google Cloud API resource
representation to a Terraform resource (or datasource) representation. Magic Modules uses that mapping
(and additional handwritten code where necessary) to generate “downstream” repositories - in particular,
the Terraform providers for Google Cloud: google
(or TPG) and google-beta (or TPGB).
Generation of the downstream repositories happens for every new commit in a PR (to a temporary branch owned by
the modular-magician robot user) and on every merge into the main branch
(to the main branch of downstreams). Generation for PR commits allows contributors to manually examine the changes,
as well as allowing automatic running of unit tests, acceptance tests, and automated checks such as breaking change
detection.
Resource types #
There are three types of resources supported by Magic Modules:
- MMv1
- Handwritten
- DCL/tpgtools
The following sections describe these tools in detail.
MMv1 #
MMv1 consists of a set of “products”; each product contains one or more “resources”.
Each product has a folder in
magic-modules/mmv1/products.
The name of the folder is the “product name”, which usually corresponds to the API subdomain covered by the
product (such as compute.googleapis.com). Each product folder contains a product configuration file
(product.yaml) and one or more resource configuration files (ResourceName.yaml). The actual name of a
ResourceName.yaml file usually matches the name of a GCP API resource in the product’s subdomain.
MMv1 resource configurations may reference handwritten code stored in
magic-modules/mmv1/templates/terraform,
which will be injected into the generated resource file. Many MMv1 resources also have one or more handwritten tests,
which are stored in the appropriate service folder inside
magic-modules/mmv1/third_party/terraform/services.
In the providers, MMv1-based resources are stored in PROVIDER/services/PRODUCT/resource_PRODUCT_RESOURCE.go, where PROVIDER
is google or google-beta, PRODUCT is the product name, and RESOURCE is the GCP API resource’s name converted to
snake case ↗.
MMv1-based files start with the following header:
*** AUTO GENERATED CODE *** Type: MMv1 ***
Handwritten #
Handwritten resources and datasources are technically part of MMv1; however, they are not generated from YAML configurations.
Instead, they are written as Go code with minimal go template “version guards” to exclude beta-only features from the google
provider.
Handwritten resources and datasources can be grouped by “service”, which generally corresponds to the API subdomain the resource or datasource interacts with.
In addition to the core implementation, handwritten resources and datasources will also have documentation, tests, and sweepers (which clean up stray resources left behind by tests). Each type of code is stored in the following locations:
- Resource & datasource implementation: In the appropriate service folder inside
magic-modules/mmv1/third_party/terraform/services - Resource documentation:
magic-modules/mmv1/third_party/terraform/website/docs/r - Datasource documentation:
magic-modules/mmv1/third_party/terraform/website/docs/d - Tests: In the appropriate service folder inside
magic-modules/mmv1/third_party/terraform/services - Sweepers:
magic-modules/mmv1/third_party/terraform/utils
In the providers, handwritten resources and datasources are stored in PROVIDER/services/SERVICE/FILENAME.go, where PROVIDER
is google or google-beta, SERVICE is the service name, and FILENAME is the name of the handwritten file in magic-modules.
Handwritten files do not have an AUTO GENERATED CODE header.
DCL aka tpgtools (maintenance mode) #
DCL / tpgtools is similar to MMv1; however, it is in maintenance mode, which means that new resources using the DCL are not being added.
DCL-based files start with the following header:
*** AUTO GENERATED CODE *** Type: DCL ***