Add TGC conversion#

This method of adding TGC support is currently experimental and not officially supported. We recommended this path only for core contributors.

terraform-google-conversion (TGC) consumes a Terraform plan and uses it to build Cloud Asset Inventory (CAI) Assets. These built assets only exist in memory locally.

TGC supports only those GCP resources that are available in both the Terraform provider and Cloud Asset Inventory.

Before you begin#

Getting a Terraform resource name from a GCP resource name#

The first step in determining if a GCP resource is supported is to identify the corresponding Terraform resource. You can often do this by searching for the GCP resource name in the Terraform google provider documentation.

Getting the CAI asset type of a GCP resource#

The second step in determining if a GCP resource is supported is to verify its inclusion in the Cloud Asset Inventory (CAI). You can complete this by searching for the resource’s asset type within CAI’s list of supported asset types.

Adding support#

Adding support for a resource has 4 steps:

1. Make changes to Magic Modules to add resource conversion code.
2. Generate terraform-google-conversion.
3. Run tests locally.
4. Make PRs for Magic Modules with your changes.

Each of these is discussed in more detail below.

1. Adding a resource to TGC#

// The type comes from https://cloud.google.com/asset-inventory/docs/supported-asset-types
const ProductResourceAssetType string = "whatever.googleapis.com/asset-type"

// ProductResourceSchemaName is the TF resource schema name for the Resource resource within Product.
const ProductResourceSchemaName string = "google_product_resource"

// Copy the resource schema from [google](https://github.com/hashicorp/terraform-provider-google) Terraform provider
func ResourceGoogleProject() *schema.Resource {
	return &schema.Resource{
		SchemaVersion: 1,

		Schema: map[string]*schema.Schema{
			...
		}
	}
}

// ...
"google_product_resource": product.ResourceName(),
// ...

func resourceConverterProductResource() ResourceConverter {
	return ResourceConverter{
		Convert:   GetProductResourceCaiObject,
	}
}

func GetProductResourceCaiObject(d TerraformResourceData, config *Config) ([]Asset, error) {
	// This function does basic conversion of a Terraform resource to a CAI Asset.
	// The asset path (name) will substitute in variables from the Terraform resource.
	// The format should match what is specified at https://cloud.google.com/asset-inventory/docs/supported-asset-types
	name, err := assetName(d, config, "//whatever.googleapis.com/projects/{{project}}/whatevers/{{name}}")
	if err != nil {
		return []Asset{}, err
	}
	if obj, err := GetProductResourceApiObject(d, config); err == nil {
		return []Asset{{
			Name: name,
			Type: ProductResourceAssetType,
			Resource: &AssetResource{
				Version:              "v1",  // or whatever the correct version is
				DiscoveryDocumentURI: "https://www.googleapis.com/path/to/rest/api/docs",
				DiscoveryName:        "Whatever",  // The term used to refer to this resource by the official documentation
				Data:                 obj,
			},
		}}, nil
	} else {
		return []Asset{}, err
	}
}

func GetProductResourceApiObject(d TerraformResourceData, config *Config) (map[string]interface{}, error) {
	obj := make(map[string]interface{})

	// copy values from the terraform resource to obj
	// return any errors encountered
	// ...

	return obj, nil
}

// ...
"google_product_resource": product.ResourceTfplan2caiConverter(),
// ...

// ProductResourceConverter for the Resource resource within Product.
type ProductResourceConverter struct {
	name   string
	schema map[string]*schema.Schema
}

// NewProductResourceConverter returns an HCL converter for compute instance.
func NewProductResourceConverter(provider *schema.Provider) common.Converter {
	schema := provider.ResourcesMap[ProductResourceSchemaName].Schema

	return &ProductResourceConverter{
		name:   ProductResourceSchemaName,
		schema: schema,
	}
}

// Convert converts asset to HCL resource blocks.
func (c *ProductResourceConverter) Convert(assets []*caiasset.Asset) ([]*common.HCLResourceBlock, error) {
	var blocks []*common.HCLResourceBlock
	for _, asset := range assets {
		if asset == nil {
			continue
		}

		if asset.Resource != nil && asset.Resource.Data != nil {
			block, err := c.convertResourceData(asset)
			if err != nil {
				return nil, err
			}
			blocks = append(blocks, block)
		}
	}
	return blocks, nil
}

func (c *ProductResourceConverter) convertResourceData(asset *caiasset.Asset) (*common.HCLResourceBlock, error) {
	if asset == nil || asset.Resource == nil || asset.Resource.Data == nil {
		return nil, fmt.Errorf("asset resource data is nil")
	}

	hclData := make(map[string]interface{})
	// copy values from the CAI asset to hclData
	// return any errors encountered
	// ...

	ctyVal, err := common.MapToCtyValWithSchema(hclData, c.schema)
	if err != nil {
		return nil, err
	}
	return &common.HCLResourceBlock{
		Labels: []string{c.name, "Whatever"},
		Value:  ctyVal,
	}, nil
}

// ...
"alloydb.googleapis.com/Backup": {
	"Default": alloydb.NewAlloydbBackupCai2hclConverter(provider),
},
// ...

2. Generate terraform-google-conversion#

To generate terraform-google-conversion code locally, run the following from the root of the magic-modules repository:

make tgc OUTPUT_PATH="/path/to/your/terraform-google-conversion"

3. Run tests locally#

Before you begin#

1. Set the following environment variable:

export WRITE_FILES=true

2. Please request the necessary permission for the GCS bucket cai_assets_metadata, which stores the nightly test metadata required to run the integration tests.

   Ensure you have the correct Go version installed. Follow Before you begin and Setup your development environment from the Magic Modules documentation.

Run unit tests#

To run the unit tests locally, run the following from the root of the terraform-google-conversion repository:

make test

Run integration tests#

In the following examples, the resource being tested is google_alloydb_backup.

To run the integration tests for the added resource locally, run the following from the root of the terraform-google-conversion repository:

make test-integration-local TESTPATH=./test/services/alloydb  TESTARGS='-run=TestAccAlloydbBackup' > alloydbBackup.log

To run one integration test for the added resource locally, run the following from the root of the terraform-google-conversion repository:

make test-integration-local TESTPATH=./test/services/alloydb  TESTARGS='-run=TestAccAlloydbBackup_alloydbBackupBasicTestExample' > alloydbBackup.log

The core integration tests in terraform-google-conversion mirror the naming of the corresponding acceptance tests in the Terraform provider. This testing process uses a crucial round-trip validation method:

1. Test Input Data

   The input for each equivalent integration test combines two elements:

   • The original Terraform resource configuration from the acceptance test step.

   • The Cloud Asset Inventory (CAI) data exported from the actual resources provisioned during the nightly execution of the Terraform provider tests.

2. Test File Generation (WRITE_FILES Flow)

   By setting the environment variable WRITE_FILES, running the integration tests locally generates a series of files in the service folder, detailing the entire conversion cycle.

   For example, running TestAccAlloydbBackup_alloydbBackupBasicTestExample successfully creates five files for each step:

The integration tests pass only when two conditions are met:

1. Every field in the original raw_config must exist within the generated export_config.
2. The export_config and the final roundtrip_config must contain the same sets of fields.

Address integration test failures#

To resolve integration test failures, you need to apply the correct override configuration to the Resource.yaml file, based on the root cause of the data mismatch or failure.

• Test Specific Overrides

  These rules are applied when a specific test cannot be reliably executed or fails due to expected differences in zero values.

  • Ignore Zero Values (in raw_config)

    Add tgc_test_ignore_extra to the resource example in Resource.yaml. This tells the test to ignore fields that have a zero value in the original Terraform configuration (raw_config) but retain a value in the exported CAI assets (export_assets).

  • Skip the Entire Test

    Add tgc_skip_test:REASON to the resource example in Resource.yaml, providing a brief explanation for why the test is being skipped.

• Resource-specific Overrides

  These rules address mismatches in how the resource is identified or where it is converted.

  • Fix CAI Resource Kind Mismatch

    If the CAI asset type differs from the Terraform API resource kind (api_resource_type_kind), add the correct cai_resource_kind to the Resource.yaml file.

  • Ignore Default Terraform Encoder

    If the default Terraform encoding applied during conversion is causing issues, disable it by adding tgc_ignore_terraform_encoder: true to the Resource.yaml file.

  • Custom Decoder (CAI → GET API object Mismatch)

    This is used when the value of a field in a CAI asset is different from the value required in the final API object during cai2hcl.

    1. Add the tgc_decoder file path to the field in Resource.yaml.

    2. Implement the custom Go code in the corresponding file under mmv1/templates/tgc_next/decoders.

  • Custom Encoder (CREATE API object -> CAI Mismatch)

    This is used when the value of a field in a CAI asset is different from the value needed in the API object when the resource is first created during tfplan2cai.

    1. Add the tgc_encoder file path to the field in Resource.yaml.

    2. Implement the custom Go code in the corresponding file under mmv1/templates/tgc_next/encoders.

• Field-Specific Overrides

  These rules are used when custom code is required because the field values or structure change during the tfplan2cai or cai2hcl conversion process.

  • Ignore Field Missing in CAI

    If a field is present in the Terraform resource but not supported or exported by Cloud Asset Inventory (CAI), add is_missing_in_cai: true to that specific field’s definition in the Resource.yaml file.

  • Ignore Default Terraform Custom Flatten

    If the default flattening logic is incorrect for TGC, disable it by adding tgc_ignore_terraform_custom_flatten: true to the field definition.

  • TGC Specific Expander / Flattener

    For custom tfplan2cai conversion logic, add custom_tgc_expand to the field.

    For custom cai2hcl conversion logic, add custom_tgc_flatten to the field.

4. Make PRs#

Now that you have your code working locally, open a PR for Magic Modules.

For the Magic Modules PR, check the build results within presubmit-generate-diffs. Click through the build links, and find the results of tgc-test, tgc-test-integration-* - the other checks only matter if you’re also making changes to the terraform provider.

If tgc-test fails, make sure you can run the unit tests successfully locally. If any of tgc-test-integration-* fails, make sure you can run the integration tests successfully locally.