[FR] Create Provider Address Environment Variables - Terraform Init

[straubt1]

Terraform Version

All versions: v1.10.x

Use Cases

Provider Sourcing

For Terraform Enterprise users, sourcing Terraform Providers from private Registries has some challenges that create friction in their journey to governing binaries within their network.

There are two main use cases described here:

1. Self-Managed Customers who run TFE from within an airgapped environment which do not have network access to the Public Registry.
2. Customers who have created private Terraform Providers that are (by design) not published to the Public Registry.

Background

When authoring Terraform (*.tf) files a Provider is required to interact with "resources". A Terraform Provider is a binary that is sourced during the terraform init command and downloaded from a Registry that implements the Provider Registry Protocol.

Terraform Providers are sourced leveraging a "Provider Address" that consists of three different settings in the format "{hostname}/{namespace}/{type}":

• "hostname" - The Registry that contains the Providers. Defaults to "registry.terraform.io".
• "namespace" - The container for providers within a Registry, usually indicates the owner. Defaults to "hashicorp".
• "type" - The name of the Provider (such as "azurerm", "aws", "google", "dns", etc...) and is unique within a particular hostname and namespace. Defaults to the resource prefix present in the code.

When a resource block is present, the Provider "type" will be everything to the left of the first underscore ("_").

Example using Defaults

The following code would have the Provider "type" of "random":

resource "random_pet" "server" { }

If there was no other code provided, this provider would be sourced from "registry.terraform.io/hashicorp/random" based on the defaults.

Example using Explicit Sourcing

Terraform provides a mechanism to explicitly set the "Provider Address" (in part or in full) using the required_providers block.

terraform {
  required_providers {
    random = {
      source = "mynamespace/random"
    }
  }
}

or

terraform {
  required_providers {
    myawesomething = {
      source = "tfe.company.com/mynamespace/myawesomething"
    }
  }
}

Attempted Solutions

Challenges

When hosting your own private Terraform Providers (developed within a company and not publicly available), or pulling public Terraform Providers into an airgapped TFE installation (a private network without egress to reach "registry.terraform.io"), the required_providers block is required within your Terraform code.

If we build on top of the example above, let's consider some Terraform Code that references several other Terraform Modules.

I have to provide explicit sourcing for my Provider:

terraform {
  required_providers {
    myawesomething = {
      source = "tfe.company.com/mynamespace/myawesomething"
    }
  }
}

This code has to be at the root module level, but it also must be present in every Terraform Module that it consumes!

Unfortunately this results in a lot of code maintenance that could be otherwise avoided.

Proposal

Recommended Solution

The goal here is to allow changing the behavior of Terraform Provider sourcing during a terraform init without requiring an update to the Terraform Code itself.

In each of the recommendation let's assume that the following code is present:

terraform {
  required_providers {
    myawesomething = {
      version = "~> 1.0.0"
    }
  }
}
resource "myawesomething_compute" "server" { 
  # specific attributes, not relevant...
}

module "server_farm" {
  source "./modules/server_farm"
  # specific attributes, not relevant...
}

The local module (which could be remote as well) at "./modules/server_farm" has a similar provider reference.

Implement New Environment Variables

Create a new configuration options that are settable via an Environment Variables that are used during terraform init when no source attribute is set in any required_providers block.

The proposed settings are:

• "TF_REGISTRY_HOSTNAME"
  • Default to "registry.terraform.io".
• "TF_REGISTRY_NAMESPACE"
  • Defaults to "hashicorp".

Effectively this would not change any current behavior when these Environment Variables are not set. It would however give great flexibility to those who who are either moving or will move how Terraform Providers are sourced within their environments.

Other Considerations

Terraform CLI Configuration File

There may be a desire to allow this configuration using the Terraform CLI Configuration File, which would be a great idea as long as these settings are available as Environment Variables.

In TFE or HCP Terraform, it is difficult to modify this CLI configuration since the platform creates it's own during a Terraform Run. Currently the only way to update this CLI configuration is with a custom Terraform Agent. Therefore it is critical that if these settings are to be implemented, that they be set with Environment Variables.

Lock File

With this change, the .terraform.lock.hcl file would maintain the actual Registry that was sourced during terraform init since the "Provider Address" is present in that lock file.

Example:

provider "registry.terraform.io/hashicorp/tfe" {
  version = "0.60.1"
  hashes = [
    "h1:qjwyzsUOdCBX12sU8qrqt/jlxkm/V0BDVeiLTtr8TBs=",
    "zh:018a44b4d8fc82d88a6bc668fbaf6d6b99a6c09db04d4091a82e800b8fd3040a",
    "zh:2ed39efb28e4f65252efc1b3c15af9c4b8245d085bedd79a5f06c7684a4c0717",
    "zh:5af13905b0aad0d348ffc286d539f141966d66911171f817a4cb5fec82e03f9b",
    "zh:645372da845c77eae1a063239c608ab3ff7bc2dfcec7d84270a9f2bc1a283d31",
    "zh:69979a63cc3634ef256c082fc824e7e475f5fd8148538ea9b0e17de6e0cf5451",
    "zh:6ef0f68da30738e9feee62b1ca67e5f4c4853078a1388b0270ff3ab9d0639eaa",
    "zh:89dfe931cc5e53884f28fc96c4ab1f7a664c5d91628606952cb3204cfee72f45",
    "zh:bcbb04d57a3cf8143b488f87ce019f4b3d8ae61b81bc9512a5f0218391981891",
    "zh:d87136e12a72efdb55722003099d35d70a5e5e8dc591bf3aded92a657d047ac5",
    "zh:d95ec293fa70e946b6cd657912b33155f8be3413e6128ed2bfa5a493f788e439",
    "zh:f7053d02240cbba0fd4c745fdfb77451c6e161908a43c532538edcb0f008a263",
    "zh:fcd8e52f7886fd0d08716409f2fe0b4e955f4b0c972a49ed3f8e67cdae9bbaec",
  ]
}

References

No response

[crw]

Thanks for this feature request! If you are viewing this issue and would like to indicate your interest, please use the 👍 reaction on the issue description to upvote this issue. We also welcome additional use case descriptions.

Thanks again!

[straubt1]

The TFE example was not meant to distract here, rather just the first example that made sense when describing a real world situation.

Just to clarify, the change requested here is to Terraform Core, not to any commercial products.

[jbardin]

Hi @straubt1,

There are a number of things we're going to have to consider closely here before deciding on an implementation:

• The provider source is not just to locate the provider during init, it is the canonical identifier for a provider within Terraform and the stored state.
• Related to the last point, required_providers is required within modules because modules within a configuration may need different providers with the same local name. The Terraform configuration on its own needs to be able to disambiguate between aws from registry.terraform.io/hashicorp/aws and registry.custom.io/custom/aws.
• We have a very long list of issues and forum posts about how the implicit registry and namespace for hashicorp providers was a mistake and confusing for users. We would need to ensure that this type of behavior does not contribute the to lack of clarity when a provider cannot be located or is incorrectly sourced.
• Many configurations source providers from multiple locations and multiple namespaces, but this assumes a single global source. If a user relied on these environment variables, then wanted to add another module which also relied on different settings of these variables, there's no way to reconcile the providers.