create_before_destroy behavior on count/for_each resources is not clearly documented

[geekofalltrades]

Terraform Version

Terraform v1.4.2
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.67.0

Your version of Terraform is out of date! The latest version
is 1.11.3. You can update by downloading from https://www.terraform.io/downloads.html

Affected Pages

• https://developer.hashicorp.com/terraform/language/meta-arguments/count
• https://developer.hashicorp.com/terraform/language/meta-arguments/for_each
• https://developer.hashicorp.com/terraform/language/meta-arguments/lifecycle

What is the docs issue?

The behavior of create_before_destroy on a resource collection type using count or for_each isn't clear from documentation. There are two ways it could be interpreted to behave:

1. Every resource in the collection is an individual resource and create_before_destroy is applied to each separately. If a plan wants to both add and delete resources with separate for_each keys or count indices from the collection, the creates and destroys happen in an undefined order.
2. create_before_destroy applies to the collection as a whole. If a plan wants to both add and delete resources with separate for_each keys or count indices from the collection, all resources to add will be created before any resources to remove are destroyed.

We had a brief production outage because we assumed the latter, but we think we observed the former. The docs do not make it clear which behavior to expect.

Proposal

Explicitly document how create_before_destroy and maybe lifecycle arguments in general are applied to resources in a collection and whether any of them apply to the collection as a whole.

References

No response

[jbardin]

Hi @geekofalltrades,

The create_before_destroy lifecycle option isn't related to count or for_each, which is why it's not mentioned, and often adding in extraneous details creates more confusion in the long run. Terraform's default ordering is to destroy then create any instance when replacement is required, and create_before_destroy inverts that order, but otherwise doesn't change the semantics of any other part of the process. Terraform also does not imply any overall ordering of expanded count or for_each resources, which means that normal instance replacement is also unordered, the same idea as what you've experienced.

Rather than artificially linking create_before_destroy to the behavior of resource expansion, perhaps it would be better to just define how resource expansion doesn't imply any batching or ordering of changes? (currently that is set via omission, there is no ordering unless explicitly defined)

[geekofalltrades]

The create_before_destroy lifecycle option isn't related to count or for_each

We thought that it was, and I don't think that was an unreasonable conclusion to reach.

resource "aws_instance" "foo" {
  for_each = toset(["a", "b"])

  # ...

  lifecycle {
    create_before_destroy = true
  }
}

The configuration structure groups these resources together into a collection, with the lifecycle block applied to the entire collection.

At least some lifecycle options have special behavior when used in collection types.

When used in a resource that uses count or for_each, you can use count.index or each.key in the expression to reference specific instances of other resources that are configured with the same count or collection.

It sounds like replace_triggered_by can allow you to target either the entire collection or individual resources within the collection.

A paragraph in the lifecycle docs about how count and for_each collections expand into separate resources to which lifecycle options are applied individually unless otherwise noted would clear it up.

[jbardin]

Yes, you can reference an expanded resource like aws_instance.foo to get the resource collection value containing all its instances, but that is not unique to or influenced by lifecycle. Whether the calling resource is expanded or not makes no difference, except whether count.index or each.key return valid values.

The behavior of replace_triggered_by only ever replaces instances, because instances are all that can be replaced. For a related example, if you try to manually force replacement of that resource using -replace aws_instance.foo, terraform will warn you that you have an Incompletely-matched force-replace resource instance and will list the individual instances which you could choose to replace instead.

Maybe to put it another way, a "resource" has no lifecycle, it is just a configuration object. Only resource instances have a lifecycle of some sort for Terraform to manage. I think a lot of users automatically insert the word "instance" or "object" wherever it's needed when the context requires talking about the managed object vs the configuration. The lifecycle docs do try to help using the word "object", as in "infrastructure object", "remote object", etc. That page could probably be re-worked to try an unify the language a bit which has been accumulated over many years.