Backport of VAULT-32657 deprecate duplicate attributes in HCL configs and policies into release/1.19.x

[hc-github-team-secure-vault-core]

Backport

This PR is auto-generated from #30386 to be assessed for backporting due to the inclusion of the label backport/1.19.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@bosouza
This person should resolve the merge-conflict(s) by either:

• Manually completing the cherry picks into this branch
• Creating a new branch and manually cherry-picking all commits being backported

merge conflict error: POST https://api.github.com/repos/hashicorp/vault/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Description

This PR deprecates the usage of duplicate attributes in HCL configuration files and policy definitions. An example of a duplicate attribute is the following, which until support is fully removed will continue parsing to the last occurence of the attribute:

path "secret" {
  capabilities = ["read"]
  capabilities = ["read", "create", "update", "delete"]
}

Following our deprecation process, these changes correspond to the deprecation phase to be announced in v1.20 (and cherry picked to v1.19, likely out in v1.19.4), implementing the following:

• CLI commands that take a configuration file will print WARNING: Duplicate keys found. A comprehensive list of those commands is: vault server, vault operator migrate, vault agent, vault proxy and vault operator diagnose;
• CLI commands using a token helper file (under $HOME/.vault) with duplicate attributes will print a similar WARNING: Duplicate keys found message;
• Any usages of a policy already stored in the system that contains duplicate attributes will print a warning on the server logs containing at least policy contains duplicate attributes. These include initial caching of policies after unseal, reading policies via the API, using a token that references such a policy among others;
• Creating or updating a policy to have duplicate attributes will cause a message to be added to the Warnings field in the server response.

This was accomplished by upgrading the hcl dependency to reject duplicate attributes, and using the new functions exposed in hashicorp/hcl#707 to create a helper function ParseAndCheckForDuplicateHclAttributes that parses the hcl and reports if there are duplicates.

Unfortunately, the library also had a behavior change that I traced back to hashicorp/hcl@e80118a not being included in the v1.0.1-vault-5 of the library we're using (as it was already included in v1.0.1-vault-4). This means that an approach we've been using of double-parsing some config fields now doesn't work the same anymore, which I'm trying to fix by removing the automatic parsing from the fields that are manually parsed. More details on the specific commit message.

Throughout the code I've left these TODO (HCL_DUP_KEYS_DEPRECATION) comments to more easily find all the places that will need to be changed/deleted after each of the following steps on the deprecation process.

For 1.20 we will only have these warnings, but on v1.21 all the operations described above involving HCL definitions with duplicate attributes will simply fail. We will add an environment variable though to allow the behavior to be rolled back to a log-only mode, at least until the final removal is completed on a future version.

Jira: VAULT-32657
ADR: VLT-006: Deprecate and remove duplicate attributes in HCL files in Vault

TODO only if you're a HashiCorp employee

• Backport Labels: If this fix needs to be backported, use the appropriate backport/ label that matches the desired release branch. Note that in the CE repo, the latest release branch will look like backport/x.x.x, but older release branches will be backport/ent/x.x.x+ent.
  • LTS: If this fixes a critical security vulnerability or severity 1 bug, it will also need to be backported to the current LTS versions of Vault. To ensure this, use all available enterprise labels.
• ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
  of a public function, even if that change is in a CE file, double check that
  applying the patch for this PR to the ENT repo and running tests doesn't
  break any tests. Sometimes ENT only tests rely on public functions in CE
  files.
• Jira: If this change has an associated Jira, it's referenced either
  in the PR description, commit message, or branch name.
• RFC: If this change has an associated RFC, please link it in the description.
• ENT PR: If this change has an associated ENT PR, please link it in the
  description. Also, make sure the changelog is in this PR, not in your ENT PR.

---

Overview of commits

• 0b91571

[github-actions]

CI Results:
All Go tests succeeded! ✅