TBS: Document discard_on_write_failure + expose it to the APM Integration

[lucabelluccini]

Potential follow up of #15159

• Docs: document discard_on_write_failure
• Integration (APM): expose the setting via
  • https://github.com/elastic/integrations/blob/a55e9eecb844d1d9192eb8584b81c687a213ef1b/packages/apm/agent/input/template.yml.hbs#L74
  • https://github.com/elastic/integrations/blob/main/packages/apm/manifest.yml

Changes required

This task will be similar to the sampling.tail.ttl task in #13525 in terms of the changes required

[carsonip]

It was intentional to have sampling.tail.discard_on_write_failure undocumented, as it was supposed to be an escape hatch for users who are facing TBS storage limit issues while we look for a long term solution.

I understand the pain to configure this config (and many more configs not exposed in apm integration package) and I agree we should make it easier.

Regarding documentation, I'm not sure if the team is committed to support this config in the long run. cc @simitt

[isaacaflores2]

• APM Integrations PR: apm: Add config for tail-based sampling discard on write integrations#13950
  • v8.19.0 Backport: apm: Backport TBS ttl and discard_on_write_failure to 8.19 integrations#14029
  • Validate agent policy for 9.1.0-SNAPSHOT using local stack with elastic-package stack up -d --version 9.1.0-SNAPSHOT
  • Validate agent policy for 8.19.0
• Docs PR: apm: Document sampling.tail.discard_on_write_failure config docs-content#1453
  • Update 8.x docs
• APM Integration policy UI Issue: [APM] Support sampling discard_on_write_failure configuration in apm integration policy kibana#221441
• Update any APM Server docs

[isaacaflores2]

I documented all the steps here to validate the APM integration changes per @rubvs recommendation.
apm-integrations-validation-steps.md

[simitt]

@isaacaflores2 IMO https://github.com/elastic/apm-server/tree/main/dev_docs would be a good place for documenting how to test integration package changes for APM. You can reduce the discard_on_write_failure specifics, or keep them in as an example.