functionaltests: Generalized upgrade tests

[ericywl]

Problem

Current upgrade tests are not automatic. Each version upgrade test tests (most of the time) only upgrades between consecutive minors e.g. 8.17 -> 8.18. This is due to the fact that version upgrades have some intricacies in the way that we assert them currently.

Our assertions include lazy rollover (no. indices per data stream) and per-index lifecycle management. This is good for catching issues that happened e.g. in 8.15, but also means that we have to manually add the tests every time a new minor comes out.

Generalized tests

What we can do is to add generalized tests with moving targets.

Generalization

These generalized tests will not go into the specifics (lazy rollover / per-index lifecycle management), and only focus on two things: document counts and data stream lifecycle. We know that in the foreseeable future, the data streams in APM should be managed solely by ILM.

Moving Targets

They will also be towards moving targets i.e. 8.x or 9.x instead of any specific minor versions. For example, if we start from 8.15, we will implement TestUpgrade_8_15_to_8_x or TestUpgrade_8_15_to_8_x_to_9_x. We will need to handpick a few source versions to do this, as implementing for all of them would defeat the purpose of having the generalized tests in the first place (not manually updating tests)

[ericywl]

I will experiment on this next week (May 12 - 16)

[ericywl]

After discussion with @simitt and @endorama, there are a few things to be done in subsequent PRs.

1. Clean up the internal package
   • Move versions.go into internal/ecclient
   • Merging files with similar functionalities together e.g. internal/kbclient/client.go and internal/kbclient/upgrade_assist.go
   • Rename ecclient -> ech, kbclient -> kibana, esclient -> elasticsearch
   • internal/asserts should probably be in the main test folder instead (?)
2. Rename functionaltests to integrationservertest
3. Double check old upgrade smoke test and remove them

One remaining question:

1. What to do with the existing smoke-test after we remove the upgrade tests?

cc @raultorrecilla