WIP: Fix message too long

[olivergondza]

When message grows too long (+128kb), it can cause build cannot launch OS processes because of that.

(Filing early to see the reproducer is working in public CI)

[ggallen]

@olivergondza , what do you think is the ultimate solution here? The trigger should check the size of CI_MESSAGE and not schedule the job if it is too big? And it should just print out a warning?

[olivergondza]

Given the symptoms are quite surprising to investigate, I want to prevent such message to ever get into environment variable.
Since that the limitation stems from linux kernel itself, and thus affecting vast majority of the deployments I would argue that +128kb message is always a problem. Also, the error is intermittent if the message length fluctuates around the critical threshold further complicating investigation and building resilient automation.

I see 2 ways to go:

• Fail the build when the message is too long. This will force users to either rethink their message format, or configure job to pass problematic messages (they know can grow in size) through file.
• Abandon the envvar approach and always use file. The length limitation will not be a problem, no sporadic build failures, but this is massive compatibility break.

[ggallen]

Another option would be to truncate the variable value and mark the build as unstable. I'm not sure if the second part is even possible or not.

This will force users to either rethink their message format, or configure job to pass problematic messages (they know can grow in size) through file.

I'm not sure this is possible. I believe the environment variable is always set. I also think in many cases users have no control over the message itself, it's coming from some publisher that's a black box to them.

I think it may also be the case that many users don't even care about the variable.

We could also potentially do both. Default to failing the job, but provide an option to change that to unstable and submit with a truncated message.

[olivergondza]

Truncate + unstable will avoid the problem, but I would still go with failing the build. Unstable primarily means "we build it, but there are test failures, warnings, compliance check problems, etc", and that it might not attract the desired attention especially if the build was unstable before. Not to mention that the build will possibly fail later, trying to parse the message or use its incomplete form propagating the error silently further down the chain and making the cause harder to discover. I like to fail fast and loud.

I also think in many cases users have no control over the message itself, it's coming from some publisher that's a black box to them.

Can be, that is why I believe they need a way to say "I know my message can be too long for an envvar, so put it in the file instead".