Not getting all logs from build #861
Comments
|
Hello @BradyShober, Does it make sense? |
|
Hi @cyrille-leclerc, When we choose to not ship logs to Elastic, this is the log output of this pipeline: When we enable log shipping to Elastic, this is the output: If I use the link in the log outputs to Kibana, I see matching output: However, I can find the expected output by using the link on the build page and going to the Logs tab |
|
As Cyrille commented, you only see the logs that come from the Jenkins Controller. That means your build Agents do not have access to the Backend, so they can not send the logs generated by the build Agent. So check that you can reach your log backend from the Build agents, then give them the rules you need in your infrastructure to allow them to reach the Backend. |
|
@kuisathaverat I hadn't explicitly mentioned it in my previous reply, but I did verify connectivity from the agent to the backend. Also if you refer to my last point in my previous reply, with some digging I can locate the logs I expect to see in elastic, something just must not be getting set appropriately to tie them to the build to get them pulled in correctly. |
|
I see, cheking the latest screenshot, the timestamp of the record in backend is in the past compared with the timestamp of the Jenkins record. There are 2 msec of difference but only in the agent. I do not think it is a problem, I just noticed it. I wonder if it could be related to the range of time we set in the link. Your pipeline is simple and it takes only a second to run the whole pipeline. Could you try to add a |
|
@kuisathaverat I should mention that yes while my sample pipeline is very quick, that is just to illustrate the issue with a simple example and this is behavior that was observed on actual builds that do take more time. I did go ahead and add the sleep and run the test again. If I go looking with the other link on the main build page and go to the Logs tab I do see the output I also checked the attributes of the logs in elastic to see if anything stood out to me, the main differences I see that may be making a difference is that the ones that are not getting pulled in do not have span.id or trace.id attributes. |
Does this mean it does not happen in all job builds? I will try to replicate it |
|
I have tested with the demo, and the following environment works as expected: Jenkins Core 2.452.2 [LTS] Only when the agent loses connectivity with the backend, those logs are not sent. All the fields are filled in any case I did not find a way that the span.id or trace.id are not processed. |
|
Can you please test the connectivity from the Jenkins build agent to the OTLP endpoint:
io.jenkins.plugins.opentelemetry.api.util.OpenTelemetryUtils.testLogRecordExporter()
|
|
I attempted this test on an agent and I was able to see the log entry in elastic. |
Thanks, so you saw this test message emitted by the Jenkins Agent but not some of the logs of the pipeline that executes on this agent, correct? |
|
Yes, I was able to see the test message from the agent. The behavior during pipeline executions is that I am able to see logs from the controller but not from the agent in the Jenkins UI, but I am able to find the logs from the agent in elastic. |
|
Do you see the build agent logs on builds that are more than 5 minutes finished? |
|
No, the logs do not show up after any period of time |
|
You only have the logs from the Jenkins Controller, so the issue is related to connectivity from the EC2 instances to your backend. In the logs of your Agents, there must be exceptions related to sending the logs. I was not able to replicate it with EC2 agents. |
|
@kuisathaverat As I previously noted, I have been able to locate the agent logs in elastic, so they are being sent, they are just not getting pulled into the Jenkins output page. Would you be able to try using the EC2 fleet plugin to reproduce? We use this with an auto scaling group instead of static agents, I am wondering if the way agents are being managed with this plugin could be interfering in some way. |
I have used a static EC2 instance, but it should be the same; in the end, it is an Agent, so it does not matter which plugin creates it.
I missed that comment in my latest review. If the logs are in the backend, they are sent. Reviewing my comments, I saw this one #861 (comment), Could you compare the |
|
There seems to have been some changes to our enterprise Kibana that is preventing me from testing with our existing cluster, so I set up a sandbox environment to do further testing. In this sandbox environment I have installed the latest plugin version and am getting this error when it tries to communicate with elastic |
|
The latest version of the plugin is tested only with Elastic Stack 8.18.0 and 9.0.0, and the recommended way to ingest the OpenTelemetry data is EDOT. Our reference implementation is in the demo folder. |
|
I did stand up a fresh 8.18.0 elastic in my sandbox, this is what I am getting this error against. I had originally been using version 3.1495.v64dcff5b_7a_6c of the plugin, but in my sandbox not even the controller logs were pulling back, but I was not seeing this error. I decided to update to the latest version since I saw in the release notes for a version in between the one I was on and the latest something about 8.18 support and thought maybe the version I was on may not have been fully compatible. After updating the plugin is when this error started. |
Jenkins and plugins versions report
Environment
What Operating System are you using (both controller, and any agents involved in the problem)?
Controller is running in a container based on jenkins/jenkins:2.440.3-lts-alpine
Agents are Windows Server 2022 and Debian 11
Reproduction steps
Expected Results
We would expect to see the same logs within Jenkins as we did before
Actual Results
The logs getting pulled in have essentially no useful content. If you use the "View logs in" link in the console output, you see the same logs as it is pulling in, however I have found that if from the build you use the "View pipeline with" link that I am able to see the logs I expect to see.
An example of what is getting pulled in:
Anything else?
No response
Are you interested in contributing a fix?
No response
The text was updated successfully, but these errors were encountered: