[Flaky Test] x-pack/filebeat/fbreceiver TestMultipleReceivers

[AndersonQ]

Flaky Test

• Test Name: x-pack/filebeat/fbreceiver TestMultipleReceivers
• Link: Failure
• Branch: Git branch the test was seen in. If a PR, the branch the PR was based off.
• Artifact Link: Failure in CI, retry also failed
• Notes: Additional details about the test. e.g. theory as to failure cause

Stack Trace

2025-04-10 15:06:03 CEST	=== Failed
2025-04-10 15:06:03 CEST	=== FAIL: x-pack/filebeat/fbreceiver TestMultipleReceivers (66.13s)
2025-04-10 15:06:03 CEST	SDK 2025/04/10 13:02:50 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 405, request to EC2 IMDS failed
2025-04-10 15:06:03 CEST	SDK 2025/04/10 13:02:53 WARN falling back to IMDSv1: operation error ec2imds: getToken, http response error StatusCode: 405, request to EC2 IMDS failed
2025-04-10 15:06:03 CEST	    oteltest.go:124:
2025-04-10 15:06:03 CEST	        	Error Trace:	C:/buildkite-agent/builds/bk-agent-prod-gcp-1744289200299222169/elastic/beats-xpack-filebeat/x-pack/filebeat/fbreceiver/receiver_test.go:206
2025-04-10 15:06:03 CEST	        	            				C:/buildkite-agent/builds/bk-agent-prod-gcp-1744289200299222169/elastic/beats-xpack-filebeat/libbeat/otelbeat/oteltest/oteltest.go:136
2025-04-10 15:06:03 CEST	        	            				C:/Users/Buildkite/.go/go-1.24.1/src/runtime/asm_amd64.s:1700
2025-04-10 15:06:03 CEST	        	Error:      	Condition failed!
2025-04-10 15:06:03 CEST	        	Messages:   	expected receivers to have logs, got logs: map[r2:[{"@timestamp":"2025-04-10T13:02:56.719Z","agent":{"ephemeral_id":"14428d21-ab6a-47f8-872c-cfe0f50bd2c6","id":"51d909ae-29f1-42d9-9996-dbec579795a1","name":"bk-agent-prod-gcp-1744289200299222169","type":"filebeat","version":"9.1.0"},"cloud":{"account":{"id":"elastic-ci-prod"},"availability_zone":"us-central1-b","instance":{"id":"1565360607196742494","name":"bk-agent-prod-gcp-1744289200299222169"},"machine":{"type":"n1-standard-4"},"project":{"id":"elastic-ci-prod"},"provider":"gcp","region":"us-central1","service":{"name":"GCE"}},"ecs":{"version":"8.0.0"},"filename":0,"host":"xxxxx","input":{"type":"benchmark"},"line":0,"message":"test","thread":0}]]
2025-04-10 15:06:03 CEST	    oteltest.go:124:
2025-04-10 15:06:03 CEST	        	Error Trace:	C:/buildkite-agent/builds/bk-agent-prod-gcp-1744289200299222169/elastic/beats-xpack-filebeat/libbeat/otelbeat/oteltest/oteltest.go:124
2025-04-10 15:06:03 CEST	        	            				C:/buildkite-agent/builds/bk-agent-prod-gcp-1744289200299222169/elastic/beats-xpack-filebeat/x-pack/filebeat/fbreceiver/receiver_test.go:191
2025-04-10 15:06:03 CEST	        	Error:      	Condition never satisfied
2025-04-10 15:06:03 CEST	        	Test:       	TestMultipleReceivers
2025-04-10 15:06:03 CEST	        	Messages:   	timeout waiting for assertion to pass

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[AndersonQ]

I found indication there might be something really going on there. Just increasing the timeout wasn't enough, see #43834 (comment).

[khushijain21]

Another instance of this flaky test
https://buildkite.com/elastic/beats-xpack-filebeat/builds/13951#01964339-8de2-4c12-885e-8e711f122c7b

[mauri870]

I looked into this test but couldn’t find anything obviously causing the failure. I can reproduce it after a dozen or so runs.

If I remove one of the receivers, the test no longer fails, so it’s definitely related to some interaction between the two receivers. Given Beats’ history with global state, it’s very likely something along those lines.