Pinging @elastic/elastic-agent (Team:Elastic-Agent)

Based on the documentation close.on_state_change.removed option is enabled by default.
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-close-removed

This option is enabled by default.

Based on the documentation close.on_state_change.removed option is enabled by default. https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-close-removed

This option is enabled by default.

Yes, it is enabled by default, hence this issue to make it disabled by default.

With close.on_state_change.removed: true (the default) files might not be ingested until the end if they're deleted before Fielbeat finishes harvesting and shipping the data.

Either my English is bad, but the following sentence in your text says that the default value should be set to true, right? If not, you should modify the text accordingly.

One way to mitigate this is to set close.on_state_change.removed: true by default

I thought about setting close.on_state_change.removed: false in the configuration. The documentation also states that clean_removed should also be disabled.

If you disable this option, you must also disable clean_removed.

Do you know how Filebeat cleans up the registry afterwards? clean_inactive does not seem to me to be the right way. Maybe if you set a very high value (one week or more),

Either my English is bad, but the following sentence in your text says that the default value should be set to true, right? If not, you should modify the text accordingly.

One way to mitigate this is to set close.on_state_change.removed: true by default

That was a typo 🤦‍♂ , it should state false. Thanks for spotting that, I'll update the description.

I thought about setting close.on_state_change.removed: false in the configuration. The documentation also states that clean_removed should also be disabled.

If you disable this option, you must also disable clean_removed.

Yes, if you do not disable the clean_removed it can happen that the file was removed from the file system, Filebeat kept it open, but the registry clean up runs, sees the file has been deleted and remove it from the registry, which leads to an inconsistent state.

Do you know how Filebeat cleans up the registry afterwards? clean_inactive does not seem to me to be the right way. Maybe if you set a very high value (one week or more),

Yes, I know. There are lots of details to it, but in short, for the Filestream input, there is a sort of garbage collector that runs periodically, it checks if the entry can be removed based on TTL, file removed, inactivity, etc. If all conditions are met, then the entry is removed from the registry.

@belimawr do you know if it is possible to make this change on Fleet managed ECK Agents running Kubernetes Integration?

Looking at the Kubernetes integration 1.80.1, I can see a "Custom configurations" field, so one could add anything supported by the Filestream input there.

Thanks so much for that confirmation! I found that as well after I stood up a lab, and added like so: