Posts Tagged ‘flume’

How to setup log rotation on Flume

Flume can be used to rotate logs file into a regular file system. Examples: In BE mode (Best Effort) exec config ‘tail("/tmp/error.log")’ ‘agentBESink("",35888)’ exec config ‘collectorSource( 35888 )’ ‘collectorSink("/tmp/nlog/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",60000 )’ in E2E (End to End) exec config ‘tail("/tmp/error.log")’ ‘agentE2ESink("",35888)’ exec config ‘collectorSource( 35888 )’ ‘collector(600000) {collectorSink("/tmp/nloge/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",600000 )}’ Explanations: collectorSink […]

How to use the Flume E2E mode

E2E provides the best reliability mode: You can tune the reliability level of the agents, simply by specifying a different kind of agent sink. There are three levels available. One of them is the E2E (End To End) mode, which relies on an acknowledgement, and will retry if no acknowledgement is received. E2E limitations: E2E […]