How to setup log rotation on Flume

Flume can be used to rotate logs file into a regular file system.


In BE mode (Best Effort)

exec config 'tail("/tmp/error.log")'     'agentBESink("",35888)'
exec config 'collectorSource( 35888 )'     'collectorSink("/tmp/nlog/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",60000 )'

in E2E (End to End)

exec config 'tail("/tmp/error.log")'      'agentE2ESink("",35888)'
exec config 'collectorSource( 35888 )'     'collector(600000) {collectorSink("/tmp/nloge/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",600000 )}'


dfsdir – in this example the directory in the collector. If it is not exist, Flume will create it.
path – in this example the logs file prefix
rollmillis – the roll time in milliseconds. Set this to be whatever you would like. When it is not provided, Flume uses flume.collector.roll.millis (which configured at /usr/lib/flume/conf/flume-conf.xml).

Needed for the E2E mode. See How to use the Flume E2E mode.

Testing script for generating logs

A script to generate data for testing the Flume’s log rotation can be:

[bash]$ for i in {1..1000000}; do echo "Flume log rotation $i" >> /tmp/error.log; sleep 4; done


In order to see only the body of the log set this in the collector’s configuration file: flume-site.xml

    <description>The output format for the data written by a Flume
One Comment
  1. Ilan Hazan says:

    If you are setting the output format flume.collector.output.format (for the data written by a Flume) to “raw”, your customized Sink will also get the log as “raw” format. In this case for Event e, the line
    is equivalent to

    Either case will bring the body of the log (only).

Leave a Reply