How to setup log rotation on Flume

Flume can be used to rotate logs file into a regular file system.

Examples:

In BE mode (Best Effort)

exec config remuscl2-vm.atomant.net 'tail("/tmp/error.log")'     'agentBESink("remuscl1-vm.atomant.net",35888)'
exec config remuscl1-vm.atomant.net 'collectorSource( 35888 )'     'collectorSink("/tmp/nlog/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",60000 )'

in E2E (End to End)

exec config remuscl2-vm.atomant.net 'tail("/tmp/error.log")'      'agentE2ESink("remuscl1-vm.atomant.net",35888)'
exec config remuscl1-vm.atomant.net 'collectorSource( 35888 )'     'collector(600000) {collectorSink("/tmp/nloge/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",600000 )}'

Explanations:

collectorSink
usage:
collectorSink[(dfsdir,path[,rollmillis])]
dfsdir – in this example the directory in the collector. If it is not exist, Flume will create it.
path – in this example the logs file prefix
rollmillis – the roll time in milliseconds. Set this to be whatever you would like. When it is not provided, Flume uses flume.collector.roll.millis (which configured at /usr/lib/flume/conf/flume-conf.xml).

collector(millis):
Needed for the E2E mode. See How to use the Flume E2E mode.

Testing script for generating logs

A script to generate data for testing the Flume’s log rotation can be:

[bash]$ for i in {1..1000000}; do echo "Flume log rotation $i" >> /tmp/error.log; sleep 4; done

Notes:

In order to see only the body of the log set this in the collector’s configuration file: flume-site.xml

<property>
    <name>flume.collector.output.format</name>
    <value>raw</value>
    <description>The output format for the data written by a Flume
    </description>
</property>
One Comment
  1. Ilan Hazan says:

    If you are setting the output format flume.collector.output.format (for the data written by a Flume) to “raw”, your customized Sink will also get the log as “raw” format. In this case for Event e, the line
    e.getBody()
    is equivalent to
    e.toString().

    Either case will bring the body of the log (only).

Leave a Reply

*