Archive for the ‘Flume’ Category

How to setup log rotation on Flume

Flume can be used to rotate logs file into a regular file system. Examples: In BE mode (Best Effort) exec config remuscl2-vm.atomant.net ‘tail("/tmp/error.log")’ ‘agentBESink("remuscl1-vm.atomant.net",35888)’ exec config remuscl1-vm.atomant.net ‘collectorSource( 35888 )’ ‘collectorSink("/tmp/nlog/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",60000 )’ in E2E (End to End) exec config remuscl2-vm.atomant.net ‘tail("/tmp/error.log")’ ‘agentE2ESink("remuscl1-vm.atomant.net",35888)’ exec config remuscl1-vm.atomant.net ‘collectorSource( 35888 )’ ‘collector(600000) {collectorSink("/tmp/nloge/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",600000 )}’ Explanations: collectorSink [...]






How to use the Flume E2E mode

E2E provides the best reliability mode: You can tune the reliability level of the agents, simply by specifying a different kind of agent sink. There are three levels available. One of them is the E2E (End To End) mode, which relies on an acknowledgement, and will retry if no acknowledgement is received. E2E limitations: E2E [...]






How to embed Flume node into your Java application

You may need to embed Flume node into your Java Program in a case you would like your application to control the incoming logs (you can then create new Sinks or Sources in the application). Embedding Flume into Java Application is possible and very easy to do. Instructions: 1. Install flume node 2. Configure your [...]






How to run the Flume’s HelloWorld example

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. For more information see extending_via_sink_source_decorator_plugins The flume’s Hello World example is an example of an experimental plug-in mechanism that allows to add new custom sources, sinks, and decorators to the Flume system. 1. The source code [...]






How to install Flume on Linux

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. For more information see Flume Installation First you need to update your Yum repository: cd /etc/yum.repos.d/ sudo wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo In order to install Flume node: sudo yum install flume-node -y In order to install Flume master: [...]