Research And Development Blog

Flume can be used to rotate logs file into a regular file system. Examples: In BE mode (Best Effort) exec config remuscl2-vm.atomant.net ‘tail("/tmp/error.log")’ ‘agentBESink("remuscl1-vm.atomant.net",35888)’ exec config remuscl1-vm.atomant.net ‘collectorSource( 35888 )’ ‘collectorSink("/tmp/nlog/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",60000 )’ in E2E (End to End) exec config remuscl2-vm.atomant.net ‘tail("/tmp/error.log")’ ‘agentE2ESink("remuscl1-vm.atomant.net",35888)’ exec config remuscl1-vm.atomant.net ‘collectorSource( 35888 )’ ‘collector(600000) {collectorSink("/tmp/nloge/%Y/%m/%d/errorlog", "mylog_%{host}_%H_",600000 )}’ Explanations: collectorSink [...]

E2E provides the best reliability mode: You can tune the reliability level of the agents, simply by specifying a different kind of agent sink. There are three levels available. One of them is the E2E (End To End) mode, which relies on an acknowledgement, and will retry if no acknowledgement is received. E2E limitations: E2E [...]

You may need to embed Flume node into your Java Program in a case you would like your application to control the incoming logs (you can then create new Sinks or Sources in the application). Embedding Flume into Java Application is possible and very easy to do. Instructions: 1. Install flume node 2. Configure your [...]

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. For more information see extending_via_sink_source_decorator_plugins The flume’s Hello World example is an example of an experimental plug-in mechanism that allows to add new custom sources, sinks, and decorators to the Flume system. 1. The source code [...]

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. For more information see Flume Installation First you need to update your Yum repository: cd /etc/yum.repos.d/ sudo wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo In order to install Flume node: sudo yum install flume-node -y In order to install Flume master: [...]