How to embed Flume node into your Java application

You may need to embed Flume node into your Java Program in a case you would like your application to control the incoming logs (you can then create new Sinks or Sources in the application).

Embedding Flume into Java Application is possible and very easy to do.

Instructions:

1. Install flume node

2. Configure your dependencies (pom.xml):

 
<dependency>
 
    <groupId>com.cloudera</groupId>
 
    <artifactId>flume</artifactId>
 
    <version>0.9.1</version>
 
    <scope>system</scope>
 
    <systemPath>${basedir}/lib/flume-0.9.1+29-core.jar</systemPath>
 
</dependency>
 
<dependency>
 
    <groupId>com.google.collections</groupId>
 
    <artifactId>google-collections</artifactId>
 
    <version>1.0</version>
 
</dependency>
 
<dependency>
 
    <artifactId>flume</artifactId>
 
    <groupId>flume</groupId>
 
    <version>0.9.1</version>
 
    <scope>system</scope>
 
    <systemPath>${basedir}/lib/flume-0.9.1-core.jar</systemPath>
 
</dependency>
 
<dependency>
 
    <artifactId>libthrift</artifactId>
 
    <groupId>libthrift</groupId>
 
    <version>0.1</version>
 
    <scope>system</scope>
 
    <systemPath>${basedir}/lib/libthrift.jar</systemPath>
 
</dependency>
 
<dependency>
 
    <groupId>org.mortbay.jetty</groupId>
 
    <artifactId>jetty</artifactId>
 
    <version>5.1.4</version>
 
    <scope>system</scope>
 
    <systemPath>${basedir}/lib/jetty-5.1.4.jar</systemPath>
 
</dependency>
 
<dependency>
 
    <groupId>com.cloudera.hadoop</groupId>
 
    <artifactId>hadoop-core</artifactId>
 
    <version>0.20.2-320</version>
 
</dependency>
 
<dependency>
 
    <groupId>com.google.collections</groupId>
 
    <artifactId>google-collections</artifactId>
 
    <version>1.0</version>
 
</dependency>
 
<dependency>
 
    <groupId>org.antlr</groupId>
 
    <artifactId>antlr</artifactId>
 
    <version>3.1.3</version>
 
</dependency>
 
<dependency>
 
    <groupId>irclib</groupId>
 
    <artifactId>irclib</artifactId>
 
    <version>1</version>
 
    <scope>system</scope>
 
    <systemPath>${basedir}/lib/irclib.jar</systemPath>
 
</dependency>
 
<dependency>
 
    <groupId>log4j</groupId>
 
    <artifactId>log4j</artifactId>
 
    <version>1.2.14</version>
 
</dependency>
 
<dependency>
 
    <groupId>org.slf4j</groupId>
 
    <artifactId>slf4j-api</artifactId>
 
    <version>1.5.8</version>
 
</dependency>
 
<dependency>
 
    <groupId>org.slf4j</groupId>
 
    <artifactId>slf4j-simple</artifactId>
 
    <version>1.5.8</version>
 
</dependency>
 
<dependency>
 
    <groupId>commons-lang</groupId>
 
    <artifactId>commons-lang</artifactId>
 
    <version>2.4</version>
 
</dependency>

3. Change your Class Path:

CLASSPATH=lib/*:/usr/local/hadoop/conf:/usr/local/remus/conf:/usr/lib/flume/lib/*

4. Update FLUME_HOME variable

FLUME_HOME=/usr/lib/flume/

export FLUME_HOME=/usr/lib/flume/

5. Call to the following function from your main function:

 
private static void StartFlume(){
 
  try {
 
     String[] args = null;
 
     FlumeNode.setup(args);
 
  } catch (Exception e) {
 
  LOG.error("Aborting: Unexpected problem with environment." + e.getMessage(), e);
 
  System.exit(-1);
 
  }
}

And that is it. You have embedded Flume into your Java Application.

References:

https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/97717fff7ef19b2a/592aaee29ff738bb

https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/ce897eb04123fbad/47273bcf0e65a7b0

Leave a Reply

*