Skip to main content

Apache Flume Installation

In this post, we briefly discuss about Apache Flume Installation and Configuration on Ubuntu machine.
The current version of Apache Flume is called as Flume NG (Next Generation) and it’s old version is renamed as Flume OG (Old Generation). In this post, we will discuss about Flume NG only.
  • JDK 1.6 or later versions of Java installed on our Ubuntu machine.
  • Memory – Sufficient memory for configurations used by sources, channels or sinks.
  • Disk Space – Sufficient disk space for configurations used by channels or sinks.
  • Directory Permissions – Read/Write permissions for directories used by agent
Apache Flume Installation On Ubuntu:

  • Download latest stable release of apache flume binary distribution from apache download mirrors at http://flume.apache.org/download.html . At the time of writing this post, apache-flume-1.5.0 is the latest version and the same (apache-flume-1.5.0.1-bin.tar.gz) is used for installation in this post.
  • Copy the apache-flume-1.5.0.1-bin.tar.gz from downloads folder to our preferred flume installation directory, usually into /usr/lib/flume and unpack the tarball. Below are the set of commands to perform these activities.
          $ sudo mkdir /usr/lib/flume
          $ sudo chmod -R 777 /usr/lib/flume
          $ cp apache-flume-1.5.0.1-bin.tar.gz /usr/lib/flume/
          $ cd /usr/lib/flume
          $ tar -xzf apache-flume-1.5.0.1-bin.tar.gz



And below is the screen shot from the installation terminal.


  • Set FLUME_HOME, FLUME_CONF_DIR environment variables in .bashrc file as shown below and add the Flume bin directory to PATH environment variable.
          $ gedit ~/.bashrc

    

  • In FLUME_CONF_DIR directory, rename flume-env.sh.template file to flume-env.sh and provide value for JAVA_HOME environment variable with Java installation directory.
  • If we are going to use memory channels while setting flume agents, it is preferable to increase the memory limits in JAVA_OPTS variable. By default, the minimum and maximum memory values are 100 MB and 200 MB respectively (Xms100m -Xmx200m). Better to increase these limits to 500 MB and 1000 MB respectively as shown below.

With these settings, we can consider flume installation as completed.

Verify Apache Flume Installation:
We can verify the flume installation with $ flume-ng –help command on terminal. If we get output similar to below then flume installation is successful.


Comments

Post a Comment

Popular posts from this blog

How to Setup Hadoop Multi-Node Cluster on Ubuntu

How to Setup Hadoop Multi-Node Cluster on Ubuntu In this tutorial, we will learn how to setup a multi-node hadoop cluster on Ubuntu 16.04. A hadoop cluster which has more than 1 datanode is a multi-node hadoop cluster, hence, the goal of this tutorial is to get 2 datanodes up and running. 1) Prerequisites Ubuntu 16.04 Hadoop-2.7.3 Java 7 SSH For this tutorial, I have two  ubuntu 16.04  systems, I call them  master  and  slave  system, one datanode will be running on each system. IP address of  Master  ->  192.168.1.37 IP address of  Slave  ->  192.168.1.38 On Master Edit hosts file with master and slave ip address. sudo gedit /etc/hosts Edit the file as below, you may remove other lines in the file. After editing save the file and close it. On Slave Edit hosts file with master and slave ip address. sudo gedit /etc/hosts Edit the file as below, you may remove other lines in the fi...
Post a Comment
Read more

Hbase installation on ubuntu

Hbase installation on ubuntu In this tutorial we will see how to install Hbase on ubuntu 16.04 by doing the following steps Step 1: Before installing Hbase, you need to First ensure that java8 is installed: sudo add-apt-repository ppa:webupd8team/java sudo apt-get update sudo apt-get install oracle-java8-installer Verify that java is correctly installed: java -version       Configuring Java Environment sudo apt-get install oracle-java8-set-default    Step 2: Ensure that you successfully installed hadoop on your machine  Check this link if you need to know how to install it.  Step 3: Download Apache Hbase Go to downloads page Choose hbase file: hbase-1.2.5-bin.tar.gz Step 4: Complete the installation process Move the downloaded file “ hbase-1.2.5-bin.tar.gz ” to your home (~) Compress it :  tar -zxvf hbase-1.2.5-bin.tar.gz Edit hbase-env.sh using this co...
Post a Comment
Read more

How To Install CouchDB and Futon on Ubuntu 14.04

How To Install CouchDB and Futon on Ubuntu 14.04    Introduction Apache CouchDB , like Redis, Cassandra, and MongoDB, is a NoSQL database . CouchDB stores data as JSON documents which are non-relational in nature. This allows users of CouchDB to store data in ways that look very similar to their real world counterparts. You can manage CouchDB from the command line or from a web interface called Futon. Futon can be used to perform administrative tasks like creating and manipulating databases, documents, and users for CouchDB. Goals By the end of this article, you will: Have CouchDB installed on a Droplet running Ubuntu 14.04 Have Futon installed on the same server Have secured the CouchDB installation Access CouchDB using Futon from your local machine, using a secure tunnel Know how to add an admin user to CouchDB Perform CRUD operations with CouchDB using Futon Perform CRUD operations with CouchDB from the command line Prerequisites Ple...
Post a Comment
Read more