Integrating LVM with Hadoop and providing Elasticity to DataNode Storage

Elasticity is the concept which we can use to increase or decrease the volume of Hadoop Data Node. Hadoop data nodes shared storage can’t be static so LVM is used to make it dynamic

Logical Volume Manager (LVM) is a device mapper framework that provides logical volume management for the Linux kernel

LVM is used for the following purposes:

• Creating single logical volumes of multiple physical volumes or entire hard disks, allowing for dynamic volume resizing.
• Managing large hard disk farms by allowing disks to be added and replaced without downtime or service disruption, in combination with hot swapping.
• On small systems (like a desktop), instead of having to estimate at installation time how big a partition might need to be, LVM allows filesystems to be easily resized as needed.
• Performing consistent backups by taking snapshots of the logical volumes.
• Encrypting multiple physical partitions with one password.

* To complete this task , we have to use Linux Partition Concept .

* I will follow the below step for this task -

<1>. Add New HardDisk

<2>. Create Logical Volume

< 3>. Format the Logical Volume

<4>. Mount the DataNode dir with the Logical Volume

<5>. Restart DataNode

<6>. Check Contribution Of DataNode In Distributed File Storage of Hadoop Cluster

Add New HardDisk to DataNode :-
Now I am using Oracle Virtual Box so we don’t need to purchase new hard disk . We will use Virtual Hard Disk concept.

We are adding new hard disk because we don’t have any unallocated space to do partitions.

To add new hard disk DataNode must be in “Stopped” state then Follow this steps -
(A) Go Storage in Settings of DataNode -

(B) Click on “Controller: SATA” & after this click on right “+” icon of “Controller: SATA”

Now our Hard Disk is attached -

Create Logical Volume

1. Create Physical Group

pvcreate /dev/sdc

pvcreate /dev/sdb

Display Physical Group

pvdisplay

2) Create Volume Group

vgcreate hadoop /dev/sdb /dev/sdb

Display Volume Group

vgdisplay

3) Create Logical Volume

lvcreate — size 10G — name hadoop_lvm hadoop

Display Logical Volume

Format the Logical Volume:-

mkfs.ext4 /dev/hadoop/hadoop_lvm

Mount the DataNode dir “/dn1 ” with the Logical Volume:-

mount /dev/hadoop/hadoop_lvm /dn1

Restart DataNode:-

hadoop-daemon.sh stop datanode
hadoop-daemon.sh start datanode

Check Contribution Of DataNode In Distributed File Storage of Hadoop Cluster:-

hadoop dfsadmin -report

Now we can increase or decrease the size of our DataNode dir by chaning the size of Logical Volume.

THANK YOU!

My LinkedIn: Himanshu Agrawal