Technology Archives - Page 4 of 5 - Princeton IT Services

Category: Technology

Loading Customer Data into HBase using a PIG script

There are different ways to load data into HBase tables like: ‘put’ to manually load data records into HBase, ImportTSV and bulk load options. Alternatively, lets try to load huge customer data file into HBase using Apache PIG. The data set has the following fields:

Using MRUnit to Develop and Test MapReduce Jobs

Conceptually, MapReduce jobs are relatively simple. In the map phase, each input record has a function applied to it, resulting in one or more key-value pairs. The reduce phase receives a group of the key-value pairs and performs some function over that group. Testing mappers and reducers should be as easy as testing any other […]

Using FileSystem API to read and write data to HDFS

Reading data from and writing data to Hadoop Distributed File System (HDFS) can be done in a lot of ways. Now let us start by using the FileSystem API to create and write to a file in HDFS, followed by an application to read a file from HDFS and write it back to the local […]

33 Frequently Used HDFS Shell Commands

# Open a terminal window to the current working directory. # /home/training # 1. Print the Hadoop version hadoop version # 2. List the contents of the root directory in HDFS # hadoop fs -ls / # 3. Report the amount of space used and # available on currently mounted filesystem # hadoop fs -df […]

Computing Moving-Average of Stocks in Hadoop HIVE

General Sense of Moving Average: Moving Average is a widely used indicator in technical analysis that helps smooth out price action by filtering out the “noise” from random price fluctuations. A moving average (MA) is a trend-following or lagging indicator because it is based on past prices.

Analyse Hadoop fsimage using the Offline Image Viewer (OIV) Tool

Hadoop fsimage is an “Image” file and its contents cannot be read easily using normal unix file system tools like cat, more etc. At times, it is very important to read the clear text version of the fsimage which holds the meta data of the file system. You can perform NameSpace Analysis, find out health […]