Xiao Qin's Research

A Middleware Approach to Teaching Computer Security (2009 - )

Project 2: Configuring Hadoop

Project Description

In order to achieve proper performance, Hadoop needs to be configured to run on a given set-up. This project will illustrate how to create and edit basic configuration files. It will also show how to manipulate configurations from within java programs. In addition, an introduction to unit testing will confirm the results of these procedures as you go.

Resources

Hadoop: The Definitive Guide. Author: Tom White. O’Reilly Media, 2009. Chapter 5.

System Requirements

1. Ubuntu version 8.04 or later.

2. Sun Java 6

3. SSH installed

4.Apache Hadoop installed

Project Tasks

This project will continue exploring how to run a Hadoop cluster by introducing configuration files. It is encouraged to try different strategies to experience different kinds of results.

1. (XX points) Create a single XML configuration file. You may use the examples shown as a model, but do not copy verbatim.

• (a) Create a property with a value of type string:
  <?xml version="1.0"?> <configuration> <property> <name>color</name> <value>yellow</value> <description>Color</description> </property></configuration>
• (b) Create a property with an integer value: <property> <name>size</name> <value>10</value> <description>Size</description> </property>
• (c) Create a property using variable expansion: <property> <name>color-size</name> <value>${color},${size}</value> <description>Color and size</description> </property>

Try adding some additional properties using other data types such as booleans and floats. Experiment with variable expansion and create different kinds of combinations.

2. (XX points) Opening and reading your configuration file into a java application. Once again, write your own code and do not use the examples verbatim.

(a) Create a configuration file object:
Start by opening up an editor and instantiating a Configuration object and adding your XML
document as a resource.
Configuration conf = new Configuration();
conf.addResource("configuration-1.xml");

(b) Write unit test assertions on your Configuration object:
Use assertions to test the values on your properties
assertThat(conf.get("color"), is("yellow"));
assertThat(conf.getInt("size", 0), is(10));
Run your program and make sure that it passes your tests.

3. (XX points) Combine multiple configuration files.

Write one or more additional configuration files and add them as resources (including your original) to a Configuration object in a new java program. At least one of your additional configuration files must include a property that is in your original. Make sure to write unit tests and confirm that your program passes them.

4. (XX points) Changing properties within the java application:

Write a new new program and add all your configuration files as resources. Then, use the setProperty() method to alter several of your properties in your configuration. Remember to run unit tests to confirm the results of these instructions. Here’s an example.

assertThat(conf.get("size"), is("10"));
System.setProperty("size", "14");
assertThat(conf.get("size"), is("14"));

Submission

You need to submit a ReadMe (including your program(s), command line output, and configuration files) as well as a detailed lab report to describe what you have done and what you have observed. You also need to provide an explanation to the observations that are interesting or surprising.