org.openimaj.hadoop.mapreduce

Class MultiStagedJob

java.lang.Object

org.openimaj.hadoop.mapreduce.MultiStagedJob

public class MultiStagedJob
extends Object

A set of hadoop jobs done in series. The final output (directory) of the nth job is used as the map input path of the (n+1)th job.

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Constructor Summary

Constructors

Constructor and Description
MultiStagedJob(org.apache.hadoop.fs.Path[] initialInput, org.apache.hadoop.fs.Path root, boolean removePreliminary, String[] args) Start a multistaged job specification.
MultiStagedJob(org.apache.hadoop.fs.Path[] initialInput, org.apache.hadoop.fs.Path root, String[] args) Start a multistaged job specification.
MultiStagedJob(String inpath, String outpath, String[] args) Conveniance function.

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.fs.Path[]	getStagePaths(String completedJobId)
void	queueStage(Stage<?,?,?,?,?,?,?,?> s) Add a stage to the end of the queue of stages.
void	removeIntermediate(boolean b)
org.apache.hadoop.fs.Path	runAll() Run all the staged jobs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultiStagedJob

public MultiStagedJob(org.apache.hadoop.fs.Path[] initialInput,
                      org.apache.hadoop.fs.Path root,
                      String[] args)

Start a multistaged job specification. The root path holds the final and all preliminary steps

Parameters:

initialInput - the initial input given to the first stage

root - the final output location

args - the arguments with which to start the job

MultiStagedJob

public MultiStagedJob(org.apache.hadoop.fs.Path[] initialInput,
                      org.apache.hadoop.fs.Path root,
                      boolean removePreliminary,
                      String[] args)

Start a multistaged job specification. The root path holds the final. and all preliminary steps

Parameters:

initialInput - the initial input given to the first stage

removePreliminary - whether all intermediate steps should be removed

root - the final output location

args - the arguments with which to start the job

MultiStagedJob

public MultiStagedJob(String inpath,
                      String outpath,
                      String[] args)
               throws IOException

Conveniance function. Finds the input paths using #SequenceFileUtility.getFilePaths and uses Path(outpath)

Parameters:

inpath -

outpath -

args - the arguments with which to start the job

Throws:

IOException

Method Detail

queueStage

public void queueStage(Stage<?,?,?,?,?,?,?,?> s)

Add a stage to the end of the queue of stages.

Parameters:

s -

runAll

public org.apache.hadoop.fs.Path runAll()
                                 throws Exception

Run all the staged jobs. The input/output of each job is at follows: initial -> stage1 -> output1 output1 -> stage2 -> output2 ... output(N-1) -> stageN -> final for each output, the directory created is scanned for part files matching the regex "part.*"

Returns:

The path to the final output for convenience (##base##/final by convention)

Throws:

Exception

getStagePaths

public org.apache.hadoop.fs.Path[] getStagePaths(String completedJobId)

Parameters:

completedJobId -

Returns:

the path to the output of the completed job

removeIntermediate

public void removeIntermediate(boolean b)

Parameters:

b - if true all but the final output and input for this multi staged job are true