org.openimaj.hadoop.tools.twitter.token.mode.pointwisemi.count

Class PairMutualInformation

java.lang.Object

org.openimaj.hadoop.mapreduce.stage.Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<OUTPUT_KEY,OUTPUT_VALUE>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,MAP_OUTPUT_KEY,MAP_OUTPUT_VALUE,OUTPUT_KEY,OUTPUT_VALUE>

org.openimaj.hadoop.mapreduce.stage.helper.TextSequenceFileStage<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

org.openimaj.hadoop.mapreduce.stage.helper.TextByteByteStage

org.openimaj.hadoop.tools.twitter.token.mode.pointwisemi.count.PairMutualInformation

public class PairMutualInformation
extends TextByteByteStage

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Field Summary

Fields

Modifier and Type	Field and Description
static String	PAIR_STATS_FILE The location of the statistics file
static String	PAIRMI_DIR The pairMI output directory
static String	TIMEDELTA The time delta between time periods
static String	TIMEPERIOD_COUNT_OUTPUT_ROOT The root directory where timeperiod pair counts will be stored
static String	TIMEPERIOD_OUTPUT_NAME Name of the timeperiod count directory

Constructor Summary

Constructors

Constructor and Description
PairMutualInformation(String[] nonHadoopArgs, long timedelta)

Method Summary

Modifier and Type	Method and Description
Class<? extends org.apache.hadoop.mapreduce.Reducer<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>>	combiner() By default this method returns the IdentityReducer class.
void	finished(org.apache.hadoop.mapreduce.Job job) Called when the stage's job is completed.
static WritablePairEnum	loadStats(org.apache.hadoop.fs.Path outpath) Load the PointwisePMI stats file from an output location (Path: outpath/ PAIR_STATS_FILE
static Map<Long,Long>	loadTimeCounts(org.apache.hadoop.fs.Path pairmiloc) Load the total pairs seen in every time period from the pairmi location provided
Class<PairEmit>	mapper() By default this method returns the IdentityMapper class.
String	outname()
Class<? extends org.apache.hadoop.mapreduce.Reducer<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>>	reducer() By default this method returns the IdentityReducer class.
void	setup(org.apache.hadoop.mapreduce.Job job) Add any final adjustments to the job's config
org.apache.hadoop.mapreduce.Job	stage(org.apache.hadoop.fs.Path[] inputs, org.apache.hadoop.fs.Path output, org.apache.hadoop.conf.Configuration conf)

Methods inherited from class org.openimaj.hadoop.mapreduce.stage.Stage

lzoCompress, setCombinerClass, setMapperClass, setReducerClass

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TIMEDELTA

public static final String TIMEDELTA

The time delta between time periods

See Also:

Constant Field Values

PAIR_STATS_FILE

public static final String PAIR_STATS_FILE

The location of the statistics file

See Also:

Constant Field Values

PAIRMI_DIR

public static final String PAIRMI_DIR

The pairMI output directory

See Also:

Constant Field Values

TIMEPERIOD_COUNT_OUTPUT_ROOT

public static final String TIMEPERIOD_COUNT_OUTPUT_ROOT

The root directory where timeperiod pair counts will be stored

See Also:

Constant Field Values

TIMEPERIOD_OUTPUT_NAME

public static final String TIMEPERIOD_OUTPUT_NAME

Name of the timeperiod count directory

See Also:

Constant Field Values

Constructor Detail

PairMutualInformation

public PairMutualInformation(String[] nonHadoopArgs,
                             long timedelta)

Parameters:

nonHadoopArgs - the arguments for configuration

timedelta -

Method Detail

setup

public void setup(org.apache.hadoop.mapreduce.Job job)
           throws IOException

Description copied from class: Stage

Add any final adjustments to the job's config

Overrides:

setup in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Throws:

IOException

mapper

public Class<PairEmit> mapper()

Description copied from class: Stage

By default this method returns the IdentityMapper class. This mapper outputs the values handed as they are.

Overrides:

mapper in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Returns:

the class of the mapper to use

combiner

public Class<? extends org.apache.hadoop.mapreduce.Reducer<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>> combiner()

Description copied from class: Stage

By default this method returns the IdentityReducer class. This combiner outputs the values handed as they are.

Overrides:

combiner in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Returns:

the class of the reducer to use

stage

public org.apache.hadoop.mapreduce.Job stage(org.apache.hadoop.fs.Path[] inputs,
                                             org.apache.hadoop.fs.Path output,
                                             org.apache.hadoop.conf.Configuration conf)
                                      throws Exception

Overrides:

stage in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Parameters:

inputs - the input paths to be expected

output - the output location

conf - the job configuration

Returns:

the job to be launched in this stage

Throws:

Exception

IOException

reducer

public Class<? extends org.apache.hadoop.mapreduce.Reducer<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>> reducer()

Description copied from class: Stage

By default this method returns the IdentityReducer class. This reducer outputs the values handed as they are.

Overrides:

reducer in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Returns:

the class of the reducer to use

outname

public String outname()

Overrides:

outname in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

Returns:

the name of the output directory of this stage. If the name is null the directory itself is used.

finished

public void finished(org.apache.hadoop.mapreduce.Job job)

Description copied from class: Stage

Called when the stage's job is completed. Might never be called in some cases. For example, when the stagerunner is told specifically not to wait for the job to finish.

Overrides:

finished in class Stage<org.apache.hadoop.mapreduce.lib.input.TextInputFormat,org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat<org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.BytesWritable>

loadStats

public static WritablePairEnum loadStats(org.apache.hadoop.fs.Path outpath)
                                  throws IOException

Load the PointwisePMI stats file from an output location (Path: outpath/ PAIR_STATS_FILE

Parameters:

outpath -

Returns:

a WritablePairEnum instance with the counter values filled

Throws:

IOException

loadTimeCounts

public static Map<Long,Long> loadTimeCounts(org.apache.hadoop.fs.Path pairmiloc)
                                     throws IOException

Load the total pairs seen in every time period from the pairmi location provided

Parameters:

pairmiloc - a directory which contains PAIRMI_DIR/ TIMEPERIOD_OUTPUT_NAME

Returns:

map of a time period to a count

Throws:

IOException