org.openimaj.hadoop.tools.twitter.token.mode.dfidf

Class CountWordsAcrossTimeperiod

java.lang.Object

org.openimaj.hadoop.mapreduce.stage.StageProvider

org.openimaj.hadoop.tools.twitter.token.mode.dfidf.CountWordsAcrossTimeperiod

public class CountWordsAcrossTimeperiod
extends StageProvider

A Stage provider wrapping the functionality of CountWordsAcrossTimeperiod.Map and CountWordsAcrossTimeperiod.Reduce. The Map expects times as keys and maps of word counts as input. The mapper emits words as keys with pairs of the timeperiods and counts within that time period This along with the TimeIndex generated by the CountTweetsInTimeperiod is used to construct a DFIDF per word per time period. One interpretation of the DFIDF score is a weighting based on counts up to a particular time period. This functionality is encoded in the CountWordsAcrossTimeperiod.NonCombinedTimesReducer Another interpretation is that the DFIDF score weights words based on word occurences at the end of some period of time, i.e. the count at some END

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CountWordsAcrossTimeperiod.Map function(timePeriodLength) map input: ,,,...> map output: [ word: , word: , ...
static class	CountWordsAcrossTimeperiod.NonCombinedTimesReducer reduce input: , ,...
static class	CountWordsAcrossTimeperiod.Reduce reduce input: , ,...

Field Summary

Fields

Modifier and Type	Field and Description
static String	ARGS_KEY arg key
static String	WORDCOUNT_DIR where the intermediat word counts should be stored

Constructor Summary

Constructors

Constructor and Description
CountWordsAcrossTimeperiod(String[] nonHadoopArgs)
CountWordsAcrossTimeperiod(String[] nonHadoopArgs, boolean combinedTimes)

Method Summary

Modifier and Type	Method and Description
SimpleSequenceFileStage<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>	stage()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ARGS_KEY

public static final String ARGS_KEY

arg key

See Also:

Constant Field Values

WORDCOUNT_DIR

public static final String WORDCOUNT_DIR

where the intermediat word counts should be stored

See Also:

Constant Field Values

Constructor Detail

CountWordsAcrossTimeperiod

public CountWordsAcrossTimeperiod(String[] nonHadoopArgs)

Parameters:

nonHadoopArgs -

CountWordsAcrossTimeperiod

public CountWordsAcrossTimeperiod(String[] nonHadoopArgs,
                                  boolean combinedTimes)

Parameters:

nonHadoopArgs -

combinedTimes - whether the mapper expects times entries with values for each word. i.e. combined times

Method Detail

stage

public SimpleSequenceFileStage<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable> stage()

Specified by:

stage in class StageProvider

Returns:

a job stage