org.openimaj.hadoop.tools.twitter.token.mode.dfidf

Class CountTweetsInTimeperiod

java.lang.Object

org.openimaj.hadoop.mapreduce.stage.StageProvider

org.openimaj.hadoop.tools.twitter.token.mode.dfidf.CountTweetsInTimeperiod

public class CountTweetsInTimeperiod
extends StageProvider

A mapper/reducer whose purpose is to do the following: function(timePeriodLength) So a word in a tweet can happen in the time period between t - 1 and t. First task: map input: tweetstatus # json twitter status with JSONPath to words map output: , -1: > reduce input: ,...,]> reduce output: ,,,...>

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	CountTweetsInTimeperiod.InMemoryCombiningReducer reduce input: ,...,]> reduce output: ,,,...>
static class	CountTweetsInTimeperiod.Map map input: tweetstatus # json twitter status with JSONPath to words map output: , -1: >
static class	CountTweetsInTimeperiod.TimeIndexReducer Identical to the IdentityReducer but constructs a time index found in TIMEINDEX_FILE

Field Summary

Fields

Modifier and Type	Field and Description
static String	GLOBAL_STATS_FILE where to find the global stats file
static String	TIMECOUNT_DIR option for the timecount dir location
static String	TIMEINDEX_FILE A time index holding tweet totals and cumulative totals for each time period
static String	TIMEINDEX_LOCATION_PROP A time index holding tweet totals and cumulative totals for each time period

Constructor Summary

Constructors

Constructor and Description
CountTweetsInTimeperiod(String[] nonHadoopArgs, boolean inMemoryCombine, long timedelta)
CountTweetsInTimeperiod(String[] nonHadoopArgs, long timedelta)

Method Summary

Modifier and Type	Method and Description
static org.apache.hadoop.fs.Path	constructIndexPath(org.apache.hadoop.fs.Path outpath)
static TimeFrequencyHolder	readTimeIndex(org.apache.hadoop.fs.Path indexOut) Read a TimeFrequencyHolder from a Path.
TextLongByteStage	stage()
static void	writeTimeIndex(TimeFrequencyHolder timeMap, org.apache.hadoop.fs.Path indexOut) Write a timeindex to a Path

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

TIMECOUNT_DIR

public static final String TIMECOUNT_DIR

option for the timecount dir location

See Also:

Constant Field Values

TIMEINDEX_FILE

public static final String TIMEINDEX_FILE

A time index holding tweet totals and cumulative totals for each time period

See Also:

Constant Field Values

GLOBAL_STATS_FILE

public static final String GLOBAL_STATS_FILE

where to find the global stats file

See Also:

Constant Field Values

TIMEINDEX_LOCATION_PROP

public static final String TIMEINDEX_LOCATION_PROP

A time index holding tweet totals and cumulative totals for each time period

See Also:

Constant Field Values

Constructor Detail

CountTweetsInTimeperiod

public CountTweetsInTimeperiod(String[] nonHadoopArgs,
                               long timedelta)

Parameters:

nonHadoopArgs - to be sent to the stage

timedelta - the time delta between which to quantise time periods

CountTweetsInTimeperiod

public CountTweetsInTimeperiod(String[] nonHadoopArgs,
                               boolean inMemoryCombine,
                               long timedelta)

Parameters:

nonHadoopArgs - to be sent to the stage

inMemoryCombine - whether an in memory combination of word counts should be performed

timedelta - the time delta between which to quantise time periods

Method Detail

stage

public TextLongByteStage stage()

Specified by:

stage in class StageProvider

Returns:

a job stage

writeTimeIndex

public static void writeTimeIndex(TimeFrequencyHolder timeMap,
                                  org.apache.hadoop.fs.Path indexOut)
                           throws IOException

Write a timeindex to a Path

Parameters:

timeMap -

indexOut -

Throws:

IOException

readTimeIndex

public static TimeFrequencyHolder readTimeIndex(org.apache.hadoop.fs.Path indexOut)
                                         throws IOException

Read a TimeFrequencyHolder from a Path. Path is assumed to be a directory containing many TimeFrequencyHolder instances.

Parameters:

indexOut -

Returns:

a new TimeFrequencyHolder

Throws:

IOException

constructIndexPath

public static org.apache.hadoop.fs.Path constructIndexPath(org.apache.hadoop.fs.Path outpath)

Parameters:

outpath -

Returns:

the index location if it exists