org.openimaj.hadoop.tools.twitter.token.outputmode.jacard

Class TimeWordJacardIndex

java.lang.Object

org.openimaj.hadoop.mapreduce.stage.Stage<org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<INPUT_KEY,INPUT_VALUE>,org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<OUTPUT_KEY,OUTPUT_VALUE>,INPUT_KEY,INPUT_VALUE,MAP_OUTPUT_KEY,MAP_OUTPUT_VALUE,OUTPUT_KEY,OUTPUT_VALUE>

org.openimaj.hadoop.mapreduce.stage.helper.SequenceFileTextStage<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>

org.openimaj.hadoop.tools.twitter.token.outputmode.jacard.TimeWordJacardIndex

public class TimeWordJacardIndex
extends SequenceFileTextStage<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>

Count word instances (not occurences) across times. Allows for investigation of how the vocabulary has changed over time.

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Constructor Summary

Constructors

Constructor and Description
TimeWordJacardIndex()

Method Summary

Modifier and Type	Method and Description
Class<? extends org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>>	mapper() By default this method returns the IdentityMapper class.
String	outname()
Class<? extends org.apache.hadoop.mapreduce.Reducer<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>>	reducer() By default this method returns the IdentityReducer class.
void	setup(org.apache.hadoop.mapreduce.Job job) Add any final adjustments to the job's config

Methods inherited from class org.openimaj.hadoop.mapreduce.stage.Stage

combiner, finished, lzoCompress, setCombinerClass, setMapperClass, setReducerClass, stage

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TimeWordJacardIndex

public TimeWordJacardIndex()

Method Detail

setup

public void setup(org.apache.hadoop.mapreduce.Job job)

Description copied from class: Stage

Add any final adjustments to the job's config

Overrides:

setup in class Stage<org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>

mapper

public Class<? extends org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text>> mapper()

Description copied from class: Stage

By default this method returns the IdentityMapper class. This mapper outputs the values handed as they are.

Overrides:

mapper in class Stage<org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>

Returns:

the class of the mapper to use

reducer

public Class<? extends org.apache.hadoop.mapreduce.Reducer<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>> reducer()

Description copied from class: Stage

By default this method returns the IdentityReducer class. This reducer outputs the values handed as they are.

Overrides:

reducer in class Stage<org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>

Returns:

the class of the reducer to use

outname

public String outname()

Overrides:

outname in class Stage<org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat<org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>,org.apache.hadoop.mapreduce.lib.output.TextOutputFormat<org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.NullWritable,org.apache.hadoop.io.Text>

Returns:

the name of the output directory of this stage. If the name is null the directory itself is used.