org.openimaj.hadoop.tools.twitter.token.outputmode.sparsecsv

Class WordIndex

java.lang.Object

org.openimaj.hadoop.mapreduce.stage.StageAppender

org.openimaj.hadoop.tools.twitter.token.outputmode.sparsecsv.WordIndex

public class WordIndex
extends StageAppender

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WordIndex.Map Emits each word with the total number of times the word was seen
static class	WordIndex.Reduce Writes each word,count

Field Summary

Fields

Modifier and Type	Field and Description
protected static String	WORDCOUNT_THRESH
protected static String	WORDCOUNT_TIMETHRESH
protected static String	WORDCOUNT_TOPN

Constructor Summary

Constructors

Constructor and Description
WordIndex()
WordIndex(int wordCountThreshold, int topNWords)
WordIndex(int wordCountThreshold, int wordTimeThreshold, int topNWords)

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)
static LinkedHashMap<String,IndependentPair<Long,Long>>	readWordCountLines(String path)
static LinkedHashMap<String,IndependentPair<Long,Long>>	readWordCountLines(String path, String ext) from a report output path get the words
void	stage(MultiStagedJob mjob)
static void	writeToMatlab(String path) Write a CSV wordIndex to a MLCell writen to a .mat data file

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

WORDCOUNT_THRESH

protected static final String WORDCOUNT_THRESH

See Also:

Constant Field Values

WORDCOUNT_TOPN

protected static final String WORDCOUNT_TOPN

See Also:

Constant Field Values

WORDCOUNT_TIMETHRESH

protected static final String WORDCOUNT_TIMETHRESH

See Also:

Constant Field Values

Constructor Detail

WordIndex

public WordIndex(int wordCountThreshold,
                 int topNWords)

WordIndex

public WordIndex(int wordCountThreshold,
                 int wordTimeThreshold,
                 int topNWords)

WordIndex

public WordIndex()

Method Detail

readWordCountLines

public static LinkedHashMap<String,IndependentPair<Long,Long>> readWordCountLines(String path)
                                                                           throws IOException

Parameters:

path -

Returns:

map of words to counts and index

Throws:

IOException

readWordCountLines

public static LinkedHashMap<String,IndependentPair<Long,Long>> readWordCountLines(String path,
                                                                                  String ext)
                                                                           throws IOException

from a report output path get the words

Parameters:

path - report output path

ext - where the words are in the path

Returns:

map of words to counts and index

Throws:

IOException

stage

public void stage(MultiStagedJob mjob)

Overrides:

stage in class StageAppender

Parameters:

mjob - what should i add myself to?

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException

writeToMatlab

public static void writeToMatlab(String path)
                          throws IOException

Write a CSV wordIndex to a MLCell writen to a .mat data file

Parameters:

path -

Throws:

IOException