org.openimaj.text.nlp

Class TweetTokeniser

java.lang.Object

org.openimaj.text.nlp.TweetTokeniser

All Implemented Interfaces:

Iterable<gov.sandia.cognition.text.token.Token>

public class TweetTokeniser
extends Object
implements Iterable<gov.sandia.cognition.text.token.Token>

A tokeniser built to work with short text, like that found in twitter. Protects various elements of the text with an assumption that if the user made the mark, it was an important mark that carries meaning because of the relatively high premium of each key stroke. Based on the twokenise by Brendan O'Connor

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Constructor Summary

Constructors

Constructor and Description
TweetTokeniser(String s)

Method Summary

Modifier and Type	Method and Description
List<String>	getProtectedStringTokens()
List<String>	getStringTokens()
List<gov.sandia.cognition.text.token.Token>	getTokens()
List<String>	getUnprotectedStringTokens()
static boolean	isValid(Locale locale) Check whether this locale is supported by this tokeniser.
static boolean	isValid(String locale) Check whether this locale (specified by the two letter country code, Locale) is supported by this tokeniser.
Iterator<gov.sandia.cognition.text.token.Token>	iterator()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

TweetTokeniser

public TweetTokeniser(String s)
               throws UnsupportedEncodingException,
                      TweetTokeniserException

Parameters:

s - Tokenise this string

Throws:

UnsupportedEncodingException

TweetTokeniserException

Method Detail

isValid

public static boolean isValid(Locale locale)

Check whether this locale is supported by this tokeniser. The unsupported languages are those which don't need space characters to delimit words, namely the CJK languages.

Parameters:

locale -

Returns:

true if the local is supported

isValid

public static boolean isValid(String locale)

Check whether this locale (specified by the two letter country code, Locale) is supported by this tokeniser. The unsupported languages are those which don't need space characters to delimit words, namely the CJK languages.

Parameters:

locale -

Returns:

true if the local is supported

iterator

public Iterator<gov.sandia.cognition.text.token.Token> iterator()

Specified by:

iterator in interface Iterable<gov.sandia.cognition.text.token.Token>

getTokens

public List<gov.sandia.cognition.text.token.Token> getTokens()

Returns:

all the tokens detected (as Token instances)

getStringTokens

public List<String> getStringTokens()

Returns:

return all tokens as a List of String

getProtectedStringTokens

public List<String> getProtectedStringTokens()

Returns:

return all tokens protected by the Twokenizer regex

getUnprotectedStringTokens

public List<String> getUnprotectedStringTokens()

Returns:

return all the tokens not protected by the Twokenizer regex