org.openimaj.hadoop.sequencefile

Class SequenceFileUtility<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>

java.lang.Object

org.openimaj.hadoop.sequencefile.SequenceFileUtility<K,V>

Type Parameters:

K - Key type

V - Value type

All Implemented Interfaces:

Iterable<Map.Entry<K,V>>

Direct Known Subclasses:

IntBytesSequenceMemoryUtility, TextBytesSequenceFileUtility, TextBytesSequenceMemoryUtility

public abstract class SequenceFileUtility<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>
extends Object
implements Iterable<Map.Entry<K,V>>

Base class for a utility class that deals with specifically typed sequence files.

Author:

Jonathon Hare (jsh2@ecs.soton.ac.uk), Sina Samangooei (ss@ecs.soton.ac.uk)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SequenceFileUtility.FilenameKeyProvider A class that provides Text keys from the name of a file
static interface	SequenceFileUtility.KeyProvider<K> Interface for objects that can make a key from a path
static class	SequenceFileUtility.MD5UUIDKeyProvider A class that provides Text keys by calculating a UUID from the MD5 of a file
static class	SequenceFileUtility.RelativePathFilenameKeyProvider A class that provides Text keys from the relative path + name of a file

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.io.SequenceFile.CompressionType	compressionType
protected org.apache.hadoop.conf.Configuration	config
protected org.apache.hadoop.fs.FileSystem	fileSystem
protected boolean	isReader
protected org.apache.hadoop.fs.Path	sequenceFilePath
protected String	uuid
protected org.apache.hadoop.io.SequenceFile.Writer	writer

Constructor Summary

Constructors

Constructor and Description
SequenceFileUtility(String uriOrPath, boolean read)
SequenceFileUtility(String uriOrPath, org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
SequenceFileUtility(String uriOrPath, org.apache.hadoop.io.SequenceFile.CompressionType compressionType, Map<String,String> metadata)
SequenceFileUtility(URI uri, boolean read)
SequenceFileUtility(URI uri, org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
SequenceFileUtility(URI uri, org.apache.hadoop.io.SequenceFile.CompressionType compressionType, Map<String,String> metadata)

Method Summary

Modifier and Type	Method and Description
void	appendData(K key, V value) Append data to a sequence file.
void	appendFile(K key, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p) Append data read from a file to the sequence file.
Map<org.apache.hadoop.fs.Path,K>	appendFiles(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, boolean recurse, org.apache.hadoop.fs.PathFilter pathFilter, SequenceFileUtility.KeyProvider<K> keyProvider) Append files to a sequenceFile.
void	close() Close the underlying writer.
static URI	convertToURI(String uriOrPath) Converts a string representing a file or uri to a uri object.
void	exportData(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dirPath) Extracts file to a directory.
void	exportData(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dirPath, NamingStrategy naming, ExtractionState extrState, boolean addExtension, long offset) Extracts file to a directory.
void	exportData(NamingStrategy np, ExtractionState nps, long offset, KeyValueDump<K,V> dump)
void	exportData(String uriOrPath) Extracts file to a directory.
void	exportData(String uriOrPath, NamingStrategy naming, ExtractionState extrState, boolean addExtension, long offset) Extracts file to a directory.
void	exportDataToZip(String uriOrPath, NamingStrategy naming, ExtractionState state, boolean addExtension, long offset) Extracts file to a directory.
void	exportDataToZip(ZipOutputStream zos, NamingStrategy naming, ExtractionState extrState, boolean addExtension, long offset) Extracts file to a zip file.
void	extract(List<RecordInformationExtractor> extractors, PrintStream stream, String delim) Go through a sequence file, applying each RecordInformationExtractor to each key, printing out the results in order to the provided PrintStream
V	find(K queryKey) Search for the record identified by queryKey.
V	find(K queryKey, long offset) Search for the record identified by queryKey.
boolean	findAndExport(K key, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path dirPath, long offset) Find a record and write the value to a file.
boolean	findAndExport(K key, String uriOrPath, long offset) Find a record and write the value to a file.
Class<? extends org.apache.hadoop.io.compress.CompressionCodec>	getCompressionCodecClass()
org.apache.hadoop.io.SequenceFile.CompressionType	getCompressionType()
static org.apache.hadoop.fs.Path[]	getFilePaths(String[] uriOrPaths, String filenamePrefix) Get a list of all the sequence files (with a given name prefix) in the set of input paths.
static org.apache.hadoop.fs.Path[]	getFilePaths(String[] uriOrPaths, String subdir, String filenamePrefix) Get a list of all the sequence files (with a given name prefix) in the set of input paths.
static org.apache.hadoop.fs.Path[]	getFilePaths(String uriOrPath, String filenamePrefix) Get a list of all the sequence files (with a given name prefix) in a directory.
static URI[]	getFiles(String uriOrPath, String filenamePrefix) Get a list of all the sequence files (with a given name prefix) in a directory.
static URI[]	getFilesRegex(String uriOrPath, String regex) Get a list of all the sequence files whose names match the given regular expression in a directory.
org.apache.hadoop.fs.FileSystem	getFileSystem(URI uri) Get the filesystem associated with a uri.
static org.apache.hadoop.fs.FileSystem	getFileSystem(URI uri, org.apache.hadoop.conf.Configuration config) Get the filesystem associated with a uri.
Map<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>	getMetadata() Return the metadata map.
long	getNumberRecords() Get number of records in file.
org.apache.hadoop.fs.Path	getPath(URI uri) Get a path from a uri.
static URI[]	getReducerFiles(String uriOrPath) Get a list of all the reducer outputs in a directory.
org.apache.hadoop.fs.Path	getSequenceFilePath()
String	getUUID() Get the UUID of this file
Iterator<Map.Entry<K,V>>	iterator()
List<K>	listKeys() Return a list of the keys in the sequence file.
Map<K,Long>	listKeysAndOffsets() Return a list of the keys in the sequence file.
static String	md5sum(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path p) Get the MD5 sum of a file
static ZipOutputStream	openZipOutputStream(String uriOrPath)
protected abstract void	printFile(V value)
protected abstract V	readFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path)
protected abstract void	writeFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path path, V value)
void	writePathMap(Map<org.apache.hadoop.fs.Path,K> map)
protected abstract void	writeZipData(ZipOutputStream zos, V value)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Field Detail

config

protected org.apache.hadoop.conf.Configuration config

fileSystem

protected org.apache.hadoop.fs.FileSystem fileSystem

sequenceFilePath

protected org.apache.hadoop.fs.Path sequenceFilePath

writer

protected org.apache.hadoop.io.SequenceFile.Writer writer

compressionType

protected org.apache.hadoop.io.SequenceFile.CompressionType compressionType

isReader

protected boolean isReader

uuid

protected String uuid

Constructor Detail

SequenceFileUtility

public SequenceFileUtility(String uriOrPath,
                           boolean read)
                    throws IOException

Throws:

IOException

SequenceFileUtility

public SequenceFileUtility(URI uri,
                           boolean read)
                    throws IOException

Throws:

IOException

SequenceFileUtility

public SequenceFileUtility(String uriOrPath,
                           org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
                    throws IOException

Throws:

IOException

SequenceFileUtility

public SequenceFileUtility(URI uri,
                           org.apache.hadoop.io.SequenceFile.CompressionType compressionType)
                    throws IOException

Throws:

IOException

SequenceFileUtility

public SequenceFileUtility(String uriOrPath,
                           org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
                           Map<String,String> metadata)
                    throws IOException

Throws:

IOException

SequenceFileUtility

public SequenceFileUtility(URI uri,
                           org.apache.hadoop.io.SequenceFile.CompressionType compressionType,
                           Map<String,String> metadata)
                    throws IOException

Throws:

IOException

Method Detail

getReducerFiles

public static URI[] getReducerFiles(String uriOrPath)
                             throws IOException

Get a list of all the reducer outputs in a directory. If the given path/uri is not a directory, then it is assumed that it is a SequenceFile and returned directly.

Parameters:

uriOrPath - the path or uri

Returns:

the reducer outputs

Throws:

IOException

getFiles

public static URI[] getFiles(String uriOrPath,
                             String filenamePrefix)
                      throws IOException

Get a list of all the sequence files (with a given name prefix) in a directory. If the given uri is not a directory, then it is assumed that it is a SequenceFile and returned directly.

Parameters:

uriOrPath - the path or uri

filenamePrefix - the prefix of the file name

Returns:

the matching files

Throws:

IOException

getFilePaths

public static org.apache.hadoop.fs.Path[] getFilePaths(String[] uriOrPaths,
                                                       String filenamePrefix)
                                                throws IOException

Get a list of all the sequence files (with a given name prefix) in the set of input paths. If a given uri is not a directory, then it is assumed that it is a SequenceFile and returned directly.

Parameters:

uriOrPaths - the paths or uris

filenamePrefix - the prefix of the file name

Returns:

the list of sequence files

Throws:

IOException

getFilePaths

public static org.apache.hadoop.fs.Path[] getFilePaths(String[] uriOrPaths,
                                                       String subdir,
                                                       String filenamePrefix)
                                                throws IOException

Get a list of all the sequence files (with a given name prefix) in the set of input paths.

Optionally a subdirectory can be provided; if provided the subdirectory is appended to each path (i.e. PATH/subdirectory).

If the given uri is not a directory, then it is assumed that it is a single SequenceFile and returned directly.

Parameters:

uriOrPaths - the URI or path to the directory/file

subdir - the optional subdirectory (may be null)

filenamePrefix - the prefix of the file name

Returns:

the list of sequence files

Throws:

IOException

getFilePaths

public static org.apache.hadoop.fs.Path[] getFilePaths(String uriOrPath,
                                                       String filenamePrefix)
                                                throws IOException

Get a list of all the sequence files (with a given name prefix) in a directory. If the given uri is not a directory, then it is assumed that it is a SequenceFile and returned directly.

Parameters:

uriOrPath - the path or uri

filenamePrefix - the prefix of the file name

Returns:

the list of sequence files

Throws:

IOException

getFilesRegex

public static URI[] getFilesRegex(String uriOrPath,
                                  String regex)
                           throws IOException

Get a list of all the sequence files whose names match the given regular expression in a directory. If the given uri is not a directory, then it is assumed that it is a SequenceFile and returned directly.

Parameters:

uriOrPath - the path or uri

regex - the regular expression to match

Returns:

a list of files

Throws:

IOException

listKeysAndOffsets

public Map<K,Long> listKeysAndOffsets()

Return a list of the keys in the sequence file. Read mode only.

Returns:

keys.

extract

public void extract(List<RecordInformationExtractor> extractors,
                    PrintStream stream,
                    String delim)

Go through a sequence file, applying each RecordInformationExtractor to each key, printing out the results in order to the provided PrintStream

Parameters:

extractors - the RecordInformationExtractors to apply

stream - the stream to write to

delim -

convertToURI

public static URI convertToURI(String uriOrPath)

Converts a string representing a file or uri to a uri object.

Parameters:

uriOrPath - uri or path to convert

Returns:

uri

getUUID

public String getUUID()

Get the UUID of this file

Returns:

UUID

getMetadata

public Map<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text> getMetadata()

Return the metadata map. Read mode only.

Returns:

metadata

listKeys

public List<K> listKeys()

Return a list of the keys in the sequence file. Read mode only.

Returns:

keys.

exportData

public void exportData(String uriOrPath)
                throws IOException

Extracts file to a directory. Read mode only.

Parameters:

uriOrPath - path or uri to extract to.

Throws:

IOException

exportData

public void exportData(String uriOrPath,
                       NamingStrategy naming,
                       ExtractionState extrState,
                       boolean addExtension,
                       long offset)
                throws IOException

Extracts file to a directory. Read mode only.

Parameters:

uriOrPath - path or uri to extract to.

naming - the naming strategy

extrState - the extraction state

addExtension - if true, then file extensions are added to each record automatically

offset - offset from which to start. Can be used to reduce number of files extracted.

Throws:

IOException

openZipOutputStream

public static ZipOutputStream openZipOutputStream(String uriOrPath)
                                           throws IOException

Throws:

IOException

exportDataToZip

public void exportDataToZip(String uriOrPath,
                            NamingStrategy naming,
                            ExtractionState state,
                            boolean addExtension,
                            long offset)
                     throws IOException

Extracts file to a directory. Read mode only.

Parameters:

uriOrPath - path or uri to extract to.

naming - the naming strategy

state - the extraction state

addExtension - if true, then file extensions are added to each record automatically

offset - offset from which to start. Can be used to reduce number of files extracted.

Throws:

IOException

exportDataToZip

public void exportDataToZip(ZipOutputStream zos,
                            NamingStrategy naming,
                            ExtractionState extrState,
                            boolean addExtension,
                            long offset)
                     throws IOException

Extracts file to a zip file. Read mode only.

Parameters:

zos - The ZipOutputStream to write to

naming - The naming strategy

extrState - The extration state

addExtension - if true, then file extensions are added to each record automatically

offset - offset from which to start. Can be used to reduce number of files extracted.

Throws:

IOException

exportData

public void exportData(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path dirPath)

Extracts file to a directory. Read mode only.

Parameters:

fs - filesystem of output file

dirPath - path to extract to

exportData

public void exportData(org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path dirPath,
                       NamingStrategy naming,
                       ExtractionState extrState,
                       boolean addExtension,
                       long offset)

Extracts file to a directory. Read mode only.

Parameters:

fs - filesystem of output file

dirPath - path to extract to

naming - the naming strategy

extrState - the extraction state

addExtension - if true, then file extensions are added to each record automatically

offset - offset from which to start. Can be used to reduce number of files extracted.

exportData

public void exportData(NamingStrategy np,
                       ExtractionState nps,
                       long offset,
                       KeyValueDump<K,V> dump)

close

public void close()
           throws IOException

Close the underlying writer. Does nothing in read mode.

Throws:

IOException

getNumberRecords

public long getNumberRecords()

Get number of records in file. Read mode only.

Returns:

number of records

getCompressionCodecClass

public Class<? extends org.apache.hadoop.io.compress.CompressionCodec> getCompressionCodecClass()

Returns:

the compression codec in use for this file.

getCompressionType

public org.apache.hadoop.io.SequenceFile.CompressionType getCompressionType()

Returns:

he compression mode used for this sequence file.

getFileSystem

public org.apache.hadoop.fs.FileSystem getFileSystem(URI uri)
                                              throws IOException

Get the filesystem associated with a uri.

Parameters:

uri -

Returns:

the filesystem

Throws:

IOException

getFileSystem

public static org.apache.hadoop.fs.FileSystem getFileSystem(URI uri,
                                                            org.apache.hadoop.conf.Configuration config)
                                                     throws IOException

Get the filesystem associated with a uri.

Parameters:

uri -

config -

Returns:

the filesystem

Throws:

IOException

getPath

public org.apache.hadoop.fs.Path getPath(URI uri)
                                  throws IOException

Get a path from a uri.

Parameters:

uri -

Returns:

the path

Throws:

IOException

md5sum

public static String md5sum(org.apache.hadoop.fs.FileSystem fs,
                            org.apache.hadoop.fs.Path p)

Get the MD5 sum of a file

Parameters:

fs -

p -

Returns:

the md5sum

readFile

protected abstract V readFile(org.apache.hadoop.fs.FileSystem fs,
                              org.apache.hadoop.fs.Path path)
                       throws IOException

Throws:

IOException

writeFile

protected abstract void writeFile(org.apache.hadoop.fs.FileSystem fs,
                                  org.apache.hadoop.fs.Path path,
                                  V value)
                           throws IOException

Throws:

IOException

writeZipData

protected abstract void writeZipData(ZipOutputStream zos,
                                     V value)
                              throws IOException

Throws:

IOException

printFile

protected abstract void printFile(V value)
                           throws IOException

Throws:

IOException

appendFile

public void appendFile(K key,
                       org.apache.hadoop.fs.FileSystem fs,
                       org.apache.hadoop.fs.Path p)
                throws IOException

Append data read from a file to the sequence file.

Parameters:

key -

fs -

p -

Throws:

IOException

appendData

public void appendData(K key,
                       V value)
                throws IOException

Append data to a sequence file.

Parameters:

key -

value -

Throws:

IOException

appendFiles

public Map<org.apache.hadoop.fs.Path,K> appendFiles(org.apache.hadoop.fs.FileSystem fs,
                                                    org.apache.hadoop.fs.Path path,
                                                    boolean recurse,
                                                    org.apache.hadoop.fs.PathFilter pathFilter,
                                                    SequenceFileUtility.KeyProvider<K> keyProvider)
                                             throws IOException

Append files to a sequenceFile.

Parameters:

fs - The filesystem of the files being added.

path - The path of the file(s) being added.

recurse - If true, then subdirectories are also searched

pathFilter - Filter for omitting files. Can be null.

keyProvider - Object that can return a key for a given file.

Returns:

Paths and their respective keys for files that were added.

Throws:

IOException

writePathMap

public void writePathMap(Map<org.apache.hadoop.fs.Path,K> map)
                  throws IOException

Throws:

IOException

find

public V find(K queryKey,
              long offset)

Search for the record identified by queryKey.

Parameters:

queryKey - the key.

offset - the offset from which to commence search

Returns:

the found value, or null.

find

public V find(K queryKey)

Search for the record identified by queryKey. Uses a linear search from the beginning of the file.

Parameters:

queryKey -

Returns:

the found value, or null.

findAndExport

public boolean findAndExport(K key,
                             String uriOrPath,
                             long offset)
                      throws IOException

Find a record and write the value to a file.

Parameters:

key -

uriOrPath -

offset -

Returns:

false if record not found, true otherwise.

Throws:

IOException

findAndExport

public boolean findAndExport(K key,
                             org.apache.hadoop.fs.FileSystem fs,
                             org.apache.hadoop.fs.Path dirPath,
                             long offset)
                      throws IOException

Find a record and write the value to a file.

Parameters:

key -

fs -

dirPath -

offset -

Returns:

false if record not found, true otherwise.

Throws:

IOException

getSequenceFilePath

public org.apache.hadoop.fs.Path getSequenceFilePath()

iterator

public Iterator<Map.Entry<K,V>> iterator()

Specified by:

iterator in interface Iterable<Map.Entry<K extends org.apache.hadoop.io.Writable,V extends org.apache.hadoop.io.Writable>>