org.openimaj.hadoop.tools

Class HadoopToolsUtil

java.lang.Object

org.openimaj.hadoop.tools.HadoopToolsUtil

public class HadoopToolsUtil
extends Object

Tools for dealing with #InOutTool instances that are hdfs files

Author:

Sina Samangooei (ss@ecs.soton.ac.uk)

Constructor Summary

Constructors

Constructor and Description
HadoopToolsUtil()

Method Summary

Modifier and Type	Method and Description
static String[]	decodeArgs(String[] args) A horrible hack to deal with hadoop's horrible hack when setting arrays of strings as configs
static String[]	encodeArgs(String[] args) A horrible hack to deal with hadoop's horrible hack when setting arrays of strings as configs
static boolean	fileExists(String path) Use hadoop filesystem to check if the given path exists
static org.apache.hadoop.fs.FileSystem	getFileSystem(org.apache.hadoop.fs.Path p) Get the FileSystem corresponding to a Path.
static org.apache.hadoop.fs.FileSystem	getFileSystem(URI uri)
static org.apache.hadoop.fs.Path[]	getInputPaths(InOutToolOptions options) Get the input paths from an InOutToolOptions.
static org.apache.hadoop.fs.Path[]	getInputPaths(String path) Get the input paths from a String.
static org.apache.hadoop.fs.Path[]	getInputPaths(String[] paths)
static org.apache.hadoop.fs.Path[]	getInputPaths(String[] paths, String subdir) All the files starting with "part" in the paths which look like: "paths[i]/subdir
static org.apache.hadoop.fs.Path	getOutputPath(InOutToolOptions options) Get the output path from an InOutToolOptions.
static org.apache.hadoop.fs.Path	getOutputPath(String path) Get the output path from a String.
static String[]	readlines(String p) Read a whole hadoop file into a string.
static void	removeFile(String f) Delete a file
static void	validateInput(InOutToolOptions tool)
static void	validateOutput(InOutToolOptions tool)
static void	validateOutput(String outpath, boolean replace)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopToolsUtil

public HadoopToolsUtil()

Method Detail

validateOutput

public static void validateOutput(InOutToolOptions tool)
                           throws org.kohsuke.args4j.CmdLineException

Parameters:

tool - options to get data from

Throws:

org.kohsuke.args4j.CmdLineException

validateOutput

public static void validateOutput(String outpath,
                                  boolean replace)
                           throws org.kohsuke.args4j.CmdLineException

Parameters:

outpath - The desired output

replace - whether the existing outputs should be removed

Throws:

org.kohsuke.args4j.CmdLineException

getFileSystem

public static org.apache.hadoop.fs.FileSystem getFileSystem(URI uri)
                                                     throws IOException

Parameters:

uri -

Returns:

the file system of a given path (HDFS or Local usually)

Throws:

IOException

getFileSystem

public static org.apache.hadoop.fs.FileSystem getFileSystem(org.apache.hadoop.fs.Path p)
                                                     throws IOException

Get the FileSystem corresponding to a Path.

Parameters:

p - the path.

Returns:

the filesystem

Throws:

IOException

validateInput

public static void validateInput(InOutToolOptions tool)
                          throws org.kohsuke.args4j.CmdLineException

Parameters:

tool -

Throws:

org.kohsuke.args4j.CmdLineException

removeFile

public static void removeFile(String f)
                       throws IOException

Delete a file

Parameters:

f - the file to delete

Throws:

IOException

getOutputPath

public static org.apache.hadoop.fs.Path getOutputPath(InOutToolOptions options)

Get the output path from an InOutToolOptions.

Parameters:

options - the InOutToolOptions.

Returns:

the output path.

getOutputPath

public static org.apache.hadoop.fs.Path getOutputPath(String path)

Get the output path from a String.

Parameters:

path - the path string

Returns:

the path

getInputPaths

public static org.apache.hadoop.fs.Path[] getInputPaths(InOutToolOptions options)
                                                 throws IOException

Get the input paths from an InOutToolOptions. This will resolve the input path and return either a Path object representing the string or, if the path string is a directory, a list of Paths representing all the "part" files.

Parameters:

options - the InOutToolOptions.

Returns:

the input path

Throws:

IOException

getInputPaths

public static org.apache.hadoop.fs.Path[] getInputPaths(String path)
                                                 throws IOException

Get the input paths from a String. This will resolve the path string and return either a Path object representing the string or, if the path string is a directory, a list of Paths representing all the "part" files.

Parameters:

path - the path string

Returns:

the paths

Throws:

IOException

getInputPaths

public static org.apache.hadoop.fs.Path[] getInputPaths(String[] paths)
                                                 throws IOException

Parameters:

paths -

Returns:

all the file starting with "part" in the paths requested

Throws:

IOException

getInputPaths

public static org.apache.hadoop.fs.Path[] getInputPaths(String[] paths,
                                                        String subdir)
                                                 throws IOException

All the files starting with "part" in the paths which look like: "paths[i]/subdir

Parameters:

paths -

subdir -

Returns:

the paths to the part files

Throws:

IOException

fileExists

public static boolean fileExists(String path)
                          throws IOException

Use hadoop filesystem to check if the given path exists

Parameters:

path - the path to the file

Returns:

true if file exists; false otherwise

Throws:

IOException

readlines

public static String[] readlines(String p)
                          throws IOException

Read a whole hadoop file into a string. This is obviously a ridiculous thing to do for all but the SMALLEST hadoop files so be very careful

Parameters:

p - a path

Returns:

the content of the path p as a string

Throws:

IOException

encodeArgs

public static String[] encodeArgs(String[] args)

A horrible hack to deal with hadoop's horrible hack when setting arrays of strings as configs

Parameters:

args -

Returns:

horribly replace each "," with #COMMA#

decodeArgs

public static String[] decodeArgs(String[] args)

A horrible hack to deal with hadoop's horrible hack when setting arrays of strings as configs

Parameters:

args -

Returns:

horribly replace each #COMMA# with ","