org.openimaj.hadoop.tools.downloader

Class HadoopDownloaderOptions

java.lang.Object

org.openimaj.hadoop.tools.downloader.HadoopDownloaderOptions

public class HadoopDownloaderOptions
extends Object

Command-line options for the downloader tool

Author:

Jonathon Hare (jsh2@ecs.soton.ac.uk)

Constructor Summary

Constructors

Constructor and Description
HadoopDownloaderOptions(String[] args) Construct with the given arguments

Method Summary

Modifier and Type	Method and Description
boolean	followRedirects()
InputMode.Parser	getInputParser()
org.apache.hadoop.fs.Path[]	getInputPaths() Get the input file(s) containing the URLs
int	getNumberOfReducers()
int	getNumberOfThreads() Get the number of threads to use in the mapper.
org.apache.hadoop.fs.Path	getOutputPath()
long	getSleep()
void	prepare(boolean initial) Prepare the options
boolean	writeFailures()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

HadoopDownloaderOptions

public HadoopDownloaderOptions(String[] args)

Construct with the given arguments

Parameters:

args - the arguments

Method Detail

prepare

public void prepare(boolean initial)

Prepare the options

Parameters:

initial - true if initial setup is being performed; false if inside the mapper

getInputPaths

public org.apache.hadoop.fs.Path[] getInputPaths()
                                          throws IOException

Get the input file(s) containing the URLs

Returns:

the input paths

Throws:

IOException

getOutputPath

public org.apache.hadoop.fs.Path getOutputPath()

Returns:

the output file location

getNumberOfReducers

public int getNumberOfReducers()

Returns:

the number of reducers

getInputParser

public InputMode.Parser getInputParser()

Returns:

the InputMode.Parser corresponding to the selected mode.

getSleep

public long getSleep()

Returns:

the time in milliseconds to sleep after downloading a file

followRedirects

public boolean followRedirects()

Returns:

true if redirects should be followed; false otherwise

writeFailures

public boolean writeFailures()

Returns:

true if failed records should be logged to a file

getNumberOfThreads

public int getNumberOfThreads()

Get the number of threads to use in the mapper. If >1 a MultithreadedMapper will be used.

Returns:

number of threads to use