org.openimaj.hadoop.tools.downloader

Class DownloadMapper

java.lang.Object

org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>

org.openimaj.hadoop.tools.downloader.DownloadMapper

public class DownloadMapper
extends org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>

A Hadoop Mapper for downloading files.

Author:

Sina Samangooei (ss@ecs.soton.ac.uk), Jonathon Hare (jsh2@ecs.soton.ac.uk)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	DownloadMapper.Counters

Nested classes/interfaces inherited from class org.apache.hadoop.mapreduce.Mapper

org.apache.hadoop.mapreduce.Mapper.Context

Constructor Summary

Constructors

Constructor and Description
DownloadMapper()

Method Summary

Modifier and Type	Method and Description
protected void	cleanup(org.apache.hadoop.mapreduce.Mapper.Context context)
void	map(org.apache.hadoop.io.LongWritable index, org.apache.hadoop.io.Text urlLine, org.apache.hadoop.mapreduce.Mapper.Context context)
protected void	setup(org.apache.hadoop.mapreduce.Mapper.Context context)

Methods inherited from class org.apache.hadoop.mapreduce.Mapper

run

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DownloadMapper

public DownloadMapper()

Method Detail

setup

protected void setup(org.apache.hadoop.mapreduce.Mapper.Context context)
              throws IOException,
                     InterruptedException

Overrides:

setup in class org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>

Throws:

IOException

InterruptedException

cleanup

protected void cleanup(org.apache.hadoop.mapreduce.Mapper.Context context)
                throws IOException,
                       InterruptedException

Overrides:

cleanup in class org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>

Throws:

IOException

InterruptedException

map

public void map(org.apache.hadoop.io.LongWritable index,
                org.apache.hadoop.io.Text urlLine,
                org.apache.hadoop.mapreduce.Mapper.Context context)

Overrides:

map in class org.apache.hadoop.mapreduce.Mapper<org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable>