com.google.appengine.tools.mapreduce

Class InputReader<I>

java.lang.Object

com.google.appengine.tools.mapreduce.InputReader<I>

Type Parameters:

I - type of values produced by this input

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

ConcatenatingInputReader, DatastoreInputReader, DatastoreKeyInputReader, ForwardingInputReader, LevelDbInputReader, LogInputReader, UnmarshallingInputReader

public abstract class InputReader<I>
extends Object
implements Serializable

Reads data from an input source.

Created by Input to read input for a given shard. Reads input for a shard as a number of slices, where the slicing is up to the caller.

beginShard() is called before any calls to beginSlice() or next() to setup the reader.

Each slice is read by calling beginSlice(), then next() and/or getProgress() any number of times in any order, then endSlice(). Between two slices, before the first slice, or after the final slice, the InputReader may go through serialization and deserialization.

If a slice is aborted, there is no guarantee whether endSlice() will be called; however, if it is not called, the InputReader will not be serialized. If the slice is retried later, the InputReader serialized after the previous slice will be deserialized again.

At the end of the final slice, endShard() will be called after endSlice().

Author:

ohler@google.com (Christian Ohler)

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
InputReader()

Method Summary

Methods

Modifier and Type	Method and Description
void	beginShard() Performs setup at the beginning of the shard.
void	beginSlice() Prepares the InputReader for reading, after possibly having gone through serialization and deserialization.
void	close() Deprecated. Override endShard instead.
void	endShard() Called after endSlice if there will not be any subsequent calls to beginSlice or next.
void	endSlice() Prepares the InputReader for serialization.
long	estimateMemoryRequirement() Returns the estimated memory that will be used by this reader in bytes.
ShardContext	getContext() Returns the current context, or null if none.
Double	getProgress() Returns the relative progress reading this input as a number from 0 to 1.
abstract I	next() Returns the next input value, or throws NoSuchElementException when there is no more input data.
void	open() Deprecated. Override beginShard instead.
void	setContext(ShardContext context) Used internally to sets the context to be used for the processing that follows.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

InputReader

public InputReader()

Method Detail

setContext

public void setContext(ShardContext context)

Used internally to sets the context to be used for the processing that follows.

getContext

public ShardContext getContext()

Returns the current context, or null if none.

next

public abstract I next()
                throws IOException,
                       NoSuchElementException

Returns the next input value, or throws NoSuchElementException when there is no more input data. This is done rather than providing a hasNext() function, to allow the implementation to deal with streams and remain serializable, even if the item being read is not.

Throws:

IOException

NoSuchElementException

getProgress

public Double getProgress()

Returns the relative progress reading this input as a number from 0 to 1. Returns null if relative progress cannot be determined.

beginSlice

public void beginSlice()
                throws IOException

Prepares the InputReader for reading, after possibly having gone through serialization and deserialization.

Throws:

IOException - in the event of failure

endSlice

public void endSlice()
              throws IOException

Prepares the InputReader for serialization.

Throws:

IOException - in the event of failure

open

@Deprecated
public void open()
          throws IOException

Deprecated. Override beginShard instead.

Throws:

IOException - in the event of failure

beginShard

public void beginShard()
                throws IOException

Performs setup at the beginning of the shard. This method is invoked before the first call to beginSlice(). It will not be invoked again unless the shard restarts. When a shard is restarted, this method is invoked and the input should be read from the beginning.

Throws:

IOException - in the event of failure

close

@Deprecated
public void close()
           throws IOException

Deprecated. Override endShard instead.

Throws:

IOException - in the event of failure

endShard

public void endShard()
              throws IOException

Called after endSlice if there will not be any subsequent calls to beginSlice or next. This may be used to teardown or finalize any state if required.

Throws:

IOException - in the event of failure

estimateMemoryRequirement

public long estimateMemoryRequirement()

Returns the estimated memory that will be used by this reader in bytes. (This is normally just set to the size of the buffers used by the implementation)