com.google.appengine.tools.mapreduce

Class Worker<C extends WorkerContext<?>>

java.lang.Object

com.google.appengine.tools.mapreduce.Worker<C>

Type Parameters:

C - type of context required by this worker

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

MapOnlyMapper, Mapper, Reducer, SortWorker

public abstract class Worker<C extends WorkerContext<?>>
extends Object
implements Serializable

Base class for Mapper and Reducer.

Created during a MapReduceJob to process a given shard. Processes data for zero or more shards, each subdivided into zero or more slices, where sharding and slicing is up to the caller.

Each shard starts with a call to beginShard(), then processes zero or more slices, then finishes with a call to endShard(). Between two shards, before the first shard, or after the final shard, the worker may go through serialization and deserialization.

If a shard is aborted, there is no guarantee whether endShard() will be called; however, if it is not called, the worker will not be serialized or used further in any way. If the shard is retried later, the worker serialized after the previous shard will be deserialized again.

A worker may be used for processing multiple shards in parallel; in that case, it is deserialized multiple times to produce multiple copies. Each copy will process only one shard at a time.

Each slice starts with a call to beginSlice(), then performs the actual work with zero or more calls to BaseMapper.map(I) or Reducer.reduce(K, com.google.appengine.tools.mapreduce.ReducerInput<V>), then endSlice(). Between two slices, before the first slice, or after the final slice, the worker may go through serialization and deserialization.

If a slice is aborted, there is no guarantee whether endSlice() will be called; however, if it is not called, the worker will not be serialized or used further in any way. If the slice is retried later, the Worker serialized after the previous slice will be deserialized again.

Slices of the same shard are processed by the same worker sequentially.

This class is really an interface that might be evolving. In order to avoid breaking users when we change the interface, we made it an abstract class.

Author:

ohler@google.com (Christian Ohler)

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
Worker()

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowSliceRetry() Indicate if a slice retry is allowed.
void	beginShard() Prepares the worker for processing a new shard, after possibly having gone through serialization and deserialization.
void	beginSlice() Prepares the worker for processing a new slice, after possibly having gone through serialization and deserialization.
void	endShard() Notifies the worker that there is no more data to process in the current shard, and prepares it for possible serialization.
void	endSlice() Notifies the worker that there is no more data to process in the current slice, and prepares it for possible serialization.
long	estimateMemoryRequirement() Returns an estimate of the amount of memory needed for this worker in bytes.
C	getContext() Returns the current context, or null if none.
void	setContext(C context) Used internally to sets the context to be used for the processing that follows.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Worker

public Worker()

Method Detail

setContext

public void setContext(C context)

Used internally to sets the context to be used for the processing that follows.

getContext

public final C getContext()

Returns the current context, or null if none.

beginShard

public void beginShard()

Prepares the worker for processing a new shard, after possibly having gone through serialization and deserialization.

endShard

public void endShard()

Notifies the worker that there is no more data to process in the current shard, and prepares it for possible serialization.

beginSlice

public void beginSlice()

Prepares the worker for processing a new slice, after possibly having gone through serialization and deserialization.

endSlice

public void endSlice()

Notifies the worker that there is no more data to process in the current slice, and prepares it for possible serialization.

estimateMemoryRequirement

public long estimateMemoryRequirement()

Returns an estimate of the amount of memory needed for this worker in bytes.

allowSliceRetry

public boolean allowSliceRetry()

Indicate if a slice retry is allowed.