com.google.appengine.tools.mapreduce.impl

Class WorkerShardTask<I,O,C extends WorkerContext<O>>

java.lang.Object

com.google.appengine.tools.mapreduce.impl.WorkerShardTask<I,O,C>

Type Parameters:

I - type of input values consumed by the worker

O - type of output values produced by the worker

C - type of context required by the worker

All Implemented Interfaces:

IncrementalTaskWithContext, IncrementalTask, Serializable

Direct Known Subclasses:

MapOnlyShardTask, MapShardTask, MergeShardTask, ReduceShardTask, SortShardTask

public abstract class WorkerShardTask<I,O,C extends WorkerContext<O>>
extends Object
implements IncrementalTaskWithContext

Base class for the specific MR stage workers. Each subclass is responsible to set the context on its input, output, and worker.

Author:

ohler@google.com (Christian Ohler)

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Long	claimedMemory
protected static MemoryLimiter	LIMITER

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	WorkerShardTask(IncrementalTaskContext context)

Method Summary

Methods

Modifier and Type	Method and Description
protected static String	abbrev(Object x)
protected abstract void	callWorker(I input)
void	cleanup() Clean up and release any resources claimed in prepare.
protected abstract long	estimateMemoryRequirement()
protected abstract String	formatLastWorkItem(I item)
IncrementalTaskContext	getContext()
protected abstract InputReader<I>	getInputReader()
abstract OutputWriter<O>	getOutputWriter()
protected abstract Worker<C>	getWorker()
boolean	isDone()
void	prepare() Called immediately before run.
void	run() Runs this task.
protected void	setFinalized()
protected abstract boolean	shouldCheckpoint(long timeElapsed)
String	toString()
protected boolean	wasFinalized()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface com.google.appengine.tools.mapreduce.impl.shardedjob.IncrementalTask

allowSliceRetry, jobCompleted

Field Detail

LIMITER

protected static final MemoryLimiter LIMITER

claimedMemory

protected transient Long claimedMemory

Constructor Detail

WorkerShardTask

protected WorkerShardTask(IncrementalTaskContext context)

Method Detail

getContext

public final IncrementalTaskContext getContext()

Specified by:

getContext in interface IncrementalTaskWithContext

toString

public String toString()

Overrides:

toString in class Object

cleanup

public void cleanup()

Description copied from interface: IncrementalTask

Clean up and release any resources claimed in prepare. Implementations of this method should not throw under any circumstances.

Specified by:

cleanup in interface IncrementalTask

prepare

public void prepare()

Description copied from interface: IncrementalTask

Called immediately before run. This method should be very limited in scope and should not block, perform IO or fail for any reason other than Rejecting the request.

Specified by:

prepare in interface IncrementalTask

run

public void run()

Description copied from interface: IncrementalTask

Runs this task. This will be invoked over and over until isDone returns true. If this throws an exception, it may be retried a limited number of times according to a retry policy specified in ShardedJobSettings

Specified by:

run in interface IncrementalTask

abbrev

protected static String abbrev(Object x)

isDone

public boolean isDone()

Specified by:

isDone in interface IncrementalTask

Returns:

true iff this task is done and run should no longer be invoked.

setFinalized

protected final void setFinalized()

wasFinalized

protected final boolean wasFinalized()

shouldCheckpoint

protected abstract boolean shouldCheckpoint(long timeElapsed)

Returns:

true iff a checkpoint should be performed.

estimateMemoryRequirement

protected abstract long estimateMemoryRequirement()

getWorker

protected abstract Worker<C> getWorker()

getInputReader

protected abstract InputReader<I> getInputReader()

callWorker

protected abstract void callWorker(I input)

formatLastWorkItem

protected abstract String formatLastWorkItem(I item)

getOutputWriter

public abstract OutputWriter<O> getOutputWriter()