com.google.appengine.tools.mapreduce.impl.sort

Class MergeShardTask

java.lang.Object

com.google.appengine.tools.mapreduce.impl.WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

com.google.appengine.tools.mapreduce.impl.sort.MergeShardTask

All Implemented Interfaces:

IncrementalTaskWithContext, IncrementalTask, Serializable

public class MergeShardTask
extends WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

Passes input to the output. (All the real work is done in those classes)

See Also:

Serialized Form

Field Summary

Fields inherited from class com.google.appengine.tools.mapreduce.impl.WorkerShardTask

claimedMemory, LIMITER

Constructor Summary

Constructors

Constructor and Description
MergeShardTask(String mrJobId, int shardNumber, int shardCount, InputReader<KeyValue<ByteBuffer,Iterator<ByteBuffer>>> in, OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>> out, int sortReadTimeMillis)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowSliceRetry(boolean abandon)
protected void	callWorker(KeyValue<ByteBuffer,Iterator<ByteBuffer>> input)
protected long	estimateMemoryRequirement()
protected String	formatLastWorkItem(KeyValue<ByteBuffer,Iterator<ByteBuffer>> item)
InputReader<KeyValue<ByteBuffer,Iterator<ByteBuffer>>>	getInputReader()
OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>>	getOutputWriter()
protected Worker<MergeContext>	getWorker()
void	jobCompleted(Status status) A job completed callback to allow resource cleanup and compaction of the finalized state.
protected boolean	shouldCheckpoint(long timeElapsed)

Methods inherited from class com.google.appengine.tools.mapreduce.impl.WorkerShardTask

abbrev, cleanup, getContext, isDone, prepare, run, setFinalized, toString, wasFinalized

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

MergeShardTask

public MergeShardTask(String mrJobId,
              int shardNumber,
              int shardCount,
              InputReader<KeyValue<ByteBuffer,Iterator<ByteBuffer>>> in,
              OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>> out,
              int sortReadTimeMillis)

Method Detail

callWorker

protected void callWorker(KeyValue<ByteBuffer,Iterator<ByteBuffer>> input)

Specified by:

callWorker in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

formatLastWorkItem

protected String formatLastWorkItem(KeyValue<ByteBuffer,Iterator<ByteBuffer>> item)

Specified by:

formatLastWorkItem in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

shouldCheckpoint

protected boolean shouldCheckpoint(long timeElapsed)

Specified by:

shouldCheckpoint in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

Returns:

true iff a checkpoint should be performed.

estimateMemoryRequirement

protected long estimateMemoryRequirement()

Specified by:

estimateMemoryRequirement in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

getWorker

protected Worker<MergeContext> getWorker()

Specified by:

getWorker in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

getOutputWriter

public OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>> getOutputWriter()

Specified by:

getOutputWriter in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

getInputReader

public InputReader<KeyValue<ByteBuffer,Iterator<ByteBuffer>>> getInputReader()

Specified by:

getInputReader in class WorkerShardTask<KeyValue<ByteBuffer,Iterator<ByteBuffer>>,KeyValue<ByteBuffer,List<ByteBuffer>>,MergeContext>

jobCompleted

public void jobCompleted(Status status)

Description copied from interface: IncrementalTask

A job completed callback to allow resource cleanup and compaction of the finalized state.

Parameters:

status - the status of the job

allowSliceRetry

public boolean allowSliceRetry(boolean abandon)

Parameters:

abandon - true if a retry is due to an abandoned lock.

Returns:

true if a slice retry after failure are permitted.