com.google.appengine.tools.mapreduce.impl.sort

Class SortShardTask

java.lang.Object

com.google.appengine.tools.mapreduce.impl.WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

com.google.appengine.tools.mapreduce.impl.sort.SortShardTask

All Implemented Interfaces:

IncrementalTaskWithContext, IncrementalTask, Serializable

public class SortShardTask
extends WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

Keeps accepting inputs and passing them to a SortWorker until the worker's buffer is full. Once this occurs endSlice is called which causes the data to be written out in sorted order.

See Also:

Serialized Form

Field Summary

Fields inherited from class com.google.appengine.tools.mapreduce.impl.WorkerShardTask

claimedMemory, LIMITER

Constructor Summary

Constructors

Constructor and Description
SortShardTask(String mrJobId, int shardNumber, int shardCount, InputReader<KeyValue<ByteBuffer,ByteBuffer>> in, SortWorker worker, OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>> out, int sortReadTimeMillis)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowSliceRetry(boolean abandon)
protected void	callWorker(KeyValue<ByteBuffer,ByteBuffer> input)
void	cleanup() Clean up and release any resources claimed in prepare.
protected long	estimateMemoryRequirement()
protected String	formatLastWorkItem(KeyValue<ByteBuffer,ByteBuffer> item)
InputReader<KeyValue<ByteBuffer,ByteBuffer>>	getInputReader()
OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>>	getOutputWriter()
protected Worker<SortContext>	getWorker()
void	jobCompleted(Status status) A job completed callback to allow resource cleanup and compaction of the finalized state.
void	prepare() Called immediately before run.
protected boolean	shouldCheckpoint(long timeElapsed)

Methods inherited from class com.google.appengine.tools.mapreduce.impl.WorkerShardTask

abbrev, getContext, isDone, run, setFinalized, toString, wasFinalized

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

SortShardTask

public SortShardTask(String mrJobId,
             int shardNumber,
             int shardCount,
             InputReader<KeyValue<ByteBuffer,ByteBuffer>> in,
             SortWorker worker,
             OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>> out,
             int sortReadTimeMillis)

Method Detail

prepare

public void prepare()

Description copied from interface: IncrementalTask

Called immediately before run. This method should be very limited in scope and should not block, perform IO or fail for any reason other than Rejecting the request.

Specified by:

prepare in interface IncrementalTask

Overrides:

prepare in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

cleanup

public void cleanup()

Description copied from interface: IncrementalTask

Clean up and release any resources claimed in prepare. Implementations of this method should not throw under any circumstances.

Specified by:

cleanup in interface IncrementalTask

Overrides:

cleanup in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

callWorker

protected void callWorker(KeyValue<ByteBuffer,ByteBuffer> input)

Specified by:

callWorker in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

formatLastWorkItem

protected String formatLastWorkItem(KeyValue<ByteBuffer,ByteBuffer> item)

Specified by:

formatLastWorkItem in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

shouldCheckpoint

protected boolean shouldCheckpoint(long timeElapsed)

Specified by:

shouldCheckpoint in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

Returns:

true iff a checkpoint should be performed.

estimateMemoryRequirement

protected long estimateMemoryRequirement()

Specified by:

estimateMemoryRequirement in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

getWorker

protected Worker<SortContext> getWorker()

Specified by:

getWorker in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

getOutputWriter

public OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>> getOutputWriter()

Specified by:

getOutputWriter in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

getInputReader

public InputReader<KeyValue<ByteBuffer,ByteBuffer>> getInputReader()

Specified by:

getInputReader in class WorkerShardTask<KeyValue<ByteBuffer,ByteBuffer>,KeyValue<ByteBuffer,List<ByteBuffer>>,SortContext>

allowSliceRetry

public boolean allowSliceRetry(boolean abandon)

Parameters:

abandon - true if a retry is due to an abandoned lock.

Returns:

true if a slice retry after failure are permitted.

jobCompleted

public void jobCompleted(Status status)

Description copied from interface: IncrementalTask

A job completed callback to allow resource cleanup and compaction of the finalized state.

Parameters:

status - the status of the job