com.google.appengine.tools.mapreduce.outputs

Class ShardingOutputWriter<K,V,WriterT extends OutputWriter<KeyValue<K,V>>>

java.lang.Object

com.google.appengine.tools.mapreduce.OutputWriter<KeyValue<K,V>>

com.google.appengine.tools.mapreduce.outputs.ShardingOutputWriter<K,V,WriterT>

Type Parameters:

K - key type

V - value type

WriterT - type of the output writer being written to

All Implemented Interfaces:

Serializable

Direct Known Subclasses:

GoogleCloudStorageMapOutputWriter

public abstract class ShardingOutputWriter<K,V,WriterT extends OutputWriter<KeyValue<K,V>>>
extends OutputWriter<KeyValue<K,V>>

Writes KeyValues out to a number of output writers each corresponding to a different shard A sharder and an Output object are provided in the constructor. The output object will be used to create the writers that will be written to. The sharder will determine which writer any given record will be written to based on the key.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected Sharder	sharder

Constructor Summary

Constructors

Constructor and Description
ShardingOutputWriter(Marshaller<K> keyMarshaller, Sharder sharder)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowSliceRetry()
void	beginShard() Will be called once before any calls to write.
void	beginSlice() Prepares the writer for writing after possibly having gone through serialization or deserialization.
protected abstract WriterT	createWriter(int shardId) Creates a new writer.
void	endShard() Called when no more output will be written to this writer.
void	endSlice() Prepares the writer for possible serialization.
abstract long	estimateMemoryRequirement() Returns the estimated memory that will be used by this writer in bytes.
protected Map<Integer,WriterT>	getShardsToWriterMap()
void	write(KeyValue<K,V> value) Writes a value to the output.

Methods inherited from class com.google.appengine.tools.mapreduce.OutputWriter

cleanup, close, getContext, open, setContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

sharder

protected final Sharder sharder

Constructor Detail

ShardingOutputWriter

public ShardingOutputWriter(Marshaller<K> keyMarshaller,
                    Sharder sharder)

Method Detail

beginShard

public void beginShard()
                throws IOException

Description copied from class: OutputWriter

Will be called once before any calls to write. Prepares the writer for processing, after possibly having gone through serialization and deserialization.

Overrides:

beginShard in class OutputWriter<KeyValue<K,V>>

Throws:

IOException - in the event of failure

endSlice

public void endSlice()
              throws IOException

Description copied from class: OutputWriter

Prepares the writer for possible serialization.

Overrides:

endSlice in class OutputWriter<KeyValue<K,V>>

Throws:

IOException - in the event of failure

beginSlice

public void beginSlice()
                throws IOException

Description copied from class: OutputWriter

Prepares the writer for writing after possibly having gone through serialization or deserialization.

Overrides:

beginSlice in class OutputWriter<KeyValue<K,V>>

Throws:

IOException - in the event of failure

write

public void write(KeyValue<K,V> value)
           throws IOException

Description copied from class: OutputWriter

Writes a value to the output.

Specified by:

write in class OutputWriter<KeyValue<K,V>>

Throws:

IOException

createWriter

protected abstract WriterT createWriter(int shardId)

Creates a new writer. (This is called at most once per shard)

Parameters:

shardId - Implementations should not assume numbers provided are either in-order or continuous.

endShard

public void endShard()
              throws IOException

Description copied from class: OutputWriter

Called when no more output will be written to this writer.

Overrides:

endShard in class OutputWriter<KeyValue<K,V>>

Throws:

IOException

estimateMemoryRequirement

public abstract long estimateMemoryRequirement()

Description copied from class: OutputWriter

Returns the estimated memory that will be used by this writer in bytes. (This is normally just set to the size of the buffers used by the implementation)

Overrides:

estimateMemoryRequirement in class OutputWriter<KeyValue<K,V>>

getShardsToWriterMap

protected Map<Integer,WriterT> getShardsToWriterMap()

allowSliceRetry

public boolean allowSliceRetry()

Overrides:

allowSliceRetry in class OutputWriter<KeyValue<K,V>>