com.google.appengine.tools.mapreduce.impl

Class GoogleCloudStorageMapOutputWriter<K,V>

java.lang.Object

com.google.appengine.tools.mapreduce.OutputWriter<KeyValue<K,V>>

com.google.appengine.tools.mapreduce.outputs.ShardingOutputWriter<K,V,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>

com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter<K,V>

Type Parameters:

K - type of intermediate keys

V - type of intermediate values

All Implemented Interfaces:

Serializable

public class GoogleCloudStorageMapOutputWriter<K,V>
extends ShardingOutputWriter<K,V,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>

An OutputWriter that is used by the map stage and writes bytes to GCS, later to be used by the sort/merge stages. Content is written in a LevelDb log Format and then using the KeyValueMarshaller to marshall the individual record.

See Also:

Serialized Form

Field Summary

Fields inherited from class com.google.appengine.tools.mapreduce.outputs.ShardingOutputWriter

sharder

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageMapOutputWriter(String bucket, String fileNamePattern, Marshaller<K> keyMarshaller, Marshaller<V> valueMarshaller, Sharder sharder)

Method Summary

Methods

Modifier and Type	Method and Description
boolean	allowSliceRetry()
com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>	createWriter(int sortShard) Creates a new writer.
long	estimateMemoryRequirement() Returns the estimated memory that will be used by this writer in bytes.
protected Map<Integer,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>	getShardsToWriterMap()

Methods inherited from class com.google.appengine.tools.mapreduce.outputs.ShardingOutputWriter

beginShard, beginSlice, endShard, endSlice, write

Methods inherited from class com.google.appengine.tools.mapreduce.OutputWriter

cleanup, close, getContext, open, setContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoogleCloudStorageMapOutputWriter

public GoogleCloudStorageMapOutputWriter(String bucket,
                                 String fileNamePattern,
                                 Marshaller<K> keyMarshaller,
                                 Marshaller<V> valueMarshaller,
                                 Sharder sharder)

Method Detail

allowSliceRetry

public boolean allowSliceRetry()

Overrides:

allowSliceRetry in class ShardingOutputWriter<K,V,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>

createWriter

public com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V> createWriter(int sortShard)

Description copied from class: ShardingOutputWriter

Creates a new writer. (This is called at most once per shard)

Specified by:

createWriter in class ShardingOutputWriter<K,V,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>

Parameters:

sortShard - Implementations should not assume numbers provided are either in-order or continuous.

getShardsToWriterMap

protected Map<Integer,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>> getShardsToWriterMap()

Overrides:

getShardsToWriterMap in class ShardingOutputWriter<K,V,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>

estimateMemoryRequirement

public long estimateMemoryRequirement()

Description copied from class: OutputWriter

Returns the estimated memory that will be used by this writer in bytes. (This is normally just set to the size of the buffers used by the implementation)

Specified by:

estimateMemoryRequirement in class ShardingOutputWriter<K,V,com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMapOutputWriter.MapOutputWriter<K,V>>