com.google.appengine.tools.mapreduce.impl

Class GoogleCloudStorageMergeOutput

java.lang.Object

com.google.appengine.tools.mapreduce.Output<KeyValue<ByteBuffer,List<ByteBuffer>>,FilesByShard>

com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMergeOutput

All Implemented Interfaces:

Serializable

public class GoogleCloudStorageMergeOutput
extends Output<KeyValue<ByteBuffer,List<ByteBuffer>>,FilesByShard>

Defines the way data is written out by the merging stage. This consists of multiple GCS files using Level Db format. Each file contains KeyValue pairs ordered by their key (same format as the sort output).

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageMergeOutput(String bucket, String mrJobId, Integer tier)

Method Summary

Methods

Modifier and Type	Method and Description
List<? extends OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>>>	createWriters(int shards) Returns a writer that writes the data the same way that the sort does, splitting the output every time the key goes backwards in sequence.
FilesByShard	finish(Collection<? extends OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>>> writers) Returns a result to be made available through MapReduceResult.getOutputResult().

Methods inherited from class com.google.appengine.tools.mapreduce.Output

getContext, setContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoogleCloudStorageMergeOutput

public GoogleCloudStorageMergeOutput(String bucket,
                             String mrJobId,
                             Integer tier)

Method Detail

createWriters

public List<? extends OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>>> createWriters(int shards)

Returns a writer that writes the data the same way that the sort does, splitting the output every time the key goes backwards in sequence. This way the output is a collection of files that are individually fully sorted. This works in conjunction with GoogleCloudStorageMergeInput to convert a large number of sorted files into a much smaller number of sorted files.

Specified by:

createWriters in class Output<KeyValue<ByteBuffer,List<ByteBuffer>>,FilesByShard>

Parameters:

shards - The number of shards which should be equal to the number of OutputWriters returned.

finish

public FilesByShard finish(Collection<? extends OutputWriter<KeyValue<ByteBuffer,List<ByteBuffer>>>> writers)

Description copied from class: Output

Returns a result to be made available through MapReduceResult.getOutputResult().

This method allows the Output to inspect the final state of its OutputWriters to produce a final result object. For example, if the OutputWriters produce blobs in blobstore, finish could collect and return a list of blob IDs.

Should return null if no such result makes sense for this kind of output.

Called after all OutputWriters have been closed (with OutputWriter.close()). It is possible for this method to be called more than once with the same writers collection. It is also possible for the job to fail after this is invoked.

The writers argument will contain the same writers that Output.createWriters(int) returned in the same order. Writers may be serialized and deserialized multiple times. typically, getWriter will have been called in a different JVM.

Specified by:

finish in class Output<KeyValue<ByteBuffer,List<ByteBuffer>>,FilesByShard>