com.google.appengine.tools.mapreduce.impl

Class GoogleCloudStorageMergeInput

java.lang.Object

com.google.appengine.tools.mapreduce.Input<KeyValue<ByteBuffer,Iterator<ByteBuffer>>>

com.google.appengine.tools.mapreduce.impl.GoogleCloudStorageMergeInput

All Implemented Interfaces:

Serializable

public class GoogleCloudStorageMergeInput
extends Input<KeyValue<ByteBuffer,Iterator<ByteBuffer>>>

Defines the way the data is read in by the merger. This consists of merging GCS files that contain sorted KeyValue values in LevelDb format. To maintain the sorted order a MergingReader is used. (Note the returned sequence will go backwards, seecreateReaders())

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageMergeInput(FilesByShard files, int mergeFanin)

Method Summary

Methods

Modifier and Type	Method and Description
List<? extends InputReader<KeyValue<ByteBuffer,Iterator<ByteBuffer>>>>	createReaders() Creates multiple merging readers for each shard using createReaderForShard(com.google.appengine.tools.mapreduce.Marshaller<com.google.appengine.tools.mapreduce.KeyValue<java.nio.ByteBuffer, ? extends java.lang.Iterable<java.nio.ByteBuffer>>>, com.google.appengine.tools.mapreduce.GoogleCloudStorageFileSet) below.

Methods inherited from class com.google.appengine.tools.mapreduce.Input

getContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoogleCloudStorageMergeInput

public GoogleCloudStorageMergeInput(FilesByShard files,
                            int mergeFanin)

Method Detail

createReaders

public List<? extends InputReader<KeyValue<ByteBuffer,Iterator<ByteBuffer>>>> createReaders()

Creates multiple merging readers for each shard using createReaderForShard(com.google.appengine.tools.mapreduce.Marshaller<com.google.appengine.tools.mapreduce.KeyValue<java.nio.ByteBuffer, ? extends java.lang.Iterable<java.nio.ByteBuffer>>>, com.google.appengine.tools.mapreduce.GoogleCloudStorageFileSet) below. These are combined into a single reader via concatenation. The resulting input stream will be in order except when the input switches from one merging reader to the next the key will very likely go backwards.

Specified by:

createReaders in class Input<KeyValue<ByteBuffer,Iterator<ByteBuffer>>>