com.google.appengine.tools.mapreduce.outputs

Class SizeSegmentedGoogleCloudStorageFileOutput

java.lang.Object

com.google.appengine.tools.mapreduce.Output<ByteBuffer,GoogleCloudStorageFileSet>

com.google.appengine.tools.mapreduce.outputs.SizeSegmentedGoogleCloudStorageFileOutput

All Implemented Interfaces:

Serializable

public final class SizeSegmentedGoogleCloudStorageFileOutput
extends Output<ByteBuffer,GoogleCloudStorageFileSet>

An Output that writes files to GCS segmented by specified size limit. For e.g. to write files with size no greater than 100 bytes pass the segmentSizeLimit as 100.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
SizeSegmentedGoogleCloudStorageFileOutput(String bucket, long segmentSizeLimit, String fileNamePattern, String mimeType)

Method Summary

Methods

Modifier and Type	Method and Description
List<? extends OutputWriter<ByteBuffer>>	createWriters(int numShards) Returns OutputWriters that write files to GCS but segments the files based on the maximum allowed file size.
GoogleCloudStorageFileSet	finish(Collection<? extends OutputWriter<ByteBuffer>> writers) Returns a list of all the filenames written by the output writers

Methods inherited from class com.google.appengine.tools.mapreduce.Output

getContext, setContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

SizeSegmentedGoogleCloudStorageFileOutput

public SizeSegmentedGoogleCloudStorageFileOutput(String bucket,
                                         long segmentSizeLimit,
                                         String fileNamePattern,
                                         String mimeType)

Parameters:

bucket - GCS bucket

segmentSizeLimit - Maximum size of the files to be written to the output

fileNamePattern - a java format string Formatter containing one int argument for the shard number and another int argument for the segment number for e.g. shard-%04d-segment-%04d.

mimeType - The string to be passed as the mimeType to GCS.

Method Detail

createWriters

public List<? extends OutputWriter<ByteBuffer>> createWriters(int numShards)

Returns OutputWriters that write files to GCS but segments the files based on the maximum allowed file size. One writer can write more than one file per shard

Specified by:

createWriters in class Output<ByteBuffer,GoogleCloudStorageFileSet>

Parameters:

numShards - The number of shards which should be equal to the number of OutputWriters returned.

finish

public GoogleCloudStorageFileSet finish(Collection<? extends OutputWriter<ByteBuffer>> writers)
                                 throws IOException

Returns a list of all the filenames written by the output writers

Specified by:

finish in class Output<ByteBuffer,GoogleCloudStorageFileSet>

Throws:

IOException