com.google.appengine.tools.mapreduce.outputs

Class GoogleCloudStorageFileOutput

java.lang.Object

com.google.appengine.tools.mapreduce.Output<ByteBuffer,GoogleCloudStorageFileSet>

com.google.appengine.tools.mapreduce.outputs.GoogleCloudStorageFileOutput

All Implemented Interfaces:

Serializable

public class GoogleCloudStorageFileOutput
extends Output<ByteBuffer,GoogleCloudStorageFileSet>

An Output that writes bytes to a set of Cloud Storage files, one per shard. Produces a single file output (usually on a per-shard basis). This format does not insert any separator characters, so it by default cannot be read back with the CloudStorageLineInputReader.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
GoogleCloudStorageFileOutput(String bucket, String fileNamePattern, String mimeType) Creates output files who's names follow the provided pattern in the specified bucket.
GoogleCloudStorageFileOutput(String bucket, String fileNamePattern, String mimeType, boolean supportSliceRetries) Creates output files who's names follow the provided pattern in the specified bucket.

Method Summary

Methods

Modifier and Type	Method and Description
List<GoogleCloudStorageFileOutputWriter>	createWriters(int numShards) Returns a list of writers, one for each shard, for this output.
GoogleCloudStorageFileSet	finish(Collection<? extends OutputWriter<ByteBuffer>> writers) Returns a list of GcsFilename that has one element for each reduce shard.

Methods inherited from class com.google.appengine.tools.mapreduce.Output

getContext, setContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

GoogleCloudStorageFileOutput

public GoogleCloudStorageFileOutput(String bucket,
                            String fileNamePattern,
                            String mimeType)

Creates output files who's names follow the provided pattern in the specified bucket. This will construct an instance that supports slice retries.

Parameters:

fileNamePattern - a Java format string Formatter containing one int argument for the shard number.

mimeType - The string to be passed as the mimeType to GCS.

GoogleCloudStorageFileOutput

public GoogleCloudStorageFileOutput(String bucket,
                            String fileNamePattern,
                            String mimeType,
                            boolean supportSliceRetries)

Creates output files who's names follow the provided pattern in the specified bucket.

Parameters:

fileNamePattern - a Java format string Formatter containing one int argument for the shard number.

mimeType - The string to be passed as the mimeType to GCS.

supportSliceRetries - indicates if slice retries should be supported by this writer. Slice retries are achieved by writing each slice to a temporary file and copying it to its destination when processing the next slice.

Method Detail

createWriters

public List<GoogleCloudStorageFileOutputWriter> createWriters(int numShards)

Description copied from class: Output

Returns a list of writers, one for each shard, for this output.

Specified by:

createWriters in class Output<ByteBuffer,GoogleCloudStorageFileSet>

Parameters:

numShards - The number of shards which should be equal to the number of OutputWriters returned.

finish

public GoogleCloudStorageFileSet finish(Collection<? extends OutputWriter<ByteBuffer>> writers)

Returns a list of GcsFilename that has one element for each reduce shard.

Specified by:

finish in class Output<ByteBuffer,GoogleCloudStorageFileSet>