Configuration: Template Parameters
Pipeline Flags
The Dataflow pipeline requires or accepts the following flags:
| Flag | Required | Default Value | Description | Example |
|---|---|---|---|---|
| input_file_format | Yes | N/A | Format of the files to load. Supported: avro and csv. |
--input_file_format=csv |
| input_file_pattern | Yes | N/A | File pattern to the file(s). The path can contain glob characters (*, ?, and [...] sets). | --input_file_pattern=gs://my-bucket/data/*.csv |
| input_schema | Yes | N/A | The input schema using dtype strings. The format for each field is field_name:dtype. Example: name:string. The fields must follow the order of the file when in csv format. Each field must be separated with ;. Example: name:string;phone:number. |
--input_schema=id:int64;first_name:string;last_name:string;salary:float |
| input_csv_file_delimiter | No | , |
The column delimiter for the input CSV file(s). Only used for CSV files. | --input_csv_file_delimiter=";" |
| input_file_contains_headers | No | 1 |
Whether the input CSV files contain a header record. Use 1 for true and 0 for false. Only used for CSV files. | --input_file_contains_headers=1 |
| alloydb_ip | Yes | N/A | The IP (or hostname) of the AlloyDB (or PostgreSQL) database. The database must be accessible by the pipeline runner. | --alloydb_ip=10.1.1.25 |
| alloydb_port | No | 5432 |
Port of the AlloyDB instance. | --alloydb_port=5432 |
| alloydb_database | No | postgres |
Name of the AlloyDB database to which data will be written. | --alloydb_database=dbname |
| alloydb_user | No | postgres |
User to login to Postgres database on AlloyDB. | --alloydb_user=myuser |
| alloydb_password | Yes | N/A | Password to login to Postgres database on AlloyDB using the provided user. | --alloydb_password=mySafePassword123! |
| alloydb_table | Yes | N/A | Name of the table within the Postgres database on AlloyDB to which data will be written. | --alloydb_table=customers |
DataFlow Template Flags
Dataflow requires some additional parameter to deploy the pipeline as a Dataflow job. For information about pipeline options that are directly required and supported by Flex Templates, see Pipeline options.
| Flag | Required | Description | Example |
|---|---|---|---|
| region | Yes | The region where you want to deploy the Dataflow job. | --region=asia-southeast-1 |
| project | Yes | The project id where you want to deploy the Dataflow job. | --project=my-gcp-project |
| job_name | Yes | Name of the Dataflow job name. | --job_name=gcs_to_alloydb |
| temp_location | Yes | Cloud Storage path for Dataflow to stage temporary job files created during the execution of the pipeline. | --temp_location=gs://bucket/path/dataflow/temp/ |
| staging_location | Yes | Cloud Storage path for Dataflow to stage staging job files created during the execution of the pipeline. | --staging_location=gs://bucket/path/dataflow/staging/ |