Setting up connection profiles

In case of minimal downtime migration, Spanner Migration Tool needs information about connection profiles which are required by the datastream. There are two connection profiles that need to be set up - source connection profile and target connection profile.

Table of contents
  1. Source connection profile
  2. Target connection profile
  3. Sharded Migrations Configuration
    1. Form based configuration
    2. JSON based configuration

Source connection profile

Source connection profile is used by datastream to connect to the source database and read the data from the source database. Users can either use an existing source connection profile or create a new connection profile from Spanner Migration Tool by specifying a new name for the connection profile and allowlisting the IPs.

Target connection profile

Target connection profile is used to connect to the GCS bucket where the datastream writes data written to. Users can either use an existing target connection profile or create a new one from Spanner Migration Tool by specifying a new name for the connection profile. Please ensure that the GCS bucket is empty in case you choose an existing connection profile to ensure consistency between source and spanner database. In case the user opts for a new target connection profile, Spanner Migration Tool creates a new GCS bucket with bucket name as the Migration Request ID.

Sharded Migrations Configuration

In case of sharded migrations, Spanner migration tool requires connection details of each shard to create a source connection profile and launch a Datastream for it. Alternatively, the user can also provide pre-created source connection profiles for SMT to use.

For an overview of how sharded migrations work, refer to this section

  • On the prepare migration page, click on Configure Datastream to get started with creating/configuring connection profiles for each shard.

Form based configuration

Data shardId is a SMT generated identifier to track migration jobs created for a physical shard. It typically does not require a user to modify it.

  • Spanner migration tool provides multiple ways of configuring a sharded migration - via JSON or form. Configure the source and target as defined above, the same concepts apply here as well.

  • Add the database to shardId configuration mapping. Note that the value of the shardId provided here will to be used to populate the migration_shard_id column added to each table for the sharded migration. This field will be used to identify the source shard of MySQL while writing the data to Spanner, and has many other applications, such as reverse replication.

  • Click on ADD MORE SHARDS to save the current shard information and configure the next. The shard counter at the top shows the total number of physical instances and logical shards configured. Refer here for details on this terminology.

  • Once all shards are configured, click on FINISH.

JSON based configuration

For JSON based configuration, creation of new resources is not supported. The connection profiles configured via JSON should already exist in Datastream.

  • Select the Text input in the Datastream details form.
  • Paste the JSON configuration of the shards.
  • Click on Finish.
  • SMT will validate the configuration, and if valid, save it.

SMT provides an example of how a JSON based configuration looks like here.

Source and Target connection profiles can be automatically generated. Refer to this documentation to know more.