Skip to main content

Troubleshooting

Overview​

The troubleshooting section aims to identify the most common problems users may face when deploying and using RAD Lab.

If you can't find a solution, please don't hesitate to create a GitHub issue.

NOTE: This is not an officially supported Google product

  1. Project quota exceeded
  2. Unable to modify Organization Policy Constraints
  3. Local Terraform Deployment ID Directory Already Exists
  4. Timeout when Destroying the deployment

Deployment Troubleshooting​

Google Organization Policies - unable to modify constraints​

Issue: When running the python3 radlab.py installation from Cloud Shell, you receive the following error:

Error: Error waiting to create Instance: Error waiting for Creating Instance: Error code 25, message: Constraint constraints/compute.vmExternalIpAccess violated for project [project_id]. Add instance projects/[project_id]/zones/us-east4-c/instances/notebooks-instance-0 to the constraint to use external IP with it.
.
.
Error Occurred - Deployment failed for ID:[deployment_id]

Solution: If you see above error in your initial deployment run, rerun the deployment via python3 radlab.py using the [deployment_id] and select Update (in Steps to Deploy RAD Lab Modules). This may have been caused as the Organization Policy constraints/compute.vmExternalIpAccess is not completely rolled out.

NOTE: Similarly, if the error occurs for any other organization policies, then the workaround is same as above.

Issue: When running the python3 radlab.py installation from Cloud Shell, you receive the following error(s):

Error: googleapi: Error 403: The caller does not have permission, forbidden
...
with google_project_organization_policy.shielded_vm_policy[0],
...

OR

with google_project_organization_policy.trustedimage_project_policy[0], ...

OR

with google_project_organization_policy.external_ip_policy[0],

Solution: The project is required to be run as part of a Google Organization in Google Cloud. Be sure that the Google Cloud user you run the python3 radlab.py script as has both of the following roles:

  • Organization Policy Administrator
  • Organization Viewer

Project Quota Exceeded​

Issue: When running the python3 radlab.py installation from Cloud Shell, you receive the following error:

 Error: Error setting billing account "[yourBillingID]" for project "projects/radlab-ds-analytics-[deployment_id]": 
 googleapi: Error 400: Precondition check failed., 
 failedPrecondition

  with module.project_radlab_ds_analytics.module.project-factory.google_project.main,

Solution: There are soft limits to the number of projects you can initially associate with your billing account. You can request a quota increase for the number of projects you are allowed to link to your billing account by filling out the form here. Please be sure you are logged in as a user with Google Cloud project Owner rights.

Note: If your billing account is relatively new, or if you are still in your free trial period, you may be required to authorize funds to pre-pay your billing account. The amount you asked to pay will vary depending on your billing history but will usually not exceed $50. Please see Why am I being asked to make a payment for more projects for details.

Timeout when Destroying the deployment​

Issue: When running the python3 radlab.py installation from Cloud Shell, you receive the following error:

╷
│ Error: Error waiting for Deleting Subnetwork: timeout while waiting for state to become 'DONE' (last state: 'RUNNING', timeout: 6m0s)
│ 
.
.
Error Occurred - Deployment failed for ID:[deployment_id]

Solution: If you see above error, rerun the deployment via python3 radlab.py using the [deployment_id] and select Delete (in Steps to Deploy RAD Lab Modules). This may have been caused if it took longer than expected to destroy any resource(s).

Operations Troubleshooting​

Local Terraform Deployment ID Directory Already Exists​

Issue: When running an Update or Delete action when running python3 radlab.py installation from Cloud Shell, you receive the following error:

File "radlab.py", line 134, in main

    os.mkdir(env_path)

FileExistsError: [Errno 17] File exists:
 '/home/[radlabAdminUser]/radlab/deployments/data_science_[deployment_id]'

Solution: You likely have a local copy of a Terraform /deployments folder with the same deploymentID in your Cloud Shell instance that needs to be removed. This can happen if you perform multiple Actions on a single deployment from the same Cloud Shell instance. Even though you are looking to a shared Google Cloud bucket to get the current Terraform state, Terraform will still try to create a local copy of the deployment first.

To solve this issue, you can safely remove the conflicting deployment folder by navigating to the ./radlab-launcher/deployments directory in Cloud Shell

cd ./deployments

And removing the module folder with the conflicting deploymentID:

rm -rfv /data_science_[deployment_id]/

Note: The above command will remove all files and sub-directories in the /data_science_[deployment_id]/ directory before removing the /data_science_[deployment_id]/ directory. The contents of this directory will sync from the Google Cloud state bucket next time you run the python3 radlab.py installation.