User guide¶

This document assumes that you have already deployed k8s-hybrid-neg-controller to your Kubernetes cluster.

Creating hybrid NEGs¶

To create hybrid NEGs for your Kubernetes workloads, add the annotation solutions.cloud.google.com/hybrid-neg to your Kubernetes Services. The controller expects that the format of the annotation value matches the format of the cloud.google.com/neg annotation used with the GKE NEG controller.

A Kubernetes Service spec can declare more than one port. Endpoints in NEGs can only have one port. This means that if you specify multiple ports in the annotation, the controller creates NEGs for each port.

The format of the solutions.cloud.google.com/hybrid-neg annotation is:

solutions.cloud.google.com/hybrid-neg: '{
  "exposed_ports": {
    "SERVICE_PORT_1": {"name": "NEG_NAME_1"},
    "SERVICE_PORT_2": {"name": "NEG_NAME_2"},
    "SERVICE_PORT_3": {"name": "NEG_NAME_3"},
  }
}'

Each instance of SERVICE_PORT_N is a distinct port number that refers to a Service port. For each Service port in the exposed_ports map, the controller creates one NEG in each zone that the controller is configured to use.

An example annotation value for service port 50051:

solutions.cloud.google.com/hybrid-neg: '{"exposed_ports": {"50051": {"name": "my-hybrid-neg"}}}'

You do not need to specify NEG names. For example, the following is a valid annotation value for a Service that exposes port 80:

solutions.cloud.google.com/hybrid-neg: '{"exposed_ports": {"80": {}}}'

If you do not specify a name, the controller generates a name for the NEGs based on the Service namespace, name, and service port number.

The format of automatically generated NEG names is:

k8s1-CLUSTER_ID-NAMESPACE-SERVICE_NAME-PORT_NUMBER-RANDOM_HASH

When the controller detects a Service with the solutions.cloud.google.com/hybrid-neg annotation, it creates NEGs based on the annotation value and the zones configured on the controller. When creating the NEGs, the controller adds the solutions.cloud.google.com/hybrid-neg-status annotation to the Service. The format of the status annotation value matches that of the cloud.google.com/neg-status annotation created by the GKE NEG controller.

The format of the solutions.cloud.google.com/hybrid-neg-status annotation is:

solutions.cloud.google.com/hybrid-neg-status: '{
  "network-endpoint-groups": {
    "SERVICE_PORT_1": "NEG_NAME_1",
    "SERVICE_PORT_2": "NEG_NAME_2",
    "SERVICE_PORT_3": "NEG_NAME_3",
  },
  "zones": ["ZONE_1", "ZONE_2", "ZONE_3"]
}'

The values of NEG_NAME_N are the NEG names - either the names you specified in the solutions.cloud.google.com/hybrid-neg annotation, or names generated by the controller.

The values of ZONE_N are the Compute Engine zones that the controller have been configured to use. The zones are either target values of the --zone-mapping flag, or the list specified with the --neg-zones flag, with the latter taking precedence.

List the NEGs created by the controller:

gcloud compute network-endpoint-groups list --filter 'networkEndpointType:NON_GCP_PRIVATE_IP_PORT'

To list the endpoints of the NEGs, use the command gcloud compute network-endpoint-groups list-network-endpoints.

Using the hybrid NEGs¶

After the hybrid NEGs have been created, you can add them as backends to your backend services on Google Cloud. Backend services are used with load balancer URL maps and with Cloud Service Mesh Route resources from the Network Services API.

Hybrid NEGs are zonal resources, so ensure that you add each zonal NEG to your backend service.

See the following tutorials on using hybrid NEGs with Google Cloud products.

External Application Load Balancer¶

Internal Application Load Balancer¶

External proxy Network Load Balancer¶

Set up a regional external proxy Network Load Balancer with hybrid connectivity

Internal proxy Network Load Balancer¶

Cloud Service Mesh with managed Traffic Director control plane¶

Deleting hybrid NEGs¶

You can delete hybrid NEGs created by k8s-hybrid-neg-controller in any of these ways:

Delete the Kubernetes Service that has the solutions.cloud.google.com/hybrid-neg annotation.
Remove the solutions.cloud.google.com/hybrid-neg annotation from the Kubernetes Service.
Remove the exposed_port dictionary entries in the solutions.cloud.google.com/hybrid-neg annotation on the Kubernetes Service. For example, you can delete all hybrid NEGs of a Service by changing the annotation to look like this:

solutions.cloud.google.com/hybrid-neg: '{"exposed_ports": {}}'

If you do any of the above, k8s-hybrid-neg-controller first detaches all endpoints from the NEGs, and then deletes the NEGs.

If a NEG is added as a backend to a backend service, the controller will be unable to delete the NEG. In this scenario, the controller will remove all endpoints from the NEG, but leave the NEG in place. For more information, see the section below on preventing leaked NEGs.

Preventing leaked NEGs¶

You are responsible for managing the lifecycle of hybrid NEGs. You could leak NEGs in these ways:

When you delete a Kubernetes Service or remove the solutions.cloud.google.com/hybrid-neg annotation from a Kubernetes Service, any associated NEGs that are referenced by a backend service won't be deleted. Remove the NEGs from the backend service to allow NEG deletion.
When a Kubernetes cluster is deleted, NEGs are not deleted.

Follow the instructions in the section above on deleting hybrid NEGs above to prevent leaked NEGs.