Configuring multi-cluster Services
source link: https://cloud.google.com/kubernetes-engine/docs/how-to/multi-cluster-services
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Configuring multi-cluster Services
This page shows you how to enable and use multi-cluster Services (MCS). To learn more about how MCS works and its benefits, see Multi-cluster Services.
The Google Kubernetes Engine (GKE) MCS feature extends the reach of the Kubernetes Service beyond the cluster boundary and lets you discover and invoke Services across multiple GKE clusters. You can export a subset of existing Services or new Services.
Note: When you export a Service with MCS, that Service is then available across all of the clusters in your fleet.Note: For MCS to function correctly, GKE deploys Pods to your nodes that have elevated RBAC permissions, such as the ability to patch all deployments, services, and endpoints. These permissions are required for GKE to propagate endpoints between multiple GKE clusters.
Google Cloud resources managed by MCS
MCS manages the following components of Google Cloud:
Cloud DNS: MCS configures Cloud DNS zones and records for each exported Service in your fleet clusters. This lets you connect to Services that are running in other clusters. These zones and records are created, read, updated, and deleted based on the Services that you choose to export across clusters.
Note: Using Cloud DNS incurs additional charges. You are billed according to Cloud DNS pricing.Firewall rules: MCS configures firewall rules that let Pods communicate with each other across clusters within your fleet. Firewall rules are created, read, updated, and deleted based on the clusters that you add to your fleet. These rules are similar to the rules that GKE creates to enable communication between Pods within a GKE cluster.
Traffic Director: MCS uses Traffic Director as a control plane to keep track of endpoints and their health across clusters.
Requirements
MCS has the following requirements:
MCS only supports exporting services from VPC-native GKE clusters on Google Cloud. For more information, see Creating a VPC-native cluster. You cannot use non VPC-native clusters.
Connectivity between clusters depends on clusters running within the same VPC network or in peered VPC networks.
MCS is not supported on clusters that use Cloud DNS as the in-cluster Service name resolver. For more information, see MCS with Cloud DNS.
Pricing
Multi-cluster Services is included as part of the GKE cluster management fee and has no extra cost for usage. You must enable the Traffic Director API, but MCS does not incur any Traffic Director endpoint charges. Anthos licensing is not required to use MCS.
Before you begin
Before you start, make sure you have performed the following tasks:
Install the Google Cloud SDK.
Enable the Google Kubernetes Engine API:
Enable the MCS, GKE fleet (hub), Resource Manager, Traffic Director, and Cloud DNS APIs:
gcloud services enable \
multiclusterservicediscovery.googleapis.com \
gkehub.googleapis.com \
cloudresourcemanager.googleapis.com \
trafficdirector.googleapis.com \
dns.googleapis.com \
--project=PROJECT_IDReplace
Note: Non-project owners must be granted thePROJECT_ID
with the project ID from the project where you plan to register your clusters to a fleet.serviceusage.services.enable
permission before they can enable APIs.
Enabling MCS in your project
MCS requires that participating GKE clusters be registered into the same fleet. Once the MCS feature is enabled for a fleet, any clusters can export Services between clusters in the fleet.
While MCS does require registration to a fleet, it does not require Anthos.
Anthos
If the Anthos API is enabled in your fleet host project as a prerequisite for other Anthos components, then any clusters registered to a fleet are charged according to Anthos pricing. You can confirm if Anthos is enabled using the following command:
gcloud services list --project=PROJECT_ID | grep anthos.googleapis.com
If the output is similar to the following, Anthos is enabled and any clusters registered to the fleet are Anthos clusters:
anthos.googleapis.com Anthos API
If this is not expected then contact your project administrator.
An empty output indicates that Anthos is not enabled.
Warning: Do not disable the Anthos API if there are other active Anthos components in use in your project or those components might experience an outage.
Enabling MCS on your GKE cluster
Enable the MCS feature for the fleet:
gcloud container hub multi-cluster-services enable \
--project PROJECT_IDReplace
PROJECT_ID
with the project ID from the project where you plan to register your clusters to a fleet.Register your GKE clusters to a fleet. We strongly recommend that you register your cluster with Workload Identity enabled. If you do not use Workload Identity, you need to complete the additional steps in Authenticating service accounts.
To register your cluster, run the following command:
gcloud container hub memberships register MEMBERSHIP_NAME \
--gke-cluster GKE_CLUSTER \
--enable-workload-identity \
--project PROJECT_IDReplace the following:
MEMBERSHIP_NAME
: the membership name that you choose to uniquely represent the cluster being registered on the hub.GKE_CLUSTER
: thelocation/name
of the GKE cluster from the current project. The location can be a zone or a region, for example:us-central1-a/my-gke-cluster
.
Grant the required Identity and Access Management (IAM) permissions for MCS Importer:
gcloud projects add-iam-policy-binding PROJECT_ID \
--member "serviceAccount:PROJECT_ID.svc.id.goog[gke-mcs/gke-mcs-importer]" \
--role "roles/compute.networkViewer"Replace
PROJECT_ID
with the project ID from the project where you registered your clusters to a fleet.Ensure that each cluster in the fleet has a namespace to share Services in. You cannot export Services in the
default
andkube-system
namespaces across clusters.If needed, create a namespace by using the following command:
kubectl create ns NAMESPACE
Replace
Note: Since Services are not imported to clusters where their exporting namespace is not present, create a Service's namespace in any clusters from which a Service will be consumed. If a namespace is missing, you can add the namespace at any time, but might take up to five minutes for a previously exported Service to be imported to a newly created namespace.NAMESPACE
with a name for the namespace.To verify that MCS is enabled, run the following command:
gcloud container hub multi-cluster-services describe
The output is similar to the following:
createTime: '2021-08-10T13:05:23.512044937Z' membershipStates: projects/PROJECT_ID/locations/global/memberships/MCS_NAME: state: code: OK description: Firewall successfully updated updateTime: '2021-08-10T13:14:45.173444870Z' name: projects/PROJECT_NAME/locations/global/features/multiclusterservicediscovery resourceState: state: ACTIVE spec: {}
If the value of
state
is notACTIVE
, see the troubleshooting section.
Authenticating service accounts
Important: You do not need to do any steps in this section if you registered a cluster to a fleet with Workload Identity enabled.
If you registered your GKE clusters to a fleet using a service
account, you need to take additional steps to authenticate the service account.
MCS deploys a component called gke-mcs-importer
. This component receives
endpoint updates from Traffic Director, so as part of enabling MCS you need to grant
your service account permission to read information
from Traffic Director.
When you use a service account, you can use the Compute Engine default service account or your own node service account:
If you are using a Compute Engine default service account, enable the following scopes:
https://www.googleapis.com/auth/compute.readonly
https://www.googleapis.com/auth/cloud-platform
To learn more about enabling scopes, see Changing the service account and access scopes for an instance.
If you are using your own node service account, assign the
roles/compute.networkViewer
role to your service account.
Using MCS
The following sections show you how to use MCS. MCS uses the Kubernetes multi-cluster services API.
Registering a Service for export
To register a Service for export to other clusters within your fleet, complete the following steps:
Create a
ServiceExport
object namedexport.yaml
:# export.yaml
kind: ServiceExport
apiVersion: net.gke.io/v1
metadata:
namespace: NAMESPACE
name: SERVICE_EXPORT_NAMEReplace the following:
NAMESPACE
: the namespace of theServiceExport
object. This namespace must match the namespace of the Service that you are exporting. Services in thedefault
andkube-system
namespaces are not exported across clusters.SERVICE_EXPORT_NAME
: the name of a Service in your cluster that you want to export to other clusters within your fleet.
Create the
ServiceExport
resource by running the following command:kubectl apply -f export.yaml
The initial export of your Service takes approximately five minutes to sync to clusters registered in your fleet. After a Service is exported, subsequent endpoint syncs happen immediately.
You can export the same Service from multiple clusters. Services with the same namespace and same name are treated the same. For the same Service, endpoints from up to five clusters are fully synced across clusters. When there are more than five clusters exporting the same Service, endpoint subsetting happens. Over all consuming clusters, Service endpoints are distributed uniformly.
Consuming cross-cluster Services
MCS supports ClusterSetIP and headless Services. Only DNS "A" records are available.
After you create a ServiceExport
object, the following domain name resolves to
your exported Service from any Pod in any fleet cluster:
SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local
The output includes the following values:
SERVICE_EXPORT_NAME
andNAMESPACE
: the values you define in yourServiceExport
object.
For ClusterSetIP Services, the domain resolves to the ClusterSetIP
. You
can find this value by locating the ServiceImport
object in a cluster in the
namespace that the ServiceExport
object was created in. The ServiceImport
object is automatically created.
For example:
kind: ServiceImport
apiVersion: net.gke.io/v1
metadata:
namespace: EXPORTED-SERVICE-NAMESPACE
name: external-svc-SERVICE-EXPORT-TARGET
status:
ports:
- name: https
port: 443
protocol: TCP
targetPort: 443
ips: CLUSTER_SET_IP
For headless Services, the domain resolves to the list of IP addresses of the endpoints in the exporting clusters. Each backend Pod with a hostname is also independently addressable with a domain name of the following form:
HOSTNAME.MEMBERSHIP_NAME.SERVICE_EXPORT_NAME.NAMESPACE.svc.clusterset.local
The output includes the following values:
SERVICE_EXPORT_NAME
andNAMESPACE
: the values you define in yourServiceExport
object.MEMBERSHIP_NAME
: the membership name in your fleet for the cluster that the Pod is in.HOSTNAME
: the hostname of the Pod.
Disabling MCS
To disable MCS, complete the following steps:
For each cluster in your fleet, delete each ServiceExport object that you created:
kubectl delete serviceexport SERVICE_EXPORT_NAME \
-n NAMESPACEReplace the following:
SERVICE_EXPORT_NAME
: the name of your ServiceExport object.NAMESPACE
: the namespace of theServiceExport
object.
Unregister your clusters from the fleet if they don't need to be registered for another purpose.
Disable the
multiclusterservicediscovery
feature:gcloud container hub multi-cluster-services disable \
--project PROJECT_IDReplace
PROJECT_ID
with the project ID from the project where you registered clusters.Disable the API for MCS:
gcloud services disable multiclusterservicediscovery.googleapis.com \
--project PROJECT_IDReplace
PROJECT_ID
with the project ID from the project where you registered clusters.
Limitations
The following limits are not enforced, and in some cases you can exceed these limits depending on the load in your clusters or project and the rate of endpoint churn. However, you might experience performance issues when these limitations are exceeded.
Exporting clusters: A single Service, identified by a namespaced name, can be safely exported from up to 5 clusters simultaneously. Beyond that limit, it's possible that only a subset of endpoints can be imported to consuming clusters. You can export different Services from different subsets of clusters.
The number of Pods behind a single Service: It's safe if you keep below 250 Pods behind a single Service. This is the same limitation that single cluster Services have. With relatively static workloads and a small number of multi-cluster Services, it might be possible to significantly exceed this number into thousands of endpoints per Service. As with single cluster Services, all endpoints are watched by
kube-proxy
on every node. When going beyond this limit, especially when exporting from multiple clusters simultaneously, larger nodes might be required.The number of multi-cluster Services simultaneously exported: We recommend that you simultaneously export no more than 50 unique Service ports, identified by a Service's namespaced name and declared ports. For example, exporting a Service that exposes ports 80 and 443 would count against 2 of the 50 unique Service port limit. Services with the same namespaced name exported from multiple clusters count as a single unique Service. The previously mentioned 2 port service would still only count against 2 ports if it were exported from 5 clusters simultaneously. Each multi-cluster Service counts toward your Backend Services quota, and each exporting cluster or zone creates a network endpoint group (NEG).
MCS with clusters in multiple projects
You cannot export a service if that service is already being exported by other clusters in a different project in the fleet with the same name and namespace. You can access the service in other clusters in the fleet in other projects, but those clusters cannot export the same service in the same namespace.
Troubleshooting
The following sections provide you with troubleshooting tips for MCS.
Viewing the featureState
Viewing the feature state can help you confirm if MCS was configured successfully. You can view the MCS feature state by using the following command:
gcloud container hub multi-cluster-services describe
The output is similar to the following:
createTime: '2021-08-10T13:05:23.512044937Z'
membershipStates:
projects/PROJECT_ID/locations/global/memberships/MCS_NAME:
state:
code: OK
description: Firewall successfully updated
updateTime: '2021-08-10T13:14:45.173444870Z'
name: projects/PROJECT_NAME/locations/global/features/multiclusterservicediscovery
resourceState:
state: ACTIVE
spec: {}
The most helpful fields for troubleshooting are code
and description
.
Codes in the featureState
A code indicates the member's general state in relation to MCS. You can find
these fields in the state.code
field. There are three possible
codes:
OK
: The membership was successfully added to MCS and is ready to use.WARNING
: MCS is in the process of reconciling membership setup. The description field can provide more information about what caused this code.FAILED
: This membership was not added to MCS. Other memberships in the fleet with anOK
code are not affected by thisFAILED
membership. The description field can provide more information about what caused this code.ERROR
: This membership is missing resources. Other memberships in the fleet with anOK
code are not affected by thisERROR
membership. The description field can provide more information about what caused this code.
Descriptions in the featureState
A description gives you further information about the membership's state in
MCS. You can find these descriptions in the state.description
field and you
can see the following descriptions:
Firewall successfully created
: This message indicates that the member's firewall rule was successfully created and or updated. The membership's code isOK
.Firewall creation pending
: This message indicates that the member's firewall rule is pending creation or update. The membership's code isWARNING
. This membership can experience issues updating and connecting to new multi-cluster Services and memberships added while the firewall rule is pending.GKE Cluster missing
: This message indicates that the member's registered GKE cluster is unavailable and or deleted. The membership's code isERROR
. This membership needs to be manually unregistered from the fleet after a GKE cluster is deleted.Project that member lives in is missing required permissions and/or has not enabled all required APIs - additional setup steps are required
: This message indicates there are internal StatusForbidden (403) errors, and the membership's code isFAILED
. This error occurs in the following scenarios:You have not enabled the necessary APIs in the member's project.
If the member's cluster lives in a separate project than the fleet, see cross-project setup to ensure you have completed all necessary steps. If you have completed all steps, ensure that the following APIs are enabled in the registration project with the following commands:
gcloud services enable multiclusterservicediscovery.googleapis.com --project PROJECT_ID
gcloud services enable dns.googleapis.com --project PROJECT_ID
gcloud services enable trafficdirector.googleapis.com --project PROJECT_ID
gcloud services enable cloudresourcemanager.googleapis.com --project PROJECT_IDReplace
PROJECT_ID
with the project ID from the project where you registered clusters.The
mcsd
orgkehub
service account requires more permissions in the member's project.The
mcsd
andgkehub
service accounts should automatically have been created in the registration project with all the required permissions. To verify the service accounts exist, run the following commands:gcloud projects get-iam-policy PROJECT_ID | grep gcp-sa-mcsd
gcloud projects get-iam-policy PROJECT_ID | grep gcp-sa-gkehubReplace
PROJECT_ID
with the project ID from the project where you registered clusters.
These commands should show you the full name of the
mcsd
andgkehub
service accounts.Multiple VPCs detected in the hub - VPC must be peered with other VPCs for successful connectivity
: This message occurs when clusters hosted in different VPCs are registered to the same fleet. Membership status isOK
. A cluster's VPC network is defined by its NetworkConfig's network. Multi-cluster Services require a flat network, and these VPCs must be actively peered for multi-cluster Services to properly connect to each other. To learn more, see Example VPC Network Peering setup.Member does not exist in the same project as hub - additional setup steps are required, errors may occur if not completed.
: This message reminds you that cross-project clusters require additional setup steps. Membership status isOK
. Cross-project memberships are defined as a member's cluster that does not exist in the same project as the fleet. For more information, see cross-project setup.Non-GKE clusters are currently not supported
: This message reminds you that MCS only supports GKE clusters. Non-GKE clusters cannot be added to MCS. Membership status isFAILED
.
Known issues
MCS with Cloud DNS for in-cluster name resolution
You must not join a fleet if your GKE cluster is configured with Cluster scope DNS or VPC scope DNS as the in-cluster service name resolver. Doing so will result in incorrect name resolution for multi-cluster services.
MCS with Shared VPC
With the current implementation of MCS, if you deploy more than one fleet in the same Shared VPC, metadata are shared between fleets. When a Service is created in one fleet, Service metadata is exported or imported in all other fleets that are part of the same Shared VPC and visible to the user.
This behavior will be fixed in an upcoming release of MCS.
Health check uses default port instead of containerPort
When you deploy a Service with a targetPort
field referencing a named port in
a Deployment, MCS configures the default port for the health check instead of
the specified containerPort
.
To avoid this issue, use numerical values in the Service field
ports.targetPort
and the Deployment field
readinessProbe.httpGet.port
instead of named values.
This behavior will be fixed in an upcoming release of MCS.
What's next
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK