Multi-Cloud Multi-Cluster Kubernetes Orchestration with Karmada

Multi-Cloud Multi-Cluster Kubernetes Orchestration with Karmada

ADM-MIDA0UI
  2023-08-19 09:30 2023-08-19 09:30 Created   2024-05-17 21:05:17 2024-05-17 21:05 Updated    

Introduction

Karmada is a multi-Kubernetes cluster application distribution management system that can create multi-cluster applications just like using Kubernetes native API.

Environment Setup

  • We are going to install Karmada kubernetes plugin into our Kind local cluster (on-prem) to work as our host cluster and then it will be joining two member clusters from two different cloud providers Google Cloud Platform and Digital Ocean.
  • Both members will join the host in push mode (agentless).

Prerequisites

Go (Latest)
Kubectl
Kind

Installation

Create a Kind cluster

1
kind create cluster

Host Machine (always)

1
2
3
4
wget https://github.com/karmada-io/karmada/releases/download/v1.2.0/kubectl-karmada-linux-amd64.tgz

tar -zxf kubectl-karmada-linux-amd64.tgz
sudo cp kubectl-karmada /usr/local/bin/

Initialize the Karmada control plane components on our Kind local cluster

1
kubectl karmada init --kubeconfig=~/.kube/config

Output:

Checking with kubectl -n karmada-system get pods

Create Google Cloud GKE Cluster and get its Kubeconfig

Notice, for GKE Kubeconfig we must install the gcloud cli and the gcloud auth plugin for kubectl in order to be able to authenticate to the GKE cluster, this is only required for GKE as Digital Ocean Cluster uses token-based authentication

gcloud requirements installation

1
2
yay -S google-cloud-cli
yay -S google-cloud-cli-gke-gcloud-auth-plugin

Add the following to your .bashrc or .zshrc

1
2
3
export USE_GKE_GCLOUD_AUTH_PLUGIN=True

source ~/.bashrc

Getting the GKE Kubeconfig file locally

1
gcloud container clusters get-credentials <clustername> --region us-central1 --project <projectname>

Then download the file ~/.kube/config locally and rename it to member1 or like config1

Create Digital Ocean Kubernetes Cluster and get its Kubeconfig

Go to actions and download config locally then rename it as well to member2 or config2

With everything done in enviornment setup, we join both clusters to our host cluster now

Join member clusters

To make names simpler we sed -i 's/do-ams3-k8s-1-27-4-do-0-ams3-1692406742545/docluster/g' config2 and sed -i 's/gke_esoteric-might-387308_us-central1_autopilot-cluster-1/gke-cluster/g' config1 because Karmada doesnt support special characters in names as well as longer than 48-character names.

There are two contexts in Karmada:

  • karmada-apiserver: kubectl config use-context karmada-apiserver
  • karmada-host: kubectl config use-context karmada-host

The karmada-apiserver is the main kubeconfig to be used when interacting with the Karmada control plane, while the karmada-host is only used for debugging Karmada installation with the host cluster.

Remember, although we have the context karmada-apiserver, this still not an actual Kubernetes cluster, rather it is a Karmada control plane API server running inside the Karmada-host Cluster.

Karmada simplifies multi-cluster access through its central access portals on the karmada-apiserver. This allows unified authentication to be managed within the Karmada control plane. Karmada achieves unified authentication using Kubernetes user camouflage feature.

When a cluster is added to Karmada, it generates a ServiceAccount called karmada-impersonator in the cluster’s karmada-cluster namespace. The associated token is then gathered into the Karmada control plane. This ServiceAccount lacks default permissions as it’s not bound to any RBAC.

So, let’s start with the GKE Cluster

Join GKE Cluster

In host cluster, run

1
kubectl karmada --kubeconfig /etc/karmada/karmada-apiserver.config  join gke-cluster --cluster-kubeconfig=~/clusters/config1

This will fail saying “no secret was found for the gke-cluster service account”, because Kubernetes no longer creates a secret token for service account automatically, so we need to create that manually. Karmada needs to use ServiceAccount and Secret to access member clusters.

So, lets make thing more clear, upon running that command, karmada will create a service account inside our GKE cluster, but the thing is, this service account which is named karmada-gke-cluster lacks a secret, and since Kubernetes decided to no longer dealing with that, we have to deal with it manually…

As shown here:

Using GKE config to edit service account and create secrets using kubectl

1
2
3
4
5
$ kubectl --kubeconfig=config2 get sa -n karmada-cluster                                                                
NAME                   SECRETS   AGE
default                0         7h51m
karmada-gke-cluste     0         109m
karmada-impersonator   0         7h51m

so let’s prepare our secret.yaml file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Secret
metadata:
  name: karmada-impersonator
  namespace: karmada-cluster
  annotations:
    kubernetes.io/service-account.name: "karmada-impersonator"
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
  name: gke-cluster-secret
  namespace: karmada-cluster
  annotations:
    kubernetes.io/service-account.name: "karmada-gke-cluster"
type: kubernetes.io/service-account-token

and edit the existing service account to append the secret name at the end

We’re done for GKE, let’s just join it as member now

This is the output to expect

1
2
$ kubectl karmada --kubeconfig /etc/karmada/karmada-apiserver.config  join gke-cluster --cluster-kubeconfig=~/clusters/config1
cluster(ubuntub) is joined successfully

Join Digital Ocean Cluster

In host cluster, run

1
kubectl karmada --kubeconfig /etc/karmada/karmada-apiserver.config  join docluster --cluster-kubeconfig=~/clusters/config2

The same process here, Karmada will create a service account for both the docluster and the karmada-impersonator, and both will be lacking secrets.

Using Digital Ocean config to edit service account and create secrets using kubectl

Secret file here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: v1
kind: Secret
metadata:
  name: karmada-impersonator
  namespace: karmada-cluster
  annotations:
    kubernetes.io/service-account.name: "karmada-impersonator"
type: kubernetes.io/service-account-token
---
apiVersion: v1
kind: Secret
metadata:
  name: docluster-secret
  namespace: karmada-cluster
  annotations:
    kubernetes.io/service-account.name: "karmada-docluster"
type: kubernetes.io/service-account-token

View member clusters

Deploy Nginx application

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
$ cat nginx-deployment.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
$ kubectl  --kubeconfig /etc/karmada/karmada-apiserver.config apply -f nginx-deployment.yaml

Karmada, works on Kubernetes manifests as they are, but requires a distribution strategy and specification of member clusters, so let’s create PropagationPolicy.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
    - apiVersion: v1
      kind: Service
      name: nginx
  placement:
    clusterAffinity:
      clusterNames:
        - ubuntub
        - ubuntuc
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - ubuntub
            weight: 1
          - targetCluster:
              clusterNames:
                - ubuntuc
            weight: 1

The same could be achieved using a simpler strategy to duplicate across all the member clusters.

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: nginx-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: nginx
  placement:
    replicaScheduling:
      replicaSchedulingType: Duplicated

So when we apply any of the above PropagationPolicies, both member clusters will have nginx deployment with two replicas.

1
kubectl  --kubeconfig /etc/karmada/karmada-apiserver.config apply -f PropagationPolicy.yaml

Application Deployment Status

In Control Cluster

Notice there are no pods here!

In Member Clusters

What to check next?

Implemeting this whole logic in a kubernetes custom controller using Kubebuilder or client-go

  • Title: Multi-Cloud Multi-Cluster Kubernetes Orchestration with Karmada
  • Author: ADM-MIDA0UI
  • Created at: 2023-08-19 09:30:00
  • Updated at: 2024-05-17 21:05:17
  • Link: https://admida0ui.de/2023/08/19/karmada/
  • License: This work is licensed under CC BY-NC-SA 4.0.
On this page
Multi-Cloud Multi-Cluster Kubernetes Orchestration with Karmada
  1. Introduction
  2. Environment Setup
    1. Prerequisites
    2. Installation
    3. Initialize the Karmada control plane components on our Kind local cluster
    4. Create Google Cloud GKE Cluster and get its Kubeconfig
    5. Create Digital Ocean Kubernetes Cluster and get its Kubeconfig
  3. Join member clusters
    1. Join GKE Cluster
    2. Join Digital Ocean Cluster
    3. View member clusters
  4. Deploy Nginx application
  5. Application Deployment Status
    1. In Control Cluster
    2. In Member Clusters
  6. What to check next?