Operators, operators everywhere

As you may have noticed, Kubernetes operators are becoming more an more popular those days. In this post we are going to explain the basics around Operators and we will develop a simple Operator using the Operator Framework SDK.

What is an Operator

An operator aims to automate actions usually performed manually while lessening the likelihood of error and simplifying complexity.

We can think of an operator as a method of packaging, deploying and managing a Kubernetes enabled application. Kubernetes enabled applications are deployed on Kubernetes and managed using the Kubernetes APIs and tooling.

Kubernetes APIs can be extended in order to enable new types of Kubernetes enabled applications. We could say that Operators are the runtime that manages such applications.

A simple Operator would define how to deploy an application, whereas an advanced one will also take care of day-2 operations like backup, upgrades, etc.

Operators use the Controller pattern, but not all Controllers are Operators. We could say it’s an Operator if it’s got:

  • Controller Pattern
  • API Extension
  • Single-App Focus

Feel free to read more about operators on the Operator FAQ by CoreOS

Kubernetes Controllers

In the Kubernetes world, the Controllers take care of routine tasks to ensure cluster’s observed state, matches cluster’s desired state.

Each Controller is responsible for a particular resource in Kubernetes. The Controller runs a control loop that watches the shared state of the cluster through the Kubernetes API server and makes changes attempting to move the current state towards the desired state.

Some examples:

  • Replication Controller
  • Cronjob Controller

Controller Components

There are two main components in a controller: Informer/SharedInformer and WorkQueue.

Informer

In order to retrieve information about an object, the Controller sends a request to the Kubernetes API server. However, querying the API repeatedly can become expensive when dealing with thousands of objects.

On top of that, the Controller doesn’t really need to send requests continuously. It only cares about CRUD events happening on the objects it’s managing.

Informers are not much used in the current Kubernetes, instead SharedInformers are used.

SharedInformer

A Informer creates a local cache for a set of resources used by itself. In Kubernetes there are multiple controllers running and caring about multiple kinds of resources though.

Having a shared cache among Controllers instead of one cache for each Controller sounds like a plan, that’s a SharedInformer.

WorkQueue

The SharedInformer can’t track what each Controller is up to, so the Controller must provide its own queuing and retrying mechanism.

Whenever a resource changes, the SharedInformer’s Event Handler puts a key into the WorkQueue so the Controller will take care of that change.

How a Controller Works

Control Loop

Every controller has a Control Loop which basically does:

  1. Processes every single item from the WorkQueue
  2. Pops an item and do whatever it needs to do with that item
  3. Pushes the item back to the WorkQueue if required
  4. Updates the item status to reflect the new changes
  5. Starts over

Code Examples

WorkQueue

  1. Stuff is put into the WorkQueue
  2. Stuff is take out from the WorkQueue in the Control Loop
  3. WorkQueue doesn’t store objects, it stores MetaNamespaceKeys

A MetaNamespaceKey is a key-value reference for an object. It has the namespace for the resource and the name for the resource.

Code Examples

SharedInformer

As we said before, is a shared data cache which distributes the data to all the Listers interested in knowing about changes happening to specific objects.

The most important part of the SharedInformer are the EventHandlers. Using an EventHandler is how you register your interest in specific object updates like addition, creation, updation or deletion.

When an update occurs, the object will be put into the WorkQueue so it gets processed by the Controller in the Control Loop.

Listers are an important part of the SharedInformers as well. Listers are designed specifically to be used within Controllers as they have access to the cache.

Listers vs Client-go

Listers have access to the cache whereas Client-go will hit the Kubernetes API server (which is expensive when dealing with thousands of objects).

Code Examples

SyncHandler A.K.A Reconciliation Loop

The first invocation of the SyncHandler will always be getting the MetaNamespaceKey for the resource it needs to work with.

With the MetaNamespaceKey the object is gathered from the cache, but well.. it’s not really an object, but a pointer to the cached object.

With the object reference we can read the object, in case the object needs to be updated, then the object have to be DeepCopied. DeepCopy is an expensive operation, making sure the object will be modified before calling DeepCopy is a good practice.

With the object reference / DeepCopy we are ready to apply our business logic.

Code Examples

Kubernetes Controllers

Some information about controllers:

  • Cronjob controller is probably the smallest one out there
  • Sample Controller will help you getting started with Kubernetes Controllers

Writing your very first Operator using the Operator Framework SDK

We will create a very simple Operator using the Operator Framework SDK.

The Operator will be in charge of deploying a simple GoLang application.

Requirements

At the moment of this writing the following versions were used:

  • golang-1.19.5
  • Operator Framework SDK v1.26.1
  • Kubernetes 1.24

Installing the Operator Framework SDK

RELEASE_VERSION=v1.26.1
# Linux
sudo curl -L https://github.com/operator-framework/operator-sdk/releases/download/${RELEASE_VERSION}/operator-sdk_linux_amd64 -o /usr/local/bin/operator-sdk
sudo chmod +x /usr/local/bin/operator-sdk

Initializing the Operator Project

First, a new new project for our Operator will be initialized.

mkdir -p ~/operators-projects/reverse-words-operator && cd $_
export GO111MODULE=on
export GOPROXY=https://proxy.golang.org
export GH_USER=<github_user>
operator-sdk init --domain=linuxera.org --repo=github.com/$GH_USER/reverse-words-operator

Create the Operator API Types

As previously discussed, Operators extend the Kubernetes API, the API itself is organized in groups and versions. Our Operator will define a new Group, object Kind and its versioning.

In the example below we will define a new API Group called apps under domain linuxera.org, a new object Kind ReverseWordsApp and its versioning v1alpha1.

operator-sdk create api --group=apps --version=v1alpha1 --kind=ReverseWordsApp --resource=true --controller=true

Now it’s time to define the structure of our new Object. The Spec properties that we will be using are:

  • replicas: Will be used to define the number of replicas for our application
  • appVersion: Will be used to define which version of the application is deployed

In the Status we will use:

  • appPods: Will track the pods associated to our current ReverseWordsApp instance
  • Different conditions

Below the code for our Types:

/*


Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// ReverseWordsAppSpec defines the desired state of ReverseWordsApp
type ReverseWordsAppSpec struct {
	Replicas   int32  `json:"replicas"`
	AppVersion string `json:"appVersion,omitempty"`
}

// ReverseWordsAppStatus defines the observed state of ReverseWordsApp
type ReverseWordsAppStatus struct {
	AppPods    []string           `json:"appPods"`
	Conditions []metav1.Condition `json:"conditions"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// ReverseWordsApp is the Schema for the reversewordsapps API
type ReverseWordsApp struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   ReverseWordsAppSpec   `json:"spec,omitempty"`
	Status ReverseWordsAppStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// ReverseWordsAppList contains a list of ReverseWordsApp
type ReverseWordsAppList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items           []ReverseWordsApp `json:"items"`
}

func init() {
	SchemeBuilder.Register(&ReverseWordsApp{}, &ReverseWordsAppList{})
}

// Conditions
const (
	// ConditionTypeReverseWordsDeploymentNotReady indicates if the Reverse Words Deployment is not ready

	ConditionTypeReverseWordsDeploymentNotReady string = "ReverseWordsDeploymentNotReady"

	// ConditionTypeReady indicates if the Reverse Words Deployment is ready
	ConditionTypeReady string = "Ready"
)

You can download the Types file:

curl -Ls https://linuxera.org/writing-operators-using-operator-framework/reversewordsapp_types.go -o ~/operators-projects/reverse-words-operator/api/v1alpha1/reversewordsapp_types.go

Replicas will be defined as an int32 and will reference the Spec property replicas. For the status AppPods will be defined as a stringList and will reference the Status property appPods.

With above changes in-place we need to add new dependencies and re-generate some boilerplate code to take into account the latest changes in our types.

go mod tidy
make manifests
make generate

Code your Operator business logic

An empty controller (well, not that empty) has been created into our project, now it’s time to modify it so it actually deploys our application the way we want.

Our application consists of a Deployment and a Service, so our Operator will deploy the Reverse Words App as follows:

  1. A Kubernetes Deployment object will be created
  2. A Kubernetes Service object will be created

Below code (commented) for our Controller:

/*


Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package controllers

import (
	"context"
	"reflect"

	"github.com/go-logr/logr"
	appsv1alpha1 "github.com/mvazquezc/reverse-words-operator/api/v1alpha1"
	appsv1 "k8s.io/api/apps/v1"
	corev1 "k8s.io/api/core/v1"
	"k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/api/meta"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/apimachinery/pkg/util/intstr"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
	"sigs.k8s.io/controller-runtime/pkg/event"
	ctrllog "sigs.k8s.io/controller-runtime/pkg/log"
	"sigs.k8s.io/controller-runtime/pkg/predicate"
)

// ReverseWordsAppReconciler reconciles a ReverseWordsApp object
type ReverseWordsAppReconciler struct {
	client.Client
	Scheme *runtime.Scheme
}

const (
	// Finalizer for our objects
	reverseWordsAppFinalizer = "finalizer.reversewordsapp.apps.linuxera.org"
	concurrentReconciles     = 10
)

// +kubebuilder:rbac:groups=apps.linuxera.org,resources=reversewordsapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.linuxera.org,resources=reversewordsapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.linuxera.org,resources=reversewordsapps/finalizers,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch
// +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch

func (r *ReverseWordsAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := ctrllog.FromContext(ctx)

	// Fetch the ReverseWordsApp instance
	instance := &appsv1alpha1.ReverseWordsApp{}
	err := r.Get(ctx, req.NamespacedName, instance)
	if err != nil {
		if errors.IsNotFound(err) {
			// Request object not found, could have been deleted after reconcile request.
			// Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
			// Return and don't requeue
			log.Info("ReverseWordsApp resource not found. Ignoring since object must be deleted")
			return ctrl.Result{}, nil
		}
		// Error reading the object - requeue the request.
		log.Error(err, "Failed to get ReverseWordsApp")
		return ctrl.Result{}, err
	}

	// Check if the CR is marked to be deleted
	isInstanceMarkedToBeDeleted := instance.GetDeletionTimestamp() != nil
	if isInstanceMarkedToBeDeleted {
		log.Info("Instance marked for deletion, running finalizers")
		if contains(instance.GetFinalizers(), reverseWordsAppFinalizer) {
			// Run the finalizer logic
			err := r.finalizeReverseWordsApp(log, instance)
			if err != nil {
				// Don't remove the finalizer if we failed to finalize the object
				return ctrl.Result{}, err
			}
			log.Info("Instance finalizers completed")
			// Remove finalizer once the finalizer logic has run
			controllerutil.RemoveFinalizer(instance, reverseWordsAppFinalizer)
			err = r.Update(ctx, instance)
			if err != nil {
				// If the object update fails, requeue
				return ctrl.Result{}, err
			}
		}
		log.Info("Instance can be deleted now")
		return ctrl.Result{}, nil
	}

	// Add Finalizers to the CR
	if !contains(instance.GetFinalizers(), reverseWordsAppFinalizer) {
		if err := r.addFinalizer(log, instance); err != nil {
			return ctrl.Result{}, err
		}
	}

	// Reconcile Deployment object
	result, err := r.reconcileDeployment(instance, log)
	if err != nil {
		return result, err
	}
	// Reconcile Service object
	result, err = r.reconcileService(instance, log)
	if err != nil {
		return result, err
	}

	// The CR status is updated in the Deployment reconcile method

	return ctrl.Result{}, nil
}

func (r *ReverseWordsAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&appsv1alpha1.ReverseWordsApp{}).
		Owns(&appsv1.Deployment{}).
		Owns(&corev1.Service{}).
		WithEventFilter(ignoreDeletionPredicate()).                                     // Filter events that do not increase generation id, like status updates
		WithOptions(controller.Options{MaxConcurrentReconciles: concurrentReconciles}). // run multiple reconcile loops in parallel
		Complete(r)
}

func (r *ReverseWordsAppReconciler) reconcileDeployment(cr *appsv1alpha1.ReverseWordsApp, log logr.Logger) (ctrl.Result, error) {
	// Define a new Deployment object
	deployment := newDeploymentForCR(cr)

	// Set ReverseWordsApp instance as the owner and controller of the Deployment
	if err := ctrl.SetControllerReference(cr, deployment, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}

	// Check if this Deployment already exists
	deploymentFound := &appsv1.Deployment{}
	err := r.Get(context.Background(), types.NamespacedName{Name: deployment.Name, Namespace: deployment.Namespace}, deploymentFound)
	if err != nil && errors.IsNotFound(err) {
		log.Info("Creating a new Deployment", "Deployment.Namespace", deployment.Namespace, "Deployment.Name", deployment.Name)
		err = r.Create(context.Background(), deployment)
		if err != nil {
			return ctrl.Result{}, err
		}
		// Requeue the object to update its status
		return ctrl.Result{Requeue: true}, nil
	} else if err != nil {
		return ctrl.Result{}, err
	} else {
		// Deployment already exists
		log.Info("Deployment already exists", "Deployment.Namespace", deploymentFound.Namespace, "Deployment.Name", deploymentFound.Name)
	}

	// Ensure deployment replicas match the desired state
	if !reflect.DeepEqual(deploymentFound.Spec.Replicas, deployment.Spec.Replicas) {
		log.Info("Current deployment replicas do not match ReverseWordsApp configured Replicas")
		// Update the replicas
		err = r.Update(context.Background(), deployment)
		if err != nil {
			log.Error(err, "Failed to update Deployment.", "Deployment.Namespace", deploymentFound.Namespace, "Deployment.Name", deploymentFound.Name)
			return ctrl.Result{}, err
		}
	}
	// Ensure deployment container image match the desired state, returns true if deployment needs to be updated
	if checkDeploymentImage(deploymentFound, deployment) {
		log.Info("Current deployment image version do not match ReverseWordsApp configured version")
		// Update the image
		err = r.Update(context.Background(), deployment)
		if err != nil {
			log.Error(err, "Failed to update Deployment.", "Deployment.Namespace", deploymentFound.Namespace, "Deployment.Name", deploymentFound.Name)
			return ctrl.Result{}, err
		}
	}

	// Check if the deployment is ready
	deploymentReady := isDeploymentReady(deploymentFound)

	// Create list options for listing deployment pods
	podList := &corev1.PodList{}
	listOpts := []client.ListOption{
		client.InNamespace(deploymentFound.Namespace),
		client.MatchingLabels(deploymentFound.Labels),
	}
	// List the pods for this ReverseWordsApp deployment
	err = r.List(context.Background(), podList, listOpts...)
	if err != nil {
		log.Error(err, "Failed to list Pods.", "Deployment.Namespace", deploymentFound.Namespace, "Deployment.Name", deploymentFound.Name)
		return ctrl.Result{}, err
	}
	// Get running Pods from listing above (if any)
	podNames := getRunningPodNames(podList.Items)
	if deploymentReady {
		// Update the status to ready
		cr.Status.AppPods = podNames
		meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{Type: appsv1alpha1.ConditionTypeReverseWordsDeploymentNotReady, Status: metav1.ConditionFalse, Reason: appsv1alpha1.ConditionTypeReverseWordsDeploymentNotReady})
		meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{Type: appsv1alpha1.ConditionTypeReady, Status: metav1.ConditionTrue, Reason: appsv1alpha1.ConditionTypeReady})
	} else {
		// Update the status to not ready
		cr.Status.AppPods = podNames
		meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{Type: appsv1alpha1.ConditionTypeReverseWordsDeploymentNotReady, Status: metav1.ConditionTrue, Reason: appsv1alpha1.ConditionTypeReverseWordsDeploymentNotReady})
		meta.SetStatusCondition(&cr.Status.Conditions, metav1.Condition{Type: appsv1alpha1.ConditionTypeReady, Status: metav1.ConditionFalse, Reason: appsv1alpha1.ConditionTypeReady})
	}
	// Reconcile the new status for the instance
	cr, err = r.updateReverseWordsAppStatus(cr, log)
	if err != nil {
		log.Error(err, "Failed to update ReverseWordsApp Status.")
		return ctrl.Result{}, err
	}
	// Deployment reconcile finished
	return ctrl.Result{}, nil
}

// updateReverseWordsAppStatus updates the Status of a given CR
func (r *ReverseWordsAppReconciler) updateReverseWordsAppStatus(cr *appsv1alpha1.ReverseWordsApp, log logr.Logger) (*appsv1alpha1.ReverseWordsApp, error) {
	reverseWordsApp := &appsv1alpha1.ReverseWordsApp{}
	err := r.Get(context.Background(), types.NamespacedName{Name: cr.Name, Namespace: cr.Namespace}, reverseWordsApp)
	if err != nil {
		return reverseWordsApp, err
	}

	if !reflect.DeepEqual(cr.Status, reverseWordsApp.Status) {
		log.Info("Updating ReverseWordsApp Status.")
		// We need to update the status
		err = r.Status().Update(context.Background(), cr)
		if err != nil {
			return cr, err
		}
		updatedReverseWordsApp := &appsv1alpha1.ReverseWordsApp{}
		err = r.Get(context.Background(), types.NamespacedName{Name: cr.Name, Namespace: cr.Namespace}, updatedReverseWordsApp)
		if err != nil {
			return cr, err
		}
		cr = updatedReverseWordsApp.DeepCopy()
	}
	return cr, nil

}

// addFinalizer adds a given finalizer to a given CR
func (r *ReverseWordsAppReconciler) addFinalizer(log logr.Logger, cr *appsv1alpha1.ReverseWordsApp) error {
	log.Info("Adding Finalizer for the ReverseWordsApp")
	controllerutil.AddFinalizer(cr, reverseWordsAppFinalizer)

	// Update CR
	err := r.Update(context.Background(), cr)
	if err != nil {
		log.Error(err, "Failed to update ReverseWordsApp with finalizer")
		return err
	}
	return nil
}

// finalizeReverseWordsApp runs required tasks before deleting the objects owned by the CR
func (r *ReverseWordsAppReconciler) finalizeReverseWordsApp(log logr.Logger, cr *appsv1alpha1.ReverseWordsApp) error {
	// TODO(user): Add the cleanup steps that the operator
	// needs to do before the CR can be deleted. Examples
	// of finalizers include performing backups and deleting
	// resources that are not owned by this CR, like a PVC.
	log.Info("Successfully finalized ReverseWordsApp")
	return nil
}

func (r *ReverseWordsAppReconciler) reconcileService(cr *appsv1alpha1.ReverseWordsApp, log logr.Logger) (ctrl.Result, error) {
	// Define a new Service object
	service := newServiceForCR(cr)

	// Set ReverseWordsApp instance as the owner and controller of the Service
	if err := controllerutil.SetControllerReference(cr, service, r.Scheme); err != nil {
		return ctrl.Result{}, err
	}

	// Check if this Service already exists
	serviceFound := &corev1.Service{}
	err := r.Get(context.Background(), types.NamespacedName{Name: service.Name, Namespace: service.Namespace}, serviceFound)
	if err != nil && errors.IsNotFound(err) {
		log.Info("Creating a new Service", "Service.Namespace", service.Namespace, "Service.Name", service.Name)
		err = r.Create(context.Background(), service)
		if err != nil {
			return ctrl.Result{}, err
		}
		// Service created successfully - don't requeue
		return ctrl.Result{}, nil
	} else if err != nil {
		return ctrl.Result{}, err
	} else {
		// Service already exists
		log.Info("Service already exists", "Service.Namespace", serviceFound.Namespace, "Service.Name", serviceFound.Name)
	}
	// Service reconcile finished
	return ctrl.Result{}, nil
}

// Returns a new deployment without replicas configured
// replicas will be configured in the sync loop
func newDeploymentForCR(cr *appsv1alpha1.ReverseWordsApp) *appsv1.Deployment {
	labels := map[string]string{
		"app": cr.Name,
	}
	replicas := cr.Spec.Replicas
	// Minimum replicas will be 1
	if replicas == 0 {
		replicas = 1
	}
	appVersion := "latest"
	if cr.Spec.AppVersion != "" {
		appVersion = cr.Spec.AppVersion
	}
	// TODO:Check if application version exists
	containerImage := "quay.io/mavazque/reversewords:" + appVersion
	probe := &corev1.Probe{
		ProbeHandler: corev1.ProbeHandler{
			HTTPGet: &corev1.HTTPGetAction{
				Path: "/health",
				Port: intstr.FromInt(8080),
			},
		},
		InitialDelaySeconds: 5,
		TimeoutSeconds:      2,
		PeriodSeconds:       15,
	}
	return &appsv1.Deployment{
		TypeMeta: metav1.TypeMeta{
			APIVersion: "apps/v1",
			Kind:       "Deployment",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      "dp-" + cr.Name,
			Namespace: cr.Namespace,
			Labels:    labels,
		},
		Spec: appsv1.DeploymentSpec{
			Replicas: &replicas,
			Selector: &metav1.LabelSelector{
				MatchLabels: labels,
			},
			Template: corev1.PodTemplateSpec{
				ObjectMeta: metav1.ObjectMeta{
					Labels: labels,
				},
				Spec: corev1.PodSpec{
					Containers: []corev1.Container{
						{
							Image: containerImage,
							Name:  "reversewords",
							Ports: []corev1.ContainerPort{
								{
									ContainerPort: 8080,
									Name:          "reversewords",
								},
							},
							LivenessProbe:  probe,
							ReadinessProbe: probe,
						},
					},
				},
			},
		},
	}
}

// Returns a new service
func newServiceForCR(cr *appsv1alpha1.ReverseWordsApp) *corev1.Service {
	labels := map[string]string{
		"app": cr.Name,
	}
	return &corev1.Service{
		TypeMeta: metav1.TypeMeta{
			APIVersion: "v1",
			Kind:       "Service",
		},
		ObjectMeta: metav1.ObjectMeta{
			Name:      "service-" + cr.Name,
			Namespace: cr.Namespace,
			Labels:    labels,
		},
		Spec: corev1.ServiceSpec{
			Type:     corev1.ServiceTypeLoadBalancer,
			Selector: labels,
			Ports: []corev1.ServicePort{
				{
					Name: "http",
					Port: 8080,
				},
			},
		},
	}
}

// isDeploymentReady returns a true bool if the deployment has all its pods ready
func isDeploymentReady(deployment *appsv1.Deployment) bool {
	configuredReplicas := deployment.Status.Replicas
	readyReplicas := deployment.Status.ReadyReplicas
	deploymentReady := false
	if configuredReplicas == readyReplicas {
		deploymentReady = true
	}
	return deploymentReady
}

// getRunningPodNames returns the pod names for the pods running in the array of pods passed in
func getRunningPodNames(pods []corev1.Pod) []string {
	// Create an empty []string, so if no podNames are returned, instead of nil we get an empty slice
	var podNames []string = make([]string, 0)
	for _, pod := range pods {
		if pod.GetObjectMeta().GetDeletionTimestamp() != nil {
			continue
		}
		if pod.Status.Phase == corev1.PodPending || pod.Status.Phase == corev1.PodRunning {
			podNames = append(podNames, pod.Name)
		}
	}
	return podNames
}

// checkDeploymentImage returns wether the deployment image is different or not
func checkDeploymentImage(current *appsv1.Deployment, desired *appsv1.Deployment) bool {
	for _, curr := range current.Spec.Template.Spec.Containers {
		for _, des := range desired.Spec.Template.Spec.Containers {
			// Only compare the images of containers with the same name
			if curr.Name == des.Name {
				if curr.Image != des.Image {
					return true
				}
			}
		}
	}
	return false
}

// contains returns true if a string is found on a slice
func contains(list []string, s string) bool {
	for _, v := range list {
		if v == s {
			return true
		}
	}
	return false
}

// Ignore changes that do not increase the resource generation
func ignoreDeletionPredicate() predicate.Predicate {
	return predicate.Funcs{
		UpdateFunc: func(e event.UpdateEvent) bool {
			// Ignore updates to CR status in which case metadata.Generation does not change
			return e.ObjectOld.GetGeneration() != e.ObjectNew.GetGeneration()
		},
		DeleteFunc: func(e event.DeleteEvent) bool {
			// Evaluates to false if the object has been confirmed deleted.
			return !e.DeleteStateUnknown
		},
	}
}

You can download the controller code, remember to change the GitHub ID before bulding the operator:

# Remember to change import: appsv1alpha1 "github.com/mvazquezc/reverse-words-operator/api/v1alpha1"

curl -Ls https://linuxera.org/writing-operators-using-operator-framework/reversewordsapp_controller.go -o ~/operators-projects/reverse-words-operator/controllers/reversewordsapp_controller.go

Setup Watch namespaces

By default, the controller will watch all namespaces, in this case we want it to watch only the namespace where it runs, in order to do so we need to update the controller options in the main.go file.

/*
Copyright 2021.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

package main

import (
	"flag"
	"fmt"
	"os"

	// Import all Kubernetes client auth plugins (e.g. Azure, GCP, OIDC, etc.)
	// to ensure that exec-entrypoint and run can make use of them.
	_ "k8s.io/client-go/plugin/pkg/client/auth"

	"k8s.io/apimachinery/pkg/runtime"
	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
	clientgoscheme "k8s.io/client-go/kubernetes/scheme"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/healthz"
	"sigs.k8s.io/controller-runtime/pkg/log/zap"

	appsv1alpha1 "github.com/mvazquezc/reverse-words-operator/api/v1alpha1"
	"github.com/mvazquezc/reverse-words-operator/controllers"
	//+kubebuilder:scaffold:imports
)

var (
	scheme   = runtime.NewScheme()
	setupLog = ctrl.Log.WithName("setup")
)

func init() {
	utilruntime.Must(clientgoscheme.AddToScheme(scheme))

	utilruntime.Must(appsv1alpha1.AddToScheme(scheme))
	//+kubebuilder:scaffold:scheme
}

func main() {
	var metricsAddr string
	var enableLeaderElection bool
	var probeAddr string
	flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
	flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
	flag.BoolVar(&enableLeaderElection, "leader-elect", false,
		"Enable leader election for controller manager. "+
			"Enabling this will ensure there is only one active controller manager.")
	opts := zap.Options{
		Development: true,
	}
	opts.BindFlags(flag.CommandLine)
	flag.Parse()

	ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
	watchNamespace, err := getWatchNamespace()
	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:                 scheme,
		MetricsBindAddress:     metricsAddr,
		Port:                   9443,
		HealthProbeBindAddress: probeAddr,
		LeaderElection:         enableLeaderElection,
		LeaderElectionID:       "1ef59d40.linuxera.org",
		Namespace:              watchNamespace, // namespaced-scope when the value is not an empty string
	})
	if err != nil {
		setupLog.Error(err, "unable to start manager")
		os.Exit(1)
	}

	if err = (&controllers.ReverseWordsAppReconciler{
		Client: mgr.GetClient(),
		Scheme: mgr.GetScheme(),
	}).SetupWithManager(mgr); err != nil {
		setupLog.Error(err, "unable to create controller", "controller", "ReverseWordsApp")
		os.Exit(1)
	}
	//+kubebuilder:scaffold:builder

	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up health check")
		os.Exit(1)
	}
	if err := mgr.AddReadyzCheck("readyz", healthz.Ping); err != nil {
		setupLog.Error(err, "unable to set up ready check")
		os.Exit(1)
	}

	setupLog.Info("starting manager")
	if err := mgr.Start(ctrl.SetupSignalHandler()); err != nil {
		setupLog.Error(err, "problem running manager")
		os.Exit(1)
	}
}

// getWatchNamespace returns the Namespace the operator should be watching for changes
func getWatchNamespace() (string, error) {
	// WatchNamespaceEnvVar is the constant for env variable WATCH_NAMESPACE
	// which specifies the Namespace to watch.
	// An empty value means the operator is running with cluster scope.
	var watchNamespaceEnvVar = "WATCH_NAMESPACE"

	ns, found := os.LookupEnv(watchNamespaceEnvVar)
	if !found {
		return "", fmt.Errorf("%s must be set", watchNamespaceEnvVar)
	}
	return ns, nil
}

You can download the main.go, remember to change the GitHub ID before bulding the operator:

# Remember to change import: appsv1alpha1 "github.com/mvazquezc/reverse-words-operator/api/v1alpha1"

curl -Ls https://linuxera.org/writing-operators-using-operator-framework/main.go -o ~/operators-projects/reverse-words-operator/main.go

Specify permissions and generate RBAC manifests

Our controller needs some RBAC permissions to interact with the resources it manages. These has been specified via RBAC Markers in our controller code:

// +kubebuilder:rbac:groups=apps.linuxera.org,resources=reversewordsapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps.linuxera.org,resources=reversewordsapps/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps.linuxera.org,resources=reversewordsapps/finalizers,verbs=get;update;patch
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list;watch
// +kubebuilder:rbac:groups=core,resources=events,verbs=create;patch

func (r *ReverseWordsAppReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {

The ClusterRole manifest at config/rbac/role.yaml is generated from the above markers via controller-gen with the following command:

go mod tidy
make manifests

Build the Operator

First, we will build the operator and once the image is built, we will push it to the Quay Registry.

Before we start building the operator, we need access to a Kubernetes cluster. If you don’t have one you can use Kind, Minikube or my prefered one, KCli

In order to get a local cluster with KCli, just run this command:

kcli create kube generic -P ctlplanes=1 -P workers=1 -P ctlplane_memory=4096 -P numcpus=2 -P worker_memory=4096 -P sdn=calico -P version=1.24 -P ingress=true -P ingress_method=nginx -P metallb=true -P domain=linuxera.org operatorscluster

Now that we have the cluster up and running we will build and push the operator.

NOTE: If you use podman instead of docker you can edit the Makefile and change docker commands by podman commands

export USERNAME=<quay-username>
make docker-build docker-push IMG=quay.io/$USERNAME/reversewords-operator:v0.0.1

Deploy the Operator

  1. Create the required CRDs in the cluster

    make install
    
  2. Deploy the operator

    NOTE: While developing you can run the operator locally (you need a valid kubeconfig) by running make run

    In order to deploy the different operator pieces, Kustomize is used. There is a Kustomization file (~/operators-projects/reverse-words-operator/config/default/kustomization.yaml) where you can define some defaults for your operator, like the namePrefix for the different objects or the namespace where it will be deployed.

    1. Edit the default kustomization file ~/operators-projects/reverse-words-operator/config/default/kustomization.yaml and specify the namespace where your operator should run by modifying the namespace property

      export NAMESPACE=operators-test
      sed -i "s/namespace: .*/namespace: $NAMESPACE/g" ~/operators-projects/reverse-words-operator/config/default/kustomization.yaml
      
    2. Create the namespace and Deploy the operator

      kubectl create ns $NAMESPACE
      export USERNAME=<quay_username>
      make deploy IMG=quay.io/$USERNAME/reversewords-operator:v0.0.1
      
    3. Patch the controller deployment so it only watches the namespace where it’s running

      kubectl -n $NAMESPACE patch deployment reverse-words-operator-controller-manager -p '{"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"kube-rbac-proxy"},{"name":"manager"}],"containers":[{"env":[{"name":"WATCH_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"name":"manager"}]}}}}'
      
  3. We should see our operator pod up and running

    1.676311934753094e+09	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "127.0.0.	1:8080"}
    1.6763119347534144e+09	INFO	setup	starting manager
    I0213 18:12:14.753860       1 leaderelection.go:248] attempting to acquire leader lease operators-test/1ef59d40.	linuxera.org...
    1.6763119347538702e+09	INFO	Starting server	{"kind": "health probe", "addr": "[::]:8081"}
    1.6763119347539454e+09	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "127.0.0.1:8080"}
    I0213 18:12:37.253145       1 leaderelection.go:258] successfully acquired lease operators-test/1ef59d40.linuxera.org
    1.6763119572531986e+09	DEBUG	events		reverse-words-operator-controller-manager-755f55cffc-9wprb_028ea174-80b7-4294-abfb-1d1aae892cd7 became leader		{"type": "Normal", "object": {"kind":"Lease","namespace":"operators-test","name":"1ef59d40.linuxera.org",	"uid":"ae3f7dff-1624-47b2-9010-c666d559473d","apiVersion":"coordination.k8s.io/v1","resourceVersion":"11102046"}, 	"reason": "LeaderElection"}
    1.6763119572534668e+09	INFO	Starting EventSource	{"controller": "reversewordsapp", "controllerGroup": "apps.	linuxera.org", "controllerKind": "ReverseWordsApp", "source": "kind source: *v1alpha1.ReverseWordsApp"}
    1.6763119572535148e+09	INFO	Starting EventSource	{"controller": "reversewordsapp", "controllerGroup": "apps.	linuxera.org", "controllerKind": "ReverseWordsApp", "source": "kind source: *v1.Deployment"}
    1.6763119572535224e+09	INFO	Starting EventSource	{"controller": "reversewordsapp", "controllerGroup": "apps.	linuxera.org", "controllerKind": "ReverseWordsApp", "source": "kind source: *v1.Service"}
    1.6763119572535274e+09	INFO	Starting Controller	{"controller": "reversewordsapp", "controllerGroup": "apps.	linuxera.org", "controllerKind": "ReverseWordsApp"}
    1.6763119573569405e+09	INFO	Starting workers	{"controller": "reversewordsapp", "controllerGroup": "apps.	linuxera.org", "controllerKind": "ReverseWordsApp", "worker count": 10}
    
  4. Now it’s time to create ReverseWordsApp instances

    cat <<EOF | kubectl -n $NAMESPACE create -f -
    apiVersion: apps.linuxera.org/v1alpha1
    kind: ReverseWordsApp
    metadata:
        name: example-reversewordsapp
    spec:
        replicas: 1
    EOF
    
    cat <<EOF | kubectl -n $NAMESPACE create -f -
    apiVersion: apps.linuxera.org/v1alpha1
    kind: ReverseWordsApp
    metadata:
        name: example-reversewordsapp-2
    spec:
        replicas: 2
    EOF
    
  5. We should see two deployments and services being created, and if wee look at the status of our object we should see the pods backing the instance

    kubectl -n $NAMESPACE get reversewordsapps example-reversewordsapp -o yaml
    
    apiVersion: apps.linuxera.org/v1alpha1
    kind: ReverseWordsApp
    metadata:
      creationTimestamp: "2023-02-13T18:13:48Z"
      finalizers:
      - finalizer.reversewordsapp.apps.linuxera.org
      generation: 1
      name: example-reversewordsapp
      namespace: operators-test
      resourceVersion: "11103763"
      uid: ef9351b2-cd44-47b0-ba66-8143f06267dc
    spec:
      replicas: 1
    status:
      appPods:
      - dp-example-reversewordsapp-75cff95fd8-5qvm9
      conditions:
      - lastTransitionTime: "2023-02-13T18:13:48Z"
        message: ""
        reason: ReverseWordsDeploymentNotReady
        status: "True"
        type: ReverseWordsDeploymentNotReady
      - lastTransitionTime: "2023-02-13T18:13:48Z"
        message: ""
        reason: Ready
        status: "False"
        type: Ready
    
  6. We can test our application now

    LB_ENDPOINT=$(kubectl -n $NAMESPACE get svc --selector='app=example-reversewordsapp' -o jsonpath='{.items[*].status.loadBalancer.ingress[*].ip}')
    
    curl -X POST -d '{"word":"PALC"}' http://$LB_ENDPOINT:8080
    {"reverse_word":"CLAP"}
    
  7. Cleanup

    kubectl -n $NAMESPACE delete reversewordsapp example-reversewordsapp example-reversewordsapp-2
    kubectl delete -f config/crd/bases/apps.linuxera.org_reversewordsapps.yaml 
    kubectl delete ns operators-test
    
  8. That’s it!

In the next episode:

Sources