In the previous post in this two-part series, I introduced the basic concepts of cloud infrastructure automation using Terraform and GCP. I also showed how to set up a GCP project and use Terraform to provision a basic GCP compute instance. 

In this post, I will review some examples of provisioning more advanced GCP resources, using more advanced Terraform techniques. 

Before Getting Started

First, if you haven’t already, follow the tutorial in the first post, as the examples here assume you have completed it. Also, make sure you enable the following GCP APIs:

• Google Kubernetes Engine (GKE)
• Identity and Access Management (IAM)
• Google Cloud Memorystore

Provision a Kubernetes Cluster

GKE is a managed Kubernetes environment on which you can deploy container-based workloads. While kubectl and similar tools can be used to automate deployments on GKE, you can’t use them to provision the cluster itself. Fortunately, Terraform can handle this for you.

In a minute, I’ll show you how to change your existing Terraform project to deploy a GKE cluster instead of a single VM. But, before doing so, I’m going to introduce some new concepts which we will use in the next version of our Terraform script: Terraform input variables and GKE service accounts.

Terraform Input Variables

Like function arguments in most programming languages, Terraform input variables introduce flexibility and reusability to Terraform resources. You can replace hard-coded values with placeholders and provide values from external files or from command-line arguments. The values used for variables are stored with the Terraform state, so if you change the value of a variable, terraform apply will know how to modify your resources accordingly. In this post, I’ll use Terraform input variables to control the cluster’s location and size.

GKE Service Accounts

Your Terraform code will define a new GCP service account that the GKE cluster nodes will use. This is different from the service account we used when running Terraform, as the manually created service account has broad permissions to deploy new GCP resources throughout the project. Your nodes will use this limited service account, which will be tuned to allow Kubernetes workloads access to specific GCP resources—without requiring special account keys.

Start by creating a new file: variables.tf. 

This file defines our input variables:

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "zone" {
  description = "GCP zone (must be within the selected region)"
  type        = string
  default     = "us-central1-a"
}

variable "cluster_size" {
  description = "Kubernetes cluster size"
  type        = number
  default     = 1
}

Then, edit your main.tf file to have the following content:

provider "google" {
  project = "tfm-tutorial"
  region  = var.region
  zone    = var.zone
}

resource "google_service_account" "my_service_account" {
  account_id   = "my-gke-cluster-sa"
  display_name = "GKE Cluster Account"
}

resource "google_container_cluster" "my_gke_cluster" {
  name               = "my-gke-cluster"
  initial_node_count = 1
  remove_default_node_pool = true

  network    = "default"
  subnetwork = "default"

  ip_allocation_policy {
    cluster_ipv4_cidr_block  = "/16"
    services_ipv4_cidr_block = "/22"
  }
}

resource "google_container_node_pool" "my_gke_primary_nodepool" {
  name       = "my-gke-node-pool"
  cluster    = google_container_cluster.my_gke_cluster.name
  node_count = var.cluster_size

  node_config {
    machine_type = "e2-standard-2"
    service_account = google_service_account.my_service_account.email
    oauth_scopes    = ["https://www.googleapis.com/auth/cloud-platform"]
  }
}

There are a few noteworthy points here:

• The variables.tf file defines three input variables and sets a type, description, and optional default value for each. 
• In the main.tf file, refer to the value of these variables using var.<variable name>, as you did when setting the GCP region and zone.
• GKE will always create a cluster with a default node pool. Terraform allows you to remove that node pool by specifying remove_default_note_pool = true; you must then replace it with a customized node pool.
• The node pool resource implicitly depends on the service account resource, as your nodes will run as this account. Refer to the account by setting service_account = google_service_account.my_service_account.email.
• As f1-micro type machines are too limited to run Kubernetes nodes, we switched to e2-standard-2 machines.

Apply Changes and Specify Input Values

Apply the changes by running `terraform apply` and confirming the change. Launching a GKE cluster and node pool takes a few minutes, so be patient. 

Once the cluster is up, run gcloud container clusters list. 

You should see your cluster, with one node:

NAME            LOCATION       MASTER_VERSION    MASTER_IP     MACHINE_TYPE   NODE_VERSION      NUM_NODES  STATUS
my-gke-cluster  us-central1-a  1.18.12-gke.1210  34.70.174.94  e2-standard-2  1.18.12-gke.1210  1          RUNNING

Now, change the value of your size variable and see what happens. Run the following:

$ terraform apply -var cluster_size=2

You should see a plan that modifies your existing node pool in place. Confirm. Then, after Terraform finishes, run gcloud container clusters list. 

The cluster should now show two nodes.

Terraform and kubectl

Configuring kubectl

Now that you have a Kubernetes cluster up and running, you should be able to access it using standard Kubernetes tooling, namely kubectl. 

If you don’t already have it installed, follow these installation instructions.

You can use gcloud to configure kubectl with access credentials to your cluster. 

Get the name and location of your cluster:

$ gcloud container clusters list


NAME        	LOCATION   	MASTER_VERSION	MASTER_IP 	MACHINE_TYPE   NODE_VERSION  	NUM_NODES  STATUS
my-gke-cluster    us-central1-a  1.18.12-gke.1210  12.34.56.78  e2-standard-2  1.18.12-gke.1210  1      	RUNNING

Then run:

$ gcloud container clusters get-credentials my-gke-cluster --zone us-central1-a

This will configure kubectl to point to your cluster by default. 

Note: If you destroy and recreate the cluster, you will need to repeat this command.

Now you should be able to run:

$ kubectl get nodes

If kubectl is properly configured, you should see the list of nodes in your cluster.

The Terraform Kubernetes Provider vs. kubectl

Terraform and Kubernetes are, in many ways, similar tools—they both enable automated deployment and follow an infrastructure-as-code approach. Terraform offers a kubernetes provider, which you can use to provision resources on existing Kubernetes clusters. Thus, in many cases, you can use Terraform instead of kubectl. However, this may lead to some confusion regarding which resources should be managed by Terraform and which should be managed by Kubernetes-native tools.

Like with similar dilemmas, I do not believe there is one correct choice here. However, here’s a good rule-of-thumb. Use Terraform to provision the infrastructure supporting the application and ensure the availability of any external dependencies. Use Kubernetes to deploy your application and its immediate dependencies. 

A monitoring service agent and the Let’s Encrypt cert-manager service used to automatically rotate SSL certificates are two good examples of generic “infrastructure” deployed to Kubernetes that is not application specific. I usually manage these with Terraform, not kubectl.

Adding a Managed Redis Instance

Imagine your application depends on a Redis backend. While it’s perfectly possible to deploy Redis into your Kubernetes cluster, Memorystore, a managed memory storage instance compatible with Redis, could be even better. It offers some benefits over deploying your own Redis nodes, including seamless scalability and high availability. 

Let’s provision an instance of Memorystore for Redis using Terraform and ensure Kubernetes pods can connect to it. 

Provision a Memorystore Instance

First, add the following block at the bottom of main.tf:

resource "google_redis_instance" "my_memorystore" {
  name               = "my-memory-store"
  memory_size_gb     = 1
  authorized_network = google_container_cluster.my_gke_cluster.network
  display_name       = "Terraform Tutorial Memory Storage"
}

This defines a Redis Memorystore instance with 1 GB of storage and authorizes your GKE cluster network to access it. 

Next, run `terraform apply` and approve the changes. After a few minutes, you should get confirmation that the new instance is up. 

Define a Kubernetes Service for Redis

To expose Redis to pods running on Kubernetes without having to configure the Redis hostname repeatedly, create an ExternalName Kubernetes service pointing to it. This abstracts the external Redis endpoint using the internal Kubernetes DNS, making it available to all pods as if it was a service deployed inside the cluster. We will do this using the kubernetes Terraform provider.
In your project directory, create a new file named k8s-resources.tf:

data "google_client_config" "provider" {}

provider "kubernetes" {
  host  = "https://${google_container_cluster.my_gke_cluster.endpoint}"
  token = data.google_client_config.provider.access_token
  cluster_ca_certificate = base64decode(
     google_container_cluster.my_gke_cluster.master_auth[0].cluster_ca_certificate,
  )
}

resource "kubernetes_service" "redis_memorystore_svc" {
  depends_on = [google_container_cluster.my_gke_cluster]
  metadata {
    name = "redis-memorystore"
  }
  spec {
    port {
      port        = 6379
      target_port = google_redis_instance.my_memorystore.port
    }
    type          = "ExternalName"
    external_name = google_redis_instance.my_memorystore.host
  }
}

This configures the kubernetes provider and points it to your cluster. A kubernetes_service resource is defined, setting redis-memorystore as a CNAME of the Redis instance hostname. The GKE cluster is specified as an explicit dependency. This ensures resource destruction order, even though there is no implicit dependency between the two resources. 

To initialize the Kubernetes provider, run:

$ terraform init

Then run:

$ terraform apply 

When the new resource creation is finished, run:

$ kubectl get services

You should see the `redis-memorystore` service listed. 

Seeing It All Work Together

For the grand finale, let’s use kubectl to deploy a small application, with Redis as a backend. 

Save the following in a file named demo-counter.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
 name: demo-counter
 labels:
   app: demo-counter
spec:
 replicas: 2
 selector:
   matchLabels:
     app: demo-counter
 template:
   metadata:
     labels:
       app: demo-counter
   spec:
     containers:
     - name: app
       image: shevron/python-redis-demo-counter:0.1.0
       ports:
       - containerPort: 8000
       env:
       - name: REDIS_HOST
         value: redis-memorystore
---
apiVersion: v1
kind: Service
metadata:
 name: demo-counter-svc
spec:
 selector:
   app: demo-counter
 ports:
   - protocol: TCP
     port: 80
     targetPort: 8000
 type: LoadBalancer

This Kubernetes file defines a deployment and a service to expose it over HTTP. The ready-made Docker image you are deploying is a sample counter application that uses Redis as a backend.

The app expects the `REDIS_HOST` environment variable to point to the Redis hostname. I set it to `redis-memorystore`(the hostname of the Redis service as exposed inside your cluster). This value shouldn’t change, even if you decide to move the Redis backend elsewhere—the Kubernetes service acts as a “pointer” to the right address.

Deploy this by running:

$ kubectl apply -f demo-counter.yaml

You should see a message confirming that a deployment and a service have been created. 

After a few moments, run:

$ kubectl get services demo-counter-svc -o \

    jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'

This should print out the public IP address of your deployed service. You can now use curl to hit your app:

$ curl -X GET https://<printed IP address>/my-counter 

0

This prints the value of `my-counter`. 

To increment the counter, send a POST request:

$ curl -X POST https://<service address>/my-counter 

1

You can repeat the above command multiple times. Sending a GET request again will show you the latest value. The counter value is stored in Redis, accessed by the Python app running on Kubernetes.

Cleaning Up

Before finishing, remember to clean up by destroying all the resources you have provisioned:

$ terraform destroy

Of course, as you’re using Terraform, you recreate the same cluster in minutes, simply by running `terraform apply`.

Summary

In this series, I demonstrated how useful and easy it is to automate resource provisioning and deployment using Terraform. I also gave you an introduction to using Terraform with GCP. While I haven’t ventured deep into Terraform, GCP, or Kubernetes, I hope that the examples I provided demonstrate a solid approach to using all these technologies together.

If you want to learn more about Terraform, and get familiar with some more advanced Terraform techniques, I recommend that you check out our How to Use Terraform Like a Pro series next.