Guide: Preview environments in Kubernetes Engine Google Cloud
Thanks to this migration we have full control of our review apps and we can control every step of the installation, plus Kubernetes lets us control each pod's resources, and is easy to scale.
Preparing the DNS
We are going to use Cloudflare as our DNS provider so we need to create two types of DNS: A and CNAME.
Create a DNS type A and use review
for the NAME but let it keep empty for content, you will fill it later.
After that create a DNS type CNAME and use *.review
for the name with the following content: review.your_company.com
. Notice the asterisk, this will be needed so you can create URLs like reviewapp1.review.your_company.com
Setting up the cluster
Research your preferred cloud provider, then go ahead and create a cluster with an installed gateway API.
Prerequisites
- Install Helm in your machine
- A Kubernetes cluster
Cert Manager and Gateway Installation
Once you have your cluster, we are going to install cert-manager
which is required to issue the certificates for your preview environments and access through HTTPS.
I'm going to show you how to do it with Cloudflare, but if you choose use another provider, the cert manager site has plenty of helpful docs.
The following command will install cert-manager in its own namespace with the gateway API support enabled:
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --set "extraArgs={--feature-gates=ExperimentalGatewayAPISupport=true}"
After that, we are going to install the Custom Resource Definitions. This helped me to avoid running errors with cert-manager pods.
helm upgrade cert-manager jetstack/cert-manager --namespace cert-manager --set installCRDs=true
It's time to install the next resources and we are going to need a Cloudflare token generated with special settings. Go to Cloudflare > Profile > API Tokens > API Tokens and create the token with these settings:
- Permissions:
- Zone - DNS - Edit
- Zone - Zone - Read
- Zone Resources
- Include - All Zones
We have to push this secret to our cluster inside the cert-manager namespace
apiVersion: v1
kind: Secret
metadata:
name: cloudflare-api-token-secret
namespace: cert-manager
type: Opaque
stringData:
api-token: <CLOUDFLARE_TOKEN>
cloudflare-api-secret.yaml
Save it and apply it kubectl apply -f cloudflare-api-token.yaml
Next step is to create the issuer that will generate the certificate for our review apps, and we need to install a cluster issuer so we can install review apps in different namespaces. It will use letsencrypt.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: acme-issuer
namespace: cert-manager
spec:
acme:
email: <YOUR_CLOUDFLARE_EMAIL>
server: https://acme-v02.api.letsencrypt.org/directory
privateKeySecretRef:
name: example-issuer-account-key
solvers:
- dns01:
cloudflare:
email: <YOUR_CLOUDFLARE_EMAIL>
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
clusterissuer.yaml
Save it and apply it kubectl apply -f cluster-issuer.yaml
Finally the gateway, that will create a load balancer with a public IP that will be needed for setting the DNS:
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
namespace: default
name: external-http
annotations:
cert-manager.io/cluster-issuer: acme-issuer
spec:
gatewayClassName: <GATEWAY_CLASS_NAME>
listeners:
- name: https
hostname: <WILDCARD_HOSTNAME>
port: 443
protocol: HTTPS
allowedRoutes:
namespaces:
from: All
tls:
mode: Terminate
certificateRefs:
- name: review-cert
kind: Secret
group: ""
gateway.yaml
You have to replace <GATEWAY_CLASS_NAME> with one provided by your cluster. In Google Cloud I used gke-l7-global-external-managed
so, as I said, it will depend on your cluster.
<WILDCARD_HOSTNAME> is the DNS type CNAME we set in Cloudflare, e.g. it could be something like *.review.your_company.com
.
Save it and apply it kubectl apply -f gateway.yaml
After some time you should be able to get a public IP when inspecting the gateway: kubectl get gateway
and that IP should be used as the content for the DNS type A you created before.
"Helmifiying" your application
If you have a dockerised application you can create a custom helm chart to make it easy to run it in the Kubernetes cluster you have created.
Run the following command to create your chart and then you can start adding the bits you need for your app:
helm create YOUR_APP
You can add dependencies, that are other charts, to your application using the Charts.yaml
file e.g. add databases as dependencies
dependencies:
- name: redis
version: 18.6.1
repository: https://charts.bitnami.com/bitnami
- name: memcached
version: 6.7.1
repository: https://charts.bitnami.com/bitnami
- name: postgresql
version: 13.2.25
repository: https://charts.bitnami.com/bitnami
- name: elasticsearch
version: 19.13.14
repository: https://charts.bitnami.com/bitnami
Charts.yaml
The advantage is that these charts are highly customised so in the values.yaml
file you can configure them as you wish:
redis:
image:
tag: 6.2
architecture: standalone
auth:
enabled: false
master:
persistence:
enabled: false
values.yaml
Build the dependencies helm dependency build
and it's time to configure your templates.
Inside the templates folder you have to configure all the Kubernetes yaml files that your app will have, my review app has deployments, jobs and config maps.
There are two special yaml files that we need for connecting with the gateway, first your app needs to get accessed by people so we need to create a Service
to expose the app, something like the following would work but depends on your needs:
apiVersion: v1
kind: Service
metadata:
name: hubbado-web
spec:
selector:
app: hubbado-web
ports:
- name: https
port: 443
protocol: TCP
targetPort: 3000
- name: http
port: 9393
protocol: TCP
targetPort: 9393
service-web.yaml
And now we need the HTTPRoute
kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: hubbado-web-http-route
spec:
parentRefs:
- kind: Gateway
name: external-http
namespace: default
hostnames:
- "<YOUR_BRANCH_NAME>.<WILDCARD_HOSTNAME>"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: hubbado-web
port: 443
http-route.yaml
This will connect the Gateway
with the Service
. Additionally you could need a HealthCheckPolicy
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
name: hubbado-web-healthcheck
namespace: {{ .Release.Namespace }}
spec:
default:
checkIntervalSec: 30
logConfig:
enabled: true
config:
type: HTTP
httpHealthCheck:
port: 9393
requestPath: /_liveness
targetRef:
group: ""
kind: Service
name: hubbado-web
health-check-policy-web.yaml
Why do we need this health check? To properly configure all the routing and tell the gateway that everything was working.
You will need more configurations depending on your application.
Now it is time to store your custom chart in a registry that allows installation later on. In our case, we used Github packages to push the chart.
- Create the package:
helm package hubbado-review-app --version "$VERSION_NUMBER"
- Logging to the registry:
echo "GITHUB_TOKEN" | helm registry login
https://ghcr.io/v2
--username GITHUB_USERNAME --password-stdin
The GITHUB_TOKEN
needs access to packages scope so be sure to configure it with that permission.
- Push the package:
helm push hubbado-review-app-$VERSION_NUMBER.tgz oci://ghcr.io/hubbado
Now it is time to use Github actions (or something similar) to deploy the app.
Github actions
We use Github actions as CI and also to deploy our review apps depending on the git branch and I would like to show you a workflow example to share how we do it:
name: Review app - Install
on: workflow_dispatch
jobs:
build:
# Configure this step to build your app using docker
deploy:
runs-on: self-hosted
needs: build
steps:
- id: 'auth'
uses: 'google-github-actions/auth@v2'
with:
credentials_json: '${{ secrets.GOOGLE_CREDENTIALS }}'
- id: 'get-credentials'
uses: 'google-github-actions/get-gke-credentials@v2'
with:
cluster_name: ${{ secrets.CLUSTER_NAME }}
location: ${{ secrets.CLUSTER_LOCATION }}
project_id: ${{ secrets.PROJECT_ID }}
- name: Log in Helm into github package registry
run: |
echo ${{ secrets.GITHUB_TOKEN }} | \
helm registry login https://ghcr.io/v2 \
--username ${{ github.repository_owner }} \
--password-stdin
- name: Pull review app chart
run: |
helm pull \
--untar oci://ghcr.io/hubbado/hubbado-review-app
- name: Install review app
run: |
helm install \
--namespace ${{ github.ref_name }} \
--create-namespace review-app hubbado-review-app \
--set image.branch=${{ github.ref_name }}
review_app-install.yaml
Now wait until your app is deployed and you will have a full review app similar to production where you can test all the changes you want.