Go Backend Deployment Workflow

An interactive, zoomable version of this diagram is available at go-backend-deploy.html.

Overview

The Go backend service uses an interactive script backend/go/deploy.sh to orchestrate building container images, publishing them to GitHub Container Registry (GHCR), updating environment-specific Kustomize overlays, and rolling out workloads to Kubernetes. This workflow is designed to:

Eliminate Docker Desktop dependencies: Fully supports daemonless Podman and native Linux Buildah.
Support low-power dev machines: Leverages remote Kaniko jobs inside the Kubernetes/K3s cluster for in-cluster container builds.
Provide flexible rollout strategies: Supports immediate hot-fixes via kubectl apply -k, strict declarative GitOps via ArgoCD, or a Dual Sync mode that combines both.

High-Level Pipeline Architecture

The deployment pipeline is organized into four logical stages:

[Developer Invocation] ──► [Build & Push Strategy] ──► [Kustomize Overlay] ──► [Deployment Execution]
    ./deploy.sh <env>          1) Local (Podman/Buildah)   k8s/overlays/<env>/     1) Direct: kubectl apply
    Check prerequisites        2) Cluster Kaniko Job       Update newTag: VERSION  2) GitOps: ArgoCD sync
                               3) Skip (reuse tag)                                 3) Both: Direct + GitOps

Step 1: Environment & Prerequisites Detection

When you run deploy.sh, it immediately resolves target environment parameters and validates local tooling:

# Default to development
./deploy.sh

# Explicit environment target
./deploy.sh staging
./deploy.sh production

Script Inputs & Environment Variables

Variable	Source	Default	Description
`ENV`	Argument `$1`	`development`	Target environment (`development`, `staging`, `uat`, `production`).
`REGISTRY`	`$DOCKER_REGISTRY`	`ghcr.io/waynecheah`	Remote container registry URL.
`IMAGE_NAME`	Hardcoded	`backend`	Repository image identifier.
`VERSION`	`$VERSION`	`$(git rev-parse --short HEAD)`	Commit hash used as immutable container tag.
`BRANCH_NAME`	`$BRANCH_NAME`	Current git branch	Branch pushed to when committing Kustomize updates.
`DOCKER_PLATFORM`	`$DOCKER_PLATFORM`	(empty)	Optional cross-compile target (e.g. `linux/amd64`).

Automated Tooling Checks

Build Engine Detection: Checks if buildah (Linux native) or podman is installed.
Kubernetes CLI: Ensures kubectl exists in $PATH.
Overlay Existence: Confirms that k8s/overlays/${ENV} exists before proceeding.

Step 2: Build Engine Selection

The script presents three build options depending on your local hardware and environment:

Select build method:
1) Local Build (using podman/buildah)
2) Remote Build (using Kaniko in K3s cluster)
3) Skip build (use existing image)

Option 1: Local Build (`Podman` / `Buildah`)

Engine: Automatically selects buildah bud on Linux or podman build on macOS.
Workflow: Compiles backend/go/Dockerfile locally, tags with ${FULL_IMAGE}:${VERSION} and :latest, then pushes to GHCR.

Cross-Platform Compilation:

# Example: Build AMD64 image from an Apple Silicon Mac
export DOCKER_PLATFORM=linux/amd64
./deploy.sh development

Best For: Fast, iterative inner loops on powerful workstations.

Option 2: Remote Build (`Kaniko` in K3s Cluster)

Engine: Ephemeral Kubernetes batch/v1 Job executing gcr.io/kaniko-project/executor:latest.
Workflow:
1. Locates ghcr-secret in the target namespace (or across cluster namespaces).
2. Constructs an authenticated Git context URL (injecting $GITHUB_TOKEN if present).
3. Cleans up any existing kaniko-build-backend job.
4. Deploys the Kaniko Job into the cluster with --cache=true and sub-path backend.
5. Waits for completion (timeout up to 40 minutes for ARM/Raspberry Pi builds) and deletes the job upon success.
Best For: Lightweight laptops without container engines, remote development, or building directly on target architecture nodes.

Option 3: Skip Build

Workflow: Leaves registry images untouched and skips straight to overlay configuration.
Best For: Re-deploying an already-built image, testing configuration changes, or promoting a known commit hash without rebuilding.

Step 3: Kustomize Overlay Tagging

Once the image is confirmed in the registry:

The script navigates to backend/go/k8s/overlays/${ENV}.

Updates kustomization.yaml with the new tag:

images:
  - name: ghcr.io/waynecheah/backend
    newTag: a1b2c3d # Updated to $VERSION

Uses kustomize edit set image ... if the kustomize CLI is present, otherwise falls back to sed -i for portability.

Step 4: Deployment Execution Modes (Deep Dive)

The deployment execution prompt asks how changes should be applied to the Kubernetes cluster:

How would you like to deploy?
1) Apply directly to cluster (kubectl apply -k)
2) Commit to Git for ArgoCD sync
3) Both
4) Skip deployment

                          [kustomize_patch]
                                  │
                 ┌────────────────┼────────────────┐
                 ▼ (Option 1)     ▼ (Option 2)     ▼ (Option 3)
           Direct Rollout    GitOps Commit     Dual Sync (Both)
          [direct_apply]     [gitops_push]     Direct + GitOps
                 │                │                │
          kubectl apply -k   git push origin       │
                 │                │                │
                 │                ▼                │
                 │          [argocd_sync]          │
                 │         ArgoCD Operator         │
                 │         Reconcile loop          │
                 │                │                │
                 └────────────────┼────────────────┘
                                  ▼
                          [Live Workloads]
                        Go Backend Pods in K3s

Mode 1: Direct Rollout (`direct`)

Command: kubectl apply -k k8s/overlays/${ENV}
How it works: Compiles the Kustomize manifests locally and pushes resource definitions directly to the active Kubernetes context.
Characteristics:
- ⚡ Instantaneous: Pods roll out immediately in seconds.
- ⚠️ GitOps Drift: If ArgoCD has selfHeal: true enabled on the application, ArgoCD will eventually detect the mismatch with Git and roll the pods back to the Git version.
When to use: Quick hot-testing or debugging in development environments.

Mode 2: GitOps via ArgoCD (`git`)

Command: Commits kustomization.yaml and runs git push origin HEAD:${BRANCH_NAME}.
Commit Message: Update backend image to ${VERSION} for ${ENV} [skip ci]
How it works:
- The [skip ci] tag prevents redundant GitHub Actions test workflows from triggering.
- ArgoCD continuously monitors the target branch specified in backend/go/argocd/application.yaml.
- When ArgoCD detects the new commit, it synchronizes the live cluster according to its environment policy:
  - development & uat: Automated sync (prune: true, selfHeal: true).
  - staging: Automated sync (prune: true, selfHeal: false, manual approval).
  - production: Strictly manual sync for safety (automated: null).
Characteristics:
- 🔒 Auditable & Declarative: Every deployment is recorded in Git commit history.
- 🛡️ Zero Drift: Cluster state strictly reflects Git repository state.
When to use: Standard team workflows and promotions to staging and production.

Mode 3: Dual Sync (`both`)

Command: Runs kubectl apply -k AND commits and pushes to Git.
How it works:
- The local cluster rolls out the new image immediately without waiting for ArgoCD’s poll cycle.
- The Git commit is pushed concurrently, updating the target branch so ArgoCD sees the Git state and cluster state as perfectly in sync.
Characteristics:
- 🚀 Zero Waiting: Immediate rollout speed of kubectl apply.
- 🤝 Zero Drift: No risk of ArgoCD rolling back your local update.
When to use: The recommended choice when developing against a cluster managed by ArgoCD.

Mode 4: Skip Deployment (`skip`)

Action: Stops execution without touching the cluster or committing to Git.
When to use: Preparing the Kustomize overlay locally for manual inspection or inclusion in a wider PR.

Decision Matrix for Developers

Scenario	Build Choice	Deploy Choice	Result
New feature on local machine with Podman	`1) Local Build`	`3) Both`	Builds locally, pushes to GHCR, updates overlay, applies to cluster, and commits to Git.
Developing on a low-spec laptop (no container engine)	`2) Remote Kaniko`	`3) Both`	Cluster builds image asynchronously via K3s Job, pushes to GHCR, and deploys.
Config/Secret change only (no Go code change)	`3) Skip Build`	`1) Direct`	Skips image compilation, immediately updates Kustomize ConfigMap/Secret on cluster.
Formal promotion to Staging / Production	`3) Skip Build` or `1) Local`	`2) GitOps`	Bumps image tag in Git; team approves rollout via ArgoCD dashboard.

ArgoCD Environment Sync Policies

Defined in backend/go/argocd/application.yaml:

Application	Target Branch	Overlay Path	Automated Sync	Self-Heal	Retries
`backend-dev`	`develop`	`k8s/overlays/development`	✅ Enabled	✅ Enabled	5 retries
`backend-uat`	`uat`	`k8s/overlays/uat`	✅ Enabled	✅ Enabled	5 retries
`backend-staging`	`staging`	`k8s/overlays/staging`	✅ Enabled	❌ Manual approval	5 retries
`backend-prod`	`main`	`k8s/overlays/production`	❌ Manual only	❌ Disabled	3 retries

Troubleshooting & Common Issues

Kaniko build fails with ghcr-secret not found: Ensure a GitHub Container Registry token secret exists in your target namespace:

kubectl create secret docker-registry ghcr-secret \
  --docker-server=ghcr.io \
  --docker-username=<GITHUB_USERNAME> \
  --docker-password=<GITHUB_PAT> \
  -n <ENV>

ArgoCD rolls back your direct kubectl apply changes: If you selected 1) Direct in an environment where ArgoCD has selfHeal: true (such as development), ArgoCD will detect cluster drift from Git and revert your pod back to the Git tag. Select 3) Both to commit the new tag and keep Git in sync.

Non-interactive CI/CD execution: You can bypass the interactive prompts in scripts or automated jobs by setting environment variables:

DEPLOY_METHOD="direct" VERSION="v1.0.0" ./deploy.sh development

Deployment Script: backend/go/deploy.sh
ArgoCD Applications: backend/go/argocd/application.yaml
K8s Overlays: backend/go/k8s/overlays/
ArgoCD Deployment Guide: docs/backend/go/infrastructure/argocd-deployment.md
Deployment Quick Reference: docs/backend/go/infrastructure/deployment-quick-ref.md

Documentations

Getting Started

API

Modules

Infrastructure

Development

Guides

Go Backend Deployment Workflow

Overview

High-Level Pipeline Architecture

Step 1: Environment & Prerequisites Detection

Script Inputs & Environment Variables

Automated Tooling Checks

Step 2: Build Engine Selection

Option 1: Local Build (`Podman` / `Buildah`)

Option 2: Remote Build (`Kaniko` in K3s Cluster)

Option 3: Skip Build

Step 3: Kustomize Overlay Tagging

Step 4: Deployment Execution Modes (Deep Dive)

Mode 1: Direct Rollout (`direct`)

Mode 2: GitOps via ArgoCD (`git`)

Mode 3: Dual Sync (`both`)

Mode 4: Skip Deployment (`skip`)

Decision Matrix for Developers

ArgoCD Environment Sync Policies

Troubleshooting & Common Issues

​Overview

​High-Level Pipeline Architecture

​Step 1: Environment & Prerequisites Detection

​Script Inputs & Environment Variables

​Automated Tooling Checks

​Step 2: Build Engine Selection

​Option 1: Local Build (Podman / Buildah)

​Option 2: Remote Build (Kaniko in K3s Cluster)

​Option 3: Skip Build

​Step 3: Kustomize Overlay Tagging

​Step 4: Deployment Execution Modes (Deep Dive)

​Mode 1: Direct Rollout (direct)

​Mode 2: GitOps via ArgoCD (git)

​Mode 3: Dual Sync (both)

​Mode 4: Skip Deployment (skip)

​Decision Matrix for Developers

​ArgoCD Environment Sync Policies

​Troubleshooting & Common Issues

​Related Files & Links

Overview

High-Level Pipeline Architecture

Step 1: Environment & Prerequisites Detection

Script Inputs & Environment Variables

Automated Tooling Checks

Step 2: Build Engine Selection

Option 1: Local Build (`Podman` / `Buildah`)

Option 2: Remote Build (`Kaniko` in K3s Cluster)

Option 3: Skip Build

Step 3: Kustomize Overlay Tagging

Step 4: Deployment Execution Modes (Deep Dive)

Mode 1: Direct Rollout (`direct`)

Mode 2: GitOps via ArgoCD (`git`)

Mode 3: Dual Sync (`both`)

Mode 4: Skip Deployment (`skip`)

Decision Matrix for Developers

ArgoCD Environment Sync Policies

Troubleshooting & Common Issues

Related Files & Links