The ECS Engineer’s Guide to Kubernetes

🗺️ Concept Mapping

⚠️ Concepts That Don't Map Cleanly

Namespaces: ECS doesn't have a direct equivalent. Namespaces are logical partitions within a cluster, not hard account-level security boundaries. They scope resource names and enable RBAC, quotas, and policy. Most teams use namespaces per-environment or per-team, then use separate clusters/accounts for stronger isolation.

Labels & Selectors: K8s uses key-value labels on every resource, and selectors to query them. This is how Services find Pods, how Deployments manage ReplicaSets, and how you organize everything. ECS tags are similar but far less central to the system.

DaemonSets: Runs exactly one Pod per node. There's no ECS equivalent — this is used for node-level agents like log collectors, monitoring agents, or CNI plugins.

StatefulSets: For stateful workloads needing stable identity, ordered deployment, and persistent storage. ECS can do persistent storage with EBS, but doesn't provide the same identity guarantees.

🖥️ Nodes core

A Node is a machine (VM or bare metal) in your cluster that runs workloads. Each node runs three things: the kubelet (an agent that receives instructions from the API server and manages pods), kube-proxy (handles network rules so pods can communicate), and a container runtime (usually containerd — Docker is deprecated as a runtime).

Nodes have allocatable resources (CPU, memory, ephemeral storage) and can be labeled and tainted to control scheduling. For example, you might label GPU nodes and use a nodeSelector to schedule ML workloads there.

# Taint a node
kubectl taint nodes gpu-node-1 gpu=true:NoSchedule

# Pod must tolerate it to be scheduled there
tolerations:
- key: "gpu"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

🫛 Pods core

A Pod is the smallest deployable unit — it's one or more containers that share a network namespace (same IP, same localhost) and can share volumes. Think of it as the K8s equivalent of an ECS Task, though typically simpler since most pods have just one container.

Pods are ephemeral by design. They can be killed, restarted, or rescheduled at any time. You should never create Pods directly in production — instead, you create a Deployment (or StatefulSet, DaemonSet, Job) which creates Pods for you and handles their lifecycle.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
  labels:
    app: web         # ← labels are how Services find this pod
spec:
  initContainers:      # ← runs before main containers
  - name: db-migrate
    image: my-app:latest
    command: ["./migrate.sh"]
  containers:
  - name: app
    image: my-app:latest
    ports:
    - containerPort: 8080
    resources:           # ← always set these in production
      requests:
        cpu: "250m"       # 0.25 CPU cores
        memory: "256Mi"
      limits:
        cpu: "500m"
        memory: "512Mi"
    livenessProbe:       # ← restarts container if it fails
      httpGet:
        path: /healthz
        port: 8080
    readinessProbe:      # ← removes from Service endpoints if it fails
      httpGet:
        path: /ready
        port: 8080
  - name: log-shipper    # ← sidecar container
    image: fluentbit:latest

📦 Namespaces core

Namespaces are virtual clusters within a physical cluster. They scope resource names (you can have a pod called "api" in both staging and production namespaces) and are the primary unit for RBAC, resource quotas, and network policies.

Default namespaces: default (where things go if you don't specify), kube-system (control plane components), kube-public (publicly readable), kube-node-lease (node heartbeats).

🏷️ Labels, Selectors & Annotations core

Labels are key-value pairs attached to resources. They're the connective tissue of K8s — Services find Pods via label selectors, Deployments manage ReplicaSets through them, NetworkPolicies target pods by label. Think of labels as queryable tags that drive system behavior.

Annotations are also key-value pairs, but they're for metadata that doesn't drive selection — build info, config for tools like Ingress controllers, links to docs, etc.

# Labels drive behavior
labels:
  app: payment-service
  env: production
  team: billing

# Annotations are informational
annotations:
  prometheus.io/scrape: "true"
  deploy.link: "https://ci.example.com/builds/421"

🚀 Deployments — Your Daily Driver

A Deployment is what you'll use for 90% of your workloads. It manages a ReplicaSet which manages Pods. When you update the Deployment (new image, env change), it creates a new ReplicaSet and progressively shifts traffic, giving you zero-downtime rolling updates — the same thing an ECS service deployment does, but with more fine-grained control.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
  namespace: production
spec:
  replicas: 3                # ← like ECS desired count
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1            # ← max pods over desired during deploy
      maxUnavailable: 0     # ← zero downtime — always have 3 ready
  selector:
    matchLabels:
      app: api-server      # ← must match template labels
  template:                    # ← this IS the pod spec
    metadata:
      labels:
        app: api-server
    spec:
      containers:
      - name: api
        image: myregistry/api:v2.3.1
        ports:
        - containerPort: 8080

kubectl rollout history deployment/api-server
kubectl rollout undo deployment/api-server           # previous version
kubectl rollout undo deployment/api-server --to-revision=3  # specific version
kubectl rollout status deployment/api-server          # watch progress

📊 StatefulSets

StatefulSets provide guarantees Deployments don't: stable, unique network identifiers (pods are named web-0, web-1, etc., not random hashes), ordered deployment and scaling, and stable persistent storage that follows the pod identity even through rescheduling.

You'd use a StatefulSet for databases, distributed systems like Kafka or Elasticsearch, or anything needing persistent identity. Each pod gets its own PersistentVolumeClaim that sticks with it.

⏱️ Jobs & CronJobs

A Job creates pods that run to completion — like an ECS RunTask. It tracks success/failure and supports parallelism. A CronJob creates Jobs on a schedule (cron syntax). Common uses: database backups, report generation, data pipeline triggers.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"      # ← 2 AM daily
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: backup
            image: backup-tool:latest
          restartPolicy: OnFailure

🌐 The K8s Networking Model

K8s enforces a flat network model: every Pod gets its own IP, every Pod can reach every other Pod directly (no NAT), and nodes can reach all pods. This is implemented by a CNI (Container Network Interface) plugin — Calico, Cilium, Flannel, etc. The CNI is a choice you make (or your managed K8s provider makes).

In ECS on EC2, you dealt with awsvpc mode and ENIs. In K8s, the CNI plugin handles all this transparently — you just use Pod IPs.

🔗 Services — Stable Networking to Pods

Pods are ephemeral — their IPs change. A Service provides a stable virtual IP (ClusterIP) and DNS name that routes to a set of pods matched by label selectors. This is like an internal ECS service discovery endpoint, but it's built in and automatic.

apiVersion: v1
kind: Service
metadata:
  name: api-server        # ← becomes dns: api-server.production.svc.cluster.local
  namespace: production
spec:
  selector:
    app: api-server        # ← routes to pods with this label
  ports:
  - port: 80                # ← port the Service listens on
    targetPort: 8080        # ← port on the pod

🚪 Ingress & Ingress Controllers

An Ingress is a resource that defines HTTP(S) routing rules — host-based and path-based routing to Services. But it doesn't do anything by itself; you need an Ingress Controller (a running pod) to implement it. Popular choices: NGINX Ingress Controller, Traefik, HAProxy, AWS ALB Ingress Controller, Envoy/Emissary.

Think of Ingress as the K8s equivalent of ALB listener rules. The newer Gateway API (graduating to GA) is replacing Ingress with a more expressive, role-oriented model — worth learning early since it's the future.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: api-server
            port:
              number: 80

🛡️ NetworkPolicies

By default, all pods can talk to all pods. NetworkPolicies are firewall rules for pod-to-pod traffic, implemented by your CNI plugin. They're the K8s equivalent of security groups, but at the pod level rather than the instance level.

If your CNI supports them (Calico and Cilium do, Flannel doesn't), you can restrict which pods can communicate with each other, implementing zero-trust networking within the cluster.

🕸️ Service Mesh (Istio, Linkerd, Cilium)

A service mesh adds a proxy sidecar (usually Envoy) to every pod, giving you mTLS (encrypted pod-to-pod communication), fine-grained traffic routing (canary deploys, A/B testing), observability (distributed tracing, metrics per-request), and circuit breaking. It's like having an ALB between every pair of services.

Istio is feature-rich but complex. Linkerd is lighter and simpler. Cilium does mesh at the eBPF/kernel level without sidecars. At a core K8s infra shop, you'll almost certainly encounter one of these.

💾 The Storage Stack

K8s storage has three layers, and understanding how they connect is important:

In practice, you usually define a StorageClass (or use the default one your cloud provides), then create PVCs in your pod specs. The StorageClass's provisioner automatically creates the underlying PV (e.g., an EBS volume on AWS, a PD on GCP). This is called dynamic provisioning, and it's what you'll use 99% of the time.

📁 Volume Types You'll Encounter

emptyDir: Ephemeral, per-pod scratch space. Exists as long as the pod exists — for temp files, caches, inter-container communication within a pod.

hostPath: Mounts a directory from the host node. Use sparingly — breaks portability. Mainly for DaemonSets that need to read node-level files (logs, docker socket).

CSI (Container Storage Interface): The plugin standard for storage. AWS EBS CSI Driver, EFS CSI Driver, etc. — these are what let K8s provision cloud storage dynamically.

⚙️ ConfigMaps

ConfigMaps store non-sensitive key-value data or entire config files. You can mount them as environment variables or as files in a volume. This replaces things like SSM Parameter Store for non-secret config.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "db.internal.svc.cluster.local"
  LOG_LEVEL: "info"
  nginx.conf: |        # ← you can store entire files
    server {
      listen 80;
      location / { proxy_pass http://app:8080; }
    }

🔐 Secrets

Secrets are like ConfigMaps but for sensitive data (passwords, tokens, TLS certs). They're base64-encoded (not encrypted!) at rest by default. For real security, enable encryption at rest in etcd or use an external secrets solution.

In production at a core K8s shop, you'll likely use the External Secrets Operator to sync secrets from Vault, AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault into K8s Secrets — giving you the best of both worlds.

⌨️ Essential Commands

# ── Context & Cluster ──────────────────────────────────
kubectl config get-contexts              # list clusters
kubectl config use-context prod-cluster  # switch cluster
kubectl config set-context --current --namespace=staging  # set default ns

# ── Viewing Resources ─────────────────────────────────
kubectl get pods                         # list pods in current ns
kubectl get pods -A                      # all namespaces
kubectl get pods -o wide                 # more detail (node, IP)
kubectl get deploy,svc,ing               # multiple resource types
kubectl describe pod my-pod              # detailed info + events
kubectl get pod my-pod -o yaml           # full manifest

# ── Debugging ─────────────────────────────────────────
kubectl logs my-pod                      # stdout logs
kubectl logs my-pod -c sidecar           # specific container
kubectl logs -f my-pod                   # follow/tail
kubectl logs -l app=api --all-containers # by label selector
kubectl exec -it my-pod -- /bin/sh       # shell into pod
kubectl port-forward svc/api 8080:80     # tunnel to your laptop
kubectl top pods                         # resource usage

# ── Creating & Modifying ──────────────────────────────
kubectl apply -f manifest.yaml           # create or update
kubectl apply -f ./k8s/                  # apply whole directory
kubectl delete -f manifest.yaml          # delete resources
kubectl scale deploy/api --replicas=5    # manual scale
kubectl edit deploy/api                  # open in editor

# ── Dry Runs & Diff ───────────────────────────────────
kubectl apply -f m.yaml --dry-run=server # server-side dry run
kubectl diff -f manifest.yaml            # show what would change

🛠️ Power User Tips

Aliases: Most K8s engineers alias kubectl to k. Set up shell completion — it's a massive time saver.

kubectx + kubens: Tools to switch clusters and namespaces fast. Essential when you're managing multiple clusters.

kubectl debug: K8s 1.23+ lets you attach ephemeral debug containers to running pods — invaluable when a pod has no shell (distroless images).

krew: A plugin manager for kubectl. Gives you access to hundreds of community plugins like kubectl neat (clean up YAML output), kubectl tree (resource hierarchy), and kubectl sniff (tcpdump on pods).

📋 kubeconfig Explained

Your kubeconfig file (~/.kube/config) stores cluster connection details: server addresses, authentication credentials, and contexts (a named combo of cluster + user + namespace). When you run aws eks update-kubeconfig, it adds the EKS cluster to this file. You can have dozens of clusters configured and switch between them with kubectl config use-context.

🧪 What You Will Build

This lab gives you a real Kubernetes API on your laptop. OrbStack provides the local Docker runtime, kind creates Kubernetes nodes as containers, kubectl talks to the cluster, and Tilt gives you the fast edit-build-deploy loop.

🚀 Prerequisites and Cluster Setup

Install OrbStack, kind, kubectl, and Tilt first. Then create a disposable local cluster for this lab.

# Verify local tooling
orb version
docker version
kind version
kubectl version --client
tilt version

# Create an isolated local Kubernetes cluster using OrbStack's Docker runtime
kind create cluster --name ecs-k8s-lab

# Confirm kubectl is talking to the lab cluster
kubectl config current-context
kubectl cluster-info
kubectl get nodes

kubectl config use-context kind-ecs-k8s-lab

📁 Create the Tiny Service

Create a local directory named local-k8s-demo. The service is intentionally small so Kubernetes stays the focus.

local-k8s-demo/
├── Dockerfile
├── package.json
├── server.js
├── k8s.yaml
└── Tiltfile

package.json

{
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.18.3"
  }
}

server.js

const express = require('express');

const app = express();
const port = process.env.PORT || 8080;
const message = process.env.MESSAGE || 'Hello from local Kubernetes';

app.get('/', (_req, res) => {
  res.json({ service: 'local-k8s-demo', message });
});

app.get('/healthz', (_req, res) => {
  res.json({ ok: true });
});

app.get('/env', (_req, res) => {
  res.json({ message, pod: process.env.HOSTNAME });
});

app.listen(port, () => {
  console.log(`local-k8s-demo listening on ${port}`);
});

Dockerfile

FROM node:20-alpine
WORKDIR /app
COPY package.json package-lock.json* ./
RUN npm install --omit=dev
COPY server.js .
EXPOSE 8080
CMD ["npm", "start"]

📦 Deploy It Manually First

This is the Kubernetes version of defining a task, running a service, and wiring a stable endpoint to disposable containers.

k8s.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: local-k8s
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: local-k8s-demo-config
  namespace: local-k8s
data:
  MESSAGE: "Hello from a ConfigMap"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: local-k8s-demo
  namespace: local-k8s
spec:
  replicas: 2
  selector:
    matchLabels:
      app: local-k8s-demo
  template:
    metadata:
      labels:
        app: local-k8s-demo
    spec:
      containers:
      - name: app
        image: local-k8s-demo:dev
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        envFrom:
        - configMapRef:
            name: local-k8s-demo-config
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 2
          periodSeconds: 5
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 250m
            memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: local-k8s-demo
  namespace: local-k8s
spec:
  type: ClusterIP
  selector:
    app: local-k8s-demo
  ports:
  - port: 80
    targetPort: 8080

# Build locally and load the image into the kind cluster
docker build -t local-k8s-demo:dev .
kind load docker-image local-k8s-demo:dev --name ecs-k8s-lab

# Apply desired state
kubectl apply -f k8s.yaml

# Inspect what Kubernetes created
kubectl get deploy,po,svc -n local-k8s
kubectl describe deploy local-k8s-demo -n local-k8s
kubectl logs -l app=local-k8s-demo -n local-k8s

# Reach the ClusterIP Service from your laptop
kubectl port-forward -n local-k8s svc/local-k8s-demo 8080:80
curl http://localhost:8080
curl http://localhost:8080/healthz
curl http://localhost:8080/env

🔍 Break It, Inspect It, Fix It

The unlock is seeing the control loop in action. Delete pods, scale the Deployment, change config, and watch Kubernetes reconcile back to desired state.

# Watch Kubernetes replace deleted pods
kubectl get pods -n local-k8s -w
kubectl delete pod -n local-k8s -l app=local-k8s-demo

# Scale like ECS desired count
kubectl scale deploy/local-k8s-demo -n local-k8s --replicas=4
kubectl get pods -n local-k8s

# Change config and restart pods to pick it up
kubectl edit configmap local-k8s-demo-config -n local-k8s
kubectl rollout restart deploy/local-k8s-demo -n local-k8s
kubectl rollout status deploy/local-k8s-demo -n local-k8s

# Inspect the objects Kubernetes is reconciling
kubectl describe pod -n local-k8s -l app=local-k8s-demo
kubectl get deploy local-k8s-demo -n local-k8s -o yaml

🔄 Add Tilt for the Inner Loop

Manual Docker build, image load, and kubectl apply are useful once. Tilt turns that into a daily development loop.

Tiltfile

docker_build('local-k8s-demo', '.')
k8s_yaml('k8s.yaml')
k8s_resource('local-k8s-demo', port_forwards='8080:80')

tilt up

Now edit server.js, save, and watch Tilt rebuild the image, update Kubernetes, stream logs, and keep the port-forward alive. Refresh http://localhost:8080 to see the change.

ECS translation: instead of build → push → deploy or a Compose-like simulation, Tilt gives you a real Kubernetes-native loop against the same API shape you will use in dev, staging, and production.

🧹 Cleanup

# Reset just the app, but keep the cluster
kubectl delete namespace local-k8s

# Or delete the whole disposable lab cluster
kind delete cluster --name ecs-k8s-lab

⎈ What Helm Does

Raw K8s manifests are static YAML. Helm solves three problems: templating (parameterize your YAML so the same chart works in dev and prod), packaging (bundle all the manifests for an app into a versioned "chart"), and release management (track what's deployed, rollback to previous versions).

A Helm Chart is a directory containing templates, a values.yaml for defaults, and metadata. A Release is an instance of a chart deployed to a cluster. You can deploy the same chart multiple times with different values.

📂 Chart Structure

my-app-chart/
├── Chart.yaml            # name, version, dependencies
├── values.yaml           # default configuration values
├── templates/
│   ├── deployment.yaml   # Go templates referencing {{ .Values.* }}
│   ├── service.yaml
│   ├── ingress.yaml
│   ├── configmap.yaml
│   ├── _helpers.tpl      # reusable template snippets
│   └── NOTES.txt         # post-install message
└── charts/               # sub-chart dependencies

💻 Key Commands

# ── Install & Upgrade ─────────────────────────────────
helm install my-release ./my-app-chart            # install from local chart
helm install my-release bitnami/postgresql         # from a chart repo
helm upgrade my-release ./my-app-chart            # upgrade with new values
helm upgrade --install my-release ./my-app-chart  # install or upgrade

# ── Values ────────────────────────────────────────────
helm install my-release ./chart -f prod-values.yaml  # override defaults
helm install my-release ./chart --set replicas=5     # inline override

# ── Inspect & Debug ───────────────────────────────────
helm template ./my-app-chart                      # render templates locally
helm get values my-release                        # see deployed values
helm history my-release                           # release history
helm rollback my-release 2                        # rollback to revision 2

# ── Repos ─────────────────────────────────────────────
helm repo add bitnami https://charts.bitnami.com/bitnami
helm search repo postgresql

🔄 Helm vs. Kustomize

Helm uses Go templating — powerful but can produce complex, hard-to-read templates. Best for packaging reusable charts (community or internal) and managing complex apps with many config variants.

Kustomize (built into kubectl as kubectl apply -k) uses a template-free overlay approach — you start with base manifests and layer on patches for each environment. No templating language, just structured YAML overlays. Many teams use Kustomize for simple internal apps and Helm for third-party software.

They're not mutually exclusive — you can use helm template to render manifests, then manage them with Kustomize overlays.

🏗️ The Two-Layer Model

In the K8s world, infrastructure management splits into two distinct layers. Understanding this boundary is critical:

Terraform has a Kubernetes provider that can manage K8s resources, but most teams avoid mixing the two layers in the same state. The dominant pattern is: Terraform provisions the cluster and node pools → a GitOps tool (ArgoCD/Flux) or Helm manages everything inside it.

📝 What Terraform Manages for K8s

# Typical Terraform for EKS
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.0"

  cluster_name    = "production"
  cluster_version = "1.29"

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  eks_managed_node_groups = {
    general = {
      instance_types = ["m6i.xlarge"]
      min_size       = 3
      max_size       = 10
      desired_size   = 5
    }
    gpu = {
      instance_types = ["g5.xlarge"]
      min_size       = 0
      max_size       = 4
      taints = { gpu = { value = "true", effect = "NO_SCHEDULE" } }
    }
  }
}

🔀 Alternatives to Terraform

Pulumi: Same concept as Terraform but you write real code (TypeScript, Python, Go) instead of HCL. Strong K8s support.

Crossplane: Manages cloud infrastructure using K8s-native CRDs. You define an RDS instance as a K8s manifest, and Crossplane's controllers provision it. "Infrastructure as K8s resources." Popular at K8s-native shops because it unifies the management plane.

CDK for Terraform (CDKTF): Write Terraform in TypeScript/Python/Go. Best of both worlds if your team knows Terraform but wants real language features.

🧩 CRDs: Custom Resource Definitions

A CRD extends the K8s API with a new resource type. Once you apply a CRD, you can create, read, update, and delete instances of that new type using kubectl, exactly like built-in resources. For example, after installing cert-manager, you get a Certificate resource type:

apiVersion: cert-manager.io/v1
kind: Certificate          # ← this is a CRD, not built into K8s
metadata:
  name: api-tls
spec:
  secretName: api-tls-secret
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - api.example.com

CRDs are just the schema definition. The behavior comes from a controller that watches for instances of that CRD and acts on them — that's where Operators come in.

🤖 Operators: The Pattern

An Operator is a CRD + a custom controller that embeds domain-specific operational knowledge. It's a program running in your cluster that watches for your custom resources and does the right thing — provisioning, scaling, backing up, failover — automatically.

For example, a PostgreSQL Operator (like CloudNativePG or Zalando's postgres-operator) watches for PostgresCluster CRDs and handles: provisioning primary + replicas, automated failover, backups to S3, point-in-time recovery, version upgrades. All the ops knowledge of running Postgres, encoded as software.

🌟 Essential Operators You'll Encounter

cert-manager: Automates TLS certificate lifecycle. Talks to Let's Encrypt, Vault, or cloud CAs. Defines Certificate, Issuer, and ClusterIssuer CRDs.

External Secrets Operator: Syncs secrets from Vault/AWS SM/GCP SM into K8s Secrets. ExternalSecret CRD.

Prometheus Operator (kube-prometheus-stack): Defines ServiceMonitor, PodMonitor, PrometheusRule CRDs. Makes it declarative to configure monitoring.

ArgoCD: GitOps controller. Application CRD defines what to deploy from which git repo.

Crossplane: Cloud infrastructure as CRDs. Provisions RDS, S3, VPCs via K8s manifests.

🏠 Local Cluster Tools

For a guided OrbStack + kind + Tilt walkthrough, use the Local K8s Lab section earlier in the path. This section is the broader tool survey.

# kind — create a cluster in seconds
kind create cluster --name dev --config kind-config.yaml
kind load docker-image my-app:latest --name dev  # load local images
kind delete cluster --name dev

# Multi-node kind config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker

🔄 Tilt — Inner Loop Development

Tilt watches your source code, automatically rebuilds container images, and hot-reloads them into your local K8s cluster. It collapses the "edit → build → push → deploy → test" cycle into "edit → see changes." Think of it as the K8s equivalent of hot-reload for your entire microservice stack.

You define a Tiltfile (written in Starlark, a Python dialect) that describes how to build and deploy your services:

# Tiltfile
docker_build('my-app', '.')    # build image from Dockerfile
k8s_yaml('k8s/deployment.yaml') # apply manifests
k8s_resource('my-app',
  port_forwards='8080:8080',   # auto port-forward
  resource_deps=['database'])   # dependency ordering

Tilt also provides a web UI showing the status of all your services, their logs, and build status in one dashboard.

🏎️ ko — Fast Go Container Builds

ko builds Go applications into container images without a Dockerfile. It compiles your Go binary and layers it onto a minimal base image, then pushes to a registry. It's dramatically faster than Docker builds for Go and produces minimal, secure images.

At a K8s infra shop where controllers and operators are written in Go, ko is likely part of the standard workflow:

# Build and push to registry
ko build ./cmd/my-controller

# Apply manifest with ko resolving image references
ko apply -f config/        # replaces image: ko://... with real image refs

# Use with kind for local dev
KO_DOCKER_REPO=kind.local ko apply -f config/

🔨 Other Build & Image Tools

Skaffold: Google's build/deploy tool (similar to Tilt). Watches code, builds images, deploys to cluster. Has more CI/CD pipeline features.

Buildpacks (pack): Builds container images from source without a Dockerfile. Detects language and framework automatically.

kaniko: Builds Docker images inside a container without Docker daemon. Essential for building images in K8s-based CI (you can't run Docker-in-Docker easily in K8s pods).

Earthly: Makefile + Dockerfile hybrid. Reproducible builds that work locally and in CI identically.

🔄 GitOps (ArgoCD & Flux)

GitOps is the dominant deployment pattern in the K8s world. The idea: Git is the single source of truth for your cluster state. A GitOps controller running in the cluster watches a Git repo, and any time the repo changes, it syncs the cluster to match.

ArgoCD is the most popular GitOps tool. It provides a web UI showing the sync status of every application, diff views, manual sync triggers, and multi-cluster support. It defines an Application CRD that points to a Git repo path and a target cluster/namespace.

Flux (from Weaveworks/CNCF) is the alternative — more lightweight, "Git-native," and composable with its own set of CRDs (GitRepository, Kustomization, HelmRelease).

📈 Autoscaling — Three Dimensions

HorizontalPodAutoscaler (HPA): Scales pod replicas based on CPU, memory, or custom metrics. Like ECS Service Auto Scaling with target tracking.

VerticalPodAutoscaler (VPA): Adjusts pod resource requests/limits over time based on actual usage. Helps right-size pods. Can be advisory-only or auto-apply.

Cluster Autoscaler / Karpenter: Adds or removes nodes based on pending pods. Cluster Autoscaler works with cloud-provider auto-scaling groups. Karpenter (from AWS, now CNCF) is more modern — it provisions nodes directly via the cloud API, faster and more flexible. If you're on EKS, Karpenter is likely what you'll use.

KEDA (Kubernetes Event-Driven Autoscaling): Scales based on external event sources — SQS queue depth, Kafka lag, Prometheus queries, cron schedules. Much richer than HPA alone.

🔒 RBAC (Role-Based Access Control)

RBAC controls who can do what in the cluster. Four resource types:

Role / ClusterRole: Define permissions (which API verbs on which resource types). Role is namespace-scoped, ClusterRole is cluster-wide.

RoleBinding / ClusterRoleBinding: Attach a Role to a subject (user, group, or ServiceAccount).

ServiceAccounts are identities for pods. Combined with IRSA (IAM Roles for Service Accounts on EKS), a ServiceAccount can map to an IAM role, giving pods fine-grained AWS permissions — replacing the ECS Task Role pattern.

🛡️ Policy Engines & Admission Control

Admission controllers intercept requests to the K8s API before they're persisted. This is where you enforce security policies, inject sidecars, mutate resources, and validate configurations.

OPA/Gatekeeper: Policy engine using Rego language. Define ConstraintTemplates for policies like "all pods must have resource limits" or "no privileged containers."

Kyverno: K8s-native policy engine. Policies are written as K8s resources (YAML), not a separate language. Lower learning curve. Can validate, mutate, and generate resources.

Pod Security Standards (PSS) / Pod Security Admission: Built-in, replaces the deprecated PodSecurityPolicies. Three levels: Privileged, Baseline, Restricted. Applied per-namespace.

📊 Observability Stack

K8s doesn't ship with built-in monitoring (unlike CloudWatch in ECS). You build an observability stack:

Metrics: Prometheus (collection) + Grafana (visualization). The Prometheus Operator makes this declarative. metrics-server provides basic CPU/memory for HPA.

Logs: Fluentd/Fluent Bit (collection) → Elasticsearch/Loki (storage) → Grafana/Kibana (query). Or a managed service like Datadog, New Relic.

Traces: OpenTelemetry (instrumentation) → Jaeger/Tempo (storage and UI). Essential for debugging microservice architectures.

The "Grafana stack" (Loki + Tempo + Mimir + Grafana) is becoming the de facto open-source standard, especially with OpenTelemetry as the unified collection layer.

🔀 Multi-Cluster & Federation

At a core K8s infra shop, you'll likely manage multiple clusters. Patterns include:

Cluster API (CAPI): Manages the lifecycle of K8s clusters themselves using K8s resources. You define a Cluster CRD, and CAPI provisions the actual infrastructure. It's "Kubernetes managing Kubernetes."

Submariner / Skupper: Cross-cluster networking, letting Services in one cluster reach Services in another.

vCluster: Virtual clusters — lightweight K8s clusters that run inside a host cluster's namespace. Great for multi-tenancy, CI environments, and dev isolation.

🗂️ By Category

Cluster Provisioning

EKS / GKE / AKS (managed) · kubeadm (manual) · Cluster API (declarative) · k3s (lightweight) · Talos (immutable OS)

Infrastructure as Code

Terraform · Pulumi · Crossplane · CDK for Terraform · OpenTofu

Package Management & Templating

Helm · Kustomize · Carvel (ytt + kapp) · Jsonnet · cdk8s

GitOps & Continuous Delivery

ArgoCD · Flux · ArgoRollouts · Flagger · Spinnaker · Tekton

Networking

Cilium · Calico · Flannel (CNI) · NGINX Ingress · Traefik · Envoy/Emissary · Gateway API · CoreDNS

Service Mesh

Istio · Linkerd · Cilium Service Mesh · Consul Connect

Security

OPA/Gatekeeper · Kyverno · Falco (runtime) · Trivy (scanning) · cert-manager · External Secrets Operator · Sealed Secrets · SPIFFE/SPIRE

Observability

Prometheus · Grafana · Loki · Tempo · Mimir · OpenTelemetry · Jaeger · Datadog · Pixie (eBPF)

Local Dev

kind · minikube · k3d · Tilt · Skaffold · Telepresence · DevSpace

Build & Image

ko · kaniko · Buildpacks · Earthly · Docker/Podman · Harbor (registry)

Autoscaling

HPA · VPA · Cluster Autoscaler · Karpenter · KEDA

Operator Frameworks

Kubebuilder · Operator SDK · Metacontroller · KUDO · controller-runtime

Multi-Cluster

Cluster API · vCluster · Submariner · Liqo · Admiralty

Developer Experience

kubectl · kubectx/kubens · k9s (terminal UI) · Lens (desktop UI) · krew (plugin manager) · stern (multi-pod log tailing)

🗺️ Recommended Learning Path

Week 1-2: Get comfortable with kubectl, deploy apps to kind, understand Pods → Deployments → Services → Ingress. Write raw manifests by hand. Break things.

Week 3-4: Learn Helm (install third-party charts, then write your own). Understand Kustomize. Set up port-forwarding, debug failed pods, read events.

Week 5-6: Dive into RBAC, NetworkPolicies, resource quotas. Understand the security model. Learn about ServiceAccounts and IRSA if on EKS.

Week 7-8: Study Operators and CRDs. Install cert-manager, understand how it works. Read the Kubebuilder book. Start reading your team's operator code.

Ongoing: GitOps with ArgoCD, observability stack, service mesh concepts, Cluster API, advanced scheduling (affinity, topology spread constraints), admission webhooks. Attend KubeCon talks on YouTube.

🧩 The Unlock: Platform vs Workload

Spark and Airflow are not just "containers you run." They are distributed systems: schedulers, workers, drivers, executors, metadata stores, logs, secrets, retry loops, autoscaling signals, and failure domains.

ECS asks: what task or service should I run? Kubernetes asks: what state should exist, and which controller reconciles it? Once you see a platform as controllers plus runtime units, the mapping becomes obvious.

🏪 Scenario: Marketplace Data Platform

Imagine a marketplace company processing clickstream events, orders, fraud signals, search relevance data, and ML features.

🗺️ Platform-Neutral Architecture

🧠 The Brain-Unlock Matrix

Read each row as: "what problem am I solving, how does each platform express it, and what is the underlying mental model?"

🟧 ECS Execution Flow

☸️ Kubernetes Execution Flow