Kubernetes operator

This page is a web-friendly tour of the operator usage guide. Every YAML and command below is adapted from the always-working reference, deploy/e2e/e2e.sh, which builds every image, deploys the operator, instruments a raw JPetStore, runs a campaign, and asserts the result in a kind cluster.

The two-CRD model

You instrument once (a target), then run tests against it (campaigns). One target can be driven by many campaigns over time.

Install

The install is three things: the two CRDs, the namespaced RBAC (ServiceAccount + Role + RoleBinding), and the controller Deployment. It also needs three images it launches or injects, each supplied via a flag:

Each flag empty falls back to a built-in default; in practice you pin them to a fixed tag you built and loaded, so the cluster uses your image instead of pulling.

First build + kind-load the three images (each build.sh takes [TAG] [KIND_CLUSTER]), then install with Helm (recommended) or kustomize.

# build + kind-load the three images the operator uses
deploy/agents-image/build.sh    0.3.0 <kind-cluster>
deploy/runner-image/build.sh    0.3.0 <kind-cluster>
deploy/dashboard-image/build.sh 0.3.0 <kind-cluster>

Install from the published repo (no clone, no build)

The chart is published to a GitHub Pages Helm repo, and its default images to GitHub Container Registry (ghcr.io/ianp94/basquin-*) — a plain helm install pulls real images with nothing to build:

helm repo add basquin https://ianp94.github.io/basquin/charts
helm repo update
helm install basquin basquin/basquin-operator \
  --namespace basquin-system --create-namespace \
  --set fullnameOverride=basquin

Build + install from a checkout (local / kind)

For a local kind cluster, build the images (below) and install from the checkout, pinning the locally-built images:

helm install basquin ./deploy/helm/basquin-operator \
  --namespace basquin-system --create-namespace \
  --set fullnameOverride=basquin \
  --set imageTag=0.3.0 \
  --set image.repository=basquin/operator \
  --set images.agents=basquin/agents \
  --set images.runner=basquin/runner \
  --set images.dashboard=basquin/dashboard

imageTag sets all four image tags at once; on a published release it defaults to the chart's appVersion, so a repo install needs no version flags.

Or install with kustomize

kubectl apply -f operator/config/crd/bases/basquin.dev_basquintargets.yaml
kubectl apply -f operator/config/crd/bases/basquin.dev_basquincampaigns.yaml
kubectl create namespace basquin-system

kubectl kustomize operator/config/default \
  | sed 's#image: controller:latest#image: basquin/operator:0.3.0#' \
  | kubectl apply -f -

# then wire the three image flags (Helm does this for you)
for arg in --agents-image=basquin/agents:0.3.0 \
           --runner-image=basquin/runner:0.3.0 \
           --dashboard-image=basquin/dashboard:0.3.0 ; do
  kubectl -n basquin-system patch deploy basquin-controller-manager --type=json \
    -p="[{\"op\":\"add\",\"path\":\"/spec/template/spec/containers/0/args/-\",\"value\":\"$arg\"}]"
done
kubectl -n basquin-system rollout status deploy/basquin-controller-manager --timeout=120s

The kubectl patch loop appends unconditionally — re-running it by hand duplicates the args (guard each with a grep -q check as deploy/e2e/e2e.sh does, or just use Helm, which wires the flags declaratively).

Instrument an app — BasquinTarget

Point a BasquinTarget at a Deployment in the same namespace. The operator patches its pod template: an initContainer copies the agents into a shared emptyDir, the agent flags are appended to the container's JVM opts env var (never replacing your heap/GC flags), the coverage port is exposed — then rolls it out.

apiVersion: basquin.dev/v1alpha1
kind: BasquinTarget
metadata:
  name: jpetstore
  namespace: basquin-system
spec:
  deploymentRef:
    name: jpetstore
  container: jpetstore          # REQUIRED when the pod has >1 container

  # CATALINA_OPTS (Tomcat) | JAVA_TOOL_OPTIONS (everything else, the default). Flags are APPENDED.
  jvmOptsVar: CATALINA_OPTS

  agents:
    threadTracker: true          # native JVMTI leak/thread oracle (default true)
    coverage:
      enabled: true              # JaCoCo tcpserver for coverage-guided-over-HTTP
      port: 6300                 # coverage port inside the pod (default 6300)
      includes: "org.mybatis.jpetstore.*"   # REQUIRED when enabled — no wildcard default

  invariants:
    mode: soft                   # soft (record + continue) | hard (fail the iteration)
    latencyMaxMs: 25
    heapDeltaMaxKb: 256

  coverageService: true          # headless Service so one driver reaches every replica by DNS

Wait for status.phase to reach Injected, then grab the coverage endpoint the campaign will consume:

kubectl -n basquin-system get basquintargets
# NAME        DEPLOYMENT   PHASE      INSTRUMENTED   AGE
# jpetstore   jpetstore    Injected   1              30s

kubectl -n basquin-system get basquintarget jpetstore -o jsonpath='{.status.coverageEndpoint}'
# jpetstore-basquin-jacoco.basquin-system.svc.cluster.local:6300

Target phases: Pending → Injecting → Injected (or Reverting / Error). With coverageService: true the operator creates a headless Service named after the Deployment it targets — <deploymentRef.name>-basquin-jacoco (not the CR's own name; they match in this example) — and writes its DNS name to status.coverageEndpoint.

Run a test — BasquinCampaign

Once the target is Injected, a BasquinCampaign drives it. The operator gates on the target being Injected, reads its status.coverageEndpoint, launches the driver Job, and aggregates status.

apiVersion: basquin.dev/v1alpha1
kind: BasquinCampaign
metadata:
  name: jpetstore-campaign
  namespace: basquin-system
spec:
  targetRef:
    name: jpetstore

  # REQUIRED — the app's in-cluster HTTP entrypoint (you create the app Service yourself).
  baseURL: http://jpetstore-app.basquin-system.svc.cluster.local:8080

  driver:
    # Bound the run — set EXACTLY ONE of iterations / duration (CEL-enforced).
    iterations: 200
    # duration: "10m"          # ...or a Go-style duration; runner exits cleanly at the deadline

    grammarConfigMap: jpetstore-grammar
    corpusConfigMap: jpetstore-corpus

    # Where the app's .class files live INSIDE the target image (for JaCoCo covered/total).
    classesPath: /usr/local/tomcat/webapps/ROOT/WEB-INF/classes

  # dashboard: {}              # per-campaign dashboard on by default — see below

baseURL is required — there's no default; the operator does not front the app with a Service of its own. Set exactly one of driver.iterations or driver.duration. classesPath must point at real .class files in the target image — a war-only image (no exploded WEB-INF/classes) has nothing to copy and the run fails loudly rather than reporting a misleading 0% coverage.

Grammar & corpus ConfigMaps

The driver's exploration surface is a grammar (structure) and a corpus (values), each delivered as a ConfigMap. The grammar ConfigMap has one key — the grammar file; grammarKey defaults to the sole key:

kubectl -n basquin-system create configmap jpetstore-grammar \
  --from-file=jpetstore.grammar=examples/grammar/jpetstore.grammar \
  --dry-run=client -o yaml | kubectl apply -f -

The corpus ConfigMap is a flat map of files keyed by basename. Because --from-file on a directory is non-recursive, pass both the top-level route-seed files and the nested values/ files explicitly:

kubectl -n basquin-system create configmap jpetstore-corpus \
  --from-file=examples/corpus/jpetstore/ \
  --from-file=examples/corpus/jpetstore/values/ \
  --dry-run=client -o yaml | kubectl apply -f -

The grammar's @-value-file basenames (e.g. itemId.txt) must line up with ConfigMap keys — that's why the flat basename convention matters.

Load / soak mode — spec.mode: load

A campaign runs in one of two modes (spec.mode, default explore). An explore run, on completion, emits its interesting "replay corpus" (the inputs that reached new coverage) as a campaign-owned ConfigMap <campaign>-corpus-out (status.corpusConfigMap). A load run replays a saved corpus at a fixed concurrency for a duration — no mutation, no coverage — watching the same invariant oracles under sustained traffic. Fuzz to discover the interesting states, then hammer those states under load.

apiVersion: basquin.dev/v1alpha1
kind: BasquinCampaign
metadata:
  name: jpetstore-load
  namespace: basquin-system
spec:
  mode: load
  targetRef:
    name: jpetstore
  baseURL: http://jpetstore-app.basquin-system.svc.cluster.local:8080
  driver:
    duration: 30m                            # load is time-bounded (requires duration, not iterations)
    concurrency: 50                          # parallel in-flight requests
    corpusConfigMap: jpetstore-campaign-corpus-out   # the corpus the explore run emitted
    # warmup: 30s                            # optional: excluded from the reported latency percentiles

Load requires driver.corpusConfigMap and forbids a grammar (CEL-enforced); its driver Job is coverage-free. Read the results from status.load:

kubectl -n basquin-system get basquincampaign jpetstore-load -o jsonpath='{.status.load}'
# {"requests":1284003,"throughputRps":"713.4","latencyMs":{"p50":8,"p90":22,"p99":61,"max":240},
#  "heapDriftKb":1840,"threadDrift":0,"violations":{"latency":12,"heap":0,"thread":0}}

# ...or via the CLI: --corpus-from reuses the corpus the explore campaign emitted (its
# <campaign>-corpus-out ConfigMap), copied into a fresh load-campaign-owned ConfigMap.
# (--corpus <dir> replays local files instead.)
basquin run --name jpetstore-load --mode load --target jpetstore --base-url http://jpetstore-app...:8080 \
  --duration 30m --concurrency 50 --corpus-from jpetstore-campaign --watch

First cut: only violations.latency is threshold-gated (against invariants.latencyMaxMs); heap/thread are reported as end-to-end drift, measured on the driver.

Dashboard

By default every campaign gets its own dashboard — the operator creates a Deployment + Service (<campaign>-dashboard, ClusterIP on port 7070), owner-referenced to the campaign so it's garbage-collected when the campaign is deleted. Reads are token-gated (DD-028): the operator mints a per-campaign token, and basquin dashboard prints a tokenized URL. Reach it with a port-forward:

kubectl -n basquin-system get basquincampaign jpetstore-campaign -o jsonpath='{.status.dashboardURL}'
# http://jpetstore-campaign-dashboard.basquin-system.svc.cluster.local:7070

kubectl -n basquin-system port-forward svc/jpetstore-campaign-dashboard 7070:7070
# then open http://localhost:7070

The dashboard outlives the run — it's GC'd with the campaign, not with the driver Job, so results stay viewable after the driver completes. On spec.dashboard you can set enabled: false to create none, or externalPush: "host:port" to fan many campaigns into one shared, long-lived dashboard for cross-campaign comparison.

Reading results

The driver writes a machine-readable summary the operator surfaces in campaign status — no dashboard scraping needed:

kubectl -n basquin-system get basquincampaign
# NAME                 TARGET      PHASE       COVERAGE   FINDINGS   AGE
# jpetstore-campaign   jpetstore   Completed   23.1       19         5m

kubectl -n basquin-system get basquincampaign jpetstore-campaign -o yaml | less
# status.phase / coveragePct / findings / dashboardURL / driverJob / startTime / completionTime

Campaign phase machine: Pending (target not yet Injected) → Provisioning (creating dashboard / driver Job) → Running (driver Job active) → Completed (Job succeeded, summary read) or Failed (driver failed, or the target went away mid-run). status.driverJob names the Job — tail its logs to triage a run.

Editing a running campaign

The campaign spec is hashed onto the driver Job. Jobs are immutable, so editing the spec triggers a fresh run — the operator deletes and recreates the driver Job with the new config. That's the intended "I changed the test" semantics: change iterations, the grammar ConfigMap ref, or an invariant, re-apply, and a new run starts. A steady, unchanged campaign is a no-op reconcile.