* feat: optional PodDisruptionBudget for Spark drivers (#2949)
Refs: https://github.com/kubeflow/spark-operator/issues/2949
Add an opt-in PodDisruptionBudget for the driver pod so a voluntary
eviction (e.g. `kubectl drain`) cannot kill the driver mid-job. The
apiserver's eviction admission handler consults a matching PDB; without
one the eviction is admitted and the operator only learns of it after
the fact via its watch. A PDB is the only lever the operator has here
that does not put it on the eviction hot path.
Gated twice, off by default so upgrades are no-op:
--enable-driver-pdb controller flag (default false)
spec.driverPodDisruptionBudget per-app *bool (nil/false = no PDB)
A bare *bool suffices: the driver is a singleton, so MinAvailable is
fixed at 1 and a selector struct has nothing to grow into.
Lifecycle (driver_pdb.go):
- ensureDriverPDB converges PDB existence to spec on every
Submitted/Running reconcile, so it recovers from operator restart
with the gate flipped on, live spec edits, and transient API
failures (retry.RetryOnConflict + controller-runtime backoff).
- createDriverPDB/deleteDriverPDB read the cache first for
idempotency; AlreadyExists/NotFound are treated as success.
- The PDB carries a controller OwnerReference plus
LabelLaunchedBySparkOperator + LabelSparkAppName, so kube GC reaps
it on app deletion and the cache informer can filter to owned PDBs.
- cleanUpOnTermination deletes the PDB on Completed/Failed; the
terminal-app reconcile re-runs it to reap PDBs orphaned when the
controller itself was drained mid-transition.
- deleteSparkResources drops the PDB before the driver pod so an
in-flight external drain does not block on a stale PDB.
RBAC: the PDB entry in cache.Options.ByObject starts an informer, which
needs list+watch even though the code only Get/Create/Deletes. Granted
get;list;watch;create;delete on poddisruptionbudgets via the kubebuilder
marker (regenerated config/rbac/role.yaml), the Helm chart helper, and
the chart rbac unit test. Missing verbs would loop the informer on
Forbidden and saturate the shared client rate limiter.
Helm: controller.driverPodDisruptionBudget.enable (default false) emits
--enable-driver-pdb=true; ci-values.yaml turns it on for e2e.
Tests: envtest specs cover the gate matrix and the convergence cases
ensureDriverPDB owns (restart-with-gate-on, spec false->true and
true->false on a running app, transient Create failure retried via an
interceptor.Funcs client). E2e submits real apps with the field set and
omitted and asserts PDB creation, GC on deletion, and absence when not
opted in.
Makefile: kind-load-spark-image preloads the Spark runtime image and
e2e-test pins IMAGE_TAG=local with the load targets as prerequisites, so
`make deploy` and `make e2e-test` are self-contained on a local Kind
cluster instead of hitting ImagePullBackOff.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Vincent Janelle <randomfrequency@gmail.com>
* test: use events.FakeRecorder in driver PDB test
NewReconciler takes an events.EventRecorder, not the legacy
record.EventRecorder. Align driver_pdb_test.go with controller_test.go.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Vincent Janelle <randomfrequency@gmail.com>
* allow for kustomize to patch pdb value; default to true in ci
Signed-off-by: Vincent Janelle <randomfrequency@gmail.com>
---------
Signed-off-by: Vincent Janelle <randomfrequency@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
