mirror of
https://github.com/hanzo-ml/spark-operator.git
synced 2026-07-27 06:55:02 +00:00
2078e740f9
* feat: optional PodDisruptionBudget for Spark drivers (#2949) Refs: https://github.com/kubeflow/spark-operator/issues/2949 Add an opt-in PodDisruptionBudget for the driver pod so a voluntary eviction (e.g. `kubectl drain`) cannot kill the driver mid-job. The apiserver's eviction admission handler consults a matching PDB; without one the eviction is admitted and the operator only learns of it after the fact via its watch. A PDB is the only lever the operator has here that does not put it on the eviction hot path. Gated twice, off by default so upgrades are no-op: --enable-driver-pdb controller flag (default false) spec.driverPodDisruptionBudget per-app *bool (nil/false = no PDB) A bare *bool suffices: the driver is a singleton, so MinAvailable is fixed at 1 and a selector struct has nothing to grow into. Lifecycle (driver_pdb.go): - ensureDriverPDB converges PDB existence to spec on every Submitted/Running reconcile, so it recovers from operator restart with the gate flipped on, live spec edits, and transient API failures (retry.RetryOnConflict + controller-runtime backoff). - createDriverPDB/deleteDriverPDB read the cache first for idempotency; AlreadyExists/NotFound are treated as success. - The PDB carries a controller OwnerReference plus LabelLaunchedBySparkOperator + LabelSparkAppName, so kube GC reaps it on app deletion and the cache informer can filter to owned PDBs. - cleanUpOnTermination deletes the PDB on Completed/Failed; the terminal-app reconcile re-runs it to reap PDBs orphaned when the controller itself was drained mid-transition. - deleteSparkResources drops the PDB before the driver pod so an in-flight external drain does not block on a stale PDB. RBAC: the PDB entry in cache.Options.ByObject starts an informer, which needs list+watch even though the code only Get/Create/Deletes. Granted get;list;watch;create;delete on poddisruptionbudgets via the kubebuilder marker (regenerated config/rbac/role.yaml), the Helm chart helper, and the chart rbac unit test. Missing verbs would loop the informer on Forbidden and saturate the shared client rate limiter. Helm: controller.driverPodDisruptionBudget.enable (default false) emits --enable-driver-pdb=true; ci-values.yaml turns it on for e2e. Tests: envtest specs cover the gate matrix and the convergence cases ensureDriverPDB owns (restart-with-gate-on, spec false->true and true->false on a running app, transient Create failure retried via an interceptor.Funcs client). E2e submits real apps with the field set and omitted and asserts PDB creation, GC on deletion, and absence when not opted in. Makefile: kind-load-spark-image preloads the Spark runtime image and e2e-test pins IMAGE_TAG=local with the load targets as prerequisites, so `make deploy` and `make e2e-test` are self-contained on a local Kind cluster instead of hitting ImagePullBackOff. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Vincent Janelle <randomfrequency@gmail.com> * test: use events.FakeRecorder in driver PDB test NewReconciler takes an events.EventRecorder, not the legacy record.EventRecorder. Align driver_pdb_test.go with controller_test.go. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Vincent Janelle <randomfrequency@gmail.com> * allow for kustomize to patch pdb value; default to true in ci Signed-off-by: Vincent Janelle <randomfrequency@gmail.com> --------- Signed-off-by: Vincent Janelle <randomfrequency@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>