zeekayandHanzo Dev
dada5a3169
Two endpoints were answering with the right status code and an empty
truth, so every fleet looked instrumented while reporting nothing.
GET /v1/health encoded apihealth.APIReply through jsonv2, which has no
default representation for time.Duration — and every Result carries one.
The marshal therefore failed on every node, every time, and the handler
fell back to {"healthy":…,"error":"health reply encode failed"}. The
status code is written before the encode, so k8s probes and dashboards
stayed green while the body carried no checks at all. Measured on Zoo,
Hanzo and Pars mainnet, node v1.34.9 and v1.36.2 alike.
The type is defined with encoding/json tags and the POST (jsonrpc) path
already encodes it through that codec, which is why POST returned the
full check set while GET returned nothing. One wire type deserves one
encoder: GET now uses encoding/json too, so the two paths agree by
construction. encoding/json also maps invalid UTF-8 in check Details to
U+FFFD instead of failing, which is the behaviour the old jsontext
AllowInvalidUTF8 option was reaching for. The buffer stays: it keeps a
partial encode from shipping a torn body.
/v1/metrics answered 200 with zero bytes for the same shape of reason.
luxfi/metric resolves NewRegistry() to a no-op registry unless the
binary is built with -tags metrics (registry_noop.go, //go:build
!metrics), and only the `full` profile passed it. Images build with the
default `minimal` profile, so every metric in luxd registered into a
black hole while --api-metrics-enabled=true still advertised metrics as
on. Whether metrics are served is the runtime flag's call; a build tag
must not silently overrule it. `metrics` becomes a base tag on every
profile.
Verified: the three new handler tests fail against the jsonv2 encoder
(checks decode empty, error field nil — the exact production symptom)
and pass after. `go tool nm` shows (*registry).Gather absent from a
default build and present once the tag is set.
Co-authored-by: Hanzo Dev <dev@hanzo.ai>
Health Checking
Health Check Types
Readiness
Readiness is a special type of health check. Readiness checks will only run until they pass for the first time. After a readiness check passes, it will never be run again. These checks are typically used to indicate that the startup of a component has finished.
Health
Health checks typically indicate that a component is operating as expected. The health of a component may flip due to any arbitrary heuristic the component exposes.
Liveness
Liveness checks are intended to indicate that a component has become unhealthy and has no way to recover.
Naming and Tags
All registered checks must have a unique name which will be included in the health check results.
Additionally, checks can optionally specify an arbitrary number of tags which can be used to group health checks together.
Special Tags
- "All" is a tag that is automatically added for every check that is registered.
- "Application" checks are checks that are globally applicable. This means that it is not possible to filter application-wide health checks from a response.
Health Check Worker
Readiness, Health, and Liveness checks are all implemented by using their own health check worker.
A health check worker starts a goroutine that updates the health of all registered checks every freq. By default freq is set to 30s.
When a health check is added it will always initially report as unhealthy.
Every health check runs in its own goroutine to maximize concurrency. It is guaranteed that no locks from the health checker are held during the execution of the health check.
When the health check worker is stopped, it will finish executing any currently running health checks and then terminate its primary goroutine. After the health check worker is stopped, the health checks will never run again.