The llmobs span views parameterize the trace_id predicate as `trace_id = $N`
and carry the real id in the request Variables map. narrowWindowByTraceID reads
the filter EXPRESSION via ExtractTraceIDsFromFilter, so it got back the literal
"$N", looked that up in distributed_trace_summary, found nothing, and — for
SignalTraces — short-circuited the whole span query to empty. Every trace-detail
observations/traces query returned zero rows fleet-wide even though the spans
existed (verified live: trace 9aa37a28...4406a held its gen_ai span the whole time).
Resolve $N/$name placeholders against q.variables before the trace_summary lookup
(mirroring the where-clause visitor). An unresolved placeholder skips the
optimization (runs the fully-substituted main query) instead of short-circuiting.
Inline literals keep the fast path.
Regression: TestNarrowWindowParameterizedTraceID (bound placeholder narrows, not
short-circuits; unbound placeholder skips, not short-circuits) + TestResolveTraceIDVars.
Require github.com/hanzoai/go-openai v1.41.0 directly now that the fork
declares its own module path; drop the sashabaranov/go-openai => fork
replace. Bump hanzoai/ai v1.813.6 -> v1.822.3. The local
replace github.com/hanzoai/cloud => ../cloud is preserved; the sibling
is on the v1.801.63 state, which drags in the owned hanzoai/go-cosyvoice
and hanzoai/go-openai-realtime forks in place of the upstream WqyJh ones.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swap every Go import of github.com/ClickHouse/clickhouse-go/v2 (and its
lib/driver, lib/chcol sub-packages) to the canonical Hanzo rebrand
github.com/hanzoai/datastore-go/v2. The datastore-go API is byte-identical
to clickhouse-go, so bare imports keep the `clickhouse` alias (path-only
change) and sub-path imports map 1:1 — no call-site logic changes.
Why: datastore-go registers sql.Register("datastore", ...) with a UNIQUE
name, whereas the upstream clickhouse-go fork registers sql.Register(
"clickhouse", ...) in its init(). Compiled into the hanzoai/cloud binary
alongside another "clickhouse" registrant, that duplicate database/sql
registration panicked cloud at boot. o11y's production/query plane now
contributes only the "datastore" name.
No sql.Open/sql.Register-by-name string changed: o11y opens the store via
the native datastore.Open(&datastore.Options{})/ParseDSN API, not a
database/sql driver name. The "clickhouse" telemetrystore provider value,
factory.MustNewName("clickhouse"), and db.system semconv stay as-is
(implementation surface, not driver registration) per repo doctrine.
Test double: bump github.com/hanzoai/clickhouse-go-mock v0.14.1 -> v0.14.2,
which is the same mock rebuilt on datastore-go so telemetrystoretest's
Provider satisfies the datastore.Conn-typed TelemetryStore interface.
Build unblock: mirror cloud's existing replace of
github.com/sashabaranov/go-openai => github.com/hanzoai/go-openai v1.40.0.
The ../cloud sibling forces hanzoai/ai v1.806.3, whose model package needs
the Hanzo go-openai fork's ReasoningContent field; replace directives don't
propagate from deps, so o11y needs its own copy. Pre-existing latent gap
surfaced by the graph recompute, not part of the driver swap.
clickhouse-go/v2 and hanzo-ds/go remain indirect (pulled by the
hanzoai/otel-collector schema migrator and the cloud sibling respectively)
— no o11y source imports them. CGO_ENABLED=0 GOWORK=off go build ./... = 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The embedded one-binary (hanzoai/cloud) runs o11y in-process behind the edge
gateway, which already validates the Hanzo IAM JWT and injects trusted identity
headers (X-Org-Id/X-User-Id/X-User-Email); the sharder gates cross-org. Yet
iamauthz round-trips BACK OUT to an external IAM Casbin enforcer (hanzo/o11y) for
the provision-time grant AND every per-read enforce — an enforcer the one-binary
carries no credentials for. Result: every /v1/o11y read 401s (provision Grant ->
Write -> add-policy fails authz_unavailable), breaking the console overview-metrics
widgets on ~9 secondary product pages.
Add localauthz: same enforced policy as iamauthz (a subject is authorized iff it
holds a required role in its org) but the relationship tuples live in-process
instead of in external IAM. Selected by config: O11Y_AUTHZ_PROVIDER=local (embed)
vs the default iam (standalone). Founding grants come from the same iamidentn
provision path, so the tuple set is populated before the per-route check on the
same request and re-populated on the first request after restart. Role metadata
stays in the local SQL store, exactly as iamauthz.
Decomplected: authorization follows the trust boundary. One IAM validates once at
the edge; the embedded service trusts the assertion instead of re-deriving it over
a synchronous external round-trip that adds a failure mode to every telemetry read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A GetIdentity error (e.g. provision: ensureOrg/Grant failure) was swallowed —
next.ServeHTTP without claims → downstream generic 'unauthenticated' 401 with NO
diagnostic. Log ::IDENTITY-GET-FAILED:: so the real cause is visible.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TypeRole's selector regex was ^([a-z-]{1,50}|\*)$ — lowercase + hyphen only, NO
digits. But the built-in role granted to every reader is O11yAdminRoleName =
'o11y-admin', which contains digits ('11'). So MustSelector('o11y-admin') failed the
regex and PANICKED (recovered→500) on every identity provision — breaking EVERY
authenticated o11y data read (query_range/services/rules/dashboards) in the embedded
runtime. Allow [a-z0-9-]; role names legitimately carry digits (o11y-admin,
otel-collector, …). Additive — existing lowercase/hyphen roles still match.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The embedded one-binary runtime (cloud) reaches PublicHandler with paths ALREADY
rewritten to /api/<res> (mount.go rewriteExternalPath strips the /v1/o11y prefix). The
ExternalPath wrapper then ran StripPrefix(routePrefix=/v1/o11y) on /api/<res>, which
404s (prefix already gone) — so EVERY embedded o11y data call (query_range, services,
rules, dashboards, and the versioned /api/vN forms) 404'd; only the health switch
passed through. Broaden that special-case: any /api/-prefixed path goes straight to the
router. Standalone (still /v1/o11y-prefixed) is unaffected — it falls through to
StripPrefix as before. Fixes console.hanzo.ai o11y-telemetry overview widgets.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
createPublicServer registered /api/vN + web routes but never called AddVersionlessAliases,
so the version-less /api/<resource> aliases lived ONLY on the o11yapiserver provider path,
NOT on the app.Server router that community.NewServer/PublicHandler assembles. The unified
cloud one-binary embeds o11y via community.NewServer, so /v1/o11y/<resource> (rewritten to
/api/<resource> by mount.go) 404'd — breaking the console's version-less o11y contract
(overview RED-metrics on ~9 product pages). Additive; no effect on /api/vN.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Complete the half-done driver convergence in sqlitesqlstore/provider.go
(origin/main had added the driver_test.go pragma guard + the http_handler
comment, but provider.go still imported modernc directly):
- Drop "modernc.org/sqlite" + "modernc.org/sqlite/lib"; import
github.com/hanzoai/sqlite, whose init registers the ONE "sqlite" driver
(mattn/SQLCipher under cgo, modernc pure-Go otherwise) — no more direct
modernc import that would double-register "sqlite" in the cgo cloud binary.
- WrapAlreadyExistsErrf now classifies via backend-neutral
sqlite.IsConstraint{Unique,PrimaryKey,ForeignKey} instead of the
modernc-typed *sqlite.Error + lib.SQLITE_CONSTRAINT_* checks.
- Build the DSN via sqlite.PragmaDSN so pragmas apply under BOTH backends.
The old hardcoded modernc `_pragma=journal_mode(wal)` form is SILENTLY
DROPPED by mattn — so under CGO=1 (how o11y links into the cloud binary)
journal_mode/busy_timeout never applied. driver_test.go now passes under
mattn too. _txlock (a driver param both backends honor) is appended.
go.mod: + hanzoai/sqlite v0.2.2 (pulls luxfi/crypto 1.19.26, mattn 1.14.47);
exclude mattn v2.0.3+incompatible (the mis-tagged pre-modules version the
cloud graph over-selects, which cannot build the cgo codec). Surgical — no
cloud-graph churn (cloud stays at its pseudo-version).
Verified: cmd/community builds -mod=readonly under CGO=0 AND CGO=1;
pkg/sqlstore tests green both backends (incl the pragma guard); 0 modernc
in the CGO=1 cmd/community graph (no double-registration). Pre-existing
unrelated: go build ./... hits a hanzoai/ai <-> go-openai ReasoningContent
skew in the cloud graph — not o11y source, not this change.
The server image is headless (O11Y_WEB_ENABLED=false, /api only); this serves
the dashboard SPA at the edge so o11y.hanzo.ai/ stays 200 with the newer backend.
- Dockerfile.site: re-serves the proven in-production SPA (0.1.0 /etc/o11y/web)
through hanzoai/static --spa (:3000), byte-identical — same pattern commerce-site
used while its from-source build was broken. TODO: swap to a from-source node
build stage once frontend/ pnpm build is green (mid rolldown-vite migration).
- docker-site.yaml: CI build+push ghcr.io/hanzoai/o11y-site on main.
Hanzo AI