* [operations/jsonnet] restructure our compiled output for multiple environments
* Clean up old files
* Ignore vendor/ path for microservices
* Include "extras" example for testing non-default outputs
* Drop extras for the size of the diff
* Allow RetryInfo to be enabled at cluster and runtime overrides
* Update manifest.md file
* Add CHANGELOG.md entry
* use RetryAfterOnResourceExhausted > 0 as cluster level control
* Additional scheduler tests
* Include scheduler integration make target
* Update integration tests
* Include tests in CI
* Include new integration tests in test-all target
* Include docker target for backend-scheduler integration
* Use t.TempDir for temporary directory
* Clean up after review
* Add missing comma
* Add e2e test coverage for rhythm
Updated e2e tests for rhythm/ingest. Also added a new `test-e2e-ingest` target in CI workflows and Makefile.
* Build image
* Refactor and update integration tests, configs, and utilities
Signed-off-by: Mario <mariorvinas@gmail.com>
* Remove unnecessary time.Sleep
* woah, forbidden
---------
Signed-off-by: Mario <mariorvinas@gmail.com>
* added kafka information to write dashboard
* fmt
* added distributor bytes recieved
* changelog
* proper unit
* use histogram quantiles instead of heatmaps
* compile mixin again
* split kafka appends between failed and success for better resolution
* New config and logic to only search RF1 blocks after a given date, and stop all flushing from ingesters
* Fix filtering function
* Fix lint issue
* Re-generate manifest
* Add RF1After query param to search
Introduced a new `RF1After` field in the search request to allow filtering blocks by replication factor and time. Updated backend logic and tests to handle this functionality. Adjusted max search duration to enhance usability and clarity in search tests.
Signed-off-by: Mario <mariorvinas@gmail.com>
* Update Docker images for protobuf gen
Signed-off-by: Mario <mariorvinas@gmail.com>
* Improve RF1After handling and validation in trace queries
Added RF1After field to trace query validation logic and related structures. Updated request parsing, test cases, and query configuration to support RF1After. Removed deprecated RF1After config from querier module, ensuring runtime determination.
Signed-off-by: Mario <mariorvinas@gmail.com>
* Undo dumb logic
* Fix config options
* You again proto?
* Add RF1After filtering logic to tag searches
Signed-off-by: Mario <mariorvinas@gmail.com>
* WTF proto
* Fix comment
---------
Signed-off-by: Mario <mariorvinas@gmail.com>
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
* Begin changes for backend-scheduler module
* Comment breadcrumb
* Initial compactor extensions and backendscheduler module
* Add initial jsonnet
* Some cleanup after discussion
* Add basic integration test for jobs fetching
* Flesh out some unit tests
* Clean up and add test coverage
* Improve coverage
* Clean up dead ring code
* Set default value for compactor job polling
* Add log message for backend scheduling
* Add note about fixing the compactor flow
* Small operational fixes
* Plumb client defaults
* Fix job handling and module dependency chicken/egg/lizard
* Update config manifest
* Move CompactionBlockSelector
* Revert tempodb compactor changes
* Revert "Revert tempodb compactor changes"
This reverts commit 239c4f2a9df2c289164321f427ac56fa4b7d92cc.
* Add backendworker module and work around comapctor dependencies
* Move job work and queue into separate package
* Add jsonnet for backend-worker
* Fix backendscheduler tests
* Update docs manifest
* Revert compaction method rename
* Drop debug
* Revert compactor module changes
* Drop debug
* Revert compactor test change
* Fix mock interface after extension
* Start e2e integration test
* Lint and clean up
* Move tmpDir creation to the top
* Commit for sharing
* Clean up debug
* Working integration test
* Implement backoff
* Add manifest
* Clean up comments
* Clean up comments
* Clean up comments
* Clean up
* Clean up comments
* Clean up test
* Clean up comment
* Clean up
* Ensure the queue is drained before schedule
* Fix metric label
* Ensure the poller context is canceled
* Set default values for worker backoff locally
* Fix metric label
* Drop worker interval in favor of backoff
* Fix prune handling
* Improve work selection
* Hack in a backend cache
* Leverage existing blocklist measurement
* Skip the work cache when listing tenants
* Test CompactWithConfig
* Move proto for new work to new file
* Tenants with zero blocks have zero priority
* Many things
* Add note
* Drop empty file
* Plumb work config for pruning
* Drop needless proto change
* Update backendscheduler tests
* Update integration tests
* Fix config and generate doc manifest
* Add config validation
* Add more config validation
* Implement backoff wrapper for update calls
* Move tenant index-writitng to the backend-worker and shard on the ring
* Include RingConfig
* Generate manifest
* Fix manifest instance_id
* Drop workerID context injection from the GRPC request
* Push job collection into work package
* Push logic for skipping blocks which are already working into work package
* Log the workerID
* Return gRPC error on invalid job status
* Metric worker retry calls
* Improve test coverage for work
* Improve test coverage for job
* Fix test logic
* Reduce surface area, improve test coverage, clean up error handling
* Include new errors
* Fix lint
* Add tests for backendworker module
* Drop comments
* Fix backoff for Next() call on the worker to ensure backoff is honored with error return
* Bring back outstanding block metric on scheduler
* Make note about losing the last work time when the tenant is pushed back to the queue
* Measure all tenants, not just ones which get scheduled
* Continue scheduling if a single tenant was not enough work to load
* thoughts
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Add a timeout to cleanup jobs which have been running too long
* Drop "last work" from tenant selector implementation for simplicity
* Revert blocklist changes
* Ensure locking on work.OnBlock
* Clean up
* Generate manifest
* Clean up comment
* Clean up
* Clean up and handle a couple TODO
* Metric jobs created
* Fix job count per tenant
* Fix signature change in grpc DialOption
* Generate manifest
* Clean up
* Spellcheck
* Add missing Prune call
* Auto start jobs which are returned, metric job retries
* Drop Type from NextRequest
* Move marshal/unmarshal to Work and drop a mutex
* Simplify mutex handling
* Test for job reload
* Clean up after rebase
* Generate manifest
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: Joe Elliott <number101010@gmail.com>
* Set CPU architecture API version for amd64 and arm64
This enables the compiler to use instructions from more recent API versions
* Strip debug symbols from production binary
Reduces Tempo binary size by ~20%
* Remove GO111MODULE=on environment variable
GO111MODULE defaults to 'on' since Go 1.16
* CHANGELOG.md
Because we are using the $(MAKE) variable in targets which are called inside
the tools containers, the value of the variable corresponds to the path of make in the host,
which may be different than the one in the container, causing make to fail locally.
* remove tempo-serverless from cmd
* cleanup Makefile and .gitignore
* remove serverless code from pkg/api/
* remove serverless code from the querier
* clean up serverless from tempo-mixin and playbook
* Remove serverless from tempo docs
* Update changelog
* go mod vendor
* Remove tempo_feature_enabled metric
* inline internalSearchBlock
* docs alias
* Block-builder PoC
* Add unit test for block-builder (#4289)
* Add unit test for block-builder
* fmt
* Update tests
* cmon
* Deterministically build blocks for partition sections (#4327)
* Pull main (#4342)
* chore: remove gofakeit dependency (#4274)
* Further reduce Labes() calls in the metrics registry (#4283)
* Respect passed headers in read path requests (#4287)
* Ingester: Validate completed blocks (#4256)
* Add validate method to block
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Add Validate usage in the ingester
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog
Signed-off-by: Joe Elliott <number101010@gmail.com>
* add test and fix replay
Signed-off-by: Joe Elliott <number101010@gmail.com>
* increment metric
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Add `invalid_utf8` to reasons spans could be rejected (#4293)
* Add `invalid_utf8` to reasons spans could be rejected
* Update changelog
* Update docs
* Ensure test covers invalid UTF-8 and not slack time
* add signals for duplicate rf1 data (#4296)
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Bump anchore/sbom-action from 0.17.5 to 0.17.7 (#4307)
Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.17.5 to 0.17.7.
- [Release notes](https://github.com/anchore/sbom-action/releases)
- [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
- [Commits](https://github.com/anchore/sbom-action/compare/v0.17.5...v0.17.7)
---
updated-dependencies:
- dependency-name: anchore/sbom-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* docs: Update readme with explore traces info (#4263)
* docs: Update readme with explore traces info
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* chore: remove spanlogger (#4312)
* chore: remove spanlogger
* Query-Frontend: Add middleware to drop headers (#4298)
* header strip ware
Signed-off-by: Joe Elliott <number101010@gmail.com>
* comment
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog
Signed-off-by: Joe Elliott <number101010@gmail.com>
* remove header strip wear from metrics summary
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Increase length of time compactions have to fail (#4315)
* increase length of time compactions have to fail
Signed-off-by: Joe Elliott <number101010@gmail.com>
* gen
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* docs: mark serverless as deprecated (#4017)
* docs: mark serverless as deprecated
* Changelog + readme
* docs: Remove duplicated examples (#4295)
This removes duplicates examples from the Configure TraceQL
metrics page.
Signed-off-by: Alex Bikfalvi <alex.bikfalvi@grafana.com>
* tempo-cli: support dropping multiple traces in a single operation (#4266)
* tempo-cli: support dropping multiple traces in a single operation
* update final log message
---------
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
* [DOC] Add clarification for metrics summary and traceQL metrics (#4316)
* Add clarification for metrics summary and traceQL metrics
* Apply suggestions from code review
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* Update docs/sources/tempo/api_docs/metrics-summary.md
---------
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* TraceQL metrics time range fixes (#4325)
* Disconnect job time range filtering from step, so that results in split backend/recent range is accurate
* changelog
* Fix to assert metrics query range before alignment because alignment may increase it, which is not the responsibility of the caller to account for (#4331)
* Add doc about configuring TLS with Helm (#4328)
* Add doc about configuring TLS with Helm
* Add memberlist and readinessProbe to example
* Include server config for listening on TLS
* Add note about scraping
* Update docs/sources/tempo/configuration/network/tls.md
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
* Update docs/sources/tempo/configuration/network/tls.md
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* Update docs/sources/tempo/configuration/network/tls.md
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* Add memcached config for TLS
---------
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* [DOC] Add TLS info to Helm chart doc (#4334)
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Alex Bikfalvi <alex.bikfalvi@grafana.com>
Co-authored-by: Javier Molina Reyes <javiermolinar@live.com>
Co-authored-by: Zach Leslie <zach.leslie@grafana.com>
Co-authored-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ryan Perry <Rperry2174@gmail.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Alex Bikfalvi <alex@bikfalvi.com>
Co-authored-by: Andrey Karpov <ndk@users.noreply.github.com>
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
Co-authored-by: Martin Disibio <martin.disibio@grafana.com>
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
* WIP: Rhythm ingest path (#4314)
* Validate distributor config. Finish encoder/decoder tests
* Repair tests
* Make SingleBinary work out of the box by defaulting to partition 0
* Fix first time startup where blockbuilder fails before ingester can create topic
* Fix initial startup cycle time and delay
* Add more failure modes to the block-builder (#4345)
* Add more tests to the block-builder
* stuff
* Add comments
* [Rhythm] Metrics generator read from kafka first pass (#4359)
* Metrics generator read from kafka first pass
* review feedback
* Multiple fixes in block-builder (#4364)
* [rhythm] git merge origin/main (#4376)
* chore: remove gofakeit dependency (#4274)
* Further reduce Labes() calls in the metrics registry (#4283)
* Respect passed headers in read path requests (#4287)
* Ingester: Validate completed blocks (#4256)
* Add validate method to block
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Add Validate usage in the ingester
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog
Signed-off-by: Joe Elliott <number101010@gmail.com>
* add test and fix replay
Signed-off-by: Joe Elliott <number101010@gmail.com>
* increment metric
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Add `invalid_utf8` to reasons spans could be rejected (#4293)
* Add `invalid_utf8` to reasons spans could be rejected
* Update changelog
* Update docs
* Ensure test covers invalid UTF-8 and not slack time
* add signals for duplicate rf1 data (#4296)
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Bump anchore/sbom-action from 0.17.5 to 0.17.7 (#4307)
Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.17.5 to 0.17.7.
- [Release notes](https://github.com/anchore/sbom-action/releases)
- [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
- [Commits](https://github.com/anchore/sbom-action/compare/v0.17.5...v0.17.7)
---
updated-dependencies:
- dependency-name: anchore/sbom-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* docs: Update readme with explore traces info (#4263)
* docs: Update readme with explore traces info
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* chore: remove spanlogger (#4312)
* chore: remove spanlogger
* Query-Frontend: Add middleware to drop headers (#4298)
* header strip ware
Signed-off-by: Joe Elliott <number101010@gmail.com>
* comment
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog
Signed-off-by: Joe Elliott <number101010@gmail.com>
* remove header strip wear from metrics summary
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Increase length of time compactions have to fail (#4315)
* increase length of time compactions have to fail
Signed-off-by: Joe Elliott <number101010@gmail.com>
* gen
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* docs: mark serverless as deprecated (#4017)
* docs: mark serverless as deprecated
* Changelog + readme
* docs: Remove duplicated examples (#4295)
This removes duplicates examples from the Configure TraceQL
metrics page.
Signed-off-by: Alex Bikfalvi <alex.bikfalvi@grafana.com>
* tempo-cli: support dropping multiple traces in a single operation (#4266)
* tempo-cli: support dropping multiple traces in a single operation
* update final log message
---------
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
* [DOC] Add clarification for metrics summary and traceQL metrics (#4316)
* Add clarification for metrics summary and traceQL metrics
* Apply suggestions from code review
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* Update docs/sources/tempo/api_docs/metrics-summary.md
---------
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
* TraceQL metrics time range fixes (#4325)
* Disconnect job time range filtering from step, so that results in split backend/recent range is accurate
* changelog
* Fix to assert metrics query range before alignment because alignment may increase it, which is not the responsibility of the caller to account for (#4331)
* Add doc about configuring TLS with Helm (#4328)
* Add doc about configuring TLS with Helm
* Add memberlist and readinessProbe to example
* Include server config for listening on TLS
* Add note about scraping
* Update docs/sources/tempo/configuration/network/tls.md
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
* Update docs/sources/tempo/configuration/network/tls.md
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* Update docs/sources/tempo/configuration/network/tls.md
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* Add memcached config for TLS
---------
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* [DOC] Add TLS info to Helm chart doc (#4334)
* fix deprecation warning by switching to DoBatchWithOptions (#4343)
Signed-off-by: Daniel Strobusch <1847260+dastrobu@users.noreply.github.com>
* bump dskit to v0.0.0-20241115082728-f2a7eb3aa0e9 to leverage benefits for context causes for DoBatch calls. (#4341)
See https://github.com/grafana/dskit/issues/576
Signed-off-by: Daniel Strobusch <1847260+dastrobu@users.noreply.github.com>
* Bump github.com/minio/minio-go/v7 from 7.0.70 to 7.0.80 (#4282)
* Bump github.com/minio/minio-go/v7 from 7.0.70 to 7.0.80
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.70 to 7.0.80.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.70...v7.0.80)
---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update serverless vendor
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zach Leslie <zach.leslie@grafana.com>
* update default config values to better align with production workloads (#4340)
* update default config values to better align with production workloads
* Update CHANGELOG.md and config docs
* Ingester memory improvements by adjusting prealloc (#4344)
* remove trace ids
Signed-off-by: Joe Elliott <number101010@gmail.com>
* linear buckets
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog
Signed-off-by: Joe Elliott <number101010@gmail.com>
* tuney tune
Signed-off-by: Joe Elliott <number101010@gmail.com>
* metric misses and increase pool size
Signed-off-by: Joe Elliott <number101010@gmail.com>
* lint
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.13.0 to 1.16.0 (#4302)
* Bump github.com/Azure/azure-sdk-for-go/sdk/azcore from 1.13.0 to 1.16.0
Bumps [github.com/Azure/azure-sdk-for-go/sdk/azcore](https://github.com/Azure/azure-sdk-for-go) from 1.13.0 to 1.16.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-go/releases)
- [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md)
- [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/azcore/v1.13.0...sdk/azcore/v1.16.0)
---
updated-dependencies:
- dependency-name: github.com/Azure/azure-sdk-for-go/sdk/azcore
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* Update serverless vendor
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Zach Leslie <zach.leslie@grafana.com>
* Use Prometheus fast regexp (#4329)
* basic integration
Signed-off-by: Joe Elliott <number101010@gmail.com>
* patch tests for new meaning
Signed-off-by: Joe Elliott <number101010@gmail.com>
* patch up more tests
Signed-off-by: Joe Elliott <number101010@gmail.com>
* add basic tests
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog + docs
Signed-off-by: Joe Elliott <number101010@gmail.com>
* remove benches
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Cleaned up + tests
Signed-off-by: Joe Elliott <number101010@gmail.com>
* comment
Signed-off-by: Joe Elliott <number101010@gmail.com>
* lint
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Update docs/sources/tempo/traceql/_index.md
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* comment
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
* Fix broken link in service-graphs docs (#4351)
* Fix minor typo in TraceQL docs (#4356)
* Bump default memcached version (#4363)
* Exemplar fixes (#4366)
* Fix exemplars based on duration to convert to seconds, fix various other issues
* changelog
* fix: initialize histogram buckets to 0 to avoid them being downsampled (#4368)
* initialized histogram buckets to 0 to avoid them being downsampled
* Ingester/Generator Live trace cleanup (#4365)
* moved trace sizes somewhere shareable
Signed-off-by: Joe Elliott <number101010@gmail.com>
* use tracesizes in ingester
Signed-off-by: Joe Elliott <number101010@gmail.com>
* make tests work
Signed-off-by: Joe Elliott <number101010@gmail.com>
* trace bytes in generator
Signed-off-by: Joe Elliott <number101010@gmail.com>
* remove traceCount
Signed-off-by: Joe Elliott <number101010@gmail.com>
* live trace shenanigans
Signed-off-by: Joe Elliott <number101010@gmail.com>
* changelog
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Update modules/generator/processor/localblocks/livetraces.go
Co-authored-by: Mario <mariorvinas@gmail.com>
* Update modules/ingester/instance.go
Co-authored-by: Mario <mariorvinas@gmail.com>
* Test cleanup. Add sz test, restore commented out and fix e2e
Signed-off-by: Joe Elliott <number101010@gmail.com>
* remove todo comment
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: Mario <mariorvinas@gmail.com>
* Bump anchore/sbom-action from 0.17.7 to 0.17.8 (#4371)
Bumps [anchore/sbom-action](https://github.com/anchore/sbom-action) from 0.17.7 to 0.17.8.
- [Release notes](https://github.com/anchore/sbom-action/releases)
- [Changelog](https://github.com/anchore/sbom-action/blob/main/RELEASE.md)
- [Commits](https://github.com/anchore/sbom-action/compare/v0.17.7...v0.17.8)
---
updated-dependencies:
- dependency-name: anchore/sbom-action
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Update for IDs change
* Only run blockbuilder if ingest enabled
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Alex Bikfalvi <alex.bikfalvi@grafana.com>
Signed-off-by: Daniel Strobusch <1847260+dastrobu@users.noreply.github.com>
Co-authored-by: Javier Molina Reyes <javiermolinar@live.com>
Co-authored-by: Zach Leslie <zach.leslie@grafana.com>
Co-authored-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ryan Perry <Rperry2174@gmail.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Alex Bikfalvi <alex@bikfalvi.com>
Co-authored-by: Andrey Karpov <ndk@users.noreply.github.com>
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
Co-authored-by: Martin Disibio <martin.disibio@grafana.com>
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
Co-authored-by: Daniel Strobusch <1847260+dastrobu@users.noreply.github.com>
Co-authored-by: Carles Garcia <carles.garciacabot@grafana.com>
* [rhythm] Changes to simplify operations (#4389)
* Use mapping for assigning partitions
* Use mapping for assigning partitions in the generator too
* Add support for SASL auth to kafka clients
* Add metrics to ingest (#4395)
* [rhythm] Extract block-builder into its own module (#4396)
* Extract block-builder into its own module
* Update /operations and examples
* No ephemeral storage
* No rolling strategy either
* fmt and compile
* Address review comment
* [rhythm] Correctly pass start/end time when appending a trace (#4410)
* Correctly pass start/end times
* Different code, same result
* [rhythm] Multiple fixes to block-builder consumption (#4413)
* Multiple fixes to cycle consumption
* fmt
* happy now?
* ups
* Rhythm: Separate non-flushing local blocks processor to store new queue data for reads (#4411)
* wip: separate non-flushing local blocks processor to store new queue data for reads
* Make real config for non-flushing local blocks processor, optional, validate wal config and use defaults if needed
* Fix defaulting of second WAL config
* [rhythm] Make ID generator more robust (#4416)
* Make ID generator more robust
* Simplify
* Update to e50f5d96b
* Fix registering of kafka read client metrics (#4502)
* [rhythm] Make ID generator more robust (#4416) (#4507)
* Make ID generator more robust
* Simplify
* Removed references to Loki and Mimir (#4509)
Signed-off-by: Joe Elliott <number101010@gmail.com>
* [Rhythm] Block builder test updates (#4510)
* Make blockbuilder tests closer to real kafka and less implementation specific by always enabling support for consumer groups, call commit control func in order
* Verify last committed offset in each test
* hide test function
* lint
* lint
* [Rhythm] Block-builder consumption loop (#4480)
* Alternate block-builder consume
* Set timeout on PollFetches, reduce initial poll delay, update 1 test to work using real consumergroup functionality
* restore metrics
* Re-add original partition lag metric, polled in separate goroutine. Fix consume loop to only consume full-duration cycles for more determinism
* merge conflict
* Review feedback
* Review feedback
* Comment
* code cleanup, lint
* logs
* code cleanup
* lint
* Review feedback
* Remove missed lookback_on_no_commit config in e2e tests and regen manifest
* Review feedback
* Fix rewind to latest commit to init correctly, it didn't work in some clusters (#4532)
* [rhythm] merge main at 71e8531 (#4531)
* Fixes
* More fixes
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Alex Bikfalvi <alex.bikfalvi@grafana.com>
Signed-off-by: Daniel Strobusch <1847260+dastrobu@users.noreply.github.com>
Co-authored-by: Javier Molina Reyes <javiermolinar@live.com>
Co-authored-by: Zach Leslie <zach.leslie@grafana.com>
Co-authored-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ryan Perry <Rperry2174@gmail.com>
Co-authored-by: Kim Nylander <104772500+knylander-grafana@users.noreply.github.com>
Co-authored-by: Suraj Nath <9503187+electron0zero@users.noreply.github.com>
Co-authored-by: Alex Bikfalvi <alex@bikfalvi.com>
Co-authored-by: Andrey Karpov <ndk@users.noreply.github.com>
Co-authored-by: Jennifer Villa <jvilla2013@gmail.com>
Co-authored-by: Martin Disibio <martin.disibio@grafana.com>
Co-authored-by: Markus Toivonen <markus.toivonen@hoxhunt.com>
Co-authored-by: Daniel Strobusch <1847260+dastrobu@users.noreply.github.com>
Co-authored-by: Carles Garcia <carles.garciacabot@grafana.com>
* Add script to generate manifest with default config
The script:
- generates the manifest with default config YAML which is included in the docs as a reference.
- runs in CI to check if the manifest needs to be updated as a result of new changes in the default config.
* Generate manifest during build step in CI
* Add Makefile command
* Add build flags
* fix(tools): fix github action
* fix(tools): use correct ref fr v4
* fix(tools): use correct ref
* fix(tools): fecth tags
* fix(tools): fetch tags
* fix(tools): use tag directly during build
* Revert "fix(tools): use tag directly during build"
This reverts commit 5cbd0552db1daeccc89bb931c3545dcd2d04ff37.
* fix(tools): inject tag as version directly during build
* Update proto for v1 backend
* Functioning blocklist with new package
* Functioning blocklist with existing package
* Generate proto to existing package
* Add wrapper around google UUID for proto marshaling
* Update blocklist for wrapper type
* Test WriteTenantIndex
* Implement json for uuid
* Almost functional backend tests
* Passing backend tests
* Improve UUID coverage and utility
* TempoDB updates for UUID and struct type changes
* Update method style for uuid.New()
* Update method style for uuid.MustParse()
* Add uuid.From helper
* Update modules for UUID and proto struct types
* Update poller integration test for UUID
* Update tempo-cli for UUID and proto struct type changes
* Lint and clean up
* Fix serverless
* Fix cli
* Fix cli
* Add fixture tests for roundtrip assurance between versions
* Clean up notes from proto definition
* Update backend/test
* Be more specific about unmarshal UUID
* Include event for compacted block
* Use better json tag for compacted meta
* Implement UnmarshalJson on CompcatedBlockMeta for embedded struct
* Test for CompactedMeta marshal/unmarshal
* Drop benchmark index
* Clean commented code
* Extend mocks to capture multiple writes
* Update backend/test fixtures
* Write both proto and json to the backend for all three files in all backends
* Fix lint
* Back UUID with google.UUID to avoid struct
* Relocate pkg/uuid to tempodb/backend
* Re-plumb UUID through tempodb
* Re-plumb UUID through cmd
* Re-plumb UUID through modules
* Re-plumb UUID through integration/
* Drop package spec from tempo.proto
* Improve encoding marshal performance
* Drop block proto and tenantindex json
* Drop additional proto include
* Implement zstd tenant index compression
* Add note about use of json in the dedicatedcolumns
* Fix ingester local block
* Spell
* Revert mock update
* Drop json fallback from tenantindex read
* Replace single-tenant test fixtures
* Update fixture replacement test
* Improve error handling and fix test
* Extend test for tenantindex round-trip
* Small fixes
* Revert "Drop json fallback from tenantindex read"
This reverts commit 834ff5b85d8a8a4d7eb560fbb67ba8f3a5a3959a.
* Add back mocks to capture multiple backend writes
* Add back proto & json writitng/reading for tenantindex
* Change TotalObjects type from int32 -> int64
The original change was int -> int32, which on 64bit machines is a
downgrade that I overlooked. Raising here to int64 to maintain the
size.
* Store write buffer at object path in mocks to improve readability in tests
* Avoid allocation in interface validation
* Update note to include intern
* Pluralize metas and compacted_metas fields
* Fallback to json only on ErrDoesNotExist
* Lint
* Drop debug
* Revert "Pluralize metas and compacted_metas fields"
This reverts commit 46a8c36e37ac33e1ffc104287d898930e02cdd97.
* Import original tenantindex and test contents for backwards compatibility
* Update changelog
* Pin the goreleaser version for Go version support
Without this change, goreleaser@latest requires a newer version of Go
than is currently supported by the project. Here we pin the version.
* Pin the version of the tools image
* Move `jsonnetfmt` and `goimports` install into tools/
This will ensure that the version we are using to compile locally is the
same version that we are using in CI. The `goimports` was already
included in the tools, and so the CI was redundant. Also not, this
moves us from the C jsonnet to the Go jsonnet, which looks to introduce
slight formatting changes.
* Compile jsonnet
* Move tk, jb, and jsonnet also into tools/
* Include tools dependency on additional make targets
* Include tools Dockerfile
* Include tools image build target
* Update tools/go.sum
* Tidy up image build
* Prepare multi-arch docker image with install script
The script is used because not all imports in the tools.go are modules
which can be imported. The version of the import is determined by
matching the nearest module path and installing the import at the
version specified in the go.mod. This will allow updates to the go.mod
to direct the versions of the tools installed in the docker image.
* Stop ignoring the tools to allow docker build
* Include entrypoint
* Use TOOLS_CMD for docker execution on a few key make targets
* Use my image
* Fix variable
* Mark workdir as git safe
* Update drone to publish a tempo-ci-tools image
* Fix --platform for image source and use golang:alpine
* tempodb: update poller tests and mocks to support new work
* include Blocklist interface for testing
* tempodb: include quick poller
* Include QuickBlocks on tempodb
* Include notes about what is being implemented
* Add initial ListBlocks() stubs and implement s3
* Fix poller commits
* Lint fix
* Fix method for unimplemented test
* Add test coverage for previous compactions
* Fix method for unimplemented test
* Lint fix
* Drop some debug printing
* Drop unused variablea and update debug message
* Relocate allocations
* Invent fewer objects
* Lint fix
* Implement gcs ListBlocks()
* Tidy up s3.go
* Import fix
* Adjust list_test.go for eyeball parse
* Include notes after discoverying why
* Wrap a couple errors for clarity
* Modify error test for additional details
* Modify gcs backend for single character delimiter
* Add note about call change
* Add note about call change and implement listv2 for s3
* Adjust uuidInBlocklist to use existing allocations
* Add config and defaults for tenant polling concurrency
* Implement concurrent tenant polling
* Implement Find() on backend readers
* Add Find to MockRawReader
* Convert QuickBlocks over to use Find()
* Implement Find() on local/cache backend readers
* Drop ListBlocks in favor of Find
* Replace Blocks() with QuickBlocks() and update usage everywhere
* Fix local.Find for path handling
* Fix backend raw tests for Find mocks
* Reorder to reduce PR diff
* Relocate super-handy block boudary method
* Extend FindFunc to indicate Done
* Parallelize the Find() calls within Blocks()
* Fix race
* Fix boundedwaitgroup init in test
* Update backend implementations
* Make note of implemenation details
* Capture errors from tenant polling
* Add feature detection on backend readers
* Use backend.HasFeature() to determine if appropriate function
* Fix error handling in poller
* Pick lint
* Extend log message to include tenant_concurrency
* Add doc for HasFeature
* Local backend supports no features
* Pick lint
* Ensure we wait for results
* Avoid early return
* Improve error handling to include all errors
* Remove HasFeature() from RawReader interface after discussion
* Drop start argument from Find signature
* Include range calculation for sharded list calls
* Enable concurrent list calls for s3.Find()
* Include useful OwnsEverythingSharder
* Remove concurrency handling from tempodb.Blocks()
* Add docs for new config options
* Fix GCS e2e object path
* Clean up comments
* Include a test to ensure retained blocks are forgotten
* Begin reimplementing ListBlock() in RawReaders after discussion
* Reimplement ListBlocks() on the backends
* Fix ListBlocks() for gcs
* Drop Find and supporting code
* Drop unused method
* Back out proto change, since `main` also has this issue
* Update docs/sources/tempo/configuration/_index.md
Co-authored-by: Mario <mariorvinas@gmail.com>
* Reimplement ListBlocks() for Azure
* Lint
* Rename config option for concurrent block list
* Relocate block boundary code to own package
* Relocate Blocklist interface to poller package
* Rely on error count for early exit condition
* Avoid polling additional tenants if we have already reached our error tollerance
* Improve tests
* Include initial benchmarks
* Update docs/sources/tempo/configuration/_index.md
Co-authored-by: Koenraad Verheyden <koenraad.verheyden@posteo.net>
* Update docs/sources/tempo/configuration/_index.md
Co-authored-by: Koenraad Verheyden <koenraad.verheyden@posteo.net>
* Use *List directly, drop Blocklist interface
* Use builtin clear() to avoid heap escape
* Adhere to new error usage
* Speed up benchmark setup
* Trade memory for time in large data sets
* Lock for race
* Update CLI handling for Blocks() call to include compacted results
* Add doc to mention the concurrency per-instance
* Tidy up backends a bit
* Implement concurrent ListBlocks() on gcs
* Clean up
* Include ListBlocksConcurrency config option for GCS
* Test for compacted migration
* Start tracing ListBlocks()
* Start tracing the poller
* Fix span attribute
* Sort the block list for test
* Use slice collection instead of channel collection for deadlock
* Revert tenant poll concurrency
* Pass context to the EnablePolling method for future use
* Handle context on pollingLoop
* Use attribtues not events for better traceql
* Drop last tenant concurrency reference
* Fix Do() revert to old code
* Update for context
* Update tracing attributes for poller
* Back out proto update
* Remove unused error
* Remove inacurate docs after refactor
* Clean up
* Include additional span attributes
* Drop unused
* Drop unused
* Address some feedback in the backend
* Skip compacted blocks on trace summary
* Include TODO about reducing the 404 on meta.json for compacted blocks
* Include integration test for poller
* Upgrade fake-gcs-server
* Use JSONRead in gcs client for fake-gcs-server when Endpoint is set
* Avoid early allocation of error for concurent operations
* Include EndOffset in gcs backend
* Include integration configs
* Fix for global min/max
* Include poller integration test in ci
* Preallocate for review comments
* Lock for all results and break early if error
* Include additional assertion for poller_test
* Adjust gcs and s3 locking
* Adjust ListBlocks interface to take a tenant string instead of a keypath
* Move OwnsEverythingSharder
* Remove some comments
* Ensure exclusive max unless global max
* Restructure for clarity
* Tidy up backends a bit
* Fix test call count after avoiding BlockMeta for known compacted blocks
* Update tests to include CompactedMeta call count
* Include error message when block UUID is below shard minimum
* Centralize backend.GlobalMaxBlockID
---------
Co-authored-by: Mario <mariorvinas@gmail.com>
Co-authored-by: Koenraad Verheyden <koenraad.verheyden@posteo.net>
* Use errorlint to check for plain error comparisons and type assertions
* Explicitly load golangci-lint config file
* Fix linter warnings about plain error comparison
* Fix linter warnings about plain error type assertions
* Implement tooManySpansError.Error() with pointer receiver
* Implement traceTooLargeError.Error() with pointer receiver
* Implement unsupportedError.Error() with pointer receiver
* Implement ParseError.Error() with pointer receiver
* Improve comment for errorlint in .golangci.yml
* Implement validationError.Error() with pointer receiver
* Check for ParseError using errors.As()
* Remove custom multi error implementation
* More readable condition for errors.Is(err, context.Canceled)
* Implement validationError.Unwrap()
* Include gofumpt and goimports in tools
* Replace gofmt with gofumpt in Makefile
* Run `make fmt`
* Adhere to goconst lint rule
* Include gofumpt note in the CONTRIBUTING.md
* Update CHANGELOG
* Attempt unit testing jsonnet microservices
Signed-off-by: Zach Leslie <zach.leslie@grafana.com>
* Config update
* Update structure for microservices test
* Wire up CI to run the jsonnet tests
* Separate compile step, use output in first test
---------
Signed-off-by: Zach Leslie <zach.leslie@grafana.com>
* Break up tests into pkg, tempodb and others in CI Job
* fix ci.yaml file
* fix Makefile
* Run tests with -v
* split tempodb/wal out from tempodb
* lower go test timeout
* keep unit-tests-tempodb name short
* Revert https://github.com/grafana/tempo/pull/2312
* Add bash in build-tempo-binaries Drone step
`tools/image-tag` script need bash and it was failing because
`golang:1.20-alpine` image used in build-tempo-binaries step
is missing bash
due to missing bash `./tools/image-tag` was failing with
`env: can't execute 'bash': No such file or directory` and
producing tempo binaries with missing version tag
Tested this locally by building tempo in `golang:1.20-alpine`
with bash, and it works.
* make drone
