* Add operators IN, NOT IN, MATCH ANY, and MATCH NONE
* Implement Static.append()
* Print string arrays with '`' instead of '"'
Example ["aa", "bb"] -> [`aa`, `bb`]
* Rewrite binary operations to array equivalent
The rewriter handles the following cases:
- { .a = "a" || .a = "b" } => { .a IN ["a", "b"] }
- { .a != "a" && .a != "b" } => { .a NOT IN ["a", "b"] }
- { .a =~ "a" || .a =~ "b" } => { .a MATCH ANY ["a", "b"] }
- { .a !~ "a" && .a !~ "b" } => { .a MATCH NONE ["a", "b"] }
* Add test cases with arrays to TestBinOp
* traceql.Parse() applies AST optimizations by default
* Rename predicate_test.go to predicates_test.go
* Add array predicates to parquetquery
* BREAKING CHANGE: more precise regex validation
BinaryOperation.validate() considered everythin a valid regex where
FieldExpression.EncodeToString(false) happened to return a valid regex.
With this change the RHS of a regex operation is required to be a string
or string array
* vp4: handle predicates with arrays correctly
* vp5: handle predicates with arrays correctly
* vp3: handle predicates with arrays correctly
* Add test cases for optimized queries
* Update query test examples with optimizable queries
* vp3: fix handling of array types
* CHANGELOG.md
* Pass array operators down to fetch layer
* Read to first virtual row after seeking on a low definition level
This helps avoid additional spans/op in queries with only
resource attributes
---------
Co-authored-by: Martin Disibio <mdisibio@gmail.com>
* Allow up to 20 dedicated strings in vparquet5, and update tempo-cli analysis tools with new 3% target heurstic for strings, similar to how we are doing for ints
* Fix num int attr default back to 5
* changelog
* Cleanup leftover int fields. Validate dedicated column overrides and treat too many columns as a warning
* Lint to return err last
* add suggest dedicated columns cli command
* add option to provide an outfile path
* lint
* remove json
* make output-file a string instead of path arg
* make not global options
* revert overrides args
* back to what it was
* Change vp5 array handling, to not separate array vs non-array types for dedicated columns. This restores the precondition that all values of the given primitive type (string,int) exist in the column, and read path operations (search, fetch, tag lookup) will work as expected
* changelog
* fix special operands search tags handling bug
* add more tests to cover more scenarios
* remove span well known handling in vparquet5 and added event dedicated columns tests
* oops typo
* add dedicated events to expected values for test
* Implement GCS delete retry when marking a block compacted
* Include retry in additional delete for ClearBlock
* Include max retries configuration for GCS compactor/retention operations
* Update config manifest
* Play golf using atomic.Int32 instead of an int32
* vParquet5: remove orphan DedicatedAttributes nodes from schema
When a scope does not have any dedicated attributes, we have to
remove the empty / orphan DedicatedAttributes group node
* CHANGELOG.md
* New analyse block data, more dedicated columns
* Add analyse block arg to output quoted/spaced list for other cli command
* Committing cli cardinality analysis
* Different dedicated columns per scope, turn dictionary on and off dynamically based on column cardinality
* Experiments
* First version of working RewriteSchema
* Simple version
* Updates
* Update to new options pattern in test branch
* Fixes after merging main
* Cleanup/simplify analyse block code for blobs
* Make blob detection in analyse block optional
* Update to same default
* Update to final parquet-go StructTag option
* Delete unused dedicated columns, update all references of schema/reader/write to account for it
* cli cleanup and support blobs in jsonnet
* Upgrade parquet-go to get remaining fix
* cleanup
* Fix event dedicated attr writes. Readd tests and cleanup
* cleanup
* lint
* Support vParquet5 for analyse block cli command
* changelog + bump encoding version
* Fix analyse blocks combination of data
* More test coverage of event dedicated columns
* Fix wal block, search, and tag lookup on event dedicated columns. More test coverage
* Revert printing scope in simple summary
* Fix delete impl for s3/gcs/azure backends to account for prefix
* Update changelog
* Omit prefix from backend.readerWriter.Find result set
The backend delete function (which prepends the prefix) is being used
in two places: first to delete indexes directly using the file path,
and secondly, the results from the Find function are iterated over to
delete the files for a tenant. If the result set from the Find
function returns the full path (including the prefix), the delete
function would prepend the prefix again and hence would not target the
correct path. Hence all Find implementations are updated to return
unprefixed key in the result set (we trim the prefix from key before passing
over to the FindFunc arg). This is consistent with the List implementation.
* Add prefix for keypath for versioned deletes
* Remove bucket prefix in backend.Find instead of entire prefix
* Make intrinsic=nil queries fail validation
* No autocomplete on invalid queries, check in missing test case for previous commit
* Fix to only validate tag lookup if parsing succeeded
* Disallow resource.service.name=nil since we dont' support that in storage. Push down operator OpExists so that != nil for performance and correctness of metrics queries, cleanup vp4
* cleanup and port all changes to vp3/5
* Add missing predicate support for OpExists
* Update autocomplete for OpExists and port changes to vp3/5
* lint
* vp2
* Cleanup/simplify
* changelog
* Review feedback, cleanup to use shared func
* Add field SpanCount to ScopeSpans
* Make new vp5 preview version
* Convert vp5 test-data
* Move rowNumberIterator to own file
* Implement virtualRowNumberIterator
* Improve tests for virtualRowNumberIterator
* Set correct span count for vp5 test data
* Use virtualRowNumberIterator instead of StatusCode colum iter
* CHANGELOG.md
* [bugfix] support attr = nil for tags
* add other vparquet versions
* added tests
* changelog
* Accumulate distinct results at a higher level
Signed-off-by: Joe Elliott <number101010@gmail.com>
* Add logic to wal blocks
Signed-off-by: Joe Elliott <number101010@gmail.com>
---------
Signed-off-by: Joe Elliott <number101010@gmail.com>
Co-authored-by: Joe Elliott <number101010@gmail.com>
* Migrate S3 backend to AWS SDK v2
* cleanup: remove all aws-sdk-go refs (incl. s3_test.go), replace with minio.NoSuchKey, clean backup, ran go mod tidy & vendor,add to changelog.md, make test-e2e-api pass
* fix: format
---------
Co-authored-by: Javi <javiermolinar@live.com>
* niliter
* mewo
* it kinda works..?
* no events and links
* rebase
* move conditions around - added more test assertions
* clean up
* testing remove this
* add nil value handling
* lint
* fix ast condition
* update test examples
* fix tests
* select all should print nil for dedicated columns with nil values
* remove prints
* fixed comments
* lint
* remove not needed params
* add support for vparquet3 and vparquet5
* vparquet2 shakefist
* fix test and lint
* add support for links and events
* fix nil iterator returning last row and add test
* rebase, lint, changelog, docs
* fix test
* fix test, add column chunk null check
* logic to return correct rn for events and links and return attr key in response
* fix test from rebase
* fix column name for wellknown and custom attr
* one more from vparquet3
* Tune benchmarks
* Add options field to dedicated attribute columns
* Dedicated columns have slice values
* New encoding vParquet5-preview4
* Reuse slices in DedicatedAttributes
* Run make generate-manifest
* Fix lint/fmt warning
* CHANGELOG.md
* Process options in DedicatedColumns.Hash()
* vp3 and vp4 ignore dedicated columns with options
* Refactor filterDedicatedColumns() functions
* Avoid collistions when hashing dedicated attribute slices
* Fix hashing for rebatchTrace() and add tests
* resourceSpanHash() takes int dedicated columns into account
* DedicatedColumns.Validate() takes int dedicated columns into account
* Remove Validate() for individual dedicated columns
{} with (debug_return_in=100s, debug_std_dev=10s, debug_data_factor=0.1)
debug_return_in - adds latency for queriers (disabled by default)
debug_std_dev - adds dispersion to wait time (0 by default)
debug_data_factor - probability of non-empty response from a querier (0 by default)
* wip: first batch of eliminating promLabels from traceql metrics
* Finish adding helper methods and fix tests
* Fix frontend, more tests
* Fix test
* cleanup
* changelog
* More cleanup
* Use proto reserved to prevent reuse of promlabels instead of just commenting it out
* Add e2e test for fixed type handling
* lint
* Cleanup prom_labels from instant series
* Relabel as breaking change and update with notes on breaking changes and rollout consideration
* feat: add support for identifying root spans with child_of links in traceToParquet
* feat: add pr in changelog
* Verify if link childs are internal to the trace
