Compare commits

..
439 Commits
Author SHA1 Message Date
zeekayandClaude Opus 4.8 f90c751b10 refactor(datastore): back the datastore migration driver with native hanzo-ds/go, drop clickhouse-go
The datastore:// migration driver is now a self-contained golang-migrate
database.Driver that opens through github.com/hanzo-ds/go (the native
ClickHouse-wire driver registered as the database/sql name "datastore").
The ClickHouse-specific migration logic (multi-statement exec,
schema_migrations DDL, cluster/engine config, Version/SetVersion/Drop/Lock)
is ported directly into database/datastore, so it no longer embeds or
imports the upstream database/clickhouse driver or
github.com/ClickHouse/clickhouse-go.

- Delete database/clickhouse (driver + test + examples) and
  internal/cli/build_clickhouse.go.
- internal/cli/build_datastore.go drops the clickhouse-go blank import.
- go.mod: drop github.com/ClickHouse/clickhouse-go, add
  github.com/hanzo-ds/go v1.0.1; tidy. Zero clickhouse-go in go.mod,
  go.sum, and the module/dependency graph.
- Port the driver test + example migrations under database/datastore
  (integration test requires a live datastore/clickhouse-wire backend).
- Update README, release workflow build tags, bug report template.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 18:40:56 -07:00
z c7a0a74aef docs(brand): add hero banner 2026-06-28 20:17:58 -07:00
z 50117bac35 chore(brand): dynamic hero banner 2026-06-28 20:17:56 -07:00
d9c67dc417 ci: run on self-hosted ARC pool (hanzo-build-linux-amd64/deploy), not GitHub-hosted (#1)
Co-authored-by: zeekay <z@hanzo.ai>
2026-06-19 20:36:45 -07:00
Antje WorringandClaude Opus 4.8 a5075e2de1 docs: tidy LLM.md indexes; CLAUDE.md -> LLM.md symlink convention
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 10:05:12 -07:00
Hanzo AI 9477979a55 deps: bump Go to 1.26.4 across go.mod, Dockerfiles, GH Actions
Workspace-wide sync. luxfi/node already shipped on 1.26.4 in v1.30.6
(commit 121aca1fa9); this is the cross-repo catch-up.
2026-06-07 12:10:59 -07:00
Hanzo AI f4abb4bba7 Merge remote-tracking branch 'upstream/master' 2026-05-19 21:54:56 -07:00
Hanzo AI 3461b6bfd7 go.mod: bump go directive to 1.26.3 (security advisory) 2026-05-12 21:25:26 -07:00
Hanzo AI 602c5edb72 chore: symlink AGENTS.md and CLAUDE.md to LLM.md
Canonical project context lives in LLM.md. Symlinks ensure
agentic coding tools (Claude Code, Cursor, etc.) find context
automatically regardless of which filename they look for.
2026-04-01 14:12:04 -07:00
Hanzo Dev a7da767fa3 chore: bump Go 1.26.1 2026-03-22 14:10:48 -07:00
Hanzo Dev b55e3ea06d docs: add LLM.md project guide 2026-03-11 10:31:22 -07:00
Dale HuiandGitHub 2bd822b3aa Merge pull request #1361 from golang-migrate/dependabot/go_modules/go_modules-6b971a9d7e
Bump go.opentelemetry.io/otel/sdk from 1.36.0 to 1.40.0 in the go_modules group across 1 directory
2026-03-11 00:50:10 -07:00
Zach Kelling 460d57267a fix: tar binary as 'migrate' not 'migrate.linux-amd64' to match upstream 2026-03-03 10:16:42 -08:00
Zach Kelling 1e4b96537d Add release workflow for CI-built binaries
Builds migrate CLI for linux/darwin (amd64/arm64) with datastore
driver support and publishes GitHub Release on tag push.
2026-03-03 09:24:50 -08:00
Zach Kelling 837dda3cc9 Add datastore database driver for Hanzo Datastore
Registers "datastore://" as a URL scheme that delegates to the
ClickHouse driver. This allows golang-migrate to natively connect
to Hanzo Datastore instances without URL rewriting in scripts.
2026-03-03 09:15:21 -08:00
dependabot[bot]andGitHub 1620c56290 Bump go.opentelemetry.io/otel/sdk
Bumps the go_modules group with 1 update in the / directory: [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go).


Updates `go.opentelemetry.io/otel/sdk` from 1.36.0 to 1.40.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.36.0...v1.40.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.40.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-01 20:47:32 +00:00
Dale HuiandGitHub 257fa847d6 Merge pull request #1320 from achanda/upgrade-pgxv5
Upgrade pgx v5 to latest version
2025-12-14 15:16:48 -08:00
001b70ac7d ci: update golangci-lint to v2 (#1341)
* ci: update golangci-lint to v2

update method:
```
golangci-lint migrate
golangci-lint run --fix
```

and fix the last ones manually

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-14 15:14:46 -08:00
leonklingeleandGitHub 89e308c340 chore: remove dependency on "hashicorp/go-multierror" (#1322)
* feat: remove dependency on "hashicorp/go-multierror"

It seems the project has been unmainted for quite some time already,
see for example https://github.com/hashicorp/go-multierror/issues/97
and https://github.com/hashicorp/go-multierror/issues/98.
This also removes an uneccessary dependency which slims down the go.mod.

Go as of version 1.20 supports appending errors by specifying
multiple "%w" formatters in the "fmt.Errorf"[^0] function.

[^0]: https://pkg.go.dev/fmt#Errorf

* feat: use errors.Join to combine multiple errors

As an update to the previous commit 59f084ea3be2e7ec9d912c1a311402b1e4c40df6,
use "errors.Join"[^0] instead of multiple "%w" formatters for "fmt.Errorf".

[^0]: https://pkg.go.dev/errors#Join

* chore: bring back unused util.go file as removing it is a breaking change
2025-11-29 11:37:42 -08:00
Abhishek Chanda fed8fe9f0a Upgrade pgx v5 to latest version 2025-11-27 20:13:25 -06:00
Dale HuiandGitHub 472ef2e75c Merge pull request #1336 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.45.0
Bump golang.org/x/crypto from 0.43.0 to 0.45.0
2025-11-27 00:56:21 -08:00
dependabot[bot]andGitHub 8d7625987a Bump golang.org/x/crypto from 0.43.0 to 0.45.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.43.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-27 08:05:11 +00:00
9f9df7c897 chore: Update cloud.google.com/go/spanner version (#1330)
* chore: Update cloud.google.com/go/spanner version

* Run `go mod tidy`

---------

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2025-11-27 00:01:15 -08:00
Dale HuiandGitHub a371c8ea2b Merge pull request #1304 from iamonah/fix/clarify-databaseName-meaning
Rename databaseName to databaseDriverName
2025-11-26 22:52:08 -08:00
Dale HuiandGitHub 43cc3b3ea5 Merge pull request #1325 from HaraldNordgren/linter_issues
fix linter issues
2025-11-26 22:38:12 -08:00
Dale HuiandGitHub f939a89925 Merge pull request #1335 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.45.0
Bump golang.org/x/crypto from 0.36.0 to 0.45.0
2025-11-26 22:30:24 -08:00
dependabot[bot]andGitHub 6dd86e0ded Bump golang.org/x/crypto from 0.36.0 to 0.45.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.36.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.36.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-20 01:45:37 +00:00
Dale HuiandGitHub 70e6d6d574 Merge pull request #1333 from matsoob/updateGoInBaseImage
Update docker image to use go 1.25, addressing vulnerabilities
2025-11-18 23:04:52 -08:00
Dale HuiandGitHub a51d0da644 Merge pull request #1334 from golang-migrate/dependabot/go_modules/github.com/dvsekhvalnov/jose2go-1.7.0
Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0
2025-11-18 23:01:27 -08:00
dependabot[bot]andGitHub 3f583f0b61 Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0
Bumps [github.com/dvsekhvalnov/jose2go](https://github.com/dvsekhvalnov/jose2go) from 1.6.0 to 1.7.0.
- [Commits](https://github.com/dvsekhvalnov/jose2go/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/dvsekhvalnov/jose2go
  dependency-version: 1.7.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-14 21:46:48 +00:00
matsoob ce3cb727b6 Update docker image 2025-11-13 23:35:48 +00:00
Harald Nordgren 1ee8337800 revert GenerateAdvisoryLockID 2025-11-11 09:42:53 +01:00
Harald Nordgren 952d08e30b fix linter issues 2025-11-09 16:54:45 +01:00
Dale HuiandGitHub 39ab04d659 Update goreleaser Go version from 1.24.x to 1.25.x 2025-11-08 21:48:01 -08:00
Dale HuiandGitHub 8b9c5f7712 Merge pull request #1310 from dhui/update_go
Add support for Go 1.25 and drop support for 1.23
2025-08-23 20:27:53 -07:00
Dale Hui b4ec9bccb1 Add support for Go 1.25 and drop support for 1.23 2025-08-23 13:42:01 -07:00
ed4bdd4614 Ensure bufferWriter is always closed in Migration.Buffer and propagate close errors (#1308)
* Ensure bufferWriter is always closed and error is returned in Buffer()

* Ensure bufferWriter is always closed and error is returned in Buffer()

* addressed review comments. used named error and joined error to return final error

* Fixed defer block

---------

Co-authored-by: Chandra kant <chandra.kant@cohesity.com>
2025-08-23 13:12:17 -07:00
Roman SharkovandGitHub 8945e853c4 Merge pull request #1303 from romshark/master
refactor: Remove go.uber.org/atomic in favor of std sync/atomic
2025-08-20 16:35:25 -07:00
Dale HuiandGitHub 7108d806dd Merge pull request #1309 from dhui/dktest_v0.4.6
Update dktest to v0.4.6 for docker vuln fix
2025-08-19 16:24:09 -07:00
Dale HuiandGitHub 682016f04c Merge pull request #1277 from sandhilt/doc/change-apt-key-to-gpg
Update apt-key to gpg
2025-08-19 14:22:16 -07:00
Dale Hui f3e6b5a737 Replace usage of deprecated docker types
This is a backwards incompatible change,
but there don't seem to be any public imports:
https://pkg.go.dev/github.com/golang-migrate/migrate/v4@v4.18.3/testing?tab=importedby
2025-08-19 14:03:55 -07:00
Dale Hui 0a17402aa2 Update dktest to v0.4.6 for docker vuln fix 2025-08-19 13:27:40 -07:00
Dale HuiandGitHub 5eee0c8030 Merge pull request #1299 from golang-migrate/dependabot/go_modules/golang.org/x/oauth2-0.27.0
Bump golang.org/x/oauth2 from 0.18.0 to 0.27.0
2025-08-19 13:15:26 -07:00
0xOnah 30690d7df3 Rename databaseName to databaseDriverName 2025-08-03 21:30:31 +01:00
dependabot[bot]andGitHub 642a24d61b Bump golang.org/x/oauth2 from 0.18.0 to 0.27.0
---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-version: 0.27.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-18 18:07:02 +00:00
sandhilt 5300864bba fix(docs): update Linux installation instructions for GPG key handling 2025-05-13 15:05:40 -03:00
Dale HuiandGitHub 278833935c Merge pull request #1186 from urbim/bugfix/sqlserverLockFix
Fixed sqlserver not actually getting a lock if lock is already set
2025-04-25 13:14:27 -07:00
Urban Malc 6b9f24bf7f Fixed sqlserver not actually getting a lock if lock is already set 2025-04-24 12:14:21 +02:00
Dale HuiandGitHub 9023d66a0b Merge pull request #1244 from alexandear-org/chore-redundant-build-tags
Remove redundant build tags
2025-04-17 15:21:39 -07:00
Dale HuiandGitHub 10494908af Merge pull request #1179 from lunfel/master
Update migrate -help output for the readme file
2025-04-17 15:17:59 -07:00
Dale Hui 033835ac1b Update to dktest v0.4.5 2025-04-17 14:58:46 -07:00
Dale HuiandGitHub 329152ed62 Merge pull request #1196 from Rambatino/patch-1
fix: typo limited not limitted
2025-04-17 13:59:24 -07:00
Oleksandr Redko 34c2b4ad9c Remove redundant build tags 2025-04-17 23:57:28 +03:00
Dale HuiandGitHub a3b7633246 Merge pull request #1259 from golang-migrate/dependabot/go_modules/golang.org/x/net-0.38.0
Bump golang.org/x/net from 0.33.0 to 0.38.0
2025-04-17 00:12:46 -07:00
dependabot[bot]andGitHub 5b97c921ed Bump golang.org/x/net from 0.33.0 to 0.38.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.33.0 to 0.38.0.
- [Commits](https://github.com/golang/net/compare/v0.33.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-04-17 02:40:34 +00:00
Dale HuiandGitHub 3c3ce91f29 Merge pull request #1258 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.35.0
Bump golang.org/x/crypto from 0.31.0 to 0.35.0
2025-04-16 19:39:14 -07:00
dependabot[bot]andGitHub f37ef795db Bump golang.org/x/crypto from 0.31.0 to 0.35.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.31.0 to 0.35.0.
- [Commits](https://github.com/golang/crypto/compare/v0.31.0...v0.35.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.35.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-04-16 22:36:30 +00:00
Dale HuiandGitHub 1af841d787 Merge pull request #1260 from dhui/update_go
Update supported Go versions and README
2025-04-16 15:35:09 -07:00
Dale Hui 7269490309 Update golangci-lint version used in GitHub Actions 2025-04-16 15:12:33 -07:00
Dale Hui fccd197e40 Mention CLI install instructions in main README
Fixes:
* https://github.com/golang-migrate/migrate/issues/973
* https://github.com/golang-migrate/migrate/issues/1020
2025-04-16 14:43:17 -07:00
Dale Hui e6d84f6f80 Drop support for Go 1.22 and add support for Go 1.24
Drop `toolchain` module directive to favor default/auto behavior
See also: https://github.com/golang-migrate/migrate/pull/1183
2025-04-16 14:38:52 -07:00
Dale HuiandGitHub 146bfa05b3 Merge pull request #1241 from hazzik/refactor/pkg/errors
refactor: replace github.com/pkg/errors with stdlib
2025-04-16 14:04:34 -07:00
Dale HuiandGitHub 604248c97e Merge pull request #1250 from golang-migrate/dependabot/go_modules/github.com/golang-jwt/jwt/v4-4.5.2
Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2
2025-03-23 16:54:14 -07:00
dependabot[bot]andGitHub a4d0a1b8d9 Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.1...v4.5.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-21 22:40:35 +00:00
Alex Zaytsev 60d73be14e refactor: replace github.com/pkg/errors with stdlib 2025-03-10 13:54:11 +10:00
Jochen SchalandaandGitHub 36d17ba2ec tests: fix various tests (#1209)
* Remove old versions of MongoDB

Removed:

| Release     | Release Date  | End of Life Date |
|-------------|---------------|------------------|
| MongoDB 3.4 | November 2016 | January 2020     |
| MongoDB 3.6 | November 2017 | April 2021       |
| MongoDB 4.0 | June 2018     | April 2022       |
| MongoDB 4.2 | August 2019   | April 2023       |

Added:

| Release     | Release Date  | End of Life Date |
|-------------|---------------|------------------|
| MongoDB 5.0 | July 2021     | October 2024     |
| MongoDB 6.0 | July 2022     | July 2025        |
| MongoDB 7.0 | August 2023   | August 2026      |
| MongoDB 8.0 | October 2024  | TBD              |

https://www.mongodb.com/legal/support-policy/lifecycles

* test(postgres): run tests with PostgreSQL 17

Add PostgreSQL 17, remove PostgreSQL 12 (EOL since November 21, 2024).

https://www.postgresql.org/support/versioning/

* test(cockroachdb): remove old versions of CockroachDB in tests

Removed:

|       Release       |                Released                |              Maintenance Support             |              Assistance Support              |         Latest        |
|:-------------------:|:--------------------------------------:|:--------------------------------------------:|:--------------------------------------------:|:---------------------:|
| 2.1                 | 6 years ago (30 Oct 2018)              | Ended 5 years ago (30 Oct 2019)              | Ended 4 years and 7 months ago (30 Apr 2020) | 2.1.11 (22 Jan 2020)  |
| 2.0                 | 6 years and 8 months ago (04 Apr 2018) | Ended 5 years and 8 months ago (04 Apr 2019) | Ended 5 years ago (04 Oct 2019)              | 2.0.7 (28 Nov 2018)   |
| 1.1                 | 7 years ago (12 Oct 2017)              | Ended 6 years ago (12 Oct 2018)              | Ended 5 years and 8 months ago (12 Apr 2019) | 1.1.9 (01 Oct 2018)   |
| 1.0                 | 7 years ago (10 May 2017)              | Ended 6 years ago (10 May 2018)              | Ended 6 years ago (10 Nov 2018)              | 1.0.7 (11 Feb 2018)   |

Added:

|       Release       |                Released                |              Maintenance Support             |              Assistance Support              |         Latest        |
|:-------------------:|:--------------------------------------:|:--------------------------------------------:|:--------------------------------------------:|:---------------------:|
| 24.3                | 3 weeks ago (18 Nov 2024)              | Ends in 11 months (18 Nov 2025)              | Ends in 1 year and 5 months (18 May 2026)    | 24.3.0 (20 Nov 2024)  |
| 24.2                | 3 months and 4 weeks ago (12 Aug 2024) | Ends in 2 months (12 Feb 2025)               | Ends in 2 months (12 Feb 2025)               | 24.2.5 (11 Nov 2024)  |
| 24.1 (Upcoming LTS) | 6 months and 3 weeks ago (20 May 2024) | Ends in 5 months (20 May 2025)               | Ends in 11 months (20 Nov 2025)              | 24.1.7 (11 Nov 2024)  |
| 23.2 (Upcoming LTS) | 10 months ago (05 Feb 2024)            | Ends in 1 month and 4 weeks (05 Feb 2025)    | Ends in 7 months and 4 weeks (05 Aug 2025)   | 23.2.16 (13 Nov 2024) |

* test(firebird): remove old versions of Firebird in tests

Removed: Firebird 2.5, see https://firebirdsql.org/en/discontinued-versions/

Added:
- Firebird 4.0 (https://firebirdsql.org/en/firebird-4-0-5/)
- Firebird 5.0 (https://firebirdsql.org/en/firebird-5-0-1/)

* test(clickhouse): run ClickHouse tests with official LTS image (24.8)
2025-02-19 16:19:12 -08:00
Dale HuiandGitHub 24a6476044 Merge pull request #1225 from stensonb/patch-1
Update FAQ.md - typo
2025-02-19 16:06:49 -08:00
Bryan StensonandGitHub a8680333c3 Update FAQ.md - typo 2025-02-01 23:26:02 -08:00
Dale HuiandGitHub d477553dcd Merge pull request #1195 from golang-migrate/dependabot/go_modules/github.com/golang-jwt/jwt/v4-4.5.1
Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.5.1
2025-01-26 17:53:05 -08:00
dependabot[bot]andGitHub e145cded4a Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.5.1
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.4.2 to 4.5.1.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.4.2...v4.5.1)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-27 01:06:51 +00:00
Dale Hui 7651c8a37e linter fixes 2025-01-26 17:04:55 -08:00
Dale Hui e5a152b9f6 Drop support for Azure SQL Edge
Will be retired on 9/30/2025 and CI tests using the docker image have been flakey
https://azure.microsoft.com/en-us/updates?id=azure-sql-edge-retirement
2025-01-26 16:43:31 -08:00
Dale HuiandGitHub 12c619ee47 Fix CI (#1222)
* Run go mod tidy

* Fix Github linting action

Changes:
* checkout before setting up go
* pin golangci-lint version

* Update supported db versions

* Use start-single-node instead of start for cockroachdb
2025-01-26 15:46:11 -08:00
Dale Hui bc06922b4d Update dktest from v0.4.3 to v0.4.4 2025-01-25 21:05:33 -08:00
Mark RamotowskiandGitHub 8b091919d4 fix: typo limited not limitted 2024-11-28 09:28:14 +00:00
Mathieu Tanguay c5137c417c Update migrate -help output for the readme file 2024-10-02 10:41:33 -04:00
Dale HuiandGitHub c378583d78 Merge pull request #1162 from rselbach/rselbach/no-sensitive-info
Don't output sensitive information on error
2024-09-23 10:45:56 -07:00
Roberto Selbach e22d0128cc Don't output sensitive information on error 2024-09-12 12:12:34 -04:00
Dale Hui 555501f7cd Update dktest from v0.4.2 to v0.4.3 2024-09-08 14:02:51 -07:00
Dale HuiandGitHub d5eb594426 Merge pull request #1101 from adityassharma-ss/patch-1
CORRECTION DONE near m.Step(2) if you want to explicitly set the number of migrations to run
2024-09-08 11:42:13 -07:00
Dale HuiandGitHub 67c71f9a9d Merge pull request #1152 from golang-migrate/dependabot/go_modules/google.golang.org/grpc-1.64.1
Bump google.golang.org/grpc from 1.64.0 to 1.64.1
2024-09-08 11:34:22 -07:00
Dale HuiandGitHub 1e02b42a7a Merge pull request #1119 from muzammilar/add-noble-numbat
Support .deb package for Noble Numbat (24.04)
2024-09-07 23:07:01 -07:00
dependabot[bot]andGitHub 5b6f623dec Bump google.golang.org/grpc from 1.64.0 to 1.64.1
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.64.1.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.64.0...v1.64.1)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-08 06:04:50 +00:00
Dale HuiandGitHub ddd7fa61da Merge pull request #1127 from joschi/tests-mysql-9
test(mysql): run tests with MySQL 8.0, 8.4, and 9.0
2024-09-07 22:59:58 -07:00
Dale HuiandGitHub 7e8f6be051 Merge pull request #1126 from joschi/tests-postgres-16
test(postgres): run tests with PostgreSQL 16
2024-09-07 22:59:25 -07:00
Dale HuiandGitHub 65a3bd56cc Merge pull request #1153 from joschi/fix-docker-imports
fix(tests): fix Docker imports
2024-09-07 22:58:32 -07:00
Dale HuiandGitHub e40e64c4ac Merge pull request #1133 from joschi/go-1.23
build: use Go 1.23 to build the project
2024-09-07 22:51:54 -07:00
Jochen Schalanda 6ceb5a9467 fix(tests): fix Docker imports 2024-09-07 22:50:55 +02:00
Jochen Schalanda 0444ec8531 test(postgres): run tests with PostgreSQL 16 2024-09-07 21:47:48 +02:00
Jochen Schalanda f4d986b9d9 test(mysql): run tests with MySQL 8.0, 8.4, and 9.0 2024-09-07 21:47:28 +02:00
Jochen Schalanda ee93fd4aff build: use Go 1.23 to build the project
https://tip.golang.org/doc/go1.23
2024-09-07 21:45:48 +02:00
Dale Hui d28e549a68 Update dktest from v0.4.1 to v0.4.2 2024-09-07 00:01:37 -07:00
Jochen SchalandaandGitHub 897be38496 chore: remove deprecated rand.Seed() in testing.docker (#1149)
* chore: remove deprecated `rand.Seed()` in testing.docker

* Use math/rand/v2

* Remove Go 1.21

Go 1.21 doesn't support `math/rand/v2`.
2024-09-06 23:46:24 -07:00
Dale Hui 1b1c6f6d14 Use newer math/rand/v2 2024-09-06 23:32:35 -07:00
Dale HuiandGitHub f44379d6f1 Merge pull request #1151 from dhui/fix-redshift-tests
Fix redshift tests
2024-09-06 00:41:25 -07:00
Dale Hui e1a7e31b1d Use forked version of postgres 8 to avoid Docker image format version issues.
See: https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1

Forked images created by pulling the original image and pushing to the fork. For more details, see: https://github.com/golang-migrate/migrate/pull/1143#issuecomment-2328069381
2024-09-05 23:28:26 -07:00
Dale Hui 899a244399 Fix redshift tests
Newer versions of postgres require a password
2024-09-05 23:26:56 -07:00
Muzammil Abdul Rehman 41e389a078 Support .deb package for Noble Numbat (24.04) 2024-07-17 12:56:34 -07:00
Dale HuiandGitHub 2477f639fd Merge pull request #1098 from gjabell/master
Upgrade go-sqlite3 to v1.14.22
2024-06-10 12:46:13 -07:00
Dale HuiandGitHub 1585b870f8 Merge pull request #1103 from Shion1305/shion/issue-1102
🧑‍💻 improve error message for invalid source, database, resolves: #1102
2024-06-10 12:34:31 -07:00
Dale HuiandGitHub b16220c938 Merge pull request #1105 from Shion1305/shion/issue-1104
#1104 Fix golanglint-ci lint config & lint errors
2024-06-10 12:28:15 -07:00
Shion Ichikawa 34594af0ab 🚨 golangci-lint: fix for errcheck lint errors 2024-06-05 23:40:12 +09:00
Shion Ichikawa 1735cb3920 🚨 golangci-lint: disable interfacer linter 2024-06-05 23:39:14 +09:00
Shion Ichikawa cabb2db836 🧑‍💻 improve error message for invalid source, database, resolves: #1102 2024-06-05 22:44:03 +09:00
Aditya SharmaandGitHub da72428fa0 Update README.md
if you want to explicitly set the number of migrations to run, the README.md needs a correction, there should "Steps()" and not Step()
2024-06-04 19:25:09 +05:30
Galen Abell c81eaf7918 Upgrade go-sqlite3 to v1.14.22 2024-05-31 17:30:06 +02:00
Dale HuiandGitHub 04b36ebaac Merge pull request #1070 from certifaction/mysql-safe-update
Make MySQL SetVersion compatible with sql_safe_update
2024-04-29 21:59:22 -07:00
Dale HuiandGitHub 45e4f1ef2f Merge pull request #1074 from golang-migrate/dependabot/go_modules/golang.org/x/net-0.23.0
Bump golang.org/x/net from 0.21.0 to 0.23.0
2024-04-29 21:55:07 -07:00
dependabot[bot]andGitHub 859d92bd77 Bump golang.org/x/net from 0.21.0 to 0.23.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0.
- [Commits](https://github.com/golang/net/compare/v0.21.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-19 13:19:09 +00:00
Mateusz Nowotyński e428ecebea Make MySQL SetVersion compatible with sql_safe_update 2024-04-19 09:34:12 +02:00
Dale HuiandGitHub 0c456c49a6 Merge pull request #1068 from goodfirm/master
chore: fix some typos in comments
2024-04-16 23:16:25 -07:00
Dale Hui f100226758 Update dktest from v0.4.0 to v0.4.1 to fix docker vulnerability 2024-04-16 23:03:36 -07:00
Dale HuiandGitHub c52377528f Merge pull request #1055 from golang-migrate/dependabot/go_modules/github.com/jackc/pgx/v5-5.5.4
Bump github.com/jackc/pgx/v5 from 5.3.1 to 5.5.4
2024-04-16 22:57:30 -07:00
dependabot[bot]andGitHub a78d1abfa8 Bump github.com/jackc/pgx/v5 from 5.3.1 to 5.5.4
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) from 5.3.1 to 5.5.4.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v5.3.1...v5.5.4)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-16 20:43:15 +00:00
Dale HuiandGitHub 837776f0c7 Merge pull request #1054 from golang-migrate/dependabot/go_modules/google.golang.org/protobuf-1.33.0
Bump google.golang.org/protobuf from 1.31.0 to 1.33.0
2024-04-16 13:38:31 -07:00
Dale HuiandGitHub 128b6509cc Merge pull request #1050 from golang-migrate/dependabot/go_modules/github.com/jackc/pgx/v4-4.18.2
Bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2
2024-04-16 13:38:07 -07:00
dependabot[bot]andGitHub 2e0872f3e8 Bump google.golang.org/protobuf from 1.31.0 to 1.33.0
Bumps google.golang.org/protobuf from 1.31.0 to 1.33.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-16 16:19:02 +00:00
dependabot[bot]andGitHub d1df97b2d1 Bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2
Bumps [github.com/jackc/pgx/v4](https://github.com/jackc/pgx) from 4.18.1 to 4.18.2.
- [Changelog](https://github.com/jackc/pgx/blob/v4.18.2/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v4.18.1...v4.18.2)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-16 16:18:25 +00:00
Tim RühsenandGitHub 1a002d0e9e Set golangci-lint to 1.54.2 (latest is broken) (#1046)
* Set golangci-lint to 1.54.2 (latest is broken)

* Testing golangci-lint latest (1.57.1)
2024-04-15 23:54:05 -07:00
Dale HuiandGitHub 2c5df8742b Merge pull request #1072 from dhui/dktesting-cleanup
Delete db docker images after tests complete for a few large db docker images
2024-04-15 23:44:37 -07:00
Dale Hui f4950c144e Fallback to dktest.DefaultCleanupTimeout if the dktest.Options doesn't have one specified 2024-04-15 22:57:23 -07:00
Dale Hui b1d02e222f Cleanup sqlserver images after tests run 2024-04-15 22:42:34 -07:00
Dale Hui 49cac86ab2 Cleanup mongodb images after tests run 2024-04-15 00:35:46 -07:00
Dale Hui 06614d9cf5 Cleanup yugabytedb images after tests run 2024-04-15 00:25:40 -07:00
Dale Hui 1b707a76f4 Cleanup cassandra images after tests run 2024-04-15 00:18:13 -07:00
Dale Hui 2884a8e851 Cleanup postgres images after tests run 2024-04-15 00:17:36 -07:00
Dale Hui 4bc6777f19 Add dktesting.Cleanup() method 2024-04-15 00:09:23 -07:00
goodfirm 9d70a397e4 chore: fix some typos in comments
Signed-off-by: goodfirm <fanyishang@yeah.net>
2024-04-10 14:36:41 +08:00
Dale Hui e9133367d7 Drop support for Go 1.20 and add support for Go 1.22 2024-03-24 22:35:06 -07:00
Dale HuiandGitHub 87ba13c398 Merge pull request #1061 from occupyhabit/master
chore: fix some typos
2024-03-24 22:31:39 -07:00
Dale Hui ff8a961068 Update yugabyte test images 2024-03-24 22:30:38 -07:00
occupyhabit 7f85f9cf64 chore: fix some typos
Signed-off-by: occupyhabit <wangmengjiao@outlook.com>
2024-03-23 21:49:28 +08:00
Dale HuiandGitHub a5dc5d1efc Merge pull request #1025 from otoolep/patch-1
rqlite is spelled with all lowercase
2024-01-10 20:56:10 -08:00
Philip O'TooleandGitHub 94b8fa5946 rqlite is spelled with all lowercase
I'm the creator of rqlite, so wanted to fix the title. See https://rqlite.io/
2024-01-05 15:34:45 -05:00
Dale HuiandGitHub 0d41589774 Merge pull request #1022 from golang-migrate/dependabot/go_modules/github.com/dvsekhvalnov/jose2go-1.6.0
Bump github.com/dvsekhvalnov/jose2go from 1.5.0 to 1.6.0
2024-01-02 12:48:02 -08:00
dependabot[bot]andGitHub a860f0cbd4 Bump github.com/dvsekhvalnov/jose2go from 1.5.0 to 1.6.0
Bumps [github.com/dvsekhvalnov/jose2go](https://github.com/dvsekhvalnov/jose2go) from 1.5.0 to 1.6.0.
- [Commits](https://github.com/dvsekhvalnov/jose2go/compare/v1.5...v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/dvsekhvalnov/jose2go
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-31 22:22:36 +00:00
Dale HuiandGitHub 516f038f30 Merge pull request #840 from alex-kuck/fix/839/sqlserver_version_table
[sqlserver] Ensure version table in provided schema
2023-12-31 13:59:38 -08:00
Alex Kucksdorf 0350a00606 [sqlserver] Always access version table with explicit schema 2023-12-25 10:29:27 +01:00
Alex Kucksdorf 814769373f [sqlserver] Ensure version table in provided schema
Closes #839
2023-12-25 10:29:26 +01:00
Dale HuiandGitHub 0815e2d770 Merge pull request #992 from fortnox-andreas/master
Add support for locking table in pgx-driver
2023-12-20 14:13:36 -08:00
Dale Hui 5aa4670cbb Fix GoReleaser deprecations
https://goreleaser.com/deprecations/#-rm-dist
https://goreleaser.com/deprecations/#sourcerlcp
2023-12-20 13:12:07 -08:00
Dale Hui d63a5c2c28 Only test against YugabyteDB LTS releases 2023-12-20 12:49:33 -08:00
Dale Hui cd17c5a808 Drop support for Go 1.19 and add support for Go 1.21 2023-12-20 11:37:41 -08:00
Dale Hui f2c4b52679 Update aws-sdk-go from v1.44.301 to v1.49.6 2023-12-20 11:33:09 -08:00
Dale HuiandGitHub c7c50118a0 Merge pull request #956 from Kenai/master
Update aws-sdk-go to adress vulerabilitiy
2023-12-20 11:29:57 -08:00
Dale HuiandGitHub 9c551d4398 Merge pull request #875 from no-name16/clickhouse_create_database_migrations
clickhouse: add to README.md and examples database creation
2023-12-20 11:24:13 -08:00
Tommy Karlsson 091ad5db99 Quote locktable from config in queries
Defer rollback of transactions
2023-12-20 16:10:29 +01:00
Dale HuiandGitHub ab24e7674d Merge branch 'master' into clickhouse_create_database_migrations 2023-12-20 00:33:47 -08:00
Dale HuiandGitHub e8edcdce66 Merge branch 'master' into master 2023-12-20 00:30:20 -08:00
Dale Hui 691f687eb8 Reformat ScyllaDB/Cassandra docs 2023-12-20 00:28:14 -08:00
Dale HuiandGitHub eb64ffa1ad Merge pull request #947 from mkorolyov/scylladb_support
add tests for scylladb. add scylladb to docs
2023-12-20 00:19:19 -08:00
Dale HuiandGitHub a94396c3a8 Merge pull request #857 from luca-nardelli/master
clickhouse: Quote db name in ensureVersionTable
2023-12-20 00:15:38 -08:00
Dale HuiandGitHub 0ba6fc3166 Merge pull request #1002 from jsabbatini-upguard/upgrade-spanner
Updated version of spanner to support sequences and generate uuid
2023-12-19 23:10:13 -08:00
Dale HuiandGitHub 20636841a9 Merge branch 'master' into master 2023-12-19 23:01:18 -08:00
Dale HuiandGitHub 0695426fbf Merge pull request #929 from SuperSandro2000/patch-1
Move supported go version to standard place
2023-12-19 22:58:49 -08:00
Dale HuiandGitHub 1d1cd48953 Merge branch 'master' into scylladb_support 2023-12-19 22:56:40 -08:00
Dale HuiandGitHub 76efa726ed Merge branch 'master' into master 2023-12-19 22:53:04 -08:00
Dale HuiandGitHub 90c5015954 Merge pull request #959 from jorng/libpq-update
Update lib/pq to fix cert permissions issues
2023-12-19 22:50:16 -08:00
Dale Hui 5026488530 Clean up require directive grouping 2023-12-19 22:41:22 -08:00
Dale Hui fb224367d7 Merge remote-tracking branch 'origin/master' into upgrade-spanner 2023-12-19 22:39:19 -08:00
Dale HuiandGitHub 47a2661dd4 Merge pull request #1015 from golang-migrate/dependabot/go_modules/google.golang.org/grpc-1.56.3
Bump google.golang.org/grpc from 1.55.0 to 1.56.3
2023-12-19 22:03:03 -08:00
Dale HuiandGitHub b39ee92ed9 Merge pull request #999 from tobyscott25/patch-1
Add syntax highlighting to Postgres example
2023-12-19 21:38:57 -08:00
dependabot[bot]andGitHub c3ebd527d8 Bump google.golang.org/grpc from 1.55.0 to 1.56.3
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.55.0 to 1.56.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.55.0...v1.56.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-20 05:37:50 +00:00
Dale HuiandGitHub f375aebc5c Merge pull request #1007 from swensone/master
Add support for rqlite
2023-12-19 21:35:18 -08:00
Dale HuiandGitHub 834fa39423 Merge pull request #1014 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.17.0
Bump golang.org/x/crypto from 0.14.0 to 0.17.0
2023-12-19 21:15:31 -08:00
Erik Swenson bfedabbc35 Merge remote-tracking branch 'upstream/master' 2023-12-19 10:41:12 -07:00
Erik Swenson 7d03609443 add 8.11 and 8.12 versions and remove debug logging 2023-12-19 10:33:43 -07:00
dependabot[bot]andGitHub 5ded96d97d Bump golang.org/x/crypto from 0.14.0 to 0.17.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-19 08:40:00 +00:00
Dale Hui b567287637 Update from alpine 3.18 to 3.19 2023-12-19 00:02:27 -08:00
Dale Hui 4078ef8941 New release prep
* Update dktest from v0.3.16 to v0.4.0 to fix docker vulnerability
* Fix linter issues
* Update outdated GitHub Actions
2023-12-18 23:44:05 -08:00
Dale HuiandGitHub 7eac9190d4 Merge pull request #1006 from testwill/typo
fix: typo
2023-12-18 21:58:57 -08:00
Erik Swenson 669437c3b8 update rqlite 8 container version to 8.0.6 2023-12-14 09:13:13 -07:00
Erik Swenson cf03803fe0 Add rqlite 8.0.0 to tested database versions 2023-12-06 16:02:36 -07:00
Erik Swenson 5163ac7824 feature: add rqlite support 2023-11-27 17:05:24 -07:00
guoguangwu ee8a8e50a7 fix: typo
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-11-27 13:45:49 +08:00
Jacopo Sabbatini 72957b6966 Updated version of spanner to support sequences and generate uuid 2023-11-14 15:04:55 +11:00
Toby ScottandGitHub 12968a75af Add syntax highlighting to Postgres example 2023-11-04 12:24:32 +11:00
Tommy Karlsson bead4a952b Added documentation and test for lock strategy 2023-09-20 14:27:26 +02:00
Andreas Emilsson 129922aae3 Added support for pgx locking table
In order to support running migrations through PgBouncer which does not
support advisory locks.
2023-09-20 09:59:55 +02:00
Rob PrentissandRob Prentiss f2e0b335d7 Update lib/pq to fix cert permissions issues
When attempting to connect to a PostgreSQL database using certificate
credentials, authentication may fail due to permissions issues on the
certificate files. When using migrate in Kubernetes, this issue may be
unavoidable when using secrets.

The github.com/lib/pq library has resolved this issue as of v1.10.6, so
this commit updates that library to the latest release version (v1.10.9)
to resolve the issue in migrate.
2023-07-21 12:00:31 -07:00
Jan Śreniawski 876a13d423 Update aws-sdk-go to adress vulerabilitiy 2023-07-18 09:00:47 +02:00
Maxim Korolyov f8afa5a6f6 small changes to retry failed by timeout CI 2023-07-17 22:41:12 +02:00
Maxim KorolyovandGitHub 45d23ed92a Merge branch 'master' into scylladb_support 2023-07-11 09:16:48 +02:00
Dale HuiandGitHub 856ea12df9 Merge pull request #953 from dhui/update_spanner
Update spanner to fix security issue
2023-07-05 23:05:13 -07:00
Dale Hui 208ac532e8 Update spanner to fix security issue
See also: https://github.com/golang-migrate/migrate/pull/952
2023-07-05 22:18:03 -07:00
Dale HuiandGitHub bad30b549d Mention migradaptor
migradaptor can be used to migrate from other db migration tools to golang-migrate

https://github.com/golang-migrate/migrate/issues/940#issuecomment-1611712379
2023-07-02 22:22:38 -07:00
Maxim Korolyov 7a72550614 add tests for scylladb. add scylladb to docs 2023-06-25 20:30:21 +02:00
Dale HuiandGitHub 691bc773cc Merge pull request #936 from golang-migrate/dependabot/go_modules/github.com/snowflakedb/gosnowflake-1.6.19
Bump github.com/snowflakedb/gosnowflake from 1.6.3 to 1.6.19
2023-06-11 14:36:47 -07:00
dependabot[bot]andGitHub 383386b35f Bump github.com/snowflakedb/gosnowflake from 1.6.3 to 1.6.19
Bumps [github.com/snowflakedb/gosnowflake](https://github.com/snowflakedb/gosnowflake) from 1.6.3 to 1.6.19.
- [Release notes](https://github.com/snowflakedb/gosnowflake/releases)
- [Changelog](https://github.com/snowflakedb/gosnowflake/blob/master/release.go)
- [Commits](https://github.com/snowflakedb/gosnowflake/compare/v1.6.3...v1.6.19)

---
updated-dependencies:
- dependency-name: github.com/snowflakedb/gosnowflake
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-09 22:58:57 +00:00
Dale Hui 315a205b59 Update Dockerfiles
Use Alpine 3.18 and Go 1.20
2023-06-06 21:33:03 -07:00
Dale Hui 23d8d33f57 Support .deb packages for Ubuntu Jammy Jellyfish and Debian bookworm 2023-06-04 21:52:01 -07:00
SandroandGitHub 92dec355ef Move supported go version to standard place 2023-06-01 11:09:09 +02:00
Dale HuiandGitHub f9c067b3e0 Merge pull request #922 from DariusParvin/patch-1
Fix typo
2023-05-30 22:15:25 -07:00
Dale HuiandGitHub 0b3b628703 Use pgx5 connection string in README 2023-05-30 22:12:01 -07:00
Dale Hui c62dbbacc2 Opt-in to goreleaser's new property
https://goreleaser.com/deprecations/#archivesrlcp
2023-05-27 16:39:56 -07:00
Dale Hui 8d96f81de6 Update mssql to fix 386 and arm7 builds 2023-05-27 16:35:51 -07:00
Dale Hui ea8bebc7ef Downgrade docker 2023-05-20 14:21:01 -07:00
Dale Hui 1020682de5 Update dktest from v0.3.15 to v0.3.16 for upstream security fixes 2023-05-20 00:34:56 -07:00
Darius ParvinandGitHub d72e5c6096 Fix typo
change `stamements` to `statements`
2023-04-26 15:09:50 -07:00
Dale HuiandGitHub 84009cf2ab Merge pull request #914 from soyayaos/master
fix: postgresql search_path can be empty (#696)
2023-04-06 22:49:01 -07:00
Vladimir Galunshchikov ddaf669dc4 fix: postgresql search_path can be empty (#696) 2023-04-06 23:48:03 +02:00
Dale HuiandGitHub e968557bc6 Merge pull request #913 from dhui/update_dktest
Update dktest from v0.3.14 to v0.3.15 for upstream security fixes
2023-04-05 10:33:54 -07:00
Dale Hui 5be1166fa3 Update dktest from v0.3.14 to v0.3.15 for upstream security fixes 2023-04-04 21:58:42 -07:00
Eli TreuherzandGitHub b94ed1a710 Add driver for pgx v5 (#848)
* Add driver for pgx v5

Fixes #827

- Move existing driver into database/pgx/v4
- Make exported items into database/pgx into links to database/pgx/v4,
  marked as Deprecated
- Remove most tests in database/pgx, leaving a couple of e2e tests
  behind to help ensure backwards compatibility.
- Mark defunct ErrDatabaseDirty as Deprecated - no code returns it
- Bump pgx/v4 dependency
- Add test runs for newer Postgres versions

* Tweak pgx tags and registration strings

* Tweak registration strings
2023-04-04 21:57:01 -07:00
Dale HuiandGitHub c34675fa54 Merge pull request #905 from dhui/update-mongodb
Update mongodb driver for security fix
2023-03-25 23:46:32 -07:00
Dale Hui 2c61c69199 Update mongodb driver for security fix
Mongodb driver v1.7.5 no longer depends on packr
https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMGOBUFFALOPACKRV2-1920670
https://github.com/mongodb/mongo-go-driver/pull/815
2023-03-25 21:02:48 -07:00
Dale HuiandGitHub 586c129bae Merge pull request #903 from dhui/update-sqlite3
Update go-sqlite3 for security fix
2023-03-25 01:32:14 -07:00
Dale Hui 2111d2f93e Update go-sqlite3 for security fix
https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMMATTNGOSQLITE3-3121037
2023-03-25 00:05:17 -07:00
GeorgeWheatcroftandGitHub b63a0d4880 fix issue of downloading and using github sourced migration files of … (#900)
* fix issue of downloading and using github sourced migration files of 100MB >=1Mb (which previously failed to download due to their size and the api call performed) by preferring the go-github client library method 'DownloadContents' for such operations (which doesn't have the same limitation)

* empty commit & push to trigger ci
2023-03-24 23:18:40 -07:00
ViktoriaandGitHub 3b02b182ee Correct a spelling mistake 2023-03-19 07:44:50 +01:00
Dale HuiandGitHub 8ec04221b4 Merge pull request #836 from m-chrome/update-pgx-deps
Update pgx deps
2023-03-15 15:05:36 -07:00
Dale HuiandGitHub 8d1322e09a Merge branch 'master' into update-pgx-deps 2023-03-15 14:14:30 -07:00
Dale HuiandGitHub 1d693e8836 Merge pull request #809 from jcuanmhoney/master
Upgrading Spanner version
2023-03-15 11:09:59 -07:00
Dale Hui 4831e0ed50 go-genproto to google-cloud-go message type migration
Followed these steps: https://github.com/googleapis/google-cloud-go/blob/main/migration.md
2023-03-15 09:13:26 -07:00
Dale HuiandGitHub 30b6b9aeac Merge branch 'master' into master 2023-03-15 08:45:02 -07:00
Dale HuiandGitHub 952f3f046e Merge pull request #896 from dhui/go-1.20
Drop support for Go 1.18 and add support for Go 1.20
2023-03-15 00:33:07 -07:00
Dale Hui e497646d9c Drop support for Go 1.18 and add support for Go 1.20 2023-03-14 23:19:03 -07:00
Dale HuiandGitHub 179dba434c Merge pull request #806 from mtsmfm/patch-1
Fix placeholder in download URL
2023-03-14 23:11:20 -07:00
Dale Hui f87b567859 Update dktest from v0.3.13 to v0.3.14 2023-03-14 22:40:05 -07:00
Dale HuiandGitHub 7cacfc2080 Merge pull request #882 from golang-migrate/dependabot/go_modules/github.com/aws/aws-sdk-go-1.34.0
Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0
2023-03-08 22:12:55 -08:00
Dale HuiandGitHub 448a9b1819 Merge pull request #885 from golang-migrate/dependabot/go_modules/golang.org/x/net-0.7.0
Bump golang.org/x/net from 0.1.0 to 0.7.0
2023-03-08 22:12:33 -08:00
Dale HuiandGitHub e7af02e753 Merge pull request #893 from MasterOdin/patch-1
Fix GH workflow badge in README
2023-03-08 22:11:52 -08:00
Matthew PevelerandGitHub e5d8970cd0 Fix GH workflow badge in README
Signed-off-by: Matthew Peveler <matt.peveler@gmail.com>
2023-03-01 10:35:29 -05:00
dependabot[bot]andGitHub fd42ff1729 Bump golang.org/x/net from 0.1.0 to 0.7.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.1.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-18 03:11:56 +00:00
dependabot[bot]andGitHub 88d30a4217 Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.33.0 to 1.34.0.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/v1.34.0/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.33.0...v1.34.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-14 16:27:52 +00:00
Viktoria 64755d0b20 Update README.md 2023-01-24 14:37:31 +03:00
Viktoria 90a3ac4f87 Remove cluster adaptation for tables to pass tests 2023-01-24 13:22:17 +03:00
Viktoria 50112e7650 Add to clickhouse README.md database creation 2023-01-24 12:40:36 +03:00
Dale HuiandGitHub dc26c41ac9 Update security policy
GitHub security advisories have been enabled, so update the security policy accordingly
2023-01-10 23:51:36 -08:00
Dale HuiandGitHub 145baa187d Merge pull request #865 from golang-migrate/dependabot/go_modules/github.com/aws/aws-sdk-go-1.33.0
Bump github.com/aws/aws-sdk-go from 1.17.7 to 1.33.0
2023-01-10 23:33:56 -08:00
dependabot[bot]andGitHub eb59b750e8 Bump github.com/aws/aws-sdk-go from 1.17.7 to 1.33.0
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.17.7 to 1.33.0.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/v1.33.0/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.17.7...v1.33.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-30 19:08:10 +00:00
Dale HuiandGitHub c9ff76f294 Merge pull request #841 from tsbkw/fix/conform-to-go-test-naming-convention
fix: Conform to go test naming convention
2022-12-06 00:05:18 -08:00
Dale HuiandGitHub b63177a7da Merge pull request #856 from morigs/fix/ybdb-err-is-retryable-panic
fix(yugabyte): panic on err retry check
2022-12-03 01:08:36 -08:00
Luca Nardelli 839421e733 Leverage quoteIdentifier from pgx 2022-11-30 14:38:22 +01:00
Luca Nardelli 9fe7383564 Quote in drop as well 2022-11-30 10:26:31 +01:00
Luca Nardelli 90273fe0aa clickhouse: Quote db name in ensureVersionTable 2022-11-29 15:46:06 +01:00
tsbkwandGitHub a7f9122726 Merge branch 'master' into fix/conform-to-go-test-naming-convention 2022-11-28 08:18:35 +09:00
Igor Morozov e7e707727d fix(yugabyte): panic on err retry check 2022-11-23 19:05:34 +04:00
TOGASHI TomokiandGitHub 31791adbc0 fix(spanner): update spanner module to fix parsing issue (#843)
* update spanner module to fix spanner parsing issue

* bump the spanner module version to v1.40.0
2022-11-08 22:27:33 -08:00
tsbkw 41f3495935 fix: Conform to go test naming convention
As reported in #835 go vet reporting the violation of
naming convention of go test.
see: https://pkg.go.dev/testing#hdr-Examples

Although this project use golangci-lint as linter and
it has functionality to skip by nolint:govet, so basically
there is no problems.
However if there is no cons, it is better to conform with
naming convention and fix the code like below.
https://github.com/golang-migrate/migrate/blob/c367ed2f12b8549c2ace442c440a9e312348951d/dktesting/example_test.go#L17
2022-10-26 19:20:29 +09:00
Michail Tsyganov 8324b455c0 Update pgx deps 2022-10-22 17:51:24 +03:00
c367ed2f12 Resolve Docker CVEs (#815)
* Bump alpine images to version 3.16 and update golang image to one based on alpine 3.15

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Bump alpine image to version 3.16

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Bump go version to 1.18

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Revert "Bump go version to 1.18"

This reverts commit 7a7d2bfb0b4711012dbecc8e19b1a3c336ea7615.

* Revert "Bump alpine image to version 3.16"

This reverts commit b13c3b84dcb7abb4038d75389dff6dd509645f33.

* Upgrade docker image and migrate binary to golang 1.19, and use build stages in docker image

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Back out unnecessary changes

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>
Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2022-10-19 00:02:33 -07:00
Isaac FolmsbeeandGitHub 5ff4dc804a Fix CockroachDB tutorial's link being relative to 'GETTING_STARTED.md' (#819)
* Fix link root

Fix the link root so that it doesn't point relatively.

* Simplify to just the / root
2022-10-18 23:16:35 -07:00
Dale HuiandGitHub 4ca6c06627 Merge pull request #817 from dschaller/patch-1
docs: minor updates to getting started docs
2022-10-18 23:06:51 -07:00
Dale HuiandGitHub 865751e1b3 Merge pull request #816 from dschaller/dschaller-patch-1
docs: use v4 godocs link in readme
2022-10-18 23:04:13 -07:00
Dale HuiandGitHub 03e74b9289 Update SECURITY.md
mention impact and urgency
2022-10-18 22:35:33 -07:00
Dale HuiandGitHub a2cd4036be Create SECURITY.md 2022-10-18 22:33:42 -07:00
Dale Hui 58eb755472 Use v4 for Go Report Card 2022-10-14 23:57:39 -07:00
Dale Hui 30411f49a1 Add support for Go 1.19 and drop support for Go 1.17 2022-10-14 22:52:48 -07:00
Dale Hui 723c60fe0d Update dktest from v0.3.11 to v0.3.13 2022-10-14 22:50:44 -07:00
Dale HuiandGitHub 1d0b934641 Merge pull request #820 from Juneezee/go1.17
build: update `go` directive in `go.mod` to 1.17
2022-10-14 22:46:39 -07:00
Eng Zer Jun 0d539f4cae build: update go directive in go.mod to 1.17
This commit enables support for module graph pruning and lazy module
loading for projects that are at Go 1.17 or higher.

Reference: https://go.dev/ref/mod#go-mod-file-go
Reference: https://go.dev/ref/mod#graph-pruning
Reference: https://go.dev/ref/mod#lazy-loading
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-06 12:38:28 +08:00
Dale HuiandGitHub 37a9174b9d Merge pull request #810 from linzhengen/bump_ci_version
bump up CI version & fix CI failed
2022-10-05 14:00:37 -07:00
linzhengen 30deade574 bump ci version & go1.19 fmt ./... 2022-10-05 23:48:20 +09:00
Dale HuiandGitHub 2d1de76527 Merge pull request #814 from tspearconquest/fix_cves
Patch CVEs
2022-10-04 23:32:30 -07:00
DerekandGitHub d5be909ccd docs: minor updates to getting started docs 2022-10-01 14:16:21 -07:00
DerekandGitHub 2fdf921b1e docs: use v4 godocs link in readme 2022-10-01 14:14:30 -07:00
Thomas Spear 405ef9df07 Run gofmt -s on the files that are failing the lint job upstream
Signed-off-by: Thomas Spear <tspear@conquestcyber.com>
2022-09-28 14:19:40 -05:00
Thomas Spear 856f9c4192 Patch CVEs
Signed-off-by: Thomas Spear <tspear@conquestcyber.com>
2022-09-28 14:10:48 -05:00
Javier Cuan-Martinez 79c741fa73 Upgrading Spanner version 2022-09-15 08:59:13 -07:00
Fumiaki MATSUSHIMA ca03021154 Fix placeholder in download URL 2022-09-08 20:17:19 +09:00
Dale HuiandGitHub 5eb1604704 Merge pull request #726 from morigs/feat/yugabyte-support
feat: add yugabyte support
2022-08-25 23:45:01 -07:00
Igor Morozov b74509abe2 docs(yugabyte): update readme (add retry options) 2022-08-25 09:44:59 +03:00
Igor Morozov 9a407b6083 style(yugabyte): rename retry options 2022-08-25 09:38:36 +03:00
Igor Morozov 5582a2083b style(yugabyte): simplify error conditions 2022-08-25 09:36:23 +03:00
Igor Morozov bf926fa253 style: fix lint errors 2022-08-24 09:50:31 +03:00
Igor MorozovandGitHub 9f21d9ca68 Merge branch 'master' into feat/yugabyte-support 2022-08-24 09:44:44 +03:00
Igor Morozov 76d03050d2 feat(yugabytedb): configure retries 2022-08-24 09:44:27 +03:00
Dale HuiandGitHub 4bd83668eb Merge pull request #796 from agcooke/bugfix/fix-docs
The example Clickhouse DSN incorrectly described how to set the usern…
2022-08-23 21:48:42 -07:00
Adrian Cooke 8c64ac7ccd The example Clickhouse DSN incorrectly described how to set the username and password.
The example:

   `clickhouse://username:password@host:port/database=clicks?x-multi-statement=true`

did not work, but:

`clickhouse://host:port?username=user&password=password&database=clicks&x-multi-statement=true`

does.

I got the above DSN format from the tests.
2022-08-19 14:00:28 +02:00
Dale HuiandGitHub 03613f14ac Merge pull request #784 from anacrolix/patch-1
Fix broken tags link
2022-08-01 22:26:50 -07:00
Matt JoinerandGitHub 162fc2bb3d Sqlite update (#785)
* Update modernc.org/sqlite

Specifically to get support for sqlite3 3.38.0.

* go mod tidy
2022-08-01 22:25:44 -07:00
Matt JoinerandGitHub c47c8cc9d9 Fix broken tags link 2022-08-01 18:31:22 +10:00
Dom HuttonandGitHub 3bba76f90a modify: Add statement timeout query parmeter for MySQL (#778)
* modify: Add statement timeout query parmeter for MySQL

* fix: parsing and assignment of statementTimeout

* fix: bad merge
2022-07-27 11:14:52 -07:00
Jon KjennbakkenandGitHub 0d585e2a56 Switch to microsoft go-mssqldb (#758)
* Switch to https://github.com/microsoft/go-mssqldb

* Fix undeclared name

* rebase

* Merge master
2022-07-27 11:12:27 -07:00
Dale HuiandGitHub eebc4c437a Merge pull request #755 from abhinavnair/replace-ioutil
Replace deprecated ioutil pkg with os & io
2022-07-26 23:59:08 -07:00
Dale HuiandGitHub fca5869487 Merge pull request #774 from linzhengen/fix_spanner_parsing_issue
fix spanner parsing issue
2022-07-26 23:23:02 -07:00
Dale HuiandGitHub af0c073f52 Merge pull request #770 from cdamoc90poe/bump-version-for-sqlite3
bump version for sqlite3 lib to 1.14.14
2022-07-26 23:20:33 -07:00
Dale HuiandGitHub edff41a91c Merge pull request #776 from SinghCoder/patch-1
chore: Correct interface name in comments.
2022-07-26 23:08:00 -07:00
Dale HuiandGitHub 5dc2e47f4e Merge pull request #780 from soyayaos/master
jq must be invoked with -r option
2022-07-26 22:52:55 -07:00
Vladimir Galunshchikov 89836f4f11 jq must be invoked with -r option 2022-07-26 18:39:07 +03:00
HarpinderandGitHub 3843f9233f Correct interface name in comments. 2022-07-23 22:03:39 +05:30
linzhengen 185a4ded0e fix spanner parsing issue 2022-07-15 14:11:39 +09:00
Cristian Damoc b2be9e7b62 bump version for sqlite3 lib to 1.14.14 2022-07-12 13:58:19 +03:00
Abhinav Nair 885d03c4af Replace deprecated ioutil pkg with os & io
As of Go 1.16, the same functionality is now provided by package io or
package os, and those implementations should be preferred in new code.

So replacing all usage of ioutil pkg with io & os.
2022-06-26 16:57:29 +08:00
Dale Hui acc11eda6d Use t.Skipf() per golangci-lint (gosimple) 2022-05-13 15:59:07 -07:00
Dale Hui b5e3e92f00 Support Go 1.18 and drop support for Go 1.16 2022-05-13 15:45:41 -07:00
Dale Hui 075d6e8130 Update dktest from v0.3.10 to v0.3.11 to fix upstream security issues 2022-05-06 21:59:35 -07:00
Igor Morozov bcd0d884bf feat: add yugabyte support 2022-04-03 00:25:18 +03:00
Dale Hui 331a15d92a Update dktest from v0.3.9 to v0.3.10 to fix upstream security issues 2022-03-16 23:49:09 -07:00
Dale HuiandGitHub cc5f8b7ea7 Merge pull request #714 from eaze/josh/test-sqlserver-with-edge
Run SQL server tests on Azure SQL Edge
2022-03-16 23:20:51 -07:00
Josh Holbrook 9d3babd356 Run SQL server tests on Azure SQL Edge 2022-03-16 16:50:29 -04:00
Dale HuiandGitHub 57aead3125 Merge pull request #659 from AndreasKl/add-with-connection-to-postgres
Add WithConnection to Postgres similar to MySQL.
2022-02-24 12:40:56 -08:00
Dale HuiandGitHub e1d604b490 Merge pull request #708 from taywrobel/cassandra-connect-timeout
Expose configuring cassandra connect timeout through query string
2022-02-22 14:25:43 -08:00
Andreas KluthandGitHub 3dfae0db3e Merge branch 'master' into add-with-connection-to-postgres 2022-02-21 10:36:13 +01:00
Taylor Wrobel 6e499cb007 Expose configuring cassandra connect timeout through query string
Exposes the cassandra client's ConnectTimeout parameter through
the cassandra database query string.

The Cassandra client has a fairly aggressive connect timeout of
600ms, which can cause flakiness in certain network environments
or in Cassandra-compatible clusters with a more complex process
for establishing connections (i.e. ScyllaDB).

The query timeout is already configurable and exposed, but does
not impact the connect timeout in the gocql driver. This change
allows for configurtion of both.
2022-02-18 16:33:06 -08:00
Dale HuiandGitHub 0bc9734138 Merge pull request #700 from Juneezee/test/t.TempDir
test: use `T.TempDir` to create temporary test directory
2022-02-09 16:41:31 -08:00
Andreas KluthandGitHub 34c9878ba3 Merge branch 'master' into add-with-connection-to-postgres 2022-02-09 21:37:14 +01:00
Eng Zer Jun 98d2f99a88 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-02-09 09:37:48 +08:00
Dale HuiandGitHub 6b95a852e1 Merge pull request #695 from xkisu/master
Incorrect DSN example for Clickhouse
2022-02-06 23:00:40 -08:00
Keith MitchellandGitHub 81cbc9c058 fix: incorrect clickhouse dns example
The current example for the Clickhouse DSN is incorrect and will result in the client trying to use the default username, database, and password regardless of what is set in the DSN. This change fixed the DNS example to be correct for the current versions of the clickhouse-go client.
2022-02-02 17:55:16 +00:00
Andreas KluthandGitHub 19c48918be Merge branch 'master' into add-with-connection-to-postgres 2022-01-20 21:45:09 +01:00
Dale HuiandGitHub 3e43c429ee Merge pull request #686 from pankona/bump-sqlite3-to-1-14-10
Bump go-sqlite3 to v1.14.10
2022-01-12 10:52:36 -08:00
Dale Hui c277388d49 Update dktest to v0.3.9 for upstream security vulnerability fixes 2022-01-12 10:48:53 -08:00
Dale HuiandGitHub bb269a7be1 Merge pull request #687 from bndw/master
docs: Include iofs in migration sources
2022-01-12 09:37:28 -08:00
bndw ed8a8b0e94 Include iofs in migration sources 2022-01-11 21:54:21 -08:00
pankona a46677ef90 bump go-sqlite3 to v1.14.10 2022-01-12 03:33:21 +00:00
Dale HuiandGitHub 576ba6897b Merge pull request #669 from kylechadha/upgrade-spanner-lib
fix(spanner): Upgrade spanner client to latest, which adds JSON_VALUE support
2021-12-17 20:40:25 -08:00
9f5ed82865 Use DELETE FROM instead of TRUNCATE for MySQL (#656)
* Use DELETE FROM instead of TRUNCATE for MySQL

* Set isolation level to SERIALIZABLE

Co-authored-by: Martin Arrieta <mail@martinarrieta.me>
2021-12-17 20:21:25 -08:00
Kyle Chadha 020eda481c Upgrade cloud.google.com/go/spanner to 1.28.0, which adds JSON_VALUE support 2021-12-09 18:53:44 -08:00
moppandGitHub 918e13ad9a Upgrade containerd and image-spec for the vulnerabilities (#662)
* upgrade containerd/containerd to 1.5.8 for vulnerability

* upgrade opencontainers/image-spec to 1.0.2 for vulnerability

* upgrade dktest to v0.3.8
2021-12-09 00:40:24 -08:00
Dale HuiandGitHub 0c500eba13 Merge pull request #664 from suzaku/simplify
len(k) is guaranteed to be greater than 1 in this case
2021-12-07 21:12:12 -08:00
Dale HuiandGitHub ca2d3e0724 Merge pull request #658 from mccalltd/update-spanner-to-1.27
Update Google Cloud Spanner to 1.27
2021-12-07 21:07:09 -08:00
suzaku 169d0fcf29 Include 2 letter query param key in unit test 2021-12-08 10:38:36 +08:00
satoru 4f82413d85 len(k) is guaranteed to be greater than 1 in this case 2021-12-08 10:38:36 +08:00
Dale HuiandGitHub 2d6f0e2ab5 Merge branch 'master' into update-spanner-to-1.27 2021-12-07 14:26:59 -08:00
Khash SajadiandGitHub b69b032d6a clarify the package name (#660)
* clarify the package name

* Adding import and connection string sections

* more clarification around the go usage
2021-12-07 13:52:37 -08:00
Dale HuiandGitHub efbef2a2bd Merge pull request #663 from suzaku/refacotring
refactor: use sort.Slice
2021-12-07 09:12:11 -08:00
satoru fc7f720c9f refactor: use sort.Slice 2021-12-07 14:49:37 +08:00
Andreas Kluth f621644741 Add WithConnection to Postgres similar to MySQL. 2021-11-29 15:50:04 +01:00
Dale Hui cf68462e13 Run gofmt on internal build dir 2021-11-23 16:30:29 -08:00
Dale Hui 090a3c4d85 go mod tidy 2021-11-23 16:28:41 -08:00
Tim McCall 579878468e Update Spanner to 1.27 2021-11-19 10:53:05 -07:00
4ba695707a Resolves #647 - Fixes typos in Mongo advisory locking parameters (#648)
* Resolves #647 - Fixes typos in Mongo advisory locking parameters

* PR Feedback for #647 - Error out when both params set

* Fixes lint issues with #647

* Additional PR Fixes #647

* Lint fixes again #647

* One more tweak to CONTRIBUTING.md

* One more tweak to CONTRIBUTING.md, again

Co-authored-by: Steve Ramage <commits@sjrx.net>
2021-11-04 22:17:35 -07:00
Dale HuiandGitHub efa49bc721 Merge pull request #646 from Greyeye/master
change github auth to use oauth token instead of basic.
2021-11-04 22:16:48 -07:00
Dale HuiandGitHub 675d588f15 Merge branch 'master' into master 2021-11-04 21:48:03 -07:00
Dale HuiandGitHub d11b43e78a Merge pull request #650 from grnhse/bump-versions
Bump go version in go.mod and bump autorest/adal
2021-11-01 23:13:45 -07:00
Juan C. Müller 5aa1a0381e Bump version of autorest/adal 2021-11-01 12:26:02 -04:00
Dale HuiandGitHub b5eae1a189 Merge pull request #651 from glacials/master
Add pkger source to README
2021-10-31 22:11:23 -07:00
Ben CarlssonandGitHub bfd05026fc Set syntax highlighting for pkger example 2021-10-31 13:26:24 -07:00
Ben CarlssonandGitHub 06c25a20bc Add pkger to README 2021-10-31 13:26:08 -07:00
Greyeye a2b0bdadbc change github auth to use oauth token instead of basic. 2021-10-26 10:57:16 +10:00
Dale HuiandGitHub 9f8bb1bb1a Merge pull request #645 from mgenware/patch-1
Use the recommended v4 in mysql README
2021-10-25 16:44:16 -07:00
Liu YuanYuanandGitHub 608e4f666b Use the recommended v4 in mysql README 2021-10-24 23:25:52 +08:00
Dale HuiandGitHub 1ba9becc5c Merge pull request #634 from Baumanar/master
Fix for failing test for AWS keyspace
2021-10-15 09:52:04 -07:00
Baumanar c81e147ad6 go mod tidy 2021-10-14 10:37:49 +02:00
Baumanar dbb574c92c fix test 2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar 79472544ff Delete all rows
https://github.com/golang-migrate/migrate/pull/579#discussion_r650178687
2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar b9b66468f5 Use ParseBool
https://github.com/golang-migrate/migrate/pull/579#discussion_r650171022
2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar 5eaf23f836 Support for AWS Keyspaces 2021-10-14 10:36:38 +02:00
Dale HuiandGitHub f25b303a88 Merge pull request #638 from dhui/gosnowflake
Update gosnowflake from v1.4.3 to v1.6.3
2021-10-13 22:13:24 -07:00
Dale Hui 41431fbdc3 Update gosnowflake from v1.4.3 to v1.6.3
Addresses:
* https://github.com/golang-migrate/migrate/issues/632
* https://github.com/golang-migrate/migrate/issues/637
2021-10-13 21:18:38 -07:00
Dale Hui 780d5551e0 Update docker client usage with breaking change 2021-10-12 22:11:20 -07:00
Dale Hui 71106e5fd2 Update dktest to v0.3.7 to fix security warnings
https://github.com/dhui/dktest/releases/tag/v0.3.7
2021-10-12 21:58:29 -07:00
Dale HuiandGitHub eea81e7876 Merge pull request #628 from golang-migrate/revert-binary-location-change-in-docker-image
revert binary file location change in docker image
2021-09-27 12:12:47 -07:00
Fontinalis 1668f8e6a0 revert binary file location change in docker image 2021-09-27 10:13:33 +02:00
Dale HuiandGitHub 10a92d0cc8 Merge pull request #620 from golang-migrate/fix-source-file-driver
fix source/file driver
2021-09-16 16:45:03 -07:00
Fontinalis 0775323324 fix source/file driver 2021-09-16 14:24:01 +02:00
Dale Hui 14f6d3a92f Update build constraints
Issue caught by linter

https://pkg.go.dev/go/build#hdr-Build_Constraints
https://pkg.go.dev/cmd/go#hdr-Build_constraints
https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
2021-09-10 21:04:11 -07:00
Dale Hui 677b5deb09 Update golangci-lint config
- Update to renamed config keys
- Remove redundant linter (maligned)
2021-09-10 21:02:44 -07:00
Dale HuiandGitHub b4047a1b47 Merge pull request #617 from kaworu/pr/kaworu/database/clickhouse/honor-DefaultMigrationsTableEngine
clickhouse: honor `DefaultMigrationsTableEngine` when created `WithInstance()`
2021-09-10 20:36:47 -07:00
Alexandre Perrin 9db2d011f3 clickhouse: honor DefaultMigrationsTableEngine when created WithInstance()
Before this patch, clickhouse.WithInstance() would not select
DefaultMigrationsTableEngine when no MigrationsTableEngine was provided
through the clickhouse.Config{} argument. This behaviour forces the
caller to always provide explicitely the config's MigrationsTableEngine.

This patch make it so DefaultMigrationsTableEngine is used when
MigrationsTableEngine is not provided through clickhouse.Config{}.

Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
2021-09-10 10:44:55 +02:00
Dale HuiandGitHub aef2747761 Fix grammar for Clickhouse cluster mode docs 2021-08-24 21:45:25 -07:00
aa119418b8 Adding support clickhouse cluster (#568)
* added url query for creating schema_migrations table on cluster

Signed-off-by: Ildar Valiullin <preved.911@gmail.com>

* changed documentation by clickhouse databases

Signed-off-by: Ildar Valiullin <preved.911@gmail.com>

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2021-08-24 21:39:30 -07:00
Dale Hui df6106d10e Support Go 1.17 and drop support for Go 1.15 2021-08-21 01:32:23 -07:00
Dale Hui b62d267202 Push builds for debian bullseye 2021-08-18 16:04:22 -07:00
Dale Hui d30a9850da Change build status badge to use GitHub Actions 2021-08-06 21:50:44 -07:00
Dale Hui bea4b82719 Fix docker image tags by undoing previous commit 2021-08-06 21:38:45 -07:00
Dale Hui ea0fa43397 Use fewer docker image tags 2021-08-06 20:41:19 -07:00
Dale HuiandGitHub 82dd9d78ce Merge pull request #609 from golang-migrate/607-fix-dockerfile
fix missing /migrate link in docker image
2021-08-06 09:58:35 -07:00
Fontinalis dba334c74e fix missing /migrate link in docker image 2021-08-06 10:44:05 +02:00
Dale HuiandGitHub 784a2f89da Merge pull request #608 from alethenorio/update-spanner-dependency
Update Google Spanner dependency
2021-08-05 21:20:58 -07:00
Eng Zer JunandGitHub 5aa3b3ad30 ci: migrate from CircleCI to GitHub Actions (#605)
* ci: migrate from CircleCI to GitHub Actions

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* ci: merge release.yaml into ci.yaml

At the time of this commit, it is not possible to make the `goreleaser`
job run in a separate workflow and detect the tag push event, see [1].

[1]: https://stackoverflow.com/a/68078768/7902371
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-05 21:12:21 -07:00
Alexandre Thenorio ab5fe5e783 Update Google Spanner dependency
Adds support for the following in SpanSQL

* ROW DELETION POLICY
* ALTER DATABASE
* Parsing keywords and functions in case insensitive
2021-08-05 19:25:26 +02:00
Dale Hui b2c89c743c Clean up Makefile targets
* move echo-source and echo-database targets so the build target is first and default
* mark echo-source and echo-database targets as .PHONY
2021-08-04 16:19:03 -07:00
Dale Hui a4c42d7390 Include source archives with releases
See: https://goreleaser.com/customization/source/
2021-08-03 21:41:15 -07:00
Dale Hui 823c2806a0 Allow prereleases via GoReleaser 2021-08-03 12:19:56 -07:00
5a601b986d Support for GitHub Actions and goreleaser (#604)
* goreleaser configuration and automatic release workflow

* update and fix goreleaser

* remove windows arm ignore

* add format override for windows to use zip files

* update environment and docker username

* the single source of truth

* fix depricated api issue ( thanks @dhui )

Co-authored-by: Mitsuo Heijo <mitsuo.heijo@gmail.com>
2021-07-30 23:09:32 -07:00
Dale HuiandGitHub 5bf05dc323 Remove -o option from go install example in CLI docs
The -o option only exists in `go build`, not `go install`. e.g. it's not a build flag
See: https://github.com/golang-migrate/migrate/issues/603#issuecomment-888432107
2021-07-28 09:06:18 -07:00
Dale HuiandGitHub 586cc866f5 Merge pull request #595 from DGollings/master
use github.com/cockroachdb/cockroach-go/v2/crdb
2021-07-27 09:22:16 -07:00
Dale Hui 614523948d Merge branch 'master' into crdb 2021-07-26 22:13:48 -07:00
Dale Hui aa47d3da8c Update MongoDB db driver to fix security issue
https://nvd.nist.gov/vuln/detail/CVE-2021-20329
2021-07-23 13:46:20 -07:00
Daniel Gollings 94649eedd8 ran go mod tidy 2021-07-14 14:44:05 +02:00
Daniel Gollings f1b7710148 use github.com/cockroachdb/cockroach-go/v2/crdb (for better go module support) 2021-07-14 14:35:05 +02:00
Sébastien CaparrosandGitHub 31cedbb961 Added a method to create a mysql database from a connection object (#583)
* Added a method to create a mysql database from a connection object

* Calling WithConnection from WithInstance to de-duplicate code

* Adding context and ping to mysql.WithConnection

* Interface type check at compile time
2021-07-06 10:45:43 -07:00
805f8c8f2d Add SqlServer MSI Support (#591)
* MSI Auth for SQL Server

Add MSI Auth option to SQL Server connection. Default if no password is provided in connection string.

* Parse resource endpoint from server for msi

update host name parsing to get just the resource endpoint for msi

* Update go-mssqldb

update go-mssqldb to resolve panic issue referenced here: https://github.com/denisenkom/go-mssqldb/pull/642

* Update sqlserver.go

switch from deprecated methods. NewServicePrincipalTokenFromManagedIdentity calls the two methods that are deprecated for us

* Update sqlserver.go

add useMsi param instead of looking for nil password

* Update sqlserver readme

Update sqlserver readme for msi auth. make useMsi a bit safer

* Update sqlserver.go

remove comment

* Update database/sqlserver/README.md

Co-authored-by: Keegan Campbell <me@kfcampbell.com>

* Update sqlserver.go

refactor resource uri logic into its own method

* Update sqlserver_test.go

add tests for msi connection. can only test whether it fails with useMsi= true, or succeeds with useMsi=false

* Update sqlserver.go

check msi.EnsureFresh return value

* Return error for multiple auth and move query filter

move migrate.FilterCustomQuery(purl).String() into one line out of if/else. return error if both useMsi=true and password are passed

* Update README.md

update readme with warning for useMsi

* Update sqlserver_test.go

Update TestMsiFalse test case as now it should fail when useMsi=false and no password is provided

Co-authored-by: Keegan Campbell <me@kfcampbell.com>
2021-07-01 23:08:44 -07:00
3dfb0ff538 used uber atomic bool instead standard in lock/unlock db (#580)
* used uber atomic bool instead standard in lock/unlock db

* fixing local lock optimization implementation

* fixing mysql local lock optimization implementation

* fixing cockroachdb local lock optimization implementation

* CAS wrapper to automatically restore

* added test for cas with restore

* added atomic lock to sqlite & mongo

* added atomic lock to sqlite & mongo

* fix/refactor test cas restore function

* disable lock for mongo

* mongo local lock before acquiring the db lock

Co-authored-by: a.prinkov <aprinkov@ourgapps.com>
2021-06-23 14:40:50 -07:00
Dale Hui 3b3c1b6aa5 Update go-github from v17 to v35 2021-05-28 14:56:29 -07:00
Dale Hui 8a1a8534dc Update snowflake DB driver to update to non-vulnerable jwt downstream package
Fixes https://github.com/golang-migrate/migrate/issues/569
2021-05-21 09:56:26 -07:00
Kyle ChadhaandGitHub 1695f40b3e fix(spanner): upgrade spanner client to latest to support NUMERIC types (#565)
* fix(spanner): upgrade spanner client to latest to support NUMERIC types

* run go mod tidy

* Add spanner example migration test case
2021-05-18 22:16:21 -07:00
Dale HuiandGitHub 2e7358e012 Merge pull request #560 from dhui/re-add-iofs-src-driver
Re-add the io/fs source driver
2021-05-10 23:38:05 -07:00
Dale Hui eac6684bf3 Fix iofs tests
Changes:
- embedded variables must be at the package level
- re-use embedded FS in example for tests
- bump Go module version to 1.16 for tests to run
    - This may cause issues building in 1.15
2021-05-10 22:49:46 -07:00
Dale Hui 53720240b4 Update docker builds to use Go 1.16 and Alpine Linux 3.13 2021-05-10 11:28:59 -07:00
Dale Hui 337ff632e3 Revert "Merge pull request #481 from johejo/revert-472-iofs_source_driver"
This reverts commit b690b78e22, reversing
changes made to 15931649a2.
2021-05-10 10:50:27 -07:00
Dale HuiandGitHub c4d82aa89a Merge pull request #558 from zikaeroh/bump-sqlite
Bump sqlite past memory corruption bug
2021-05-06 22:35:36 -07:00
zikaeroh 7a631fe801 Bump sqlite past memory corruption bug 2021-05-06 21:32:04 -07:00
Dale Hui ffdcb52998 Update sqlite and sqlite3 docs with comparison 2021-05-05 10:12:41 -07:00
Andrew GillisandGitHub 512097b404 Add support for modernc.org/sqlite (#555)
* Add support for modernc.org/sqlite

This is a CGo-free port of SQLite

* Remove sqlite from build target
2021-05-05 10:04:18 -07:00
Dale HuiandGitHub af256da8ca Merge pull request #550 from spacefill/547-postgres-pgx-ensureVersionTable-must-use-config-schema-name
[pgx][postgres] Config.SchemaName refactoring usage
2021-04-25 13:45:05 -07:00
Stéphane Klein d0af5824ad postgres,pgx: fix SchemaName parameter is ignored ##547 2021-04-25 07:34:19 +02:00
Bruno MouraandGitHub 4720914286 clickhouse: make table engine for schema versions table configurable (#551)
* clickhouse: make the table engine used for schema_migrations configurable

* clickhouse: tests creation with configurable table engine

* clickhouse: refactor tests to run with multiple table engines

* clickhouse: clarify the use case for the x-migrations-table-engine option

* clickhouse: fix naming consistency for the migrations table engine

* clickhouse: fix tests function wrap pattern
2021-04-21 22:03:11 -07:00
Stéphane KleinandGitHub 3d5d577b61 Postgres - add x-migrations-table-quoted url query option (#95) (#533)
* Postgres and pgx - Add x-migrations-table-quoted url query option to postgres and pgx drivers (#95)

By default, gomigrate quote migrations table name, if `x-migrations-table-quoted` is enabled, then you must to quote migrations table name manually, for instance `"gomigrate"."schema_migrations"`

* Work In Progress
2021-04-19 14:57:50 -07:00
Alexey PrinkovandGitHub d68a3abedb added tests for clickhouse database (#532)
* added tests for clickhouse

* added atomic lock flag to clickhouse & remove extra test

* added clickhouse test package
2021-04-05 15:37:50 -07:00
testtest959andGitHub 511ae9f5b6 Postgres: Add a check to determine if table already exists to elide CREATE query (#526)
* Squash commits

* Format

* Minor refactoring

* Address PR feedback; Add mustRun

* Fix a test assert
2021-03-18 21:03:57 -07:00
Dale HuiandGitHub 63aff4b03d Merge pull request #529 from linux019/upgrade_ch
upgrade clickhouse driver lib
2021-03-16 20:23:20 -07:00
= 2839a3d6f2 upgrade clickhouse driver lib 2021-03-16 18:51:56 +02:00
Dale HuiandGitHub 4573d98dce Merge pull request #527 from jeffwidman/patch-2
Fix typo
2021-03-15 23:58:57 -07:00
Jeff WidmanandGitHub a728e7a674 Fix typo 2021-03-15 21:45:44 -07:00
Dale HuiandGitHub 707fc52dfe Merge pull request #522 from Juneezee/remove-all-from-drop
Move `-all` flag usage from `drop` to `down`
2021-03-13 21:36:01 -08:00
Jeff WidmanandGitHub 6ecf7bcd0a Point docs link to v4 (#524)
* Point docs link to `v4`

It's very confusing that when I land directly on `https://pkg.go.dev/github.com/golang-migrate/migrate` I get a the message:
> You are look at a deprecated version, use the latest version instead

But when I click the `latest version` link, it goes to a completely dead/wrong place.

So I think for almost anyone who actually wants to look at the real docs, it's a lot simpler to link straight to `v4` by default. If they want a different version, they can change that. But let's have the default point to a helpful place, and then you can go looking for the broken version, rather than the other way around.

* Update the badge link as well
2021-03-13 21:30:45 -08:00
Dylan BargteilandGitHub 2f0c0752ed Add pgx driver package (#517)
* replace lib/pq dependency with pgx in postgres driver

* fix pgx error handling

pgconn is a low-level library, and errors from queries flow through higher-level pgx library

* update postgres tests to use pgx driver and error reporting

* update quoteIdentifier implementation to match lib/pq

* move changes for pgx support into new driver package

* add pgx driver to CLI

* restore examples used in testing for pgx driver

* replace postgres driver name with pgx in DSN in tests and docs

* consolidate connection string formatting in pgx testing

* lock pgx package at v4

Prior to v4, pgx did not support go modules. Version locking the import is the safest way to ensure we get the (as of now) latest version of the module. Additionally v4 implements some breaking changes to connection and error handling, which are also addressed in this commit.

* update parsed URL scheme instead of using string operation

* add pgx for builds using make

* add pgx to top level docs

* remove autogenerated TOC
2021-03-13 21:29:10 -08:00
Eng Zer Jun aa2dad421f fix: move -all flag usage from drop to down
The `-all` flag usage should be used in `down` only, not `drop`.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-03-13 15:22:09 +08:00
Dale HuiandGitHub 7dc03e3c0d Merge pull request #509 from nronas/master
Allow the ability to pass timezone when generating migration files
2021-02-24 09:36:46 -08:00
John Tzikas 6cdb8c1b23 Allow the abillity to pass timezone when generating migration files 2021-02-23 20:20:14 +02:00
Dale Hui e57a024960 Support Go 1.16 and drop support for Go 1.15
- Document CLI install instructions for Go 1.16
- Update golangci-lint from v1.33.0 to v1.37.0
2021-02-18 21:33:36 -08:00
Dale Hui 8709f196bf Restore CircleCI build cache after checkout and remove branch from cache key 2021-02-09 21:46:40 -08:00
Dale Hui 70d6a03251 Upgrade dktest from v0.3.3 to v0.3.4
Addresses: https://github.com/golang-migrate/migrate/issues/506
2021-02-09 20:48:31 -08:00
Dale Hui ff92da45f1 Use strconv.ParseBool() to parse the x-multi-statement option in the postgres driver 2021-02-03 22:59:24 -08:00
Dale HuiandGitHub a185b9e78f Merge pull request #495 from AnatolyRugalev/postgresql-support-multistatement
Support multi-statement execution for PostgreSQL
2021-02-02 23:18:28 -08:00
Anatoly RugalevandGitHub 42abb24ae0 Merge branch 'master' into postgresql-support-multistatement 2021-02-02 13:24:47 +03:00
Anatoly Rugalev 38ffbfb778 Addressed review comments 2021-02-02 11:24:33 +01:00
BenandGitHub dab829b564 Add default GitHub API client (#501) 2021-01-30 23:07:53 -08:00
BenandGitHub eb62a386cc sqlite3: Fix error when path to database contains whitespace (#500) 2021-01-28 23:36:56 -08:00
6d8df972da Fix issue when gitlab return only first 20 migrations (reopen) (#497)
* Fix issue when gitlab return only first 20 migrations

* CircleCI issue. repush

* gofmt with -s

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2021-01-28 14:35:14 -08:00
Dale HuiandGitHub 550193e3e0 Fix CircleCI badge link 2021-01-26 00:27:32 -08:00
Dale Hui 5ed79873f7 Update badge to reflect move from TravisCI to CircleCI 2021-01-26 00:23:44 -08:00
Dale Hui 20b5378936 Use CircleCI
Squashed commit of the following:

commit 5f63ff0333a9608530b4bb79b44af837224f80a5
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:36:57 2021 -0800

    Remove unnecessary parameter to checkout

commit b87711057588bfe05fb01d8ff9b679200b2c3015
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:30:37 2021 -0800

    Give up trying to only push coverage for Go 1.15 via YAML templates

commit 6f366c35d11a903fdd8d92738de26073fe7a9c0f
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:27:20 2021 -0800

    YAML template merging only works with maps, not scalars, so specify the current directory for "checkout"

commit e65ff967018508f36e9b9d88f20fe135d8ce6c3e
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:21:30 2021 -0800

    Use `go get` instead of `go install` and add template for steps

commit fef2a9e7f9bfbb6070b849199e7d0cced8c6747a
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 19:59:24 2021 -0800

    Push code coverage to coveralls.io
        - Only push code coverage for latest support version of Go

commit 48d78bb2576aa8662daaa3e116d0e19dafa77279
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:24:13 2020 -0800

    golangci-lint runs slowly on circle ci machines...

commit d47e04f9aad971a72f169089369ed9e976fbf94f
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:20:35 2020 -0800

    Cleanup setup and remove debugging commands

commit 063337f2faff23b6ca636d77acbd4c504ac6914b
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:17:57 2020 -0800

    try using ~/bin

commit 57f34fbf5ce38ff8fdfdb4be43958d43db01d9f6
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:13:55 2020 -0800

    Install golangci-lint in another dir

commit 2b5d6d65ee4807e3c2df0afffc29498defe5a6c3
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:03:08 2020 -0800

    More go env debugging

commit 6719f663d582e59a551bb10ff7365aef814c1d85
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:59:24 2020 -0800

    Sparate downloading and running of golangci-lint.sh

commit 6e9ec9ff0c2d7353b711af5ad685bb4d07ff7292
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:54:44 2020 -0800

    debug gimme env

commit 706a43ca67e012c7982deb115a7168fc2f9067c3
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:47:01 2020 -0800

    debug go env

commit 18694c6419ceeaf0b649939249a618f3876ef530
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:39:53 2020 -0800

    Use .x version of Go

commit 2e4a87b03d1e0b8cf66719bea672304ae008d881
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:36:01 2020 -0800

    correctly use gimme

commit 99a1b838df8832243fc85cdb30d69a07042f9f93
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:31:45 2020 -0800

    Install golangci-lint after installing requested version of Go

commit cecf219e557d5cac26f3f1f43955cfcdb397bb50
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:29:48 2020 -0800

    Still don't support docker layer caching

commit 244ec8fe0b6e5d28cb9c16159bbc3d36ba5ec523
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:28:33 2020 -0800

    Try using machine executor

commit 44d3b29a1e8f953dcf4a2051e0d1037e7b62f85b
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:07:09 2020 -0800

    Fix docker run

commit 525983e81d6a3080e2b28bfdb601581346926450
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:03:39 2020 -0800

    properly specify build-arg

commit d5eed206a6b11c2606690ffdee4639e5399cdfc7
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 20:54:06 2020 -0800

    Try running CircleCI tests in a Docker container

commit 35e62d1c4ccdeb528495f9d28f26a049bcc93bd8
Merge: 90268de a53e6fc
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:50:58 2020 -0800

    Merge remote-tracking branch 'origin/master' into circleci

commit 90268dedc2cf72ab498ebfd26465551a16463ee7
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:47:46 2020 -0800

    Update golangci lint to v1.33.0

commit d5e0747a53ddfdb7b1ce6ff4bacb236bfc1b7794
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:44:45 2020 -0800

    Don't have access to docker layer caching

commit 6aaecb3b761809b6cf2b4f365f2f9d62f410661d
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:43:52 2020 -0800

    Don't have access to xlarge

commit 275de107de76ced3d6b20e913ffddff84919c3e5
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:39:14 2020 -0800

    CircleCI job names can't have a '.' character

commit e1d339f0eb10519f6002939147201060415a98bb
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:30:51 2020 -0800

    Update CircleCI config

commit d65194971c028af94e7a3cd171eb6221a08dde05
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Fri Aug 2 00:05:59 2019 -0700

    Add config for CircleCI

commit 19d7355548c2a6d64225ea63368e10e19eb43d19
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Jul 30 00:12:41 2019 -0700

    Update golangci-lint from v1.16.0 to v1.17.1
2021-01-25 21:08:52 -08:00
Anatoly Rugalev 3a5169ac6d Support multi-statement execution for PostgreSQL 2021-01-18 12:16:37 +03:00
Mitsuo HeijoandGitHub a53e6fc425 fix test on windows (#482)
* fix test on windows

On Windows, the test fails because os.Remove cannot remove unclosed and still-handled files.

* use t.Error instead of t.Log
2020-11-24 22:53:21 -08:00
Dale HuiandGitHub b690b78e22 Merge pull request #481 from johejo/revert-472-iofs_source_driver
Revert "Add io/fs#FS Driver #471"
2020-11-21 21:40:35 -08:00
Mitsuo HeijoandGitHub 67546bfa67 Revert "Add io/fs#FS Driver #471 (#472)"
This reverts commit fedbe42fc8.
2020-11-21 23:27:36 +09:00
Charles MartinotandGitHub 15931649a2 Mysql - Fix crash with ansi_quotes sql_mode enabled (#476)
* Fix crash with ansi_quotes

* Run lint
2020-11-18 12:38:28 -08:00
Mitsuo HeijoandGitHub fedbe42fc8 Add io/fs#FS Driver #471 (#472)
* Add io/fs#FS Driver #471

* Refactor iofs.Iofs to be independent of httpfs

* Rename Iofs to IoFS, accepts fs.FS and use fs.DirEntry.Info() to undo the changes in ErrDuplicateMigration

* fix documents and migration directory for testing

* Fix iofs.Driver so that it can be used by embedding it like httpfs.PartialDriver

* iofs.Driver closes the file system if possible

* Rename receiver

* Refactor file driver to use iofs.Driver

* iofs build constraint is not needed so remove it

* Properly return an error in a corner case

* Increase golangci-lint timeout

* Fix example code

* Revert "Increase golangci-lint timeout"

This reverts commit 2cd12a63b50b7ac48d12dd0adc1ec4f4b49c3b93.

* unexport driver, add iofs.PartialDriver, remove type alias from file driver

* do not panic in Open, return error
2020-11-18 10:06:11 -08:00
Dale Hui 52e7117d37 Use flagset's NArg() and Arg() methods instead of global flag NArg() and Arg() for sub-commands
Fixes: https://github.com/golang-migrate/migrate/issues/467
2020-11-04 23:37:05 -08:00
Yannick HeinrichandGitHub 8a56273871 Add help statement for cli (#460)
* Add help statement for cli

Refers to #372

* Make constant for argument

* Create a flagset for each command

* Factor error handling

* Factor flagset creation

* fix: Typo and bad behaviour

* Fix lint error

* Fix according to the comments

* Fix according to the comments
2020-11-03 14:54:12 -08:00
Dale Hui 110c0f74b4 Update dktest to v0.3.3 which also updates lib/pq to v1.8.0 2020-10-30 22:26:55 -07:00
EvanandGitHub f76b94ab7d Document drop -f in migrate -help (#459)
* Document drop -f in migrate -help

* Document drop -f in migrate -help 2
2020-10-17 01:44:43 -07:00
Vladimir DimitrovandGitHub 91deba8df0 Update driver.go (#454)
* Update driver.go

* addressing comments
2020-10-16 22:19:05 -07:00
Maarten BezemerandGitHub f18b16515a postgres - Force MigrationsTable in 'public' schema (#414)
* Force MigrationsTable on public schema

* Revert "Force MigrationsTable on public schema"

This reverts commit e5b25e8858ed42d19d420cb8bd16795c060994d2.

* Add example to fix search_path issue

* Use POSTGRESQL_URL in example and fix typo

* Make example simpler by using the `search_path` query parameter of the DB URL
2020-10-15 20:21:00 -07:00
Dale HuiandGitHub f8f021730e Merge pull request #458 from xortive/patch-1
Fix typo in doc comment
2020-10-13 09:19:03 -07:00
MattandGitHub a17aa6e83d Fix typo in doc comment 2020-10-13 11:55:41 -04:00
Dale HuiandGitHub 5d84bec1c8 Merge pull request #456 from hypnoglow/iter-err
Check iterator error on Drop where applicable
2020-10-10 20:46:14 -07:00
Igor Zibarev 5f932c05c8 Check iterator error on Drop where applicable 2020-10-10 23:53:20 +03:00
Dale HuiandGitHub 0521ebeb71 Warn about installing the CLI from within a module
Addresses: https://github.com/golang-migrate/migrate/pull/257
2020-10-07 16:51:03 -07:00
Dale Hui 99ca856a7c Run go fix 2020-10-07 15:31:28 -07:00
Dale Hui 04a572114a Parse out custom TLS params and register the TLS config before calling mysql.ParseDSN
Fixes: https://github.com/golang-migrate/migrate/issues/411
2020-10-06 19:49:59 -07:00
Dale Hui 8c531c7836 Use spanner cloud emulator to run tests
Addresses: https://github.com/golang-migrate/migrate/issues/410
2020-10-01 17:58:26 -07:00
Dale Hui fa3e11943a Add local locks for spanner db driver 2020-10-01 17:58:23 -07:00
d6881f1826 Add Bitbucket Cloud Support as source (#450)
* Added bitbucket cloud support as source for migration

* added test

* address PR Comments

* Add entry in README.md; update go.mod and go.sum

Co-authored-by: abhigupta91 <abhigupta0818@gmail.com>
2020-09-28 22:08:20 -07:00
208 changed files with 10406 additions and 1830 deletions
+56
View File
@@ -0,0 +1,56 @@
# Golang CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-go/ for more details
version: 2.1
jobs:
"golang-1_15": &template
machine:
# https://circleci.com/docs/2.0/configuration-reference/#available-machine-images
image: ubuntu-2004:202010-01
# docker_layer_caching: true
# https://circleci.com/docs/2.0/configuration-reference/#resource_class
resource_class: medium
# Leave working directory unspecified and use defaults:
# https://circleci.com/blog/go-v1.11-modules-and-circleci/
# working_directory: /go/src/github.com/golang-migrate/migrate
environment:
GO111MODULE: "on"
GO_VERSION: "1.15.x"
steps:
# - setup_remote_docker:
# version: 19.03.13
# docker_layer_caching: true
- run: curl -sL -o ~/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
- run: curl -sfL -o ~/bin/golangci-lint.sh https://install.goreleaser.com/github.com/golangci/golangci-lint.sh
- run: chmod +x ~/bin/gimme ~/bin/golangci-lint.sh
- run: eval "$(gimme $GO_VERSION)"
- run: golangci-lint.sh -b ~/bin v1.37.0
- checkout
- restore_cache:
keys:
- go-mod-v1-{{ arch }}-{{ checksum "go.sum" }}
- run: golangci-lint run
- run: make test COVERAGE_DIR=/tmp/coverage
- save_cache:
key: go-mod-v1-{{ arch }}-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod"
- run: go get github.com/mattn/goveralls
- run: goveralls -service=circle-ci -coverprofile /tmp/coverage/combined.txt
"golang-1_16":
<<: *template
environment:
GO_VERSION: "1.16.x"
workflows:
version: 2
build:
jobs:
- "golang-1_15"
- "golang-1_16"
+1 -1
View File
@@ -25,7 +25,7 @@ e.g. s3, github, go-bindata, gcs, file
Obtained by running: `migrate -help`
**Loaded Database Drivers**
e.g. spanner, stub, clickhouse, cockroachdb, crdb-postgres, postgres, postgresql, redshift, cassandra, cockroach, mysql
e.g. spanner, stub, datastore, cockroachdb, crdb-postgres, postgres, postgresql, pgx, redshift, cassandra, cockroach, mysql
Obtained by running: `migrate -help`
**Go Version**
+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1280" height="640" viewBox="0 0 1280 640" role="img" aria-label="migrate">
<rect width="1280" height="640" fill="#0A0A0A"/>
<svg x="96" y="215" width="210" height="210" viewBox="0 0 67 67"><path d="M22.21 67V44.6369H0V67H22.21Z" fill="#fff"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z" fill="#fff"/><path d="M22.21 0H0V22.3184H22.21V0Z" fill="#fff"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z" fill="#fff"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z" fill="#fff"/></svg>
<text x="378" y="276" font-family="Inter,system-ui,-apple-system,sans-serif" font-size="78" font-weight="800" letter-spacing="-2" fill="#ffffff">migrate</text>
<text x="378" y="322" font-family="Inter,system-ui,sans-serif" font-size="30" fill="#ffffff" opacity=".66">Database migrations. CLI and Golang library.</text>
<rect x="378" y="338" width="806" height="3" rx="1.5" fill="#ffffff" opacity=".9"/>
<text x="378" y="390" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">github.com/hanzoai</text>
<text x="1184" y="390" text-anchor="end" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">hanzo.ai</text>
</svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

+111
View File
@@ -0,0 +1,111 @@
name: CI
on:
push:
pull_request:
jobs:
lint:
name: lint
runs-on: hanzo-build-linux-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.26.4.x"
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.6.2
test:
runs-on: hanzo-build-linux-amd64
strategy:
matrix:
go: ["1.24.x", "1.25.x"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Run test
run: make test COVERAGE_DIR=/tmp/coverage
- name: Send goveralls coverage
uses: shogo82148/actions-goveralls@v1
with:
path-to-profile: /tmp/coverage/combined.txt
flag-name: Go-${{ matrix.go }}
parallel: true
check-coverage:
name: Check coverage
needs: [test]
runs-on: hanzo-build-linux-amd64
steps:
- uses: shogo82148/actions-goveralls@v1
with:
parallel-finished: true
goreleaser:
name: Release a new version
needs: [lint, test]
runs-on: hanzo-build-linux-amd64
environment: GoReleaser
# This job only runs when
# 1. When the previous `lint` and `test` jobs has completed successfully
# 2. When the repository is not a fork, i.e. it will only run on the official golang-migrate/migrate
# 3. When the workflow is triggered by a tag with `v` prefix
if: ${{ success() && github.repository == 'golang-migrate/migrate' && startsWith(github.ref, 'refs/tags/v') }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ruby/setup-ruby@v1
with:
ruby-version: 2.7
- uses: actions/setup-go@v5
with:
go-version: "1.26.4.x"
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
username: golangmigrate
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: echo "SOURCE=$(make echo-source)" >> $GITHUB_ENV
- run: echo "DATABASE=$(make echo-database)" >> $GITHUB_ENV
- uses: goreleaser/goreleaser-action@v5
with:
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: gem install package_cloud
- run: package_cloud push golang-migrate/migrate/ubuntu/bionic dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/focal dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/jammy dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/noble dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/buster dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/bullseye dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/bookworm dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
+52
View File
@@ -0,0 +1,52 @@
name: Release
on:
push:
tags:
- 'v*'
permissions:
contents: write
packages: write
jobs:
release:
runs-on: hanzo-build-linux-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.26.4'
- name: Build CLI binaries
run: |
VERSION=${GITHUB_REF_NAME#v}
mkdir -p cli/build
for pair in "linux/amd64" "linux/arm64" "darwin/amd64" "darwin/arm64"; do
OS=${pair%/*}
ARCH=${pair#*/}
OUTPUT="cli/build/migrate.${OS}-${ARCH}"
echo "Building $OUTPUT..."
CGO_ENABLED=0 GOOS=$OS GOARCH=$ARCH \
go build -a -o "$OUTPUT" \
-ldflags="-s -w -X main.Version=$VERSION" \
-tags 'datastore file' \
./cmd/migrate
ln -f "$OUTPUT" cli/build/migrate
tar czf "${OUTPUT}.tar.gz" -C cli/build migrate
done
cd cli/build && sha256sum *.tar.gz > sha256sum.txt
- name: Create GitHub Release
env:
GH_TOKEN: ${{ github.token }}
run: |
gh release create ${{ github.ref_name }} \
cli/build/*.tar.gz \
cli/build/sha256sum.txt \
--title "${{ github.ref_name }}" \
--generate-notes \
--latest
+3 -1
View File
@@ -6,4 +6,6 @@ cli/migrate
.godoc.pid
vendor/
.vscode/
.idea/
.idea
dist/
migrate
+30 -20
View File
@@ -1,27 +1,37 @@
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
timeout: 2m
version: "2"
linters:
enable:
#- golint
- interfacer
- unconvert
#- dupl
- goconst
- gofmt
- misspell
- maligned
- unparam
- nakedret
- prealloc
#- gosec
linters-settings:
misspell:
locale: US
- revive
- unconvert
- unparam
settings:
misspell:
locale: US
revive:
rules:
- name: redundant-build-tag
exclusions:
generated: lax
rules:
- path: (.+)\.go$
text: G104
paths:
- third_party$
- builtin$
- examples$
issues:
max-same: 0
max-per-linter: 0
exclude-use-default: false
exclude:
# gosec: Duplicated errcheck checks
- G104
max-issues-per-linter: 0
max-same-issues: 0
formatters:
enable:
- gofmt
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
+102
View File
@@ -0,0 +1,102 @@
project_name: migrate
before:
hooks:
- go mod tidy
builds:
- env:
- CGO_ENABLED=0
goos:
- linux
- windows
- darwin
goarch:
- amd64
- arm
- arm64
- 386
goarm:
- 7
main: ./cmd/migrate
ldflags:
- '-w -s -X main.Version={{ .Version }} -extldflags "static"'
flags:
- "-tags={{ .Env.DATABASE }} {{ .Env.SOURCE }}"
- "-trimpath"
nfpms:
- homepage: "https://github.com/golang-migrate/migrate"
maintainer: "dhui@users.noreply.github.com"
license: MIT
description: "Database migrations"
formats:
- deb
file_name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
dockers:
- goos: linux
goarch: amd64
dockerfile: Dockerfile.github-actions
use: buildx
ids:
- migrate
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
build_flag_templates:
- '--label=org.opencontainers.image.created={{ .Date }}'
- '--label=org.opencontainers.image.title={{ .ProjectName }}'
- '--label=org.opencontainers.image.revision={{ .FullCommit }}'
- '--label=org.opencontainers.image.version={{ .Version }}'
- "--label=org.opencontainers.image.source={{ .GitURL }}"
- "--platform=linux/amd64"
- goos: linux
goarch: arm64
dockerfile: Dockerfile.github-actions
use: buildx
ids:
- migrate
image_templates:
- 'migrate/migrate:{{ .Tag }}-arm64'
build_flag_templates:
- '--label=org.opencontainers.image.created={{ .Date }}'
- '--label=org.opencontainers.image.title={{ .ProjectName }}'
- '--label=org.opencontainers.image.revision={{ .FullCommit }}'
- '--label=org.opencontainers.image.version={{ .Version }}'
- "--label=org.opencontainers.image.source={{ .GitURL }}"
- "--platform=linux/arm64"
docker_manifests:
- name_template: 'migrate/migrate:{{ .Tag }}'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
- name_template: 'migrate/migrate:{{ .Major }}'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
- name_template: 'migrate/migrate:latest'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
archives:
- name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
format_overrides:
- goos: windows
format: zip
checksum:
name_template: 'sha256sum.txt'
release:
draft: true
prerelease: auto
source:
enabled: true
format: zip
changelog:
skip: false
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
- Merge pull request
- Merge branch
- go mod tidy
snapshot:
name_template: "{{ .Tag }}-next"
Symlink
+1
View File
@@ -0,0 +1 @@
LLM.md
Symlink
+1
View File
@@ -0,0 +1 @@
LLM.md
+2 -2
View File
@@ -1,4 +1,4 @@
FROM golang:1.15-alpine3.12 AS builder
FROM golang:1.26.4-alpine3.21 AS builder
ARG VERSION
RUN apk add --no-cache git gcc musl-dev make
@@ -15,7 +15,7 @@ COPY . ./
RUN make build-docker
FROM alpine:3.12
FROM alpine:3.21
RUN apk add --no-cache ca-certificates
+17
View File
@@ -0,0 +1,17 @@
ARG DOCKER_IMAGE
FROM $DOCKER_IMAGE
RUN apk add --no-cache git gcc musl-dev make
WORKDIR /go/src/github.com/golang-migrate/migrate
ENV GO111MODULE=on
ENV COVERAGE_DIR=/tmp/coverage
COPY go.mod go.sum ./
RUN go mod download
COPY . ./
CMD ["make", "test"]
+11
View File
@@ -0,0 +1,11 @@
FROM alpine:3.19
RUN apk add --no-cache ca-certificates
COPY migrate /usr/local/bin/migrate
RUN ln -s /usr/local/bin/migrate /usr/bin/migrate
RUN ln -s /usr/local/bin/migrate /migrate
ENTRYPOINT ["migrate"]
CMD ["--help"]
+2 -2
View File
@@ -50,7 +50,7 @@
and whenever we want, not just once at the beginning of all tests.
#### Can I maintain my driver in my own repository?
Yes, technically thats possible. We want to encourage you to contribute your driver to this respository though.
Yes, technically thats possible. We want to encourage you to contribute your driver to this repository though.
The driver's functionality is dictated by migrate's interfaces. That means there should really
just be one driver for a database/ source. We want to prevent a future where several drivers doing the exact same thing,
just implemented a bit differently, co-exist somewhere on GitHub. If users have to do research first to find the
@@ -65,7 +65,7 @@
and then "force" the expected version.
#### What happens if two programs try and update the database at the same time?
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations at the same time
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations on
the same database at the same time. For example, the MySQL driver uses the `GET_LOCK` function, while the Postgres driver uses
the `pg_advisory_lock` function.
+4 -4
View File
@@ -1,7 +1,7 @@
# Getting started
Before you start, you should understand the concept of forward/up and reverse/down database migrations.
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases)
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases).
## Create migrations
Create some migrations using migrate CLI. Here is an example:
@@ -10,7 +10,7 @@ migrate create -ext sql -dir db/migrations -seq create_users_table
```
Once you create your files, you should fill them.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created his migration later gets it merged to the repository first.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created their migration later gets it merged to the repository first.
Developers and Teams should keep an eye on such cases (especially during code review).
[Here](https://github.com/golang-migrate/migrate/issues/179#issuecomment-475821264) is the issue summary if you would like to read more.
@@ -28,9 +28,9 @@ migrate -database YOUR_DATABASE_URL -path PATH_TO_YOUR_MIGRATIONS up
Just add the code to your app and you're ready to go!
Before commiting your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
Before committing your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
(e.g. if you created a table in a migration but reverse migration did not delete it, you will encounter an error when running the forward migration again)
It's also worth checking your migrations in a separate, containerized environment. You can find some tools in the end of this document.
It's also worth checking your migrations in a separate, containerized environment. You can find some tools at the [end of this document](#further-reading).
**IMPORTANT:** If you would like to run multiple instances of your app on different machines be sure to use a database that supports locking when running migrations. Otherwise you may encounter issues.
+39
View File
@@ -0,0 +1,39 @@
# Hanzo Golang Migrate
## Overview
Go module: github.com/golang-migrate/migrate/v4
## Tech Stack
- **Language**: Go
## Build & Run
```bash
go build ./...
go test ./...
```
## Structure
```
golang-migrate/
CONTRIBUTING.md
Dockerfile
Dockerfile.circleci
Dockerfile.github-actions
FAQ.md
GETTING_STARTED.md
LICENSE
MIGRATIONS.md
Makefile
README.md
SECURITY.md
cli/
cmd/
database/
dktesting/
```
## Key Files
- `README.md` -- Project documentation
- `go.mod` -- Go module definition
- `Makefile` -- Build automation
- `Dockerfile` -- Container build
+10 -4
View File
@@ -1,6 +1,6 @@
SOURCE ?= file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver firebird neo4j
DATABASE_TEST ?= $(DATABASE) sqlite sqlcipher
SOURCE ?= file go_bindata github github_ee bitbucket aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb yugabytedb datastore mongodb sqlserver firebird neo4j pgx pgx5 rqlite
DATABASE_TEST ?= $(DATABASE) sqlite sqlite3 sqlcipher
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
REPO_OWNER ?= $(shell cd .. && basename "$$(pwd)")
@@ -98,6 +98,12 @@ release:
git tag v$(V)
@read -p "Press enter to confirm and push to origin ..." && git push origin v$(V)
echo-source:
@echo "$(SOURCE)"
echo-database:
@echo "$(DATABASE)"
define external_deps
@echo '-- $(1)'; go list -f '{{join .Deps "\n"}}' $(1) | grep -v github.com/$(REPO_OWNER)/migrate | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'
@@ -107,7 +113,7 @@ endef
.PHONY: build build-docker build-cli clean test-short test test-with-flags html-coverage \
restore-import-paths rewrite-import-paths list-external-deps release \
docs kill-docs open-docs kill-orphaned-docker-containers
docs kill-docs open-docs kill-orphaned-docker-containers echo-source echo-database
SHELL = /bin/sh
RAND = $(shell echo $$RANDOM)
+26 -11
View File
@@ -1,11 +1,13 @@
[![Build Status](https://img.shields.io/travis/com/golang-migrate/migrate/master.svg)](https://travis-ci.com/golang-migrate/migrate)
[![GoDoc](https://godoc.org/github.com/golang-migrate/migrate?status.svg)](https://godoc.org/github.com/golang-migrate/migrate)
<p align="center"><img src=".github/hero.svg" alt="migrate" width="880"></p>
[![GitHub Workflow Status (branch)](https://img.shields.io/github/actions/workflow/status/golang-migrate/migrate/ci.yaml?branch=master)](https://github.com/golang-migrate/migrate/actions/workflows/ci.yaml?query=branch%3Amaster)
[![GoDoc](https://pkg.go.dev/badge/github.com/golang-migrate/migrate)](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)
[![Coverage Status](https://img.shields.io/coveralls/github/golang-migrate/migrate/master.svg)](https://coveralls.io/github/golang-migrate/migrate?branch=master)
[![packagecloud.io](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/golang-migrate/migrate?filter=debs)
[![Docker Pulls](https://img.shields.io/docker/pulls/migrate/migrate.svg)](https://hub.docker.com/r/migrate/migrate/)
![Supported Go Versions](https://img.shields.io/badge/Go-1.14%2C%201.15-lightgrey.svg)
![Supported Go Versions](https://img.shields.io/badge/Go-1.24%2C%201.25-lightgrey.svg)
[![GitHub Release](https://img.shields.io/github/release/golang-migrate/migrate.svg)](https://github.com/golang-migrate/migrate/releases)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate)](https://goreportcard.com/report/github.com/golang-migrate/migrate)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate/v4)](https://goreportcard.com/report/github.com/golang-migrate/migrate/v4)
# migrate
@@ -24,21 +26,26 @@ Forked from [mattes/migrate](https://github.com/mattes/migrate)
Database drivers run migrations. [Add a new database?](database/driver.go)
* [PostgreSQL](database/postgres)
* [PGX v4](database/pgx)
* [PGX v5](database/pgx/v5)
* [Redshift](database/redshift)
* [Ql](database/ql)
* [Cassandra](database/cassandra)
* [SQLite](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [Cassandra / ScyllaDB](database/cassandra)
* [SQLite](database/sqlite)
* [SQLite3](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [SQLCipher](database/sqlcipher)
* [MySQL/ MariaDB](database/mysql)
* [MySQL / MariaDB](database/mysql)
* [Neo4j](database/neo4j)
* [MongoDB](database/mongodb)
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
* [Google Cloud Spanner](database/spanner)
* [CockroachDB](database/cockroachdb)
* [ClickHouse](database/clickhouse)
* [YugabyteDB](database/yugabytedb)
* [Datastore](database/datastore)
* [Firebird](database/firebird)
* [MS SQL Server](database/sqlserver)
* [rqlite](database/rqlite)
### Database URLs
@@ -66,9 +73,12 @@ $
Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go)
* [Filesystem](source/file) - read from filesystem
* [io/fs](source/iofs) - read from a Go [io/fs](https://pkg.go.dev/io/fs#FS)
* [Go-Bindata](source/go_bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata))
* [pkger](source/pkger) - read from embedded binary data ([markbates/pkger](https://github.com/markbates/pkger))
* [GitHub](source/github) - read from remote GitHub repositories
* [GitHub Enterprise](source/github_ee) - read from remote GitHub Enterprise repositories
* [Bitbucket](source/bitbucket) - read from remote Bitbucket repositories
* [Gitlab](source/gitlab) - read from remote Gitlab repositories
* [AWS S3](source/aws_s3) - read from Amazon Web Services S3
* [Google Cloud Storage](source/google_cloud_storage) - read from Google Cloud Platform Storage
@@ -79,7 +89,7 @@ Source drivers read migrations from local or remote sources. [Add a new source?]
* Handles ctrl+c (SIGINT) gracefully.
* No config search paths, no config files, no magic ENV var injections.
__[CLI Documentation](cmd/migrate)__
[CLI Documentation](cmd/migrate) (includes CLI install instructions)
### Basic usage
@@ -103,7 +113,7 @@ $ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
* Uses `io.Reader` streams internally for low memory overhead.
* Thread-safe and no goroutine leaks.
__[Go Documentation](https://godoc.org/github.com/golang-migrate/migrate)__
__[Go Documentation](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)__
```go
import (
@@ -137,7 +147,7 @@ func main() {
m, err := migrate.NewWithDatabaseInstance(
"file:///migrations",
"postgres", driver)
m.Steps(2)
m.Up() // or m.Steps(2) if you want to explicitly set the number of migrations to run
}
```
@@ -163,6 +173,11 @@ Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-file
[Best practices: How to write migrations.](MIGRATIONS.md)
## Coming from another db migration tool?
Check out [migradaptor](https://github.com/musinit/migradaptor/).
*Note: migradaptor is not affiliated or supported by this project*
## Versions
Version | Supported? | Import | Notes
+16
View File
@@ -0,0 +1,16 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| master | :white_check_mark: |
| 4.x | :white_check_mark: |
| 3.x | :x: |
| < 3.0 | :x: |
## Reporting a Vulnerability
We prefer [coordinated disclosures](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure). To start one, create a GitHub security advisory following [these instructions](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability)
Please suggest potential impact and urgency in your reports.
+22 -11
View File
@@ -7,7 +7,7 @@
[Release Downloads](https://github.com/golang-migrate/migrate/releases)
```bash
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$os-$arch.tar.gz | tar xvz
```
### MacOS
@@ -27,8 +27,8 @@ $ scoop install migrate
### Linux (*.deb package)
```bash
$ curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
$ echo "deb https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ curl -fsSL https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo gpg --dearmor -o /etc/apt/keyrings/migrate.gpg
$ echo "deb [signed-by=/etc/apt/keyrings/migrate.gpg] https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ apt-get update
$ apt-get install -y migrate
```
@@ -41,13 +41,19 @@ $ apt-get install -y migrate
$ go get -u -d github.com/golang-migrate/migrate/cmd/migrate
$ cd $GOPATH/src/github.com/golang-migrate/migrate/cmd/migrate
$ git checkout $TAG # e.g. v4.1.0
$ # Go 1.15 and below
$ go build -tags 'postgres' -ldflags="-X main.Version=$(git describe --tags)" -o $GOPATH/bin/migrate $GOPATH/src/github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.16+
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@$TAG
```
#### Unversioned
```bash
$ # Go 1.15 and below
$ go get -tags 'postgres' -u github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.16+
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
```
#### Notes
@@ -57,9 +63,10 @@ $ go get -tags 'postgres' -u github.com/golang-migrate/migrate/cmd/migrate
to build the cli for use with other databases, replace the `postgres` build tag
with the appropriate database tag(s) for the databases desired. The tags
correspond to the names of the sub-packages underneath the
[`database`](../database) package.
[`database`](../../database) package.
1. Similarly to the database build tags, if you need to support other sources, use the appropriate build tag(s).
1. Support for build constraints will be removed in the future: https://github.com/golang-migrate/migrate/issues/60
1. For versions of Go 1.15 and lower, [make sure](https://github.com/golang-migrate/migrate/pull/257#issuecomment-705249902) you're not installing the `migrate` CLI from a module. e.g. there should not be any `go.mod` files in your current directory or any directory from your current directory to the root
## Usage
@@ -79,14 +86,18 @@ Options:
-help Print usage
Commands:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string.
create [-ext E] [-dir D] [-seq] [-digits N] [-format] [-tz] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string. Note: migrations with the same time cause "duplicate migration version" error.
Use -tz option to specify the timezone that will be used when generating non-sequential migrations (defaults: UTC).
goto V Migrate to version V
up [N] Apply all or N up migrations
down [N] Apply all or N down migrations
drop Drop everything inside database
down [N] [-all] Apply all or N down migrations
Use -all to apply all down migrations
drop [-f] Drop everything inside database
Use -f to bypass confirmation
force V Set version V but don't run migration (ignores dirty state)
version Print current migration version
```
@@ -120,7 +131,7 @@ $ migrate -database "$MY_MIGRATE_DATABASE"
Check out https://stedolan.github.io/jq/
```bash
$ migrate -database "$(cat config.json | jq '.database')"
$ migrate -database "$(cat config.json | jq -r '.database')"
```
### YAML files
+10 -3
View File
@@ -1,12 +1,17 @@
# Cassandra
# Cassandra / ScyllaDB
* Drop command will not work on Cassandra 2.X because it rely on
* `Drop()` method will not work on Cassandra 2.X because it rely on
system_schema table which comes with 3.X
* Other commands should work properly but are **not tested**
* Other methods should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
**ScyllaDB**
* No additional configuration is required since it is a drop-in replacement for Cassandra.
* The `Drop()` method` works for ScyllaDB 5.1
## Usage
`cassandra://host:port/keyspace?param1=value&param2=value2`
@@ -20,12 +25,14 @@ system_schema table which comes with 3.X
| `consistency` | ALL | Migration consistency
| `protocol` | | Cassandra protocol version (3 or 4)
| `timeout` | 1 minute | Migration timeout
| `connect-timeout` | 600ms | Initial connection timeout to the cluster |
| `username` | nil | Username to use when authenticating. |
| `password` | nil | Password to use when authenticating. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
| `disable-host-lookup`| false | Disable initial host lookup. |
`timeout` is parsed using [time.ParseDuration(s string)](https://golang.org/pkg/time/#ParseDuration)
+39 -17
View File
@@ -4,16 +4,15 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/gocql/gocql"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/hashicorp/go-multierror"
)
func init() {
@@ -45,7 +44,7 @@ type Config struct {
type Cassandra struct {
session *gocql.Session
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -132,6 +131,14 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
}
cluster.Timeout = timeout
}
if len(u.Query().Get("connect-timeout")) > 0 {
var connectTimeout time.Duration
connectTimeout, err = time.ParseDuration(u.Query().Get("connect-timeout"))
if err != nil {
return nil, err
}
cluster.ConnectTimeout = connectTimeout
}
if len(u.Query().Get("sslmode")) > 0 {
if u.Query().Get("sslmode") != "disable" {
@@ -155,6 +162,14 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
}
}
if len(u.Query().Get("disable-host-lookup")) > 0 {
if flag, err := strconv.ParseBool(u.Query().Get("disable-host-lookup")); err != nil && flag {
cluster.DisableInitialHostLookup = true
} else if err != nil {
return nil, err
}
}
session, err := cluster.CreateSession()
if err != nil {
return nil, err
@@ -182,15 +197,16 @@ func (c *Cassandra) Close() error {
}
func (c *Cassandra) Lock() error {
if c.isLocked {
if !c.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
c.isLocked = true
return nil
}
func (c *Cassandra) Unlock() error {
c.isLocked = false
if !c.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
@@ -213,7 +229,7 @@ func (c *Cassandra) Run(migration io.Reader) error {
return err
}
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -226,16 +242,26 @@ func (c *Cassandra) Run(migration io.Reader) error {
}
func (c *Cassandra) SetVersion(version int, dirty bool) error {
query := `TRUNCATE "` + c.config.MigrationsTable + `"`
if err := c.session.Query(query).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
// DELETE instead of TRUNCATE because AWS Keyspaces does not support it
// see: https://docs.aws.amazon.com/keyspaces/latest/devguide/cassandra-apis.html
squery := `SELECT version FROM "` + c.config.MigrationsTable + `"`
dquery := `DELETE FROM "` + c.config.MigrationsTable + `" WHERE version = ?`
iter := c.session.Query(squery).Iter()
var previous int
for iter.Scan(&previous) {
if err := c.session.Query(dquery, previous).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(dquery)}
}
}
if err := iter.Close(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(squery)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
query := `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
if err := c.session.Query(query, version, dirty).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -288,11 +314,7 @@ func (c *Cassandra) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -314,7 +336,7 @@ func parseConsistency(consistencyStr string) (consistency gocql.Consistency, err
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("Failed to parse consistency \"%s\": %v", consistencyStr, r)
err = fmt.Errorf("failed to parse consistency \"%s\": %v", consistencyStr, r)
}
}
}()
+17 -1
View File
@@ -25,9 +25,11 @@ var (
// Although Cassandra 2.x is supported by the Apache Foundation,
// the migrate db driver only supports Cassandra 3.x since it uses
// the system_schema keyspace.
// last ScyllaDB version tested is 5.1.11
specs = []dktesting.ContainerSpec{
{ImageName: "cassandra:3.0", Options: opts},
{ImageName: "cassandra:3.11", Options: opts},
{ImageName: "scylladb/scylla:5.1.11", Options: opts},
}
)
@@ -59,6 +61,20 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
@@ -79,7 +95,7 @@ func Test(t *testing.T) {
})
}
func TestMigrate(t *testing.T) {
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
-19
View File
@@ -1,19 +0,0 @@
# ClickHouse
`clickhouse://host:port?username=user&password=qwerty&database=clicks&x-multi-statement=true`
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
| `x-multi-statement` | false | Enable multiple statements to be ran in a single migration (See note below) |
## Notes
* The Clickhouse driver does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
-260
View File
@@ -1,260 +0,0 @@
package clickhouse
import (
"database/sql"
"fmt"
"io"
"io/ioutil"
"net/url"
"strconv"
"strings"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/hashicorp/go-multierror"
)
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
ErrNilConfig = fmt.Errorf("no config")
)
type Config struct {
DatabaseName string
MigrationsTable string
MultiStatementEnabled bool
MultiStatementMaxSize int
}
func init() {
database.Register("clickhouse", &ClickHouse{})
}
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ch := &ClickHouse{
conn: conn,
config: config,
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
type ClickHouse struct {
conn *sql.DB
config *Config
}
func (ch *ClickHouse) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = "tcp"
conn, err := sql.Open("clickhouse", q.String())
if err != nil {
return nil, err
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
ch = &ClickHouse{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
DatabaseName: purl.Query().Get("database"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
MultiStatementMaxSize: multiStatementMaxSize,
},
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
func (ch *ClickHouse) init() error {
if len(ch.config.DatabaseName) == 0 {
if err := ch.conn.QueryRow("SELECT currentDatabase()").Scan(&ch.config.DatabaseName); err != nil {
return err
}
}
if len(ch.config.MigrationsTable) == 0 {
ch.config.MigrationsTable = DefaultMigrationsTable
}
if ch.config.MultiStatementMaxSize <= 0 {
ch.config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
return ch.ensureVersionTable()
}
func (ch *ClickHouse) Run(r io.Reader) error {
if ch.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(r, multiStmtDelimiter, ch.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if _, e := ch.conn.Exec(string(m)); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
return true
}); e != nil {
return e
}
return err
}
migration, err := ioutil.ReadAll(r)
if err != nil {
return err
}
if _, err := ch.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ch *ClickHouse) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ch.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ch.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ch *ClickHouse) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ch.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ch.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the ClickHouse type.
func (ch *ClickHouse) ensureVersionTable() (err error) {
if err = ch.Lock(); err != nil {
return err
}
defer func() {
if e := ch.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
var (
table string
query = "SHOW TABLES FROM " + ch.config.DatabaseName + " LIKE '" + ch.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ch.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
query = `
CREATE TABLE ` + ch.config.MigrationsTable + ` (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=TinyLog
`
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ch *ClickHouse) Drop() (err error) {
query := "SHOW TABLES FROM " + ch.config.DatabaseName
tables, err := ch.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
}
}()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + ch.config.DatabaseName + "." + table
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
return nil
}
func (ch *ClickHouse) Lock() error { return nil }
func (ch *ClickHouse) Unlock() error { return nil }
func (ch *ClickHouse) Close() error { return ch.conn.Close() }
+3 -3
View File
@@ -15,7 +15,7 @@ CREATE USER IF NOT EXISTS cockroach;
GRANT ALL ON DATABASE example TO cockroach;
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convienience:
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
```
export COCKROACHDB_URL='cockroachdb://cockroach:@localhost:26257/example?sslmode=disable'
```
@@ -50,7 +50,7 @@ And in the `.down.sql` let's delete it:
```
DROP TABLE IF EXISTS example.users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](/GETTING_STARTED.md#create-migrations)
## Run migrations
```
@@ -139,4 +139,4 @@ func main() {
}
}
```
You can find details [here](README.md#use-in-your-go-project)
You can find details [here](README.md#use-in-your-go-project)
+60 -70
View File
@@ -3,23 +3,18 @@ package cockroachdb
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
)
"sync/atomic"
import (
"github.com/cockroachdb/cockroach-go/crdb"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
import (
"github.com/cockroachdb/cockroach-go/v2/crdb"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/lib/pq"
)
func init() {
@@ -46,7 +41,7 @@ type Config struct {
type CockroachDb struct {
db *sql.DB
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -152,75 +147,71 @@ func (c *CockroachDb) Close() error {
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Lock() error {
err := crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = multierror.Append(err, errClose)
return database.CasRestoreOnErr(&c.isLocked, false, true, database.ErrLocked, func() (err error) {
return crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
return nil
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return nil
})
})
if err != nil {
return err
} else {
c.isLocked = true
return nil
}
}
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Unlock() error {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
c.isLocked = false
return nil
}
return database.CasRestoreOnErr(&c.isLocked, true, false, database.ErrNotLocked, func() (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
c.isLocked = false
return nil
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
return nil
}
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
})
}
func (c *CockroachDb) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -285,7 +276,7 @@ func (c *CockroachDb) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -300,6 +291,9 @@ func (c *CockroachDb) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -324,11 +318,7 @@ func (c *CockroachDb) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+6 -6
View File
@@ -26,13 +26,13 @@ import (
const defaultPort = 26257
var (
opts = dktest.Options{Cmd: []string{"start", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Released versions: https://www.cockroachlabs.com/docs/releases/
opts = dktest.Options{Cmd: []string{"start-single-node", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.cockroachlabs.com/docs/releases/release-support-policy#supported-versions
specs = []dktesting.ContainerSpec{
{ImageName: "cockroachdb/cockroach:v1.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v1.1.9", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.1.3", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v24.3", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v24.1", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.2", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.1", Options: opts},
}
)
+31
View File
@@ -0,0 +1,31 @@
# Datastore
`datastore://host:port?username=user&password=password&database=clicks&x-multi-statement=true`
Hanzo Datastore is a ClickHouse-wire-compatible database engine. This driver
connects through the native `github.com/hanzo-ds/go` driver, which registers the
`database/sql` driver name `datastore`.
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `x-migrations-table-engine`| Engine to use for the migrations table, defaults to TinyLog |
| `x-cluster-name` | Name of cluster for creating `schema_migrations` table cluster wide |
| `x-multi-statement` | Enable multiple statements to be ran in a single migration (See note below) |
| `x-multi-statement-max-size` | Maximum size of a single migration in bytes when `x-multi-statement` is set (defaults to 10 MB) |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
## Notes
* The Datastore driver does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
* Using the default TinyLog table engine for the `schema_migrations` table prevents backing up the table. If you need to back up the database, run the migrations with `x-migrations-table-engine=MergeTree`.
* Datastore cluster mode is not officially supported, but you can try enabling `schema_migrations` table replication by specifying a `x-cluster-name`:
* When `x-cluster-name` is specified, `x-migrations-table-engine` also should be specified.
* When `x-cluster-name` is specified, only the `schema_migrations` table is replicated across the cluster. You still need to write your migrations so that the application tables are replicated within the cluster.
* If you want to create a database inside a migration, note that the `schema_migrations` table will live in the `default` database, so you cannot use `USE <database_name>` inside a migration. In this case you may omit the database in the connection string (example [here](examples/migrations/003_create_database.up.sql)).
+337
View File
@@ -0,0 +1,337 @@
// Package datastore implements the golang-migrate database.Driver interface for
// Hanzo Datastore (a ClickHouse-wire-compatible database engine). It registers
// the "datastore://" URL scheme and connects through the native
// github.com/hanzo-ds/go driver, which registers the database/sql driver name
// "datastore".
//
// Usage:
//
// import _ "github.com/golang-migrate/migrate/v4/database/datastore"
//
// Then use a URL like:
//
// datastore://host:9000?username=default&database=default&x-multi-statement=true
package datastore
import (
"database/sql"
"errors"
"fmt"
"io"
"net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
var (
multiStmtDelimiter = []byte(";")
// DriverName is the database/sql driver registered by github.com/hanzo-ds/go.
DriverName = "datastore"
DefaultMigrationsTable = "schema_migrations"
DefaultMigrationsTableEngine = "TinyLog"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
ErrNilConfig = fmt.Errorf("no config")
)
type Config struct {
DatabaseName string
ClusterName string
MigrationsTable string
MigrationsTableEngine string
MultiStatementEnabled bool
MultiStatementMaxSize int
}
func init() {
database.Register("datastore", &Datastore{})
}
// WithInstance creates a Datastore driver from an existing *sql.DB opened
// against the "datastore" driver.
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ds := &Datastore{
conn: conn,
config: config,
}
if err := ds.init(); err != nil {
return nil, err
}
return ds, nil
}
// Datastore is a golang-migrate database.Driver backed by the native
// github.com/hanzo-ds/go ClickHouse-wire driver.
type Datastore struct {
conn *sql.DB
config *Config
isLocked atomic.Bool
}
func (ds *Datastore) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = DriverName
conn, err := sql.Open(DriverName, q.String())
if err != nil {
return nil, err
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
migrationsTableEngine := DefaultMigrationsTableEngine
if s := purl.Query().Get("x-migrations-table-engine"); len(s) > 0 {
migrationsTableEngine = s
}
ds = &Datastore{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
MigrationsTableEngine: migrationsTableEngine,
DatabaseName: purl.Query().Get("database"),
ClusterName: purl.Query().Get("x-cluster-name"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
MultiStatementMaxSize: multiStatementMaxSize,
},
}
if err := ds.init(); err != nil {
return nil, err
}
return ds, nil
}
func (ds *Datastore) init() error {
if len(ds.config.DatabaseName) == 0 {
if err := ds.conn.QueryRow("SELECT currentDatabase()").Scan(&ds.config.DatabaseName); err != nil {
return err
}
}
if len(ds.config.MigrationsTable) == 0 {
ds.config.MigrationsTable = DefaultMigrationsTable
}
if ds.config.MultiStatementMaxSize <= 0 {
ds.config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
if len(ds.config.MigrationsTableEngine) == 0 {
ds.config.MigrationsTableEngine = DefaultMigrationsTableEngine
}
return ds.ensureVersionTable()
}
func (ds *Datastore) Run(r io.Reader) error {
if ds.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(r, multiStmtDelimiter, ds.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if _, e := ds.conn.Exec(string(m)); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
return true
}); e != nil {
return e
}
return err
}
migration, err := io.ReadAll(r)
if err != nil {
return err
}
if _, err := ds.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ds *Datastore) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ds.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ds.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ds *Datastore) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ds.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ds.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Datastore type.
func (ds *Datastore) ensureVersionTable() (err error) {
if err = ds.Lock(); err != nil {
return err
}
defer func() {
if e := ds.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
var (
table string
query = "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName) + " LIKE '" + ds.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ds.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
if len(ds.config.ClusterName) > 0 {
query = fmt.Sprintf(`
CREATE TABLE %s ON CLUSTER %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.ClusterName, ds.config.MigrationsTableEngine)
} else {
query = fmt.Sprintf(`
CREATE TABLE %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.MigrationsTableEngine)
}
if strings.HasSuffix(ds.config.MigrationsTableEngine, "Tree") {
query = fmt.Sprintf(`%s ORDER BY sequence`, query)
}
if _, err := ds.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Drop() (err error) {
query := "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName)
tables, err := ds.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + quoteIdentifier(ds.config.DatabaseName) + "." + quoteIdentifier(table)
if _, err := ds.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Lock() error {
if !ds.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
func (ds *Datastore) Unlock() error {
if !ds.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (ds *Datastore) Close() error { return ds.conn.Close() }
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+226
View File
@@ -0,0 +1,226 @@
package datastore_test
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"fmt"
"log"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/datastore"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
const defaultPort = 9000
var (
tableEngines = []string{"TinyLog", "MergeTree"}
opts = dktest.Options{
Env: map[string]string{"CLICKHOUSE_USER": "user", "CLICKHOUSE_PASSWORD": "password", "CLICKHOUSE_DB": "db"},
PortRequired: true, ReadyFunc: isReady,
}
specs = []dktesting.ContainerSpec{
{ImageName: "clickhouse:24.8", Options: opts},
}
)
func datastoreConnectionString(host, port, engine string) string {
if engine != "" {
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&x-migrations-table-engine=%v&debug=false",
host, port, engine)
}
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&debug=false",
host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
return false
}
db, err := sql.Open("datastore", datastoreConnectionString(ip, port, ""))
if err != nil {
log.Println("open error", err)
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn:
return false
default:
fmt.Println(err)
}
return false
}
return true
}
func TestCases(t *testing.T) {
for _, engine := range tableEngines {
t.Run("Test_"+engine, func(t *testing.T) { testSimple(t, engine) })
t.Run("Migrate_"+engine, func(t *testing.T) { testMigrate(t, engine) })
t.Run("Version_"+engine, func(t *testing.T) { testVersion(t, engine) })
t.Run("Drop_"+engine, func(t *testing.T) { testDrop(t, engine) })
}
t.Run("WithInstanceDefaultConfigValues", func(t *testing.T) { testSimpleWithInstanceDefaultConfigValues(t) })
}
func testSimple(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testSimpleWithInstanceDefaultConfigValues(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, "")
conn, err := sql.Open("datastore", addr)
if err != nil {
t.Fatal(err)
}
d, err := datastore.WithInstance(conn, &datastore.Config{})
if err != nil {
_ = conn.Close()
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testMigrate(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "db", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func testVersion(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
expectedVersion := 1
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
err = d.SetVersion(expectedVersion, false)
if err != nil {
t.Fatal(err)
}
version, _, err := d.Version()
if err != nil {
t.Fatal(err)
}
if version != expectedVersion {
t.Fatal("Version mismatch")
}
})
}
func testDrop(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
err = d.Drop()
if err != nil {
t.Fatal(err)
}
})
}
@@ -1,3 +1,3 @@
CREATE TABLE test_1 (
Date Date
Date Date
) Engine=Memory;
@@ -1,3 +1,3 @@
CREATE TABLE test_2 (
Date Date
Date Date
) Engine=Memory;
@@ -0,0 +1,10 @@
DROP TABLE IF EXISTS driver_ratings;
DROP TABLE IF EXISTS user_ratings;
DROP TABLE IF EXISTS orders;
DROP TABLE IF EXISTS driver_ratings_queue;
DROP TABLE IF EXISTS user_ratings_queue;
DROP TABLE IF EXISTS orders_queue;
DROP VIEW IF EXISTS user_ratings_queue_mv;
DROP VIEW IF EXISTS driver_ratings_queue_mv;
DROP VIEW IF EXISTS orders_queue_mv;
DROP DATABASE IF EXISTS analytics;
@@ -0,0 +1,81 @@
CREATE DATABASE IF NOT EXISTS analytics;
CREATE TABLE IF NOT EXISTS analytics.driver_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.driver_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'driver-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.driver_ratings_queue_mv TO analytics.driver_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.driver_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.user_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY userID
ORDER BY (inserted_time);
CREATE TABLE analytics.user_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'user-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'JSON',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.user_ratings_queue_mv TO analytics.user_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.user_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.orders(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.orders_queue(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'orders',
kafka_group_name = 'order_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.orders_queue_mv TO orders AS
SELECT from_place, to_place, userID, driverID, orderID
FROM analytics.orders_queue;
+14 -14
View File
@@ -1,4 +1,4 @@
// Package database provides the Database interface.
// Package database provides the Driver interface.
// All database drivers must implement this interface, register themselves,
// optionally provide a `WithInstance` function and pass the tests
// in package database/testing.
@@ -25,22 +25,22 @@ var drivers = make(map[string]Driver)
// Driver is the interface every database driver must implement.
//
// How to implement a database driver?
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
//
// Guidelines:
// * Don't try to correct user input. Don't assume things.
// - Don't try to correct user input. Don't assume things.
// When in doubt, return an error and explain the situation to the user.
// * All configuration input must come from the URL string in func Open()
// - All configuration input must come from the URL string in func Open()
// or the Config{} struct in WithInstance. Don't os.Getenv().
type Driver interface {
// Open returns a new driver instance configured with parameters
+17 -16
View File
@@ -1,18 +1,19 @@
// +build go1.9
//go:build go1.9
package firebird
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/nakagami/firebirdsql"
"io"
"io/ioutil"
nurl "net/url"
)
func init() {
@@ -36,7 +37,7 @@ type Firebird struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -106,20 +107,21 @@ func (f *Firebird) Close() error {
}
func (f *Firebird) Lock() error {
if f.isLocked {
if !f.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
f.isLocked = true
return nil
}
func (f *Firebird) Unlock() error {
f.isLocked = false
if !f.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (f *Firebird) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -178,7 +180,7 @@ func (f *Firebird) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -193,6 +195,9 @@ func (f *Firebird) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// delete one by one ...
for _, t := range tableNames {
@@ -218,11 +223,7 @@ func (f *Firebird) ensureVersionTable() (err error) {
defer func() {
if e := f.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+3 -2
View File
@@ -38,8 +38,9 @@ var (
},
}
specs = []dktesting.ContainerSpec{
{ImageName: "jacobalberty/firebird:2.5-ss", Options: opts},
{ImageName: "jacobalberty/firebird:3.0", Options: opts},
{ImageName: "jacobalberty/firebird:v3.0", Options: opts},
{ImageName: "jacobalberty/firebird:v4.0", Options: opts},
{ImageName: "jacobalberty/firebird:v5.0", Options: opts},
}
)
+2 -2
View File
@@ -15,8 +15,8 @@
| `x-transaction-mode` | `TransactionMode` | If set to `true` wrap commands in [transaction](https://docs.mongodb.com/manual/core/transactions). Available only for replica set. Driver is using [strconv.ParseBool](https://golang.org/pkg/strconv/#ParseBool) for parsing|
| `x-advisory-locking` | `true` | Feature flag for advisory locking, if set to false, disable advisory locking |
| `x-advisory-lock-collection` | `migrate_advisory_lock` | The name of the collection to use for advisory locking.|
| `x-advisory-lock-timout` | `15` | The max time in seconds that the advisory lock will wait if the db is already locked. |
| `x-advisory-lock-timout-interval` | `10` | The max timeout in seconds interval that the advisory lock will wait if the db is already locked. |
| `x-advisory-lock-timeout` | `15` | The max time in seconds that migrate will wait to acquire a lock before failing. |
| `x-advisory-lock-timeout-interval` | `10` | The max time in seconds between attempts to acquire the advisory lock, the lock is attempted to be acquired using an exponential backoff algorithm. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as. Can be omitted |
| `password` | | The user's password. Can be omitted |
+91 -72
View File
@@ -2,20 +2,21 @@ package mongodb
import (
"context"
"errors"
"fmt"
"io"
"net/url"
"os"
"strconv"
"sync/atomic"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/x/mongo/driver/connstring"
"io"
"io/ioutil"
"net/url"
os "os"
"strconv"
"time"
)
func init() {
@@ -35,14 +36,16 @@ const LockIndexName = "lock_unique_key" // the name of the inde
const contextWaitTimeout = 5 * time.Second // how long to wait for the request to mongo to block/wait for.
var (
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
ErrLockTimeoutConfigConflict = fmt.Errorf("both x-advisory-lock-timeout-interval and x-advisory-lock-timout-interval were specified")
)
type Mongo struct {
client *mongo.Client
db *mongo.Database
config *Config
client *mongo.Client
db *mongo.Database
config *Config
isLocked atomic.Bool
}
type Locking struct {
@@ -111,7 +114,7 @@ func WithInstance(instance *mongo.Client, config *Config) (database.Driver, erro
}
func (m *Mongo) Open(dsn string) (database.Driver, error) {
//connstring is experimental package, but it used for parse connection string in mongo.Connect function
// connstring is experimental package, but it used for parse connection string in mongo.Connect function
uri, err := connstring.Parse(dsn)
if err != nil {
return nil, err
@@ -135,7 +138,22 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
if err != nil {
return nil, err
}
maxLockingIntervals, err := parseInt(unknown.Get("x-advisory-lock-timout-interval"), DefaultLockTimeoutInterval)
lockTimeoutIntervalValue := unknown.Get("x-advisory-lock-timeout-interval")
// The initial release had a typo for this argument but for backwards compatibility sake, we will keep supporting it
// and we will error out if both values are set.
lockTimeoutIntervalValueFromTypo := unknown.Get("x-advisory-lock-timout-interval")
lockTimeout := lockTimeoutIntervalValue
if lockTimeoutIntervalValue != "" && lockTimeoutIntervalValueFromTypo != "" {
return nil, ErrLockTimeoutConfigConflict
} else if lockTimeoutIntervalValueFromTypo != "" {
lockTimeout = lockTimeoutIntervalValueFromTypo
}
maxLockCheckInterval, err := parseInt(lockTimeout, DefaultLockTimeoutInterval)
if err != nil {
return nil, err
}
@@ -155,7 +173,7 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
CollectionName: lockCollection,
Timeout: lockingTimout,
Enabled: advisoryLockingFlag,
Interval: maxLockingIntervals,
Interval: maxLockCheckInterval,
},
})
if err != nil {
@@ -164,7 +182,7 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
return mc, nil
}
//Parse the url param, convert it to boolean
// Parse the url param, convert it to boolean
// returns error if param invalid. returns defaultValue if param not present
func parseBoolean(urlParam string, defaultValue bool) (bool, error) {
@@ -181,7 +199,7 @@ func parseBoolean(urlParam string, defaultValue bool) (bool, error) {
return defaultValue, nil
}
//Parse the url param, convert it to int
// Parse the url param, convert it to int
// returns error if param invalid. returns defaultValue if param not present
func parseInt(urlParam string, defaultValue int) (int, error) {
@@ -223,7 +241,7 @@ func (m *Mongo) Version() (version int, dirty bool, err error) {
}
func (m *Mongo) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -250,8 +268,8 @@ func (m *Mongo) executeCommandsWithTransaction(ctx context.Context, cmds []bson.
return &database.Error{OrigErr: err, Err: "failed to start transaction"}
}
if err := m.executeCommands(sessionContext, cmds); err != nil {
//When command execution is failed, it's aborting transaction
//If you tried to call abortTransaction, it`s return error that transaction already aborted
// When command execution is failed, it's aborting transaction
// If you tried to call abortTransaction, it`s return error that transaction already aborted
return err
}
if err := sessionContext.CommitTransaction(sessionContext); err != nil {
@@ -307,11 +325,7 @@ func (m *Mongo) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -327,55 +341,60 @@ func (m *Mongo) ensureVersionTable() (err error) {
// Utilizes advisory locking on the config.LockingCollection collection
// This uses a unique index on the `locking_key` field.
func (m *Mongo) Lock() error {
if !m.config.Locking.Enabled {
return database.CasRestoreOnErr(&m.isLocked, false, true, database.ErrLocked, func() error {
if !m.config.Locking.Enabled {
return nil
}
pid := os.Getpid()
hostname, err := os.Hostname()
if err != nil {
hostname = fmt.Sprintf("Could not determine hostname. Error: %s", err.Error())
}
newLockObj := lockObj{
Key: lockKeyUniqueValue,
Pid: pid,
Hostname: hostname,
CreatedAt: time.Now(),
}
operation := func() error {
timeout, cancelFunc := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).InsertOne(timeout, newLockObj)
defer cancelFunc()
return err
}
exponentialBackOff := backoff.NewExponentialBackOff()
duration := time.Duration(m.config.Locking.Timeout) * time.Second
exponentialBackOff.MaxElapsedTime = duration
exponentialBackOff.MaxInterval = time.Duration(m.config.Locking.Interval) * time.Second
err = backoff.Retry(operation, exponentialBackOff)
if err != nil {
return database.ErrLocked
}
return nil
}
pid := os.Getpid()
hostname, err := os.Hostname()
if err != nil {
hostname = fmt.Sprintf("Could not determine hostname. Error: %s", err.Error())
}
newLockObj := lockObj{
Key: lockKeyUniqueValue,
Pid: pid,
Hostname: hostname,
CreatedAt: time.Now(),
}
operation := func() error {
timeout, cancelFunc := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).InsertOne(timeout, newLockObj)
defer cancelFunc()
return err
}
exponentialBackOff := backoff.NewExponentialBackOff()
duration := time.Duration(m.config.Locking.Timeout) * time.Second
exponentialBackOff.MaxElapsedTime = duration
exponentialBackOff.MaxInterval = time.Duration(m.config.Locking.Interval) * time.Second
err = backoff.Retry(operation, exponentialBackOff)
if err != nil {
return database.ErrLocked
}
return nil
})
}
func (m *Mongo) Unlock() error {
if !m.config.Locking.Enabled {
return database.CasRestoreOnErr(&m.isLocked, true, false, database.ErrNotLocked, func() error {
if !m.config.Locking.Enabled {
return nil
}
filter := findFilter{
Key: lockKeyUniqueValue,
}
ctx, cancel := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).DeleteMany(ctx, filter)
defer cancel()
if err != nil {
return err
}
return nil
}
filter := findFilter{
Key: lockKeyUniqueValue,
}
ctx, cancel := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).DeleteMany(ctx, filter)
defer cancel()
if err != nil {
return err
}
return nil
})
}
+33 -19
View File
@@ -32,10 +32,10 @@ var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.mongodb.com/support-policy
specs = []dktesting.ContainerSpec{
{ImageName: "mongo:3.4", Options: opts},
{ImageName: "mongo:3.6", Options: opts},
{ImageName: "mongo:4.0", Options: opts},
{ImageName: "mongo:4.2", Options: opts},
{ImageName: "mongo:5.0", Options: opts},
{ImageName: "mongo:6.0", Options: opts},
{ImageName: "mongo:7.0", Options: opts},
{ImageName: "mongo:8.0", Options: opts},
}
)
@@ -74,6 +74,22 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testWithAuth", testWithAuth)
t.Run("testLockWorks", testLockWorks)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -99,7 +115,7 @@ func Test(t *testing.T) {
})
}
func TestMigrate(t *testing.T) {
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -125,7 +141,7 @@ func TestMigrate(t *testing.T) {
})
}
func TestWithAuth(t *testing.T) {
func testWithAuth(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -180,7 +196,7 @@ func TestWithAuth(t *testing.T) {
})
}
func TestLockWorks(t *testing.T) {
func testLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -221,18 +237,7 @@ func TestLockWorks(t *testing.T) {
t.Fatal(err)
}
// disable locking, validate wer can lock twice
mc.config.Locking.Enabled = false
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
// re-enable locking,
// enable locking,
//try to hit a lock conflict
mc.config.Locking.Enabled = true
mc.config.Locking.Timeout = 1
@@ -252,6 +257,15 @@ func TestTransaction(t *testing.T) {
{ImageName: "mongo:4", Options: dktest.Options{PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"mongod", "--bind_ip_all", "--replSet", "rs0"}}},
}
t.Cleanup(func() {
for _, spec := range transactionSpecs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
dktesting.ParallelTest(t, transactionSpecs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
+5 -4
View File
@@ -6,6 +6,7 @@
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-no-lock` | `NoLock` | Set to `true` to skip `GET_LOCK`/`RELEASE_LOCK` statements. Useful for [multi-master MySQL flavors](https://www.percona.com/doc/percona-xtradb-cluster/LATEST/features/pxc-strict-mode.html#explicit-table-locking). Only run migrations from one host when this is enabled. |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds, functionally similar to [Server-side SELECT statement timeouts](https://dev.mysql.com/blog-archive/server-side-select-statement-timeouts/) but enforced by the client. Available for all versions of MySQL, not just >=5.7. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
@@ -13,7 +14,7 @@
| `port` | | The port to bind to. |
| `tls` | | TLS / SSL encrypted connection parameter; see [go-sql-driver](https://github.com/go-sql-driver/mysql#tls). Use any name (e.g. `migrate`) if you want to use a custom TLS config (`x-tls-` queries). |
| `x-tls-ca` | | The location of the CA (certificate authority) file. |
| `x-tls-cert` | | The location of the client certicicate file. Must be used with `x-tls-key`. |
| `x-tls-cert` | | The location of the client certificate file. Must be used with `x-tls-key`. |
| `x-tls-key` | | The location of the private key file. Must be used with `x-tls-cert`. |
| `x-tls-insecure-skip-verify` | | Whether or not to use SSL (true\|false) |
@@ -28,9 +29,9 @@ import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate"
"github.com/golang-migrate/migrate/database/mysql"
_ "github.com/golang-migrate/migrate/source/file"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/mysql"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
+183 -138
View File
@@ -1,4 +1,4 @@
// +build go1.9
//go:build go1.9
package mysql
@@ -7,23 +7,22 @@ import (
"crypto/tls"
"crypto/x509"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"os"
"strconv"
"strings"
)
"sync/atomic"
"time"
import (
"github.com/go-sql-driver/mysql"
"github.com/hashicorp/go-multierror"
)
import (
"github.com/golang-migrate/migrate/v4/database"
)
var _ database.Driver = (*Mysql)(nil) // explicit compile time type check
func init() {
database.Register("mysql", &Mysql{})
}
@@ -35,13 +34,14 @@ var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrAppendPEM = fmt.Errorf("failed to append PEM")
ErrTLSCertKeyConfig = fmt.Errorf("To use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
ErrTLSCertKeyConfig = fmt.Errorf("to use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
)
type Config struct {
MigrationsTable string
DatabaseName string
NoLock bool
MigrationsTable string
DatabaseName string
NoLock bool
StatementTimeout time.Duration
}
type Mysql struct {
@@ -49,25 +49,31 @@ type Mysql struct {
// just do everything over a single conn anyway.
conn *sql.Conn
db *sql.DB
isLocked bool
isLocked atomic.Bool
config *Config
}
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
// connection instance must have `multiStatements` set to true
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Mysql, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
if err := conn.PingContext(ctx); err != nil {
return nil, err
}
mx := &Mysql{
conn: conn,
db: nil,
config: config,
}
if config.DatabaseName == "" {
query := `SELECT DATABASE()`
var databaseName sql.NullString
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -82,21 +88,33 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err := mx.ensureVersionTable(); err != nil {
return nil, err
}
return mx, nil
}
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
mx := &Mysql{
conn: conn,
db: instance,
config: config,
}
if err := mx.ensureVersionTable(); err != nil {
mx, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
mx.db = instance
return mx, nil
}
@@ -118,6 +136,68 @@ func extractCustomQueryParams(c *mysql.Config) (map[string]string, error) {
}
func urlToMySQLConfig(url string) (*mysql.Config, error) {
// Need to parse out custom TLS parameters and call
// mysql.RegisterTLSConfig() before mysql.ParseDSN() is called
// which consumes the registered tls.Config
// Fixes: https://github.com/golang-migrate/migrate/issues/411
//
// Can't use url.Parse() since it fails to parse MySQL DSNs
// mysql.ParseDSN() also searches for "?" to find query parameters:
// https://github.com/go-sql-driver/mysql/blob/46351a8/dsn.go#L344
if idx := strings.LastIndex(url, "?"); idx > 0 {
rawParams := url[idx+1:]
parsedParams, err := nurl.ParseQuery(rawParams)
if err != nil {
return nil, err
}
ctls := parsedParams.Get("tls")
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := os.ReadFile(parsedParams.Get("x-tls-ca"))
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := parsedParams.Get("x-tls-cert"), parsedParams.Get("x-tls-key"); ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
insecureSkipVerifyStr := parsedParams.Get("x-tls-insecure-skip-verify")
if len(insecureSkipVerifyStr) > 0 {
x, err := strconv.ParseBool(insecureSkipVerifyStr)
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
err = mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
if err != nil {
return nil, err
}
}
}
}
config, err := mysql.ParseDSN(strings.TrimPrefix(url, "mysql://"))
if err != nil {
return nil, err
@@ -140,52 +220,6 @@ func urlToMySQLConfig(url string) (*mysql.Config, error) {
}
config.Passwd = password
// use custom TLS?
ctls := config.TLSConfig
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := ioutil.ReadFile(config.Params["x-tls-ca"])
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := config.Params["x-tls-cert"], config.Params["x-tls-key"]; ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
if len(config.Params["x-tls-insecure-skip-verify"]) > 0 {
x, err := strconv.ParseBool(config.Params["x-tls-insecure-skip-verify"])
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
err = mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
if err != nil {
return nil, err
}
}
}
return config, nil
}
@@ -208,15 +242,25 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
}
}
statementTimeoutParam := customParams["x-statement-timeout"]
statementTimeout := 0
if statementTimeoutParam != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutParam)
if err != nil {
return nil, fmt.Errorf("could not parse x-statement-timeout as float: %w", err)
}
}
db, err := sql.Open("mysql", config.FormatDSN())
if err != nil {
return nil, err
}
mx, err := WithInstance(db, &Config{
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
NoLock: noLock,
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
NoLock: noLock,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
})
if err != nil {
return nil, err
@@ -227,7 +271,11 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
func (m *Mysql) Close() error {
connErr := m.conn.Close()
dbErr := m.db.Close()
var dbErr error
if m.db != nil {
dbErr = m.db.Close()
}
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -235,72 +283,70 @@ func (m *Mysql) Close() error {
}
func (m *Mysql) Lock() error {
if m.isLocked {
return database.ErrLocked
}
return database.CasRestoreOnErr(&m.isLocked, false, true, database.ErrLocked, func() error {
if m.config.NoLock {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
if !success {
return database.ErrLocked
}
if m.config.NoLock {
m.isLocked = true
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
if success {
m.isLocked = true
return nil
}
return database.ErrLocked
})
}
func (m *Mysql) Unlock() error {
if !m.isLocked {
return database.CasRestoreOnErr(&m.isLocked, true, false, database.ErrNotLocked, func() error {
if m.config.NoLock {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
return nil
}
if m.config.NoLock {
m.isLocked = false
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
m.isLocked = false
return nil
})
}
func (m *Mysql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
ctx := context.Background()
if m.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, m.config.StatementTimeout)
defer cancel()
}
query := string(migr[:])
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
if _, err := m.conn.ExecContext(ctx, query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
@@ -308,15 +354,15 @@ func (m *Mysql) Run(migration io.Reader) error {
}
func (m *Mysql) SetVersion(version int, dirty bool) error {
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{})
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "TRUNCATE `" + m.config.MigrationsTable + "`"
query := "DELETE FROM `" + m.config.MigrationsTable + "` LIMIT 1"
if _, err := tx.ExecContext(context.Background(), query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -328,7 +374,7 @@ func (m *Mysql) SetVersion(version int, dirty bool) error {
query := "INSERT INTO `" + m.config.MigrationsTable + "` (version, dirty) VALUES (?, ?)"
if _, err := tx.ExecContext(context.Background(), query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -370,7 +416,7 @@ func (m *Mysql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -385,6 +431,9 @@ func (m *Mysql) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// disable checking foreign key constraints until finished
@@ -420,17 +469,13 @@ func (m *Mysql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
// check if migration table exists
var result string
query := `SHOW TABLES LIKE "` + m.config.MigrationsTable + `"`
query := `SHOW TABLES LIKE '` + m.config.MigrationsTable + `'`
if err := m.conn.QueryRowContext(context.Background(), query).Scan(&result); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
+100 -13
View File
@@ -2,26 +2,28 @@ package mysql
import (
"context"
"crypto/ed25519"
"crypto/x509"
"database/sql"
sqldriver "database/sql/driver"
"encoding/pem"
"errors"
"fmt"
"log"
"math/big"
"math/rand"
"net/url"
"os"
"strconv"
"testing"
)
import (
"github.com/dhui/dktest"
"github.com/go-sql-driver/mysql"
"github.com/stretchr/testify/assert"
)
import (
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/stretchr/testify/assert"
)
const defaultPort = 3306
@@ -31,12 +33,21 @@ var (
Env: map[string]string{"MYSQL_ROOT_PASSWORD": "root", "MYSQL_DATABASE": "public"},
PortRequired: true, ReadyFunc: isReady,
}
optsAnsiQuotes = dktest.Options{
Env: map[string]string{"MYSQL_ROOT_PASSWORD": "root", "MYSQL_DATABASE": "public"},
PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"--sql-mode=ANSI_QUOTES"},
}
// Supported versions: https://www.mysql.com/support/supportedplatforms/database.html
specs = []dktesting.ContainerSpec{
{ImageName: "mysql:5.5", Options: opts},
{ImageName: "mysql:5.6", Options: opts},
{ImageName: "mysql:5.7", Options: opts},
{ImageName: "mysql:8", Options: opts},
{ImageName: "mysql:8.0", Options: opts},
{ImageName: "mysql:8.4", Options: opts},
{ImageName: "mysql:9.0", Options: opts},
}
specsAnsiQuotes = []dktesting.ContainerSpec{
{ImageName: "mysql:8.0", Options: optsAnsiQuotes},
{ImageName: "mysql:8.4", Options: optsAnsiQuotes},
{ImageName: "mysql:9.0", Options: optsAnsiQuotes},
}
)
@@ -69,7 +80,7 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -102,7 +113,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -139,6 +150,44 @@ func TestMigrate(t *testing.T) {
})
}
func TestMigrateAnsiQuotes(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specsAnsiQuotes, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "public", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
// check ensureVersionTable
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
// check again
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
})
}
func TestLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -284,7 +333,42 @@ func TestExtractCustomQueryParams(t *testing.T) {
}
}
func createTmpCert(t *testing.T) string {
tmpCertFile, err := os.CreateTemp("", "migrate_test_cert")
if err != nil {
t.Fatal("Failed to create temp cert file:", err)
}
t.Cleanup(func() {
if err := os.Remove(tmpCertFile.Name()); err != nil {
t.Log("Failed to cleanup temp cert file:", err)
}
})
r := rand.New(rand.NewSource(0))
pub, priv, err := ed25519.GenerateKey(r)
if err != nil {
t.Fatal("Failed to generate ed25519 key for temp cert file:", err)
}
tmpl := x509.Certificate{
SerialNumber: big.NewInt(0),
}
derBytes, err := x509.CreateCertificate(r, &tmpl, &tmpl, pub, priv)
if err != nil {
t.Fatal("Failed to generate temp cert file:", err)
}
if err := pem.Encode(tmpCertFile, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}); err != nil {
t.Fatal("Failed to encode ")
}
if err := tmpCertFile.Close(); err != nil {
t.Fatal("Failed to close temp cert file:", err)
}
return tmpCertFile.Name()
}
func TestURLToMySQLConfig(t *testing.T) {
tmpCertFilename := createTmpCert(t)
tmpCertFilenameEscaped := url.PathEscape(tmpCertFilename)
testcases := []struct {
name string
urlStr string
@@ -305,7 +389,7 @@ func TestURLToMySQLConfig(t *testing.T) {
// Not supported yet: https://github.com/go-sql-driver/mysql/issues/591
// {name: "user/password - user with encoded :",
// urlStr: "mysql://username%3A:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
// expectedDSN: "username::pasword@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
// expectedDSN: "username::password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - user with encoded @",
urlStr: "mysql://username%40:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
@@ -315,6 +399,9 @@ func TestURLToMySQLConfig(t *testing.T) {
{name: "user/password - password with encoded @",
urlStr: "mysql://username:password%40@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:password@@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "custom tls",
urlStr: "mysql://username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true&tls=custom&x-tls-ca=" + tmpCertFilenameEscaped,
expectedDSN: "username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true&tls=custom&x-tls-ca=" + tmpCertFilenameEscaped},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
+7 -8
View File
@@ -2,16 +2,15 @@ package neo4j
import (
"bytes"
"errors"
"fmt"
"io"
"io/ioutil"
neturl "net/url"
"strconv"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/hashicorp/go-multierror"
"github.com/neo4j/neo4j-go-driver/neo4j"
)
@@ -142,7 +141,7 @@ func (n *Neo4j) Run(migration io.Reader) (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = multierror.Append(err, cerr)
err = errors.Join(err, cerr)
}
}()
@@ -173,7 +172,7 @@ func (n *Neo4j) Run(migration io.Reader) (err error) {
return err
}
body, err := ioutil.ReadAll(migration)
body, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -189,7 +188,7 @@ func (n *Neo4j) SetVersion(version int, dirty bool) (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = multierror.Append(err, cerr)
err = errors.Join(err, cerr)
}
}()
@@ -214,7 +213,7 @@ func (n *Neo4j) Version() (version int, dirty bool, err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = multierror.Append(err, cerr)
err = errors.Join(err, cerr)
}
}()
@@ -262,7 +261,7 @@ func (n *Neo4j) Drop() (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = multierror.Append(err, cerr)
err = errors.Join(err, cerr)
}
}()
@@ -279,7 +278,7 @@ func (n *Neo4j) ensureVersionConstraint() (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = multierror.Append(err, cerr)
err = errors.Join(err, cerr)
}
}()
+9 -8
View File
@@ -8,13 +8,15 @@ import (
)
const reservedChars = "!#$%&'()*+,/:;=?@[]"
const reservedCharTestNamePrefix = "reserved char "
const baseUsername = "username"
const scheme = "database://"
// TestUserUnencodedReservedURLChars documents the behavior of using unencoded reserved characters in usernames with
// net/url Parse()
func TestUserUnencodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
@@ -64,7 +66,7 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run("reserved char "+tc.char, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
s := scheme + baseUsername + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -98,13 +100,12 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
}
func TestUserEncodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
for _, c := range reservedChars {
c := string(c)
t.Run("reserved char "+c, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := scheme + baseUsername + encodedChar + urlSuffixAndSep
expectedUsername := baseUsername + c
@@ -126,7 +127,7 @@ func TestUserEncodedReservedURLChars(t *testing.T) {
// with net/url Parse()
func TestPasswordUnencodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := "database://" + username + ":"
schemeAndUsernameAndSep := scheme + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
@@ -174,7 +175,7 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run("reserved char "+tc.char, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
s := schemeAndUsernameAndSep + basePassword + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -213,13 +214,13 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
func TestPasswordEncodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := "database://" + username + ":"
schemeAndUsernameAndSep := scheme + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
for _, c := range reservedChars {
c := string(c)
t.Run("reserved char "+c, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := schemeAndUsernameAndSep + basePassword + encodedChar + urlSuffixAndSep
expectedPassword := basePassword + c
+43
View File
@@ -0,0 +1,43 @@
# pgx
This package is for [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). A backend for the newer [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5) is [also available](v5).
`pgx://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `x-lock-strategy` | `LockStrategy` | Strategy used for locking during migration (default: advisory) |
| `x-lock-table` | `LockTable` | Name of the table which maintains the migration lock (default: schema_lock) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS users;
@@ -0,0 +1,5 @@
CREATE TABLE users (
user_id integer unique,
name varchar(40),
email varchar(40)
);
@@ -0,0 +1 @@
ALTER TABLE users DROP COLUMN IF EXISTS city;
@@ -0,0 +1,3 @@
ALTER TABLE users ADD COLUMN city varchar(100);
@@ -0,0 +1 @@
DROP INDEX IF EXISTS users_email_index;
@@ -0,0 +1,3 @@
CREATE UNIQUE INDEX CONCURRENTLY users_email_index ON users (email);
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS books;
@@ -0,0 +1,5 @@
CREATE TABLE books (
user_id integer,
name varchar(40),
author varchar(40)
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS movies;
@@ -0,0 +1,5 @@
CREATE TABLE movies (
user_id integer,
name varchar(40),
director varchar(40)
);
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
+617
View File
@@ -0,0 +1,617 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgconn"
"github.com/jackc/pgerrcode"
_ "github.com/jackc/pgx/v4/stdlib"
"github.com/lib/pq"
)
const (
LockStrategyAdvisory = "advisory"
LockStrategyTable = "table"
)
func init() {
db := Postgres{}
database.Register("pgx", &db)
database.Register("pgx4", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
DefaultLockTable = "schema_lock"
DefaultLockStrategy = LockStrategyAdvisory
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
ErrDatabaseDirty = fmt.Errorf("database is dirty")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
LockTable string
LockStrategy string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if len(config.LockTable) == 0 {
config.LockTable = DefaultLockTable
}
if len(config.LockStrategy) == 0 {
config.LockStrategy = DefaultLockStrategy
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureLockTable(); err != nil {
return nil, err
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v4", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
lockStrategy := purl.Query().Get("x-lock-strategy")
lockTable := purl.Query().Get("x-lock-table")
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
LockStrategy: lockStrategy,
LockTable: lockTable,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.applyAdvisoryLock()
case LockStrategyTable:
return p.applyTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
}
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.releaseAdvisoryLock()
case LockStrategyTable:
return p.releaseTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
}
})
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) applyAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
}
func (p *Postgres) applyTableLock() error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
defer func() {
errRollback := tx.Rollback()
if errRollback != nil {
err = errors.Join(err, errRollback)
}
}()
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked {
return database.ErrLocked
}
query = "INSERT INTO " + pq.QuoteIdentifier(p.config.LockTable) + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return tx.Commit()
}
func (p *Postgres) releaseAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) releaseTableLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "DELETE FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
if _, err := p.db.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
// do not drop lock table
if tableName == p.config.LockTable && p.config.LockStrategy == LockStrategyTable {
continue
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) ensureLockTable() error {
if p.config.LockStrategy != LockStrategyTable {
return nil
}
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := p.db.QueryRow(query, p.config.LockTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE ` + pq.QuoteIdentifier(p.config.LockTable) + ` (lock_id BIGINT NOT NULL PRIMARY KEY)`
if _, err := p.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+786
View File
@@ -0,0 +1,786 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMigrateLockTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-lock-strategy=table", "x-lock-table=lock_table")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
+41
View File
@@ -0,0 +1,41 @@
# pgx
This package is for [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5). A backend for the older [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). is [also available](..).
`pgx5://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
+480
View File
@@ -0,0 +1,480 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgerrcode"
"github.com/jackc/pgx/v5/pgconn"
_ "github.com/jackc/pgx/v5/stdlib"
)
func init() {
db := Postgres{}
database.Register("pgx5", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v5", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+761
View File
@@ -0,0 +1,761 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://../examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
+10
View File
@@ -5,7 +5,10 @@
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
@@ -27,3 +30,10 @@
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
+25 -6
View File
@@ -7,7 +7,7 @@ Our user here is `postgres`, password `password`, and host is `localhost`.
```
psql -h localhost -U postgres -w -c "create database example;"
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convienience:
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
```
export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/example?sslmode=disable'
```
@@ -27,7 +27,7 @@ If there were no errors, we should have two files available under `db/migrations
Note the `sql` extension that we provided.
In the `.up.sql` file let's create the table:
```
```sql
CREATE TABLE IF NOT EXISTS users(
user_id serial PRIMARY KEY,
username VARCHAR (50) UNIQUE NOT NULL,
@@ -36,7 +36,7 @@ CREATE TABLE IF NOT EXISTS users(
);
```
And in the `.down.sql` let's delete it:
```
```sql
DROP TABLE IF EXISTS users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](../../GETTING_STARTED.md#create-migrations)
@@ -79,7 +79,7 @@ Again, it should create for us two migrations files:
In Postgres, when we want our queries to be done in a transaction, we need to wrap it with `BEGIN` and `COMMIT` commands.
In our example, we are going to add a column to our database that can only accept enumerable values or NULL.
Migration up:
```
```sql
BEGIN;
CREATE TYPE enum_mood AS ENUM (
@@ -92,7 +92,7 @@ ALTER TABLE users ADD COLUMN mood enum_mood;
COMMIT;
```
Migration down:
```
```sql
BEGIN;
ALTER TABLE users DROP COLUMN mood;
@@ -124,7 +124,7 @@ Indexes:
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
```go
import (
"log"
@@ -146,3 +146,22 @@ func main() {
}
```
You can find details [here](README.md#use-in-your-go-project)
## Fix issue where migrations run twice
When the schema and role names are the same, you might run into issues if you create this schema using migrations.
This is caused by the fact that the [default `search_path`](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) is `"$user", public`.
In the first run (with an empty database) the migrate table is created in `public`.
When the migrations create the `$user` schema, the next run will store (a new) migrate table in this schema (due to order of schemas in `search_path`) and tries to apply all migrations again (most likely failing).
To solve this you need to change the default `search_path` by removing the `$user` component, so the migrate table is always stored in the (available) `public` schema.
This can be done using the [`search_path` query parameter in the URL](https://github.com/jexia/migrate/blob/fix-postgres-version-table/database/postgres/README.md#postgres).
For example to force the migrations table in the public schema you can use:
```
export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/example?sslmode=disable&search_path=public'
```
Note that you need to explicitly add the schema names to the table names in your migrations when you to modify the tables of the non-public schema.
Alternatively you can add the non-public schema manually (before applying the migrations) if that is possible in your case and let the tool store the migrations table in this schema as well.
+175 -74
View File
@@ -1,21 +1,23 @@
// +build go1.9
//go:build go1.9
package postgres
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
multierror "github.com/hashicorp/go-multierror"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/lib/pq"
)
@@ -25,7 +27,12 @@ func init() {
database.Register("postgresql", &db)
}
var DefaultMigrationsTable = "schema_migrations"
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
@@ -35,35 +42,40 @@ var (
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
StatementTimeout time.Duration
MigrationsTable string
MigrationsTableQuoted bool
MultiStatementEnabled bool
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postgres, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
if err := conn.PingContext(ctx); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -76,31 +88,37 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
var schemaName sql.NullString
if err := conn.QueryRowContext(ctx, query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
if !schemaName.Valid {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
config.SchemaName = schemaName.String
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
@@ -111,6 +129,26 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return px, nil
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
px, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
px.db = instance
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
@@ -123,6 +161,17 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
@@ -132,10 +181,32 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
})
if err != nil {
@@ -147,7 +218,11 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
var dbErr error
if p.db != nil {
dbErr = p.db.Close()
}
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -156,56 +231,68 @@ func (p *Postgres) Close() error {
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
if p.isLocked {
return database.ErrLocked
}
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.SchemaName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
p.isLocked = true
return nil
return nil
})
}
func (p *Postgres) Unlock() error {
if !p.isLocked {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.SchemaName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
p.isLocked = false
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
// run migration
query := string(migr[:])
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pq.Error); ok {
var line uint
@@ -223,17 +310,16 @@ func (p *Postgres) Run(migration io.Reader) error {
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: migr, Line: line}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.Replace(s, "\r\n", "\n", -1)
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -272,10 +358,10 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.MigrationsTable)
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -284,11 +370,10 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.MigrationsTable) +
` (version, dirty) VALUES ($1, $2)`
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -302,7 +387,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` LIMIT 1`
query := `SELECT version, dirty FROM ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
@@ -330,7 +415,7 @@ func (p *Postgres) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -345,6 +430,9 @@ func (p *Postgres) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -369,15 +457,28 @@ func (p *Postgres) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
query := `CREATE TABLE IF NOT EXISTS ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` (version bigint not null primary key, dirty boolean not null)`
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+342 -25
View File
@@ -6,16 +6,17 @@ import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"log"
"github.com/golang-migrate/migrate/v4"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
@@ -34,16 +35,17 @@ var (
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:9.5", Options: opts},
{ImageName: "postgres:9.6", Options: opts},
{ImageName: "postgres:10", Options: opts},
{ImageName: "postgres:11", Options: opts},
{ImageName: "postgres:12", Options: opts},
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string) string {
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?sslmode=disable", pgPassword, host, port)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -74,7 +76,41 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testMultipleStatements", testMultipleStatements)
t.Run("testMultipleStatementsInMultiStatementMode", testMultipleStatementsInMultiStatementMode)
t.Run("testErrorParsing", testErrorParsing)
t.Run("testFilterCustomQuery", testFilterCustomQuery)
t.Run("testWithSchema", testWithSchema)
t.Run("testMigrationTableOption", testMigrationTableOption)
t.Run("testFailToCreateTableWithoutPermissions", testFailToCreateTableWithoutPermissions)
t.Run("testCheckBeforeCreateTable", testCheckBeforeCreateTable)
t.Run("testParallelSchema", testParallelSchema)
t.Run("testPostgresLock", testPostgresLock)
t.Run("testWithInstanceConcurrent", testWithInstanceConcurrent)
t.Run("testWithConnection", testWithConnection)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -96,7 +132,7 @@ func Test(t *testing.T) {
})
}
func TestMigrate(t *testing.T) {
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -122,7 +158,7 @@ func TestMigrate(t *testing.T) {
})
}
func TestMultiStatement(t *testing.T) {
func testMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -155,7 +191,40 @@ func TestMultiStatement(t *testing.T) {
})
}
func TestErrorParsing(t *testing.T) {
func testMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func testErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -184,7 +253,7 @@ func TestErrorParsing(t *testing.T) {
})
}
func TestFilterCustomQuery(t *testing.T) {
func testFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -206,7 +275,7 @@ func TestFilterCustomQuery(t *testing.T) {
})
}
func TestWithSchema(t *testing.T) {
func testWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -276,7 +345,222 @@ func TestWithSchema(t *testing.T) {
})
}
func TestParallelSchema(t *testing.T) {
func testMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func testFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func testCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func testParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -344,11 +628,7 @@ func TestParallelSchema(t *testing.T) {
})
}
func TestWithInstance(t *testing.T) {
}
func TestPostgres_Lock(t *testing.T) {
func testPostgresLock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -388,7 +668,7 @@ func TestPostgres_Lock(t *testing.T) {
})
}
func TestWithInstance_Concurrent(t *testing.T) {
func testWithInstanceConcurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -430,6 +710,44 @@ func TestWithInstance_Concurrent(t *testing.T) {
}
})
}
func testWithConnection(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
db, err := sql.Open("postgres", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
ctx := context.Background()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatal(err)
}
p, err := WithConnection(ctx, conn, &Config{})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := p.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, p, []byte("SELECT 1"))
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
@@ -480,10 +798,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.Replace(input, "\n", "\r\n", -1)
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.Replace(input, "FROM", "FRÖM", -1)
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
@@ -508,5 +826,4 @@ func Test_computeLineFromPos(t *testing.T) {
run(true, true)
})
}
}
+17 -19
View File
@@ -2,13 +2,12 @@ package ql
import (
"database/sql"
"errors"
"fmt"
"github.com/hashicorp/go-multierror"
"io"
"io/ioutil"
"strings"
nurl "net/url"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
@@ -34,7 +33,7 @@ type Config struct {
type Ql struct {
db *sql.DB
isLocked bool
isLocked atomic.Bool
config *Config
}
@@ -72,11 +71,7 @@ func (m *Ql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -133,9 +128,10 @@ func (m *Ql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
@@ -148,6 +144,10 @@ func (m *Ql) Drop() (err error) {
}
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
@@ -161,21 +161,19 @@ func (m *Ql) Drop() (err error) {
return nil
}
func (m *Ql) Lock() error {
if m.isLocked {
if !m.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
m.isLocked = true
return nil
}
func (m *Ql) Unlock() error {
if !m.isLocked {
return nil
if !m.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
m.isLocked = false
return nil
}
func (m *Ql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -190,7 +188,7 @@ func (m *Ql) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -218,7 +216,7 @@ func (m *Ql) SetVersion(version int, dirty bool) error {
m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+2 -20
View File
@@ -3,8 +3,6 @@ package ql
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -15,15 +13,7 @@ import (
)
func Test(t *testing.T) {
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
p := &Ql{}
addr := fmt.Sprintf("ql://%s", filepath.Join(dir, "ql.db"))
@@ -45,15 +35,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
db, err := sql.Open("ql", filepath.Join(dir, "ql.db"))
+17 -17
View File
@@ -1,20 +1,20 @@
// +build go1.9
//go:build go1.9
package redshift
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
@@ -36,7 +36,7 @@ type Config struct {
}
type Redshift struct {
isLocked bool
isLocked atomic.Bool
conn *sql.Conn
db *sql.DB
@@ -126,20 +126,21 @@ func (p *Redshift) Close() error {
// Redshift does not support advisory lock functions: https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html
func (p *Redshift) Lock() error {
if p.isLocked {
if !p.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
p.isLocked = true
return nil
}
func (p *Redshift) Unlock() error {
p.isLocked = false
if !p.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (p *Redshift) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -173,7 +174,7 @@ func (p *Redshift) Run(migration io.Reader) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.Replace(s, "\r\n", "\n", -1)
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -215,7 +216,7 @@ func (p *Redshift) SetVersion(version int, dirty bool) error {
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -227,7 +228,7 @@ func (p *Redshift) SetVersion(version int, dirty bool) error {
query = `INSERT INTO "` + p.config.MigrationsTable + `" (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -269,7 +270,7 @@ func (p *Redshift) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -284,6 +285,9 @@ func (p *Redshift) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -308,11 +312,7 @@ func (p *Redshift) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+16 -7
View File
@@ -28,10 +28,19 @@ import (
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "redshift"
)
var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true,
ReadyFunc: isReady,
}
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:8", Options: opts},
{ImageName: "migrate/postgres8:8", Options: opts},
}
)
@@ -44,7 +53,7 @@ func pgConnectionString(host, port string) string {
}
func connectionString(schema, host, port string) string {
return fmt.Sprintf("%s://postgres@%s:%s/postgres?sslmode=disable", schema, host, port)
return fmt.Sprintf("%s://postgres:%s@%s:%s/postgres?sslmode=disable", schema, pgPassword, host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -192,7 +201,7 @@ func TestFilterCustomQuery(t *testing.T) {
t.Fatal(err)
}
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", ip, port)
addr := fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-custom=foobar", pgPassword, ip, port)
p := &Redshift{}
d, err := p.Open(addr)
if err != nil {
@@ -234,7 +243,7 @@ func TestWithSchema(t *testing.T) {
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", ip, port))
d2, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foobar", pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
@@ -369,10 +378,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.Replace(input, "\n", "\r\n", -1)
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.Replace(input, "FROM", "FRÖM", -1)
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
+18
View File
@@ -0,0 +1,18 @@
# rqlite
`rqlite://admin:secret@server1.example.com:4001/?level=strong&timeout=5`
The `rqlite` url scheme is used for both secure and insecure connections. If connecting to an insecure database, pass `x-connect-insecure` in your URL query, or use `WithInstance` to pass an established connection.
The migrations table name is configurable through the `x-migrations-table` URL query parameter, or by using `WithInstance` and passing `MigrationsTable` through `Config`.
Other connect parameters are directly passed through to the database driver. For examples of connection strings, see https://github.com/rqlite/gorqlite#examples.
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-connect-insecure` | n/a: set on instance | Boolean to indicate whether to use an insecure connection. Defaults to `false`. |
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table. Defaults to `schema_migrations`. |
## Notes
* Uses the https://github.com/rqlite/gorqlite driver
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1,3 @@
CREATE TABLE pets (
name string
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1 @@
ALTER TABLE pets ADD predator bool;
+328
View File
@@ -0,0 +1,328 @@
package rqlite
import (
"errors"
"fmt"
"io"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/rqlite/gorqlite"
)
func init() {
database.Register("rqlite", &Rqlite{})
}
const (
// DefaultMigrationsTable defines the default rqlite migrations table
DefaultMigrationsTable = "schema_migrations"
// DefaultConnectInsecure defines the default setting for connect insecure
DefaultConnectInsecure = false
)
// ErrNilConfig is returned if no configuration was passed to WithInstance
var ErrNilConfig = fmt.Errorf("no config")
// ErrBadConfig is returned if configuration was invalid
var ErrBadConfig = fmt.Errorf("bad parameter")
// Config defines the driver configuration
type Config struct {
// ConnectInsecure sets whether the connection uses TLS. Ineffectual when using WithInstance
ConnectInsecure bool
// MigrationsTable configures the migrations table name
MigrationsTable string
}
type Rqlite struct {
db *gorqlite.Connection
isLocked atomic.Bool
config *Config
}
// WithInstance creates a rqlite database driver with an existing gorqlite database connection
// and a Config struct
func WithInstance(instance *gorqlite.Connection, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
// we use the consistency level check as a database ping
if _, err := instance.ConsistencyLevel(); err != nil {
return nil, err
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
driver := &Rqlite{
db: instance,
config: config,
}
if err := driver.ensureVersionTable(); err != nil {
return nil, err
}
return driver, nil
}
// OpenURL creates a rqlite database driver from a connect URL
func OpenURL(url string) (database.Driver, error) {
d := &Rqlite{}
return d.Open(url)
}
func (r *Rqlite) ensureVersionTable() (err error) {
if err = r.Lock(); err != nil {
return err
}
defer func() {
if e := r.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
stmts := []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (version uint64, dirty bool)`, r.config.MigrationsTable),
fmt.Sprintf(`CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version)`, r.config.MigrationsTable),
}
if _, err := r.db.Write(stmts); err != nil {
return err
}
return nil
}
// Open returns a new driver instance configured with parameters
// coming from the URL string. Migrate will call this function
// only once per instance.
func (r *Rqlite) Open(url string) (database.Driver, error) {
dburl, config, err := parseUrl(url)
if err != nil {
return nil, err
}
r.config = config
r.db, err = gorqlite.Open(dburl.String())
if err != nil {
return nil, err
}
if err := r.ensureVersionTable(); err != nil {
return nil, err
}
return r, nil
}
// Close closes the underlying database instance managed by the driver.
// Migrate will call this function only once per instance.
func (r *Rqlite) Close() error {
r.db.Close()
return nil
}
// Lock should acquire a database lock so that only one migration process
// can run at a time. Migrate will call this function before Run is called.
// If the implementation can't provide this functionality, return nil.
// Return database.ErrLocked if database is already locked.
func (r *Rqlite) Lock() error {
if !r.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
// Unlock should release the lock. Migrate will call this function after
// all migrations have been run.
func (r *Rqlite) Unlock() error {
if !r.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
// Run applies a migration to the database. migration is guaranteed to be not nil.
func (r *Rqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if _, err := r.db.WriteOne(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// SetVersion saves version and dirty state.
// Migrate will call this function before and after each call to Run.
// version must be >= -1. -1 means NilVersion.
func (r *Rqlite) SetVersion(version int, dirty bool) error {
deleteQuery := fmt.Sprintf(`DELETE FROM %s`, r.config.MigrationsTable)
statements := []gorqlite.ParameterizedStatement{
{
Query: deleteQuery,
},
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
insertQuery := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, r.config.MigrationsTable)
if version >= 0 || (version == database.NilVersion && dirty) {
statements = append(statements, gorqlite.ParameterizedStatement{
Query: insertQuery,
Arguments: []interface{}{
version,
dirty,
},
})
}
wr, err := r.db.WriteParameterized(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i].Query)}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(deleteQuery + "\n" + insertQuery)}
}
return nil
}
// Version returns the currently active version and if the database is dirty.
// When no migration has been applied, it must return version -1.
// Dirty means, a previous migration failed and user interaction is required.
func (r *Rqlite) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM " + r.config.MigrationsTable + " LIMIT 1"
qr, err := r.db.QueryOne(query)
if err != nil {
return database.NilVersion, false, nil
}
if !qr.Next() {
return database.NilVersion, false, nil
}
if err := qr.Scan(&version, &dirty); err != nil {
return database.NilVersion, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty, nil
}
// Drop deletes everything in the database.
// Note that this is a breaking action, a new call to Open() is necessary to
// ensure subsequent calls work as expected.
func (r *Rqlite) Drop() error {
query := `SELECT name FROM sqlite_master WHERE type = 'table'`
tables, err := r.db.QueryOne(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
statements := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
statement := fmt.Sprintf(`DROP TABLE %s`, tableName)
statements = append(statements, statement)
}
}
// return if nothing to do
if len(statements) <= 0 {
return nil
}
wr, err := r.db.Write(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i])}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(strings.Join(statements, "\n"))}
}
return nil
}
func parseUrl(url string) (*nurl.URL, *Config, error) {
parsedUrl, err := nurl.Parse(url)
if err != nil {
return nil, nil, err
}
config, err := parseConfigFromQuery(parsedUrl.Query())
if err != nil {
return nil, nil, err
}
if parsedUrl.Scheme != "rqlite" {
return nil, nil, fmt.Errorf("bad scheme: %w", ErrBadConfig)
}
// adapt from rqlite to http/https schemes
if config.ConnectInsecure {
parsedUrl.Scheme = "http"
} else {
parsedUrl.Scheme = "https"
}
filteredUrl := migrate.FilterCustomQuery(parsedUrl)
return filteredUrl, config, nil
}
func parseConfigFromQuery(queryVals nurl.Values) (*Config, error) {
c := Config{
ConnectInsecure: DefaultConnectInsecure,
MigrationsTable: DefaultMigrationsTable,
}
migrationsTable := queryVals.Get("x-migrations-table")
if migrationsTable != "" {
if strings.HasPrefix(migrationsTable, "sqlite_") {
return nil, fmt.Errorf("invalid value for x-migrations-table: %w", ErrBadConfig)
}
c.MigrationsTable = migrationsTable
}
connectInsecureStr := queryVals.Get("x-connect-insecure")
if connectInsecureStr != "" {
connectInsecure, err := strconv.ParseBool(connectInsecureStr)
if err != nil {
return nil, fmt.Errorf("invalid value for x-connect-insecure: %w", ErrBadConfig)
}
c.ConnectInsecure = connectInsecure
}
return &c, nil
}
+324
View File
@@ -0,0 +1,324 @@
package rqlite
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"testing"
"github.com/dhui/dktest"
"github.com/rqlite/gorqlite"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
var defaultPort uint16 = 4001
var opts = dktest.Options{
Env: map[string]string{"NODE_ID": "1"},
PortRequired: true,
ReadyFunc: isReady,
}
var specs = []dktesting.ContainerSpec{
{ImageName: "rqlite/rqlite:7.21.4", Options: opts},
{ImageName: "rqlite/rqlite:8.0.6", Options: opts},
{ImageName: "rqlite/rqlite:8.11.1", Options: opts},
{ImageName: "rqlite/rqlite:8.12.3", Options: opts},
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
fmt.Println("error getting port")
return false
}
statusString := fmt.Sprintf("http://%s:%s/status", ip, port)
fmt.Println(statusString)
var readyResp struct {
Store struct {
Ready bool `json:"ready"`
} `json:"store"`
}
resp, err := http.Get(statusString)
if err != nil {
fmt.Println("error getting status")
return false
}
if resp.StatusCode != 200 {
fmt.Println("statusCode != 200")
return false
}
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("error reading body")
return false
}
if err := json.Unmarshal(body, &readyResp); err != nil {
fmt.Println("error unmarshaling body")
return false
}
return readyResp.Store.Ready
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
r := &Rqlite{}
d, err := r.Open(connectString)
assert.NoError(t, err)
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
driver, err := OpenURL(connectString)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
return
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
dt.TestMigrate(t, m)
})
}
func TestBadConnectInsecureParam(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?x-connect-insecure=foo", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestBadProtocol(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("postgres://%s:%s/database", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestNoConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
_, err = WithInstance(db, nil)
assert.ErrorIs(t, err, ErrNilConfig)
})
}
func TestWithInstanceEmptyConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
driver, err := WithInstance(db, &Config{})
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", DefaultMigrationsTable))
assert.NoError(t, err)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
})
}
func TestMigrationTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
config := Config{MigrationsTable: "my_migration_table"}
driver, err := WithInstance(db, &config)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
_, err = db.WriteOne(`INSERT INTO pets (name, predator) VALUES ("franklin", true)`)
assert.NoError(t, err)
res, err := db.QueryOne(`SELECT name, predator FROM pets LIMIT 1`)
assert.NoError(t, err)
_ = res.Next()
// make sure we can use the migrated table
var petName string
var petPredator int
err = res.Scan(&petName, &petPredator)
assert.NoError(t, err)
assert.Equal(t, petName, "franklin")
assert.Equal(t, petPredator, 1)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
})
}
func TestParseUrl(t *testing.T) {
tests := []struct {
name string
passedUrl string
expectedUrl string
expectedConfig *Config
expectedErr string
}{
{
"defaults",
"rqlite://localhost:4001",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"configure migration table",
"rqlite://localhost:4001?x-migrations-table=foo",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: "foo"},
"",
},
{
"configure connect insecure",
"rqlite://localhost:4001?x-connect-insecure=true",
"http://localhost:4001",
&Config{ConnectInsecure: true, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"invalid migration table",
"rqlite://localhost:4001?x-migrations-table=sqlite_bar",
"",
nil,
"invalid value for x-migrations-table: bad parameter",
},
{
"invalid connect insecure",
"rqlite://localhost:4001?x-connect-insecure=baz",
"",
nil,
"invalid value for x-connect-insecure: bad parameter",
},
{
"invalid url",
string([]byte{0x7f}),
"",
nil,
"parse \"\\x7f\": net/url: invalid control character in URL",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
actualUrl, actualConfig, actualErr := parseUrl(tt.passedUrl)
if tt.expectedUrl != "" {
assert.Equal(t, tt.expectedUrl, actualUrl.String())
} else {
assert.Nil(t, actualUrl)
}
assert.Equal(t, tt.expectedConfig, actualConfig)
if tt.expectedErr == "" {
assert.NoError(t, actualErr)
} else {
assert.EqualError(t, actualErr, tt.expectedErr)
}
})
}
}
+16 -16
View File
@@ -3,15 +3,15 @@ package snowflake
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
sf "github.com/snowflakedb/gosnowflake"
)
@@ -37,7 +37,7 @@ type Config struct {
}
type Snowflake struct {
isLocked bool
isLocked atomic.Bool
conn *sql.Conn
db *sql.DB
@@ -158,20 +158,21 @@ func (p *Snowflake) Close() error {
}
func (p *Snowflake) Lock() error {
if p.isLocked {
if !p.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
p.isLocked = true
return nil
}
func (p *Snowflake) Unlock() error {
p.isLocked = false
if !p.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (p *Snowflake) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -205,7 +206,7 @@ func (p *Snowflake) Run(migration io.Reader) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.Replace(s, "\r\n", "\n", -1)
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -247,7 +248,7 @@ func (p *Snowflake) SetVersion(version int, dirty bool) error {
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -261,7 +262,7 @@ func (p *Snowflake) SetVersion(version int, dirty bool) error {
` + strconv.FormatBool(dirty) + `)`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -303,7 +304,7 @@ func (p *Snowflake) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -318,6 +319,9 @@ func (p *Snowflake) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -342,11 +346,7 @@ func (p *Snowflake) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+1 -1
View File
@@ -33,7 +33,7 @@ as described in [README.md#database-urls](../../README.md#database-urls)
## DDL with comments
At the moment the GCP Spanner backed does not seem to allow for comments (See https://issuetracker.google.com/issues/159730604)
so in order to be able to use migration with DDL containing comments `x-clean-stamements` is required
so in order to be able to use migration with DDL containing comments `x-clean-statements` is required
## Multiple statements
@@ -0,0 +1 @@
DROP TABLE Transactions
@@ -0,0 +1,6 @@
CREATE TABLE Transactions (
UserId INT64,
TransactionId STRING(40),
Total NUMERIC
) PRIMARY KEY(UserId, TransactionId),
INTERLEAVE IN PARENT Users ON DELETE CASCADE
+22 -17
View File
@@ -1,16 +1,16 @@
package spanner
import (
"context"
"errors"
"fmt"
"io"
"io/ioutil"
"log"
nurl "net/url"
"regexp"
"strconv"
"strings"
"golang.org/x/net/context"
"sync/atomic"
"cloud.google.com/go/spanner"
sdb "cloud.google.com/go/spanner/admin/database/apiv1"
@@ -19,9 +19,8 @@ import (
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
adminpb "cloud.google.com/go/spanner/admin/database/apiv1/databasepb"
"google.golang.org/api/iterator"
adminpb "google.golang.org/genproto/googleapis/spanner/admin/database/v1"
)
func init() {
@@ -34,10 +33,12 @@ const DefaultMigrationsTable = "SchemaMigrations"
// Driver errors
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
ErrDatabaseDirty = fmt.Errorf("database is dirty")
ErrNilConfig = errors.New("no config")
ErrNoDatabaseName = errors.New("no database name")
ErrNoSchema = errors.New("no schema")
ErrDatabaseDirty = errors.New("database is dirty")
ErrLockHeld = errors.New("unable to obtain lock")
ErrLockNotHeld = errors.New("unable to release already released lock")
)
// Config used for a Spanner instance
@@ -56,6 +57,8 @@ type Spanner struct {
db *DB
config *Config
lock atomic.Bool
}
type DB struct {
@@ -143,17 +146,23 @@ func (s *Spanner) Close() error {
// Lock implements database.Driver but doesn't do anything because Spanner only
// enqueues the UpdateDatabaseDdlRequest.
func (s *Spanner) Lock() error {
return nil
if swapped := s.lock.CompareAndSwap(false, true); swapped {
return nil
}
return ErrLockHeld
}
// Unlock implements database.Driver but no action required, see Lock.
func (s *Spanner) Unlock() error {
return nil
if swapped := s.lock.CompareAndSwap(true, false); swapped {
return nil
}
return ErrLockNotHeld
}
// Run implements database.Driver
func (s *Spanner) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -289,11 +298,7 @@ func (s *Spanner) ensureVersionTable() (err error) {
defer func() {
if e := s.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+49 -36
View File
@@ -8,52 +8,57 @@ import (
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
_ "github.com/golang-migrate/migrate/v4/source/file"
"cloud.google.com/go/spanner/spannertest"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func Test(t *testing.T) {
if testing.Short() {
t.Skip("skipping test in short mode.")
}
db, ok := os.LookupEnv("SPANNER_DATABASE")
if !ok {
t.Skip("SPANNER_DATABASE not set, skipping test.")
}
s := &Spanner{}
addr := fmt.Sprintf("spanner://%s", db)
d, err := s.Open(addr)
// withSpannerEmulator is not thread-safe and cannot be used with parallel tests since it sets the emulator
func withSpannerEmulator(t *testing.T, testFunc func(t *testing.T)) {
t.Helper()
srv, err := spannertest.NewServer("localhost:0")
if err != nil {
t.Fatal(err)
t.Fatal("Failed to create Spanner emulator:", err)
}
dt.Test(t, d, []byte("SELECT 1"))
// This is not thread-safe
if err := os.Setenv("SPANNER_EMULATOR_HOST", srv.Addr); err != nil {
t.Fatal("Failed to set SPANNER_EMULATOR_HOST env var:", err)
}
defer srv.Close()
testFunc(t)
}
const db = "projects/abc/instances/def/databases/testdb"
func Test(t *testing.T) {
withSpannerEmulator(t, func(t *testing.T) {
uri := fmt.Sprintf("spanner://%s", db)
s := &Spanner{}
d, err := s.Open(uri)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("CREATE TABLE test (id BOOL) PRIMARY KEY (id)"))
})
}
func TestMigrate(t *testing.T) {
if testing.Short() {
t.Skip("skipping test in short mode.")
}
db, ok := os.LookupEnv("SPANNER_DATABASE")
if !ok {
t.Skip("SPANNER_DATABASE not set, skipping test.")
}
s := &Spanner{}
addr := fmt.Sprintf("spanner://%s", db)
d, err := s.Open(addr)
if err != nil {
t.Fatal(err)
}
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", db, d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
withSpannerEmulator(t, func(t *testing.T) {
s := &Spanner{}
uri := fmt.Sprintf("spanner://%s", db)
d, err := s.Open(uri)
if err != nil {
t.Fatal(err)
}
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", uri, d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestCleanStatements(t *testing.T) {
@@ -146,6 +151,14 @@ func TestCleanStatements(t *testing.T) {
SET OPTIONS (allow_commit_timestamp=true);`,
expected: []string{"ALTER TABLE users ALTER COLUMN created SET OPTIONS (allow_commit_timestamp = true)"},
},
{
name: "column with NUMERIC type",
multiStatement: `CREATE TABLE table_name (
id STRING(255) NOT NULL,
sum NUMERIC,
) PRIMARY KEY (id)`,
expected: []string{"CREATE TABLE table_name (\n id STRING(255) NOT NULL,\n sum NUMERIC,\n) PRIMARY KEY(id)"},
},
}
for _, tc := range testCases {
+16 -17
View File
@@ -2,16 +2,16 @@ package sqlcipher
import (
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/mutecomm/go-sqlcipher/v4"
)
@@ -34,7 +34,7 @@ type Config struct {
type Sqlite struct {
db *sql.DB
isLocked bool
isLocked atomic.Bool
config *Config
}
@@ -72,11 +72,7 @@ func (m *Sqlite) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -140,9 +136,10 @@ func (m *Sqlite) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
@@ -153,6 +150,10 @@ func (m *Sqlite) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
@@ -172,23 +173,21 @@ func (m *Sqlite) Drop() (err error) {
}
func (m *Sqlite) Lock() error {
if m.isLocked {
if !m.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
m.isLocked = true
return nil
}
func (m *Sqlite) Unlock() error {
if !m.isLocked {
return nil
if !m.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
m.isLocked = false
return nil
}
func (m *Sqlite) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -207,7 +206,7 @@ func (m *Sqlite) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -242,7 +241,7 @@ func (m *Sqlite) SetVersion(version int, dirty bool) error {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+6 -48
View File
@@ -3,8 +3,6 @@ package sqlcipher
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -17,15 +15,7 @@ import (
)
func Test(t *testing.T) {
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s", filepath.Join(dir, "sqlite3.db"))
@@ -37,15 +27,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
db, err := sql.Open("sqlite3", filepath.Join(dir, "sqlite3.db"))
@@ -72,15 +54,7 @@ func TestMigrate(t *testing.T) {
}
func TestMigrationTable(t *testing.T) {
dir, err := ioutil.TempDir("", "sqlite3-driver-test-migration-table")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
@@ -120,15 +94,7 @@ func TestMigrationTable(t *testing.T) {
}
func TestNoTxWrap(t *testing.T) {
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s?x-no-tx-wrap=true", filepath.Join(dir, "sqlite3.db"))
@@ -142,19 +108,11 @@ func TestNoTxWrap(t *testing.T) {
}
func TestNoTxWrapInvalidValue(t *testing.T) {
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s?x-no-tx-wrap=yeppers", filepath.Join(dir, "sqlite3.db"))
_, err = p.Open(addr)
_, err := p.Open(addr)
if assert.Error(t, err) {
assert.Contains(t, err.Error(), "x-no-tx-wrap")
assert.Contains(t, err.Error(), "invalid syntax")
+17
View File
@@ -0,0 +1,17 @@
# sqlite
`sqlite://path/to/database?query`
Unlike other migrate database drivers, the sqlite driver will automatically wrap each migration in an implicit transaction by default. Migrations must not contain explicit `BEGIN` or `COMMIT` statements. This behavior may change in a future major release. (See below for a workaround.)
The auxiliary query parameters listed below may be supplied to tailor migrate behavior. All auxiliary query parameters are optional.
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table. Defaults to `schema_migrations`. |
| `x-no-tx-wrap` | `NoTxWrap` | Disable implicit transactions when `true`. Migrations may, and should, contain explicit `BEGIN` and `COMMIT` statements. |
## Notes
* Uses the `modernc.org/sqlite` sqlite db driver (pure Go)
* Has [limited `GOOS` and `GOARCH` support](https://pkg.go.dev/modernc.org/sqlite?utm_source=godoc#hdr-Supported_platforms_and_architectures)
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1,3 @@
CREATE TABLE pets (
name string
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1 @@
ALTER TABLE pets ADD predator bool;
+264
View File
@@ -0,0 +1,264 @@
package sqlite
import (
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
_ "modernc.org/sqlite"
)
func init() {
database.Register("sqlite", &Sqlite{})
}
var DefaultMigrationsTable = "schema_migrations"
var (
ErrDatabaseDirty = fmt.Errorf("database is dirty")
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
)
type Config struct {
MigrationsTable string
DatabaseName string
NoTxWrap bool
}
type Sqlite struct {
db *sql.DB
isLocked atomic.Bool
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
mx := &Sqlite{
db: instance,
config: config,
}
if err := mx.ensureVersionTable(); err != nil {
return nil, err
}
return mx, nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Sqlite type.
func (m *Sqlite) ensureVersionTable() (err error) {
if err = m.Lock(); err != nil {
return err
}
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
query := fmt.Sprintf(`
CREATE TABLE IF NOT EXISTS %s (version uint64,dirty bool);
CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version);
`, m.config.MigrationsTable, m.config.MigrationsTable)
if _, err := m.db.Exec(query); err != nil {
return err
}
return nil
}
func (m *Sqlite) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
dbfile := strings.Replace(migrate.FilterCustomQuery(purl).String(), "sqlite://", "", 1)
db, err := sql.Open("sqlite", dbfile)
if err != nil {
return nil, err
}
qv := purl.Query()
migrationsTable := qv.Get("x-migrations-table")
if len(migrationsTable) == 0 {
migrationsTable = DefaultMigrationsTable
}
noTxWrap := false
if v := qv.Get("x-no-tx-wrap"); v != "" {
noTxWrap, err = strconv.ParseBool(v)
if err != nil {
return nil, fmt.Errorf("x-no-tx-wrap: %s", err)
}
}
mx, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
NoTxWrap: noTxWrap,
})
if err != nil {
return nil, err
}
return mx, nil
}
func (m *Sqlite) Close() error {
return m.db.Close()
}
func (m *Sqlite) Drop() (err error) {
query := `SELECT name FROM sqlite_master WHERE type = 'table';`
tables, err := m.db.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
err = m.executeQuery(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
query := "VACUUM"
_, err = m.db.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
return nil
}
func (m *Sqlite) Lock() error {
if !m.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
func (m *Sqlite) Unlock() error {
if !m.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (m *Sqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if m.config.NoTxWrap {
return m.executeQueryNoTx(query)
}
return m.executeQuery(query)
}
func (m *Sqlite) executeQuery(query string) error {
tx, err := m.db.Begin()
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (m *Sqlite) executeQueryNoTx(query string) error {
if _, err := m.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (m *Sqlite) SetVersion(version int, dirty bool) error {
tx, err := m.db.Begin()
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "DELETE FROM " + m.config.MigrationsTable
if _, err := tx.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (m *Sqlite) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM " + m.config.MigrationsTable + " LIMIT 1"
err = m.db.QueryRow(query).Scan(&version, &dirty)
if err != nil {
return database.NilVersion, false, nil
}
return version, dirty, nil
}
+133
View File
@@ -0,0 +1,133 @@
package sqlite
import (
"database/sql"
"fmt"
"path/filepath"
"testing"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
_ "github.com/golang-migrate/migrate/v4/source/file"
_ "modernc.org/sqlite"
)
func Test(t *testing.T) {
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://%s", filepath.Join(dir, "sqlite.db"))
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
}
func TestMigrate(t *testing.T) {
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
db, err := sql.Open("sqlite", filepath.Join(dir, "sqlite.db"))
if err != nil {
return
}
defer func() {
if err := db.Close(); err != nil {
return
}
}()
driver, err := WithInstance(db, &Config{})
if err != nil {
t.Fatal(err)
}
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
}
func TestMigrationTable(t *testing.T) {
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
db, err := sql.Open("sqlite", filepath.Join(dir, "sqlite.db"))
if err != nil {
return
}
defer func() {
if err := db.Close(); err != nil {
return
}
}()
config := &Config{
MigrationsTable: "my_migration_table",
}
driver, err := WithInstance(db, config)
if err != nil {
t.Fatal(err)
}
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
if err != nil {
t.Fatal(err)
}
t.Log("UP")
err = m.Up()
if err != nil {
t.Fatal(err)
}
_, err = db.Query(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
if err != nil {
t.Fatal(err)
}
}
func TestNoTxWrap(t *testing.T) {
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://%s?x-no-tx-wrap=true", filepath.Join(dir, "sqlite.db"))
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
// An explicit BEGIN statement would ordinarily fail without x-no-tx-wrap.
// (Transactions in sqlite may not be nested.)
dt.Test(t, d, []byte("BEGIN; CREATE TABLE t (Qty int, Name string); COMMIT;"))
}
func TestNoTxWrapInvalidValue(t *testing.T) {
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://%s?x-no-tx-wrap=yeppers", filepath.Join(dir, "sqlite.db"))
_, err := p.Open(addr)
if assert.Error(t, err) {
assert.Contains(t, err.Error(), "x-no-tx-wrap")
assert.Contains(t, err.Error(), "invalid syntax")
}
}
func TestMigrateWithDirectoryNameContainsWhitespaces(t *testing.T) {
dir := t.TempDir()
dbPath := filepath.Join(dir, "sqlite.db")
t.Logf("DB path : %s\n", dbPath)
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://file:%s", dbPath)
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
}
+5 -2
View File
@@ -2,12 +2,15 @@
`sqlite3://path/to/database?query`
Unlike other migrate database drivers, the sqlite3 driver will automatically wrap each migration in an implicit transaction by default. Migrations must not contain explicit `BEGIN` or `COMMIT` statements. This behaviour may change in a future major release. (See below for a workaround.)
Unlike other migrate database drivers, the sqlite3 driver will automatically wrap each migration in an implicit transaction by default. Migrations must not contain explicit `BEGIN` or `COMMIT` statements. This behavior may change in a future major release. (See below for a workaround.)
Refer to [upstream documentation](https://github.com/mattn/go-sqlite3/blob/master/README.md#connection-string) for a complete list of query parameters supported by the sqlite3 database driver. The auxiliary query parameters listed below may be supplied to tailor migrate behaviour. All auxiliary query parameters are optional.
Refer to [upstream documentation](https://github.com/mattn/go-sqlite3/blob/master/README.md#connection-string) for a complete list of query parameters supported by the sqlite3 database driver. The auxiliary query parameters listed below may be supplied to tailor migrate behavior. All auxiliary query parameters are optional.
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table. Defaults to `schema_migrations`. |
| `x-no-tx-wrap` | `NoTxWrap` | Disable implicit transactions when `true`. Migrations may, and should, contain explicit `BEGIN` and `COMMIT` statements. |
## Notes
* Uses the `github.com/mattn/go-sqlite3` sqlite db driver (cgo)
+16 -17
View File
@@ -2,16 +2,16 @@ package sqlite3
import (
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/mattn/go-sqlite3"
)
@@ -34,7 +34,7 @@ type Config struct {
type Sqlite struct {
db *sql.DB
isLocked bool
isLocked atomic.Bool
config *Config
}
@@ -72,11 +72,7 @@ func (m *Sqlite) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -140,9 +136,10 @@ func (m *Sqlite) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
@@ -153,6 +150,10 @@ func (m *Sqlite) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
@@ -172,23 +173,21 @@ func (m *Sqlite) Drop() (err error) {
}
func (m *Sqlite) Lock() error {
if m.isLocked {
if !m.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
m.isLocked = true
return nil
}
func (m *Sqlite) Unlock() error {
if !m.isLocked {
return nil
if !m.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
m.isLocked = false
return nil
}
func (m *Sqlite) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -207,7 +206,7 @@ func (m *Sqlite) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -242,7 +241,7 @@ func (m *Sqlite) SetVersion(version int, dirty bool) error {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}

Some files were not shown because too many files have changed in this diff Show More