Compare commits

..
619 Commits
Author SHA1 Message Date
zeekayandClaude Opus 4.8 f90c751b10 refactor(datastore): back the datastore migration driver with native hanzo-ds/go, drop clickhouse-go
The datastore:// migration driver is now a self-contained golang-migrate
database.Driver that opens through github.com/hanzo-ds/go (the native
ClickHouse-wire driver registered as the database/sql name "datastore").
The ClickHouse-specific migration logic (multi-statement exec,
schema_migrations DDL, cluster/engine config, Version/SetVersion/Drop/Lock)
is ported directly into database/datastore, so it no longer embeds or
imports the upstream database/clickhouse driver or
github.com/ClickHouse/clickhouse-go.

- Delete database/clickhouse (driver + test + examples) and
  internal/cli/build_clickhouse.go.
- internal/cli/build_datastore.go drops the clickhouse-go blank import.
- go.mod: drop github.com/ClickHouse/clickhouse-go, add
  github.com/hanzo-ds/go v1.0.1; tidy. Zero clickhouse-go in go.mod,
  go.sum, and the module/dependency graph.
- Port the driver test + example migrations under database/datastore
  (integration test requires a live datastore/clickhouse-wire backend).
- Update README, release workflow build tags, bug report template.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 18:40:56 -07:00
z c7a0a74aef docs(brand): add hero banner 2026-06-28 20:17:58 -07:00
z 50117bac35 chore(brand): dynamic hero banner 2026-06-28 20:17:56 -07:00
d9c67dc417 ci: run on self-hosted ARC pool (hanzo-build-linux-amd64/deploy), not GitHub-hosted (#1)
Co-authored-by: zeekay <z@hanzo.ai>
2026-06-19 20:36:45 -07:00
Antje WorringandClaude Opus 4.8 a5075e2de1 docs: tidy LLM.md indexes; CLAUDE.md -> LLM.md symlink convention
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 10:05:12 -07:00
Hanzo AI 9477979a55 deps: bump Go to 1.26.4 across go.mod, Dockerfiles, GH Actions
Workspace-wide sync. luxfi/node already shipped on 1.26.4 in v1.30.6
(commit 121aca1fa9); this is the cross-repo catch-up.
2026-06-07 12:10:59 -07:00
Hanzo AI f4abb4bba7 Merge remote-tracking branch 'upstream/master' 2026-05-19 21:54:56 -07:00
Hanzo AI 3461b6bfd7 go.mod: bump go directive to 1.26.3 (security advisory) 2026-05-12 21:25:26 -07:00
Hanzo AI 602c5edb72 chore: symlink AGENTS.md and CLAUDE.md to LLM.md
Canonical project context lives in LLM.md. Symlinks ensure
agentic coding tools (Claude Code, Cursor, etc.) find context
automatically regardless of which filename they look for.
2026-04-01 14:12:04 -07:00
Hanzo Dev a7da767fa3 chore: bump Go 1.26.1 2026-03-22 14:10:48 -07:00
Hanzo Dev b55e3ea06d docs: add LLM.md project guide 2026-03-11 10:31:22 -07:00
Dale HuiandGitHub 2bd822b3aa Merge pull request #1361 from golang-migrate/dependabot/go_modules/go_modules-6b971a9d7e
Bump go.opentelemetry.io/otel/sdk from 1.36.0 to 1.40.0 in the go_modules group across 1 directory
2026-03-11 00:50:10 -07:00
Zach Kelling 460d57267a fix: tar binary as 'migrate' not 'migrate.linux-amd64' to match upstream 2026-03-03 10:16:42 -08:00
Zach Kelling 1e4b96537d Add release workflow for CI-built binaries
Builds migrate CLI for linux/darwin (amd64/arm64) with datastore
driver support and publishes GitHub Release on tag push.
2026-03-03 09:24:50 -08:00
Zach Kelling 837dda3cc9 Add datastore database driver for Hanzo Datastore
Registers "datastore://" as a URL scheme that delegates to the
ClickHouse driver. This allows golang-migrate to natively connect
to Hanzo Datastore instances without URL rewriting in scripts.
2026-03-03 09:15:21 -08:00
dependabot[bot]andGitHub 1620c56290 Bump go.opentelemetry.io/otel/sdk
Bumps the go_modules group with 1 update in the / directory: [go.opentelemetry.io/otel/sdk](https://github.com/open-telemetry/opentelemetry-go).


Updates `go.opentelemetry.io/otel/sdk` from 1.36.0 to 1.40.0
- [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases)
- [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.36.0...v1.40.0)

---
updated-dependencies:
- dependency-name: go.opentelemetry.io/otel/sdk
  dependency-version: 1.40.0
  dependency-type: indirect
  dependency-group: go_modules
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-03-01 20:47:32 +00:00
Dale HuiandGitHub 257fa847d6 Merge pull request #1320 from achanda/upgrade-pgxv5
Upgrade pgx v5 to latest version
2025-12-14 15:16:48 -08:00
001b70ac7d ci: update golangci-lint to v2 (#1341)
* ci: update golangci-lint to v2

update method:
```
golangci-lint migrate
golangci-lint run --fix
```

and fix the last ones manually

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-14 15:14:46 -08:00
leonklingeleandGitHub 89e308c340 chore: remove dependency on "hashicorp/go-multierror" (#1322)
* feat: remove dependency on "hashicorp/go-multierror"

It seems the project has been unmainted for quite some time already,
see for example https://github.com/hashicorp/go-multierror/issues/97
and https://github.com/hashicorp/go-multierror/issues/98.
This also removes an uneccessary dependency which slims down the go.mod.

Go as of version 1.20 supports appending errors by specifying
multiple "%w" formatters in the "fmt.Errorf"[^0] function.

[^0]: https://pkg.go.dev/fmt#Errorf

* feat: use errors.Join to combine multiple errors

As an update to the previous commit 59f084ea3be2e7ec9d912c1a311402b1e4c40df6,
use "errors.Join"[^0] instead of multiple "%w" formatters for "fmt.Errorf".

[^0]: https://pkg.go.dev/errors#Join

* chore: bring back unused util.go file as removing it is a breaking change
2025-11-29 11:37:42 -08:00
Abhishek Chanda fed8fe9f0a Upgrade pgx v5 to latest version 2025-11-27 20:13:25 -06:00
Dale HuiandGitHub 472ef2e75c Merge pull request #1336 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.45.0
Bump golang.org/x/crypto from 0.43.0 to 0.45.0
2025-11-27 00:56:21 -08:00
dependabot[bot]andGitHub 8d7625987a Bump golang.org/x/crypto from 0.43.0 to 0.45.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.43.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-27 08:05:11 +00:00
9f9df7c897 chore: Update cloud.google.com/go/spanner version (#1330)
* chore: Update cloud.google.com/go/spanner version

* Run `go mod tidy`

---------

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2025-11-27 00:01:15 -08:00
Dale HuiandGitHub a371c8ea2b Merge pull request #1304 from iamonah/fix/clarify-databaseName-meaning
Rename databaseName to databaseDriverName
2025-11-26 22:52:08 -08:00
Dale HuiandGitHub 43cc3b3ea5 Merge pull request #1325 from HaraldNordgren/linter_issues
fix linter issues
2025-11-26 22:38:12 -08:00
Dale HuiandGitHub f939a89925 Merge pull request #1335 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.45.0
Bump golang.org/x/crypto from 0.36.0 to 0.45.0
2025-11-26 22:30:24 -08:00
dependabot[bot]andGitHub 6dd86e0ded Bump golang.org/x/crypto from 0.36.0 to 0.45.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.36.0 to 0.45.0.
- [Commits](https://github.com/golang/crypto/compare/v0.36.0...v0.45.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.45.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-20 01:45:37 +00:00
Dale HuiandGitHub 70e6d6d574 Merge pull request #1333 from matsoob/updateGoInBaseImage
Update docker image to use go 1.25, addressing vulnerabilities
2025-11-18 23:04:52 -08:00
Dale HuiandGitHub a51d0da644 Merge pull request #1334 from golang-migrate/dependabot/go_modules/github.com/dvsekhvalnov/jose2go-1.7.0
Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0
2025-11-18 23:01:27 -08:00
dependabot[bot]andGitHub 3f583f0b61 Bump github.com/dvsekhvalnov/jose2go from 1.6.0 to 1.7.0
Bumps [github.com/dvsekhvalnov/jose2go](https://github.com/dvsekhvalnov/jose2go) from 1.6.0 to 1.7.0.
- [Commits](https://github.com/dvsekhvalnov/jose2go/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/dvsekhvalnov/jose2go
  dependency-version: 1.7.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-11-14 21:46:48 +00:00
matsoob ce3cb727b6 Update docker image 2025-11-13 23:35:48 +00:00
Harald Nordgren 1ee8337800 revert GenerateAdvisoryLockID 2025-11-11 09:42:53 +01:00
Harald Nordgren 952d08e30b fix linter issues 2025-11-09 16:54:45 +01:00
Dale HuiandGitHub 39ab04d659 Update goreleaser Go version from 1.24.x to 1.25.x 2025-11-08 21:48:01 -08:00
Dale HuiandGitHub 8b9c5f7712 Merge pull request #1310 from dhui/update_go
Add support for Go 1.25 and drop support for 1.23
2025-08-23 20:27:53 -07:00
Dale Hui b4ec9bccb1 Add support for Go 1.25 and drop support for 1.23 2025-08-23 13:42:01 -07:00
ed4bdd4614 Ensure bufferWriter is always closed in Migration.Buffer and propagate close errors (#1308)
* Ensure bufferWriter is always closed and error is returned in Buffer()

* Ensure bufferWriter is always closed and error is returned in Buffer()

* addressed review comments. used named error and joined error to return final error

* Fixed defer block

---------

Co-authored-by: Chandra kant <chandra.kant@cohesity.com>
2025-08-23 13:12:17 -07:00
Roman SharkovandGitHub 8945e853c4 Merge pull request #1303 from romshark/master
refactor: Remove go.uber.org/atomic in favor of std sync/atomic
2025-08-20 16:35:25 -07:00
Dale HuiandGitHub 7108d806dd Merge pull request #1309 from dhui/dktest_v0.4.6
Update dktest to v0.4.6 for docker vuln fix
2025-08-19 16:24:09 -07:00
Dale HuiandGitHub 682016f04c Merge pull request #1277 from sandhilt/doc/change-apt-key-to-gpg
Update apt-key to gpg
2025-08-19 14:22:16 -07:00
Dale Hui f3e6b5a737 Replace usage of deprecated docker types
This is a backwards incompatible change,
but there don't seem to be any public imports:
https://pkg.go.dev/github.com/golang-migrate/migrate/v4@v4.18.3/testing?tab=importedby
2025-08-19 14:03:55 -07:00
Dale Hui 0a17402aa2 Update dktest to v0.4.6 for docker vuln fix 2025-08-19 13:27:40 -07:00
Dale HuiandGitHub 5eee0c8030 Merge pull request #1299 from golang-migrate/dependabot/go_modules/golang.org/x/oauth2-0.27.0
Bump golang.org/x/oauth2 from 0.18.0 to 0.27.0
2025-08-19 13:15:26 -07:00
0xOnah 30690d7df3 Rename databaseName to databaseDriverName 2025-08-03 21:30:31 +01:00
dependabot[bot]andGitHub 642a24d61b Bump golang.org/x/oauth2 from 0.18.0 to 0.27.0
---
updated-dependencies:
- dependency-name: golang.org/x/oauth2
  dependency-version: 0.27.0
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-07-18 18:07:02 +00:00
sandhilt 5300864bba fix(docs): update Linux installation instructions for GPG key handling 2025-05-13 15:05:40 -03:00
Dale HuiandGitHub 278833935c Merge pull request #1186 from urbim/bugfix/sqlserverLockFix
Fixed sqlserver not actually getting a lock if lock is already set
2025-04-25 13:14:27 -07:00
Urban Malc 6b9f24bf7f Fixed sqlserver not actually getting a lock if lock is already set 2025-04-24 12:14:21 +02:00
Dale HuiandGitHub 9023d66a0b Merge pull request #1244 from alexandear-org/chore-redundant-build-tags
Remove redundant build tags
2025-04-17 15:21:39 -07:00
Dale HuiandGitHub 10494908af Merge pull request #1179 from lunfel/master
Update migrate -help output for the readme file
2025-04-17 15:17:59 -07:00
Dale Hui 033835ac1b Update to dktest v0.4.5 2025-04-17 14:58:46 -07:00
Dale HuiandGitHub 329152ed62 Merge pull request #1196 from Rambatino/patch-1
fix: typo limited not limitted
2025-04-17 13:59:24 -07:00
Oleksandr Redko 34c2b4ad9c Remove redundant build tags 2025-04-17 23:57:28 +03:00
Dale HuiandGitHub a3b7633246 Merge pull request #1259 from golang-migrate/dependabot/go_modules/golang.org/x/net-0.38.0
Bump golang.org/x/net from 0.33.0 to 0.38.0
2025-04-17 00:12:46 -07:00
dependabot[bot]andGitHub 5b97c921ed Bump golang.org/x/net from 0.33.0 to 0.38.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.33.0 to 0.38.0.
- [Commits](https://github.com/golang/net/compare/v0.33.0...v0.38.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-version: 0.38.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-04-17 02:40:34 +00:00
Dale HuiandGitHub 3c3ce91f29 Merge pull request #1258 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.35.0
Bump golang.org/x/crypto from 0.31.0 to 0.35.0
2025-04-16 19:39:14 -07:00
dependabot[bot]andGitHub f37ef795db Bump golang.org/x/crypto from 0.31.0 to 0.35.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.31.0 to 0.35.0.
- [Commits](https://github.com/golang/crypto/compare/v0.31.0...v0.35.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-version: 0.35.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-04-16 22:36:30 +00:00
Dale HuiandGitHub 1af841d787 Merge pull request #1260 from dhui/update_go
Update supported Go versions and README
2025-04-16 15:35:09 -07:00
Dale Hui 7269490309 Update golangci-lint version used in GitHub Actions 2025-04-16 15:12:33 -07:00
Dale Hui fccd197e40 Mention CLI install instructions in main README
Fixes:
* https://github.com/golang-migrate/migrate/issues/973
* https://github.com/golang-migrate/migrate/issues/1020
2025-04-16 14:43:17 -07:00
Dale Hui e6d84f6f80 Drop support for Go 1.22 and add support for Go 1.24
Drop `toolchain` module directive to favor default/auto behavior
See also: https://github.com/golang-migrate/migrate/pull/1183
2025-04-16 14:38:52 -07:00
Dale HuiandGitHub 146bfa05b3 Merge pull request #1241 from hazzik/refactor/pkg/errors
refactor: replace github.com/pkg/errors with stdlib
2025-04-16 14:04:34 -07:00
Dale HuiandGitHub 604248c97e Merge pull request #1250 from golang-migrate/dependabot/go_modules/github.com/golang-jwt/jwt/v4-4.5.2
Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2
2025-03-23 16:54:14 -07:00
dependabot[bot]andGitHub a4d0a1b8d9 Bump github.com/golang-jwt/jwt/v4 from 4.5.1 to 4.5.2
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.1 to 4.5.2.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.1...v4.5.2)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-03-21 22:40:35 +00:00
Alex Zaytsev 60d73be14e refactor: replace github.com/pkg/errors with stdlib 2025-03-10 13:54:11 +10:00
Jochen SchalandaandGitHub 36d17ba2ec tests: fix various tests (#1209)
* Remove old versions of MongoDB

Removed:

| Release     | Release Date  | End of Life Date |
|-------------|---------------|------------------|
| MongoDB 3.4 | November 2016 | January 2020     |
| MongoDB 3.6 | November 2017 | April 2021       |
| MongoDB 4.0 | June 2018     | April 2022       |
| MongoDB 4.2 | August 2019   | April 2023       |

Added:

| Release     | Release Date  | End of Life Date |
|-------------|---------------|------------------|
| MongoDB 5.0 | July 2021     | October 2024     |
| MongoDB 6.0 | July 2022     | July 2025        |
| MongoDB 7.0 | August 2023   | August 2026      |
| MongoDB 8.0 | October 2024  | TBD              |

https://www.mongodb.com/legal/support-policy/lifecycles

* test(postgres): run tests with PostgreSQL 17

Add PostgreSQL 17, remove PostgreSQL 12 (EOL since November 21, 2024).

https://www.postgresql.org/support/versioning/

* test(cockroachdb): remove old versions of CockroachDB in tests

Removed:

|       Release       |                Released                |              Maintenance Support             |              Assistance Support              |         Latest        |
|:-------------------:|:--------------------------------------:|:--------------------------------------------:|:--------------------------------------------:|:---------------------:|
| 2.1                 | 6 years ago (30 Oct 2018)              | Ended 5 years ago (30 Oct 2019)              | Ended 4 years and 7 months ago (30 Apr 2020) | 2.1.11 (22 Jan 2020)  |
| 2.0                 | 6 years and 8 months ago (04 Apr 2018) | Ended 5 years and 8 months ago (04 Apr 2019) | Ended 5 years ago (04 Oct 2019)              | 2.0.7 (28 Nov 2018)   |
| 1.1                 | 7 years ago (12 Oct 2017)              | Ended 6 years ago (12 Oct 2018)              | Ended 5 years and 8 months ago (12 Apr 2019) | 1.1.9 (01 Oct 2018)   |
| 1.0                 | 7 years ago (10 May 2017)              | Ended 6 years ago (10 May 2018)              | Ended 6 years ago (10 Nov 2018)              | 1.0.7 (11 Feb 2018)   |

Added:

|       Release       |                Released                |              Maintenance Support             |              Assistance Support              |         Latest        |
|:-------------------:|:--------------------------------------:|:--------------------------------------------:|:--------------------------------------------:|:---------------------:|
| 24.3                | 3 weeks ago (18 Nov 2024)              | Ends in 11 months (18 Nov 2025)              | Ends in 1 year and 5 months (18 May 2026)    | 24.3.0 (20 Nov 2024)  |
| 24.2                | 3 months and 4 weeks ago (12 Aug 2024) | Ends in 2 months (12 Feb 2025)               | Ends in 2 months (12 Feb 2025)               | 24.2.5 (11 Nov 2024)  |
| 24.1 (Upcoming LTS) | 6 months and 3 weeks ago (20 May 2024) | Ends in 5 months (20 May 2025)               | Ends in 11 months (20 Nov 2025)              | 24.1.7 (11 Nov 2024)  |
| 23.2 (Upcoming LTS) | 10 months ago (05 Feb 2024)            | Ends in 1 month and 4 weeks (05 Feb 2025)    | Ends in 7 months and 4 weeks (05 Aug 2025)   | 23.2.16 (13 Nov 2024) |

* test(firebird): remove old versions of Firebird in tests

Removed: Firebird 2.5, see https://firebirdsql.org/en/discontinued-versions/

Added:
- Firebird 4.0 (https://firebirdsql.org/en/firebird-4-0-5/)
- Firebird 5.0 (https://firebirdsql.org/en/firebird-5-0-1/)

* test(clickhouse): run ClickHouse tests with official LTS image (24.8)
2025-02-19 16:19:12 -08:00
Dale HuiandGitHub 24a6476044 Merge pull request #1225 from stensonb/patch-1
Update FAQ.md - typo
2025-02-19 16:06:49 -08:00
Bryan StensonandGitHub a8680333c3 Update FAQ.md - typo 2025-02-01 23:26:02 -08:00
Dale HuiandGitHub d477553dcd Merge pull request #1195 from golang-migrate/dependabot/go_modules/github.com/golang-jwt/jwt/v4-4.5.1
Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.5.1
2025-01-26 17:53:05 -08:00
dependabot[bot]andGitHub e145cded4a Bump github.com/golang-jwt/jwt/v4 from 4.4.2 to 4.5.1
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.4.2 to 4.5.1.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.4.2...v4.5.1)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-01-27 01:06:51 +00:00
Dale Hui 7651c8a37e linter fixes 2025-01-26 17:04:55 -08:00
Dale Hui e5a152b9f6 Drop support for Azure SQL Edge
Will be retired on 9/30/2025 and CI tests using the docker image have been flakey
https://azure.microsoft.com/en-us/updates?id=azure-sql-edge-retirement
2025-01-26 16:43:31 -08:00
Dale HuiandGitHub 12c619ee47 Fix CI (#1222)
* Run go mod tidy

* Fix Github linting action

Changes:
* checkout before setting up go
* pin golangci-lint version

* Update supported db versions

* Use start-single-node instead of start for cockroachdb
2025-01-26 15:46:11 -08:00
Dale Hui bc06922b4d Update dktest from v0.4.3 to v0.4.4 2025-01-25 21:05:33 -08:00
Mark RamotowskiandGitHub 8b091919d4 fix: typo limited not limitted 2024-11-28 09:28:14 +00:00
Mathieu Tanguay c5137c417c Update migrate -help output for the readme file 2024-10-02 10:41:33 -04:00
Dale HuiandGitHub c378583d78 Merge pull request #1162 from rselbach/rselbach/no-sensitive-info
Don't output sensitive information on error
2024-09-23 10:45:56 -07:00
Roberto Selbach e22d0128cc Don't output sensitive information on error 2024-09-12 12:12:34 -04:00
Dale Hui 555501f7cd Update dktest from v0.4.2 to v0.4.3 2024-09-08 14:02:51 -07:00
Dale HuiandGitHub d5eb594426 Merge pull request #1101 from adityassharma-ss/patch-1
CORRECTION DONE near m.Step(2) if you want to explicitly set the number of migrations to run
2024-09-08 11:42:13 -07:00
Dale HuiandGitHub 67c71f9a9d Merge pull request #1152 from golang-migrate/dependabot/go_modules/google.golang.org/grpc-1.64.1
Bump google.golang.org/grpc from 1.64.0 to 1.64.1
2024-09-08 11:34:22 -07:00
Dale HuiandGitHub 1e02b42a7a Merge pull request #1119 from muzammilar/add-noble-numbat
Support .deb package for Noble Numbat (24.04)
2024-09-07 23:07:01 -07:00
dependabot[bot]andGitHub 5b6f623dec Bump google.golang.org/grpc from 1.64.0 to 1.64.1
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.64.1.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.64.0...v1.64.1)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-09-08 06:04:50 +00:00
Dale HuiandGitHub ddd7fa61da Merge pull request #1127 from joschi/tests-mysql-9
test(mysql): run tests with MySQL 8.0, 8.4, and 9.0
2024-09-07 22:59:58 -07:00
Dale HuiandGitHub 7e8f6be051 Merge pull request #1126 from joschi/tests-postgres-16
test(postgres): run tests with PostgreSQL 16
2024-09-07 22:59:25 -07:00
Dale HuiandGitHub 65a3bd56cc Merge pull request #1153 from joschi/fix-docker-imports
fix(tests): fix Docker imports
2024-09-07 22:58:32 -07:00
Dale HuiandGitHub e40e64c4ac Merge pull request #1133 from joschi/go-1.23
build: use Go 1.23 to build the project
2024-09-07 22:51:54 -07:00
Jochen Schalanda 6ceb5a9467 fix(tests): fix Docker imports 2024-09-07 22:50:55 +02:00
Jochen Schalanda 0444ec8531 test(postgres): run tests with PostgreSQL 16 2024-09-07 21:47:48 +02:00
Jochen Schalanda f4d986b9d9 test(mysql): run tests with MySQL 8.0, 8.4, and 9.0 2024-09-07 21:47:28 +02:00
Jochen Schalanda ee93fd4aff build: use Go 1.23 to build the project
https://tip.golang.org/doc/go1.23
2024-09-07 21:45:48 +02:00
Dale Hui d28e549a68 Update dktest from v0.4.1 to v0.4.2 2024-09-07 00:01:37 -07:00
Jochen SchalandaandGitHub 897be38496 chore: remove deprecated rand.Seed() in testing.docker (#1149)
* chore: remove deprecated `rand.Seed()` in testing.docker

* Use math/rand/v2

* Remove Go 1.21

Go 1.21 doesn't support `math/rand/v2`.
2024-09-06 23:46:24 -07:00
Dale Hui 1b1c6f6d14 Use newer math/rand/v2 2024-09-06 23:32:35 -07:00
Dale HuiandGitHub f44379d6f1 Merge pull request #1151 from dhui/fix-redshift-tests
Fix redshift tests
2024-09-06 00:41:25 -07:00
Dale Hui e1a7e31b1d Use forked version of postgres 8 to avoid Docker image format version issues.
See: https://docs.docker.com/engine/deprecated/#pushing-and-pulling-with-image-manifest-v2-schema-1

Forked images created by pulling the original image and pushing to the fork. For more details, see: https://github.com/golang-migrate/migrate/pull/1143#issuecomment-2328069381
2024-09-05 23:28:26 -07:00
Dale Hui 899a244399 Fix redshift tests
Newer versions of postgres require a password
2024-09-05 23:26:56 -07:00
Muzammil Abdul Rehman 41e389a078 Support .deb package for Noble Numbat (24.04) 2024-07-17 12:56:34 -07:00
Dale HuiandGitHub 2477f639fd Merge pull request #1098 from gjabell/master
Upgrade go-sqlite3 to v1.14.22
2024-06-10 12:46:13 -07:00
Dale HuiandGitHub 1585b870f8 Merge pull request #1103 from Shion1305/shion/issue-1102
🧑‍💻 improve error message for invalid source, database, resolves: #1102
2024-06-10 12:34:31 -07:00
Dale HuiandGitHub b16220c938 Merge pull request #1105 from Shion1305/shion/issue-1104
#1104 Fix golanglint-ci lint config & lint errors
2024-06-10 12:28:15 -07:00
Shion Ichikawa 34594af0ab 🚨 golangci-lint: fix for errcheck lint errors 2024-06-05 23:40:12 +09:00
Shion Ichikawa 1735cb3920 🚨 golangci-lint: disable interfacer linter 2024-06-05 23:39:14 +09:00
Shion Ichikawa cabb2db836 🧑‍💻 improve error message for invalid source, database, resolves: #1102 2024-06-05 22:44:03 +09:00
Aditya SharmaandGitHub da72428fa0 Update README.md
if you want to explicitly set the number of migrations to run, the README.md needs a correction, there should "Steps()" and not Step()
2024-06-04 19:25:09 +05:30
Galen Abell c81eaf7918 Upgrade go-sqlite3 to v1.14.22 2024-05-31 17:30:06 +02:00
Dale HuiandGitHub 04b36ebaac Merge pull request #1070 from certifaction/mysql-safe-update
Make MySQL SetVersion compatible with sql_safe_update
2024-04-29 21:59:22 -07:00
Dale HuiandGitHub 45e4f1ef2f Merge pull request #1074 from golang-migrate/dependabot/go_modules/golang.org/x/net-0.23.0
Bump golang.org/x/net from 0.21.0 to 0.23.0
2024-04-29 21:55:07 -07:00
dependabot[bot]andGitHub 859d92bd77 Bump golang.org/x/net from 0.21.0 to 0.23.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.21.0 to 0.23.0.
- [Commits](https://github.com/golang/net/compare/v0.21.0...v0.23.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-19 13:19:09 +00:00
Mateusz Nowotyński e428ecebea Make MySQL SetVersion compatible with sql_safe_update 2024-04-19 09:34:12 +02:00
Dale HuiandGitHub 0c456c49a6 Merge pull request #1068 from goodfirm/master
chore: fix some typos in comments
2024-04-16 23:16:25 -07:00
Dale Hui f100226758 Update dktest from v0.4.0 to v0.4.1 to fix docker vulnerability 2024-04-16 23:03:36 -07:00
Dale HuiandGitHub c52377528f Merge pull request #1055 from golang-migrate/dependabot/go_modules/github.com/jackc/pgx/v5-5.5.4
Bump github.com/jackc/pgx/v5 from 5.3.1 to 5.5.4
2024-04-16 22:57:30 -07:00
dependabot[bot]andGitHub a78d1abfa8 Bump github.com/jackc/pgx/v5 from 5.3.1 to 5.5.4
Bumps [github.com/jackc/pgx/v5](https://github.com/jackc/pgx) from 5.3.1 to 5.5.4.
- [Changelog](https://github.com/jackc/pgx/blob/master/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v5.3.1...v5.5.4)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v5
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-16 20:43:15 +00:00
Dale HuiandGitHub 837776f0c7 Merge pull request #1054 from golang-migrate/dependabot/go_modules/google.golang.org/protobuf-1.33.0
Bump google.golang.org/protobuf from 1.31.0 to 1.33.0
2024-04-16 13:38:31 -07:00
Dale HuiandGitHub 128b6509cc Merge pull request #1050 from golang-migrate/dependabot/go_modules/github.com/jackc/pgx/v4-4.18.2
Bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2
2024-04-16 13:38:07 -07:00
dependabot[bot]andGitHub 2e0872f3e8 Bump google.golang.org/protobuf from 1.31.0 to 1.33.0
Bumps google.golang.org/protobuf from 1.31.0 to 1.33.0.

---
updated-dependencies:
- dependency-name: google.golang.org/protobuf
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-16 16:19:02 +00:00
dependabot[bot]andGitHub d1df97b2d1 Bump github.com/jackc/pgx/v4 from 4.18.1 to 4.18.2
Bumps [github.com/jackc/pgx/v4](https://github.com/jackc/pgx) from 4.18.1 to 4.18.2.
- [Changelog](https://github.com/jackc/pgx/blob/v4.18.2/CHANGELOG.md)
- [Commits](https://github.com/jackc/pgx/compare/v4.18.1...v4.18.2)

---
updated-dependencies:
- dependency-name: github.com/jackc/pgx/v4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-04-16 16:18:25 +00:00
Tim RühsenandGitHub 1a002d0e9e Set golangci-lint to 1.54.2 (latest is broken) (#1046)
* Set golangci-lint to 1.54.2 (latest is broken)

* Testing golangci-lint latest (1.57.1)
2024-04-15 23:54:05 -07:00
Dale HuiandGitHub 2c5df8742b Merge pull request #1072 from dhui/dktesting-cleanup
Delete db docker images after tests complete for a few large db docker images
2024-04-15 23:44:37 -07:00
Dale Hui f4950c144e Fallback to dktest.DefaultCleanupTimeout if the dktest.Options doesn't have one specified 2024-04-15 22:57:23 -07:00
Dale Hui b1d02e222f Cleanup sqlserver images after tests run 2024-04-15 22:42:34 -07:00
Dale Hui 49cac86ab2 Cleanup mongodb images after tests run 2024-04-15 00:35:46 -07:00
Dale Hui 06614d9cf5 Cleanup yugabytedb images after tests run 2024-04-15 00:25:40 -07:00
Dale Hui 1b707a76f4 Cleanup cassandra images after tests run 2024-04-15 00:18:13 -07:00
Dale Hui 2884a8e851 Cleanup postgres images after tests run 2024-04-15 00:17:36 -07:00
Dale Hui 4bc6777f19 Add dktesting.Cleanup() method 2024-04-15 00:09:23 -07:00
goodfirm 9d70a397e4 chore: fix some typos in comments
Signed-off-by: goodfirm <fanyishang@yeah.net>
2024-04-10 14:36:41 +08:00
Dale Hui e9133367d7 Drop support for Go 1.20 and add support for Go 1.22 2024-03-24 22:35:06 -07:00
Dale HuiandGitHub 87ba13c398 Merge pull request #1061 from occupyhabit/master
chore: fix some typos
2024-03-24 22:31:39 -07:00
Dale Hui ff8a961068 Update yugabyte test images 2024-03-24 22:30:38 -07:00
occupyhabit 7f85f9cf64 chore: fix some typos
Signed-off-by: occupyhabit <wangmengjiao@outlook.com>
2024-03-23 21:49:28 +08:00
Dale HuiandGitHub a5dc5d1efc Merge pull request #1025 from otoolep/patch-1
rqlite is spelled with all lowercase
2024-01-10 20:56:10 -08:00
Philip O'TooleandGitHub 94b8fa5946 rqlite is spelled with all lowercase
I'm the creator of rqlite, so wanted to fix the title. See https://rqlite.io/
2024-01-05 15:34:45 -05:00
Dale HuiandGitHub 0d41589774 Merge pull request #1022 from golang-migrate/dependabot/go_modules/github.com/dvsekhvalnov/jose2go-1.6.0
Bump github.com/dvsekhvalnov/jose2go from 1.5.0 to 1.6.0
2024-01-02 12:48:02 -08:00
dependabot[bot]andGitHub a860f0cbd4 Bump github.com/dvsekhvalnov/jose2go from 1.5.0 to 1.6.0
Bumps [github.com/dvsekhvalnov/jose2go](https://github.com/dvsekhvalnov/jose2go) from 1.5.0 to 1.6.0.
- [Commits](https://github.com/dvsekhvalnov/jose2go/compare/v1.5...v1.6.0)

---
updated-dependencies:
- dependency-name: github.com/dvsekhvalnov/jose2go
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-31 22:22:36 +00:00
Dale HuiandGitHub 516f038f30 Merge pull request #840 from alex-kuck/fix/839/sqlserver_version_table
[sqlserver] Ensure version table in provided schema
2023-12-31 13:59:38 -08:00
Alex Kucksdorf 0350a00606 [sqlserver] Always access version table with explicit schema 2023-12-25 10:29:27 +01:00
Alex Kucksdorf 814769373f [sqlserver] Ensure version table in provided schema
Closes #839
2023-12-25 10:29:26 +01:00
Dale HuiandGitHub 0815e2d770 Merge pull request #992 from fortnox-andreas/master
Add support for locking table in pgx-driver
2023-12-20 14:13:36 -08:00
Dale Hui 5aa4670cbb Fix GoReleaser deprecations
https://goreleaser.com/deprecations/#-rm-dist
https://goreleaser.com/deprecations/#sourcerlcp
2023-12-20 13:12:07 -08:00
Dale Hui d63a5c2c28 Only test against YugabyteDB LTS releases 2023-12-20 12:49:33 -08:00
Dale Hui cd17c5a808 Drop support for Go 1.19 and add support for Go 1.21 2023-12-20 11:37:41 -08:00
Dale Hui f2c4b52679 Update aws-sdk-go from v1.44.301 to v1.49.6 2023-12-20 11:33:09 -08:00
Dale HuiandGitHub c7c50118a0 Merge pull request #956 from Kenai/master
Update aws-sdk-go to adress vulerabilitiy
2023-12-20 11:29:57 -08:00
Dale HuiandGitHub 9c551d4398 Merge pull request #875 from no-name16/clickhouse_create_database_migrations
clickhouse: add to README.md and examples database creation
2023-12-20 11:24:13 -08:00
Tommy Karlsson 091ad5db99 Quote locktable from config in queries
Defer rollback of transactions
2023-12-20 16:10:29 +01:00
Dale HuiandGitHub ab24e7674d Merge branch 'master' into clickhouse_create_database_migrations 2023-12-20 00:33:47 -08:00
Dale HuiandGitHub e8edcdce66 Merge branch 'master' into master 2023-12-20 00:30:20 -08:00
Dale Hui 691f687eb8 Reformat ScyllaDB/Cassandra docs 2023-12-20 00:28:14 -08:00
Dale HuiandGitHub eb64ffa1ad Merge pull request #947 from mkorolyov/scylladb_support
add tests for scylladb. add scylladb to docs
2023-12-20 00:19:19 -08:00
Dale HuiandGitHub a94396c3a8 Merge pull request #857 from luca-nardelli/master
clickhouse: Quote db name in ensureVersionTable
2023-12-20 00:15:38 -08:00
Dale HuiandGitHub 0ba6fc3166 Merge pull request #1002 from jsabbatini-upguard/upgrade-spanner
Updated version of spanner to support sequences and generate uuid
2023-12-19 23:10:13 -08:00
Dale HuiandGitHub 20636841a9 Merge branch 'master' into master 2023-12-19 23:01:18 -08:00
Dale HuiandGitHub 0695426fbf Merge pull request #929 from SuperSandro2000/patch-1
Move supported go version to standard place
2023-12-19 22:58:49 -08:00
Dale HuiandGitHub 1d1cd48953 Merge branch 'master' into scylladb_support 2023-12-19 22:56:40 -08:00
Dale HuiandGitHub 76efa726ed Merge branch 'master' into master 2023-12-19 22:53:04 -08:00
Dale HuiandGitHub 90c5015954 Merge pull request #959 from jorng/libpq-update
Update lib/pq to fix cert permissions issues
2023-12-19 22:50:16 -08:00
Dale Hui 5026488530 Clean up require directive grouping 2023-12-19 22:41:22 -08:00
Dale Hui fb224367d7 Merge remote-tracking branch 'origin/master' into upgrade-spanner 2023-12-19 22:39:19 -08:00
Dale HuiandGitHub 47a2661dd4 Merge pull request #1015 from golang-migrate/dependabot/go_modules/google.golang.org/grpc-1.56.3
Bump google.golang.org/grpc from 1.55.0 to 1.56.3
2023-12-19 22:03:03 -08:00
Dale HuiandGitHub b39ee92ed9 Merge pull request #999 from tobyscott25/patch-1
Add syntax highlighting to Postgres example
2023-12-19 21:38:57 -08:00
dependabot[bot]andGitHub c3ebd527d8 Bump google.golang.org/grpc from 1.55.0 to 1.56.3
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.55.0 to 1.56.3.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](https://github.com/grpc/grpc-go/compare/v1.55.0...v1.56.3)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-20 05:37:50 +00:00
Dale HuiandGitHub f375aebc5c Merge pull request #1007 from swensone/master
Add support for rqlite
2023-12-19 21:35:18 -08:00
Dale HuiandGitHub 834fa39423 Merge pull request #1014 from golang-migrate/dependabot/go_modules/golang.org/x/crypto-0.17.0
Bump golang.org/x/crypto from 0.14.0 to 0.17.0
2023-12-19 21:15:31 -08:00
Erik Swenson bfedabbc35 Merge remote-tracking branch 'upstream/master' 2023-12-19 10:41:12 -07:00
Erik Swenson 7d03609443 add 8.11 and 8.12 versions and remove debug logging 2023-12-19 10:33:43 -07:00
dependabot[bot]andGitHub 5ded96d97d Bump golang.org/x/crypto from 0.14.0 to 0.17.0
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.14.0 to 0.17.0.
- [Commits](https://github.com/golang/crypto/compare/v0.14.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/crypto
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-12-19 08:40:00 +00:00
Dale Hui b567287637 Update from alpine 3.18 to 3.19 2023-12-19 00:02:27 -08:00
Dale Hui 4078ef8941 New release prep
* Update dktest from v0.3.16 to v0.4.0 to fix docker vulnerability
* Fix linter issues
* Update outdated GitHub Actions
2023-12-18 23:44:05 -08:00
Dale HuiandGitHub 7eac9190d4 Merge pull request #1006 from testwill/typo
fix: typo
2023-12-18 21:58:57 -08:00
Erik Swenson 669437c3b8 update rqlite 8 container version to 8.0.6 2023-12-14 09:13:13 -07:00
Erik Swenson cf03803fe0 Add rqlite 8.0.0 to tested database versions 2023-12-06 16:02:36 -07:00
Erik Swenson 5163ac7824 feature: add rqlite support 2023-11-27 17:05:24 -07:00
guoguangwu ee8a8e50a7 fix: typo
Signed-off-by: guoguangwu <guoguangwu@magic-shield.com>
2023-11-27 13:45:49 +08:00
Jacopo Sabbatini 72957b6966 Updated version of spanner to support sequences and generate uuid 2023-11-14 15:04:55 +11:00
Toby ScottandGitHub 12968a75af Add syntax highlighting to Postgres example 2023-11-04 12:24:32 +11:00
Tommy Karlsson bead4a952b Added documentation and test for lock strategy 2023-09-20 14:27:26 +02:00
Andreas Emilsson 129922aae3 Added support for pgx locking table
In order to support running migrations through PgBouncer which does not
support advisory locks.
2023-09-20 09:59:55 +02:00
Rob PrentissandRob Prentiss f2e0b335d7 Update lib/pq to fix cert permissions issues
When attempting to connect to a PostgreSQL database using certificate
credentials, authentication may fail due to permissions issues on the
certificate files. When using migrate in Kubernetes, this issue may be
unavoidable when using secrets.

The github.com/lib/pq library has resolved this issue as of v1.10.6, so
this commit updates that library to the latest release version (v1.10.9)
to resolve the issue in migrate.
2023-07-21 12:00:31 -07:00
Jan Śreniawski 876a13d423 Update aws-sdk-go to adress vulerabilitiy 2023-07-18 09:00:47 +02:00
Maxim Korolyov f8afa5a6f6 small changes to retry failed by timeout CI 2023-07-17 22:41:12 +02:00
Maxim KorolyovandGitHub 45d23ed92a Merge branch 'master' into scylladb_support 2023-07-11 09:16:48 +02:00
Dale HuiandGitHub 856ea12df9 Merge pull request #953 from dhui/update_spanner
Update spanner to fix security issue
2023-07-05 23:05:13 -07:00
Dale Hui 208ac532e8 Update spanner to fix security issue
See also: https://github.com/golang-migrate/migrate/pull/952
2023-07-05 22:18:03 -07:00
Dale HuiandGitHub bad30b549d Mention migradaptor
migradaptor can be used to migrate from other db migration tools to golang-migrate

https://github.com/golang-migrate/migrate/issues/940#issuecomment-1611712379
2023-07-02 22:22:38 -07:00
Maxim Korolyov 7a72550614 add tests for scylladb. add scylladb to docs 2023-06-25 20:30:21 +02:00
Dale HuiandGitHub 691bc773cc Merge pull request #936 from golang-migrate/dependabot/go_modules/github.com/snowflakedb/gosnowflake-1.6.19
Bump github.com/snowflakedb/gosnowflake from 1.6.3 to 1.6.19
2023-06-11 14:36:47 -07:00
dependabot[bot]andGitHub 383386b35f Bump github.com/snowflakedb/gosnowflake from 1.6.3 to 1.6.19
Bumps [github.com/snowflakedb/gosnowflake](https://github.com/snowflakedb/gosnowflake) from 1.6.3 to 1.6.19.
- [Release notes](https://github.com/snowflakedb/gosnowflake/releases)
- [Changelog](https://github.com/snowflakedb/gosnowflake/blob/master/release.go)
- [Commits](https://github.com/snowflakedb/gosnowflake/compare/v1.6.3...v1.6.19)

---
updated-dependencies:
- dependency-name: github.com/snowflakedb/gosnowflake
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-06-09 22:58:57 +00:00
Dale Hui 315a205b59 Update Dockerfiles
Use Alpine 3.18 and Go 1.20
2023-06-06 21:33:03 -07:00
Dale Hui 23d8d33f57 Support .deb packages for Ubuntu Jammy Jellyfish and Debian bookworm 2023-06-04 21:52:01 -07:00
SandroandGitHub 92dec355ef Move supported go version to standard place 2023-06-01 11:09:09 +02:00
Dale HuiandGitHub f9c067b3e0 Merge pull request #922 from DariusParvin/patch-1
Fix typo
2023-05-30 22:15:25 -07:00
Dale HuiandGitHub 0b3b628703 Use pgx5 connection string in README 2023-05-30 22:12:01 -07:00
Dale Hui c62dbbacc2 Opt-in to goreleaser's new property
https://goreleaser.com/deprecations/#archivesrlcp
2023-05-27 16:39:56 -07:00
Dale Hui 8d96f81de6 Update mssql to fix 386 and arm7 builds 2023-05-27 16:35:51 -07:00
Dale Hui ea8bebc7ef Downgrade docker 2023-05-20 14:21:01 -07:00
Dale Hui 1020682de5 Update dktest from v0.3.15 to v0.3.16 for upstream security fixes 2023-05-20 00:34:56 -07:00
Darius ParvinandGitHub d72e5c6096 Fix typo
change `stamements` to `statements`
2023-04-26 15:09:50 -07:00
Dale HuiandGitHub 84009cf2ab Merge pull request #914 from soyayaos/master
fix: postgresql search_path can be empty (#696)
2023-04-06 22:49:01 -07:00
Vladimir Galunshchikov ddaf669dc4 fix: postgresql search_path can be empty (#696) 2023-04-06 23:48:03 +02:00
Dale HuiandGitHub e968557bc6 Merge pull request #913 from dhui/update_dktest
Update dktest from v0.3.14 to v0.3.15 for upstream security fixes
2023-04-05 10:33:54 -07:00
Dale Hui 5be1166fa3 Update dktest from v0.3.14 to v0.3.15 for upstream security fixes 2023-04-04 21:58:42 -07:00
Eli TreuherzandGitHub b94ed1a710 Add driver for pgx v5 (#848)
* Add driver for pgx v5

Fixes #827

- Move existing driver into database/pgx/v4
- Make exported items into database/pgx into links to database/pgx/v4,
  marked as Deprecated
- Remove most tests in database/pgx, leaving a couple of e2e tests
  behind to help ensure backwards compatibility.
- Mark defunct ErrDatabaseDirty as Deprecated - no code returns it
- Bump pgx/v4 dependency
- Add test runs for newer Postgres versions

* Tweak pgx tags and registration strings

* Tweak registration strings
2023-04-04 21:57:01 -07:00
Dale HuiandGitHub c34675fa54 Merge pull request #905 from dhui/update-mongodb
Update mongodb driver for security fix
2023-03-25 23:46:32 -07:00
Dale Hui 2c61c69199 Update mongodb driver for security fix
Mongodb driver v1.7.5 no longer depends on packr
https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMGOBUFFALOPACKRV2-1920670
https://github.com/mongodb/mongo-go-driver/pull/815
2023-03-25 21:02:48 -07:00
Dale HuiandGitHub 586c129bae Merge pull request #903 from dhui/update-sqlite3
Update go-sqlite3 for security fix
2023-03-25 01:32:14 -07:00
Dale Hui 2111d2f93e Update go-sqlite3 for security fix
https://security.snyk.io/vuln/SNYK-GOLANG-GITHUBCOMMATTNGOSQLITE3-3121037
2023-03-25 00:05:17 -07:00
GeorgeWheatcroftandGitHub b63a0d4880 fix issue of downloading and using github sourced migration files of … (#900)
* fix issue of downloading and using github sourced migration files of 100MB >=1Mb (which previously failed to download due to their size and the api call performed) by preferring the go-github client library method 'DownloadContents' for such operations (which doesn't have the same limitation)

* empty commit & push to trigger ci
2023-03-24 23:18:40 -07:00
ViktoriaandGitHub 3b02b182ee Correct a spelling mistake 2023-03-19 07:44:50 +01:00
Dale HuiandGitHub 8ec04221b4 Merge pull request #836 from m-chrome/update-pgx-deps
Update pgx deps
2023-03-15 15:05:36 -07:00
Dale HuiandGitHub 8d1322e09a Merge branch 'master' into update-pgx-deps 2023-03-15 14:14:30 -07:00
Dale HuiandGitHub 1d693e8836 Merge pull request #809 from jcuanmhoney/master
Upgrading Spanner version
2023-03-15 11:09:59 -07:00
Dale Hui 4831e0ed50 go-genproto to google-cloud-go message type migration
Followed these steps: https://github.com/googleapis/google-cloud-go/blob/main/migration.md
2023-03-15 09:13:26 -07:00
Dale HuiandGitHub 30b6b9aeac Merge branch 'master' into master 2023-03-15 08:45:02 -07:00
Dale HuiandGitHub 952f3f046e Merge pull request #896 from dhui/go-1.20
Drop support for Go 1.18 and add support for Go 1.20
2023-03-15 00:33:07 -07:00
Dale Hui e497646d9c Drop support for Go 1.18 and add support for Go 1.20 2023-03-14 23:19:03 -07:00
Dale HuiandGitHub 179dba434c Merge pull request #806 from mtsmfm/patch-1
Fix placeholder in download URL
2023-03-14 23:11:20 -07:00
Dale Hui f87b567859 Update dktest from v0.3.13 to v0.3.14 2023-03-14 22:40:05 -07:00
Dale HuiandGitHub 7cacfc2080 Merge pull request #882 from golang-migrate/dependabot/go_modules/github.com/aws/aws-sdk-go-1.34.0
Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0
2023-03-08 22:12:55 -08:00
Dale HuiandGitHub 448a9b1819 Merge pull request #885 from golang-migrate/dependabot/go_modules/golang.org/x/net-0.7.0
Bump golang.org/x/net from 0.1.0 to 0.7.0
2023-03-08 22:12:33 -08:00
Dale HuiandGitHub e7af02e753 Merge pull request #893 from MasterOdin/patch-1
Fix GH workflow badge in README
2023-03-08 22:11:52 -08:00
Matthew PevelerandGitHub e5d8970cd0 Fix GH workflow badge in README
Signed-off-by: Matthew Peveler <matt.peveler@gmail.com>
2023-03-01 10:35:29 -05:00
dependabot[bot]andGitHub fd42ff1729 Bump golang.org/x/net from 0.1.0 to 0.7.0
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.1.0 to 0.7.0.
- [Release notes](https://github.com/golang/net/releases)
- [Commits](https://github.com/golang/net/compare/v0.1.0...v0.7.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-18 03:11:56 +00:00
dependabot[bot]andGitHub 88d30a4217 Bump github.com/aws/aws-sdk-go from 1.33.0 to 1.34.0
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.33.0 to 1.34.0.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/v1.34.0/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.33.0...v1.34.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-02-14 16:27:52 +00:00
Viktoria 64755d0b20 Update README.md 2023-01-24 14:37:31 +03:00
Viktoria 90a3ac4f87 Remove cluster adaptation for tables to pass tests 2023-01-24 13:22:17 +03:00
Viktoria 50112e7650 Add to clickhouse README.md database creation 2023-01-24 12:40:36 +03:00
Dale HuiandGitHub dc26c41ac9 Update security policy
GitHub security advisories have been enabled, so update the security policy accordingly
2023-01-10 23:51:36 -08:00
Dale HuiandGitHub 145baa187d Merge pull request #865 from golang-migrate/dependabot/go_modules/github.com/aws/aws-sdk-go-1.33.0
Bump github.com/aws/aws-sdk-go from 1.17.7 to 1.33.0
2023-01-10 23:33:56 -08:00
dependabot[bot]andGitHub eb59b750e8 Bump github.com/aws/aws-sdk-go from 1.17.7 to 1.33.0
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.17.7 to 1.33.0.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/v1.33.0/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.17.7...v1.33.0)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-30 19:08:10 +00:00
Dale HuiandGitHub c9ff76f294 Merge pull request #841 from tsbkw/fix/conform-to-go-test-naming-convention
fix: Conform to go test naming convention
2022-12-06 00:05:18 -08:00
Dale HuiandGitHub b63177a7da Merge pull request #856 from morigs/fix/ybdb-err-is-retryable-panic
fix(yugabyte): panic on err retry check
2022-12-03 01:08:36 -08:00
Luca Nardelli 839421e733 Leverage quoteIdentifier from pgx 2022-11-30 14:38:22 +01:00
Luca Nardelli 9fe7383564 Quote in drop as well 2022-11-30 10:26:31 +01:00
Luca Nardelli 90273fe0aa clickhouse: Quote db name in ensureVersionTable 2022-11-29 15:46:06 +01:00
tsbkwandGitHub a7f9122726 Merge branch 'master' into fix/conform-to-go-test-naming-convention 2022-11-28 08:18:35 +09:00
Igor Morozov e7e707727d fix(yugabyte): panic on err retry check 2022-11-23 19:05:34 +04:00
TOGASHI TomokiandGitHub 31791adbc0 fix(spanner): update spanner module to fix parsing issue (#843)
* update spanner module to fix spanner parsing issue

* bump the spanner module version to v1.40.0
2022-11-08 22:27:33 -08:00
tsbkw 41f3495935 fix: Conform to go test naming convention
As reported in #835 go vet reporting the violation of
naming convention of go test.
see: https://pkg.go.dev/testing#hdr-Examples

Although this project use golangci-lint as linter and
it has functionality to skip by nolint:govet, so basically
there is no problems.
However if there is no cons, it is better to conform with
naming convention and fix the code like below.
https://github.com/golang-migrate/migrate/blob/c367ed2f12b8549c2ace442c440a9e312348951d/dktesting/example_test.go#L17
2022-10-26 19:20:29 +09:00
Michail Tsyganov 8324b455c0 Update pgx deps 2022-10-22 17:51:24 +03:00
c367ed2f12 Resolve Docker CVEs (#815)
* Bump alpine images to version 3.16 and update golang image to one based on alpine 3.15

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Bump alpine image to version 3.16

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Bump go version to 1.18

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Revert "Bump go version to 1.18"

This reverts commit 7a7d2bfb0b4711012dbecc8e19b1a3c336ea7615.

* Revert "Bump alpine image to version 3.16"

This reverts commit b13c3b84dcb7abb4038d75389dff6dd509645f33.

* Upgrade docker image and migrate binary to golang 1.19, and use build stages in docker image

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>

* Back out unnecessary changes

Signed-off-by: Thomas Spear <tspear@conquestcyber.com>
Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2022-10-19 00:02:33 -07:00
Isaac FolmsbeeandGitHub 5ff4dc804a Fix CockroachDB tutorial's link being relative to 'GETTING_STARTED.md' (#819)
* Fix link root

Fix the link root so that it doesn't point relatively.

* Simplify to just the / root
2022-10-18 23:16:35 -07:00
Dale HuiandGitHub 4ca6c06627 Merge pull request #817 from dschaller/patch-1
docs: minor updates to getting started docs
2022-10-18 23:06:51 -07:00
Dale HuiandGitHub 865751e1b3 Merge pull request #816 from dschaller/dschaller-patch-1
docs: use v4 godocs link in readme
2022-10-18 23:04:13 -07:00
Dale HuiandGitHub 03e74b9289 Update SECURITY.md
mention impact and urgency
2022-10-18 22:35:33 -07:00
Dale HuiandGitHub a2cd4036be Create SECURITY.md 2022-10-18 22:33:42 -07:00
Dale Hui 58eb755472 Use v4 for Go Report Card 2022-10-14 23:57:39 -07:00
Dale Hui 30411f49a1 Add support for Go 1.19 and drop support for Go 1.17 2022-10-14 22:52:48 -07:00
Dale Hui 723c60fe0d Update dktest from v0.3.11 to v0.3.13 2022-10-14 22:50:44 -07:00
Dale HuiandGitHub 1d0b934641 Merge pull request #820 from Juneezee/go1.17
build: update `go` directive in `go.mod` to 1.17
2022-10-14 22:46:39 -07:00
Eng Zer Jun 0d539f4cae build: update go directive in go.mod to 1.17
This commit enables support for module graph pruning and lazy module
loading for projects that are at Go 1.17 or higher.

Reference: https://go.dev/ref/mod#go-mod-file-go
Reference: https://go.dev/ref/mod#graph-pruning
Reference: https://go.dev/ref/mod#lazy-loading
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-06 12:38:28 +08:00
Dale HuiandGitHub 37a9174b9d Merge pull request #810 from linzhengen/bump_ci_version
bump up CI version & fix CI failed
2022-10-05 14:00:37 -07:00
linzhengen 30deade574 bump ci version & go1.19 fmt ./... 2022-10-05 23:48:20 +09:00
Dale HuiandGitHub 2d1de76527 Merge pull request #814 from tspearconquest/fix_cves
Patch CVEs
2022-10-04 23:32:30 -07:00
DerekandGitHub d5be909ccd docs: minor updates to getting started docs 2022-10-01 14:16:21 -07:00
DerekandGitHub 2fdf921b1e docs: use v4 godocs link in readme 2022-10-01 14:14:30 -07:00
Thomas Spear 405ef9df07 Run gofmt -s on the files that are failing the lint job upstream
Signed-off-by: Thomas Spear <tspear@conquestcyber.com>
2022-09-28 14:19:40 -05:00
Thomas Spear 856f9c4192 Patch CVEs
Signed-off-by: Thomas Spear <tspear@conquestcyber.com>
2022-09-28 14:10:48 -05:00
Javier Cuan-Martinez 79c741fa73 Upgrading Spanner version 2022-09-15 08:59:13 -07:00
Fumiaki MATSUSHIMA ca03021154 Fix placeholder in download URL 2022-09-08 20:17:19 +09:00
Dale HuiandGitHub 5eb1604704 Merge pull request #726 from morigs/feat/yugabyte-support
feat: add yugabyte support
2022-08-25 23:45:01 -07:00
Igor Morozov b74509abe2 docs(yugabyte): update readme (add retry options) 2022-08-25 09:44:59 +03:00
Igor Morozov 9a407b6083 style(yugabyte): rename retry options 2022-08-25 09:38:36 +03:00
Igor Morozov 5582a2083b style(yugabyte): simplify error conditions 2022-08-25 09:36:23 +03:00
Igor Morozov bf926fa253 style: fix lint errors 2022-08-24 09:50:31 +03:00
Igor MorozovandGitHub 9f21d9ca68 Merge branch 'master' into feat/yugabyte-support 2022-08-24 09:44:44 +03:00
Igor Morozov 76d03050d2 feat(yugabytedb): configure retries 2022-08-24 09:44:27 +03:00
Dale HuiandGitHub 4bd83668eb Merge pull request #796 from agcooke/bugfix/fix-docs
The example Clickhouse DSN incorrectly described how to set the usern…
2022-08-23 21:48:42 -07:00
Adrian Cooke 8c64ac7ccd The example Clickhouse DSN incorrectly described how to set the username and password.
The example:

   `clickhouse://username:password@host:port/database=clicks?x-multi-statement=true`

did not work, but:

`clickhouse://host:port?username=user&password=password&database=clicks&x-multi-statement=true`

does.

I got the above DSN format from the tests.
2022-08-19 14:00:28 +02:00
Dale HuiandGitHub 03613f14ac Merge pull request #784 from anacrolix/patch-1
Fix broken tags link
2022-08-01 22:26:50 -07:00
Matt JoinerandGitHub 162fc2bb3d Sqlite update (#785)
* Update modernc.org/sqlite

Specifically to get support for sqlite3 3.38.0.

* go mod tidy
2022-08-01 22:25:44 -07:00
Matt JoinerandGitHub c47c8cc9d9 Fix broken tags link 2022-08-01 18:31:22 +10:00
Dom HuttonandGitHub 3bba76f90a modify: Add statement timeout query parmeter for MySQL (#778)
* modify: Add statement timeout query parmeter for MySQL

* fix: parsing and assignment of statementTimeout

* fix: bad merge
2022-07-27 11:14:52 -07:00
Jon KjennbakkenandGitHub 0d585e2a56 Switch to microsoft go-mssqldb (#758)
* Switch to https://github.com/microsoft/go-mssqldb

* Fix undeclared name

* rebase

* Merge master
2022-07-27 11:12:27 -07:00
Dale HuiandGitHub eebc4c437a Merge pull request #755 from abhinavnair/replace-ioutil
Replace deprecated ioutil pkg with os & io
2022-07-26 23:59:08 -07:00
Dale HuiandGitHub fca5869487 Merge pull request #774 from linzhengen/fix_spanner_parsing_issue
fix spanner parsing issue
2022-07-26 23:23:02 -07:00
Dale HuiandGitHub af0c073f52 Merge pull request #770 from cdamoc90poe/bump-version-for-sqlite3
bump version for sqlite3 lib to 1.14.14
2022-07-26 23:20:33 -07:00
Dale HuiandGitHub edff41a91c Merge pull request #776 from SinghCoder/patch-1
chore: Correct interface name in comments.
2022-07-26 23:08:00 -07:00
Dale HuiandGitHub 5dc2e47f4e Merge pull request #780 from soyayaos/master
jq must be invoked with -r option
2022-07-26 22:52:55 -07:00
Vladimir Galunshchikov 89836f4f11 jq must be invoked with -r option 2022-07-26 18:39:07 +03:00
HarpinderandGitHub 3843f9233f Correct interface name in comments. 2022-07-23 22:03:39 +05:30
linzhengen 185a4ded0e fix spanner parsing issue 2022-07-15 14:11:39 +09:00
Cristian Damoc b2be9e7b62 bump version for sqlite3 lib to 1.14.14 2022-07-12 13:58:19 +03:00
Abhinav Nair 885d03c4af Replace deprecated ioutil pkg with os & io
As of Go 1.16, the same functionality is now provided by package io or
package os, and those implementations should be preferred in new code.

So replacing all usage of ioutil pkg with io & os.
2022-06-26 16:57:29 +08:00
Dale Hui acc11eda6d Use t.Skipf() per golangci-lint (gosimple) 2022-05-13 15:59:07 -07:00
Dale Hui b5e3e92f00 Support Go 1.18 and drop support for Go 1.16 2022-05-13 15:45:41 -07:00
Dale Hui 075d6e8130 Update dktest from v0.3.10 to v0.3.11 to fix upstream security issues 2022-05-06 21:59:35 -07:00
Igor Morozov bcd0d884bf feat: add yugabyte support 2022-04-03 00:25:18 +03:00
Dale Hui 331a15d92a Update dktest from v0.3.9 to v0.3.10 to fix upstream security issues 2022-03-16 23:49:09 -07:00
Dale HuiandGitHub cc5f8b7ea7 Merge pull request #714 from eaze/josh/test-sqlserver-with-edge
Run SQL server tests on Azure SQL Edge
2022-03-16 23:20:51 -07:00
Josh Holbrook 9d3babd356 Run SQL server tests on Azure SQL Edge 2022-03-16 16:50:29 -04:00
Dale HuiandGitHub 57aead3125 Merge pull request #659 from AndreasKl/add-with-connection-to-postgres
Add WithConnection to Postgres similar to MySQL.
2022-02-24 12:40:56 -08:00
Dale HuiandGitHub e1d604b490 Merge pull request #708 from taywrobel/cassandra-connect-timeout
Expose configuring cassandra connect timeout through query string
2022-02-22 14:25:43 -08:00
Andreas KluthandGitHub 3dfae0db3e Merge branch 'master' into add-with-connection-to-postgres 2022-02-21 10:36:13 +01:00
Taylor Wrobel 6e499cb007 Expose configuring cassandra connect timeout through query string
Exposes the cassandra client's ConnectTimeout parameter through
the cassandra database query string.

The Cassandra client has a fairly aggressive connect timeout of
600ms, which can cause flakiness in certain network environments
or in Cassandra-compatible clusters with a more complex process
for establishing connections (i.e. ScyllaDB).

The query timeout is already configurable and exposed, but does
not impact the connect timeout in the gocql driver. This change
allows for configurtion of both.
2022-02-18 16:33:06 -08:00
Dale HuiandGitHub 0bc9734138 Merge pull request #700 from Juneezee/test/t.TempDir
test: use `T.TempDir` to create temporary test directory
2022-02-09 16:41:31 -08:00
Andreas KluthandGitHub 34c9878ba3 Merge branch 'master' into add-with-connection-to-postgres 2022-02-09 21:37:14 +01:00
Eng Zer Jun 98d2f99a88 test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-02-09 09:37:48 +08:00
Dale HuiandGitHub 6b95a852e1 Merge pull request #695 from xkisu/master
Incorrect DSN example for Clickhouse
2022-02-06 23:00:40 -08:00
Keith MitchellandGitHub 81cbc9c058 fix: incorrect clickhouse dns example
The current example for the Clickhouse DSN is incorrect and will result in the client trying to use the default username, database, and password regardless of what is set in the DSN. This change fixed the DNS example to be correct for the current versions of the clickhouse-go client.
2022-02-02 17:55:16 +00:00
Andreas KluthandGitHub 19c48918be Merge branch 'master' into add-with-connection-to-postgres 2022-01-20 21:45:09 +01:00
Dale HuiandGitHub 3e43c429ee Merge pull request #686 from pankona/bump-sqlite3-to-1-14-10
Bump go-sqlite3 to v1.14.10
2022-01-12 10:52:36 -08:00
Dale Hui c277388d49 Update dktest to v0.3.9 for upstream security vulnerability fixes 2022-01-12 10:48:53 -08:00
Dale HuiandGitHub bb269a7be1 Merge pull request #687 from bndw/master
docs: Include iofs in migration sources
2022-01-12 09:37:28 -08:00
bndw ed8a8b0e94 Include iofs in migration sources 2022-01-11 21:54:21 -08:00
pankona a46677ef90 bump go-sqlite3 to v1.14.10 2022-01-12 03:33:21 +00:00
Dale HuiandGitHub 576ba6897b Merge pull request #669 from kylechadha/upgrade-spanner-lib
fix(spanner): Upgrade spanner client to latest, which adds JSON_VALUE support
2021-12-17 20:40:25 -08:00
9f5ed82865 Use DELETE FROM instead of TRUNCATE for MySQL (#656)
* Use DELETE FROM instead of TRUNCATE for MySQL

* Set isolation level to SERIALIZABLE

Co-authored-by: Martin Arrieta <mail@martinarrieta.me>
2021-12-17 20:21:25 -08:00
Kyle Chadha 020eda481c Upgrade cloud.google.com/go/spanner to 1.28.0, which adds JSON_VALUE support 2021-12-09 18:53:44 -08:00
moppandGitHub 918e13ad9a Upgrade containerd and image-spec for the vulnerabilities (#662)
* upgrade containerd/containerd to 1.5.8 for vulnerability

* upgrade opencontainers/image-spec to 1.0.2 for vulnerability

* upgrade dktest to v0.3.8
2021-12-09 00:40:24 -08:00
Dale HuiandGitHub 0c500eba13 Merge pull request #664 from suzaku/simplify
len(k) is guaranteed to be greater than 1 in this case
2021-12-07 21:12:12 -08:00
Dale HuiandGitHub ca2d3e0724 Merge pull request #658 from mccalltd/update-spanner-to-1.27
Update Google Cloud Spanner to 1.27
2021-12-07 21:07:09 -08:00
suzaku 169d0fcf29 Include 2 letter query param key in unit test 2021-12-08 10:38:36 +08:00
satoru 4f82413d85 len(k) is guaranteed to be greater than 1 in this case 2021-12-08 10:38:36 +08:00
Dale HuiandGitHub 2d6f0e2ab5 Merge branch 'master' into update-spanner-to-1.27 2021-12-07 14:26:59 -08:00
Khash SajadiandGitHub b69b032d6a clarify the package name (#660)
* clarify the package name

* Adding import and connection string sections

* more clarification around the go usage
2021-12-07 13:52:37 -08:00
Dale HuiandGitHub efbef2a2bd Merge pull request #663 from suzaku/refacotring
refactor: use sort.Slice
2021-12-07 09:12:11 -08:00
satoru fc7f720c9f refactor: use sort.Slice 2021-12-07 14:49:37 +08:00
Andreas Kluth f621644741 Add WithConnection to Postgres similar to MySQL. 2021-11-29 15:50:04 +01:00
Dale Hui cf68462e13 Run gofmt on internal build dir 2021-11-23 16:30:29 -08:00
Dale Hui 090a3c4d85 go mod tidy 2021-11-23 16:28:41 -08:00
Tim McCall 579878468e Update Spanner to 1.27 2021-11-19 10:53:05 -07:00
4ba695707a Resolves #647 - Fixes typos in Mongo advisory locking parameters (#648)
* Resolves #647 - Fixes typos in Mongo advisory locking parameters

* PR Feedback for #647 - Error out when both params set

* Fixes lint issues with #647

* Additional PR Fixes #647

* Lint fixes again #647

* One more tweak to CONTRIBUTING.md

* One more tweak to CONTRIBUTING.md, again

Co-authored-by: Steve Ramage <commits@sjrx.net>
2021-11-04 22:17:35 -07:00
Dale HuiandGitHub efa49bc721 Merge pull request #646 from Greyeye/master
change github auth to use oauth token instead of basic.
2021-11-04 22:16:48 -07:00
Dale HuiandGitHub 675d588f15 Merge branch 'master' into master 2021-11-04 21:48:03 -07:00
Dale HuiandGitHub d11b43e78a Merge pull request #650 from grnhse/bump-versions
Bump go version in go.mod and bump autorest/adal
2021-11-01 23:13:45 -07:00
Juan C. Müller 5aa1a0381e Bump version of autorest/adal 2021-11-01 12:26:02 -04:00
Dale HuiandGitHub b5eae1a189 Merge pull request #651 from glacials/master
Add pkger source to README
2021-10-31 22:11:23 -07:00
Ben CarlssonandGitHub bfd05026fc Set syntax highlighting for pkger example 2021-10-31 13:26:24 -07:00
Ben CarlssonandGitHub 06c25a20bc Add pkger to README 2021-10-31 13:26:08 -07:00
Greyeye a2b0bdadbc change github auth to use oauth token instead of basic. 2021-10-26 10:57:16 +10:00
Dale HuiandGitHub 9f8bb1bb1a Merge pull request #645 from mgenware/patch-1
Use the recommended v4 in mysql README
2021-10-25 16:44:16 -07:00
Liu YuanYuanandGitHub 608e4f666b Use the recommended v4 in mysql README 2021-10-24 23:25:52 +08:00
Dale HuiandGitHub 1ba9becc5c Merge pull request #634 from Baumanar/master
Fix for failing test for AWS keyspace
2021-10-15 09:52:04 -07:00
Baumanar c81e147ad6 go mod tidy 2021-10-14 10:37:49 +02:00
Baumanar dbb574c92c fix test 2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar 79472544ff Delete all rows
https://github.com/golang-migrate/migrate/pull/579#discussion_r650178687
2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar b9b66468f5 Use ParseBool
https://github.com/golang-migrate/migrate/pull/579#discussion_r650171022
2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar 5eaf23f836 Support for AWS Keyspaces 2021-10-14 10:36:38 +02:00
Dale HuiandGitHub f25b303a88 Merge pull request #638 from dhui/gosnowflake
Update gosnowflake from v1.4.3 to v1.6.3
2021-10-13 22:13:24 -07:00
Dale Hui 41431fbdc3 Update gosnowflake from v1.4.3 to v1.6.3
Addresses:
* https://github.com/golang-migrate/migrate/issues/632
* https://github.com/golang-migrate/migrate/issues/637
2021-10-13 21:18:38 -07:00
Dale Hui 780d5551e0 Update docker client usage with breaking change 2021-10-12 22:11:20 -07:00
Dale Hui 71106e5fd2 Update dktest to v0.3.7 to fix security warnings
https://github.com/dhui/dktest/releases/tag/v0.3.7
2021-10-12 21:58:29 -07:00
Dale HuiandGitHub eea81e7876 Merge pull request #628 from golang-migrate/revert-binary-location-change-in-docker-image
revert binary file location change in docker image
2021-09-27 12:12:47 -07:00
Fontinalis 1668f8e6a0 revert binary file location change in docker image 2021-09-27 10:13:33 +02:00
Dale HuiandGitHub 10a92d0cc8 Merge pull request #620 from golang-migrate/fix-source-file-driver
fix source/file driver
2021-09-16 16:45:03 -07:00
Fontinalis 0775323324 fix source/file driver 2021-09-16 14:24:01 +02:00
Dale Hui 14f6d3a92f Update build constraints
Issue caught by linter

https://pkg.go.dev/go/build#hdr-Build_Constraints
https://pkg.go.dev/cmd/go#hdr-Build_constraints
https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
2021-09-10 21:04:11 -07:00
Dale Hui 677b5deb09 Update golangci-lint config
- Update to renamed config keys
- Remove redundant linter (maligned)
2021-09-10 21:02:44 -07:00
Dale HuiandGitHub b4047a1b47 Merge pull request #617 from kaworu/pr/kaworu/database/clickhouse/honor-DefaultMigrationsTableEngine
clickhouse: honor `DefaultMigrationsTableEngine` when created `WithInstance()`
2021-09-10 20:36:47 -07:00
Alexandre Perrin 9db2d011f3 clickhouse: honor DefaultMigrationsTableEngine when created WithInstance()
Before this patch, clickhouse.WithInstance() would not select
DefaultMigrationsTableEngine when no MigrationsTableEngine was provided
through the clickhouse.Config{} argument. This behaviour forces the
caller to always provide explicitely the config's MigrationsTableEngine.

This patch make it so DefaultMigrationsTableEngine is used when
MigrationsTableEngine is not provided through clickhouse.Config{}.

Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
2021-09-10 10:44:55 +02:00
Dale HuiandGitHub aef2747761 Fix grammar for Clickhouse cluster mode docs 2021-08-24 21:45:25 -07:00
aa119418b8 Adding support clickhouse cluster (#568)
* added url query for creating schema_migrations table on cluster

Signed-off-by: Ildar Valiullin <preved.911@gmail.com>

* changed documentation by clickhouse databases

Signed-off-by: Ildar Valiullin <preved.911@gmail.com>

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2021-08-24 21:39:30 -07:00
Dale Hui df6106d10e Support Go 1.17 and drop support for Go 1.15 2021-08-21 01:32:23 -07:00
Dale Hui b62d267202 Push builds for debian bullseye 2021-08-18 16:04:22 -07:00
Dale Hui d30a9850da Change build status badge to use GitHub Actions 2021-08-06 21:50:44 -07:00
Dale Hui bea4b82719 Fix docker image tags by undoing previous commit 2021-08-06 21:38:45 -07:00
Dale Hui ea0fa43397 Use fewer docker image tags 2021-08-06 20:41:19 -07:00
Dale HuiandGitHub 82dd9d78ce Merge pull request #609 from golang-migrate/607-fix-dockerfile
fix missing /migrate link in docker image
2021-08-06 09:58:35 -07:00
Fontinalis dba334c74e fix missing /migrate link in docker image 2021-08-06 10:44:05 +02:00
Dale HuiandGitHub 784a2f89da Merge pull request #608 from alethenorio/update-spanner-dependency
Update Google Spanner dependency
2021-08-05 21:20:58 -07:00
Eng Zer JunandGitHub 5aa3b3ad30 ci: migrate from CircleCI to GitHub Actions (#605)
* ci: migrate from CircleCI to GitHub Actions

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* ci: merge release.yaml into ci.yaml

At the time of this commit, it is not possible to make the `goreleaser`
job run in a separate workflow and detect the tag push event, see [1].

[1]: https://stackoverflow.com/a/68078768/7902371
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-05 21:12:21 -07:00
Alexandre Thenorio ab5fe5e783 Update Google Spanner dependency
Adds support for the following in SpanSQL

* ROW DELETION POLICY
* ALTER DATABASE
* Parsing keywords and functions in case insensitive
2021-08-05 19:25:26 +02:00
Dale Hui b2c89c743c Clean up Makefile targets
* move echo-source and echo-database targets so the build target is first and default
* mark echo-source and echo-database targets as .PHONY
2021-08-04 16:19:03 -07:00
Dale Hui a4c42d7390 Include source archives with releases
See: https://goreleaser.com/customization/source/
2021-08-03 21:41:15 -07:00
Dale Hui 823c2806a0 Allow prereleases via GoReleaser 2021-08-03 12:19:56 -07:00
5a601b986d Support for GitHub Actions and goreleaser (#604)
* goreleaser configuration and automatic release workflow

* update and fix goreleaser

* remove windows arm ignore

* add format override for windows to use zip files

* update environment and docker username

* the single source of truth

* fix depricated api issue ( thanks @dhui )

Co-authored-by: Mitsuo Heijo <mitsuo.heijo@gmail.com>
2021-07-30 23:09:32 -07:00
Dale HuiandGitHub 5bf05dc323 Remove -o option from go install example in CLI docs
The -o option only exists in `go build`, not `go install`. e.g. it's not a build flag
See: https://github.com/golang-migrate/migrate/issues/603#issuecomment-888432107
2021-07-28 09:06:18 -07:00
Dale HuiandGitHub 586cc866f5 Merge pull request #595 from DGollings/master
use github.com/cockroachdb/cockroach-go/v2/crdb
2021-07-27 09:22:16 -07:00
Dale Hui 614523948d Merge branch 'master' into crdb 2021-07-26 22:13:48 -07:00
Dale Hui aa47d3da8c Update MongoDB db driver to fix security issue
https://nvd.nist.gov/vuln/detail/CVE-2021-20329
2021-07-23 13:46:20 -07:00
Daniel Gollings 94649eedd8 ran go mod tidy 2021-07-14 14:44:05 +02:00
Daniel Gollings f1b7710148 use github.com/cockroachdb/cockroach-go/v2/crdb (for better go module support) 2021-07-14 14:35:05 +02:00
Sébastien CaparrosandGitHub 31cedbb961 Added a method to create a mysql database from a connection object (#583)
* Added a method to create a mysql database from a connection object

* Calling WithConnection from WithInstance to de-duplicate code

* Adding context and ping to mysql.WithConnection

* Interface type check at compile time
2021-07-06 10:45:43 -07:00
805f8c8f2d Add SqlServer MSI Support (#591)
* MSI Auth for SQL Server

Add MSI Auth option to SQL Server connection. Default if no password is provided in connection string.

* Parse resource endpoint from server for msi

update host name parsing to get just the resource endpoint for msi

* Update go-mssqldb

update go-mssqldb to resolve panic issue referenced here: https://github.com/denisenkom/go-mssqldb/pull/642

* Update sqlserver.go

switch from deprecated methods. NewServicePrincipalTokenFromManagedIdentity calls the two methods that are deprecated for us

* Update sqlserver.go

add useMsi param instead of looking for nil password

* Update sqlserver readme

Update sqlserver readme for msi auth. make useMsi a bit safer

* Update sqlserver.go

remove comment

* Update database/sqlserver/README.md

Co-authored-by: Keegan Campbell <me@kfcampbell.com>

* Update sqlserver.go

refactor resource uri logic into its own method

* Update sqlserver_test.go

add tests for msi connection. can only test whether it fails with useMsi= true, or succeeds with useMsi=false

* Update sqlserver.go

check msi.EnsureFresh return value

* Return error for multiple auth and move query filter

move migrate.FilterCustomQuery(purl).String() into one line out of if/else. return error if both useMsi=true and password are passed

* Update README.md

update readme with warning for useMsi

* Update sqlserver_test.go

Update TestMsiFalse test case as now it should fail when useMsi=false and no password is provided

Co-authored-by: Keegan Campbell <me@kfcampbell.com>
2021-07-01 23:08:44 -07:00
3dfb0ff538 used uber atomic bool instead standard in lock/unlock db (#580)
* used uber atomic bool instead standard in lock/unlock db

* fixing local lock optimization implementation

* fixing mysql local lock optimization implementation

* fixing cockroachdb local lock optimization implementation

* CAS wrapper to automatically restore

* added test for cas with restore

* added atomic lock to sqlite & mongo

* added atomic lock to sqlite & mongo

* fix/refactor test cas restore function

* disable lock for mongo

* mongo local lock before acquiring the db lock

Co-authored-by: a.prinkov <aprinkov@ourgapps.com>
2021-06-23 14:40:50 -07:00
Dale Hui 3b3c1b6aa5 Update go-github from v17 to v35 2021-05-28 14:56:29 -07:00
Dale Hui 8a1a8534dc Update snowflake DB driver to update to non-vulnerable jwt downstream package
Fixes https://github.com/golang-migrate/migrate/issues/569
2021-05-21 09:56:26 -07:00
Kyle ChadhaandGitHub 1695f40b3e fix(spanner): upgrade spanner client to latest to support NUMERIC types (#565)
* fix(spanner): upgrade spanner client to latest to support NUMERIC types

* run go mod tidy

* Add spanner example migration test case
2021-05-18 22:16:21 -07:00
Dale HuiandGitHub 2e7358e012 Merge pull request #560 from dhui/re-add-iofs-src-driver
Re-add the io/fs source driver
2021-05-10 23:38:05 -07:00
Dale Hui eac6684bf3 Fix iofs tests
Changes:
- embedded variables must be at the package level
- re-use embedded FS in example for tests
- bump Go module version to 1.16 for tests to run
    - This may cause issues building in 1.15
2021-05-10 22:49:46 -07:00
Dale Hui 53720240b4 Update docker builds to use Go 1.16 and Alpine Linux 3.13 2021-05-10 11:28:59 -07:00
Dale Hui 337ff632e3 Revert "Merge pull request #481 from johejo/revert-472-iofs_source_driver"
This reverts commit b690b78e22, reversing
changes made to 15931649a2.
2021-05-10 10:50:27 -07:00
Dale HuiandGitHub c4d82aa89a Merge pull request #558 from zikaeroh/bump-sqlite
Bump sqlite past memory corruption bug
2021-05-06 22:35:36 -07:00
zikaeroh 7a631fe801 Bump sqlite past memory corruption bug 2021-05-06 21:32:04 -07:00
Dale Hui ffdcb52998 Update sqlite and sqlite3 docs with comparison 2021-05-05 10:12:41 -07:00
Andrew GillisandGitHub 512097b404 Add support for modernc.org/sqlite (#555)
* Add support for modernc.org/sqlite

This is a CGo-free port of SQLite

* Remove sqlite from build target
2021-05-05 10:04:18 -07:00
Dale HuiandGitHub af256da8ca Merge pull request #550 from spacefill/547-postgres-pgx-ensureVersionTable-must-use-config-schema-name
[pgx][postgres] Config.SchemaName refactoring usage
2021-04-25 13:45:05 -07:00
Stéphane Klein d0af5824ad postgres,pgx: fix SchemaName parameter is ignored ##547 2021-04-25 07:34:19 +02:00
Bruno MouraandGitHub 4720914286 clickhouse: make table engine for schema versions table configurable (#551)
* clickhouse: make the table engine used for schema_migrations configurable

* clickhouse: tests creation with configurable table engine

* clickhouse: refactor tests to run with multiple table engines

* clickhouse: clarify the use case for the x-migrations-table-engine option

* clickhouse: fix naming consistency for the migrations table engine

* clickhouse: fix tests function wrap pattern
2021-04-21 22:03:11 -07:00
Stéphane KleinandGitHub 3d5d577b61 Postgres - add x-migrations-table-quoted url query option (#95) (#533)
* Postgres and pgx - Add x-migrations-table-quoted url query option to postgres and pgx drivers (#95)

By default, gomigrate quote migrations table name, if `x-migrations-table-quoted` is enabled, then you must to quote migrations table name manually, for instance `"gomigrate"."schema_migrations"`

* Work In Progress
2021-04-19 14:57:50 -07:00
Alexey PrinkovandGitHub d68a3abedb added tests for clickhouse database (#532)
* added tests for clickhouse

* added atomic lock flag to clickhouse & remove extra test

* added clickhouse test package
2021-04-05 15:37:50 -07:00
testtest959andGitHub 511ae9f5b6 Postgres: Add a check to determine if table already exists to elide CREATE query (#526)
* Squash commits

* Format

* Minor refactoring

* Address PR feedback; Add mustRun

* Fix a test assert
2021-03-18 21:03:57 -07:00
Dale HuiandGitHub 63aff4b03d Merge pull request #529 from linux019/upgrade_ch
upgrade clickhouse driver lib
2021-03-16 20:23:20 -07:00
= 2839a3d6f2 upgrade clickhouse driver lib 2021-03-16 18:51:56 +02:00
Dale HuiandGitHub 4573d98dce Merge pull request #527 from jeffwidman/patch-2
Fix typo
2021-03-15 23:58:57 -07:00
Jeff WidmanandGitHub a728e7a674 Fix typo 2021-03-15 21:45:44 -07:00
Dale HuiandGitHub 707fc52dfe Merge pull request #522 from Juneezee/remove-all-from-drop
Move `-all` flag usage from `drop` to `down`
2021-03-13 21:36:01 -08:00
Jeff WidmanandGitHub 6ecf7bcd0a Point docs link to v4 (#524)
* Point docs link to `v4`

It's very confusing that when I land directly on `https://pkg.go.dev/github.com/golang-migrate/migrate` I get a the message:
> You are look at a deprecated version, use the latest version instead

But when I click the `latest version` link, it goes to a completely dead/wrong place.

So I think for almost anyone who actually wants to look at the real docs, it's a lot simpler to link straight to `v4` by default. If they want a different version, they can change that. But let's have the default point to a helpful place, and then you can go looking for the broken version, rather than the other way around.

* Update the badge link as well
2021-03-13 21:30:45 -08:00
Dylan BargteilandGitHub 2f0c0752ed Add pgx driver package (#517)
* replace lib/pq dependency with pgx in postgres driver

* fix pgx error handling

pgconn is a low-level library, and errors from queries flow through higher-level pgx library

* update postgres tests to use pgx driver and error reporting

* update quoteIdentifier implementation to match lib/pq

* move changes for pgx support into new driver package

* add pgx driver to CLI

* restore examples used in testing for pgx driver

* replace postgres driver name with pgx in DSN in tests and docs

* consolidate connection string formatting in pgx testing

* lock pgx package at v4

Prior to v4, pgx did not support go modules. Version locking the import is the safest way to ensure we get the (as of now) latest version of the module. Additionally v4 implements some breaking changes to connection and error handling, which are also addressed in this commit.

* update parsed URL scheme instead of using string operation

* add pgx for builds using make

* add pgx to top level docs

* remove autogenerated TOC
2021-03-13 21:29:10 -08:00
Eng Zer Jun aa2dad421f fix: move -all flag usage from drop to down
The `-all` flag usage should be used in `down` only, not `drop`.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-03-13 15:22:09 +08:00
Dale HuiandGitHub 7dc03e3c0d Merge pull request #509 from nronas/master
Allow the ability to pass timezone when generating migration files
2021-02-24 09:36:46 -08:00
John Tzikas 6cdb8c1b23 Allow the abillity to pass timezone when generating migration files 2021-02-23 20:20:14 +02:00
Dale Hui e57a024960 Support Go 1.16 and drop support for Go 1.15
- Document CLI install instructions for Go 1.16
- Update golangci-lint from v1.33.0 to v1.37.0
2021-02-18 21:33:36 -08:00
Dale Hui 8709f196bf Restore CircleCI build cache after checkout and remove branch from cache key 2021-02-09 21:46:40 -08:00
Dale Hui 70d6a03251 Upgrade dktest from v0.3.3 to v0.3.4
Addresses: https://github.com/golang-migrate/migrate/issues/506
2021-02-09 20:48:31 -08:00
Dale Hui ff92da45f1 Use strconv.ParseBool() to parse the x-multi-statement option in the postgres driver 2021-02-03 22:59:24 -08:00
Dale HuiandGitHub a185b9e78f Merge pull request #495 from AnatolyRugalev/postgresql-support-multistatement
Support multi-statement execution for PostgreSQL
2021-02-02 23:18:28 -08:00
Anatoly RugalevandGitHub 42abb24ae0 Merge branch 'master' into postgresql-support-multistatement 2021-02-02 13:24:47 +03:00
Anatoly Rugalev 38ffbfb778 Addressed review comments 2021-02-02 11:24:33 +01:00
BenandGitHub dab829b564 Add default GitHub API client (#501) 2021-01-30 23:07:53 -08:00
BenandGitHub eb62a386cc sqlite3: Fix error when path to database contains whitespace (#500) 2021-01-28 23:36:56 -08:00
6d8df972da Fix issue when gitlab return only first 20 migrations (reopen) (#497)
* Fix issue when gitlab return only first 20 migrations

* CircleCI issue. repush

* gofmt with -s

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2021-01-28 14:35:14 -08:00
Dale HuiandGitHub 550193e3e0 Fix CircleCI badge link 2021-01-26 00:27:32 -08:00
Dale Hui 5ed79873f7 Update badge to reflect move from TravisCI to CircleCI 2021-01-26 00:23:44 -08:00
Dale Hui 20b5378936 Use CircleCI
Squashed commit of the following:

commit 5f63ff0333a9608530b4bb79b44af837224f80a5
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:36:57 2021 -0800

    Remove unnecessary parameter to checkout

commit b87711057588bfe05fb01d8ff9b679200b2c3015
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:30:37 2021 -0800

    Give up trying to only push coverage for Go 1.15 via YAML templates

commit 6f366c35d11a903fdd8d92738de26073fe7a9c0f
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:27:20 2021 -0800

    YAML template merging only works with maps, not scalars, so specify the current directory for "checkout"

commit e65ff967018508f36e9b9d88f20fe135d8ce6c3e
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:21:30 2021 -0800

    Use `go get` instead of `go install` and add template for steps

commit fef2a9e7f9bfbb6070b849199e7d0cced8c6747a
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 19:59:24 2021 -0800

    Push code coverage to coveralls.io
        - Only push code coverage for latest support version of Go

commit 48d78bb2576aa8662daaa3e116d0e19dafa77279
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:24:13 2020 -0800

    golangci-lint runs slowly on circle ci machines...

commit d47e04f9aad971a72f169089369ed9e976fbf94f
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:20:35 2020 -0800

    Cleanup setup and remove debugging commands

commit 063337f2faff23b6ca636d77acbd4c504ac6914b
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:17:57 2020 -0800

    try using ~/bin

commit 57f34fbf5ce38ff8fdfdb4be43958d43db01d9f6
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:13:55 2020 -0800

    Install golangci-lint in another dir

commit 2b5d6d65ee4807e3c2df0afffc29498defe5a6c3
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:03:08 2020 -0800

    More go env debugging

commit 6719f663d582e59a551bb10ff7365aef814c1d85
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:59:24 2020 -0800

    Sparate downloading and running of golangci-lint.sh

commit 6e9ec9ff0c2d7353b711af5ad685bb4d07ff7292
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:54:44 2020 -0800

    debug gimme env

commit 706a43ca67e012c7982deb115a7168fc2f9067c3
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:47:01 2020 -0800

    debug go env

commit 18694c6419ceeaf0b649939249a618f3876ef530
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:39:53 2020 -0800

    Use .x version of Go

commit 2e4a87b03d1e0b8cf66719bea672304ae008d881
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:36:01 2020 -0800

    correctly use gimme

commit 99a1b838df8832243fc85cdb30d69a07042f9f93
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:31:45 2020 -0800

    Install golangci-lint after installing requested version of Go

commit cecf219e557d5cac26f3f1f43955cfcdb397bb50
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:29:48 2020 -0800

    Still don't support docker layer caching

commit 244ec8fe0b6e5d28cb9c16159bbc3d36ba5ec523
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:28:33 2020 -0800

    Try using machine executor

commit 44d3b29a1e8f953dcf4a2051e0d1037e7b62f85b
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:07:09 2020 -0800

    Fix docker run

commit 525983e81d6a3080e2b28bfdb601581346926450
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:03:39 2020 -0800

    properly specify build-arg

commit d5eed206a6b11c2606690ffdee4639e5399cdfc7
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 20:54:06 2020 -0800

    Try running CircleCI tests in a Docker container

commit 35e62d1c4ccdeb528495f9d28f26a049bcc93bd8
Merge: 90268de a53e6fc
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:50:58 2020 -0800

    Merge remote-tracking branch 'origin/master' into circleci

commit 90268dedc2cf72ab498ebfd26465551a16463ee7
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:47:46 2020 -0800

    Update golangci lint to v1.33.0

commit d5e0747a53ddfdb7b1ce6ff4bacb236bfc1b7794
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:44:45 2020 -0800

    Don't have access to docker layer caching

commit 6aaecb3b761809b6cf2b4f365f2f9d62f410661d
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:43:52 2020 -0800

    Don't have access to xlarge

commit 275de107de76ced3d6b20e913ffddff84919c3e5
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:39:14 2020 -0800

    CircleCI job names can't have a '.' character

commit e1d339f0eb10519f6002939147201060415a98bb
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:30:51 2020 -0800

    Update CircleCI config

commit d65194971c028af94e7a3cd171eb6221a08dde05
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Fri Aug 2 00:05:59 2019 -0700

    Add config for CircleCI

commit 19d7355548c2a6d64225ea63368e10e19eb43d19
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Jul 30 00:12:41 2019 -0700

    Update golangci-lint from v1.16.0 to v1.17.1
2021-01-25 21:08:52 -08:00
Anatoly Rugalev 3a5169ac6d Support multi-statement execution for PostgreSQL 2021-01-18 12:16:37 +03:00
Mitsuo HeijoandGitHub a53e6fc425 fix test on windows (#482)
* fix test on windows

On Windows, the test fails because os.Remove cannot remove unclosed and still-handled files.

* use t.Error instead of t.Log
2020-11-24 22:53:21 -08:00
Dale HuiandGitHub b690b78e22 Merge pull request #481 from johejo/revert-472-iofs_source_driver
Revert "Add io/fs#FS Driver #471"
2020-11-21 21:40:35 -08:00
Mitsuo HeijoandGitHub 67546bfa67 Revert "Add io/fs#FS Driver #471 (#472)"
This reverts commit fedbe42fc8.
2020-11-21 23:27:36 +09:00
Charles MartinotandGitHub 15931649a2 Mysql - Fix crash with ansi_quotes sql_mode enabled (#476)
* Fix crash with ansi_quotes

* Run lint
2020-11-18 12:38:28 -08:00
Mitsuo HeijoandGitHub fedbe42fc8 Add io/fs#FS Driver #471 (#472)
* Add io/fs#FS Driver #471

* Refactor iofs.Iofs to be independent of httpfs

* Rename Iofs to IoFS, accepts fs.FS and use fs.DirEntry.Info() to undo the changes in ErrDuplicateMigration

* fix documents and migration directory for testing

* Fix iofs.Driver so that it can be used by embedding it like httpfs.PartialDriver

* iofs.Driver closes the file system if possible

* Rename receiver

* Refactor file driver to use iofs.Driver

* iofs build constraint is not needed so remove it

* Properly return an error in a corner case

* Increase golangci-lint timeout

* Fix example code

* Revert "Increase golangci-lint timeout"

This reverts commit 2cd12a63b50b7ac48d12dd0adc1ec4f4b49c3b93.

* unexport driver, add iofs.PartialDriver, remove type alias from file driver

* do not panic in Open, return error
2020-11-18 10:06:11 -08:00
Dale Hui 52e7117d37 Use flagset's NArg() and Arg() methods instead of global flag NArg() and Arg() for sub-commands
Fixes: https://github.com/golang-migrate/migrate/issues/467
2020-11-04 23:37:05 -08:00
Yannick HeinrichandGitHub 8a56273871 Add help statement for cli (#460)
* Add help statement for cli

Refers to #372

* Make constant for argument

* Create a flagset for each command

* Factor error handling

* Factor flagset creation

* fix: Typo and bad behaviour

* Fix lint error

* Fix according to the comments

* Fix according to the comments
2020-11-03 14:54:12 -08:00
Dale Hui 110c0f74b4 Update dktest to v0.3.3 which also updates lib/pq to v1.8.0 2020-10-30 22:26:55 -07:00
EvanandGitHub f76b94ab7d Document drop -f in migrate -help (#459)
* Document drop -f in migrate -help

* Document drop -f in migrate -help 2
2020-10-17 01:44:43 -07:00
Vladimir DimitrovandGitHub 91deba8df0 Update driver.go (#454)
* Update driver.go

* addressing comments
2020-10-16 22:19:05 -07:00
Maarten BezemerandGitHub f18b16515a postgres - Force MigrationsTable in 'public' schema (#414)
* Force MigrationsTable on public schema

* Revert "Force MigrationsTable on public schema"

This reverts commit e5b25e8858ed42d19d420cb8bd16795c060994d2.

* Add example to fix search_path issue

* Use POSTGRESQL_URL in example and fix typo

* Make example simpler by using the `search_path` query parameter of the DB URL
2020-10-15 20:21:00 -07:00
Dale HuiandGitHub f8f021730e Merge pull request #458 from xortive/patch-1
Fix typo in doc comment
2020-10-13 09:19:03 -07:00
MattandGitHub a17aa6e83d Fix typo in doc comment 2020-10-13 11:55:41 -04:00
Dale HuiandGitHub 5d84bec1c8 Merge pull request #456 from hypnoglow/iter-err
Check iterator error on Drop where applicable
2020-10-10 20:46:14 -07:00
Igor Zibarev 5f932c05c8 Check iterator error on Drop where applicable 2020-10-10 23:53:20 +03:00
Dale HuiandGitHub 0521ebeb71 Warn about installing the CLI from within a module
Addresses: https://github.com/golang-migrate/migrate/pull/257
2020-10-07 16:51:03 -07:00
Dale Hui 99ca856a7c Run go fix 2020-10-07 15:31:28 -07:00
Dale Hui 04a572114a Parse out custom TLS params and register the TLS config before calling mysql.ParseDSN
Fixes: https://github.com/golang-migrate/migrate/issues/411
2020-10-06 19:49:59 -07:00
Dale Hui 8c531c7836 Use spanner cloud emulator to run tests
Addresses: https://github.com/golang-migrate/migrate/issues/410
2020-10-01 17:58:26 -07:00
Dale Hui fa3e11943a Add local locks for spanner db driver 2020-10-01 17:58:23 -07:00
d6881f1826 Add Bitbucket Cloud Support as source (#450)
* Added bitbucket cloud support as source for migration

* added test

* address PR Comments

* Add entry in README.md; update go.mod and go.sum

Co-authored-by: abhigupta91 <abhigupta0818@gmail.com>
2020-09-28 22:08:20 -07:00
Dale HuiandGitHub c6026053bc Merge pull request #448 from andyNewman42/locking
Add advisory locking to mongodb
2020-09-25 21:05:56 -07:00
Andy Newman 2c2f6910f4 Rename mongo flag for enabling locking. Add validation around parsing mongo params. 2020-09-25 15:02:49 -07:00
Andy Newman 73760ed4cb Implement PR feedback
rename constants
rename query params
add new timeout interval query param
add new feature flag
use timeout context
2020-09-24 15:38:57 -07:00
Andy Newman 7208a16bf3 Add advisory locking to mongodb 2020-09-24 15:38:57 -07:00
Dale HuiandGitHub 50439fee71 Merge pull request #447 from markokoleznik-cs/patch-1
Fix typo
2020-09-17 10:21:45 -07:00
markokoleznik-csandGitHub 5d91d93b7e typo 2020-09-17 07:59:02 +02:00
Dale HuiandGitHub 6819a17049 Merge pull request #439 from antonklava/add-mysql-no-lock
`NoLock` config option for mysql driver
2020-09-09 10:21:01 -07:00
Anton Klava 8a67f7e535 mysql - strconv.ParseBool for x-no-lock parame
`strconv.ParseBool` should ensure that if x-no-lock is set to any
non-booleanic value an error is returned. The parameter still
defaults to false.
2020-09-09 11:57:28 +02:00
Anton Klava aefffb6f6c NoLock config option for mysql driver
Skipping `GET_LOCK`/`RELEASE_LOCK` is useful when running MySQL as
multi-master. This change simply disables the locking.

The internal client lock is left in place as a sanity check.
2020-09-08 09:25:42 +02:00
Dale HuiandGitHub cb28f6bffb Merge pull request #436 from klingtnet/add-sqlcipher-database-driver
Add database driver for SQLCipher
2020-09-02 17:23:05 -07:00
Andreas Linz 7845cb88da Mention SQLCipher in the list of supported databases 2020-09-02 20:48:32 +02:00
Andreas Linz 7129dfa5d9 Add build file for sqlcipher 2020-09-02 20:44:52 +02:00
Andreas Linz 84eea4aaf2 Add sqlcipher to the DATABASE_TEST variable 2020-09-02 20:43:49 +02:00
Andreas Linz 39cb3234a7 Link to sqlite3 driver 2020-09-02 20:43:03 +02:00
Andreas Linz 85d673f95f Add database driver for sqlcipher
This is a copy of the existing sqlite3 driver, except that it imports `_
"github.com/mutecomm/go-sqlcipher/v4"` as database driver.  This allows
to use migrate with encrypted sqlite databases.
2020-09-02 18:29:45 +02:00
Roman LeventovandGitHub f1cf81e851 Include file path in errors from PartialDriver.ReadUp() and ReadDown() (#421)
* Include file path in errors from PartialDriver.ReadUp() and ReadDown()

* Use errors.Is(...) instead of os.Is...() in migrate.go: https://github.com/golang/go/issues/41122
2020-08-29 16:44:38 -07:00
Dale Hui c6ebff6469 Update golangci-lint from v1.27.0 to v1.30.0 2020-08-19 02:04:00 -07:00
Dale Hui c904438dda Add support for Go 1.15 and drop support for Go 1.13 2020-08-19 00:58:47 -07:00
Dale HuiandGitHub ed3f30cf12 Merge pull request #427 from wyardley/wyardley-426
fix(spanner): resolve "want scalar type" error
2020-08-19 00:11:11 -07:00
William Yardley e5eb088eaf fix(spanner): resolve "want scalar type" error
- Update spanner library to cloud.google.com/go/spanner@v1.9.0
This prevents erroring on DDL statements like:

ALTER TABLE users ALTER COLUMN created
  SET OPTIONS (allow_commit_timestamp = true);
https://github.com/googleapis/google-cloud-go/pull/2656

- Add test case which fails with older version of the library, and
succeeds with this one.

Fixes: #426
2020-08-18 22:35:07 -07:00
Danny HermesandGitHub 6035eb5469 Replacing Github with GitHub. (#429)
* Replacing `Github` with `GitHub`.

Done via:

```
$ git grep -l Github | xargs sed -i '' s/Github/GitHub/g
```

* Reverting changes to Go files.
2020-08-17 13:55:44 -07:00
Dale HuiandGitHub ce34d50b16 Merge pull request #428 from wyardley/wyardley-docs
docs(spanner): clarify how to set parameters
2020-08-16 11:05:38 -07:00
William Yardley 0010bc8495 docs(spanner): clarify how to set paramters
- include param in DSN example
- link to README docs on param (option) format
- switch to 'parameter' instead of 'option', in the main README, since
this is preferred
2020-08-16 05:40:20 -07:00
Dale Hui 31c22870d3 Use latest version of neo4j db driver to fix 32-bit builds
Relevant fix: https://github.com/neo4j/neo4j-go-driver/pull/129
2020-08-03 17:32:42 -07:00
Dale Hui 49f67a8ad4 Use make for docker builds so all releases use the same db and source drivers 2020-08-03 16:28:46 -07:00
Dale Hui 23608f1e63 Update to Alpine 3.12 for docker builds 2020-08-03 16:24:32 -07:00
Dale HuiandGitHub d72c200a64 Merge pull request #419 from dawidd6/patch-1
Makefile: provide a simpler build task
2020-07-28 12:39:51 -07:00
Dawid Dziurla ea98a5bc9f Makefile: provide a simpler build task 2020-07-28 18:55:15 +02:00
Dale HuiandGitHub 07052cd704 Merge pull request #417 from orijtech/master
go.mod: upgrade dependencies and rid vulnerable thrift version@v0.12.0
2020-07-26 23:18:33 -07:00
Emmanuel T Odeke a258a44f2a go.mod: upgrade dependencies and rid vulnerable thrift version@v0.12.0
Coming here, after digging through
google/exposure-notifications-server#749
in which the version of github.com/apache/thrift was reported by
@whaber as having known critical vulnerabilities.

Transitively however, this version was pinning to
cloud.google.com/go@v0.37.4 indirectly, which then imported
versions of opencensus:
* go.opencensus.io@v0.19.
* go.opencensus.io@v0.20.1
that imported
    github.com/apache/thrift@v0.12.0

The target package to upgrade was
    github.com/denisenkom/go-mssqldb v0.0.0-20200620013148-b91950f658ec
which now uses
    cloud.google.com/go@v0.61.0
2020-07-25 19:56:06 -07:00
Dale Hui f34bc2cba0 Distribute releases to Ubuntu 20.04 LTS Focal Fossa
Stop distributing release to non-LTS EOL'd Ubuntu 18.10 Cosmic Cuttlefish
Addresses: https://github.com/golang-migrate/migrate/issues/416
2020-07-21 11:43:31 -07:00
Dale Hui f75042ac5d Run go mod tidy 2020-07-20 21:30:45 -07:00
Dale HuiandGitHub bf6c1ec3d3 Merge pull request #415 from arjantop-cai/neo4j-no-cgo
Allow compilation of neo4j support without CGO.
2020-07-20 21:27:20 -07:00
Arjan Topolovec 271087a63b Allow compilation of neo4j support without CGO.
Since 1.8.0 neo4j driver no longer requires CGO.
2020-07-17 14:07:37 +02:00
Dale Hui 53ef02da13 Use multistmt package to parse multi-statement migrations for clickhouse, cassandra, and neo4j
Addresses: https://github.com/golang-migrate/migrate/issues/406
2020-07-07 23:41:00 -07:00
Dale Hui 620492183f Add helper package to parse multi-statement migrations
Addresses: https://github.com/golang-migrate/migrate/issues/406
2020-07-07 22:50:09 -07:00
Dale HuiandGitHub 8eb1d3031c Merge pull request #409 from alethenorio/spanner-comments
Add support for comments in Spanner migrations
2020-06-30 03:48:41 -07:00
Alexandre Thenorio e6faa29d36 Add support for comments in Spanner migrations
At the moment GCP Spanner backend does not seem to support comments
(issue being tracked at
https://issuetracker.google.com/issues/159730604).
By adding support for parsing the migration DDL with spansql we are
able to support comments and remove them before applying on the
database
2020-06-30 09:25:33 +02:00
Richard LindhoutandGitHub 8b6e3f1a2e Fix small typo in GETTING_STARTED.md (#407)
* Update GETTING_STARTED.md

* Update GETTING_STARTED.md
2020-06-22 10:09:05 -07:00
Dmitry IvankovandGitHub 1f973cc3ff Bump spanner lib to v1.5.0 (#403)
* Bump spanner lib to v1.5.0

Will enable spanner emulator support
https://cloud.google.com/spanner/docs/emulator#go

Current version supports SPANNER_EMULATOR_HOST but fails at run-time
with spanner permission errors.

* go mod tidy
2020-06-09 12:34:55 -07:00
Dale HuiandGitHub 64180398eb Add disclaimer to snowflake db driver 2020-06-03 23:16:43 -07:00
Dale HuiandGitHub 79396ffccc Merge pull request #402 from abhinavcohesity/master
Adding support for schema management in snowflake
2020-06-03 23:14:04 -07:00
Abhinav Kumar b00a0cc77b Adding support for schema management in snowflake 2020-06-03 13:55:24 +05:30
Dale HuiandGitHub 7236e82911 Merge pull request #397 from hnnsgstfssn/add-pkger-readme
source/pkger: add readme
2020-05-25 23:15:14 -07:00
Hannes Gustafsson 7b58ded3b3 source/pkger: add readme 2020-05-24 01:32:26 +01:00
Dale Hui 3c7150850e Update golangci-lint from v1.24.0 to v1.27.0 2020-05-21 15:23:17 -07:00
Dale Hui cd9f7bce91 Update neo4j db driver from v1.8.0-beta01 to v1.8.0-beta02
https://github.com/neo4j/neo4j-go-driver/compare/v1.8.0-beta01...v1.8.0-beta02
2020-05-21 13:37:35 -07:00
Dale Hui 805c416251 Use neo4j v1.8.0-beta01 client
The neo4j v1.8 client drops cgo dependencies (seabolt)
2020-05-21 13:32:52 -07:00
Dale HuiandGitHub 371e18dde4 Merge pull request #391 from shiwano/shiwano/mongodb-srv
Register mongodb+srv scheme
2020-05-18 10:45:39 -07:00
Shogo IwanoandGitHub 524edc9f0b Add -f option to drop command (#390)
* Add -y option to drop command

* Replace -y -> -f

* Update -f help text of drop command
2020-05-18 10:42:14 -07:00
Shogo Iwano 306d4577a5 Update README.md 2020-05-18 22:32:26 +09:00
Shogo Iwano 894821e1ef Register mongodb+srv scheme 2020-05-17 10:29:46 +09:00
Dale HuiandGitHub b88b868b91 Merge pull request #386 from KadenLNelson/update-cassandra-ssl-options
Update Cassandra SSL options
2020-05-12 10:19:35 -07:00
Kaden L. Nelson b8ee1b9733 update cassandra ssl options 2020-05-11 19:24:52 -07:00
hnnsgstfssnandGitHub f5a22be0a0 Add pkger source driver support (#377)
* Add pkger source driver support

As go-bindata has been abandoned [1] there are open requests, #116, for
alternative sources with similar functionality. The Buffalo project [2]
created packr and recently pkger [3] was announced [4] with the
intention to supersede packr.

This change adds support for using pkger as a source.

The implementation relies on httpfs.PartialDriver for pretty much all
functionality.

[1] https://github.com/jteeuwen/go-bindata/issues/5
[2] https://gobuffalo.io/
[3] https://github.com/markbates/pkger
[4] https://blog.gobuffalo.io/introducing-pkger-static-file-embedding-in-go-1ce76dc79c65

* pkger: rename Instance to Pkger

* pkger: make WithInstance accept *Pkger

* pkger: refactor and add access to global pkging.Pkger instance

* pkger: fix typo and cleanup debug logging
2020-04-21 16:43:53 -07:00
Dimas Ragil TandGitHub 423c2d35ee Add neo4j driver on params instance (#373)
* Add neo4j driver on instance params

* Check constraint if exist return nil

* Create new driver if nil

* Remove URL and AuthToken from neo4j config

* Change call db constraint

* Add neo4j image 4.0

* Handle create new driver

* Use labels instead constraints

* Turn off TLS encryption on test
2020-04-16 15:16:07 -07:00
Dale Hui 833d00711e Run go mod tidy 2020-04-15 02:41:33 -07:00
Dale HuiandGitHub ace4ca8556 Merge pull request #375 from jmramos02/master
Upgrade multierror to v1.1.0
2020-04-15 02:37:30 -07:00
Jose Mari Ramos 8a7f164b67 Upgrade multierror to v1.1.0 2020-04-15 10:25:13 +08:00
Dale Hui 5180ec47dc [DB-DRIVER][neo4j] Properly handle null ts 2020-04-06 05:09:41 -07:00
Dale Hui 4598673138 Update golangci-lint from v1.23.6 to v1.24.0 2020-04-06 00:07:15 -07:00
Dale Hui 2c5115907f [DB-DRIVER][neo4j] Track timestamp when schema version was last updated 2020-04-06 00:07:15 -07:00
Dale Hui 3d2a99ec64 Correctly mark migrations as dirty when the first down migration fails
Fixes: https://github.com/golang-migrate/migrate/issues/330

Other changes:
  - Cleanup code by using the database.NilVersion constant instead of the magic number -1
2020-04-06 00:07:15 -07:00
Dale HuiandGitHub 20a1dcfe80 Fix command for installing with Go toolchain (versioned)
Thanks pointing out the issue @JKKGBE !
2020-04-03 22:59:35 -07:00
Dale HuiandGitHub 43f6d84de7 Make "force db version" answer more descriptive 2020-03-26 15:01:32 -07:00
Dale HuiandGitHub 00b39b2b6c Merge pull request #361 from mknycha/add-docs-info-about-dirty-database
Docs: add info about troubleshooting dirty database
2020-03-25 18:22:46 -07:00
Marcin K 3a349451e2 Docs: add info about troubleshooting dirty database
[#337]
2020-03-24 19:16:26 +01:00
Dale HuiandGitHub 098a3e7efe Merge pull request #360 from Rocketmakers/drop-prompt
Prompt before dropping the entire database
2020-03-18 12:30:57 -07:00
Nicholas Soper ec26edc735 Give positive feedback when dropping schema 2020-03-18 09:02:00 +00:00
Nicholas Soper 1f66249648 Prompt before dropping the entire database 2020-03-17 17:37:27 +00:00
Dale HuiandGitHub 5cc28f061e Merge pull request #356 from jace-ys/master
Move migrate binary to /usr/local/bin in Dockerfile
2020-03-14 17:24:58 -07:00
jace-ys 6cdebb0871 Move migrate binary to /usr/local/bin in Dockerfile
Move the migrate binary to `/usr/local/bin` in the Dockerfile, so
it can be invoked from anywhere - this makes it easier to write
entrypoint scripts which call `migrate` that can be used
consistently both locally and in Docker containers.

To preserve backwards compatibility for anything that depends on
`migrate` existing at its existing path, we create a symlink to the
`/usr/local/bin/migrate` binary.
2020-03-14 19:21:12 +00:00
8e142dfaa7 Improved migration creation + Windows fix (#352)
* CLI: Improve migration creation.

* Validates migration version on creation, to avoid creation of duplicated versions
* Uses `os.OpenFile` with `O_CREATE|O_EXCL` to create files to avoid file collisions
* Uses `filepath.Join` to concatenate paths, making `cleanPath()` not necessary
* Prints generated filenames
* Fixes #238
* Supersedes #250

* CLI: change `createCmd` to return error and accept `print` parameter

* feat: print out an abs path

Better for Windows OS when specify -dir as /subdir, user will see C:/subdir/0001_name.up.sql rather than /subdir/0001_name.up.sql

* test: fixed abs path test fail for OS Windows

abs path tests fails because filepath.IsAbs() treats `/subdir` path as invalid abs path at windows when drive letter is not present

* feat: print absolute path for created files

Better for Windows OS systems where `/path` can be interpreted in different ways depending on working dir

* test: corrected tests for OS Windows

OS Windows has different interpretation of `/path`, it depends on working dir.
If working dir D:\test it interprets `/path` as `D:\path`

* test: fixed `dir invalid` test

Linux OS has less restriction on a filepath than Windows, so path invalid in windows is perfectly valid for Linux. The only invalid dir name in Linux is one ending with null string terminator (\000)

* refac(cli): *Cmd() now returns an error and not uses log.fatalErr(err)

* docs: added godoc, migarate usage updated

* refac(cli): code refactored

* refac: removed unnecessary path covert

* docs: comment added

* test: fixed code review issue, noErrorExpected var removed

https://github.com/golang-migrate/migrate/pull/352#discussion_r389409789

* docs: fixed godoc

Co-authored-by: Kiyoshi '13k' Murata <kbmurata@gmail.com>
2020-03-10 16:59:11 -07:00
Dale Hui 9b3db6cd7d Fix README markdown lint issues 2020-03-08 15:28:53 -07:00
Dale Hui fb3ffecbb5 Add support for Go 1.14 and drop support for Go 1.12 2020-03-08 15:28:29 -07:00
Dale Hui 1de555b72d Update cockroach db
Fixes: https://github.com/golang-migrate/migrate/issues/308
2020-03-04 08:34:50 -08:00
Dale HuiandGitHub 39cc0970bd Merge pull request #350 from saj/saj/sqlite-no-implicit-tx
sqlite3: Allow users to disable implicit transactions
2020-03-02 20:08:16 -08:00
Saj Goonatilleke 831d8c9ae6 sqlite3: Explicitly reject invalid x-no-tx-wrap values 2020-03-02 22:28:18 +11:00
Saj Goonatilleke d75ded973a sqlite3: Add README 2020-02-29 05:47:13 +11:00
Saj Goonatilleke 583757ab2d sqlite3: Allow users to disable implicit transactions
An sqlite3 database may be opened with the x-no-tx-wrap=true query
parameter if the user would prefer to retain full control over the
bounds of their transactions.  With x-no-tx-wrap enabled, migrations
should contain explicit BEGIN and COMMIT statements.  This (optional)
behaviour for the sqlite3 driver is consistent with the default
behaviour for other drivers.

Fixes #346.
2020-02-29 05:14:59 +11:00
Dale HuiandGitHub 6dc95bd201 Merge pull request #348 from martinlindhe/master
Add Windows install instructions using scoop
2020-02-25 08:59:46 -08:00
Martin Lindhe b4872fb19d Add Windows install instructions using scoop 2020-02-25 11:45:36 +01:00
Dale Hui 777550e4d7 Actually release windows 386 build 2020-02-17 08:46:08 -08:00
Dale HuiandGitHub 51f9aabf68 Merge pull request #343 from martinlindhe/win386
build-cli: Also build GOOS=windows GOARCH=386, fixes #339
2020-02-17 02:23:11 -08:00
Dale HuiandGitHub cbadf63c13 Merge branch 'master' into win386 2020-02-16 22:53:07 -08:00
Dale Hui d9d6d85ecd POSTGRES_PASSWORD is now required when using postgres via docker
See: https://github.com/docker-library/postgres/issues/681
2020-02-16 01:37:09 -08:00
Dale Hui 9dcc6f061f Drop support for postgres 9.4 and add support for postgres 12
https://www.postgresql.org/about/news/2011/
https://www.postgresql.org/about/news/1976/
2020-02-16 00:38:23 -08:00
Martin Lindhe 115177e679 build-cli: Also build GOOS=windows GOARCH=386, fixes #339 2020-02-16 05:11:07 +01:00
Dale Hui 67536b7f1c Update github.com/fsouza/fake-gcs-server from v1.7.0 to v1.17.0 2020-02-14 01:14:05 -08:00
Dale Hui 9e6ca5ccaa Update cloud.google.com/go from v0.37.4 to v0.53.0 2020-02-14 00:44:21 -08:00
Dale Hui 7291b60643 Update golangci-lint from v1.22.2 to v1.23.6 2020-02-14 00:08:02 -08:00
Dale Hui 64db556342 Update dktest from v0.3.1 to v0.3.2
Addresses: https://github.com/golang-migrate/migrate/issues/338
2020-02-13 23:35:01 -08:00
Dale HuiandGitHub 607b1142fa Merge pull request #336 from Glyphack/add-errnochange-check-up
Add ErrNoChange check when using Up.
2020-02-12 00:44:59 -08:00
glyphack 22b2dbf747 Add ErrNoChange check when using Up. https://github.com/golang-migrate/migrate/issues/100 2020-02-12 00:55:09 +03:30
Dale HuiandGitHub e1085cfef2 Merge pull request #334 from bishtawi/bishtawi/aws-session-override
Allow passing in your own aws s3client object
2020-02-04 18:30:44 -08:00
Stephen Bishtawi 1637f18132 Cleanup 2020-02-04 17:58:29 -08:00
Stephen Bishtawi 453547cf81 Cleanup 2020-02-04 09:40:17 -08:00
Stephen Bishtawi a50ea56eda Cleanup 2020-02-03 12:32:27 -08:00
Dale HuiandGitHub 402f080149 Merge pull request #335 from bcomnes/patch-1
Fix URL in tutorial
2020-01-31 15:48:44 -08:00
Bret ComnesandGitHub a74470d619 Fix URL in tutorial
This link needed to be updated.
2020-01-31 14:13:30 -07:00
Stephen Bishtawi df9067fec5 Update 2020-01-31 10:56:16 -08:00
Stephen Bishtawi 5d38bead9a Implement WithInstance pattern 2020-01-31 10:24:31 -08:00
Stephen Bishtawi c2d3db8c2b More testing 2020-01-30 13:00:45 -08:00
Stephen Bishtawi 0f10e74f69 Add test 2020-01-30 11:09:08 -08:00
Stephen Bishtawi 459f15b500 Fix when prefix is empty 2020-01-29 17:31:18 -08:00
Stephen Bishtawi 5439c3a702 Allow using a custom aws session 2020-01-29 14:57:07 -08:00
Mantas VidutisandDale Hui 69093192ef Allow for multistatement migrations in Neo4j (#328)
* allow for multistatement migrations

* multi statement option

* parse bool and package const

* default false multi, change package var to byte[]
2020-01-22 23:59:32 -08:00
f29923306d add statement timeout query param for postgres (#327)
* add statement timeout query param for postgres

* Update database/postgres/postgres.go

Co-Authored-By: Roopak Venkatakrishnan <roopak.v@gmail.com>

* add enable flag

* remove deafult catch errors

* reformat

* only use background context for 0 timeout

* refactor

* refactor

Co-authored-by: Pranay Suresh <53918678+psuresh309@users.noreply.github.com>
Co-authored-by: Roopak Venkatakrishnan <roopak.v@gmail.com>
2020-01-21 11:22:41 -08:00
Dale Hui 8b62949069 Update go-sql-driver from v1.4.1 to v1.5.0 2020-01-20 22:34:00 -08:00
Dale Hui 8346ecefea Update pq from v1.0.0 to v1.3.0 2020-01-20 22:33:01 -08:00
Mantas VidutisandDale Hui 4ff83b9d74 Fix issues with Neo driver (#324)
* initial pass

* new cli build file

* add ready function for docker start

* lowercase j

* explicit return

* test migration

* undo dockerfile change

* neo4j in dockerfile

* remove auth from url once transferred to auth token

* assorted review notes

* initial docs attempt

* unlock behavior more in line with lock behavior

* consistent naming

* linting

* seabolt in travis ci

* stdlib logger

* install seabolt as sudo

* enabled cgo

* correct version #

* remove neo4j todo

* named return values for errors

* review notes

* add libssl for alpine

* updates for review

* updates for neo driver

* go mod tidy
2020-01-19 18:56:51 -08:00
Dale Hui dde480b416 Reference seabolt in neo4j db driver README 2020-01-14 22:36:26 -08:00
Dale Hui 57c0e0d7c0 Rename "downloader" Docker build stage to "builder" 2020-01-14 22:24:00 -08:00
Dale Hui 9ca70b31ef Split up Go module fetching in Docker builds to take advantage of Docker build cache 2020-01-14 22:24:00 -08:00
Dale Hui 07bed0e214 Shallow clone seabolt repo 2020-01-14 22:24:00 -08:00
Dale Hui 3de5c9af24 Support SQLite in Docker CLI 2020-01-14 22:24:00 -08:00
Dale Hui c32255c13d Fix broken Docker CLI
Need to copy over seabolt shared/static library
2020-01-14 22:24:00 -08:00
Dale Hui 407d0cec58 Run go mod tidy 2020-01-14 22:24:00 -08:00
Mantas VidutisandDale Hui 3061f4564a Neo4J Support (#320)
* initial pass

* new cli build file

* add ready function for docker start

* lowercase j

* explicit return

* test migration

* undo dockerfile change

* neo4j in dockerfile

* remove auth from url once transferred to auth token

* assorted review notes

* initial docs attempt

* unlock behavior more in line with lock behavior

* consistent naming

* linting

* seabolt in travis ci

* stdlib logger

* install seabolt as sudo

* enabled cgo

* correct version #

* remove neo4j todo

* named return values for errors

* review notes

* add libssl for alpine

* updates for review
2020-01-14 21:59:22 -08:00
Dale Hui 9e2a0cdc38 Use new ql driver location 2020-01-03 02:59:33 -08:00
Dale Hui 49afea4d86 Don't pass params to SQL statements using fmt.Sprintf() 2020-01-03 02:59:33 -08:00
Dale Hui 4e593fd2a9 Remove unnecessary byte slice arg from TestMigrate() in database/testing 2020-01-03 00:15:52 -08:00
Oras Al-KubaisiandDale Hui 3001e743c3 More examples in Mongo (#268)
* adding/removing a new field in the schema

* example of updating field value from other fields

* added new line at end of files

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2020-01-03 00:12:08 -08:00
Dale Hui f6d44b4b24 Use Golang 1.13 and Alpine Linux 3.11 to build standard Docker images 2020-01-02 22:49:04 -08:00
Dale HuiandGitHub 0eedf49c28 Merge pull request #297 from mknycha/add-tutorial-for-cockroachdb
Add a tutorial for cockroachdb
2020-01-02 21:22:25 -08:00
Marcin K d993b8eaee Prevent shell interpretation in the exported variables 2020-01-02 20:17:56 +01:00
Dale Hui 4146594df3 Rename httpfs package files for improved readability 2020-01-02 00:10:22 -08:00
Dale Hui 8261b52d33 Use httpfs.PartialDriver as main godoc_vfs source implementation 2020-01-02 00:10:22 -08:00
Dale Hui a4820755ab Use httpfs.PartialDriver as main file source implementation 2020-01-01 16:26:24 -08:00
Dale Hui c27f930c6e Update golangci-lint from v1.20.0 to v1.22.2 2019-12-31 19:24:20 -08:00
Dale HuiandGitHub 6355956008 Merge pull request #316 from pcarranza/pc/add-arm-binaries
Add arm binaries to the build-cli phase
2019-12-31 19:10:11 -08:00
Dale HuiandGitHub 2f4bfa29e5 Merge branch 'master' into pc/add-arm-binaries 2019-12-31 17:39:25 -08:00
Dale Hui 4100c1d952 Update dktest from v0.3.0 to v0.3.1 2019-12-31 17:11:17 -08:00
Dale Hui 7965ee37ab Fix failing sqlserver tests by using latest images
Don't need to specify Ubuntu in tag since all mcr.microsoft.com/mssql/server images are based on Ubuntu anyways
2019-12-31 17:10:49 -08:00
Pablo Carranza fb317bb597 Add arm binaries to the build-cli phase 2019-12-29 11:18:51 +01:00
Dale Hui 6b1121a658 Tweak source.httpfs README 2019-12-24 15:38:36 -08:00
Dale Hui 525949de9f Add additional information in source.ErrDuplicateMigration struct 2019-12-24 15:35:08 -08:00
Dale HuiandGitHub 09f532b773 Merge pull request #293 from fln/master
Add http.FileSystem migration source driver support
2019-12-24 15:30:35 -08:00
Julius Kriukas dc7fa068e3 Drop errors.Is() usage to support go 1.12.x 2019-12-19 15:46:12 +02:00
Julius Kriukas e57ac62141 Rename PartialDriver receiver variable to p 2019-12-19 15:36:14 +02:00
Julius Kriukas 516bfe1c27 Drop delayed error reporting 2019-12-19 15:30:00 +02:00
Julius Kriukas 0491d12e50 Rename Migrator -> PartialDriver 2019-12-19 15:13:02 +02:00
Julius Kriukas b7e05731ea Add new test with example driver implemenation 2019-12-16 12:01:52 +02:00
Julius Kriukas 0bea9205d9 Improve driver.Open() error message 2019-12-16 11:06:52 +02:00
Julius Kriukas bf817e5de6 Fix README.md and documentation typo 2019-12-16 10:15:28 +02:00
Julius Kriukas bfa4bdf7a0 Split New() tests 2019-12-16 10:11:31 +02:00
Julius Kriukas 9e851de675 Split source and tests into two files to match data types 2019-12-13 17:41:47 +02:00
Julius Kriukas 98a84cdaec Fix documentation 2019-12-13 17:31:06 +02:00
Julius Kriukas 46206b7330 Split Migrator and Driver tests 2019-12-13 02:09:14 +02:00
Julius Kriukas cf23a7307e Split Migrator and Driver structures 2019-12-13 02:01:24 +02:00
Julius Kriukas 515a6d1664 Move ErrDuplicateMigration to the source package 2019-12-13 01:04:06 +02:00
Julius Kriukas 1fc246e82d Prefer field mutation over struct replacement 2019-12-13 00:59:23 +02:00
Julius Kriukas 33cf17647b Use string concatenation instead of fmt.Sprintf() 2019-12-13 00:20:02 +02:00
Julius Kriukas df5e294c5c Simplify not test.ok to else 2019-12-13 00:13:23 +02:00
Julius Kriukas 29df549a5b Add ErrDuplicateMigration typed error 2019-12-12 16:49:30 +02:00
Julius Kriukas 49ae0330b0 Test for error on First() when no migration files are present 2019-12-12 16:13:43 +02:00
Julius Kriukas 20e6881fd8 Cover subdir ignore code path 2019-12-12 15:42:56 +02:00
Julius Kriukas 1ed56cb468 Use external package for tests 2019-12-12 15:36:24 +02:00
Julius Kriukas 73ba745097 Replace WithInstance with Init, make main struct public 2019-12-12 12:06:18 +02:00
Julius Kriukas 7b6a813149 Replace test messages with sub-test names 2019-12-12 11:00:26 +02:00
Julius Kriukas 9c45ae5db0 Switch to if ok pattern for ReadUp and ReadDown 2019-12-12 10:36:01 +02:00
Julius Kriukas 9c9a49920d Drop coverage.out 2019-12-11 17:51:18 +02:00
Julius KriukasandJulius Kriukas 41dfe1d575 Add httpfs.New() constructor with delayed errors
This PR adds a new httpfs source driver constructor New(). It is
identical to the WithInstance() constructor but will postpone any errors
until the driver is actually being used.

This allows clients of this package to have less error checks without
loosing correctness.

In addition this PR adds more tests, improves test code coverage, and
makes golinter happy by removing deferred Close call.
2019-12-06 15:55:07 +02:00
Julius KriukasandJulius Kriukas b5bd00dc8a Add http.FileSystem migration source driver support
This PR adds a new migration source driver capable of reading files from
any source that implements http.FileSystem interface.

Notable http.FileSystem interface implementations:

* http.Dir() - wrapper over local file-system
* github.com/shurcooL/vfsgen

Because user of this package is responsible for getting an
implementation of http.FileSystem, this driver does not support creating
instances from driver URLs.
2019-12-06 15:55:07 +02:00
Marcin K 89b51584c5 Add a reference to the tutorial in README 2019-11-29 08:00:00 +01:00
Marcin K 63f26f6fb9 Add a simple tutorial for cockroachdb 2019-11-29 08:00:00 +01:00
Dale HuiandGitHub 41b578a45e Merge pull request #305 from bxcodec/fixClickHouse
hotfix(dependency): resolve clickhouse dependency breaking changes
2019-11-21 00:26:34 -08:00
Iman Tumorang 9d1abc92fa fix(dependency): resolve clickhouse dependency breaking changes 2019-11-21 10:18:48 +08:00
Dale Hui ae62039386 Add official support for firebird back 2019-10-17 08:15:47 -07:00
Dale Hui a7c2a28bec Deploy using Go 1.13 2019-10-17 07:53:29 -07:00
Dale Hui dc9b4dab5c Only populate db config values if it's not specified
See also: https://github.com/golang-migrate/migrate/issues/262
2019-10-17 07:48:52 -07:00
Dale HuiandGitHub 7f20f01a3b Merge pull request #288 from jon-whit/master
Trim statement leading and trailing whitespace.
2019-10-14 20:53:54 -07:00
Dale Hui 57b2ba7115 Actually use schema and db names specified in postgres db config
Addresses: https://github.com/golang-migrate/migrate/issues/262
2019-10-13 23:24:10 -07:00
Dale Hui bbe4ec3cde Update golangci-lint to v1.20.0 2019-10-12 15:14:27 -07:00
Dale Hui 84f30c64d0 Specify the expected min go version as 1.12 2019-10-12 15:14:21 -07:00
Jonathan Whitaker 241ad96665 Trim statement leading and trailing whitespace.
The changes herein make sure Spanner DDL statements have leading and
trailing whitespace characters trimmed.
2019-10-03 14:35:37 -06:00
Dale HuiandGitHub fc060940b4 Merge pull request #287 from ctuong/pg-lock-comment
Update postgres driver Lock comment to reflect actual behavior
2019-09-25 00:57:45 -07:00
Calvin Tuong 9ef241d273 Update postgres driver Lock comment to reflect actual behavior 2019-09-24 13:57:01 -07:00
Dale Hui 002e0aced4 Update golangci-lint to v1.18.0 for Go 1.13 compatibility 2019-09-17 02:21:45 -07:00
Dale Hui be590bf1b9 Drop support for Go 1.11 and add support for Go 1.13 2019-09-09 08:00:12 -07:00
256 changed files with 14444 additions and 2184 deletions
+56
View File
@@ -0,0 +1,56 @@
# Golang CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-go/ for more details
version: 2.1
jobs:
"golang-1_15": &template
machine:
# https://circleci.com/docs/2.0/configuration-reference/#available-machine-images
image: ubuntu-2004:202010-01
# docker_layer_caching: true
# https://circleci.com/docs/2.0/configuration-reference/#resource_class
resource_class: medium
# Leave working directory unspecified and use defaults:
# https://circleci.com/blog/go-v1.11-modules-and-circleci/
# working_directory: /go/src/github.com/golang-migrate/migrate
environment:
GO111MODULE: "on"
GO_VERSION: "1.15.x"
steps:
# - setup_remote_docker:
# version: 19.03.13
# docker_layer_caching: true
- run: curl -sL -o ~/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
- run: curl -sfL -o ~/bin/golangci-lint.sh https://install.goreleaser.com/github.com/golangci/golangci-lint.sh
- run: chmod +x ~/bin/gimme ~/bin/golangci-lint.sh
- run: eval "$(gimme $GO_VERSION)"
- run: golangci-lint.sh -b ~/bin v1.37.0
- checkout
- restore_cache:
keys:
- go-mod-v1-{{ arch }}-{{ checksum "go.sum" }}
- run: golangci-lint run
- run: make test COVERAGE_DIR=/tmp/coverage
- save_cache:
key: go-mod-v1-{{ arch }}-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod"
- run: go get github.com/mattn/goveralls
- run: goveralls -service=circle-ci -coverprofile /tmp/coverage/combined.txt
"golang-1_16":
<<: *template
environment:
GO_VERSION: "1.16.x"
workflows:
version: 2
build:
jobs:
- "golang-1_15"
- "golang-1_16"
-1
View File
@@ -2,7 +2,6 @@
FAQ.md
README.md
LICENSE
Makefile
.gitignore
.travis.yml
CONTRIBUTING.md
+1 -1
View File
@@ -25,7 +25,7 @@ e.g. s3, github, go-bindata, gcs, file
Obtained by running: `migrate -help`
**Loaded Database Drivers**
e.g. spanner, stub, clickhouse, cockroachdb, crdb-postgres, postgres, postgresql, redshift, cassandra, cockroach, mysql
e.g. spanner, stub, datastore, cockroachdb, crdb-postgres, postgres, postgresql, pgx, redshift, cassandra, cockroach, mysql
Obtained by running: `migrate -help`
**Go Version**
+9
View File
@@ -0,0 +1,9 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1280" height="640" viewBox="0 0 1280 640" role="img" aria-label="migrate">
<rect width="1280" height="640" fill="#0A0A0A"/>
<svg x="96" y="215" width="210" height="210" viewBox="0 0 67 67"><path d="M22.21 67V44.6369H0V67H22.21Z" fill="#fff"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z" fill="#fff"/><path d="M22.21 0H0V22.3184H22.21V0Z" fill="#fff"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z" fill="#fff"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z" fill="#fff"/></svg>
<text x="378" y="276" font-family="Inter,system-ui,-apple-system,sans-serif" font-size="78" font-weight="800" letter-spacing="-2" fill="#ffffff">migrate</text>
<text x="378" y="322" font-family="Inter,system-ui,sans-serif" font-size="30" fill="#ffffff" opacity=".66">Database migrations. CLI and Golang library.</text>
<rect x="378" y="338" width="806" height="3" rx="1.5" fill="#ffffff" opacity=".9"/>
<text x="378" y="390" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">github.com/hanzoai</text>
<text x="1184" y="390" text-anchor="end" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">hanzo.ai</text>
</svg>

After

Width:  |  Height:  |  Size: 1.3 KiB

+111
View File
@@ -0,0 +1,111 @@
name: CI
on:
push:
pull_request:
jobs:
lint:
name: lint
runs-on: hanzo-build-linux-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: "1.26.4.x"
- name: golangci-lint
uses: golangci/golangci-lint-action@v9
with:
version: v2.6.2
test:
runs-on: hanzo-build-linux-amd64
strategy:
matrix:
go: ["1.24.x", "1.25.x"]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go }}
- name: Run test
run: make test COVERAGE_DIR=/tmp/coverage
- name: Send goveralls coverage
uses: shogo82148/actions-goveralls@v1
with:
path-to-profile: /tmp/coverage/combined.txt
flag-name: Go-${{ matrix.go }}
parallel: true
check-coverage:
name: Check coverage
needs: [test]
runs-on: hanzo-build-linux-amd64
steps:
- uses: shogo82148/actions-goveralls@v1
with:
parallel-finished: true
goreleaser:
name: Release a new version
needs: [lint, test]
runs-on: hanzo-build-linux-amd64
environment: GoReleaser
# This job only runs when
# 1. When the previous `lint` and `test` jobs has completed successfully
# 2. When the repository is not a fork, i.e. it will only run on the official golang-migrate/migrate
# 3. When the workflow is triggered by a tag with `v` prefix
if: ${{ success() && github.repository == 'golang-migrate/migrate' && startsWith(github.ref, 'refs/tags/v') }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: ruby/setup-ruby@v1
with:
ruby-version: 2.7
- uses: actions/setup-go@v5
with:
go-version: "1.26.4.x"
- uses: docker/setup-qemu-action@v3
- uses: docker/setup-buildx-action@v3
- uses: docker/login-action@v3
with:
username: golangmigrate
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: echo "SOURCE=$(make echo-source)" >> $GITHUB_ENV
- run: echo "DATABASE=$(make echo-database)" >> $GITHUB_ENV
- uses: goreleaser/goreleaser-action@v5
with:
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: gem install package_cloud
- run: package_cloud push golang-migrate/migrate/ubuntu/bionic dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/focal dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/jammy dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/noble dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/buster dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/bullseye dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/bookworm dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
+52
View File
@@ -0,0 +1,52 @@
name: Release
on:
push:
tags:
- 'v*'
permissions:
contents: write
packages: write
jobs:
release:
runs-on: hanzo-build-linux-amd64
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: '1.26.4'
- name: Build CLI binaries
run: |
VERSION=${GITHUB_REF_NAME#v}
mkdir -p cli/build
for pair in "linux/amd64" "linux/arm64" "darwin/amd64" "darwin/arm64"; do
OS=${pair%/*}
ARCH=${pair#*/}
OUTPUT="cli/build/migrate.${OS}-${ARCH}"
echo "Building $OUTPUT..."
CGO_ENABLED=0 GOOS=$OS GOARCH=$ARCH \
go build -a -o "$OUTPUT" \
-ldflags="-s -w -X main.Version=$VERSION" \
-tags 'datastore file' \
./cmd/migrate
ln -f "$OUTPUT" cli/build/migrate
tar czf "${OUTPUT}.tar.gz" -C cli/build migrate
done
cd cli/build && sha256sum *.tar.gz > sha256sum.txt
- name: Create GitHub Release
env:
GH_TOKEN: ${{ github.token }}
run: |
gh release create ${{ github.ref_name }} \
cli/build/*.tar.gz \
cli/build/sha256sum.txt \
--title "${{ github.ref_name }}" \
--generate-notes \
--latest
+3
View File
@@ -6,3 +6,6 @@ cli/migrate
.godoc.pid
vendor/
.vscode/
.idea
dist/
migrate
+30 -20
View File
@@ -1,27 +1,37 @@
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
deadline: 2m
version: "2"
linters:
enable:
#- golint
- interfacer
- unconvert
#- dupl
- goconst
- gofmt
- misspell
- maligned
- unparam
- nakedret
- prealloc
#- gosec
linters-settings:
misspell:
locale: US
- revive
- unconvert
- unparam
settings:
misspell:
locale: US
revive:
rules:
- name: redundant-build-tag
exclusions:
generated: lax
rules:
- path: (.+)\.go$
text: G104
paths:
- third_party$
- builtin$
- examples$
issues:
max-same: 0
max-per-linter: 0
exclude-use-default: false
exclude:
# gosec: Duplicated errcheck checks
- G104
max-issues-per-linter: 0
max-same-issues: 0
formatters:
enable:
- gofmt
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
+102
View File
@@ -0,0 +1,102 @@
project_name: migrate
before:
hooks:
- go mod tidy
builds:
- env:
- CGO_ENABLED=0
goos:
- linux
- windows
- darwin
goarch:
- amd64
- arm
- arm64
- 386
goarm:
- 7
main: ./cmd/migrate
ldflags:
- '-w -s -X main.Version={{ .Version }} -extldflags "static"'
flags:
- "-tags={{ .Env.DATABASE }} {{ .Env.SOURCE }}"
- "-trimpath"
nfpms:
- homepage: "https://github.com/golang-migrate/migrate"
maintainer: "dhui@users.noreply.github.com"
license: MIT
description: "Database migrations"
formats:
- deb
file_name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
dockers:
- goos: linux
goarch: amd64
dockerfile: Dockerfile.github-actions
use: buildx
ids:
- migrate
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
build_flag_templates:
- '--label=org.opencontainers.image.created={{ .Date }}'
- '--label=org.opencontainers.image.title={{ .ProjectName }}'
- '--label=org.opencontainers.image.revision={{ .FullCommit }}'
- '--label=org.opencontainers.image.version={{ .Version }}'
- "--label=org.opencontainers.image.source={{ .GitURL }}"
- "--platform=linux/amd64"
- goos: linux
goarch: arm64
dockerfile: Dockerfile.github-actions
use: buildx
ids:
- migrate
image_templates:
- 'migrate/migrate:{{ .Tag }}-arm64'
build_flag_templates:
- '--label=org.opencontainers.image.created={{ .Date }}'
- '--label=org.opencontainers.image.title={{ .ProjectName }}'
- '--label=org.opencontainers.image.revision={{ .FullCommit }}'
- '--label=org.opencontainers.image.version={{ .Version }}'
- "--label=org.opencontainers.image.source={{ .GitURL }}"
- "--platform=linux/arm64"
docker_manifests:
- name_template: 'migrate/migrate:{{ .Tag }}'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
- name_template: 'migrate/migrate:{{ .Major }}'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
- name_template: 'migrate/migrate:latest'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
archives:
- name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
format_overrides:
- goos: windows
format: zip
checksum:
name_template: 'sha256sum.txt'
release:
draft: true
prerelease: auto
source:
enabled: true
format: zip
changelog:
skip: false
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
- Merge pull request
- Merge branch
- go mod tidy
snapshot:
name_template: "{{ .Tag }}-next"
+14 -11
View File
@@ -6,8 +6,8 @@ matrix:
- go: master
include:
# Supported versions of Go: https://golang.org/dl/
- go: "1.11.x"
- go: "1.12.x"
- go: "1.14.x"
- go: "1.15.x"
- go: master
go_import_path: github.com/golang-migrate/migrate
@@ -34,7 +34,7 @@ before_install:
- sudo apt-get update
- sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce
# Install golangci-lint
- curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.17.1
- curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.30.0
- echo "TRAVIS_GO_VERSION=${TRAVIS_GO_VERSION}"
install:
@@ -57,13 +57,16 @@ deploy:
secure: hWH1HLPpzpfA8pXQ93T1qKQVFSpQp0as/JLQ7D91jHuJ8p+RxVeqblDrR6HQY/95R/nyiE9GJmvUolSuw5h449LSrGxPtVWhdh6EnkxlQHlen5XeMhVjRjFV0sE9qGe8v7uAkiTfRO61ktTWHrEAvw5qpyqnNISodmZS78XIasPODQbNlzwINhWhDTHIjXGb4FpizYaL3OGCanrxfR9fQyCaqKGGBjRq3Mfq8U6Yd4mApmsE+uJxgaZV8K5zBqpkSzQRWhcVGNL5DuLsU3gfSJOo7kZeA2G71SHffH577dBoqtCZ4VFv169CoUZehLWCb+7XKJZmHXVujCURATSySLGUOPc6EoLFAn3YtsCA04mS4bZVo5FZPWVwfhjmkhtDR4f6wscKp7r1HsFHSOgm59QfETQdrn4MnZ44H2Jd39axqndn5DvK9EcZVjPHynOPnueXP2u6mTuUgh2VyyWBCDO3CNo0fGlo7VJI69IkIWNSD87K9cHZWYMClyKZkUzS+PmRAhHRYbVd+9ZjKOmnU36kUHNDG/ft1D4ogsY+rhVtXB4lgWDM5adri+EIScYdYnB1/pQexLBigcJY9uE7nQTR0U6QgVNYvun7uRNs40E0c4voSfmPdFO0FlOD2y1oQhnaXfWLbu9nMcTcs4RFGrcC7NzkUN4/WjG8s285V6w=
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
file:
- cli/build/migrate.linux-amd64.tar.gz
- cli/build/migrate.linux-armv7.tar.gz
- cli/build/migrate.linux-arm64.tar.gz
- cli/build/migrate.darwin-amd64.tar.gz
- cli/build/migrate.windows-amd64.exe.tar.gz
- cli/build/migrate.windows-386.exe.tar.gz
- cli/build/sha256sum.txt
- dependency_tree.txt
- provider: packagecloud
@@ -75,7 +78,7 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -87,7 +90,7 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -95,11 +98,11 @@ deploy:
username: golang-migrate
token:
secure: aICwu3gJ1sJ1QVCD3elpg+Jxzt4P+Zj1uoh5f0sOwnjDNIZ4FwUT1cMrWloP8P2KD0iyCOawuZER27o/kQ21oX2OxHvQbYPReA2znLm7lHzCmypAAOHPxpgnQ4rMGHHJXd+OsxtdclGs67c+EbdBfoRRbK400Qz/vjPJEDeH4mh02ZHC2nw4Nk/wV4jjBIkIt9dGEx6NgOA17FCMa3MaPHlHeFIzU7IfTlDHbS0mCCYbg/wafWBWcbGqtZLWAYtJDmfjrAStmDLdAX5J5PsB7taGSGPZHmPmpGoVgrKt/tb9Xz1rFBGslTpGROOiO4CiMAvkEKFn8mxrBGjfSBqp7Dp3eeSalKXB1DJAbEXx2sEbMcvmnoR9o43meaAn+ZRts8lRL8S/skBloe6Nk8bx3NlJCGB9WPK1G56b7c/fZnJxQbrCw6hxDfbZwm8S2YPviFTo/z1BfZDhRsL74reKsN2kgnGo2W/k38vvzIpsssQ9DHN1b0TLCxolCNPtQ7oHcQ1ohcjP2UgYXk0FhqDoL+9LQva/DU4N9sKH0UbAaqsMVSErLeG8A4aauuFcVrWRBaDYyTag4dQqzTulEy7iru2kDDIBgSQ1gMW/yoBOIPK4oi6MtbTf1X39fzXFLS1cDd3LW61yAu3YrbjAetpfx2frIvrRAiL9TxWA1gnrs5o=
dist: ubuntu/cosmic
dist: ubuntu/focal
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -111,7 +114,7 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -123,13 +126,13 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
- provider: script
script: ./docker-deploy.sh
skip_cleanup: true
on:
go: "1.12.x"
go: "1.15.x"
repo: golang-migrate/migrate
tags: true
Symlink
+1
View File
@@ -0,0 +1 @@
LLM.md
Symlink
+1
View File
@@ -0,0 +1 @@
LLM.md
+13 -10
View File
@@ -1,23 +1,26 @@
FROM golang:1.12-alpine3.10 AS downloader
FROM golang:1.26.4-alpine3.21 AS builder
ARG VERSION
RUN apk add --no-cache git gcc musl-dev
RUN apk add --no-cache git gcc musl-dev make
WORKDIR /go/src/github.com/golang-migrate/migrate
ENV GO111MODULE=on
COPY go.mod go.sum ./
RUN go mod download
COPY . ./
ENV GO111MODULE=on
ENV DATABASES="postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver"
ENV SOURCES="file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab"
RUN make build-docker
RUN go build -a -o build/migrate.linux-386 -ldflags="-s -w -X main.Version=${VERSION}" -tags "$DATABASES $SOURCES" ./cmd/migrate
FROM alpine:3.10
FROM alpine:3.21
RUN apk add --no-cache ca-certificates
COPY --from=downloader /go/src/github.com/golang-migrate/migrate/build/migrate.linux-386 /migrate
COPY --from=builder /go/src/github.com/golang-migrate/migrate/build/migrate.linux-386 /usr/local/bin/migrate
RUN ln -s /usr/local/bin/migrate /migrate
ENTRYPOINT ["/migrate"]
ENTRYPOINT ["migrate"]
CMD ["--help"]
+17
View File
@@ -0,0 +1,17 @@
ARG DOCKER_IMAGE
FROM $DOCKER_IMAGE
RUN apk add --no-cache git gcc musl-dev make
WORKDIR /go/src/github.com/golang-migrate/migrate
ENV GO111MODULE=on
ENV COVERAGE_DIR=/tmp/coverage
COPY go.mod go.sum ./
RUN go mod download
COPY . ./
CMD ["make", "test"]
+11
View File
@@ -0,0 +1,11 @@
FROM alpine:3.19
RUN apk add --no-cache ca-certificates
COPY migrate /usr/local/bin/migrate
RUN ln -s /usr/local/bin/migrate /usr/bin/migrate
RUN ln -s /usr/local/bin/migrate /migrate
ENTRYPOINT ["migrate"]
CMD ["--help"]
+8 -5
View File
@@ -16,7 +16,7 @@
NilMigration defines a migration without a body. NilVersion is defined as const -1.
#### What is the difference between uint(version) and int(targetVersion)?
version refers to an existing migration version coming from a source and therefor can never be negative.
version refers to an existing migration version coming from a source and therefore can never be negative.
targetVersion can either be a version OR represent a NilVersion, which equals -1.
#### What's the difference between Next/Previous and Up/Down?
@@ -50,10 +50,10 @@
and whenever we want, not just once at the beginning of all tests.
#### Can I maintain my driver in my own repository?
Yes, technically thats possible. We want to encourage you to contribute your driver to this respository though.
Yes, technically thats possible. We want to encourage you to contribute your driver to this repository though.
The driver's functionality is dictated by migrate's interfaces. That means there should really
just be one driver for a database/ source. We want to prevent a future where several drivers doing the exact same thing,
just implemented a bit differently, co-exist somewhere on Github. If users have to do research first to find the
just implemented a bit differently, co-exist somewhere on GitHub. If users have to do research first to find the
"best" available driver for a database in order to get started, we would have failed as an open source community.
#### Can I mix multiple sources during a batch of migrations?
@@ -65,7 +65,7 @@
and then "force" the expected version.
#### What happens if two programs try and update the database at the same time?
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations at the same time
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations on
the same database at the same time. For example, the MySQL driver uses the `GET_LOCK` function, while the Postgres driver uses
the `pg_advisory_lock` function.
@@ -73,4 +73,7 @@ Database-specific locking features are used by *some* database drivers to preven
No, it is done automatically.
#### Can I use migrate with a non-Go project?
Yes, you can use the migrate CLI in a non-Go project, but there are probably other libraries/frameworks available that offer better test and deploy integrations in that language/framework.
Yes, you can use the migrate CLI in a non-Go project, but there are probably other libraries/frameworks available that offer better test and deploy integrations in that language/framework.
#### I have got an error `Dirty database version 1. Fix and force version`. What should I do?
Keep calm and refer to [the getting started docs](GETTING_STARTED.md#forcing-your-database-version).
+15 -5
View File
@@ -1,7 +1,7 @@
# Getting started
Before you start, you should understand the concept of forward/up and reverse/down database migrations.
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases)
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases).
## Create migrations
Create some migrations using migrate CLI. Here is an example:
@@ -10,7 +10,7 @@ migrate create -ext sql -dir db/migrations -seq create_users_table
```
Once you create your files, you should fill them.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created his migration later gets it merged to the repository first.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created their migration later gets it merged to the repository first.
Developers and Teams should keep an eye on such cases (especially during code review).
[Here](https://github.com/golang-migrate/migrate/issues/179#issuecomment-475821264) is the issue summary if you would like to read more.
@@ -23,17 +23,27 @@ This way if one of commands fails, our database will remain unchanged.
Run your migrations through the CLI or your app and check if they applied expected changes.
Just to give you an idea:
```
migrate -database YOUR_DATBASE_URL -path PATH_TO_YOUR_MIGRATIONS up
migrate -database YOUR_DATABASE_URL -path PATH_TO_YOUR_MIGRATIONS up
```
Just add the code to your app and you're ready to go!
Before commiting your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
Before committing your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
(e.g. if you created a table in a migration but reverse migration did not delete it, you will encounter an error when running the forward migration again)
It's also worth checking your migrations in a separate, containerized environment. You can find some tools in the end of this document.
It's also worth checking your migrations in a separate, containerized environment. You can find some tools at the [end of this document](#further-reading).
**IMPORTANT:** If you would like to run multiple instances of your app on different machines be sure to use a database that supports locking when running migrations. Otherwise you may encounter issues.
## Forcing your database version
In case you run a migration that contained an error, migrate will not let you run other migrations on the same database. You will see an error like `Dirty database version 1. Fix and force version`, even when you fix the erred migration. This means your database was marked as 'dirty'.
You need to investigate the migration error - was your migration applied partially, or was it not applied at all? Once you know, you should force your database to a version reflecting it's real state. You can do so with `force` command:
```
migrate -path PATH_TO_YOUR_MIGRATIONS -database YOUR_DATABASE_URL force VERSION
```
Once you force the version and your migration was fixed, your database is 'clean' again and you can proceed with your migrations.
For details and example of usage see [this comment](https://github.com/golang-migrate/migrate/issues/282#issuecomment-530743258).
## Further reading:
- [PostgreSQL tutorial](database/postgres/TUTORIAL.md)
- [Best practices](MIGRATIONS.md)
+39
View File
@@ -0,0 +1,39 @@
# Hanzo Golang Migrate
## Overview
Go module: github.com/golang-migrate/migrate/v4
## Tech Stack
- **Language**: Go
## Build & Run
```bash
go build ./...
go test ./...
```
## Structure
```
golang-migrate/
CONTRIBUTING.md
Dockerfile
Dockerfile.circleci
Dockerfile.github-actions
FAQ.md
GETTING_STARTED.md
LICENSE
MIGRATIONS.md
Makefile
README.md
SECURITY.md
cli/
cmd/
database/
dktesting/
```
## Key Files
- `README.md` -- Project documentation
- `go.mod` -- Go module definition
- `Makefile` -- Build automation
- `Dockerfile` -- Container build
+21 -6
View File
@@ -1,15 +1,24 @@
SOURCE ?= file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver
SOURCE ?= file go_bindata github github_ee bitbucket aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb yugabytedb datastore mongodb sqlserver firebird neo4j pgx pgx5 rqlite
DATABASE_TEST ?= $(DATABASE) sqlite sqlite3 sqlcipher
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
REPO_OWNER ?= $(shell cd .. && basename "$$(pwd)")
COVERAGE_DIR ?= .coverage
build:
CGO_ENABLED=0 go build -ldflags='-X main.Version=$(VERSION)' -tags '$(DATABASE) $(SOURCE)' ./cmd/migrate
build-docker:
CGO_ENABLED=0 go build -a -o build/migrate.linux-386 -ldflags="-s -w -X main.Version=${VERSION}" -tags "$(DATABASE) $(SOURCE)" ./cmd/migrate
build-cli: clean
-mkdir ./cli/build
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o ../../cli/build/migrate.linux-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -a -o ../../cli/build/migrate.linux-armv7 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -o ../../cli/build/migrate.linux-arm64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -a -o ../../cli/build/migrate.darwin-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=windows GOARCH=386 go build -a -o ../../cli/build/migrate.windows-386.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -a -o ../../cli/build/migrate.windows-amd64.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cli/build && find . -name 'migrate*' | xargs -I{} tar czf {}.tar.gz {}
cd ./cli/build && shasum -a 256 * > sha256sum.txt
@@ -32,7 +41,7 @@ test:
test-with-flags:
@echo SOURCE: $(SOURCE)
@echo DATABASE: $(DATABASE)
@echo DATABASE_TEST: $(DATABASE_TEST)
@go test $(TEST_FLAGS) ./...
@@ -89,6 +98,12 @@ release:
git tag v$(V)
@read -p "Press enter to confirm and push to origin ..." && git push origin v$(V)
echo-source:
@echo "$(SOURCE)"
echo-database:
@echo "$(DATABASE)"
define external_deps
@echo '-- $(1)'; go list -f '{{join .Deps "\n"}}' $(1) | grep -v github.com/$(REPO_OWNER)/migrate | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'
@@ -96,10 +111,10 @@ define external_deps
endef
.PHONY: build-cli clean test-short test test-with-flags html-coverage \
.PHONY: build build-docker build-cli clean test-short test test-with-flags html-coverage \
restore-import-paths rewrite-import-paths list-external-deps release \
docs kill-docs open-docs kill-orphaned-docker-containers
docs kill-docs open-docs kill-orphaned-docker-containers echo-source echo-database
SHELL = /bin/bash
SHELL = /bin/sh
RAND = $(shell echo $$RANDOM)
+34 -17
View File
@@ -1,11 +1,13 @@
[![Build Status](https://img.shields.io/travis/com/golang-migrate/migrate/master.svg)](https://travis-ci.com/golang-migrate/migrate)
[![GoDoc](https://godoc.org/github.com/golang-migrate/migrate?status.svg)](https://godoc.org/github.com/golang-migrate/migrate)
<p align="center"><img src=".github/hero.svg" alt="migrate" width="880"></p>
[![GitHub Workflow Status (branch)](https://img.shields.io/github/actions/workflow/status/golang-migrate/migrate/ci.yaml?branch=master)](https://github.com/golang-migrate/migrate/actions/workflows/ci.yaml?query=branch%3Amaster)
[![GoDoc](https://pkg.go.dev/badge/github.com/golang-migrate/migrate)](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)
[![Coverage Status](https://img.shields.io/coveralls/github/golang-migrate/migrate/master.svg)](https://coveralls.io/github/golang-migrate/migrate?branch=master)
[![packagecloud.io](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/golang-migrate/migrate?filter=debs)
[![Docker Pulls](https://img.shields.io/docker/pulls/migrate/migrate.svg)](https://hub.docker.com/r/migrate/migrate/)
![Supported Go Versions](https://img.shields.io/badge/Go-1.11%2C%201.12-lightgrey.svg)
![Supported Go Versions](https://img.shields.io/badge/Go-1.24%2C%201.25-lightgrey.svg)
[![GitHub Release](https://img.shields.io/github/release/golang-migrate/migrate.svg)](https://github.com/golang-migrate/migrate/releases)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate)](https://goreportcard.com/report/github.com/golang-migrate/migrate)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate/v4)](https://goreportcard.com/report/github.com/golang-migrate/migrate/v4)
# migrate
@@ -24,24 +26,30 @@ Forked from [mattes/migrate](https://github.com/mattes/migrate)
Database drivers run migrations. [Add a new database?](database/driver.go)
* [PostgreSQL](database/postgres)
* [PGX v4](database/pgx)
* [PGX v5](database/pgx/v5)
* [Redshift](database/redshift)
* [Ql](database/ql)
* [Cassandra](database/cassandra)
* [SQLite](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [MySQL/ MariaDB](database/mysql)
* [Neo4j](database/neo4j) ([todo #167](https://github.com/mattes/migrate/issues/167))
* [Cassandra / ScyllaDB](database/cassandra)
* [SQLite](database/sqlite)
* [SQLite3](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [SQLCipher](database/sqlcipher)
* [MySQL / MariaDB](database/mysql)
* [Neo4j](database/neo4j)
* [MongoDB](database/mongodb)
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
* [Google Cloud Spanner](database/spanner)
* [CockroachDB](database/cockroachdb)
* [ClickHouse](database/clickhouse)
* [Firebird](database/firebird) ([todo #49](https://github.com/golang-migrate/migrate/issues/49))
* [YugabyteDB](database/yugabytedb)
* [Datastore](database/datastore)
* [Firebird](database/firebird)
* [MS SQL Server](database/sqlserver)
* [rqlite](database/rqlite)
### Database URLs
Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: `dbdriver://username:password@host:port/dbname?option1=true&option2=false`
Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: `dbdriver://username:password@host:port/dbname?param1=true&param2=false`
Any [reserved URL characters](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters) need to be escaped. Note, the `%` character also [needs to be escaped](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_the_percent_character)
@@ -65,9 +73,12 @@ $
Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go)
* [Filesystem](source/file) - read from filesystem
* [io/fs](source/iofs) - read from a Go [io/fs](https://pkg.go.dev/io/fs#FS)
* [Go-Bindata](source/go_bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata))
* [Github](source/github) - read from remote Github repositories
* [Github Enterprise](source/github_ee) - read from remote Github Enterprise repositories
* [pkger](source/pkger) - read from embedded binary data ([markbates/pkger](https://github.com/markbates/pkger))
* [GitHub](source/github) - read from remote GitHub repositories
* [GitHub Enterprise](source/github_ee) - read from remote GitHub Enterprise repositories
* [Bitbucket](source/bitbucket) - read from remote Bitbucket repositories
* [Gitlab](source/gitlab) - read from remote Gitlab repositories
* [AWS S3](source/aws_s3) - read from Amazon Web Services S3
* [Google Cloud Storage](source/google_cloud_storage) - read from Google Cloud Platform Storage
@@ -78,7 +89,7 @@ Source drivers read migrations from local or remote sources. [Add a new source?]
* Handles ctrl+c (SIGINT) gracefully.
* No config search paths, no config files, no magic ENV var injections.
__[CLI Documentation](cmd/migrate)__
[CLI Documentation](cmd/migrate) (includes CLI install instructions)
### Basic usage
@@ -102,7 +113,7 @@ $ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
* Uses `io.Reader` streams internally for low memory overhead.
* Thread-safe and no goroutine leaks.
__[Go Documentation](https://godoc.org/github.com/golang-migrate/migrate)__
__[Go Documentation](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)__
```go
import (
@@ -136,7 +147,7 @@ func main() {
m, err := migrate.NewWithDatabaseInstance(
"file:///migrations",
"postgres", driver)
m.Steps(2)
m.Up() // or m.Steps(2) if you want to explicitly set the number of migrations to run
}
```
@@ -146,7 +157,8 @@ Go to [getting started](GETTING_STARTED.md)
## Tutorials
- [PostgreSQL](database/postgres/TUTORIAL.md)
* [CockroachDB](database/cockroachdb/TUTORIAL.md)
* [PostgreSQL](database/postgres/TUTORIAL.md)
(more tutorials to come)
@@ -161,6 +173,11 @@ Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-file
[Best practices: How to write migrations.](MIGRATIONS.md)
## Coming from another db migration tool?
Check out [migradaptor](https://github.com/musinit/migradaptor/).
*Note: migradaptor is not affiliated or supported by this project*
## Versions
Version | Supported? | Import | Notes
+16
View File
@@ -0,0 +1,16 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| master | :white_check_mark: |
| 4.x | :white_check_mark: |
| 3.x | :x: |
| < 3.0 | :x: |
## Reporting a Vulnerability
We prefer [coordinated disclosures](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure). To start one, create a GitHub security advisory following [these instructions](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability)
Please suggest potential impact and urgency in your reports.
+31 -12
View File
@@ -7,7 +7,7 @@
[Release Downloads](https://github.com/golang-migrate/migrate/releases)
```bash
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$os-$arch.tar.gz | tar xvz
```
### MacOS
@@ -16,11 +16,19 @@ $ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/m
$ brew install golang-migrate
```
### Windows
Using [scoop](https://scoop.sh/)
```bash
$ scoop install migrate
```
### Linux (*.deb package)
```bash
$ curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
$ echo "deb https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ curl -fsSL https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo gpg --dearmor -o /etc/apt/keyrings/migrate.gpg
$ echo "deb [signed-by=/etc/apt/keyrings/migrate.gpg] https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ apt-get update
$ apt-get install -y migrate
```
@@ -33,13 +41,19 @@ $ apt-get install -y migrate
$ go get -u -d github.com/golang-migrate/migrate/cmd/migrate
$ cd $GOPATH/src/github.com/golang-migrate/migrate/cmd/migrate
$ git checkout $TAG # e.g. v4.1.0
$ go build -tags 'postgres' -ldflags="-X main.Version=$(git describe --tags)" -o $GOPATH/bin/migrate github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.15 and below
$ go build -tags 'postgres' -ldflags="-X main.Version=$(git describe --tags)" -o $GOPATH/bin/migrate $GOPATH/src/github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.16+
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@$TAG
```
#### Unversioned
```bash
$ # Go 1.15 and below
$ go get -tags 'postgres' -u github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.16+
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
```
#### Notes
@@ -49,9 +63,10 @@ $ go get -tags 'postgres' -u github.com/golang-migrate/migrate/cmd/migrate
to build the cli for use with other databases, replace the `postgres` build tag
with the appropriate database tag(s) for the databases desired. The tags
correspond to the names of the sub-packages underneath the
[`database`](../database) package.
[`database`](../../database) package.
1. Similarly to the database build tags, if you need to support other sources, use the appropriate build tag(s).
1. Support for build constraints will be removed in the future: https://github.com/golang-migrate/migrate/issues/60
1. For versions of Go 1.15 and lower, [make sure](https://github.com/golang-migrate/migrate/pull/257#issuecomment-705249902) you're not installing the `migrate` CLI from a module. e.g. there should not be any `go.mod` files in your current directory or any directory from your current directory to the root
## Usage
@@ -71,14 +86,18 @@ Options:
-help Print usage
Commands:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string.
create [-ext E] [-dir D] [-seq] [-digits N] [-format] [-tz] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string. Note: migrations with the same time cause "duplicate migration version" error.
Use -tz option to specify the timezone that will be used when generating non-sequential migrations (defaults: UTC).
goto V Migrate to version V
up [N] Apply all or N up migrations
down [N] Apply all or N down migrations
drop Drop everything inside database
down [N] [-all] Apply all or N down migrations
Use -all to apply all down migrations
drop [-f] Drop everything inside database
Use -f to bypass confirmation
force V Set version V but don't run migration (ignores dirty state)
version Print current migration version
```
@@ -112,7 +131,7 @@ $ migrate -database "$MY_MIGRATE_DATABASE"
Check out https://stedolan.github.io/jq/
```bash
$ migrate -database "$(cat config.json | jq '.database')"
$ migrate -database "$(cat config.json | jq -r '.database')"
```
### YAML files
+11 -4
View File
@@ -1,12 +1,17 @@
# Cassandra
# Cassandra / ScyllaDB
* Drop command will not work on Cassandra 2.X because it rely on
* `Drop()` method will not work on Cassandra 2.X because it rely on
system_schema table which comes with 3.X
* Other commands should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* Other methods should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
**ScyllaDB**
* No additional configuration is required since it is a drop-in replacement for Cassandra.
* The `Drop()` method` works for ScyllaDB 5.1
## Usage
`cassandra://host:port/keyspace?param1=value&param2=value2`
@@ -20,12 +25,14 @@ system_schema table which comes with 3.X
| `consistency` | ALL | Migration consistency
| `protocol` | | Cassandra protocol version (3 or 4)
| `timeout` | 1 minute | Migration timeout
| `connect-timeout` | 600ms | Initial connection timeout to the cluster |
| `username` | nil | Username to use when authenticating. |
| `password` | nil | Password to use when authenticating. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
| `disable-host-lookup`| false | Disable initial host lookup. |
`timeout` is parsed using [time.ParseDuration(s string)](https://golang.org/pkg/time/#ParseDuration)
+99 -44
View File
@@ -4,15 +4,15 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/gocql/gocql"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/golang-migrate/migrate/v4/database/multistmt"
)
func init() {
@@ -20,6 +20,12 @@ func init() {
database.Register("cassandra", db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var DefaultMigrationsTable = "schema_migrations"
var (
@@ -33,11 +39,12 @@ type Config struct {
MigrationsTable string
KeyspaceName string
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Cassandra struct {
session *gocql.Session
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -58,6 +65,10 @@ func WithInstance(session *gocql.Session, config *Config) (database.Driver, erro
config.MigrationsTable = DefaultMigrationsTable
}
if config.MultiStatementMaxSize <= 0 {
config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
c := &Cassandra{
session: session,
config: config,
@@ -120,17 +131,42 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
}
cluster.Timeout = timeout
}
if len(u.Query().Get("connect-timeout")) > 0 {
var connectTimeout time.Duration
connectTimeout, err = time.ParseDuration(u.Query().Get("connect-timeout"))
if err != nil {
return nil, err
}
cluster.ConnectTimeout = connectTimeout
}
if len(u.Query().Get("sslmode")) > 0 && len(u.Query().Get("sslrootcert")) > 0 && len(u.Query().Get("sslcert")) > 0 && len(u.Query().Get("sslkey")) > 0 {
if len(u.Query().Get("sslmode")) > 0 {
if u.Query().Get("sslmode") != "disable" {
cluster.SslOpts = &gocql.SslOptions{
CaPath: u.Query().Get("sslrootcert"),
CertPath: u.Query().Get("sslcert"),
KeyPath: u.Query().Get("sslkey"),
sslOpts := &gocql.SslOptions{}
if len(u.Query().Get("sslrootcert")) > 0 {
sslOpts.CaPath = u.Query().Get("sslrootcert")
}
if len(u.Query().Get("sslcert")) > 0 {
sslOpts.CertPath = u.Query().Get("sslcert")
}
if len(u.Query().Get("sslkey")) > 0 {
sslOpts.KeyPath = u.Query().Get("sslkey")
}
if u.Query().Get("sslmode") == "verify-full" {
cluster.SslOpts.EnableHostVerification = true
sslOpts.EnableHostVerification = true
}
cluster.SslOpts = sslOpts
}
}
if len(u.Query().Get("disable-host-lookup")) > 0 {
if flag, err := strconv.ParseBool(u.Query().Get("disable-host-lookup")); err != nil && flag {
cluster.DisableInitialHostLookup = true
} else if err != nil {
return nil, err
}
}
@@ -139,10 +175,19 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
return nil, err
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := u.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
return WithInstance(session, &Config{
KeyspaceName: strings.TrimPrefix(u.Path, "/"),
MigrationsTable: u.Query().Get("x-migrations-table"),
MultiStatementEnabled: u.Query().Get("x-multi-statement") == "true",
MultiStatementMaxSize: multiStatementMaxSize,
})
}
@@ -152,44 +197,44 @@ func (c *Cassandra) Close() error {
}
func (c *Cassandra) Lock() error {
if c.isLocked {
if !c.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
c.isLocked = true
return nil
}
func (c *Cassandra) Unlock() error {
c.isLocked = false
if !c.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (c *Cassandra) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if c.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, c.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if e := c.session.Query(tq).Exec(); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if c.config.MultiStatementEnabled {
// split query by semi-colon
queries := strings.Split(query, ";")
for _, q := range queries {
tq := strings.TrimSpace(q)
if tq == "" {
continue
}
if err := c.session.Query(tq).Exec(); err != nil {
// TODO: cast to Cassandra error and get line number
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
}
return nil
}
if err := c.session.Query(query).Exec(); err != nil {
if err := c.session.Query(string(migr)).Exec(); err != nil {
// TODO: cast to Cassandra error and get line number
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
@@ -197,12 +242,26 @@ func (c *Cassandra) Run(migration io.Reader) error {
}
func (c *Cassandra) SetVersion(version int, dirty bool) error {
query := `TRUNCATE "` + c.config.MigrationsTable + `"`
if err := c.session.Query(query).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
// DELETE instead of TRUNCATE because AWS Keyspaces does not support it
// see: https://docs.aws.amazon.com/keyspaces/latest/devguide/cassandra-apis.html
squery := `SELECT version FROM "` + c.config.MigrationsTable + `"`
dquery := `DELETE FROM "` + c.config.MigrationsTable + `" WHERE version = ?`
iter := c.session.Query(squery).Iter()
var previous int
for iter.Scan(&previous) {
if err := c.session.Query(dquery, previous).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(dquery)}
}
}
if version >= 0 {
query = `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
if err := iter.Close(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(squery)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query := `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
if err := c.session.Query(query, version, dirty).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -255,11 +314,7 @@ func (c *Cassandra) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -281,7 +336,7 @@ func parseConsistency(consistencyStr string) (consistency gocql.Consistency, err
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("Failed to parse consistency \"%s\": %v", consistencyStr, r)
err = fmt.Errorf("failed to parse consistency \"%s\": %v", consistencyStr, r)
}
}
}()
+18 -2
View File
@@ -25,9 +25,11 @@ var (
// Although Cassandra 2.x is supported by the Apache Foundation,
// the migrate db driver only supports Cassandra 3.x since it uses
// the system_schema keyspace.
// last ScyllaDB version tested is 5.1.11
specs = []dktesting.ContainerSpec{
{ImageName: "cassandra:3.0", Options: opts},
{ImageName: "cassandra:3.11", Options: opts},
{ImageName: "scylladb/scylla:5.1.11", Options: opts},
}
)
@@ -59,6 +61,20 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
@@ -79,7 +95,7 @@ func Test(t *testing.T) {
})
}
func TestMigrate(t *testing.T) {
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
@@ -101,6 +117,6 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("SELECT table_name from system_schema.tables"))
dt.TestMigrate(t, m)
})
}
-19
View File
@@ -1,19 +0,0 @@
# ClickHouse
`clickhouse://host:port?username=user&password=qwerty&database=clicks&x-multi-statement=true`
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
| `x-multi-statement` | false | Enable multiple statements to be ran in a single migration (See note below) |
## Notes
* The Clickhouse driver does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
-236
View File
@@ -1,236 +0,0 @@
package clickhouse
import (
"database/sql"
"fmt"
"io"
"io/ioutil"
"net/url"
"strings"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
)
var DefaultMigrationsTable = "schema_migrations"
var ErrNilConfig = fmt.Errorf("no config")
type Config struct {
DatabaseName string
MigrationsTable string
MultiStatementEnabled bool
}
func init() {
database.Register("clickhouse", &ClickHouse{})
}
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ch := &ClickHouse{
conn: conn,
config: config,
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
type ClickHouse struct {
conn *sql.DB
config *Config
}
func (ch *ClickHouse) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = "tcp"
conn, err := sql.Open("clickhouse", q.String())
if err != nil {
return nil, err
}
ch = &ClickHouse{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
DatabaseName: purl.Query().Get("database"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
},
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
func (ch *ClickHouse) init() error {
if len(ch.config.DatabaseName) == 0 {
if err := ch.conn.QueryRow("SELECT currentDatabase()").Scan(&ch.config.DatabaseName); err != nil {
return err
}
}
if len(ch.config.MigrationsTable) == 0 {
ch.config.MigrationsTable = DefaultMigrationsTable
}
return ch.ensureVersionTable()
}
func (ch *ClickHouse) Run(r io.Reader) error {
migration, err := ioutil.ReadAll(r)
if err != nil {
return err
}
if ch.config.MultiStatementEnabled {
// split query by semi-colon
queries := strings.Split(string(migration), ";")
for _, q := range queries {
tq := strings.TrimSpace(q)
if tq == "" {
continue
}
if _, err := ch.conn.Exec(q); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: []byte(q)}
}
}
return nil
}
if _, err := ch.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ch *ClickHouse) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ch.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ch.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ch *ClickHouse) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ch.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ch.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the ClickHouse type.
func (ch *ClickHouse) ensureVersionTable() (err error) {
if err = ch.Lock(); err != nil {
return err
}
defer func() {
if e := ch.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
var (
table string
query = "SHOW TABLES FROM " + ch.config.DatabaseName + " LIKE '" + ch.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ch.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
query = `
CREATE TABLE ` + ch.config.MigrationsTable + ` (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=TinyLog
`
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ch *ClickHouse) Drop() (err error) {
query := "SHOW TABLES FROM " + ch.config.DatabaseName
tables, err := ch.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
}
}()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + ch.config.DatabaseName + "." + table
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
return nil
}
func (ch *ClickHouse) Lock() error { return nil }
func (ch *ClickHouse) Unlock() error { return nil }
func (ch *ClickHouse) Close() error { return ch.conn.Close() }
+142
View File
@@ -0,0 +1,142 @@
# CockroachDB tutorial for beginners (insecure cluster)
## Create/configure database
First, let's start a local cluster - follow step 1. and 2. from [the docs](https://www.cockroachlabs.com/docs/stable/start-a-local-cluster.html#step-1-start-the-first-node).
Once you have it, create a database. Here I am going to create a database called `example`.
Our user here is `cockroach`. We are not going to use a password, since it's not supported for insecure cluster.
```
cockroach sql --insecure --host=localhost:26257
```
```
CREATE DATABASE example;
CREATE USER IF NOT EXISTS cockroach;
GRANT ALL ON DATABASE example TO cockroach;
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
```
export COCKROACHDB_URL='cockroachdb://cockroach:@localhost:26257/example?sslmode=disable'
```
`sslmode=disable` means that the connection with our database will not be encrypted. This is needed to connect to an insecure node.
**NOTE:** Do not use COCKROACH_URL as a variable name here, it's already in use for discrete parameters and you may run into connection problems. For more info check out [docs](https://www.cockroachlabs.com/docs/stable/connection-parameters.html#connect-using-discrete-parameters).
You can find further description of database URLs [here](README.md#database-urls).
## Create migrations
Let's create a table called `users`:
```
migrate create -ext sql -dir db/migrations -seq create_users_table
```
If there were no errors, we should have two files available under `db/migrations` folder:
- 000001_create_users_table.down.sql
- 000001_create_users_table.up.sql
Note the `sql` extension that we provided.
In the `.up.sql` file let's create the table:
```
CREATE TABLE IF NOT EXISTS example.users
(
user_id INT PRIMARY KEY,
username VARCHAR (50) UNIQUE NOT NULL,
password VARCHAR (50) NOT NULL,
email VARCHAR (300) UNIQUE NOT NULL
);
```
And in the `.down.sql` let's delete it:
```
DROP TABLE IF EXISTS example.users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](/GETTING_STARTED.md#create-migrations)
## Run migrations
```
migrate -database ${COCKROACHDB_URL} -path db/migrations up
```
Let's check if the table was created properly by running `cockroach sql --insecure --host=localhost:26257 -e "show columns from example.users;"`.
The output you are supposed to see:
```
column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden
+-------------+--------------+-------------+----------------+-----------------------+----------------------------------------------+-----------+
user_id | INT8 | false | NULL | | {primary,users_username_key,users_email_key} | false
username | VARCHAR(50) | false | NULL | | {users_username_key} | false
password | VARCHAR(50) | false | NULL | | {} | false
email | VARCHAR(300) | false | NULL | | {users_email_key} | false
(4 rows)
```
Now let's check if running reverse migration also works:
```
migrate -database ${COCKROACHDB_URL} -path db/migrations down
```
Make sure to check if your database changed as expected in this case as well.
## Database transactions
To show database transactions usage, let's create another set of migrations by running:
```
migrate create -ext sql -dir db/migrations -seq add_mood_to_users
```
Again, it should create for us two migrations files:
- 000002_add_mood_to_users.down.sql
- 000002_add_mood_to_users.up.sql
In Cockroach, when we want our queries to be done in a transaction, we need to wrap it with `BEGIN` and `COMMIT` commands, similar to PostgreSQL.
In our example, we are going to add a column to our database that can only accept enumerable values or NULL.
Migration up:
```
BEGIN;
ALTER TABLE example.users ADD COLUMN mood STRING;
ALTER TABLE example.users ADD CONSTRAINT check_mood CHECK (mood IN ('happy', 'sad', 'neutral'));
COMMIT;
```
Migration down:
```
ALTER TABLE example.users DROP COLUMN mood;
```
Now we can run our new migration and check the database:
```
migrate -database ${COCKROACHDB_URL} -path db/migrations up
cockroach sql --insecure --host=localhost:26257 -e "show columns from example.users;"
```
Expected output:
```
column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden
+-------------+--------------+-------------+----------------+-----------------------+----------------------------------------------+-----------+
user_id | INT8 | false | NULL | | {primary,users_username_key,users_email_key} | false
username | VARCHAR(50) | false | NULL | | {users_username_key} | false
password | VARCHAR(50) | false | NULL | | {} | false
email | VARCHAR(300) | false | NULL | | {users_email_key} | false
mood | STRING | true | NULL | | {} | false
(5 rows)
```
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
import (
"log"
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/cockroachdb"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
m, err := migrate.New(
"file://db/migrations",
"cockroachdb://cockroach:@localhost:26257/example?sslmode=disable")
if err != nil {
log.Fatal(err)
}
if err := m.Up(); err != nil {
log.Fatal(err)
}
}
```
You can find details [here](README.md#use-in-your-go-project)
+75 -80
View File
@@ -3,23 +3,18 @@ package cockroachdb
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
)
"sync/atomic"
import (
"github.com/cockroachdb/cockroach-go/crdb"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
import (
"github.com/cockroachdb/cockroach-go/v2/crdb"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/lib/pq"
)
func init() {
@@ -46,7 +41,7 @@ type Config struct {
type CockroachDb struct {
db *sql.DB
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -61,17 +56,19 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return nil, err
}
query := `SELECT current_database()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if config.DatabaseName == "" {
query := `SELECT current_database()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
config.DatabaseName = databaseName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
@@ -150,75 +147,71 @@ func (c *CockroachDb) Close() error {
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Lock() error {
err := crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = multierror.Append(err, errClose)
return database.CasRestoreOnErr(&c.isLocked, false, true, database.ErrLocked, func() (err error) {
return crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
return nil
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return nil
})
})
if err != nil {
return err
} else {
c.isLocked = true
return nil
}
}
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Unlock() error {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
c.isLocked = false
return nil
}
return database.CasRestoreOnErr(&c.isLocked, true, false, database.ErrNotLocked, func() (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
c.isLocked = false
return nil
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
return nil
}
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
})
}
func (c *CockroachDb) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -238,7 +231,10 @@ func (c *CockroachDb) SetVersion(version int, dirty bool) error {
return err
}
if version >= 0 {
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
if _, err := tx.Exec(`INSERT INTO "`+c.config.MigrationsTable+`" (version, dirty) VALUES ($1, $2)`, version, dirty); err != nil {
return err
}
@@ -280,7 +276,7 @@ func (c *CockroachDb) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -295,6 +291,9 @@ func (c *CockroachDb) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -319,11 +318,7 @@ func (c *CockroachDb) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+7 -7
View File
@@ -26,13 +26,13 @@ import (
const defaultPort = 26257
var (
opts = dktest.Options{Cmd: []string{"start", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Released versions: https://www.cockroachlabs.com/docs/releases/
opts = dktest.Options{Cmd: []string{"start-single-node", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.cockroachlabs.com/docs/releases/release-support-policy#supported-versions
specs = []dktesting.ContainerSpec{
{ImageName: "cockroachdb/cockroach:v1.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v1.1.9", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.1.3", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v24.3", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v24.1", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.2", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.1", Options: opts},
}
)
@@ -121,7 +121,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("SELECT 1"))
dt.TestMigrate(t, m)
})
}
+31
View File
@@ -0,0 +1,31 @@
# Datastore
`datastore://host:port?username=user&password=password&database=clicks&x-multi-statement=true`
Hanzo Datastore is a ClickHouse-wire-compatible database engine. This driver
connects through the native `github.com/hanzo-ds/go` driver, which registers the
`database/sql` driver name `datastore`.
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `x-migrations-table-engine`| Engine to use for the migrations table, defaults to TinyLog |
| `x-cluster-name` | Name of cluster for creating `schema_migrations` table cluster wide |
| `x-multi-statement` | Enable multiple statements to be ran in a single migration (See note below) |
| `x-multi-statement-max-size` | Maximum size of a single migration in bytes when `x-multi-statement` is set (defaults to 10 MB) |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
## Notes
* The Datastore driver does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
* Using the default TinyLog table engine for the `schema_migrations` table prevents backing up the table. If you need to back up the database, run the migrations with `x-migrations-table-engine=MergeTree`.
* Datastore cluster mode is not officially supported, but you can try enabling `schema_migrations` table replication by specifying a `x-cluster-name`:
* When `x-cluster-name` is specified, `x-migrations-table-engine` also should be specified.
* When `x-cluster-name` is specified, only the `schema_migrations` table is replicated across the cluster. You still need to write your migrations so that the application tables are replicated within the cluster.
* If you want to create a database inside a migration, note that the `schema_migrations` table will live in the `default` database, so you cannot use `USE <database_name>` inside a migration. In this case you may omit the database in the connection string (example [here](examples/migrations/003_create_database.up.sql)).
+337
View File
@@ -0,0 +1,337 @@
// Package datastore implements the golang-migrate database.Driver interface for
// Hanzo Datastore (a ClickHouse-wire-compatible database engine). It registers
// the "datastore://" URL scheme and connects through the native
// github.com/hanzo-ds/go driver, which registers the database/sql driver name
// "datastore".
//
// Usage:
//
// import _ "github.com/golang-migrate/migrate/v4/database/datastore"
//
// Then use a URL like:
//
// datastore://host:9000?username=default&database=default&x-multi-statement=true
package datastore
import (
"database/sql"
"errors"
"fmt"
"io"
"net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
var (
multiStmtDelimiter = []byte(";")
// DriverName is the database/sql driver registered by github.com/hanzo-ds/go.
DriverName = "datastore"
DefaultMigrationsTable = "schema_migrations"
DefaultMigrationsTableEngine = "TinyLog"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
ErrNilConfig = fmt.Errorf("no config")
)
type Config struct {
DatabaseName string
ClusterName string
MigrationsTable string
MigrationsTableEngine string
MultiStatementEnabled bool
MultiStatementMaxSize int
}
func init() {
database.Register("datastore", &Datastore{})
}
// WithInstance creates a Datastore driver from an existing *sql.DB opened
// against the "datastore" driver.
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ds := &Datastore{
conn: conn,
config: config,
}
if err := ds.init(); err != nil {
return nil, err
}
return ds, nil
}
// Datastore is a golang-migrate database.Driver backed by the native
// github.com/hanzo-ds/go ClickHouse-wire driver.
type Datastore struct {
conn *sql.DB
config *Config
isLocked atomic.Bool
}
func (ds *Datastore) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = DriverName
conn, err := sql.Open(DriverName, q.String())
if err != nil {
return nil, err
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
migrationsTableEngine := DefaultMigrationsTableEngine
if s := purl.Query().Get("x-migrations-table-engine"); len(s) > 0 {
migrationsTableEngine = s
}
ds = &Datastore{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
MigrationsTableEngine: migrationsTableEngine,
DatabaseName: purl.Query().Get("database"),
ClusterName: purl.Query().Get("x-cluster-name"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
MultiStatementMaxSize: multiStatementMaxSize,
},
}
if err := ds.init(); err != nil {
return nil, err
}
return ds, nil
}
func (ds *Datastore) init() error {
if len(ds.config.DatabaseName) == 0 {
if err := ds.conn.QueryRow("SELECT currentDatabase()").Scan(&ds.config.DatabaseName); err != nil {
return err
}
}
if len(ds.config.MigrationsTable) == 0 {
ds.config.MigrationsTable = DefaultMigrationsTable
}
if ds.config.MultiStatementMaxSize <= 0 {
ds.config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
if len(ds.config.MigrationsTableEngine) == 0 {
ds.config.MigrationsTableEngine = DefaultMigrationsTableEngine
}
return ds.ensureVersionTable()
}
func (ds *Datastore) Run(r io.Reader) error {
if ds.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(r, multiStmtDelimiter, ds.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if _, e := ds.conn.Exec(string(m)); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
return true
}); e != nil {
return e
}
return err
}
migration, err := io.ReadAll(r)
if err != nil {
return err
}
if _, err := ds.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ds *Datastore) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ds.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ds.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ds *Datastore) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ds.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ds.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Datastore type.
func (ds *Datastore) ensureVersionTable() (err error) {
if err = ds.Lock(); err != nil {
return err
}
defer func() {
if e := ds.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
var (
table string
query = "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName) + " LIKE '" + ds.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ds.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
if len(ds.config.ClusterName) > 0 {
query = fmt.Sprintf(`
CREATE TABLE %s ON CLUSTER %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.ClusterName, ds.config.MigrationsTableEngine)
} else {
query = fmt.Sprintf(`
CREATE TABLE %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.MigrationsTableEngine)
}
if strings.HasSuffix(ds.config.MigrationsTableEngine, "Tree") {
query = fmt.Sprintf(`%s ORDER BY sequence`, query)
}
if _, err := ds.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Drop() (err error) {
query := "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName)
tables, err := ds.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + quoteIdentifier(ds.config.DatabaseName) + "." + quoteIdentifier(table)
if _, err := ds.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Lock() error {
if !ds.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
func (ds *Datastore) Unlock() error {
if !ds.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (ds *Datastore) Close() error { return ds.conn.Close() }
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+226
View File
@@ -0,0 +1,226 @@
package datastore_test
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"fmt"
"log"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/datastore"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
const defaultPort = 9000
var (
tableEngines = []string{"TinyLog", "MergeTree"}
opts = dktest.Options{
Env: map[string]string{"CLICKHOUSE_USER": "user", "CLICKHOUSE_PASSWORD": "password", "CLICKHOUSE_DB": "db"},
PortRequired: true, ReadyFunc: isReady,
}
specs = []dktesting.ContainerSpec{
{ImageName: "clickhouse:24.8", Options: opts},
}
)
func datastoreConnectionString(host, port, engine string) string {
if engine != "" {
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&x-migrations-table-engine=%v&debug=false",
host, port, engine)
}
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&debug=false",
host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
return false
}
db, err := sql.Open("datastore", datastoreConnectionString(ip, port, ""))
if err != nil {
log.Println("open error", err)
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn:
return false
default:
fmt.Println(err)
}
return false
}
return true
}
func TestCases(t *testing.T) {
for _, engine := range tableEngines {
t.Run("Test_"+engine, func(t *testing.T) { testSimple(t, engine) })
t.Run("Migrate_"+engine, func(t *testing.T) { testMigrate(t, engine) })
t.Run("Version_"+engine, func(t *testing.T) { testVersion(t, engine) })
t.Run("Drop_"+engine, func(t *testing.T) { testDrop(t, engine) })
}
t.Run("WithInstanceDefaultConfigValues", func(t *testing.T) { testSimpleWithInstanceDefaultConfigValues(t) })
}
func testSimple(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testSimpleWithInstanceDefaultConfigValues(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, "")
conn, err := sql.Open("datastore", addr)
if err != nil {
t.Fatal(err)
}
d, err := datastore.WithInstance(conn, &datastore.Config{})
if err != nil {
_ = conn.Close()
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testMigrate(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "db", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func testVersion(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
expectedVersion := 1
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
err = d.SetVersion(expectedVersion, false)
if err != nil {
t.Fatal(err)
}
version, _, err := d.Version()
if err != nil {
t.Fatal(err)
}
if version != expectedVersion {
t.Fatal("Version mismatch")
}
})
}
func testDrop(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
err = d.Drop()
if err != nil {
t.Fatal(err)
}
})
}
@@ -1,3 +1,3 @@
CREATE TABLE test_1 (
Date Date
Date Date
) Engine=Memory;
@@ -1,3 +1,3 @@
CREATE TABLE test_2 (
Date Date
Date Date
) Engine=Memory;
@@ -0,0 +1,10 @@
DROP TABLE IF EXISTS driver_ratings;
DROP TABLE IF EXISTS user_ratings;
DROP TABLE IF EXISTS orders;
DROP TABLE IF EXISTS driver_ratings_queue;
DROP TABLE IF EXISTS user_ratings_queue;
DROP TABLE IF EXISTS orders_queue;
DROP VIEW IF EXISTS user_ratings_queue_mv;
DROP VIEW IF EXISTS driver_ratings_queue_mv;
DROP VIEW IF EXISTS orders_queue_mv;
DROP DATABASE IF EXISTS analytics;
@@ -0,0 +1,81 @@
CREATE DATABASE IF NOT EXISTS analytics;
CREATE TABLE IF NOT EXISTS analytics.driver_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.driver_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'driver-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.driver_ratings_queue_mv TO analytics.driver_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.driver_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.user_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY userID
ORDER BY (inserted_time);
CREATE TABLE analytics.user_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'user-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'JSON',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.user_ratings_queue_mv TO analytics.user_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.user_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.orders(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.orders_queue(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'orders',
kafka_group_name = 'order_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.orders_queue_mv TO orders AS
SELECT from_place, to_place, userID, driverID, orderID
FROM analytics.orders_queue;
+17 -16
View File
@@ -1,4 +1,4 @@
// Package database provides the Database interface.
// Package database provides the Driver interface.
// All database drivers must implement this interface, register themselves,
// optionally provide a `WithInstance` function and pass the tests
// in package database/testing.
@@ -13,7 +13,8 @@ import (
)
var (
ErrLocked = fmt.Errorf("can't acquire lock")
ErrLocked = fmt.Errorf("can't acquire lock")
ErrNotLocked = fmt.Errorf("can't unlock, as not currently locked")
)
const NilVersion int = -1
@@ -24,22 +25,22 @@ var drivers = make(map[string]Driver)
// Driver is the interface every database driver must implement.
//
// How to implement a database driver?
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a migrate/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
//
// Guidelines:
// * Don't try to correct user input. Don't assume things.
// - Don't try to correct user input. Don't assume things.
// When in doubt, return an error and explain the situation to the user.
// * All configuration input must come from the URL string in func Open()
// - All configuration input must come from the URL string in func Open()
// or the Config{} struct in WithInstance. Don't os.Getenv().
type Driver interface {
// Open returns a new driver instance configured with parameters
@@ -61,7 +62,7 @@ type Driver interface {
// all migrations have been run.
Unlock() error
// Run applies a migration to the database. migration is garantueed to be not nil.
// Run applies a migration to the database. migration is guaranteed to be not nil.
Run(migration io.Reader) error
// SetVersion saves version and dirty state.
-6
View File
@@ -10,9 +10,3 @@
| `role` | | Role name |
| `tzname` | | Time Zone name. (For Firebird 4.0+) |
| `wire_crypt` | | Enable wire data encryption or not. For Firebird 3.0+ (default is true) |
## Not officially supported
If you need a `migrate` CLI that supports Firebird, you'll need need to build it yourself. There are also no guarantees that the integration will work since we're not testing with Firebird.
For more info, see: https://github.com/golang-migrate/migrate/issues/49#issuecomment-487193672
+24 -20
View File
@@ -1,18 +1,19 @@
// +build go1.9
//go:build go1.9
package firebird
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/nakagami/firebirdsql"
"io"
"io/ioutil"
nurl "net/url"
)
func init() {
@@ -36,7 +37,7 @@ type Firebird struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -106,20 +107,21 @@ func (f *Firebird) Close() error {
}
func (f *Firebird) Lock() error {
if f.isLocked {
if !f.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
f.isLocked = true
return nil
}
func (f *Firebird) Unlock() error {
f.isLocked = false
if !f.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (f *Firebird) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -134,17 +136,20 @@ func (f *Firebird) Run(migration io.Reader) error {
}
func (f *Firebird) SetVersion(version int, dirty bool) error {
if version < 0 {
return nil
}
// Always re-write the schema version to prevent empty schema version
// for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
// TODO: parameterize this SQL statement
// https://firebirdsql.org/refdocs/langrefupd20-execblock.html
// VALUES (?, ?) doesn't work
query := fmt.Sprintf(`EXECUTE BLOCK AS BEGIN
DELETE FROM "%v";
INSERT INTO "%v" (version, dirty) VALUES (%v, %v);
END;`,
f.config.MigrationsTable, f.config.MigrationsTable, version, btoi(dirty))
if _, err := f.conn.ExecContext(context.Background(), query, version, btoi(dirty)); err != nil {
if _, err := f.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -175,7 +180,7 @@ func (f *Firebird) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -190,6 +195,9 @@ func (f *Firebird) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// delete one by one ...
for _, t := range tableNames {
@@ -215,11 +223,7 @@ func (f *Firebird) ensureVersionTable() (err error) {
defer func() {
if e := f.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+4 -3
View File
@@ -38,8 +38,9 @@ var (
},
}
specs = []dktesting.ContainerSpec{
{ImageName: "jacobalberty/firebird:2.5-ss", Options: opts},
{ImageName: "jacobalberty/firebird:3.0", Options: opts},
{ImageName: "jacobalberty/firebird:v3.0", Options: opts},
{ImageName: "jacobalberty/firebird:v4.0", Options: opts},
{ImageName: "jacobalberty/firebird:v5.0", Options: opts},
}
)
@@ -121,7 +122,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("SELECT Count(*) FROM rdb$relations"))
dt.TestMigrate(t, m)
})
}
+5 -1
View File
@@ -7,12 +7,16 @@
# Usage
`mongodb://user:password@host:port/dbname?query`
`mongodb://user:password@host:port/dbname?query` (`mongodb+srv://` also works, but behaves a bit differently. See [docs](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format) for more information)
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-collection` | `MigrationsCollection` | Name of the migrations collection |
| `x-transaction-mode` | `TransactionMode` | If set to `true` wrap commands in [transaction](https://docs.mongodb.com/manual/core/transactions). Available only for replica set. Driver is using [strconv.ParseBool](https://golang.org/pkg/strconv/#ParseBool) for parsing|
| `x-advisory-locking` | `true` | Feature flag for advisory locking, if set to false, disable advisory locking |
| `x-advisory-lock-collection` | `migrate_advisory_lock` | The name of the collection to use for advisory locking.|
| `x-advisory-lock-timeout` | `15` | The max time in seconds that migrate will wait to acquire a lock before failing. |
| `x-advisory-lock-timeout-interval` | `10` | The max time in seconds between attempts to acquire the advisory lock, the lock is attempted to be acquired using an exponential backoff algorithm. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as. Can be omitted |
| `password` | | The user's password. Can be omitted |
@@ -0,0 +1,16 @@
[
{
"update": "users",
"updates": [
{
"q": {},
"u": {
"$unset": {
"status": ""
}
},
"multi": true
}
]
}
]
@@ -0,0 +1,16 @@
[
{
"update": "users",
"updates": [
{
"q": {},
"u": {
"$set": {
"status": "active"
}
},
"multi": true
}
]
}
]
@@ -0,0 +1,14 @@
[
{
"update": "users",
"updates": [
{
"q": {},
"u": {
"fullname": ""
},
"multi": true
}
]
}
]
@@ -0,0 +1,23 @@
[
{
"aggregate": "users",
"pipeline": [
{
"$project": {
"_id": 1,
"firstname": 1,
"lastname": 1,
"username": 1,
"password": 1,
"email": 1,
"active": 1,
"fullname": { "$concat": ["$firstname", " ", "$lastname"] }
}
},
{
"$out": "users"
}
],
"cursor": {}
}
]
+226 -17
View File
@@ -2,12 +2,16 @@ package mongodb
import (
"context"
"errors"
"fmt"
"io"
"io/ioutil"
"net/url"
"os"
"strconv"
"sync/atomic"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/golang-migrate/migrate/v4/database"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
@@ -16,34 +20,61 @@ import (
)
func init() {
database.Register("mongodb", &Mongo{})
db := Mongo{}
database.Register("mongodb", &db)
database.Register("mongodb+srv", &db)
}
var DefaultMigrationsCollection = "schema_migrations"
const DefaultLockingCollection = "migrate_advisory_lock" // the collection to use for advisory locking by default.
const lockKeyUniqueValue = 0 // the unique value to lock on. If multiple clients try to insert the same key, it will fail (locked).
const DefaultLockTimeout = 15 // the default maximum time to wait for a lock to be released.
const DefaultLockTimeoutInterval = 10 // the default maximum intervals time for the locking timout.
const DefaultAdvisoryLockingFlag = true // the default value for the advisory locking feature flag. Default is true.
const LockIndexName = "lock_unique_key" // the name of the index which adds unique constraint to the locking_key field.
const contextWaitTimeout = 5 * time.Second // how long to wait for the request to mongo to block/wait for.
var (
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
ErrLockTimeoutConfigConflict = fmt.Errorf("both x-advisory-lock-timeout-interval and x-advisory-lock-timout-interval were specified")
)
type Mongo struct {
client *mongo.Client
db *mongo.Database
config *Config
client *mongo.Client
db *mongo.Database
config *Config
isLocked atomic.Bool
}
type Locking struct {
CollectionName string
Timeout int
Enabled bool
Interval int
}
type Config struct {
DatabaseName string
MigrationsCollection string
TransactionMode bool
Locking Locking
}
type versionInfo struct {
Version int `bson:"version"`
Dirty bool `bson:"dirty"`
}
type lockObj struct {
Key int `bson:"locking_key"`
Pid int `bson:"pid"`
Hostname string `bson:"hostname"`
CreatedAt time.Time `bson:"created_at"`
}
type findFilter struct {
Key int `bson:"locking_key"`
}
func WithInstance(instance *mongo.Client, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
@@ -54,17 +85,36 @@ func WithInstance(instance *mongo.Client, config *Config) (database.Driver, erro
if len(config.MigrationsCollection) == 0 {
config.MigrationsCollection = DefaultMigrationsCollection
}
if len(config.Locking.CollectionName) == 0 {
config.Locking.CollectionName = DefaultLockingCollection
}
if config.Locking.Timeout <= 0 {
config.Locking.Timeout = DefaultLockTimeout
}
if config.Locking.Interval <= 0 {
config.Locking.Interval = DefaultLockTimeoutInterval
}
mc := &Mongo{
client: instance,
db: instance.Database(config.DatabaseName),
config: config,
}
if mc.config.Locking.Enabled {
if err := mc.ensureLockTable(); err != nil {
return nil, err
}
}
if err := mc.ensureVersionTable(); err != nil {
return nil, err
}
return mc, nil
}
func (m *Mongo) Open(dsn string) (database.Driver, error) {
//connsting is experimental package, but it used for parse connection string in mongo.Connect function
// connstring is experimental package, but it used for parse connection string in mongo.Connect function
uri, err := connstring.Parse(dsn)
if err != nil {
return nil, err
@@ -72,16 +122,46 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
if len(uri.Database) == 0 {
return nil, ErrNoDatabaseName
}
unknown := url.Values(uri.UnknownOptions)
migrationsCollection := unknown.Get("x-migrations-collection")
transactionMode, _ := strconv.ParseBool(unknown.Get("x-transaction-mode"))
lockCollection := unknown.Get("x-advisory-lock-collection")
transactionMode, err := parseBoolean(unknown.Get("x-transaction-mode"), false)
if err != nil {
return nil, err
}
advisoryLockingFlag, err := parseBoolean(unknown.Get("x-advisory-locking"), DefaultAdvisoryLockingFlag)
if err != nil {
return nil, err
}
lockingTimout, err := parseInt(unknown.Get("x-advisory-lock-timeout"), DefaultLockTimeout)
if err != nil {
return nil, err
}
lockTimeoutIntervalValue := unknown.Get("x-advisory-lock-timeout-interval")
// The initial release had a typo for this argument but for backwards compatibility sake, we will keep supporting it
// and we will error out if both values are set.
lockTimeoutIntervalValueFromTypo := unknown.Get("x-advisory-lock-timout-interval")
lockTimeout := lockTimeoutIntervalValue
if lockTimeoutIntervalValue != "" && lockTimeoutIntervalValueFromTypo != "" {
return nil, ErrLockTimeoutConfigConflict
} else if lockTimeoutIntervalValueFromTypo != "" {
lockTimeout = lockTimeoutIntervalValueFromTypo
}
maxLockCheckInterval, err := parseInt(lockTimeout, DefaultLockTimeoutInterval)
if err != nil {
return nil, err
}
client, err := mongo.Connect(context.TODO(), options.Client().ApplyURI(dsn))
if err != nil {
return nil, err
}
if err = client.Ping(context.TODO(), nil); err != nil {
return nil, err
}
@@ -89,6 +169,12 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
DatabaseName: uri.Database,
MigrationsCollection: migrationsCollection,
TransactionMode: transactionMode,
Locking: Locking{
CollectionName: lockCollection,
Timeout: lockingTimout,
Enabled: advisoryLockingFlag,
Interval: maxLockCheckInterval,
},
})
if err != nil {
return nil, err
@@ -96,6 +182,39 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
return mc, nil
}
// Parse the url param, convert it to boolean
// returns error if param invalid. returns defaultValue if param not present
func parseBoolean(urlParam string, defaultValue bool) (bool, error) {
// if parameter passed, parse it (otherwise return default value)
if urlParam != "" {
result, err := strconv.ParseBool(urlParam)
if err != nil {
return false, err
}
return result, nil
}
// if no url Param passed, return default value
return defaultValue, nil
}
// Parse the url param, convert it to int
// returns error if param invalid. returns defaultValue if param not present
func parseInt(urlParam string, defaultValue int) (int, error) {
// if parameter passed, parse it (otherwise return default value)
if urlParam != "" {
result, err := strconv.Atoi(urlParam)
if err != nil {
return -1, err
}
return result, nil
}
// if no url Param passed, return default value
return defaultValue, nil
}
func (m *Mongo) SetVersion(version int, dirty bool) error {
migrationsCollection := m.db.Collection(m.config.MigrationsCollection)
if err := migrationsCollection.Drop(context.TODO()); err != nil {
@@ -122,7 +241,7 @@ func (m *Mongo) Version() (version int, dirty bool, err error) {
}
func (m *Mongo) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -149,8 +268,8 @@ func (m *Mongo) executeCommandsWithTransaction(ctx context.Context, cmds []bson.
return &database.Error{OrigErr: err, Err: "failed to start transaction"}
}
if err := m.executeCommands(sessionContext, cmds); err != nil {
//When command execution is failed, it's aborting transaction
//If you tried to call abortTransaction, it`s return error that transaction already aborted
// When command execution is failed, it's aborting transaction
// If you tried to call abortTransaction, it`s return error that transaction already aborted
return err
}
if err := sessionContext.CommitTransaction(sessionContext); err != nil {
@@ -182,10 +301,100 @@ func (m *Mongo) Drop() error {
return m.db.Drop(context.TODO())
}
func (m *Mongo) Lock() error {
func (m *Mongo) ensureLockTable() error {
indexes := m.db.Collection(m.config.Locking.CollectionName).Indexes()
indexOptions := options.Index().SetUnique(true).SetName(LockIndexName)
_, err := indexes.CreateOne(context.TODO(), mongo.IndexModel{
Options: indexOptions,
Keys: findFilter{Key: -1},
})
if err != nil {
return err
}
return nil
}
func (m *Mongo) Unlock() error {
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the MongoDb type.
func (m *Mongo) ensureVersionTable() (err error) {
if err = m.Lock(); err != nil {
return err
}
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
if err != nil {
return err
}
if _, _, err = m.Version(); err != nil {
return err
}
return nil
}
// Utilizes advisory locking on the config.LockingCollection collection
// This uses a unique index on the `locking_key` field.
func (m *Mongo) Lock() error {
return database.CasRestoreOnErr(&m.isLocked, false, true, database.ErrLocked, func() error {
if !m.config.Locking.Enabled {
return nil
}
pid := os.Getpid()
hostname, err := os.Hostname()
if err != nil {
hostname = fmt.Sprintf("Could not determine hostname. Error: %s", err.Error())
}
newLockObj := lockObj{
Key: lockKeyUniqueValue,
Pid: pid,
Hostname: hostname,
CreatedAt: time.Now(),
}
operation := func() error {
timeout, cancelFunc := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).InsertOne(timeout, newLockObj)
defer cancelFunc()
return err
}
exponentialBackOff := backoff.NewExponentialBackOff()
duration := time.Duration(m.config.Locking.Timeout) * time.Second
exponentialBackOff.MaxElapsedTime = duration
exponentialBackOff.MaxInterval = time.Duration(m.config.Locking.Interval) * time.Second
err = backoff.Retry(operation, exponentialBackOff)
if err != nil {
return database.ErrLocked
}
return nil
})
}
func (m *Mongo) Unlock() error {
return database.CasRestoreOnErr(&m.isLocked, true, false, database.ErrNotLocked, func() error {
if !m.config.Locking.Enabled {
return nil
}
filter := findFilter{
Key: lockKeyUniqueValue,
}
ctx, cancel := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).DeleteMany(ctx, filter)
defer cancel()
if err != nil {
return err
}
return nil
})
}
+89 -8
View File
@@ -32,10 +32,10 @@ var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.mongodb.com/support-policy
specs = []dktesting.ContainerSpec{
{ImageName: "mongo:3.4", Options: opts},
{ImageName: "mongo:3.6", Options: opts},
{ImageName: "mongo:4.0", Options: opts},
{ImageName: "mongo:4.2", Options: opts},
{ImageName: "mongo:5.0", Options: opts},
{ImageName: "mongo:6.0", Options: opts},
{ImageName: "mongo:7.0", Options: opts},
{ImageName: "mongo:8.0", Options: opts},
}
)
@@ -74,6 +74,22 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testWithAuth", testWithAuth)
t.Run("testLockWorks", testLockWorks)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -92,14 +108,14 @@ func Test(t *testing.T) {
}
}()
dt.TestNilVersion(t, d)
//TestLockAndUnlock(t, d) driver doesn't support lock on database level
dt.TestLockAndUnlock(t, d)
dt.TestRun(t, d, bytes.NewReader([]byte(`[{"insert":"hello","documents":[{"wild":"world"}]}]`)))
dt.TestSetVersion(t, d)
dt.TestDrop(t, d)
})
}
func TestMigrate(t *testing.T) {
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -121,11 +137,11 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte(`[{"insert":"hello","documents":[{"wild":"world"}]}]`))
dt.TestMigrate(t, m)
})
}
func TestWithAuth(t *testing.T) {
func testWithAuth(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -180,11 +196,76 @@ func TestWithAuth(t *testing.T) {
})
}
func testLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := mongoConnectionString(ip, port)
p := &Mongo{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.TestRun(t, d, bytes.NewReader([]byte(`[{"insert":"hello","documents":[{"wild":"world"}]}]`)))
mc := d.(*Mongo)
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Unlock()
if err != nil {
t.Fatal(err)
}
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Unlock()
if err != nil {
t.Fatal(err)
}
// enable locking,
//try to hit a lock conflict
mc.config.Locking.Enabled = true
mc.config.Locking.Timeout = 1
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Lock()
if err == nil {
t.Fatal("should have failed, mongo should be locked already")
}
})
}
func TestTransaction(t *testing.T) {
transactionSpecs := []dktesting.ContainerSpec{
{ImageName: "mongo:4", Options: dktest.Options{PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"mongod", "--bind_ip_all", "--replSet", "rs0"}}},
}
t.Cleanup(func() {
for _, spec := range transactionSpecs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
dktesting.ParallelTest(t, transactionSpecs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
+46
View File
@@ -0,0 +1,46 @@
// Package multistmt provides methods for parsing multi-statement database migrations
package multistmt
import (
"bufio"
"bytes"
"io"
)
// StartBufSize is the default starting size of the buffer used to scan and parse multi-statement migrations
var StartBufSize = 4096
// Handler handles a single migration parsed from a multi-statement migration.
// It's given the single migration to handle and returns whether or not further statements
// from the multi-statement migration should be parsed and handled.
type Handler func(migration []byte) bool
func splitWithDelimiter(delimiter []byte) func(d []byte, atEOF bool) (int, []byte, error) {
return func(d []byte, atEOF bool) (int, []byte, error) {
// SplitFunc inspired by bufio.ScanLines() implementation
if atEOF {
if len(d) == 0 {
return 0, nil, nil
}
return len(d), d, nil
}
if i := bytes.Index(d, delimiter); i >= 0 {
return i + len(delimiter), d[:i+len(delimiter)], nil
}
return 0, nil, nil
}
}
// Parse parses the given multi-statement migration
func Parse(reader io.Reader, delimiter []byte, maxMigrationSize int, h Handler) error {
scanner := bufio.NewScanner(reader)
scanner.Buffer(make([]byte, 0, StartBufSize), maxMigrationSize)
scanner.Split(splitWithDelimiter(delimiter))
for scanner.Scan() {
cont := h(scanner.Bytes())
if !cont {
break
}
}
return scanner.Err()
}
+57
View File
@@ -0,0 +1,57 @@
package multistmt_test
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4/database/multistmt"
)
const maxMigrationSize = 1024
func TestParse(t *testing.T) {
testCases := []struct {
name string
multiStmt string
delimiter string
expected []string
expectedErr error
}{
{name: "single statement, no delimiter", multiStmt: "single statement, no delimiter", delimiter: ";",
expected: []string{"single statement, no delimiter"}, expectedErr: nil},
{name: "single statement, one delimiter", multiStmt: "single statement, one delimiter;", delimiter: ";",
expected: []string{"single statement, one delimiter;"}, expectedErr: nil},
{name: "two statements, no trailing delimiter", multiStmt: "statement one; statement two", delimiter: ";",
expected: []string{"statement one;", " statement two"}, expectedErr: nil},
{name: "two statements, with trailing delimiter", multiStmt: "statement one; statement two;", delimiter: ";",
expected: []string{"statement one;", " statement two;"}, expectedErr: nil},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
stmts := make([]string, 0, len(tc.expected))
err := multistmt.Parse(strings.NewReader(tc.multiStmt), []byte(tc.delimiter), maxMigrationSize, func(b []byte) bool {
stmts = append(stmts, string(b))
return true
})
assert.Equal(t, tc.expectedErr, err)
assert.Equal(t, tc.expected, stmts)
})
}
}
func TestParseDiscontinue(t *testing.T) {
multiStmt := "statement one; statement two"
delimiter := ";"
expected := []string{"statement one;"}
stmts := make([]string, 0, len(expected))
err := multistmt.Parse(strings.NewReader(multiStmt), []byte(delimiter), maxMigrationSize, func(b []byte) bool {
stmts = append(stmts, string(b))
return false
})
assert.Nil(t, err)
assert.Equal(t, expected, stmts)
}
+6 -4
View File
@@ -5,6 +5,8 @@
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-no-lock` | `NoLock` | Set to `true` to skip `GET_LOCK`/`RELEASE_LOCK` statements. Useful for [multi-master MySQL flavors](https://www.percona.com/doc/percona-xtradb-cluster/LATEST/features/pxc-strict-mode.html#explicit-table-locking). Only run migrations from one host when this is enabled. |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds, functionally similar to [Server-side SELECT statement timeouts](https://dev.mysql.com/blog-archive/server-side-select-statement-timeouts/) but enforced by the client. Available for all versions of MySQL, not just >=5.7. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
@@ -12,7 +14,7 @@
| `port` | | The port to bind to. |
| `tls` | | TLS / SSL encrypted connection parameter; see [go-sql-driver](https://github.com/go-sql-driver/mysql#tls). Use any name (e.g. `migrate`) if you want to use a custom TLS config (`x-tls-` queries). |
| `x-tls-ca` | | The location of the CA (certificate authority) file. |
| `x-tls-cert` | | The location of the client certicicate file. Must be used with `x-tls-key`. |
| `x-tls-cert` | | The location of the client certificate file. Must be used with `x-tls-key`. |
| `x-tls-key` | | The location of the private key file. Must be used with `x-tls-cert`. |
| `x-tls-insecure-skip-verify` | | Whether or not to use SSL (true\|false) |
@@ -27,9 +29,9 @@ import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate"
"github.com/golang-migrate/migrate/database/mysql"
_ "github.com/golang-migrate/migrate/source/file"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/mysql"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
+208 -138
View File
@@ -1,4 +1,4 @@
// +build go1.9
//go:build go1.9
package mysql
@@ -7,23 +7,22 @@ import (
"crypto/tls"
"crypto/x509"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"os"
"strconv"
"strings"
)
"sync/atomic"
"time"
import (
"github.com/go-sql-driver/mysql"
"github.com/hashicorp/go-multierror"
)
import (
"github.com/golang-migrate/migrate/v4/database"
)
var _ database.Driver = (*Mysql)(nil) // explicit compile time type check
func init() {
database.Register("mysql", &Mysql{})
}
@@ -35,12 +34,14 @@ var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrAppendPEM = fmt.Errorf("failed to append PEM")
ErrTLSCertKeyConfig = fmt.Errorf("To use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
ErrTLSCertKeyConfig = fmt.Errorf("to use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
)
type Config struct {
MigrationsTable string
DatabaseName string
MigrationsTable string
DatabaseName string
NoLock bool
StatementTimeout time.Duration
}
type Mysql struct {
@@ -48,48 +49,45 @@ type Mysql struct {
// just do everything over a single conn anyway.
conn *sql.Conn
db *sql.DB
isLocked bool
isLocked atomic.Bool
config *Config
}
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
// connection instance must have `multiStatements` set to true
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Mysql, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
query := `SELECT DATABASE()`
var databaseName sql.NullString
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName.String) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName.String
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
if err := conn.PingContext(ctx); err != nil {
return nil, err
}
mx := &Mysql{
conn: conn,
db: instance,
db: nil,
config: config,
}
if config.DatabaseName == "" {
query := `SELECT DATABASE()`
var databaseName sql.NullString
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName.String) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName.String
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if err := mx.ensureVersionTable(); err != nil {
return nil, err
}
@@ -97,6 +95,29 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return mx, nil
}
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
mx, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
mx.db = instance
return mx, nil
}
// extractCustomQueryParams extracts the custom query params (ones that start with "x-") from
// mysql.Config.Params (connection parameters) as to not interfere with connecting to MySQL
func extractCustomQueryParams(c *mysql.Config) (map[string]string, error) {
@@ -115,6 +136,68 @@ func extractCustomQueryParams(c *mysql.Config) (map[string]string, error) {
}
func urlToMySQLConfig(url string) (*mysql.Config, error) {
// Need to parse out custom TLS parameters and call
// mysql.RegisterTLSConfig() before mysql.ParseDSN() is called
// which consumes the registered tls.Config
// Fixes: https://github.com/golang-migrate/migrate/issues/411
//
// Can't use url.Parse() since it fails to parse MySQL DSNs
// mysql.ParseDSN() also searches for "?" to find query parameters:
// https://github.com/go-sql-driver/mysql/blob/46351a8/dsn.go#L344
if idx := strings.LastIndex(url, "?"); idx > 0 {
rawParams := url[idx+1:]
parsedParams, err := nurl.ParseQuery(rawParams)
if err != nil {
return nil, err
}
ctls := parsedParams.Get("tls")
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := os.ReadFile(parsedParams.Get("x-tls-ca"))
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := parsedParams.Get("x-tls-cert"), parsedParams.Get("x-tls-key"); ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
insecureSkipVerifyStr := parsedParams.Get("x-tls-insecure-skip-verify")
if len(insecureSkipVerifyStr) > 0 {
x, err := strconv.ParseBool(insecureSkipVerifyStr)
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
err = mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
if err != nil {
return nil, err
}
}
}
}
config, err := mysql.ParseDSN(strings.TrimPrefix(url, "mysql://"))
if err != nil {
return nil, err
@@ -137,52 +220,6 @@ func urlToMySQLConfig(url string) (*mysql.Config, error) {
}
config.Passwd = password
// use custom TLS?
ctls := config.TLSConfig
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := ioutil.ReadFile(config.Params["x-tls-ca"])
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := config.Params["x-tls-cert"], config.Params["x-tls-key"]; ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
if len(config.Params["x-tls-insecure-skip-verify"]) > 0 {
x, err := strconv.ParseBool(config.Params["x-tls-insecure-skip-verify"])
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
err = mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
if err != nil {
return nil, err
}
}
}
return config, nil
}
@@ -197,14 +234,33 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
return nil, err
}
noLockParam, noLock := customParams["x-no-lock"], false
if noLockParam != "" {
noLock, err = strconv.ParseBool(noLockParam)
if err != nil {
return nil, fmt.Errorf("could not parse x-no-lock as bool: %w", err)
}
}
statementTimeoutParam := customParams["x-statement-timeout"]
statementTimeout := 0
if statementTimeoutParam != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutParam)
if err != nil {
return nil, fmt.Errorf("could not parse x-statement-timeout as float: %w", err)
}
}
db, err := sql.Open("mysql", config.FormatDSN())
if err != nil {
return nil, err
}
mx, err := WithInstance(db, &Config{
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
NoLock: noLock,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
})
if err != nil {
return nil, err
@@ -215,7 +271,11 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
func (m *Mysql) Close() error {
connErr := m.conn.Close()
dbErr := m.db.Close()
var dbErr error
if m.db != nil {
dbErr = m.db.Close()
}
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -223,62 +283,70 @@ func (m *Mysql) Close() error {
}
func (m *Mysql) Lock() error {
if m.isLocked {
return database.ErrLocked
}
return database.CasRestoreOnErr(&m.isLocked, false, true, database.ErrLocked, func() error {
if m.config.NoLock {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
if !success {
return database.ErrLocked
}
if success {
m.isLocked = true
return nil
}
return database.ErrLocked
})
}
func (m *Mysql) Unlock() error {
if !m.isLocked {
return database.CasRestoreOnErr(&m.isLocked, true, false, database.ErrNotLocked, func() error {
if m.config.NoLock {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
m.isLocked = false
return nil
})
}
func (m *Mysql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
ctx := context.Background()
if m.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, m.config.StatementTimeout)
defer cancel()
}
query := string(migr[:])
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
if _, err := m.conn.ExecContext(ctx, query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
@@ -286,24 +354,27 @@ func (m *Mysql) Run(migration io.Reader) error {
}
func (m *Mysql) SetVersion(version int, dirty bool) error {
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{})
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "TRUNCATE `" + m.config.MigrationsTable + "`"
query := "DELETE FROM `" + m.config.MigrationsTable + "` LIMIT 1"
if _, err := tx.ExecContext(context.Background(), query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query := "INSERT INTO `" + m.config.MigrationsTable + "` (version, dirty) VALUES (?, ?)"
if _, err := tx.ExecContext(context.Background(), query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -345,7 +416,7 @@ func (m *Mysql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -360,6 +431,9 @@ func (m *Mysql) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// disable checking foreign key constraints until finished
@@ -395,17 +469,13 @@ func (m *Mysql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
// check if migration table exists
var result string
query := `SHOW TABLES LIKE "` + m.config.MigrationsTable + `"`
query := `SHOW TABLES LIKE '` + m.config.MigrationsTable + `'`
if err := m.conn.QueryRowContext(context.Background(), query).Scan(&result); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
+154 -14
View File
@@ -2,24 +2,28 @@ package mysql
import (
"context"
"crypto/ed25519"
"crypto/x509"
"database/sql"
sqldriver "database/sql/driver"
"encoding/pem"
"errors"
"fmt"
"log"
"math/big"
"math/rand"
"net/url"
"os"
"strconv"
"testing"
)
import (
"github.com/dhui/dktest"
"github.com/go-sql-driver/mysql"
"github.com/stretchr/testify/assert"
)
import (
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/stretchr/testify/assert"
)
const defaultPort = 3306
@@ -29,12 +33,21 @@ var (
Env: map[string]string{"MYSQL_ROOT_PASSWORD": "root", "MYSQL_DATABASE": "public"},
PortRequired: true, ReadyFunc: isReady,
}
optsAnsiQuotes = dktest.Options{
Env: map[string]string{"MYSQL_ROOT_PASSWORD": "root", "MYSQL_DATABASE": "public"},
PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"--sql-mode=ANSI_QUOTES"},
}
// Supported versions: https://www.mysql.com/support/supportedplatforms/database.html
specs = []dktesting.ContainerSpec{
{ImageName: "mysql:5.5", Options: opts},
{ImageName: "mysql:5.6", Options: opts},
{ImageName: "mysql:5.7", Options: opts},
{ImageName: "mysql:8", Options: opts},
{ImageName: "mysql:8.0", Options: opts},
{ImageName: "mysql:8.4", Options: opts},
{ImageName: "mysql:9.0", Options: opts},
}
specsAnsiQuotes = []dktesting.ContainerSpec{
{ImageName: "mysql:8.0", Options: optsAnsiQuotes},
{ImageName: "mysql:8.4", Options: optsAnsiQuotes},
{ImageName: "mysql:9.0", Options: optsAnsiQuotes},
}
)
@@ -67,7 +80,7 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -100,7 +113,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -124,7 +137,45 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("SELECT 1"))
dt.TestMigrate(t, m)
// check ensureVersionTable
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
// check again
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
})
}
func TestMigrateAnsiQuotes(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specsAnsiQuotes, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "public", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
// check ensureVersionTable
if err := d.(*Mysql).ensureVersionTable(); err != nil {
@@ -175,6 +226,57 @@ func TestLockWorks(t *testing.T) {
})
}
func TestNoLockParamValidation(t *testing.T) {
ip := "127.0.0.1"
port := 3306
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
_, err := p.Open(addr + "?x-no-lock=not-a-bool")
if !errors.Is(err, strconv.ErrSyntax) {
t.Fatal("Expected syntax error when passing a non-bool as x-no-lock parameter")
}
}
func TestNoLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
lock := d.(*Mysql)
p = &Mysql{}
d, err = p.Open(addr + "?x-no-lock=true")
if err != nil {
t.Fatal(err)
}
noLock := d.(*Mysql)
// Should be possible to take real lock and no-lock at the same time
if err = lock.Lock(); err != nil {
t.Fatal(err)
}
if err = noLock.Lock(); err != nil {
t.Fatal(err)
}
if err = lock.Unlock(); err != nil {
t.Fatal(err)
}
if err = noLock.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestExtractCustomQueryParams(t *testing.T) {
testcases := []struct {
name string
@@ -231,7 +333,42 @@ func TestExtractCustomQueryParams(t *testing.T) {
}
}
func createTmpCert(t *testing.T) string {
tmpCertFile, err := os.CreateTemp("", "migrate_test_cert")
if err != nil {
t.Fatal("Failed to create temp cert file:", err)
}
t.Cleanup(func() {
if err := os.Remove(tmpCertFile.Name()); err != nil {
t.Log("Failed to cleanup temp cert file:", err)
}
})
r := rand.New(rand.NewSource(0))
pub, priv, err := ed25519.GenerateKey(r)
if err != nil {
t.Fatal("Failed to generate ed25519 key for temp cert file:", err)
}
tmpl := x509.Certificate{
SerialNumber: big.NewInt(0),
}
derBytes, err := x509.CreateCertificate(r, &tmpl, &tmpl, pub, priv)
if err != nil {
t.Fatal("Failed to generate temp cert file:", err)
}
if err := pem.Encode(tmpCertFile, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}); err != nil {
t.Fatal("Failed to encode ")
}
if err := tmpCertFile.Close(); err != nil {
t.Fatal("Failed to close temp cert file:", err)
}
return tmpCertFile.Name()
}
func TestURLToMySQLConfig(t *testing.T) {
tmpCertFilename := createTmpCert(t)
tmpCertFilenameEscaped := url.PathEscape(tmpCertFilename)
testcases := []struct {
name string
urlStr string
@@ -252,7 +389,7 @@ func TestURLToMySQLConfig(t *testing.T) {
// Not supported yet: https://github.com/go-sql-driver/mysql/issues/591
// {name: "user/password - user with encoded :",
// urlStr: "mysql://username%3A:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
// expectedDSN: "username::pasword@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
// expectedDSN: "username::password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - user with encoded @",
urlStr: "mysql://username%40:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
@@ -262,6 +399,9 @@ func TestURLToMySQLConfig(t *testing.T) {
{name: "user/password - password with encoded @",
urlStr: "mysql://username:password%40@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:password@@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "custom tls",
urlStr: "mysql://username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true&tls=custom&x-tls-ca=" + tmpCertFilenameEscaped,
expectedDSN: "username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true&tls=custom&x-tls-ca=" + tmpCertFilenameEscaped},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
+20
View File
@@ -0,0 +1,20 @@
# neo4j
The Neo4j driver (bolt) does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param.
This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
The queries **should** run in a single transaction, so partial migrations should not be a concern, but this is untested.
`neo4j://user:password@host:port/`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-multi-statement` | `MultiStatement` | Enable multiple statements to be ran in a single migration (See note above) |
| `user` | Contained within `AuthConfig` | The user to sign in as |
| `password` | Contained within `AuthConfig` | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 7687) |
| | `MigrationsLabel` | Name of the migrations node label |
## Supported versions
Only Neo4j v3.5+ is [supported](https://github.com/neo4j/neo4j-go-driver/issues/64#issuecomment-625133600)
+97
View File
@@ -0,0 +1,97 @@
## Create migrations
Let's create nodes called `Users`:
```
migrate create -ext cypher -dir db/migrations -seq create_user_nodes
```
If there were no errors, we should have two files available under `db/migrations` folder:
- 000001_create_user_nodes.down.cypher
- 000001_create_user_nodes.up.cypher
Note the `cypher` extension that we provided.
In the `.up.cypher` file let's create the table:
```
CREATE (u1:User {name: "Peter"})
CREATE (u2:User {name: "Paul"})
CREATE (u3:User {name: "Mary"})
```
And in the `.down.sql` let's delete it:
```
MATCH (u:User) WHERE u.name IN ["Peter", "Paul", "Mary"] DELETE u
```
Ideally your migrations should be idempotent. You can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)
## Run migrations
```
migrate -database ${NEO4J_URL} -path db/migrations up
```
Let's check if the table was created properly by running `bin/cypher-shell -u neo4j -p password`, then `neo4j> MATCH (u:User)`
The output you are supposed to see:
```
+-----------------------------------------------------------------+
| u |
+-----------------------------------------------------------------+
| (:User {name: "Peter") |
| (:User {name: "Paul") |
| (:User {name: "Mary") |
+-----------------------------------------------------------------+
```
Great! Now let's check if running reverse migration also works:
```
migrate -database ${NEO4J_URL} -path db/migrations down
```
Make sure to check if your database changed as expected in this case as well.
## Database transactions
To show database transactions usage, let's create another set of migrations by running:
```
migrate create -ext cypher -dir db/migrations -seq add_mood_to_users
```
Again, it should create for us two migrations files:
- 000002_add_mood_to_users.down.cypher
- 000002_add_mood_to_users.up.cypher
In Neo4j, when we want our queries to be done in a transaction, we need to wrap it with `:BEGIN` and `:COMMIT` commands.
Migration up:
```
:BEGIN
MATCH (u:User)
SET u.mood = "Cheery"
:COMMIT
```
Migration down:
```
:BEGIN
MATCH (u:User)
SET u.mood = null
:COMMIT
```
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
import (
"log"
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/neo4j"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
m, err := migrate.New(
"file://db/migrations",
"neo4j://neo4j:password@localhost:7687/")
if err != nil {
log.Fatal(err)
}
if err := m.Up(); err != nil {
log.Fatal(err)
}
}
```
@@ -0,0 +1 @@
DROP CONSTRAINT ON (m:Movie) ASSERT m.Name IS UNIQUE
@@ -0,0 +1 @@
CREATE CONSTRAINT ON (m:Movie) ASSERT m.Name IS UNIQUE
@@ -0,0 +1,2 @@
MATCH (m:Movie)
DELETE m
@@ -0,0 +1,2 @@
CREATE (:Movie {name: "Footloose"})
CREATE (:Movie {name: "Ghost"})
@@ -0,0 +1,3 @@
CREATE (:Movie {name: "Hollow Man"});
CREATE (:Movie {name: "Mystic River"});
;;;
+303
View File
@@ -0,0 +1,303 @@
package neo4j
import (
"bytes"
"errors"
"fmt"
"io"
neturl "net/url"
"strconv"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/neo4j/neo4j-go-driver/neo4j"
)
func init() {
db := Neo4j{}
database.Register("neo4j", &db)
}
const DefaultMigrationsLabel = "SchemaMigration"
var (
StatementSeparator = []byte(";")
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
)
type Config struct {
MigrationsLabel string
MultiStatement bool
MultiStatementMaxSize int
}
type Neo4j struct {
driver neo4j.Driver
lock uint32
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(driver neo4j.Driver, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
nDriver := &Neo4j{
driver: driver,
config: config,
}
if err := nDriver.ensureVersionConstraint(); err != nil {
return nil, err
}
return nDriver, nil
}
func (n *Neo4j) Open(url string) (database.Driver, error) {
uri, err := neturl.Parse(url)
if err != nil {
return nil, err
}
password, _ := uri.User.Password()
authToken := neo4j.BasicAuth(uri.User.Username(), password, "")
uri.User = nil
uri.Scheme = "bolt"
msQuery := uri.Query().Get("x-multi-statement")
// Whether to turn on/off TLS encryption.
tlsEncrypted := uri.Query().Get("x-tls-encrypted")
multi := false
encrypted := false
if msQuery != "" {
multi, err = strconv.ParseBool(uri.Query().Get("x-multi-statement"))
if err != nil {
return nil, err
}
}
if tlsEncrypted != "" {
encrypted, err = strconv.ParseBool(tlsEncrypted)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := uri.Query().Get("x-multi-statement-max-size"); s != "" {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
uri.RawQuery = ""
driver, err := neo4j.NewDriver(uri.String(), authToken, func(config *neo4j.Config) {
config.Encrypted = encrypted
})
if err != nil {
return nil, err
}
return WithInstance(driver, &Config{
MigrationsLabel: DefaultMigrationsLabel,
MultiStatement: multi,
MultiStatementMaxSize: multiStatementMaxSize,
})
}
func (n *Neo4j) Close() error {
return n.driver.Close()
}
// local locking in order to pass tests, Neo doesn't support database locking
func (n *Neo4j) Lock() error {
if !atomic.CompareAndSwapUint32(&n.lock, 0, 1) {
return database.ErrLocked
}
return nil
}
func (n *Neo4j) Unlock() error {
if !atomic.CompareAndSwapUint32(&n.lock, 1, 0) {
return database.ErrNotLocked
}
return nil
}
func (n *Neo4j) Run(migration io.Reader) (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
if n.config.MultiStatement {
_, err = session.WriteTransaction(func(transaction neo4j.Transaction) (interface{}, error) {
var stmtRunErr error
if err := multistmt.Parse(migration, StatementSeparator, n.config.MultiStatementMaxSize, func(stmt []byte) bool {
trimStmt := bytes.TrimSpace(stmt)
if len(trimStmt) == 0 {
return true
}
trimStmt = bytes.TrimSuffix(trimStmt, StatementSeparator)
if len(trimStmt) == 0 {
return true
}
result, err := transaction.Run(string(trimStmt), nil)
if _, err := neo4j.Collect(result, err); err != nil {
stmtRunErr = err
return false
}
return true
}); err != nil {
return nil, err
}
return nil, stmtRunErr
})
return err
}
body, err := io.ReadAll(migration)
if err != nil {
return err
}
_, err = neo4j.Collect(session.Run(string(body[:]), nil))
return err
}
func (n *Neo4j) SetVersion(version int, dirty bool) (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
query := fmt.Sprintf("MERGE (sm:%s {version: $version}) SET sm.dirty = $dirty, sm.ts = datetime()",
n.config.MigrationsLabel)
_, err = neo4j.Collect(session.Run(query, map[string]interface{}{"version": version, "dirty": dirty}))
if err != nil {
return err
}
return nil
}
type MigrationRecord struct {
Version int
Dirty bool
}
func (n *Neo4j) Version() (version int, dirty bool, err error) {
session, err := n.driver.Session(neo4j.AccessModeRead)
if err != nil {
return database.NilVersion, false, err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
query := fmt.Sprintf(`MATCH (sm:%s) RETURN sm.version AS version, sm.dirty AS dirty
ORDER BY COALESCE(sm.ts, datetime({year: 0})) DESC, sm.version DESC LIMIT 1`,
n.config.MigrationsLabel)
result, err := session.ReadTransaction(func(transaction neo4j.Transaction) (interface{}, error) {
result, err := transaction.Run(query, nil)
if err != nil {
return nil, err
}
if result.Next() {
record := result.Record()
mr := MigrationRecord{}
versionResult, ok := record.Get("version")
if !ok {
mr.Version = database.NilVersion
} else {
mr.Version = int(versionResult.(int64))
}
dirtyResult, ok := record.Get("dirty")
if ok {
mr.Dirty = dirtyResult.(bool)
}
return mr, nil
}
return nil, result.Err()
})
if err != nil {
return database.NilVersion, false, err
}
if result == nil {
return database.NilVersion, false, err
}
mr := result.(MigrationRecord)
return mr.Version, mr.Dirty, err
}
func (n *Neo4j) Drop() (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
if _, err := neo4j.Collect(session.Run("MATCH (n) DETACH DELETE n", nil)); err != nil {
return err
}
return nil
}
func (n *Neo4j) ensureVersionConstraint() (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
/**
Get constraint and check to avoid error duplicate
using db.labels() to support Neo4j 3 and 4.
Neo4J 3 doesn't support db.constraints() YIELD name
*/
res, err := neo4j.Collect(session.Run(fmt.Sprintf("CALL db.labels() YIELD label WHERE label=\"%s\" RETURN label", n.config.MigrationsLabel), nil))
if err != nil {
return err
}
if len(res) == 1 {
return nil
}
query := fmt.Sprintf("CREATE CONSTRAINT ON (a:%s) ASSERT a.version IS UNIQUE", n.config.MigrationsLabel)
if _, err := neo4j.Collect(session.Run(query, nil)); err != nil {
return err
}
return nil
}
+138
View File
@@ -0,0 +1,138 @@
package neo4j
import (
"bytes"
"context"
"fmt"
"log"
"testing"
"github.com/dhui/dktest"
"github.com/neo4j/neo4j-go-driver/neo4j"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady,
Env: map[string]string{"NEO4J_AUTH": "neo4j/migratetest", "NEO4J_ACCEPT_LICENSE_AGREEMENT": "yes"}}
specs = []dktesting.ContainerSpec{
{ImageName: "neo4j:4.0", Options: opts},
{ImageName: "neo4j:4.0-enterprise", Options: opts},
{ImageName: "neo4j:3.5", Options: opts},
{ImageName: "neo4j:3.5-enterprise", Options: opts},
}
)
func neoConnectionString(host, port string) string {
return fmt.Sprintf("bolt://neo4j:migratetest@%s:%s", host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(7687)
if err != nil {
return false
}
driver, err := neo4j.NewDriver(
neoConnectionString(ip, port),
neo4j.BasicAuth("neo4j", "migratetest", ""),
func(config *neo4j.Config) {
config.Encrypted = false
})
if err != nil {
return false
}
defer func() {
if err := driver.Close(); err != nil {
log.Println("close error:", err)
}
}()
session, err := driver.Session(neo4j.AccessModeRead)
if err != nil {
return false
}
result, err := session.Run("RETURN 1", nil)
if err != nil {
return false
} else if result.Err() != nil {
return false
}
return true
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(7687)
if err != nil {
t.Fatal(err)
}
n := &Neo4j{}
d, err := n.Open(neoConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("MATCH (a) RETURN a"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(7687)
if err != nil {
t.Fatal(err)
}
n := &Neo4j{}
neoUrl := neoConnectionString(ip, port) + "/?x-multi-statement=true"
d, err := n.Open(neoUrl)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "neo4j", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMalformed(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(7687)
if err != nil {
t.Fatal(err)
}
n := &Neo4j{}
d, err := n.Open(neoConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
migration := bytes.NewReader([]byte("CREATE (a {qid: 1) RETURN a"))
if err := d.Run(migration); err == nil {
t.Fatal("expected failure for malformed migration")
}
})
}
+9 -8
View File
@@ -8,13 +8,15 @@ import (
)
const reservedChars = "!#$%&'()*+,/:;=?@[]"
const reservedCharTestNamePrefix = "reserved char "
const baseUsername = "username"
const scheme = "database://"
// TestUserUnencodedReservedURLChars documents the behavior of using unencoded reserved characters in usernames with
// net/url Parse()
func TestUserUnencodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
@@ -64,7 +66,7 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run("reserved char "+tc.char, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
s := scheme + baseUsername + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -98,13 +100,12 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
}
func TestUserEncodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
for _, c := range reservedChars {
c := string(c)
t.Run("reserved char "+c, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := scheme + baseUsername + encodedChar + urlSuffixAndSep
expectedUsername := baseUsername + c
@@ -126,7 +127,7 @@ func TestUserEncodedReservedURLChars(t *testing.T) {
// with net/url Parse()
func TestPasswordUnencodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := "database://" + username + ":"
schemeAndUsernameAndSep := scheme + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
@@ -174,7 +175,7 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run("reserved char "+tc.char, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
s := schemeAndUsernameAndSep + basePassword + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -213,13 +214,13 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
func TestPasswordEncodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := "database://" + username + ":"
schemeAndUsernameAndSep := scheme + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
for _, c := range reservedChars {
c := string(c)
t.Run("reserved char "+c, func(t *testing.T) {
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := schemeAndUsernameAndSep + basePassword + encodedChar + urlSuffixAndSep
expectedPassword := basePassword + c
+43
View File
@@ -0,0 +1,43 @@
# pgx
This package is for [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). A backend for the newer [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5) is [also available](v5).
`pgx://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `x-lock-strategy` | `LockStrategy` | Strategy used for locking during migration (default: advisory) |
| `x-lock-table` | `LockTable` | Name of the table which maintains the migration lock (default: schema_lock) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS users;
@@ -0,0 +1,5 @@
CREATE TABLE users (
user_id integer unique,
name varchar(40),
email varchar(40)
);
@@ -0,0 +1 @@
ALTER TABLE users DROP COLUMN IF EXISTS city;
@@ -0,0 +1,3 @@
ALTER TABLE users ADD COLUMN city varchar(100);
@@ -0,0 +1 @@
DROP INDEX IF EXISTS users_email_index;
@@ -0,0 +1,3 @@
CREATE UNIQUE INDEX CONCURRENTLY users_email_index ON users (email);
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS books;
@@ -0,0 +1,5 @@
CREATE TABLE books (
user_id integer,
name varchar(40),
author varchar(40)
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS movies;
@@ -0,0 +1,5 @@
CREATE TABLE movies (
user_id integer,
name varchar(40),
director varchar(40)
);
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
+617
View File
@@ -0,0 +1,617 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgconn"
"github.com/jackc/pgerrcode"
_ "github.com/jackc/pgx/v4/stdlib"
"github.com/lib/pq"
)
const (
LockStrategyAdvisory = "advisory"
LockStrategyTable = "table"
)
func init() {
db := Postgres{}
database.Register("pgx", &db)
database.Register("pgx4", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
DefaultLockTable = "schema_lock"
DefaultLockStrategy = LockStrategyAdvisory
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
ErrDatabaseDirty = fmt.Errorf("database is dirty")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
LockTable string
LockStrategy string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if len(config.LockTable) == 0 {
config.LockTable = DefaultLockTable
}
if len(config.LockStrategy) == 0 {
config.LockStrategy = DefaultLockStrategy
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureLockTable(); err != nil {
return nil, err
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v4", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
lockStrategy := purl.Query().Get("x-lock-strategy")
lockTable := purl.Query().Get("x-lock-table")
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
LockStrategy: lockStrategy,
LockTable: lockTable,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.applyAdvisoryLock()
case LockStrategyTable:
return p.applyTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
}
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.releaseAdvisoryLock()
case LockStrategyTable:
return p.releaseTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
}
})
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) applyAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
}
func (p *Postgres) applyTableLock() error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
defer func() {
errRollback := tx.Rollback()
if errRollback != nil {
err = errors.Join(err, errRollback)
}
}()
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked {
return database.ErrLocked
}
query = "INSERT INTO " + pq.QuoteIdentifier(p.config.LockTable) + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return tx.Commit()
}
func (p *Postgres) releaseAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) releaseTableLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "DELETE FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
if _, err := p.db.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
// do not drop lock table
if tableName == p.config.LockTable && p.config.LockStrategy == LockStrategyTable {
continue
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) ensureLockTable() error {
if p.config.LockStrategy != LockStrategyTable {
return nil
}
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := p.db.QueryRow(query, p.config.LockTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE ` + pq.QuoteIdentifier(p.config.LockTable) + ` (lock_id BIGINT NOT NULL PRIMARY KEY)`
if _, err := p.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+786
View File
@@ -0,0 +1,786 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMigrateLockTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-lock-strategy=table", "x-lock-table=lock_table")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
+41
View File
@@ -0,0 +1,41 @@
# pgx
This package is for [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5). A backend for the older [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). is [also available](..).
`pgx5://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
+480
View File
@@ -0,0 +1,480 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgerrcode"
"github.com/jackc/pgx/v5/pgconn"
_ "github.com/jackc/pgx/v5/stdlib"
)
func init() {
db := Postgres{}
database.Register("pgx5", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v5", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
+761
View File
@@ -0,0 +1,761 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://../examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
+11
View File
@@ -5,6 +5,10 @@
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
@@ -26,3 +30,10 @@
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
+28 -9
View File
@@ -7,9 +7,9 @@ Our user here is `postgres`, password `password`, and host is `localhost`.
```
psql -h localhost -U postgres -w -c "create database example;"
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convienience:
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
```
export POSTGRESQL_URL=postgres://postgres:password@localhost:5432/example?sslmode=disable
export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/example?sslmode=disable'
```
`sslmode=disable` means that the connection with our database will not be encrypted. Enabling it is left as an exercise.
@@ -27,7 +27,7 @@ If there were no errors, we should have two files available under `db/migrations
Note the `sql` extension that we provided.
In the `.up.sql` file let's create the table:
```
```sql
CREATE TABLE IF NOT EXISTS users(
user_id serial PRIMARY KEY,
username VARCHAR (50) UNIQUE NOT NULL,
@@ -36,10 +36,10 @@ CREATE TABLE IF NOT EXISTS users(
);
```
And in the `.down.sql` let's delete it:
```
```sql
DROP TABLE IF EXISTS users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](../../GETTING_STARTED.md#create-migrations)
## Run migrations
```
@@ -79,7 +79,7 @@ Again, it should create for us two migrations files:
In Postgres, when we want our queries to be done in a transaction, we need to wrap it with `BEGIN` and `COMMIT` commands.
In our example, we are going to add a column to our database that can only accept enumerable values or NULL.
Migration up:
```
```sql
BEGIN;
CREATE TYPE enum_mood AS ENUM (
@@ -92,7 +92,7 @@ ALTER TABLE users ADD COLUMN mood enum_mood;
COMMIT;
```
Migration down:
```
```sql
BEGIN;
ALTER TABLE users DROP COLUMN mood;
@@ -124,7 +124,7 @@ Indexes:
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
```go
import (
"log"
@@ -145,4 +145,23 @@ func main() {
}
}
```
You can find details [here](README.md#use-in-your-go-project)
You can find details [here](README.md#use-in-your-go-project)
## Fix issue where migrations run twice
When the schema and role names are the same, you might run into issues if you create this schema using migrations.
This is caused by the fact that the [default `search_path`](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) is `"$user", public`.
In the first run (with an empty database) the migrate table is created in `public`.
When the migrations create the `$user` schema, the next run will store (a new) migrate table in this schema (due to order of schemas in `search_path`) and tries to apply all migrations again (most likely failing).
To solve this you need to change the default `search_path` by removing the `$user` component, so the migrate table is always stored in the (available) `public` schema.
This can be done using the [`search_path` query parameter in the URL](https://github.com/jexia/migrate/blob/fix-postgres-version-table/database/postgres/README.md#postgres).
For example to force the migrations table in the public schema you can use:
```
export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/example?sslmode=disable&search_path=public'
```
Note that you need to explicitly add the schema names to the table names in your migrations when you to modify the tables of the non-public schema.
Alternatively you can add the non-public schema manually (before applying the migrations) if that is possible in your case and let the tool store the migrations table in this schema as well.
+213 -89
View File
@@ -1,20 +1,23 @@
// +build go1.9
//go:build go1.9
package postgres
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
multierror "github.com/hashicorp/go-multierror"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/lib/pq"
)
@@ -24,7 +27,12 @@ func init() {
database.Register("postgresql", &db)
}
var DefaultMigrationsTable = "schema_migrations"
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
@@ -34,67 +42,83 @@ var (
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
MigrationsTable string
MigrationsTableQuoted bool
MultiStatementEnabled bool
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked bool
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postgres, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
if err := conn.PingContext(ctx); err != nil {
return nil, err
}
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName sql.NullString
if err := conn.QueryRowContext(ctx, query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if !schemaName.Valid {
return nil, ErrNoSchema
}
config.SchemaName = schemaName.String
}
config.DatabaseName = databaseName
query = `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
@@ -105,6 +129,26 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return px, nil
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
px, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
px.db = instance
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
@@ -117,10 +161,52 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
})
if err != nil {
@@ -132,7 +218,11 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
var dbErr error
if p.db != nil {
dbErr = p.db.Close()
}
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -141,53 +231,69 @@ func (p *Postgres) Close() error {
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
if p.isLocked {
return database.ErrLocked
}
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.SchemaName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
// This will either obtain the lock immediately and return true,
// or return false if the lock cannot be acquired immediately.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
p.isLocked = true
return nil
return nil
})
}
func (p *Postgres) Unlock() error {
if !p.isLocked {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.SchemaName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
p.isLocked = false
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
// run migration
query := string(migr[:])
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pq.Error); ok {
var line uint
var col uint
@@ -204,17 +310,16 @@ func (p *Postgres) Run(migration io.Reader) error {
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: migr, Line: line}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.Replace(s, "\r\n", "\n", -1)
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -253,19 +358,22 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.MigrationsTable)
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` (version, dirty) VALUES ($1, $2)`
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -279,7 +387,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` LIMIT 1`
query := `SELECT version, dirty FROM ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
@@ -307,7 +415,7 @@ func (p *Postgres) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -322,6 +430,9 @@ func (p *Postgres) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -346,15 +457,28 @@ func (p *Postgres) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
query := `CREATE TABLE IF NOT EXISTS ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` (version bigint not null primary key, dirty boolean not null)`
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+360 -32
View File
@@ -6,37 +6,46 @@ import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"log"
"github.com/golang-migrate/migrate/v4"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:9.4", Options: opts},
{ImageName: "postgres:9.5", Options: opts},
{ImageName: "postgres:9.6", Options: opts},
{ImageName: "postgres:10", Options: opts},
{ImageName: "postgres:11", Options: opts},
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string) string {
return fmt.Sprintf("postgres://postgres@%s:%s/postgres?sslmode=disable", host, port)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -67,7 +76,41 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testMultipleStatements", testMultipleStatements)
t.Run("testMultipleStatementsInMultiStatementMode", testMultipleStatementsInMultiStatementMode)
t.Run("testErrorParsing", testErrorParsing)
t.Run("testFilterCustomQuery", testFilterCustomQuery)
t.Run("testWithSchema", testWithSchema)
t.Run("testMigrationTableOption", testMigrationTableOption)
t.Run("testFailToCreateTableWithoutPermissions", testFailToCreateTableWithoutPermissions)
t.Run("testCheckBeforeCreateTable", testCheckBeforeCreateTable)
t.Run("testParallelSchema", testParallelSchema)
t.Run("testPostgresLock", testPostgresLock)
t.Run("testWithInstanceConcurrent", testWithInstanceConcurrent)
t.Run("testWithConnection", testWithConnection)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -89,7 +132,7 @@ func Test(t *testing.T) {
})
}
func TestMigrate(t *testing.T) {
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -111,11 +154,11 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("SELECT 1"))
dt.TestMigrate(t, m)
})
}
func TestMultiStatement(t *testing.T) {
func testMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -148,7 +191,40 @@ func TestMultiStatement(t *testing.T) {
})
}
func TestErrorParsing(t *testing.T) {
func testMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func testErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -177,14 +253,15 @@ func TestErrorParsing(t *testing.T) {
})
}
func TestFilterCustomQuery(t *testing.T) {
func testFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", ip, port)
addr := fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-custom=foobar",
pgPassword, ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
@@ -198,7 +275,7 @@ func TestFilterCustomQuery(t *testing.T) {
})
}
func TestWithSchema(t *testing.T) {
func testWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -226,7 +303,8 @@ func TestWithSchema(t *testing.T) {
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", ip, port))
d2, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foobar",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
@@ -240,7 +318,7 @@ func TestWithSchema(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if version != -1 {
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
@@ -267,7 +345,222 @@ func TestWithSchema(t *testing.T) {
})
}
func TestParallelSchema(t *testing.T) {
func testMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func testFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func testCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func testParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -295,7 +588,8 @@ func TestParallelSchema(t *testing.T) {
}
// re-connect using that schemas
dfoo, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foo", ip, port))
dfoo, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
@@ -305,7 +599,8 @@ func TestParallelSchema(t *testing.T) {
}
}()
dbar, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=bar", ip, port))
dbar, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=bar",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
@@ -333,11 +628,7 @@ func TestParallelSchema(t *testing.T) {
})
}
func TestWithInstance(t *testing.T) {
}
func TestPostgres_Lock(t *testing.T) {
func testPostgresLock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -377,7 +668,7 @@ func TestPostgres_Lock(t *testing.T) {
})
}
func TestWithInstance_Concurrent(t *testing.T) {
func testWithInstanceConcurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -419,6 +710,44 @@ func TestWithInstance_Concurrent(t *testing.T) {
}
})
}
func testWithConnection(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
db, err := sql.Open("postgres", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
ctx := context.Background()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatal(err)
}
p, err := WithConnection(ctx, conn, &Config{})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := p.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, p, []byte("SELECT 1"))
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
@@ -469,10 +798,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.Replace(input, "\n", "\r\n", -1)
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.Replace(input, "FROM", "FRÖM", -1)
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
@@ -497,5 +826,4 @@ func Test_computeLineFromPos(t *testing.T) {
run(true, true)
})
}
}
+26 -24
View File
@@ -2,17 +2,16 @@ package ql
import (
"database/sql"
"errors"
"fmt"
"github.com/hashicorp/go-multierror"
"io"
"io/ioutil"
"strings"
nurl "net/url"
"strings"
"sync/atomic"
_ "github.com/cznic/ql/driver"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
_ "modernc.org/ql/driver"
)
func init() {
@@ -34,7 +33,7 @@ type Config struct {
type Ql struct {
db *sql.DB
isLocked bool
isLocked atomic.Bool
config *Config
}
@@ -72,11 +71,7 @@ func (m *Ql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
@@ -85,7 +80,7 @@ func (m *Ql) ensureVersionTable() (err error) {
return err
}
if _, err := tx.Exec(fmt.Sprintf(`
CREATE TABLE IF NOT EXISTS %s (version uint64,dirty bool);
CREATE TABLE IF NOT EXISTS %s (version uint64, dirty bool);
CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version);
`, m.config.MigrationsTable, m.config.MigrationsTable)); err != nil {
if err := tx.Rollback(); err != nil {
@@ -133,9 +128,10 @@ func (m *Ql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
@@ -148,6 +144,10 @@ func (m *Ql) Drop() (err error) {
}
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
@@ -161,21 +161,19 @@ func (m *Ql) Drop() (err error) {
return nil
}
func (m *Ql) Lock() error {
if m.isLocked {
if !m.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
m.isLocked = true
return nil
}
func (m *Ql) Unlock() error {
if !m.isLocked {
return nil
if !m.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
m.isLocked = false
return nil
}
func (m *Ql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -190,7 +188,7 @@ func (m *Ql) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -210,11 +208,15 @@ func (m *Ql) SetVersion(version int, dirty bool) error {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (%d, %t)`, m.config.MigrationsTable, version, dirty)
if _, err := tx.Exec(query); err != nil {
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (uint64(?1), ?2)`,
m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+4 -22
View File
@@ -3,27 +3,17 @@ package ql
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
_ "github.com/cznic/ql/driver"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
_ "github.com/golang-migrate/migrate/v4/source/file"
_ "modernc.org/ql/driver"
)
func Test(t *testing.T) {
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
p := &Ql{}
addr := fmt.Sprintf("ql://%s", filepath.Join(dir, "ql.db"))
@@ -45,15 +35,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dir := t.TempDir()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
db, err := sql.Open("ql", filepath.Join(dir, "ql.db"))
@@ -77,5 +59,5 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("CREATE TABLE t (Qty int, Name string);"))
dt.TestMigrate(t, m)
}
+32 -27
View File
@@ -1,20 +1,20 @@
// +build go1.9
//go:build go1.9
package redshift
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
@@ -36,7 +36,7 @@ type Config struct {
}
type Redshift struct {
isLocked bool
isLocked atomic.Bool
conn *sql.Conn
db *sql.DB
@@ -53,17 +53,19 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return nil, err
}
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
config.DatabaseName = databaseName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
@@ -124,20 +126,21 @@ func (p *Redshift) Close() error {
// Redshift does not support advisory lock functions: https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html
func (p *Redshift) Lock() error {
if p.isLocked {
if !p.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
p.isLocked = true
return nil
}
func (p *Redshift) Unlock() error {
p.isLocked = false
if !p.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (p *Redshift) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
@@ -171,7 +174,7 @@ func (p *Redshift) Run(migration io.Reader) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.Replace(s, "\r\n", "\n", -1)
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -213,16 +216,19 @@ func (p *Redshift) SetVersion(version int, dirty bool) error {
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO "` + p.config.MigrationsTable + `" (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = multierror.Append(err, errRollback)
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -264,7 +270,7 @@ func (p *Redshift) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
err = errors.Join(err, errClose)
}
}()
@@ -279,6 +285,9 @@ func (p *Redshift) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -303,11 +312,7 @@ func (p *Redshift) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
err = errors.Join(err, e)
}
}()
+19 -9
View File
@@ -22,15 +22,25 @@ import (
)
import (
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "redshift"
)
var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true,
ReadyFunc: isReady,
}
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:8", Options: opts},
{ImageName: "migrate/postgres8:8", Options: opts},
}
)
@@ -43,7 +53,7 @@ func pgConnectionString(host, port string) string {
}
func connectionString(schema, host, port string) string {
return fmt.Sprintf("%s://postgres@%s:%s/postgres?sslmode=disable", schema, host, port)
return fmt.Sprintf("%s://postgres:%s@%s:%s/postgres?sslmode=disable", schema, pgPassword, host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -118,7 +128,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m, []byte("SELECT 1"))
dt.TestMigrate(t, m)
})
}
@@ -191,7 +201,7 @@ func TestFilterCustomQuery(t *testing.T) {
t.Fatal(err)
}
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", ip, port)
addr := fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-custom=foobar", pgPassword, ip, port)
p := &Redshift{}
d, err := p.Open(addr)
if err != nil {
@@ -233,7 +243,7 @@ func TestWithSchema(t *testing.T) {
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", ip, port))
d2, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foobar", pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
@@ -247,7 +257,7 @@ func TestWithSchema(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if version != -1 {
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
@@ -368,10 +378,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.Replace(input, "\n", "\r\n", -1)
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.Replace(input, "FROM", "FRÖM", -1)
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
+18
View File
@@ -0,0 +1,18 @@
# rqlite
`rqlite://admin:secret@server1.example.com:4001/?level=strong&timeout=5`
The `rqlite` url scheme is used for both secure and insecure connections. If connecting to an insecure database, pass `x-connect-insecure` in your URL query, or use `WithInstance` to pass an established connection.
The migrations table name is configurable through the `x-migrations-table` URL query parameter, or by using `WithInstance` and passing `MigrationsTable` through `Config`.
Other connect parameters are directly passed through to the database driver. For examples of connection strings, see https://github.com/rqlite/gorqlite#examples.
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-connect-insecure` | n/a: set on instance | Boolean to indicate whether to use an insecure connection. Defaults to `false`. |
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table. Defaults to `schema_migrations`. |
## Notes
* Uses the https://github.com/rqlite/gorqlite driver
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1,3 @@
CREATE TABLE pets (
name string
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1 @@
ALTER TABLE pets ADD predator bool;
+328
View File
@@ -0,0 +1,328 @@
package rqlite
import (
"errors"
"fmt"
"io"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/rqlite/gorqlite"
)
func init() {
database.Register("rqlite", &Rqlite{})
}
const (
// DefaultMigrationsTable defines the default rqlite migrations table
DefaultMigrationsTable = "schema_migrations"
// DefaultConnectInsecure defines the default setting for connect insecure
DefaultConnectInsecure = false
)
// ErrNilConfig is returned if no configuration was passed to WithInstance
var ErrNilConfig = fmt.Errorf("no config")
// ErrBadConfig is returned if configuration was invalid
var ErrBadConfig = fmt.Errorf("bad parameter")
// Config defines the driver configuration
type Config struct {
// ConnectInsecure sets whether the connection uses TLS. Ineffectual when using WithInstance
ConnectInsecure bool
// MigrationsTable configures the migrations table name
MigrationsTable string
}
type Rqlite struct {
db *gorqlite.Connection
isLocked atomic.Bool
config *Config
}
// WithInstance creates a rqlite database driver with an existing gorqlite database connection
// and a Config struct
func WithInstance(instance *gorqlite.Connection, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
// we use the consistency level check as a database ping
if _, err := instance.ConsistencyLevel(); err != nil {
return nil, err
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
driver := &Rqlite{
db: instance,
config: config,
}
if err := driver.ensureVersionTable(); err != nil {
return nil, err
}
return driver, nil
}
// OpenURL creates a rqlite database driver from a connect URL
func OpenURL(url string) (database.Driver, error) {
d := &Rqlite{}
return d.Open(url)
}
func (r *Rqlite) ensureVersionTable() (err error) {
if err = r.Lock(); err != nil {
return err
}
defer func() {
if e := r.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
stmts := []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (version uint64, dirty bool)`, r.config.MigrationsTable),
fmt.Sprintf(`CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version)`, r.config.MigrationsTable),
}
if _, err := r.db.Write(stmts); err != nil {
return err
}
return nil
}
// Open returns a new driver instance configured with parameters
// coming from the URL string. Migrate will call this function
// only once per instance.
func (r *Rqlite) Open(url string) (database.Driver, error) {
dburl, config, err := parseUrl(url)
if err != nil {
return nil, err
}
r.config = config
r.db, err = gorqlite.Open(dburl.String())
if err != nil {
return nil, err
}
if err := r.ensureVersionTable(); err != nil {
return nil, err
}
return r, nil
}
// Close closes the underlying database instance managed by the driver.
// Migrate will call this function only once per instance.
func (r *Rqlite) Close() error {
r.db.Close()
return nil
}
// Lock should acquire a database lock so that only one migration process
// can run at a time. Migrate will call this function before Run is called.
// If the implementation can't provide this functionality, return nil.
// Return database.ErrLocked if database is already locked.
func (r *Rqlite) Lock() error {
if !r.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
// Unlock should release the lock. Migrate will call this function after
// all migrations have been run.
func (r *Rqlite) Unlock() error {
if !r.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
// Run applies a migration to the database. migration is guaranteed to be not nil.
func (r *Rqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if _, err := r.db.WriteOne(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// SetVersion saves version and dirty state.
// Migrate will call this function before and after each call to Run.
// version must be >= -1. -1 means NilVersion.
func (r *Rqlite) SetVersion(version int, dirty bool) error {
deleteQuery := fmt.Sprintf(`DELETE FROM %s`, r.config.MigrationsTable)
statements := []gorqlite.ParameterizedStatement{
{
Query: deleteQuery,
},
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
insertQuery := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, r.config.MigrationsTable)
if version >= 0 || (version == database.NilVersion && dirty) {
statements = append(statements, gorqlite.ParameterizedStatement{
Query: insertQuery,
Arguments: []interface{}{
version,
dirty,
},
})
}
wr, err := r.db.WriteParameterized(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i].Query)}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(deleteQuery + "\n" + insertQuery)}
}
return nil
}
// Version returns the currently active version and if the database is dirty.
// When no migration has been applied, it must return version -1.
// Dirty means, a previous migration failed and user interaction is required.
func (r *Rqlite) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM " + r.config.MigrationsTable + " LIMIT 1"
qr, err := r.db.QueryOne(query)
if err != nil {
return database.NilVersion, false, nil
}
if !qr.Next() {
return database.NilVersion, false, nil
}
if err := qr.Scan(&version, &dirty); err != nil {
return database.NilVersion, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty, nil
}
// Drop deletes everything in the database.
// Note that this is a breaking action, a new call to Open() is necessary to
// ensure subsequent calls work as expected.
func (r *Rqlite) Drop() error {
query := `SELECT name FROM sqlite_master WHERE type = 'table'`
tables, err := r.db.QueryOne(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
statements := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
statement := fmt.Sprintf(`DROP TABLE %s`, tableName)
statements = append(statements, statement)
}
}
// return if nothing to do
if len(statements) <= 0 {
return nil
}
wr, err := r.db.Write(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i])}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(strings.Join(statements, "\n"))}
}
return nil
}
func parseUrl(url string) (*nurl.URL, *Config, error) {
parsedUrl, err := nurl.Parse(url)
if err != nil {
return nil, nil, err
}
config, err := parseConfigFromQuery(parsedUrl.Query())
if err != nil {
return nil, nil, err
}
if parsedUrl.Scheme != "rqlite" {
return nil, nil, fmt.Errorf("bad scheme: %w", ErrBadConfig)
}
// adapt from rqlite to http/https schemes
if config.ConnectInsecure {
parsedUrl.Scheme = "http"
} else {
parsedUrl.Scheme = "https"
}
filteredUrl := migrate.FilterCustomQuery(parsedUrl)
return filteredUrl, config, nil
}
func parseConfigFromQuery(queryVals nurl.Values) (*Config, error) {
c := Config{
ConnectInsecure: DefaultConnectInsecure,
MigrationsTable: DefaultMigrationsTable,
}
migrationsTable := queryVals.Get("x-migrations-table")
if migrationsTable != "" {
if strings.HasPrefix(migrationsTable, "sqlite_") {
return nil, fmt.Errorf("invalid value for x-migrations-table: %w", ErrBadConfig)
}
c.MigrationsTable = migrationsTable
}
connectInsecureStr := queryVals.Get("x-connect-insecure")
if connectInsecureStr != "" {
connectInsecure, err := strconv.ParseBool(connectInsecureStr)
if err != nil {
return nil, fmt.Errorf("invalid value for x-connect-insecure: %w", ErrBadConfig)
}
c.ConnectInsecure = connectInsecure
}
return &c, nil
}
+324
View File
@@ -0,0 +1,324 @@
package rqlite
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"testing"
"github.com/dhui/dktest"
"github.com/rqlite/gorqlite"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
var defaultPort uint16 = 4001
var opts = dktest.Options{
Env: map[string]string{"NODE_ID": "1"},
PortRequired: true,
ReadyFunc: isReady,
}
var specs = []dktesting.ContainerSpec{
{ImageName: "rqlite/rqlite:7.21.4", Options: opts},
{ImageName: "rqlite/rqlite:8.0.6", Options: opts},
{ImageName: "rqlite/rqlite:8.11.1", Options: opts},
{ImageName: "rqlite/rqlite:8.12.3", Options: opts},
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
fmt.Println("error getting port")
return false
}
statusString := fmt.Sprintf("http://%s:%s/status", ip, port)
fmt.Println(statusString)
var readyResp struct {
Store struct {
Ready bool `json:"ready"`
} `json:"store"`
}
resp, err := http.Get(statusString)
if err != nil {
fmt.Println("error getting status")
return false
}
if resp.StatusCode != 200 {
fmt.Println("statusCode != 200")
return false
}
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("error reading body")
return false
}
if err := json.Unmarshal(body, &readyResp); err != nil {
fmt.Println("error unmarshaling body")
return false
}
return readyResp.Store.Ready
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
r := &Rqlite{}
d, err := r.Open(connectString)
assert.NoError(t, err)
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
driver, err := OpenURL(connectString)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
return
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
dt.TestMigrate(t, m)
})
}
func TestBadConnectInsecureParam(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?x-connect-insecure=foo", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestBadProtocol(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("postgres://%s:%s/database", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestNoConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
_, err = WithInstance(db, nil)
assert.ErrorIs(t, err, ErrNilConfig)
})
}
func TestWithInstanceEmptyConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
driver, err := WithInstance(db, &Config{})
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", DefaultMigrationsTable))
assert.NoError(t, err)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
})
}
func TestMigrationTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
config := Config{MigrationsTable: "my_migration_table"}
driver, err := WithInstance(db, &config)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
_, err = db.WriteOne(`INSERT INTO pets (name, predator) VALUES ("franklin", true)`)
assert.NoError(t, err)
res, err := db.QueryOne(`SELECT name, predator FROM pets LIMIT 1`)
assert.NoError(t, err)
_ = res.Next()
// make sure we can use the migrated table
var petName string
var petPredator int
err = res.Scan(&petName, &petPredator)
assert.NoError(t, err)
assert.Equal(t, petName, "franklin")
assert.Equal(t, petPredator, 1)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
})
}
func TestParseUrl(t *testing.T) {
tests := []struct {
name string
passedUrl string
expectedUrl string
expectedConfig *Config
expectedErr string
}{
{
"defaults",
"rqlite://localhost:4001",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"configure migration table",
"rqlite://localhost:4001?x-migrations-table=foo",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: "foo"},
"",
},
{
"configure connect insecure",
"rqlite://localhost:4001?x-connect-insecure=true",
"http://localhost:4001",
&Config{ConnectInsecure: true, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"invalid migration table",
"rqlite://localhost:4001?x-migrations-table=sqlite_bar",
"",
nil,
"invalid value for x-migrations-table: bad parameter",
},
{
"invalid connect insecure",
"rqlite://localhost:4001?x-connect-insecure=baz",
"",
nil,
"invalid value for x-connect-insecure: bad parameter",
},
{
"invalid url",
string([]byte{0x7f}),
"",
nil,
"parse \"\\x7f\": net/url: invalid control character in URL",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
actualUrl, actualConfig, actualErr := parseUrl(tt.passedUrl)
if tt.expectedUrl != "" {
assert.Equal(t, tt.expectedUrl, actualUrl.String())
} else {
assert.Nil(t, actualUrl)
}
assert.Equal(t, tt.expectedConfig, actualConfig)
if tt.expectedErr == "" {
assert.NoError(t, actualErr)
} else {
assert.EqualError(t, actualErr, tt.expectedErr)
}
})
}
}

Some files were not shown because too many files have changed in this diff Show More