Compare commits

..
898 Commits
Author SHA1 Message Date
Dale HuiandGitHub 1ba9becc5c Merge pull request #634 from Baumanar/master
Fix for failing test for AWS keyspace
2021-10-15 09:52:04 -07:00
Baumanar c81e147ad6 go mod tidy 2021-10-14 10:37:49 +02:00
Baumanar dbb574c92c fix test 2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar 79472544ff Delete all rows
https://github.com/golang-migrate/migrate/pull/579#discussion_r650178687
2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar b9b66468f5 Use ParseBool
https://github.com/golang-migrate/migrate/pull/579#discussion_r650171022
2021-10-14 10:36:38 +02:00
Florent SoltandBaumanar 5eaf23f836 Support for AWS Keyspaces 2021-10-14 10:36:38 +02:00
Dale HuiandGitHub f25b303a88 Merge pull request #638 from dhui/gosnowflake
Update gosnowflake from v1.4.3 to v1.6.3
2021-10-13 22:13:24 -07:00
Dale Hui 41431fbdc3 Update gosnowflake from v1.4.3 to v1.6.3
Addresses:
* https://github.com/golang-migrate/migrate/issues/632
* https://github.com/golang-migrate/migrate/issues/637
2021-10-13 21:18:38 -07:00
Dale Hui 780d5551e0 Update docker client usage with breaking change 2021-10-12 22:11:20 -07:00
Dale Hui 71106e5fd2 Update dktest to v0.3.7 to fix security warnings
https://github.com/dhui/dktest/releases/tag/v0.3.7
2021-10-12 21:58:29 -07:00
Dale HuiandGitHub eea81e7876 Merge pull request #628 from golang-migrate/revert-binary-location-change-in-docker-image
revert binary file location change in docker image
2021-09-27 12:12:47 -07:00
Fontinalis 1668f8e6a0 revert binary file location change in docker image 2021-09-27 10:13:33 +02:00
Dale HuiandGitHub 10a92d0cc8 Merge pull request #620 from golang-migrate/fix-source-file-driver
fix source/file driver
2021-09-16 16:45:03 -07:00
Fontinalis 0775323324 fix source/file driver 2021-09-16 14:24:01 +02:00
Dale Hui 14f6d3a92f Update build constraints
Issue caught by linter

https://pkg.go.dev/go/build#hdr-Build_Constraints
https://pkg.go.dev/cmd/go#hdr-Build_constraints
https://go.googlesource.com/proposal/+/master/design/draft-gobuild.md
2021-09-10 21:04:11 -07:00
Dale Hui 677b5deb09 Update golangci-lint config
- Update to renamed config keys
- Remove redundant linter (maligned)
2021-09-10 21:02:44 -07:00
Dale HuiandGitHub b4047a1b47 Merge pull request #617 from kaworu/pr/kaworu/database/clickhouse/honor-DefaultMigrationsTableEngine
clickhouse: honor `DefaultMigrationsTableEngine` when created `WithInstance()`
2021-09-10 20:36:47 -07:00
Alexandre Perrin 9db2d011f3 clickhouse: honor DefaultMigrationsTableEngine when created WithInstance()
Before this patch, clickhouse.WithInstance() would not select
DefaultMigrationsTableEngine when no MigrationsTableEngine was provided
through the clickhouse.Config{} argument. This behaviour forces the
caller to always provide explicitely the config's MigrationsTableEngine.

This patch make it so DefaultMigrationsTableEngine is used when
MigrationsTableEngine is not provided through clickhouse.Config{}.

Signed-off-by: Alexandre Perrin <alex@kaworu.ch>
2021-09-10 10:44:55 +02:00
Dale HuiandGitHub aef2747761 Fix grammar for Clickhouse cluster mode docs 2021-08-24 21:45:25 -07:00
aa119418b8 Adding support clickhouse cluster (#568)
* added url query for creating schema_migrations table on cluster

Signed-off-by: Ildar Valiullin <preved.911@gmail.com>

* changed documentation by clickhouse databases

Signed-off-by: Ildar Valiullin <preved.911@gmail.com>

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2021-08-24 21:39:30 -07:00
Dale Hui df6106d10e Support Go 1.17 and drop support for Go 1.15 2021-08-21 01:32:23 -07:00
Dale Hui b62d267202 Push builds for debian bullseye 2021-08-18 16:04:22 -07:00
Dale Hui d30a9850da Change build status badge to use GitHub Actions 2021-08-06 21:50:44 -07:00
Dale Hui bea4b82719 Fix docker image tags by undoing previous commit 2021-08-06 21:38:45 -07:00
Dale Hui ea0fa43397 Use fewer docker image tags 2021-08-06 20:41:19 -07:00
Dale HuiandGitHub 82dd9d78ce Merge pull request #609 from golang-migrate/607-fix-dockerfile
fix missing /migrate link in docker image
2021-08-06 09:58:35 -07:00
Fontinalis dba334c74e fix missing /migrate link in docker image 2021-08-06 10:44:05 +02:00
Dale HuiandGitHub 784a2f89da Merge pull request #608 from alethenorio/update-spanner-dependency
Update Google Spanner dependency
2021-08-05 21:20:58 -07:00
Eng Zer JunandGitHub 5aa3b3ad30 ci: migrate from CircleCI to GitHub Actions (#605)
* ci: migrate from CircleCI to GitHub Actions

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>

* ci: merge release.yaml into ci.yaml

At the time of this commit, it is not possible to make the `goreleaser`
job run in a separate workflow and detect the tag push event, see [1].

[1]: https://stackoverflow.com/a/68078768/7902371
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-08-05 21:12:21 -07:00
Alexandre Thenorio ab5fe5e783 Update Google Spanner dependency
Adds support for the following in SpanSQL

* ROW DELETION POLICY
* ALTER DATABASE
* Parsing keywords and functions in case insensitive
2021-08-05 19:25:26 +02:00
Dale Hui b2c89c743c Clean up Makefile targets
* move echo-source and echo-database targets so the build target is first and default
* mark echo-source and echo-database targets as .PHONY
2021-08-04 16:19:03 -07:00
Dale Hui a4c42d7390 Include source archives with releases
See: https://goreleaser.com/customization/source/
2021-08-03 21:41:15 -07:00
Dale Hui 823c2806a0 Allow prereleases via GoReleaser 2021-08-03 12:19:56 -07:00
5a601b986d Support for GitHub Actions and goreleaser (#604)
* goreleaser configuration and automatic release workflow

* update and fix goreleaser

* remove windows arm ignore

* add format override for windows to use zip files

* update environment and docker username

* the single source of truth

* fix depricated api issue ( thanks @dhui )

Co-authored-by: Mitsuo Heijo <mitsuo.heijo@gmail.com>
2021-07-30 23:09:32 -07:00
Dale HuiandGitHub 5bf05dc323 Remove -o option from go install example in CLI docs
The -o option only exists in `go build`, not `go install`. e.g. it's not a build flag
See: https://github.com/golang-migrate/migrate/issues/603#issuecomment-888432107
2021-07-28 09:06:18 -07:00
Dale HuiandGitHub 586cc866f5 Merge pull request #595 from DGollings/master
use github.com/cockroachdb/cockroach-go/v2/crdb
2021-07-27 09:22:16 -07:00
Dale Hui 614523948d Merge branch 'master' into crdb 2021-07-26 22:13:48 -07:00
Dale Hui aa47d3da8c Update MongoDB db driver to fix security issue
https://nvd.nist.gov/vuln/detail/CVE-2021-20329
2021-07-23 13:46:20 -07:00
Daniel Gollings 94649eedd8 ran go mod tidy 2021-07-14 14:44:05 +02:00
Daniel Gollings f1b7710148 use github.com/cockroachdb/cockroach-go/v2/crdb (for better go module support) 2021-07-14 14:35:05 +02:00
Sébastien CaparrosandGitHub 31cedbb961 Added a method to create a mysql database from a connection object (#583)
* Added a method to create a mysql database from a connection object

* Calling WithConnection from WithInstance to de-duplicate code

* Adding context and ping to mysql.WithConnection

* Interface type check at compile time
2021-07-06 10:45:43 -07:00
805f8c8f2d Add SqlServer MSI Support (#591)
* MSI Auth for SQL Server

Add MSI Auth option to SQL Server connection. Default if no password is provided in connection string.

* Parse resource endpoint from server for msi

update host name parsing to get just the resource endpoint for msi

* Update go-mssqldb

update go-mssqldb to resolve panic issue referenced here: https://github.com/denisenkom/go-mssqldb/pull/642

* Update sqlserver.go

switch from deprecated methods. NewServicePrincipalTokenFromManagedIdentity calls the two methods that are deprecated for us

* Update sqlserver.go

add useMsi param instead of looking for nil password

* Update sqlserver readme

Update sqlserver readme for msi auth. make useMsi a bit safer

* Update sqlserver.go

remove comment

* Update database/sqlserver/README.md

Co-authored-by: Keegan Campbell <me@kfcampbell.com>

* Update sqlserver.go

refactor resource uri logic into its own method

* Update sqlserver_test.go

add tests for msi connection. can only test whether it fails with useMsi= true, or succeeds with useMsi=false

* Update sqlserver.go

check msi.EnsureFresh return value

* Return error for multiple auth and move query filter

move migrate.FilterCustomQuery(purl).String() into one line out of if/else. return error if both useMsi=true and password are passed

* Update README.md

update readme with warning for useMsi

* Update sqlserver_test.go

Update TestMsiFalse test case as now it should fail when useMsi=false and no password is provided

Co-authored-by: Keegan Campbell <me@kfcampbell.com>
2021-07-01 23:08:44 -07:00
3dfb0ff538 used uber atomic bool instead standard in lock/unlock db (#580)
* used uber atomic bool instead standard in lock/unlock db

* fixing local lock optimization implementation

* fixing mysql local lock optimization implementation

* fixing cockroachdb local lock optimization implementation

* CAS wrapper to automatically restore

* added test for cas with restore

* added atomic lock to sqlite & mongo

* added atomic lock to sqlite & mongo

* fix/refactor test cas restore function

* disable lock for mongo

* mongo local lock before acquiring the db lock

Co-authored-by: a.prinkov <aprinkov@ourgapps.com>
2021-06-23 14:40:50 -07:00
Dale Hui 3b3c1b6aa5 Update go-github from v17 to v35 2021-05-28 14:56:29 -07:00
Dale Hui 8a1a8534dc Update snowflake DB driver to update to non-vulnerable jwt downstream package
Fixes https://github.com/golang-migrate/migrate/issues/569
2021-05-21 09:56:26 -07:00
Kyle ChadhaandGitHub 1695f40b3e fix(spanner): upgrade spanner client to latest to support NUMERIC types (#565)
* fix(spanner): upgrade spanner client to latest to support NUMERIC types

* run go mod tidy

* Add spanner example migration test case
2021-05-18 22:16:21 -07:00
Dale HuiandGitHub 2e7358e012 Merge pull request #560 from dhui/re-add-iofs-src-driver
Re-add the io/fs source driver
2021-05-10 23:38:05 -07:00
Dale Hui eac6684bf3 Fix iofs tests
Changes:
- embedded variables must be at the package level
- re-use embedded FS in example for tests
- bump Go module version to 1.16 for tests to run
    - This may cause issues building in 1.15
2021-05-10 22:49:46 -07:00
Dale Hui 53720240b4 Update docker builds to use Go 1.16 and Alpine Linux 3.13 2021-05-10 11:28:59 -07:00
Dale Hui 337ff632e3 Revert "Merge pull request #481 from johejo/revert-472-iofs_source_driver"
This reverts commit b690b78e22, reversing
changes made to 15931649a2.
2021-05-10 10:50:27 -07:00
Dale HuiandGitHub c4d82aa89a Merge pull request #558 from zikaeroh/bump-sqlite
Bump sqlite past memory corruption bug
2021-05-06 22:35:36 -07:00
zikaeroh 7a631fe801 Bump sqlite past memory corruption bug 2021-05-06 21:32:04 -07:00
Dale Hui ffdcb52998 Update sqlite and sqlite3 docs with comparison 2021-05-05 10:12:41 -07:00
Andrew GillisandGitHub 512097b404 Add support for modernc.org/sqlite (#555)
* Add support for modernc.org/sqlite

This is a CGo-free port of SQLite

* Remove sqlite from build target
2021-05-05 10:04:18 -07:00
Dale HuiandGitHub af256da8ca Merge pull request #550 from spacefill/547-postgres-pgx-ensureVersionTable-must-use-config-schema-name
[pgx][postgres] Config.SchemaName refactoring usage
2021-04-25 13:45:05 -07:00
Stéphane Klein d0af5824ad postgres,pgx: fix SchemaName parameter is ignored ##547 2021-04-25 07:34:19 +02:00
Bruno MouraandGitHub 4720914286 clickhouse: make table engine for schema versions table configurable (#551)
* clickhouse: make the table engine used for schema_migrations configurable

* clickhouse: tests creation with configurable table engine

* clickhouse: refactor tests to run with multiple table engines

* clickhouse: clarify the use case for the x-migrations-table-engine option

* clickhouse: fix naming consistency for the migrations table engine

* clickhouse: fix tests function wrap pattern
2021-04-21 22:03:11 -07:00
Stéphane KleinandGitHub 3d5d577b61 Postgres - add x-migrations-table-quoted url query option (#95) (#533)
* Postgres and pgx - Add x-migrations-table-quoted url query option to postgres and pgx drivers (#95)

By default, gomigrate quote migrations table name, if `x-migrations-table-quoted` is enabled, then you must to quote migrations table name manually, for instance `"gomigrate"."schema_migrations"`

* Work In Progress
2021-04-19 14:57:50 -07:00
Alexey PrinkovandGitHub d68a3abedb added tests for clickhouse database (#532)
* added tests for clickhouse

* added atomic lock flag to clickhouse & remove extra test

* added clickhouse test package
2021-04-05 15:37:50 -07:00
testtest959andGitHub 511ae9f5b6 Postgres: Add a check to determine if table already exists to elide CREATE query (#526)
* Squash commits

* Format

* Minor refactoring

* Address PR feedback; Add mustRun

* Fix a test assert
2021-03-18 21:03:57 -07:00
Dale HuiandGitHub 63aff4b03d Merge pull request #529 from linux019/upgrade_ch
upgrade clickhouse driver lib
2021-03-16 20:23:20 -07:00
= 2839a3d6f2 upgrade clickhouse driver lib 2021-03-16 18:51:56 +02:00
Dale HuiandGitHub 4573d98dce Merge pull request #527 from jeffwidman/patch-2
Fix typo
2021-03-15 23:58:57 -07:00
Jeff WidmanandGitHub a728e7a674 Fix typo 2021-03-15 21:45:44 -07:00
Dale HuiandGitHub 707fc52dfe Merge pull request #522 from Juneezee/remove-all-from-drop
Move `-all` flag usage from `drop` to `down`
2021-03-13 21:36:01 -08:00
Jeff WidmanandGitHub 6ecf7bcd0a Point docs link to v4 (#524)
* Point docs link to `v4`

It's very confusing that when I land directly on `https://pkg.go.dev/github.com/golang-migrate/migrate` I get a the message:
> You are look at a deprecated version, use the latest version instead

But when I click the `latest version` link, it goes to a completely dead/wrong place.

So I think for almost anyone who actually wants to look at the real docs, it's a lot simpler to link straight to `v4` by default. If they want a different version, they can change that. But let's have the default point to a helpful place, and then you can go looking for the broken version, rather than the other way around.

* Update the badge link as well
2021-03-13 21:30:45 -08:00
Dylan BargteilandGitHub 2f0c0752ed Add pgx driver package (#517)
* replace lib/pq dependency with pgx in postgres driver

* fix pgx error handling

pgconn is a low-level library, and errors from queries flow through higher-level pgx library

* update postgres tests to use pgx driver and error reporting

* update quoteIdentifier implementation to match lib/pq

* move changes for pgx support into new driver package

* add pgx driver to CLI

* restore examples used in testing for pgx driver

* replace postgres driver name with pgx in DSN in tests and docs

* consolidate connection string formatting in pgx testing

* lock pgx package at v4

Prior to v4, pgx did not support go modules. Version locking the import is the safest way to ensure we get the (as of now) latest version of the module. Additionally v4 implements some breaking changes to connection and error handling, which are also addressed in this commit.

* update parsed URL scheme instead of using string operation

* add pgx for builds using make

* add pgx to top level docs

* remove autogenerated TOC
2021-03-13 21:29:10 -08:00
Eng Zer Jun aa2dad421f fix: move -all flag usage from drop to down
The `-all` flag usage should be used in `down` only, not `drop`.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-03-13 15:22:09 +08:00
Dale HuiandGitHub 7dc03e3c0d Merge pull request #509 from nronas/master
Allow the ability to pass timezone when generating migration files
2021-02-24 09:36:46 -08:00
John Tzikas 6cdb8c1b23 Allow the abillity to pass timezone when generating migration files 2021-02-23 20:20:14 +02:00
Dale Hui e57a024960 Support Go 1.16 and drop support for Go 1.15
- Document CLI install instructions for Go 1.16
- Update golangci-lint from v1.33.0 to v1.37.0
2021-02-18 21:33:36 -08:00
Dale Hui 8709f196bf Restore CircleCI build cache after checkout and remove branch from cache key 2021-02-09 21:46:40 -08:00
Dale Hui 70d6a03251 Upgrade dktest from v0.3.3 to v0.3.4
Addresses: https://github.com/golang-migrate/migrate/issues/506
2021-02-09 20:48:31 -08:00
Dale Hui ff92da45f1 Use strconv.ParseBool() to parse the x-multi-statement option in the postgres driver 2021-02-03 22:59:24 -08:00
Dale HuiandGitHub a185b9e78f Merge pull request #495 from AnatolyRugalev/postgresql-support-multistatement
Support multi-statement execution for PostgreSQL
2021-02-02 23:18:28 -08:00
Anatoly RugalevandGitHub 42abb24ae0 Merge branch 'master' into postgresql-support-multistatement 2021-02-02 13:24:47 +03:00
Anatoly Rugalev 38ffbfb778 Addressed review comments 2021-02-02 11:24:33 +01:00
BenandGitHub dab829b564 Add default GitHub API client (#501) 2021-01-30 23:07:53 -08:00
BenandGitHub eb62a386cc sqlite3: Fix error when path to database contains whitespace (#500) 2021-01-28 23:36:56 -08:00
6d8df972da Fix issue when gitlab return only first 20 migrations (reopen) (#497)
* Fix issue when gitlab return only first 20 migrations

* CircleCI issue. repush

* gofmt with -s

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2021-01-28 14:35:14 -08:00
Dale HuiandGitHub 550193e3e0 Fix CircleCI badge link 2021-01-26 00:27:32 -08:00
Dale Hui 5ed79873f7 Update badge to reflect move from TravisCI to CircleCI 2021-01-26 00:23:44 -08:00
Dale Hui 20b5378936 Use CircleCI
Squashed commit of the following:

commit 5f63ff0333a9608530b4bb79b44af837224f80a5
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:36:57 2021 -0800

    Remove unnecessary parameter to checkout

commit b87711057588bfe05fb01d8ff9b679200b2c3015
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:30:37 2021 -0800

    Give up trying to only push coverage for Go 1.15 via YAML templates

commit 6f366c35d11a903fdd8d92738de26073fe7a9c0f
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:27:20 2021 -0800

    YAML template merging only works with maps, not scalars, so specify the current directory for "checkout"

commit e65ff967018508f36e9b9d88f20fe135d8ce6c3e
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 20:21:30 2021 -0800

    Use `go get` instead of `go install` and add template for steps

commit fef2a9e7f9bfbb6070b849199e7d0cced8c6747a
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Jan 25 19:59:24 2021 -0800

    Push code coverage to coveralls.io
        - Only push code coverage for latest support version of Go

commit 48d78bb2576aa8662daaa3e116d0e19dafa77279
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:24:13 2020 -0800

    golangci-lint runs slowly on circle ci machines...

commit d47e04f9aad971a72f169089369ed9e976fbf94f
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:20:35 2020 -0800

    Cleanup setup and remove debugging commands

commit 063337f2faff23b6ca636d77acbd4c504ac6914b
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:17:57 2020 -0800

    try using ~/bin

commit 57f34fbf5ce38ff8fdfdb4be43958d43db01d9f6
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:13:55 2020 -0800

    Install golangci-lint in another dir

commit 2b5d6d65ee4807e3c2df0afffc29498defe5a6c3
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 22:03:08 2020 -0800

    More go env debugging

commit 6719f663d582e59a551bb10ff7365aef814c1d85
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:59:24 2020 -0800

    Sparate downloading and running of golangci-lint.sh

commit 6e9ec9ff0c2d7353b711af5ad685bb4d07ff7292
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:54:44 2020 -0800

    debug gimme env

commit 706a43ca67e012c7982deb115a7168fc2f9067c3
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:47:01 2020 -0800

    debug go env

commit 18694c6419ceeaf0b649939249a618f3876ef530
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:39:53 2020 -0800

    Use .x version of Go

commit 2e4a87b03d1e0b8cf66719bea672304ae008d881
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:36:01 2020 -0800

    correctly use gimme

commit 99a1b838df8832243fc85cdb30d69a07042f9f93
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:31:45 2020 -0800

    Install golangci-lint after installing requested version of Go

commit cecf219e557d5cac26f3f1f43955cfcdb397bb50
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:29:48 2020 -0800

    Still don't support docker layer caching

commit 244ec8fe0b6e5d28cb9c16159bbc3d36ba5ec523
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:28:33 2020 -0800

    Try using machine executor

commit 44d3b29a1e8f953dcf4a2051e0d1037e7b62f85b
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:07:09 2020 -0800

    Fix docker run

commit 525983e81d6a3080e2b28bfdb601581346926450
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 21:03:39 2020 -0800

    properly specify build-arg

commit d5eed206a6b11c2606690ffdee4639e5399cdfc7
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Dec 22 20:54:06 2020 -0800

    Try running CircleCI tests in a Docker container

commit 35e62d1c4ccdeb528495f9d28f26a049bcc93bd8
Merge: 90268de a53e6fc
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:50:58 2020 -0800

    Merge remote-tracking branch 'origin/master' into circleci

commit 90268dedc2cf72ab498ebfd26465551a16463ee7
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:47:46 2020 -0800

    Update golangci lint to v1.33.0

commit d5e0747a53ddfdb7b1ce6ff4bacb236bfc1b7794
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:44:45 2020 -0800

    Don't have access to docker layer caching

commit 6aaecb3b761809b6cf2b4f365f2f9d62f410661d
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:43:52 2020 -0800

    Don't have access to xlarge

commit 275de107de76ced3d6b20e913ffddff84919c3e5
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:39:14 2020 -0800

    CircleCI job names can't have a '.' character

commit e1d339f0eb10519f6002939147201060415a98bb
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Mon Dec 21 20:30:51 2020 -0800

    Update CircleCI config

commit d65194971c028af94e7a3cd171eb6221a08dde05
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Fri Aug 2 00:05:59 2019 -0700

    Add config for CircleCI

commit 19d7355548c2a6d64225ea63368e10e19eb43d19
Author: Dale Hui <dhui@users.noreply.github.com>
Date:   Tue Jul 30 00:12:41 2019 -0700

    Update golangci-lint from v1.16.0 to v1.17.1
2021-01-25 21:08:52 -08:00
Anatoly Rugalev 3a5169ac6d Support multi-statement execution for PostgreSQL 2021-01-18 12:16:37 +03:00
Mitsuo HeijoandGitHub a53e6fc425 fix test on windows (#482)
* fix test on windows

On Windows, the test fails because os.Remove cannot remove unclosed and still-handled files.

* use t.Error instead of t.Log
2020-11-24 22:53:21 -08:00
Dale HuiandGitHub b690b78e22 Merge pull request #481 from johejo/revert-472-iofs_source_driver
Revert "Add io/fs#FS Driver #471"
2020-11-21 21:40:35 -08:00
Mitsuo HeijoandGitHub 67546bfa67 Revert "Add io/fs#FS Driver #471 (#472)"
This reverts commit fedbe42fc8.
2020-11-21 23:27:36 +09:00
Charles MartinotandGitHub 15931649a2 Mysql - Fix crash with ansi_quotes sql_mode enabled (#476)
* Fix crash with ansi_quotes

* Run lint
2020-11-18 12:38:28 -08:00
Mitsuo HeijoandGitHub fedbe42fc8 Add io/fs#FS Driver #471 (#472)
* Add io/fs#FS Driver #471

* Refactor iofs.Iofs to be independent of httpfs

* Rename Iofs to IoFS, accepts fs.FS and use fs.DirEntry.Info() to undo the changes in ErrDuplicateMigration

* fix documents and migration directory for testing

* Fix iofs.Driver so that it can be used by embedding it like httpfs.PartialDriver

* iofs.Driver closes the file system if possible

* Rename receiver

* Refactor file driver to use iofs.Driver

* iofs build constraint is not needed so remove it

* Properly return an error in a corner case

* Increase golangci-lint timeout

* Fix example code

* Revert "Increase golangci-lint timeout"

This reverts commit 2cd12a63b50b7ac48d12dd0adc1ec4f4b49c3b93.

* unexport driver, add iofs.PartialDriver, remove type alias from file driver

* do not panic in Open, return error
2020-11-18 10:06:11 -08:00
Dale Hui 52e7117d37 Use flagset's NArg() and Arg() methods instead of global flag NArg() and Arg() for sub-commands
Fixes: https://github.com/golang-migrate/migrate/issues/467
2020-11-04 23:37:05 -08:00
Yannick HeinrichandGitHub 8a56273871 Add help statement for cli (#460)
* Add help statement for cli

Refers to #372

* Make constant for argument

* Create a flagset for each command

* Factor error handling

* Factor flagset creation

* fix: Typo and bad behaviour

* Fix lint error

* Fix according to the comments

* Fix according to the comments
2020-11-03 14:54:12 -08:00
Dale Hui 110c0f74b4 Update dktest to v0.3.3 which also updates lib/pq to v1.8.0 2020-10-30 22:26:55 -07:00
EvanandGitHub f76b94ab7d Document drop -f in migrate -help (#459)
* Document drop -f in migrate -help

* Document drop -f in migrate -help 2
2020-10-17 01:44:43 -07:00
Vladimir DimitrovandGitHub 91deba8df0 Update driver.go (#454)
* Update driver.go

* addressing comments
2020-10-16 22:19:05 -07:00
Maarten BezemerandGitHub f18b16515a postgres - Force MigrationsTable in 'public' schema (#414)
* Force MigrationsTable on public schema

* Revert "Force MigrationsTable on public schema"

This reverts commit e5b25e8858ed42d19d420cb8bd16795c060994d2.

* Add example to fix search_path issue

* Use POSTGRESQL_URL in example and fix typo

* Make example simpler by using the `search_path` query parameter of the DB URL
2020-10-15 20:21:00 -07:00
Dale HuiandGitHub f8f021730e Merge pull request #458 from xortive/patch-1
Fix typo in doc comment
2020-10-13 09:19:03 -07:00
MattandGitHub a17aa6e83d Fix typo in doc comment 2020-10-13 11:55:41 -04:00
Dale HuiandGitHub 5d84bec1c8 Merge pull request #456 from hypnoglow/iter-err
Check iterator error on Drop where applicable
2020-10-10 20:46:14 -07:00
Igor Zibarev 5f932c05c8 Check iterator error on Drop where applicable 2020-10-10 23:53:20 +03:00
Dale HuiandGitHub 0521ebeb71 Warn about installing the CLI from within a module
Addresses: https://github.com/golang-migrate/migrate/pull/257
2020-10-07 16:51:03 -07:00
Dale Hui 99ca856a7c Run go fix 2020-10-07 15:31:28 -07:00
Dale Hui 04a572114a Parse out custom TLS params and register the TLS config before calling mysql.ParseDSN
Fixes: https://github.com/golang-migrate/migrate/issues/411
2020-10-06 19:49:59 -07:00
Dale Hui 8c531c7836 Use spanner cloud emulator to run tests
Addresses: https://github.com/golang-migrate/migrate/issues/410
2020-10-01 17:58:26 -07:00
Dale Hui fa3e11943a Add local locks for spanner db driver 2020-10-01 17:58:23 -07:00
d6881f1826 Add Bitbucket Cloud Support as source (#450)
* Added bitbucket cloud support as source for migration

* added test

* address PR Comments

* Add entry in README.md; update go.mod and go.sum

Co-authored-by: abhigupta91 <abhigupta0818@gmail.com>
2020-09-28 22:08:20 -07:00
Dale HuiandGitHub c6026053bc Merge pull request #448 from andyNewman42/locking
Add advisory locking to mongodb
2020-09-25 21:05:56 -07:00
Andy Newman 2c2f6910f4 Rename mongo flag for enabling locking. Add validation around parsing mongo params. 2020-09-25 15:02:49 -07:00
Andy Newman 73760ed4cb Implement PR feedback
rename constants
rename query params
add new timeout interval query param
add new feature flag
use timeout context
2020-09-24 15:38:57 -07:00
Andy Newman 7208a16bf3 Add advisory locking to mongodb 2020-09-24 15:38:57 -07:00
Dale HuiandGitHub 50439fee71 Merge pull request #447 from markokoleznik-cs/patch-1
Fix typo
2020-09-17 10:21:45 -07:00
markokoleznik-csandGitHub 5d91d93b7e typo 2020-09-17 07:59:02 +02:00
Dale HuiandGitHub 6819a17049 Merge pull request #439 from antonklava/add-mysql-no-lock
`NoLock` config option for mysql driver
2020-09-09 10:21:01 -07:00
Anton Klava 8a67f7e535 mysql - strconv.ParseBool for x-no-lock parame
`strconv.ParseBool` should ensure that if x-no-lock is set to any
non-booleanic value an error is returned. The parameter still
defaults to false.
2020-09-09 11:57:28 +02:00
Anton Klava aefffb6f6c NoLock config option for mysql driver
Skipping `GET_LOCK`/`RELEASE_LOCK` is useful when running MySQL as
multi-master. This change simply disables the locking.

The internal client lock is left in place as a sanity check.
2020-09-08 09:25:42 +02:00
Dale HuiandGitHub cb28f6bffb Merge pull request #436 from klingtnet/add-sqlcipher-database-driver
Add database driver for SQLCipher
2020-09-02 17:23:05 -07:00
Andreas Linz 7845cb88da Mention SQLCipher in the list of supported databases 2020-09-02 20:48:32 +02:00
Andreas Linz 7129dfa5d9 Add build file for sqlcipher 2020-09-02 20:44:52 +02:00
Andreas Linz 84eea4aaf2 Add sqlcipher to the DATABASE_TEST variable 2020-09-02 20:43:49 +02:00
Andreas Linz 39cb3234a7 Link to sqlite3 driver 2020-09-02 20:43:03 +02:00
Andreas Linz 85d673f95f Add database driver for sqlcipher
This is a copy of the existing sqlite3 driver, except that it imports `_
"github.com/mutecomm/go-sqlcipher/v4"` as database driver.  This allows
to use migrate with encrypted sqlite databases.
2020-09-02 18:29:45 +02:00
Roman LeventovandGitHub f1cf81e851 Include file path in errors from PartialDriver.ReadUp() and ReadDown() (#421)
* Include file path in errors from PartialDriver.ReadUp() and ReadDown()

* Use errors.Is(...) instead of os.Is...() in migrate.go: https://github.com/golang/go/issues/41122
2020-08-29 16:44:38 -07:00
Dale Hui c6ebff6469 Update golangci-lint from v1.27.0 to v1.30.0 2020-08-19 02:04:00 -07:00
Dale Hui c904438dda Add support for Go 1.15 and drop support for Go 1.13 2020-08-19 00:58:47 -07:00
Dale HuiandGitHub ed3f30cf12 Merge pull request #427 from wyardley/wyardley-426
fix(spanner): resolve "want scalar type" error
2020-08-19 00:11:11 -07:00
William Yardley e5eb088eaf fix(spanner): resolve "want scalar type" error
- Update spanner library to cloud.google.com/go/spanner@v1.9.0
This prevents erroring on DDL statements like:

ALTER TABLE users ALTER COLUMN created
  SET OPTIONS (allow_commit_timestamp = true);
https://github.com/googleapis/google-cloud-go/pull/2656

- Add test case which fails with older version of the library, and
succeeds with this one.

Fixes: #426
2020-08-18 22:35:07 -07:00
Danny HermesandGitHub 6035eb5469 Replacing Github with GitHub. (#429)
* Replacing `Github` with `GitHub`.

Done via:

```
$ git grep -l Github | xargs sed -i '' s/Github/GitHub/g
```

* Reverting changes to Go files.
2020-08-17 13:55:44 -07:00
Dale HuiandGitHub ce34d50b16 Merge pull request #428 from wyardley/wyardley-docs
docs(spanner): clarify how to set parameters
2020-08-16 11:05:38 -07:00
William Yardley 0010bc8495 docs(spanner): clarify how to set paramters
- include param in DSN example
- link to README docs on param (option) format
- switch to 'parameter' instead of 'option', in the main README, since
this is preferred
2020-08-16 05:40:20 -07:00
Dale Hui 31c22870d3 Use latest version of neo4j db driver to fix 32-bit builds
Relevant fix: https://github.com/neo4j/neo4j-go-driver/pull/129
2020-08-03 17:32:42 -07:00
Dale Hui 49f67a8ad4 Use make for docker builds so all releases use the same db and source drivers 2020-08-03 16:28:46 -07:00
Dale Hui 23608f1e63 Update to Alpine 3.12 for docker builds 2020-08-03 16:24:32 -07:00
Dale HuiandGitHub d72c200a64 Merge pull request #419 from dawidd6/patch-1
Makefile: provide a simpler build task
2020-07-28 12:39:51 -07:00
Dawid Dziurla ea98a5bc9f Makefile: provide a simpler build task 2020-07-28 18:55:15 +02:00
Dale HuiandGitHub 07052cd704 Merge pull request #417 from orijtech/master
go.mod: upgrade dependencies and rid vulnerable thrift version@v0.12.0
2020-07-26 23:18:33 -07:00
Emmanuel T Odeke a258a44f2a go.mod: upgrade dependencies and rid vulnerable thrift version@v0.12.0
Coming here, after digging through
google/exposure-notifications-server#749
in which the version of github.com/apache/thrift was reported by
@whaber as having known critical vulnerabilities.

Transitively however, this version was pinning to
cloud.google.com/go@v0.37.4 indirectly, which then imported
versions of opencensus:
* go.opencensus.io@v0.19.
* go.opencensus.io@v0.20.1
that imported
    github.com/apache/thrift@v0.12.0

The target package to upgrade was
    github.com/denisenkom/go-mssqldb v0.0.0-20200620013148-b91950f658ec
which now uses
    cloud.google.com/go@v0.61.0
2020-07-25 19:56:06 -07:00
Dale Hui f34bc2cba0 Distribute releases to Ubuntu 20.04 LTS Focal Fossa
Stop distributing release to non-LTS EOL'd Ubuntu 18.10 Cosmic Cuttlefish
Addresses: https://github.com/golang-migrate/migrate/issues/416
2020-07-21 11:43:31 -07:00
Dale Hui f75042ac5d Run go mod tidy 2020-07-20 21:30:45 -07:00
Dale HuiandGitHub bf6c1ec3d3 Merge pull request #415 from arjantop-cai/neo4j-no-cgo
Allow compilation of neo4j support without CGO.
2020-07-20 21:27:20 -07:00
Arjan Topolovec 271087a63b Allow compilation of neo4j support without CGO.
Since 1.8.0 neo4j driver no longer requires CGO.
2020-07-17 14:07:37 +02:00
Dale Hui 53ef02da13 Use multistmt package to parse multi-statement migrations for clickhouse, cassandra, and neo4j
Addresses: https://github.com/golang-migrate/migrate/issues/406
2020-07-07 23:41:00 -07:00
Dale Hui 620492183f Add helper package to parse multi-statement migrations
Addresses: https://github.com/golang-migrate/migrate/issues/406
2020-07-07 22:50:09 -07:00
Dale HuiandGitHub 8eb1d3031c Merge pull request #409 from alethenorio/spanner-comments
Add support for comments in Spanner migrations
2020-06-30 03:48:41 -07:00
Alexandre Thenorio e6faa29d36 Add support for comments in Spanner migrations
At the moment GCP Spanner backend does not seem to support comments
(issue being tracked at
https://issuetracker.google.com/issues/159730604).
By adding support for parsing the migration DDL with spansql we are
able to support comments and remove them before applying on the
database
2020-06-30 09:25:33 +02:00
Richard LindhoutandGitHub 8b6e3f1a2e Fix small typo in GETTING_STARTED.md (#407)
* Update GETTING_STARTED.md

* Update GETTING_STARTED.md
2020-06-22 10:09:05 -07:00
Dmitry IvankovandGitHub 1f973cc3ff Bump spanner lib to v1.5.0 (#403)
* Bump spanner lib to v1.5.0

Will enable spanner emulator support
https://cloud.google.com/spanner/docs/emulator#go

Current version supports SPANNER_EMULATOR_HOST but fails at run-time
with spanner permission errors.

* go mod tidy
2020-06-09 12:34:55 -07:00
Dale HuiandGitHub 64180398eb Add disclaimer to snowflake db driver 2020-06-03 23:16:43 -07:00
Dale HuiandGitHub 79396ffccc Merge pull request #402 from abhinavcohesity/master
Adding support for schema management in snowflake
2020-06-03 23:14:04 -07:00
Abhinav Kumar b00a0cc77b Adding support for schema management in snowflake 2020-06-03 13:55:24 +05:30
Dale HuiandGitHub 7236e82911 Merge pull request #397 from hnnsgstfssn/add-pkger-readme
source/pkger: add readme
2020-05-25 23:15:14 -07:00
Hannes Gustafsson 7b58ded3b3 source/pkger: add readme 2020-05-24 01:32:26 +01:00
Dale Hui 3c7150850e Update golangci-lint from v1.24.0 to v1.27.0 2020-05-21 15:23:17 -07:00
Dale Hui cd9f7bce91 Update neo4j db driver from v1.8.0-beta01 to v1.8.0-beta02
https://github.com/neo4j/neo4j-go-driver/compare/v1.8.0-beta01...v1.8.0-beta02
2020-05-21 13:37:35 -07:00
Dale Hui 805c416251 Use neo4j v1.8.0-beta01 client
The neo4j v1.8 client drops cgo dependencies (seabolt)
2020-05-21 13:32:52 -07:00
Dale HuiandGitHub 371e18dde4 Merge pull request #391 from shiwano/shiwano/mongodb-srv
Register mongodb+srv scheme
2020-05-18 10:45:39 -07:00
Shogo IwanoandGitHub 524edc9f0b Add -f option to drop command (#390)
* Add -y option to drop command

* Replace -y -> -f

* Update -f help text of drop command
2020-05-18 10:42:14 -07:00
Shogo Iwano 306d4577a5 Update README.md 2020-05-18 22:32:26 +09:00
Shogo Iwano 894821e1ef Register mongodb+srv scheme 2020-05-17 10:29:46 +09:00
Dale HuiandGitHub b88b868b91 Merge pull request #386 from KadenLNelson/update-cassandra-ssl-options
Update Cassandra SSL options
2020-05-12 10:19:35 -07:00
Kaden L. Nelson b8ee1b9733 update cassandra ssl options 2020-05-11 19:24:52 -07:00
hnnsgstfssnandGitHub f5a22be0a0 Add pkger source driver support (#377)
* Add pkger source driver support

As go-bindata has been abandoned [1] there are open requests, #116, for
alternative sources with similar functionality. The Buffalo project [2]
created packr and recently pkger [3] was announced [4] with the
intention to supersede packr.

This change adds support for using pkger as a source.

The implementation relies on httpfs.PartialDriver for pretty much all
functionality.

[1] https://github.com/jteeuwen/go-bindata/issues/5
[2] https://gobuffalo.io/
[3] https://github.com/markbates/pkger
[4] https://blog.gobuffalo.io/introducing-pkger-static-file-embedding-in-go-1ce76dc79c65

* pkger: rename Instance to Pkger

* pkger: make WithInstance accept *Pkger

* pkger: refactor and add access to global pkging.Pkger instance

* pkger: fix typo and cleanup debug logging
2020-04-21 16:43:53 -07:00
Dimas Ragil TandGitHub 423c2d35ee Add neo4j driver on params instance (#373)
* Add neo4j driver on instance params

* Check constraint if exist return nil

* Create new driver if nil

* Remove URL and AuthToken from neo4j config

* Change call db constraint

* Add neo4j image 4.0

* Handle create new driver

* Use labels instead constraints

* Turn off TLS encryption on test
2020-04-16 15:16:07 -07:00
Dale Hui 833d00711e Run go mod tidy 2020-04-15 02:41:33 -07:00
Dale HuiandGitHub ace4ca8556 Merge pull request #375 from jmramos02/master
Upgrade multierror to v1.1.0
2020-04-15 02:37:30 -07:00
Jose Mari Ramos 8a7f164b67 Upgrade multierror to v1.1.0 2020-04-15 10:25:13 +08:00
Dale Hui 5180ec47dc [DB-DRIVER][neo4j] Properly handle null ts 2020-04-06 05:09:41 -07:00
Dale Hui 4598673138 Update golangci-lint from v1.23.6 to v1.24.0 2020-04-06 00:07:15 -07:00
Dale Hui 2c5115907f [DB-DRIVER][neo4j] Track timestamp when schema version was last updated 2020-04-06 00:07:15 -07:00
Dale Hui 3d2a99ec64 Correctly mark migrations as dirty when the first down migration fails
Fixes: https://github.com/golang-migrate/migrate/issues/330

Other changes:
  - Cleanup code by using the database.NilVersion constant instead of the magic number -1
2020-04-06 00:07:15 -07:00
Dale HuiandGitHub 20a1dcfe80 Fix command for installing with Go toolchain (versioned)
Thanks pointing out the issue @JKKGBE !
2020-04-03 22:59:35 -07:00
Dale HuiandGitHub 43f6d84de7 Make "force db version" answer more descriptive 2020-03-26 15:01:32 -07:00
Dale HuiandGitHub 00b39b2b6c Merge pull request #361 from mknycha/add-docs-info-about-dirty-database
Docs: add info about troubleshooting dirty database
2020-03-25 18:22:46 -07:00
Marcin K 3a349451e2 Docs: add info about troubleshooting dirty database
[#337]
2020-03-24 19:16:26 +01:00
Dale HuiandGitHub 098a3e7efe Merge pull request #360 from Rocketmakers/drop-prompt
Prompt before dropping the entire database
2020-03-18 12:30:57 -07:00
Nicholas Soper ec26edc735 Give positive feedback when dropping schema 2020-03-18 09:02:00 +00:00
Nicholas Soper 1f66249648 Prompt before dropping the entire database 2020-03-17 17:37:27 +00:00
Dale HuiandGitHub 5cc28f061e Merge pull request #356 from jace-ys/master
Move migrate binary to /usr/local/bin in Dockerfile
2020-03-14 17:24:58 -07:00
jace-ys 6cdebb0871 Move migrate binary to /usr/local/bin in Dockerfile
Move the migrate binary to `/usr/local/bin` in the Dockerfile, so
it can be invoked from anywhere - this makes it easier to write
entrypoint scripts which call `migrate` that can be used
consistently both locally and in Docker containers.

To preserve backwards compatibility for anything that depends on
`migrate` existing at its existing path, we create a symlink to the
`/usr/local/bin/migrate` binary.
2020-03-14 19:21:12 +00:00
8e142dfaa7 Improved migration creation + Windows fix (#352)
* CLI: Improve migration creation.

* Validates migration version on creation, to avoid creation of duplicated versions
* Uses `os.OpenFile` with `O_CREATE|O_EXCL` to create files to avoid file collisions
* Uses `filepath.Join` to concatenate paths, making `cleanPath()` not necessary
* Prints generated filenames
* Fixes #238
* Supersedes #250

* CLI: change `createCmd` to return error and accept `print` parameter

* feat: print out an abs path

Better for Windows OS when specify -dir as /subdir, user will see C:/subdir/0001_name.up.sql rather than /subdir/0001_name.up.sql

* test: fixed abs path test fail for OS Windows

abs path tests fails because filepath.IsAbs() treats `/subdir` path as invalid abs path at windows when drive letter is not present

* feat: print absolute path for created files

Better for Windows OS systems where `/path` can be interpreted in different ways depending on working dir

* test: corrected tests for OS Windows

OS Windows has different interpretation of `/path`, it depends on working dir.
If working dir D:\test it interprets `/path` as `D:\path`

* test: fixed `dir invalid` test

Linux OS has less restriction on a filepath than Windows, so path invalid in windows is perfectly valid for Linux. The only invalid dir name in Linux is one ending with null string terminator (\000)

* refac(cli): *Cmd() now returns an error and not uses log.fatalErr(err)

* docs: added godoc, migarate usage updated

* refac(cli): code refactored

* refac: removed unnecessary path covert

* docs: comment added

* test: fixed code review issue, noErrorExpected var removed

https://github.com/golang-migrate/migrate/pull/352#discussion_r389409789

* docs: fixed godoc

Co-authored-by: Kiyoshi '13k' Murata <kbmurata@gmail.com>
2020-03-10 16:59:11 -07:00
Dale Hui 9b3db6cd7d Fix README markdown lint issues 2020-03-08 15:28:53 -07:00
Dale Hui fb3ffecbb5 Add support for Go 1.14 and drop support for Go 1.12 2020-03-08 15:28:29 -07:00
Dale Hui 1de555b72d Update cockroach db
Fixes: https://github.com/golang-migrate/migrate/issues/308
2020-03-04 08:34:50 -08:00
Dale HuiandGitHub 39cc0970bd Merge pull request #350 from saj/saj/sqlite-no-implicit-tx
sqlite3: Allow users to disable implicit transactions
2020-03-02 20:08:16 -08:00
Saj Goonatilleke 831d8c9ae6 sqlite3: Explicitly reject invalid x-no-tx-wrap values 2020-03-02 22:28:18 +11:00
Saj Goonatilleke d75ded973a sqlite3: Add README 2020-02-29 05:47:13 +11:00
Saj Goonatilleke 583757ab2d sqlite3: Allow users to disable implicit transactions
An sqlite3 database may be opened with the x-no-tx-wrap=true query
parameter if the user would prefer to retain full control over the
bounds of their transactions.  With x-no-tx-wrap enabled, migrations
should contain explicit BEGIN and COMMIT statements.  This (optional)
behaviour for the sqlite3 driver is consistent with the default
behaviour for other drivers.

Fixes #346.
2020-02-29 05:14:59 +11:00
Dale HuiandGitHub 6dc95bd201 Merge pull request #348 from martinlindhe/master
Add Windows install instructions using scoop
2020-02-25 08:59:46 -08:00
Martin Lindhe b4872fb19d Add Windows install instructions using scoop 2020-02-25 11:45:36 +01:00
Dale Hui 777550e4d7 Actually release windows 386 build 2020-02-17 08:46:08 -08:00
Dale HuiandGitHub 51f9aabf68 Merge pull request #343 from martinlindhe/win386
build-cli: Also build GOOS=windows GOARCH=386, fixes #339
2020-02-17 02:23:11 -08:00
Dale HuiandGitHub cbadf63c13 Merge branch 'master' into win386 2020-02-16 22:53:07 -08:00
Dale Hui d9d6d85ecd POSTGRES_PASSWORD is now required when using postgres via docker
See: https://github.com/docker-library/postgres/issues/681
2020-02-16 01:37:09 -08:00
Dale Hui 9dcc6f061f Drop support for postgres 9.4 and add support for postgres 12
https://www.postgresql.org/about/news/2011/
https://www.postgresql.org/about/news/1976/
2020-02-16 00:38:23 -08:00
Martin Lindhe 115177e679 build-cli: Also build GOOS=windows GOARCH=386, fixes #339 2020-02-16 05:11:07 +01:00
Dale Hui 67536b7f1c Update github.com/fsouza/fake-gcs-server from v1.7.0 to v1.17.0 2020-02-14 01:14:05 -08:00
Dale Hui 9e6ca5ccaa Update cloud.google.com/go from v0.37.4 to v0.53.0 2020-02-14 00:44:21 -08:00
Dale Hui 7291b60643 Update golangci-lint from v1.22.2 to v1.23.6 2020-02-14 00:08:02 -08:00
Dale Hui 64db556342 Update dktest from v0.3.1 to v0.3.2
Addresses: https://github.com/golang-migrate/migrate/issues/338
2020-02-13 23:35:01 -08:00
Dale HuiandGitHub 607b1142fa Merge pull request #336 from Glyphack/add-errnochange-check-up
Add ErrNoChange check when using Up.
2020-02-12 00:44:59 -08:00
glyphack 22b2dbf747 Add ErrNoChange check when using Up. https://github.com/golang-migrate/migrate/issues/100 2020-02-12 00:55:09 +03:30
Dale HuiandGitHub e1085cfef2 Merge pull request #334 from bishtawi/bishtawi/aws-session-override
Allow passing in your own aws s3client object
2020-02-04 18:30:44 -08:00
Stephen Bishtawi 1637f18132 Cleanup 2020-02-04 17:58:29 -08:00
Stephen Bishtawi 453547cf81 Cleanup 2020-02-04 09:40:17 -08:00
Stephen Bishtawi a50ea56eda Cleanup 2020-02-03 12:32:27 -08:00
Dale HuiandGitHub 402f080149 Merge pull request #335 from bcomnes/patch-1
Fix URL in tutorial
2020-01-31 15:48:44 -08:00
Bret ComnesandGitHub a74470d619 Fix URL in tutorial
This link needed to be updated.
2020-01-31 14:13:30 -07:00
Stephen Bishtawi df9067fec5 Update 2020-01-31 10:56:16 -08:00
Stephen Bishtawi 5d38bead9a Implement WithInstance pattern 2020-01-31 10:24:31 -08:00
Stephen Bishtawi c2d3db8c2b More testing 2020-01-30 13:00:45 -08:00
Stephen Bishtawi 0f10e74f69 Add test 2020-01-30 11:09:08 -08:00
Stephen Bishtawi 459f15b500 Fix when prefix is empty 2020-01-29 17:31:18 -08:00
Stephen Bishtawi 5439c3a702 Allow using a custom aws session 2020-01-29 14:57:07 -08:00
Mantas VidutisandDale Hui 69093192ef Allow for multistatement migrations in Neo4j (#328)
* allow for multistatement migrations

* multi statement option

* parse bool and package const

* default false multi, change package var to byte[]
2020-01-22 23:59:32 -08:00
f29923306d add statement timeout query param for postgres (#327)
* add statement timeout query param for postgres

* Update database/postgres/postgres.go

Co-Authored-By: Roopak Venkatakrishnan <roopak.v@gmail.com>

* add enable flag

* remove deafult catch errors

* reformat

* only use background context for 0 timeout

* refactor

* refactor

Co-authored-by: Pranay Suresh <53918678+psuresh309@users.noreply.github.com>
Co-authored-by: Roopak Venkatakrishnan <roopak.v@gmail.com>
2020-01-21 11:22:41 -08:00
Dale Hui 8b62949069 Update go-sql-driver from v1.4.1 to v1.5.0 2020-01-20 22:34:00 -08:00
Dale Hui 8346ecefea Update pq from v1.0.0 to v1.3.0 2020-01-20 22:33:01 -08:00
Mantas VidutisandDale Hui 4ff83b9d74 Fix issues with Neo driver (#324)
* initial pass

* new cli build file

* add ready function for docker start

* lowercase j

* explicit return

* test migration

* undo dockerfile change

* neo4j in dockerfile

* remove auth from url once transferred to auth token

* assorted review notes

* initial docs attempt

* unlock behavior more in line with lock behavior

* consistent naming

* linting

* seabolt in travis ci

* stdlib logger

* install seabolt as sudo

* enabled cgo

* correct version #

* remove neo4j todo

* named return values for errors

* review notes

* add libssl for alpine

* updates for review

* updates for neo driver

* go mod tidy
2020-01-19 18:56:51 -08:00
Dale Hui dde480b416 Reference seabolt in neo4j db driver README 2020-01-14 22:36:26 -08:00
Dale Hui 57c0e0d7c0 Rename "downloader" Docker build stage to "builder" 2020-01-14 22:24:00 -08:00
Dale Hui 9ca70b31ef Split up Go module fetching in Docker builds to take advantage of Docker build cache 2020-01-14 22:24:00 -08:00
Dale Hui 07bed0e214 Shallow clone seabolt repo 2020-01-14 22:24:00 -08:00
Dale Hui 3de5c9af24 Support SQLite in Docker CLI 2020-01-14 22:24:00 -08:00
Dale Hui c32255c13d Fix broken Docker CLI
Need to copy over seabolt shared/static library
2020-01-14 22:24:00 -08:00
Dale Hui 407d0cec58 Run go mod tidy 2020-01-14 22:24:00 -08:00
Mantas VidutisandDale Hui 3061f4564a Neo4J Support (#320)
* initial pass

* new cli build file

* add ready function for docker start

* lowercase j

* explicit return

* test migration

* undo dockerfile change

* neo4j in dockerfile

* remove auth from url once transferred to auth token

* assorted review notes

* initial docs attempt

* unlock behavior more in line with lock behavior

* consistent naming

* linting

* seabolt in travis ci

* stdlib logger

* install seabolt as sudo

* enabled cgo

* correct version #

* remove neo4j todo

* named return values for errors

* review notes

* add libssl for alpine

* updates for review
2020-01-14 21:59:22 -08:00
Dale Hui 9e2a0cdc38 Use new ql driver location 2020-01-03 02:59:33 -08:00
Dale Hui 49afea4d86 Don't pass params to SQL statements using fmt.Sprintf() 2020-01-03 02:59:33 -08:00
Dale Hui 4e593fd2a9 Remove unnecessary byte slice arg from TestMigrate() in database/testing 2020-01-03 00:15:52 -08:00
Oras Al-KubaisiandDale Hui 3001e743c3 More examples in Mongo (#268)
* adding/removing a new field in the schema

* example of updating field value from other fields

* added new line at end of files

Co-authored-by: Dale Hui <dhui@users.noreply.github.com>
2020-01-03 00:12:08 -08:00
Dale Hui f6d44b4b24 Use Golang 1.13 and Alpine Linux 3.11 to build standard Docker images 2020-01-02 22:49:04 -08:00
Dale HuiandGitHub 0eedf49c28 Merge pull request #297 from mknycha/add-tutorial-for-cockroachdb
Add a tutorial for cockroachdb
2020-01-02 21:22:25 -08:00
Marcin K d993b8eaee Prevent shell interpretation in the exported variables 2020-01-02 20:17:56 +01:00
Dale Hui 4146594df3 Rename httpfs package files for improved readability 2020-01-02 00:10:22 -08:00
Dale Hui 8261b52d33 Use httpfs.PartialDriver as main godoc_vfs source implementation 2020-01-02 00:10:22 -08:00
Dale Hui a4820755ab Use httpfs.PartialDriver as main file source implementation 2020-01-01 16:26:24 -08:00
Dale Hui c27f930c6e Update golangci-lint from v1.20.0 to v1.22.2 2019-12-31 19:24:20 -08:00
Dale HuiandGitHub 6355956008 Merge pull request #316 from pcarranza/pc/add-arm-binaries
Add arm binaries to the build-cli phase
2019-12-31 19:10:11 -08:00
Dale HuiandGitHub 2f4bfa29e5 Merge branch 'master' into pc/add-arm-binaries 2019-12-31 17:39:25 -08:00
Dale Hui 4100c1d952 Update dktest from v0.3.0 to v0.3.1 2019-12-31 17:11:17 -08:00
Dale Hui 7965ee37ab Fix failing sqlserver tests by using latest images
Don't need to specify Ubuntu in tag since all mcr.microsoft.com/mssql/server images are based on Ubuntu anyways
2019-12-31 17:10:49 -08:00
Pablo Carranza fb317bb597 Add arm binaries to the build-cli phase 2019-12-29 11:18:51 +01:00
Dale Hui 6b1121a658 Tweak source.httpfs README 2019-12-24 15:38:36 -08:00
Dale Hui 525949de9f Add additional information in source.ErrDuplicateMigration struct 2019-12-24 15:35:08 -08:00
Dale HuiandGitHub 09f532b773 Merge pull request #293 from fln/master
Add http.FileSystem migration source driver support
2019-12-24 15:30:35 -08:00
Julius Kriukas dc7fa068e3 Drop errors.Is() usage to support go 1.12.x 2019-12-19 15:46:12 +02:00
Julius Kriukas e57ac62141 Rename PartialDriver receiver variable to p 2019-12-19 15:36:14 +02:00
Julius Kriukas 516bfe1c27 Drop delayed error reporting 2019-12-19 15:30:00 +02:00
Julius Kriukas 0491d12e50 Rename Migrator -> PartialDriver 2019-12-19 15:13:02 +02:00
Julius Kriukas b7e05731ea Add new test with example driver implemenation 2019-12-16 12:01:52 +02:00
Julius Kriukas 0bea9205d9 Improve driver.Open() error message 2019-12-16 11:06:52 +02:00
Julius Kriukas bf817e5de6 Fix README.md and documentation typo 2019-12-16 10:15:28 +02:00
Julius Kriukas bfa4bdf7a0 Split New() tests 2019-12-16 10:11:31 +02:00
Julius Kriukas 9e851de675 Split source and tests into two files to match data types 2019-12-13 17:41:47 +02:00
Julius Kriukas 98a84cdaec Fix documentation 2019-12-13 17:31:06 +02:00
Julius Kriukas 46206b7330 Split Migrator and Driver tests 2019-12-13 02:09:14 +02:00
Julius Kriukas cf23a7307e Split Migrator and Driver structures 2019-12-13 02:01:24 +02:00
Julius Kriukas 515a6d1664 Move ErrDuplicateMigration to the source package 2019-12-13 01:04:06 +02:00
Julius Kriukas 1fc246e82d Prefer field mutation over struct replacement 2019-12-13 00:59:23 +02:00
Julius Kriukas 33cf17647b Use string concatenation instead of fmt.Sprintf() 2019-12-13 00:20:02 +02:00
Julius Kriukas df5e294c5c Simplify not test.ok to else 2019-12-13 00:13:23 +02:00
Julius Kriukas 29df549a5b Add ErrDuplicateMigration typed error 2019-12-12 16:49:30 +02:00
Julius Kriukas 49ae0330b0 Test for error on First() when no migration files are present 2019-12-12 16:13:43 +02:00
Julius Kriukas 20e6881fd8 Cover subdir ignore code path 2019-12-12 15:42:56 +02:00
Julius Kriukas 1ed56cb468 Use external package for tests 2019-12-12 15:36:24 +02:00
Julius Kriukas 73ba745097 Replace WithInstance with Init, make main struct public 2019-12-12 12:06:18 +02:00
Julius Kriukas 7b6a813149 Replace test messages with sub-test names 2019-12-12 11:00:26 +02:00
Julius Kriukas 9c45ae5db0 Switch to if ok pattern for ReadUp and ReadDown 2019-12-12 10:36:01 +02:00
Julius Kriukas 9c9a49920d Drop coverage.out 2019-12-11 17:51:18 +02:00
Julius KriukasandJulius Kriukas 41dfe1d575 Add httpfs.New() constructor with delayed errors
This PR adds a new httpfs source driver constructor New(). It is
identical to the WithInstance() constructor but will postpone any errors
until the driver is actually being used.

This allows clients of this package to have less error checks without
loosing correctness.

In addition this PR adds more tests, improves test code coverage, and
makes golinter happy by removing deferred Close call.
2019-12-06 15:55:07 +02:00
Julius KriukasandJulius Kriukas b5bd00dc8a Add http.FileSystem migration source driver support
This PR adds a new migration source driver capable of reading files from
any source that implements http.FileSystem interface.

Notable http.FileSystem interface implementations:

* http.Dir() - wrapper over local file-system
* github.com/shurcooL/vfsgen

Because user of this package is responsible for getting an
implementation of http.FileSystem, this driver does not support creating
instances from driver URLs.
2019-12-06 15:55:07 +02:00
Marcin K 89b51584c5 Add a reference to the tutorial in README 2019-11-29 08:00:00 +01:00
Marcin K 63f26f6fb9 Add a simple tutorial for cockroachdb 2019-11-29 08:00:00 +01:00
Dale HuiandGitHub 41b578a45e Merge pull request #305 from bxcodec/fixClickHouse
hotfix(dependency): resolve clickhouse dependency breaking changes
2019-11-21 00:26:34 -08:00
Iman Tumorang 9d1abc92fa fix(dependency): resolve clickhouse dependency breaking changes 2019-11-21 10:18:48 +08:00
Dale Hui ae62039386 Add official support for firebird back 2019-10-17 08:15:47 -07:00
Dale Hui a7c2a28bec Deploy using Go 1.13 2019-10-17 07:53:29 -07:00
Dale Hui dc9b4dab5c Only populate db config values if it's not specified
See also: https://github.com/golang-migrate/migrate/issues/262
2019-10-17 07:48:52 -07:00
Dale HuiandGitHub 7f20f01a3b Merge pull request #288 from jon-whit/master
Trim statement leading and trailing whitespace.
2019-10-14 20:53:54 -07:00
Dale Hui 57b2ba7115 Actually use schema and db names specified in postgres db config
Addresses: https://github.com/golang-migrate/migrate/issues/262
2019-10-13 23:24:10 -07:00
Dale Hui bbe4ec3cde Update golangci-lint to v1.20.0 2019-10-12 15:14:27 -07:00
Dale Hui 84f30c64d0 Specify the expected min go version as 1.12 2019-10-12 15:14:21 -07:00
Jonathan Whitaker 241ad96665 Trim statement leading and trailing whitespace.
The changes herein make sure Spanner DDL statements have leading and
trailing whitespace characters trimmed.
2019-10-03 14:35:37 -06:00
Dale HuiandGitHub fc060940b4 Merge pull request #287 from ctuong/pg-lock-comment
Update postgres driver Lock comment to reflect actual behavior
2019-09-25 00:57:45 -07:00
Calvin Tuong 9ef241d273 Update postgres driver Lock comment to reflect actual behavior 2019-09-24 13:57:01 -07:00
Dale Hui 002e0aced4 Update golangci-lint to v1.18.0 for Go 1.13 compatibility 2019-09-17 02:21:45 -07:00
Dale Hui be590bf1b9 Drop support for Go 1.11 and add support for Go 1.13 2019-09-09 08:00:12 -07:00
Peter DotchevandDale Hui 3dc81826fd Improve error message in case no migration is found (#158)
* Improve error message in case no migration is found

* Return similar error

* Just log the error

* Fix error message
2019-08-29 20:09:02 -07:00
Dale HuiandGitHub b6df2cab18 Merge pull request #277 from ch3rub1m/bugfix-remove-debug-output
Remove debug output
2019-08-29 00:45:16 -07:00
ch3rub1m 0601111d88 Remove debug output 2019-08-28 18:36:44 +08:00
Dale Hui 0064ee83cf Properly filter out custom query params in MySQL DB driver
Addresses: https://github.com/golang-migrate/migrate/issues/272
2019-08-22 00:03:26 -07:00
Dale Hui a354c6d446 Run go mod tidy 2019-08-22 00:00:09 -07:00
Erik DubbelboerandDale Hui e5b4be7771 Let database.Open() use schemeFromURL as well (#271)
* Let database.Open() use schemeFromURL as well

Otherwise it will fail on MySQL DSNs.

Moved schemeFromURL into the database package. Also removed databaseSchemeFromURL
and sourceSchemeFromURL as they were just calling schemeFromURL.

Fixes https://github.com/golang-migrate/migrate/pull/265#issuecomment-522301237

* Moved url functions into internal/url

Also merged the test cases.

* Add some database tests to improve coverage

* Fix suggestions
2019-08-20 09:59:15 -07:00
Sebastiaan van StijnandDale Hui d5960ade4a Docker: restore "from env" behavior, and perform API version negotiation (#270)
* NewDockerContainer: restore "from env" behavior

commit c31948caeb replaced the deprecated
`dockerclient.NewEnvClient` with `dockerclient.NewClientWithOpts`, but
did not add the `FromEnv` option to keep  the old behavior.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

* NewDockerContainer: use API version negotiation

This helps in situations where the daemon is older than
the API client. API version negotiation checks the maximum
supported API version by the daemon, and downgrades to that
API version if needed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-20 08:42:07 -07:00
Dale Hui d068374972 Test against MS SQL Server 2019-CTP3.1 (Ubuntu) 2019-08-18 23:37:33 -07:00
Dale Hui 947936279a Update firebird README to reflect lack of official support 2019-08-18 23:34:07 -07:00
Dale Hui 42492c82ca Update golangci-lint from v1.16.0 to v1.17.1 2019-08-17 10:50:42 -07:00
Dale HuiandGitHub 97cfb4d2a6 Merge pull request #267 from zikaeroh/fix-mongodb-dep
Update MongoDB driver to 1.1.0, use correct import path
2019-08-17 10:20:45 -07:00
zikaeroh d80e0e2f7f Merge branch 'master' into fix-mongodb-dep 2019-08-17 08:13:16 -07:00
Erik DubbelboerandDale Hui eb7d0dd6a1 Fix in the URL parser with go 1.12.8 and github.com/go-sql-driver/mysql (#265)
* Fix in the URL parser with go 1.12.8 and github.com/go-sql-driver/mysql

Change schemeFromURL to just split the url by :// to find the scheme.
It's not required to parse the whole URL. MySQL DSNs aren't valid URLs.

Fixes #264

* The mysql driver itself also used net/url.Parse

* Also fix TestPasswordUnencodedReservedURLChars

* Keep backwards compatibility with url encoded username and passwords

* Fix suggestions

* Reuse old function names
2019-08-16 23:09:12 -07:00
zikaeroh 9e406c40b5 Update MongoDB driver to 1.1.0, use correct import path 2019-08-14 17:40:52 -07:00
Dale HuiandGitHub b071731cc2 Merge pull request #258 from mknycha/add-tutorial-for-getting-started
Add getting started docs and Postgres tutorial
2019-08-13 08:31:01 -07:00
Marcin K d3ec78d996 Use different word 2019-08-13 16:40:03 +02:00
Marcin K 98c383cc42 Update info regarding migrations inconsistency 2019-08-13 16:38:16 +02:00
Marcin K 12f7e22e36 Put tutorial specific info into a separate file 2019-08-11 22:06:04 +02:00
Dale HuiandGitHub 6c7d3123c2 Merge pull request #260 from groyoh/exit_status
Change exit status to 2 on command error
2019-08-10 02:16:59 -07:00
Marcin K 86e4e4965d FAQ: Add question regarding non-Go project usage 2019-08-09 18:50:24 +02:00
Marcin K d0c2e446b7 Remove hint regarding Makefile
As it is too much environment-specific
2019-08-09 18:46:57 +02:00
Marcin K bb1e9dd3ae Describe database transactions 2019-08-09 18:46:04 +02:00
Yohan Robert dc5b8a46c6 Add documentation for exit 2 2019-08-09 15:15:29 +02:00
Marcin K d90f7d51aa Cosmetics 2019-08-08 21:03:54 +02:00
Marcin K d6fa9be6e6 Add a caveat regarding DB lock when running migrations 2019-08-08 21:03:00 +02:00
Marcin K fb4fb978e1 Use idempotent queries 2019-08-08 20:34:21 +02:00
Marcin K 604f1254ba Use sequential migrations naming in example
And add migration conflicts description
2019-08-08 20:11:05 +02:00
Marcin K b49854bde5 TUTORIAL: Use password explicitly 2019-08-07 22:36:20 +02:00
Marcin K fb771c8dd8 Rephrase, list created files 2019-08-07 22:35:22 +02:00
Marcin K 44b07c5f7b Use reverse/down migration instead of rollback 2019-08-07 22:01:04 +02:00
Marcin K e98f9f6a24 FAQ: Remove driver specific info 2019-08-07 21:52:15 +02:00
Yohan Robert 95623b991f Change exit status to 2 on command error
The default exit status when a command cannot be found was 0.
This can cause migrate to fail "silently" in case of typos.
Given that providing an invalid flag causes the program to exit
with a exit status set to 2, the default exit status was also set to 2.
2019-08-07 20:13:16 +02:00
Marcin K 1d402e46a8 Tutorial: Remove info about migration version table 2019-08-07 01:09:26 +02:00
Marcin K d7e0e40e0a FAQ: Add one more question 2019-08-07 01:07:29 +02:00
Marcin K 74b9fdb3cb Add reference to README 2019-08-07 01:01:15 +02:00
Marcin K 01107cb677 Add tutorial file 2019-08-07 01:00:58 +02:00
Dale Hui e93eaeb3fe Another attempt at fixing builds
Addresses: https://github.com/golang-migrate/migrate/issues/254
2019-07-30 00:06:35 -07:00
Dale Hui b55570703d Fix broken builds
Addresses: https://github.com/golang-migrate/migrate/issues/254
2019-07-29 23:59:39 -07:00
Dale HuiandGitHub fd16003059 Add Go Report Card badge 2019-07-24 22:58:07 -07:00
Dale Hui 71c6afaf98 Reduce docker binary size by stripping symbols 2019-07-12 10:54:42 -07:00
Dale Hui bb99610517 Update Docker image to use Alpine 3.10 2019-07-11 23:22:30 -07:00
Dale HuiandGitHub ff1977cbd4 Update migration docs
Correct and clarify information around empty migrations
Addresses: https://github.com/golang-migrate/migrate/issues/244
2019-07-11 23:05:33 -07:00
Dale HuiandGitHub 14a3bcdaa8 Merge pull request #236 from KlotzAndrew/migrate-down-less-destructive
migrate down less destructive
2019-06-26 10:55:06 -07:00
Andrew Klotz ffea024e7c migrate down less destructive
default behaviour for down is to apply all down migrations, which is
comparable to dropping a database - and usually not the desired default
action

proposed changes:

 * `down` prompts for a confirm `y` before applying all down migrations, defaulting to doing nothing
 * `down --all` does the current behaviour, applying all down migrations
 * `down N` is unchanged

 * `down N --all` errors
 * `down --all N` errors
2019-06-26 00:00:54 -04:00
Dale Hui d4e8e9ac07 Deprecate migrate's MultiError in favor of github.com/hashicorp/go-multierror 2019-06-24 08:58:28 -07:00
Dale HuiandGitHub 5399892051 Merge pull request #240 from k-yomo/fix/typo
Fix typo in README for CLI
2019-06-23 22:08:45 -07:00
k-yomo 8d9d92b7b1 Fix typo in README for CLI 2019-06-23 18:46:05 +09:00
Dale Hui cde3cfc30d Fix broken sqlite tests
Since v4.3.0, Migrate.Drop() no longer recreates the migrate schema version table.
See: https://github.com/golang-migrate/migrate/releases/tag/v4.3.0
2019-06-21 00:15:01 -07:00
Dale Hui 481bf0fb03 Run tests so that coverage profiles don't need to be manually concatenated
Addresses: https://github.com/golang-migrate/migrate/issues/2
2019-06-20 23:07:47 -07:00
Dale Hui 9f6c7e5fc9 Properly clean/normalize user specified directory for create command
Addresses: https://github.com/golang-migrate/migrate/issues/238
2019-06-20 21:35:51 -07:00
Kasparas GaldikasandDale Hui 7c76166697 Github Enterprise support (#234)
* exported Github struct fields and ReadDirectory method

* github ee implementation, tests and docs

* build fixes

* Github Enterprise API endpoint based on docs

* addressing PR comments

* code review

* make linter happy

* parseBool() takes fallback

* pr comments

* tweaks to Config{}
2019-06-16 12:48:01 -07:00
Dale Hui 0d13e794e4 Clean up CLI docs
- Move main CLI docs from cli to cmd/migrate
    - Clean up markdown
    - Update example CLI usage in Docker
2019-06-13 01:30:58 -07:00
Dale HuiandGitHub 8437fe6dc6 Merge pull request #227 from zhevron/mssql-params
Fix incorrect syntax errors for MSSQL driver
2019-05-27 10:25:36 -07:00
Thomas Lokshall dd0ead011e update Dockerfile referencing renamed mssql driver 2019-05-27 08:20:47 +02:00
Thomas Lokshall ffbd6893f6 sqlserver: update comment referencing mssql 2019-05-27 08:20:35 +02:00
Thomas Lokshall bad962cf21 Merge branch 'mssql-params' of https://github.com/zhevron/migrate into mssql-params 2019-05-24 15:16:24 +02:00
Thomas Lokshall 293bfec844 rename mssql driver to sqlserver 2019-05-24 15:16:12 +02:00
Thomas LokshallandGitHub bc0b686fc0 Merge branch 'master' into mssql-params 2019-05-23 11:12:50 +02:00
Thomas Lokshall e08ae0e996 mssql: remove mssql registration, document only support for sqlserver driver 2019-05-23 09:43:13 +02:00
Alex PliutauandDale Hui e877644ed1 Fix MongoDB docs: x-migrations-collection (#225) 2019-05-22 18:29:44 -07:00
Dale HuiandGitHub 5ed8a441d4 Merge pull request #229 from aryzing/patch-1
Typo
2019-05-22 18:25:41 -07:00
Eduard Bardají PuigandGitHub 6ff5b4ed4c Typo 2019-05-22 20:25:18 +01:00
Thomas Lokshall 3d4dad5655 mssql: use sqlserver driver instead of deprecated mssql driver 2019-05-22 14:10:37 +02:00
Thomas Lokshall ec3ae16f2e mssql: fix sp_MSforeachtable parameter 2019-05-22 14:09:50 +02:00
Thomas Lokshall 9f282f4ee3 use correct mssql parameter syntax 2019-05-22 10:34:53 +02:00
Ferdy PruisandDale Hui e85c5f51b9 Disable foreign_key_checks in Drop command for MySQL (#224)
* Disable FOREIGN_KEY_CHECKS in MySQL when dropping all tables.

* Lowercased system variable

* Discard error enabling foreign_key_checks, dropping is already successful at this point

* Explicitly discard error
2019-05-21 10:56:37 -07:00
Dale HuiandGitHub 2327ddb52d Fix reference to MS SQL Server support in README 2019-05-21 10:54:54 -07:00
Dale HuiandGitHub 7b3cd164d7 Merge pull request #222 from nathan-c/master
Add MS SQL Server Support
2019-05-21 10:52:41 -07:00
Nathan Collard aeb7f633f3 mssql: fix code review comments 2019-05-21 08:44:17 +01:00
Nathan Collard 7817573893 Merge remote-tracking branch 'root/master' 2019-05-21 08:21:33 +01:00
Dale Hui 923901c81f Cleanup regex usage in spanner DB driver
- Only compile a regex once for re-use
    - Don't use regex unnecessarily (removing trailing semicolon)
    - Add test for parsing multiple statements in a migration
2019-05-20 08:28:07 -07:00
Dale Hui bd81e32d1a Mark testing package as deprecated 2019-05-19 16:52:41 -07:00
nathan-c e211c5b5f5 mssql: increase pull timeout 2019-05-19 21:36:50 +01:00
nathan-c 98e5f88b9f mssql: fix error parsing and add tests 2019-05-19 16:08:15 +01:00
nathan-c 5ac583ba7b mssql: fix linter errors 2019-05-19 15:21:25 +01:00
nathan-c 3bd91a774e mssql: reorder readme 2019-05-19 15:15:07 +01:00
nathan-c 95290f2fbd mssql: change migration table name and add readme 2019-05-19 15:09:56 +01:00
nathan-c 14e6c50d2e fix mssql build flag 2019-05-19 14:47:09 +01:00
nathan-c c273636efe parse mssql errors 2019-05-19 14:08:10 +01:00
nathan-c 2fbec2f760 update mssql driver module 2019-05-19 13:43:47 +01:00
nathan-c ae500dbe45 finish implementing mssql db and add tests 2019-05-19 13:37:40 +01:00
nathan-c 50e972912d Merge remote-tracking branch 'evosure/mssql' 2019-05-18 14:39:55 +01:00
Dale HuiandGitHub e364488761 Merge pull request #220 from Flaque/patch-1
Fix Broken Markdown in cli/README.md
2019-05-16 12:15:54 -07:00
Evan ConradandGitHub c7e9414bfe Fix Broken Markdown in cli/README.md 2019-05-16 11:30:01 -07:00
Dale HuiandGitHub f2e5e31837 Remove offical Firebird support from README 2019-04-26 22:52:56 -07:00
Dale Hui 10e2545ceb Update dependencies to resolve: https://github.com/golang-migrate/migrate/issues/202 2019-04-26 19:09:13 -07:00
Dale Hui 89970b8951 Bring built-in and supported sources and databases into parity 2019-04-26 19:04:37 -07:00
Dale Hui 66510b82c4 Remove official support for Firebird
- In the future, support for Firebird should also include v2.5
2019-04-26 19:04:05 -07:00
Kirill MuratovandDale Hui 6c96ef02df Enabled maligned, staticcheck and errcheck linters (#206)
* Enabled maligned check

* Enabled staticcheck

* Fixes for golint

* Enabled errcheck linter

* Added fixes for error check

* Added errcheck for tests

* Fixed test

* Increased golangci-lint deadline for travis

* Increased golangci-lint deadline for travis

* Decreased golangci-lint deadline for travis

* Revert for backward compatibility

* Using log.Println() instead of fmt.Println()

* Handling os.RemoveAll() errors

* Using t.Error(error) instead of t.Errorf("%v", err)

* Using t.Fatal(error) instead of t.Fatalf("%v", err)

* Using fmt.Sprint(sum) instead of t.Srintf("%v", sum)

* Refactoring

* Revert for backward compatibility

* Revert

* go mod tidy

* Added error logging

* Added error logging

* Added error handling

* Added error handling

* Added error logging

* Fix error logging

* Added error handling

* Fix

* Added logging for migr.Buffer()

* Fixes

* Firebird test disabled

* Fixed nolint comment

* Updated firebird docker image version

* Disabled test for firebird 2.5

* Fixed // nolint
2019-04-26 15:47:16 -07:00
Dale Hui ddc72468fc Update packages that depend on the wrong golint import path
Addresses: https://github.com/golang-migrate/migrate/issues/202
2019-04-09 00:31:17 -07:00
Dale Hui 16197b67e3 Update to golangci-lint v1.16.0
- Fix newly raised issues
2019-04-07 00:51:27 -07:00
Dale Hui 6e2056f5b4 Include firebird db driver in default builds 2019-04-05 23:25:57 -07:00
Dale Hui 8d63b3f95b Remove spurious list item 2019-03-28 21:35:54 -07:00
Dale Hui 43e4503343 Update golangci-lint config and implore contributors to use it
- Run golangci-lint on tests and fix found issues
2019-03-28 18:55:12 -07:00
Dale HuiandGitHub c1404f0f98 Merge pull request #194 from kmuratov/master
Enabled golangci-lint
2019-03-28 11:57:10 -07:00
Kirill Muratov 46ee425bac Updated golangci.yml 2019-03-28 19:30:02 +03:00
Kirill Muratov 81f457f379 Added newline 2019-03-26 22:05:47 +03:00
Kirill Muratov 07c594d988 Added scopelint check 2019-03-26 22:01:12 +03:00
Kirill Muratov be4c129168 Added interfacer, unconvert, goconst, nakedret, prealloc 2019-03-23 00:40:35 +03:00
Kirill Muratov ef4a805b02 Added misspell linter 2019-03-22 23:57:00 +03:00
Kirill Muratov 5059f0bee6 Fixes for govet 2019-03-22 23:05:19 +03:00
Kirill Muratov c31948caeb Fixed dockerclient.NewEnvClient is depricated 2019-03-22 22:47:24 +03:00
Kirill Muratov 2758eb210f Fixes for unused linter 2019-03-22 22:08:29 +03:00
Kirill Muratov 465060c9a7 Fixes for gosimple linter 2019-03-22 22:02:08 +03:00
Kirill Muratov 8f6bfb0295 Added golangci-lint config 2019-03-22 21:11:31 +03:00
Dale HuiandGitHub cac03318eb List Gitlab as a supported source 2019-03-21 01:54:59 -07:00
Dale HuiandGitHub 7bf5fd7859 Add missing backtick 2019-03-21 01:54:41 -07:00
Kirill MuratovandDale Hui 2c9773ed89 Added Firebird support (#191)
* Added Firebird support

* Fixed typo

* Refactoring

* Schema migrations table name don't have to be upper case

* Fixed readme

* Added Firebird 2.5 support

* Removed SchemaName

* Refactoring
2019-03-21 01:51:49 -07:00
Cyrille HemidyandDale Hui 46fb82ce3d Fmt (#189)
* fix gofmt -s

* fix gofmt -s + fix mispelling
2019-03-13 12:15:21 -07:00
Bence PatyiandDale Hui 144aaaa986 Gitlab source (#184)
* New Gitlab source

* fix: use net package to build url string

* add: gitlab source to Makefile and Dockerfile
2019-03-10 18:38:23 -07:00
Dale HuiandGitHub 14f26d8818 Update supported Go versions badge
Drop 1.10 and add 1.12
2019-03-07 11:42:22 -08:00
Dale Hui f8f7beea54 Deploy using Go 1.12 2019-03-02 02:04:53 -08:00
Dale HuiandGitHub c6c22b34c8 Fix broken "versions" table formatting in README 2019-03-01 18:52:20 -08:00
Dale Hui 752cdc3b11 Update "versions" table in the README
- Reflect supported versions
    - With Go 1.12 released, module support is no longer an issue
2019-03-01 18:49:12 -08:00
Dale Hui f5c4ccff1f Update dependencies 2019-03-01 17:45:37 -08:00
Dale Hui 5e004dbef4 Fix issues with new version of MongoDB driver:
- "connect=single" option is no longer supported
        - "connect=direct" is the equivalent
        - https://jira.mongodb.org/browse/GODRIVER-665
    - Count() needs to be passed a non-nil filter/document
        - https://jira.mongodb.org/browse/GODRIVER-572
2019-03-01 17:44:40 -08:00
Dale Hui 4fba554b47 Drop support for go1.10
- Remove dep package pinnings
2019-03-01 14:37:54 -08:00
Dale Hui 6593dc8dbf Support Go 1.12 2019-02-27 23:44:23 -08:00
Lukas JoergensenandDale Hui 480a5a634a postgres: Move lock out of ensureVersionTable, for consistency with other SQL operations (#173)
* Consistently lock in ensureVersionTable and do not call ensureVersionTable from Drop across all database implementations

* Add test for dropping postgres databases

* Fix failing database tests

* Fix CockroachDb test, lock table should be created before versionTable

* Add Initialize() to Driver interface, and add integration tests for Drop() between database implementations and migrate

* Remove Initialize, document breaking behaviour of Drop

* Revert introduction of Initialize method

* Removed Initialize in Stub as well

* Remove call to non-existent Initialize and make sure to close re-initialized database connections

* Revert changes to TestDrop in database/testing

* Split Test and TestMigrate into different test entrypoints

* Remove unused import in migrate_testing

* Remove erroneous code to fix tests

* Add stub source imports to database tests

* Add Stub source to migrate tests

* Use example migrations for tests

* Add file driver to database tests

* Align database directory layout

* Add file source driver to Cassandra

* Review changes

* Minor syntactic change for cleaner diff
2019-02-26 15:56:57 -08:00
Dale Hui f213007d4d Use alpine 3.9 for building Docker images 2019-02-19 03:25:47 -08:00
Dale Hui 0f8263de2e Remove unnecessary panics from core implementation 2019-02-19 02:07:07 -08:00
Dale HuiandGitHub 9b449be538 Merge pull request #176 from jszwedko/quote-postgres-identifiers
Quote postgres table identifiers when `DROP`ing
2019-02-18 03:14:11 -08:00
Jesse Szwedko 3db0395e0a Replace " quoting with pq.QuoteIdentifier for postgres driver 2019-02-18 11:44:38 +01:00
Jesse Szwedko fe66cd7734 Quote postgres table identifiers when DROPing
Handles table identifiers that are invalid without quotes (such as
uppercase letters).
2019-02-18 11:29:10 +01:00
Dale HuiandGitHub 4190b85018 Merge pull request #175 from plutov/master
Add mongodb driver to Dockerfile
2019-02-18 02:05:12 -08:00
Alex Pliutau 0a355bbee0 Add mongodb driver to Dockerfile 2019-02-18 10:15:23 +01:00
Brian Wigginton 407d1e3982 bump go.sum 2019-02-15 16:12:57 -06:00
Brian Wigginton 9b551a1c10 initial mssql support 2019-02-15 16:12:57 -06:00
Dale HuiandGitHub c0f025d2f6 Merge pull request #167 from arpando/fix/DocumentNilError
Fix document nil error
2019-02-05 03:48:34 -08:00
Arcadio Pando 56620dc3dd Fix document nil error 2019-02-04 17:37:22 +01:00
Dale HuiandGitHub 598addc6d7 Merge pull request #166 from bcho/fix/sqlite3-migration-table
sqlite3: fix wrong schema table usage
2019-02-02 11:19:58 -08:00
hbc 95e701da6a sqlite3: fix wrong schema table usage
fixes: https://github.com/golang-migrate/migrate/issues/165
2019-02-02 13:22:40 +08:00
Dale HuiandGitHub f6d624c729 Merge pull request #150 from tsenart/atomic-migrations-table-creation
postgres: Make `ensureVersionTable` atomic
2019-01-15 11:55:50 -08:00
Dale Hui 5ad19c5071 Run dep ensure
- Updates Gopkg.lock after github.com/hashicorp/go-multierror package was manually added to Gopkg.toml
2019-01-15 11:33:17 -08:00
Tomás Senart 41a595f62a fixup! Add github.com/hashicorp/go-multierror@v1.0.0 to Gopkg.yaml 2019-01-15 11:28:49 +01:00
Tomás Senart f58da1d4dc fixup! More explicit conditionals 2019-01-15 11:28:49 +01:00
Tomás Senart b57dbceec3 postgres: Revert back to use CREATE IF NOT EXISTS 2019-01-15 11:28:49 +01:00
Tomás Senart 06c80ae2da postgres: Use multierror.Error in ensureVersionTable 2019-01-15 11:28:49 +01:00
Tomás Senart 0beddcd2d0 postgres: Be explicit about max idle and open conns 2019-01-15 11:28:48 +01:00
Tomás Senart 94f5aa00fc postgres: Query table existance in ensureVersionTable
This commit reverts back to querying the existance of the table instead
of using CREATE IF NOT EXISTS because we want to support versions of
Postgres older than 9.1 which is when this feature was introduced.
2019-01-15 11:28:48 +01:00
Tomás Senart 5bf80d7ef2 postgres: Add comment about connection pool in TestWithInstance_Concurrent 2019-01-15 11:28:48 +01:00
Tomás Senart 09a1959309 postgres: Preserve Unlock error when outer err isn't nil 2019-01-15 11:28:48 +01:00
Tomás Senart 15713e64cb postgres: Use sync.WaitGroup in TestWithInstance_Concurrent 2019-01-15 11:28:48 +01:00
Tomás Senart 519dae2639 postgres: Make ensureVersionTable atomic
Fixes https://github.com/golang-migrate/migrate/issues/55
2019-01-15 11:28:48 +01:00
Tomás Senart ce7a2234ee fixup! Pass config to WithInstance 2019-01-15 11:28:48 +01:00
Tomás Senart a06a92d620 postgres: TestWithInstance_Concurrent 2019-01-15 11:28:47 +01:00
Dale Hui 1ca4da9a33 Run go mod tidy 2019-01-14 12:56:51 -08:00
Dale Hui 52fc08fdf5 Update supported DB versions 2019-01-12 16:13:26 -08:00
Dale Hui 2feaaaddda Update dktest from v0.2.0 to v0.3.0
- Update ReadyFuncs accordingly
2019-01-08 22:39:26 -08:00
Dale Hui 9870a84b99 Update Cassandra DB driver (gocql) 2019-01-08 11:25:23 -08:00
Dale Hui bbec63fda7 Use one of the 2 ports exposed by MySQL Docker images
- MySQL driver tests should pass more consistently now
2019-01-08 05:11:07 -08:00
Dale Hui 809c7f0ac8 Use dktest to run docker tests
- Leaving migrate/testing in case there are unknown consumers
    - Add migrate/dktesting package
    - Update tests to use migrate/dktesting instead of migrate/testing
2019-01-08 05:11:07 -08:00
Dale Hui fd47ba9a9a Remove TODO for MongoDB database driver 2019-01-08 05:01:48 -08:00
Dale HuiandGitHub b1a199a272 Merge pull request #146 from bobrovde/mongodb-driver
added mongodb driver
2019-01-08 02:34:59 -08:00
Dale HuiandGitHub fc8d57d128 Merge branch 'master' into mongodb-driver 2019-01-07 22:50:51 -08:00
Dale HuiandGitHub f24ea4d6c9 Update TravisCI badge after migration from .org to .com 2019-01-07 22:45:35 -08:00
DBobrov 12b93a3a07 Change checking of mongo replica set initialization 2019-01-07 15:17:35 +03:00
DBobrov 3fa0df7553 Refactor mongodb transactions test 2019-01-07 11:47:37 +03:00
DBobrov 13978a108a Add test for migrations in transaction mode 2019-01-06 12:45:22 +03:00
Dale Hui d549fd5265 Fix memory leak in tests
- Stop leaking tickers. As the docs state, using time.Tick() will leak tickers,
      so use time.NewTicker() with time.Ticker.Stop().
    - Be more efficient by using time.NewTimer() with time.Timer.Stop() instead of time.After().
2019-01-05 13:52:25 -08:00
DBobrov 30d2d94a22 Fix review comments 2019-01-05 11:10:25 +03:00
Dale Hui caa5719a86 Update pinned (via dep) MySQL driver from v1.4.0 to v1.4.0 2019-01-04 17:47:53 -08:00
DBobrov 436b81b357 Replaced bsonx.Doc with bson.D 2019-01-04 22:01:21 +03:00
Dale Hui 600f2eda93 Fix broken build
- `gem install` no longer recognizes the `--no-ri` and `--no-rdoc` options. Use `--no-document` instead.
    - References:
        - https://github.com/rubygems/rubygems/pull/2354
        - https://github.com/bundler/bundler/pull/6624
2019-01-03 16:35:22 -08:00
DBobrov 309edb6da6 - Add dependency for dep
- Add transaction mode for migrations
2019-01-03 18:00:23 +03:00
DBobrov bcd4f6e7dd added mongodb driver 2019-01-01 11:14:27 +03:00
Dale HuiandGitHub fd50054781 Merge pull request #145 from bobrovde/fix-clickhouse
Changed clickhouse migrations table struct
2018-12-30 13:42:39 -08:00
DBobrov 26064905d4 Changed migrations table struct of field version to fix support database.NilVersion and default version format. 2018-12-28 15:53:55 +03:00
Dale Hui 24176463f4 Update docker/docker checksum for Go 1.11.4
Addresses: https://github.com/golang-migrate/migrate/issues/138
2018-12-21 12:53:53 -08:00
Dale HuiandGitHub 418d41c2a1 Merge pull request #143 from coolaj86/url-opaque
use URL.Opaque when available
2018-12-20 21:37:54 -08:00
AJ ONeal 01291f7cb5 use URL.Opaque when available 2018-12-20 15:08:49 -07:00
Dale HuiandGitHub e8b8975ec3 Merge pull request #141 from fsouza/upgrade-mysql-driver
Upgrade go-sql-driver/mysql to v1.4.1
2018-12-19 11:31:31 -08:00
Francisco Souza 6a53d7d10d Upgrade go-sql-driver/mysql to v1.4.1
Fixes #139.
2018-12-18 18:01:52 -05:00
Dale HuiandGitHub 65870ff3e8 Merge pull request #126 from a-h/patch-1
Include details about migration locking in the FAQ
2018-12-14 17:36:07 -08:00
6f1479fa41 Update FAQ.md
Co-Authored-By: a-h <a-h@users.noreply.github.com>
2018-12-14 16:28:04 +00:00
Dale Hui f694b616aa Add supported major versions table to README
- Consolidates info about migrate major versions
    - https://github.com/golang-migrate/migrate/issues/134#issuecomment-446664565
2018-12-13 22:53:07 -08:00
Dale Hui 1bd41c2a0b README copy tweak
- helpers -> snippets
2018-12-13 22:41:26 -08:00
Dale HuiandGitHub f1d88ac6c1 Add guide for installing a specific version of migrate using the Go toolchain 2018-11-28 01:00:46 -08:00
Dale HuiandGitHub 0d5d918563 Fix CLI install with Go toolchain docs
`-d` option shouldn't be specified with `go get` when installing
2018-11-28 00:44:59 -08:00
Dale Hui 547bc41026 Merge branch 'deb' 2018-11-27 13:54:07 -08:00
Dale Hui d27dc63026 Move before_deploy steps to after_success to avoid re-building the binary before every deploy
- https://docs.travis-ci.com/user/job-lifecycle/#deploying-your-code
2018-11-27 13:48:57 -08:00
Dale Hui 83f5b9f644 Don't need to move and remove files before caching build data in Travis 2018-11-27 13:48:57 -08:00
Dale Hui addb3c1a79 .deb package changes
- Install binary to /usr/local/bin instead of /usr/bin
    - Support recent/stable Debian and Ubuntu distros that work with .deb packages
2018-11-27 13:48:57 -08:00
Dale HuiandGitHub 88e7eafcf9 Merge pull request #131 from vearutop/cli-layout
Refactor cli package to allow `go get` one-liner with a proper binary name
2018-11-27 12:15:59 -08:00
Viacheslav Poturaev 5f88d4bf22 Copy cli main package under migrate directory, keep deprecated original main package for backwards compatibility, fixes #70 2018-11-21 09:01:35 +01:00
Dale HuiandGitHub 7f00868584 Merge pull request #128 from andrei-m/fix-redshift
Fix Redshift migrations driver
2018-11-14 10:32:07 -08:00
Dale Hui 6ebc6c7dae Unlock in postgres db driver should also use the schema name 2018-11-14 02:18:57 -08:00
Dale Hui f38fe38d45 Use git describe --tags to set the version number when building the CLI 2018-11-12 02:23:08 -08:00
Dale Hui 323427e951 Update copyright in LICENSE 2018-11-08 01:45:19 -08:00
Dale Hui 06d3ebde1c Use strings.NewReader() instead of bytes.NewReader() when source is a string
- Group imports
2018-11-05 22:50:08 -08:00
Dale HuiandGitHub 1df8057f97 Merge pull request #127 from vporoshok/master
Add support for multi-schema migrations in Postgres
2018-11-05 22:34:54 -08:00
Evgeniy Bastrykov d2d449ad78 Review comments 2018-11-05 16:03:54 +04:00
Evgeniy Bastrykov 16d63e3a76 Add support for multi-schema migrations in Postgres
There is lock conflict on parallel migrations in different postgres
schemas. To avoid this conflicts function GenerateAdvisoryLockId added
variadic params to change lock id with schema name. Schema name taked
with postgres CURRENT_SCHEMA function. Null byte used as separator
between database and schema name, because any other symbol may be used
in both of it.

Closes #118
2018-11-05 12:56:38 +04:00
Andrei Mackenzie cc573a74dc Remove unused errors 2018-11-03 19:27:55 -04:00
Andrei Mackenzie 1d54cf5f18 Replace 'redshift' with 'redshift2'
This addresses https://github.com/golang-migrate/migrate/issues/90 . The
exported Redshift object no longer exports an embedde 'Driver' however,
so some more work is needed to make this backwards compatible.
2018-11-03 19:01:37 -04:00
Andrei Mackenzie 4e098f74cd Use 'DELETE FROM' over 'TRUNCATE' in Redshift
TRUNCATE commits the current transaction, which breaks the expection of
the 'Commit()' that follows.

See:
https://github.com/golang-migrate/migrate/issues/90
https://docs.aws.amazon.com/redshift/latest/dg/r_TRUNCATE.html
2018-11-03 18:52:09 -04:00
Andrei Mackenzie b0300df5e7 Use the most Redshift-like Postgres version available on Dockerhub
Redshift is based on Postgres version 8.0.2.
2018-11-03 18:49:31 -04:00
Andrei Mackenzie 9c132fe622 Update README with Redshift-specific info 2018-11-03 18:47:12 -04:00
Andrei Mackenzie 22334834ac Support the 'redshift' URL scheme
This brings the 'redshift2' package in alignment with the existing
'redshift' package.
2018-11-03 18:44:11 -04:00
Andrei Mackenzie 3373343f23 Remove advisory locking from the Redshift implementation
Redshift does not support advisory lock functions. The closest
capability is in-transaction table locks, which aren't quite right here
because the transaction scope is established within SetVersion, not
higher up above the Lock-before/Unlock-after SetVersion.

Local locking is left intact to satisfy expected "can't Lock twice
before Unlocking" behavior asserted in shared tests.
2018-11-03 18:31:41 -04:00
Andrei Mackenzie 1d0bedc454 Rename 'Postgres' to 'Redshift' in 'redshift2' 2018-11-03 18:09:53 -04:00
Andrei Mackenzie 8381ea0307 Clone the 'postgres' driver as 'redshift2'
Avoid stepping on the 'redshift' driver for the time being. Driver
registration is also modified to identify the clone as 'redshift2'
rather than 'postgres'.
2018-11-03 18:04:22 -04:00
Adrian HeskethandGitHub 2e0b820637 Include details about migration locking in the FAQ 2018-11-02 15:41:41 +00:00
Dale HuiandGitHub 9f5e1bd505 Merge pull request #124 from lopezator/master
cli: Update README.md with MacOS install info
2018-10-29 11:02:56 -07:00
Dale Hui 7db5c9a274 Remove references to dep from docs 2018-10-29 10:56:55 -07:00
David López 4fad212a25 cli: Update README.md with MacOS install info
Document in MacOS install section of README.md that brew packag is now available. References mattes/migrate#156 https://github.com/Homebrew/homebrew-core/pull/30032
2018-10-29 17:27:48 +01:00
Dale HuiandGitHub f6b3bf548d Merge pull request #121 from wmetaw/master
Add NewDB to spanner.go
2018-10-28 16:58:22 -07:00
Ryo Takashima ca888b96e3 Add NewDB to spanner.go 2018-10-28 22:00:05 +09:00
Dale HuiandGitHub 2fc1610248 Merge pull request #115 from weszeloos/add_support_for_tls_in_cassandra
Add support for TLS on Cassandra.
2018-10-16 15:09:30 -07:00
Wessel Oosthuizen bc28cee1dc Add support for TLS on Cassandra. 2018-10-16 11:18:38 +02:00
Dale Hui ea1928c904 Update dependencies 2018-10-15 13:26:18 -07:00
Dale HuiandGitHub 8c3f34e0c9 Merge pull request #114 from albenik/master
github.com/docker/docker package updated to api v1.37
2018-10-15 13:09:43 -07:00
Veniamin Albaev 956441176f github.com/docker/docker package updated to api v1.37
To avoid github.com/sirupsen/logrus package name case collision with go mod
2018-10-15 17:39:43 +03:00
Dale Hui 83d01b7e1c Run go mod tidy 2018-10-10 16:16:23 -07:00
Dale Hui fc2b2e9c64 Use v4 for Go module support 2018-10-10 16:16:23 -07:00
Dale Hui 60b45ba428 Revert "Docker builds need to use dep to manage dependencies"
This reverts commit 4937cd088f.
2018-10-10 16:16:23 -07:00
Dale Hui 107db3f407 Revert "Remove support for Go modules"
This reverts commit 5e96539f55.
2018-10-10 16:15:33 -07:00
Dale Hui 4937cd088f Docker builds need to use dep to manage dependencies 2018-10-10 16:11:44 -07:00
Dale Hui 5e96539f55 Remove support for Go modules 2018-10-10 13:34:03 -07:00
Dale HuiandGitHub 708d67fa02 Merge pull request #112 from xiamengyu/master
Fix typo in comments
2018-10-10 13:26:36 -07:00
xiamengyu1 a95b13d804 Fix typo 2018-10-10 16:26:27 +08:00
Dale HuiandGitHub 1e60261d3f Merge pull request #110 from MikeFitzgerald/master
Fix postgres drop
2018-10-04 00:30:33 -07:00
Mike Fitzgerald 0d4f1c0212 Fix postgres drop 2018-10-04 01:22:39 -04:00
Dale HuiandGitHub 4d30c4c87f Add Go version to but report template 2018-09-19 03:10:08 -07:00
Dale Hui 6c02d65563 Update commented out imports to use v3 2018-09-14 14:12:07 -07:00
Dale Hui 16dea48c82 Fix module support by opting in to modules 2018-09-14 01:14:42 -07:00
Dale Hui f517029483 Test latest version of Go as well, but allow failures 2018-09-13 16:14:43 -07:00
Dale Hui 6f233b3b58 Update dependencies 2018-09-13 16:14:43 -07:00
Dale HuiandGitHub f8c85735f4 Merge pull request #98 from Kay-Zee/Kay-Zee/close-sql-db-connections-on-driver-close
Add sql.DB to Postgres and Mysql structs to allow closing in Close()
2018-09-07 01:26:43 -07:00
Kay-Zee 1e4c50b70d Add sql.DB to Postgres and Mysql structs to allow closing in Close() 2018-09-06 17:16:02 -07:00
Dale Hui 93d53a5ae8 Use Go 1.11 (with modules enabled on alpine linux 3.8) for docker builds 2018-09-04 22:18:49 -07:00
Dale Hui 16f2b1736e Support Go 1.11 and drop support for Go 1.9
- Support modules in Go 1.11
2018-09-04 19:11:19 -07:00
Dale Hui 4d60607b0b Update issue templates 2018-08-29 12:14:54 -07:00
Dale Hui beb09d85fa Upgrade dep from v0.4.1 to v0.5.0 2018-08-15 12:08:25 -07:00
Dale HuiandGitHub 8ed902364c Merge pull request #91 from stabacov/master
Fixes MySQL custom TLS connection failure.
2018-08-15 00:03:14 -07:00
Tatsuhito KATO 02c83fb38b more descriptive. 2018-08-15 02:42:18 +00:00
Tatsuhito KATO 9049e49f9e x-tls-cert and x-tls-key must set both. 2018-08-14 07:43:14 +00:00
Tatsuhito KATO 9d36dd9841 Add detail of tls query. 2018-08-14 07:42:10 +00:00
Tatsuhito KATO 9905791932 Fixes MySQL custom TLS connection failure. 2018-08-13 11:43:19 +00:00
Dale Hui dd00ca926e Add support for postgres 10 and drop support for postgres 9.2
https://www.postgresql.org/support/versioning/
2018-08-12 15:54:21 -07:00
Dale HuiandGitHub 4092e160a4 Merge pull request #85 from HaraldNordgren/improve_tests
Update tests for 'Migrate', 'Steps' and 'UpAndDown' to assert exact migration sequence
2018-08-08 15:57:04 -07:00
Dale HuiandGitHub 6092802b3a Merge pull request #88 from HaraldNordgren/error_output
Improve error output for missing or malformed '-database' and '-source' parameters
2018-08-08 15:24:39 -07:00
Harald Nordgren 42f074062b Update tests for 'Migrate', 'Steps' and 'UpAndDown' to assert exact migration sequence 2018-08-08 22:24:37 +02:00
Harald Nordgren 112191042b Improve error output for missing or malformed '-database' and '-source' parameters 2018-08-08 21:27:12 +02:00
Dale Hui 1566dd06cd Pass the error through instead of overriding it 2018-08-07 16:22:05 -07:00
Dale HuiandGitHub 251bf76be4 Merge pull request #89 from HaraldNordgren/improve_documentation
Document that '-source' is a required parameter
2018-08-07 15:59:20 -07:00
Harald Nordgren a6eddccfe5 Document that '-source' is a required parameter 2018-08-07 09:53:36 +02:00
Dale Hui 3bd43bedad Reference mattes/migrate now that golang-migrate/migrate has been detached
https://github.com/golang-migrate/migrate/issues/81
2018-08-07 00:17:39 -07:00
Dale HuiandGitHub 4750e57ce1 Merge pull request #82 from UnitedTraders/clickhouse-multiline
Add support for the multiline clickhouse queries
2018-08-06 04:09:10 -06:00
Anton Markelov df7ce674e9 add support for the multiline clickhouse queries 2018-08-03 16:08:47 +10:00
Dale Hui e968e8a5f0 Merge branch 'update_pkgs' 2018-07-24 18:30:53 -07:00
Dale HuiandGitHub c8705dd876 Merge pull request #78 from dhui/mysql_dsn
Allow DB connection URLs to contain encoded reserved URL characters
2018-07-24 18:22:04 -07:00
Dale Hui c7a29e331f Another round of package updates couldn't hurt... 2018-07-24 18:19:22 -07:00
Dale Hui ffcbbd5fea Override dependency on docker/distribution to fix build
https://github.com/golang/dep/issues/1890#issuecomment-404036309
2018-07-24 18:13:11 -07:00
Dale Hui ebf9c0bf91 Update package dependencies 2018-07-24 18:13:11 -07:00
Dale Hui df658e8fa8 Don't urlencode the username and password for MySQL
Addresses: https://github.com/golang-migrate/migrate/pull/69
2018-07-24 17:56:17 -07:00
Dale Hui b5edb8e50c Improve error messaging when URL parsing fails
- Improve docs around escaping/encoding reserved URL characters
    - Add tests for net/url Parse() and reserved characters to document understanding
    - Addresses: https://github.com/golang-migrate/migrate/issues/77
2018-07-24 17:55:58 -07:00
Dale Hui 97eb0d191b Fix broken GenerateAdvisoryLockId() test
- Add it to the test suite to ensure that it doesn't break again
2018-07-24 17:55:58 -07:00
Dale Hui 857d7a620d Revert "Merge pull request #69 from bcho/fix/mysql-url-password-encode"
This reverts commit 78d696c1e5, reversing
changes made to 18583d5a91.
2018-07-24 17:55:58 -07:00
Dale HuiandGitHub fad64ed381 Merge pull request #76 from agquick/master
Support for multiple statements in Cassandra cql migration files
2018-07-24 16:02:44 -07:00
Alan Graham 4244ebea6a Add detail to README for Cassandra flag, improve if block 2018-07-24 19:30:44 +00:00
Alan Graham 327822c29a Added Cassandra URL flag to enable multi statement migrations 2018-07-20 13:27:54 +00:00
Dale HuiandGitHub feea45c205 Merge pull request #75 from LUSHDigital/master
Lock go-sql-driver/mysql to v1.4.0
2018-07-17 16:42:01 -07:00
Alan Graham c4b81d1210 Support for multiple statements in Cassandra cql migration files based on semi-colon and newline. 2018-07-17 15:19:54 +00:00
Dan Richards 3988074cae Lock digest. 2018-07-16 12:09:29 +01:00
Dan Richards ad2b69e115 Typo 2018-07-16 12:08:24 +01:00
Dan Richards f9eb960848 Just lock go-sql-driver/mysql. 2018-07-16 12:06:27 +01:00
Dan Richards 5927d6e5f4 Locked go-sql-driver/mysql to v1.4.0 2018-07-16 12:01:52 +01:00
Dale Hui 6bb3e494e5 Disable CGO to build binaries statically on the building platform
- Previously, only cross-compiled binaries were built statically
    - Addresses: https://github.com/golang-migrate/migrate/issues/45
2018-07-14 13:31:59 -07:00
Dale Hui 5d5d15e7a3 Link to comparison of transactional DDLs 2018-07-09 19:25:29 -07:00
Dale Hui b742ec3de4 Use unix line endings 2018-07-09 19:21:40 -07:00
Dale Hui 75f0aade0d Reduce docker image size by removing unnecessary chown
https://github.com/moby/moby/issues/5505
2018-07-09 14:52:59 -07:00
Dale HuiandGitHub 3872c8f41d Merge pull request #71 from qdentity/postgres-parse-error
Add postgres lib/pq error parsing
2018-07-09 12:26:24 -07:00
Damir Vandic ae2fd4fa7b Support postgres non-ascii error parsing 2018-07-09 11:39:29 +02:00
Damir Vandic 3eb26a65d3 Add postgres lib/pq error parsing 2018-07-07 12:17:38 +02:00
Dale Hui 157433893c Update CLI installation docs to reduce confusion
https://github.com/golang-migrate/migrate/issues/64#issuecomment-402300146
2018-07-05 10:55:44 -07:00
Dale Hui 236dac0c96 Add latest tag to docker image during build 2018-07-03 14:09:26 -07:00
Dale HuiandGitHub 78d696c1e5 Merge pull request #69 from bcho/fix/mysql-url-password-encode
prevent net/url encoding the user password
2018-06-29 14:17:45 -07:00
Chao Dengandhbc 48a4062f75 prevent net/url encoding the user password 2018-06-29 21:07:10 +08:00
Dale Hui 18583d5a91 Update docs
- Addresses: https://github.com/golang-migrate/migrate/issues/67
    - Add badge for supported Go versions
    - Suggest dep for dependency management
2018-06-27 11:45:54 -07:00
Dale HuiandGitHub 642913dbf4 Merge pull request #66 from dadlerj/cli-readme
Update CLI usage in readme
2018-06-26 13:25:18 -07:00
dadlerj 2ca5473e52 Update CLI usage in readme 2018-06-26 09:16:04 -07:00
Dale Hui 6070aa8281 Push latest image to docker hub 2018-06-19 01:42:37 -07:00
Dale HuiandGitHub 78dc468555 Merge pull request #47 from dhui/docker_badge
Add badge for num docker pulls
2018-06-19 01:33:06 -07:00
Dale Hui bcd996f3df Use environment variables since there's no way to pass secure data to deploy script provider 2018-06-19 00:39:54 -07:00
Dale Hui a1d959d0e9 Use latest version of docker 2018-06-18 23:07:54 -07:00
Dale Hui 71e97bccc7 Fix TravisCI conditional release 2018-06-18 21:42:31 -07:00
Dale Hui 3239b18671 Fix time format flag to be backwards compatible
https://github.com/golang-migrate/migrate/pull/17#issuecomment-397960867
2018-06-18 00:06:17 -07:00
Dale Hui c3794da4ed Make -ext flag required for the create command 2018-06-17 23:18:23 -07:00
Dale Hui 872a561495 Fix broken CLI build 2018-06-17 23:16:38 -07:00
Dale Hui d1adc8b15f Add golang/x/tools (godoc/vfs) as a dependency 2018-06-17 19:59:39 -07:00
Dale HuiandGitHub e7d20b35ee Merge pull request #57 from SamWhited/vfs
source/vfs: add virtual file system source
2018-06-17 19:22:06 -07:00
Dale Hui 21f566a60a Add docker hub credentials for uploading docker image 2018-06-17 17:42:03 -07:00
Sam Whited 7562e9ad5d source/vfs: add virtual file system source
Fixes #56
2018-06-17 19:35:13 -05:00
Dale Hui 0f16f7f6a2 Rename sources to reflect updated build constraint names 2018-06-12 16:45:56 +08:00
Dale Hui dfa9fc05cb Use correct build contraints (must be alphanumeric)
Fixes: https://github.com/golang-migrate/migrate/issues/52
2018-06-12 16:18:27 +08:00
Dale Hui c4e855b58c List source and database drivers in help text 2018-06-12 16:17:52 +08:00
Dale Hui 0d9f636356 Use dep to build docker image 2018-06-11 19:18:05 +08:00
Dale HuiandGitHub 387f79be13 Merge pull request #48 from dhui/static_build
Build binaries statically
2018-06-04 09:24:35 -07:00
Dale Hui 87f3e4a910 Build binaries statically
Addresses: https://github.com/golang-migrate/migrate/issues/45
2018-06-04 19:27:37 +08:00
Dale Hui 01ea635328 Add badge for num docker pulls 2018-06-04 18:54:43 +08:00
Dale HuiandGitHub 3d1bb8a4b4 Merge pull request #13 from Teddy-Schmitz/postgres-wait
postgres driver should wait for lock
2018-05-31 19:04:30 -07:00
Teddy Schmitz 046d1cb767 postgres driver should wait for lock 2018-06-01 09:11:24 +08:00
Dale Hui 3d7b06e24c Override google cloud package version (was pinned to v0.19.0 by github.com/fsouza/fake-gcs-server) 2018-05-31 15:26:18 -07:00
Dale Hui 063b69db2f Update OpenCensus dependency 2018-05-31 15:05:31 -07:00
Dale Hui c0b0829d5a Use "vanity" import to allow forks to build
https://docs.travis-ci.com/user/languages/go/#Go-Import-Path
2018-05-31 14:57:38 -07:00
Dale Hui 3cf42dd36d Don't need to re-solve package dependencies every time. Only use Gopkg.lock
Reference: https://github.com/golang/dep/blob/master/docs/ensure-mechanics.md#-no-vendor-and--vendor-only
2018-05-31 14:57:38 -07:00
Dale Hui 89e558ee4f Don't cache go source since dependencies are managed by dep now 2018-05-31 14:57:38 -07:00
Dale Hui f277ef7428 Update github.com/go-sql-driver/mysql and github.com/gocql/gocql 2018-05-31 14:57:38 -07:00
Dale Hui b7647a749e Use the master branch for go-sql-driver/mysql since there hasn't been a release in a yr and a half 2018-05-31 14:57:38 -07:00
Dale Hui 4cc6aeb274 Update gocql package 2018-05-31 14:57:38 -07:00
Dale Hui 3de903f51f Use latest minor revision of Go 1.10 in TravisCI 2018-05-31 14:57:38 -07:00
Dale Hui 4b4f49b1eb Update docs regarding usage of dep 2018-05-31 14:57:38 -07:00
Dale Hui 4407720b95 Remove "deps" target from Makefile 2018-05-31 14:57:38 -07:00
Dale Hui 36dc939f91 Use dep for TravisCI builds 2018-05-31 14:57:38 -07:00
Dale Hui 40cd326ed7 Use dep to manage dependencies 2018-05-31 14:57:38 -07:00
Dale Hui a890d44d2f CockroachDB driver improvements
- Correctly return database.ErrLocked when the DB is locked
    - Group imports
    - gofmt goodness
2018-05-31 14:44:48 -07:00
Dale HuiandGitHub c98426505e Merge pull request #43 from wmetaw/master
Fixed unkeyed fields in file.go for GoogleAppEngine
2018-05-31 14:23:14 -07:00
Ryo Takashima 6fa8039b55 Fixed unkeyed fields for GoogleAppEngine 2018-05-31 15:40:15 +09:00
Dale HuiandGitHub faf4307fcc Merge pull request #32 from kenjones-cisco/feature/cassandra-withinstance
Minor tweaks to remove duplication
2018-05-17 01:01:47 -07:00
kenjones 1512e41e41 Minor tweaks to remove duplication
Adds missing connection close for cassandra tests

Revert to default timeout of 600ms
2018-05-16 20:09:12 -04:00
Dale Hui 55a25c5e0e Cassandra DB driver test improvements
- shorten connect timeout to allow more connection retry attempts
    - close session after it's created
    - group imports
2018-05-16 15:12:36 -07:00
Dale HuiandGitHub f815731412 Merge pull request #30 from Decemberlabs/master
Add WithSession helper for Cassandra driver
2018-05-16 14:26:12 -07:00
Andrés Rodríguez d607064fd2 Switch Cassandra's package lock to instance lock 2018-05-16 16:26:23 -03:00
Andrés Rodríguez 8b0f876416 Rename Cassandra's WithSession to WithInstance 2018-05-16 16:23:21 -03:00
Andrés Rodríguez bbd8f410d9 Correct method name in comment 2018-05-16 10:54:56 -03:00
Dale Hui fb3d809e81 Cleanup CLI docs 2018-05-12 15:25:19 -07:00
Dale HuiandGitHub 95db494b30 Merge pull request #28 from tayhalla/patch-1
Update README.md
2018-05-12 14:58:12 -07:00
Taylor HallidayandGitHub 5ebdb74ba8 Update README.md
There's no homebrew support. Even the referenced TODO says so, so not sure why it's in the Readme. Let's encourage others to support the development of it rather than think the support for it is broken.
2018-05-12 07:22:21 -07:00
Dale HuiandGitHub 07c20c5357 Merge pull request #27 from Its-Alex/master
Fix dockerfile version and deps
2018-05-10 01:13:05 -07:00
ItsAlex 31daa8dad3 fix dockerfile version and deps
Signed-off-by: Alexandre MARRE <me@itsalex.fr>
2018-05-10 09:12:23 +02:00
Dale HuiandGitHub a0b03c3a68 Merge pull request #24 from Its-Alex/master
Dockerization
2018-05-09 11:22:34 -07:00
ItsAlex fb7bb4f87a add dockerization Fix #23
Signed-off-by: Alexandre MARRE <me@itsalex.fr>
2018-05-09 09:48:20 +02:00
Dale HuiandGitHub 81e9fff605 Merge pull request #22 from thurt/add-ref-to-github
add ref (SHA, branch, or tag) to the github migration source
2018-05-04 10:13:18 +02:00
thurt e6baca2ffb add ref (SHA, branch, or tag) to the github migration source 2018-05-04 03:16:12 +00:00
Dale HuiandGitHub 2b28a0bc0a Merge pull request #17 from bobmeister/cli_create_yyyymmddhhmmss_migration_files
-datetime option to create files in yyyymmddhhmmss format
2018-04-17 10:50:48 -07:00
Robert Haupt 1f389f1422 Removed commented code 2018-04-17 11:17:10 -06:00
Robert Haupt 8cdee40695 Use Go time format string if version other than timestamp is desired. 2018-04-17 11:09:08 -06:00
Robert Haupt c6c6874d72 Added -datetime option to create up/down files with version in yyyymmddhhmmss format. 2018-04-16 11:12:06 -06:00
Andrés Rodríguez 78c47074a3 Add WithSession helper for Cassandra driver
Also changes method receivers names for the Cassandra struct to "c"
since "p" makes no sense in this context.
2018-03-13 15:49:08 -03:00
Dale Hui 22f249514d Merge branch 'postgres-lock-fix'
Addresses: https://github.com/golang-migrate/migrate/pull/4
2018-02-20 16:12:54 -08:00
Dale HuiandGitHub 8f6826c9e3 Merge pull request #8 from rdallman/fix-lockyloo-golang
Fix lockyloo golang
2018-02-20 15:51:18 -08:00
Dale Hui 1519c59188 Rename Postgres.db to Postgres.conn 2018-02-20 15:40:20 -08:00
Dale HuiandGitHub a9176e9969 Merge branch 'master' into fix-lockyloo-golang 2018-02-20 15:37:20 -08:00
Dale Hui ab7dfb38ac Merge remote-tracking branch 'origin/master' into postgres-lock-fix 2018-02-20 15:32:34 -08:00
Dale Hui aa21a21647 Support Go 1.10 and drop support for 1.8
- Test using latest minor versions of the last 2 major (supported) releases
        - TravisCI's gimme doesn't support Go 1.10.x until a minor version is released
    - Explicitly use YAML strings so Go versions aren't interpreted as numbers
2018-02-20 15:20:43 -08:00
Reed Allman 094bad431e s/db/conn/ 2018-02-09 15:14:05 -08:00
Reed Allman a8ef2b8cfe update CI and build tags to go1.9 2018-02-09 00:32:47 -08:00
Reed Allman 636c911d33 fixes mysql lock failure
I believe this closes #297 as well.

I have been working on adding testing of migrations and it requires acquiring
the lock in mysql multiple times to go up and down. After nailing this down to
GET_LOCK returning a failure for every subsequent GET_LOCK call after the
first, I decided it was time to rtfm and lo and behold:

https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_release-lock

RELEASE_LOCK will not work if called from a different thread than GET_LOCK.
The simplest solution using the golang database/sql pkg appears to be to just
get a single conn to use for every operation. since migrations are happening
sequentially, I don't think this will be a performance hit (possible
improvement). now calling Lock() and Unlock() multiple times will work;
prior to this patch, every call to RELEASE_LOCK would fail. this required
minimal changes to use the *sql.Conn methods instead of the *sql.DB methods.

other changes:

* upped time out to 10s on GET_LOCK, 1s timeout can too easily leave us in a
state where we think we have the lock but it has timed out (during the
operation).
* fixes SetVersion which was not using the tx it was intending to, and fixed a
bug where the transaction could have been left open since Rollback or Commit
may never have been called.

I added a test but it does not seem to come up in the previous patch, at least
easily, I tried some shenanigans. At least, this behavior should be fixed with
this patch and hopefully the test / comment is advisory enough.

thank you for maintaining this library, it has been relatively easy to
integrate with and most stuff works (pg works great :)
2018-02-09 00:27:01 -08:00
Dale Hui 53dd06057f Add release badge 2018-01-21 02:07:25 -08:00
Dale Hui 1b840d9c79 Remove unused invalid formatter 2018-01-21 02:04:36 -08:00
Dale Hui 012f051f2e Use cached badges when possible and make badges reflect the state of the master branch 2018-01-21 01:40:41 -08:00
Dale Hui 8684cad808 Update README to reflect the current status of the sqlite driver 2018-01-21 00:20:10 -08:00
Dale Hui 15611d4559 Fix cli builds
- Removed sqlite3 support from binaries
        - It was never working anyways: https://github.com/mattes/migrate/issues/244
    - Don't use cgo as is requires a cross-compiler to build for other platforms
        - cgo was originally added for this reason: https://github.com/mattes/migrate/pull/238#issuecomment-306291988
2018-01-20 23:59:36 -08:00
Dale Hui d542e144dc Manually fetch github.com/kshvakov/clickhouse 2018-01-20 10:30:57 -08:00
Dale Hui f8fd159fbc Fork packagecloud releases 2018-01-20 02:10:22 -08:00
Dale Hui 2eb9570b52 Build release binaries (using G 1.9) and upload to Github using new key 2018-01-20 01:49:02 -08:00
Dale Hui f5ae4b42db Support creating migrations using sequences in addition to timestamps 2018-01-20 00:29:06 -08:00
Dale Hui 24dd870559 MySQL and Postgres db driver test improvements
- Log unexpected connection errors while waiting for the docker image to start
    - Don't leave connections open
2018-01-20 00:11:35 -08:00
Dale Hui 97ef1b8393 Remove redundant delay since the readyFn is sufficient 2018-01-20 00:11:35 -08:00
Dale Hui bdb332a5f8 Remove unused slowReader 2018-01-19 22:38:32 -08:00
Dale Hui 8cbdc28e66 Source code needs to be kept for goveralls to use but removed before caching in travis-ci 2018-01-19 20:53:20 -08:00
Dale Hui 4ebf080da8 Speed up travis-ci builds by caching dependencies 2018-01-19 20:32:15 -08:00
Dale Hui 53606261bf Make coverage directory configurable 2018-01-19 20:32:15 -08:00
Dale Hui 50d049d5f5 Only use coverage profiles from test runs to create combined coverage profile.
- Prevent issue where the combined coverage profile is combined with itself
2018-01-19 18:04:46 -08:00
Daniel CormierandDale Hui 395391525b Changed source.Up from untyped to a typed const
Changed source.Up from being an untyped string const to being a typed source.Direction const.
2018-01-19 16:24:25 -08:00
Dale Hui 59f3f6d6c2 Increase test timeout from default 10m to 20m since some tests are slow as molasses in January
- Example build being killed by Go's test binary: https://travis-ci.org/golang-migrate/migrate/jobs/331027558
2018-01-19 15:42:02 -08:00
Dale Hui 12cc536541 Drop support for Go 1.7
- Only the latest 2 major release version is supported. https://golang.org/doc/devel/release.html#policy
2018-01-19 15:36:13 -08:00
Dale Hui 51ec421ad8 Remove debugging print from mysql driver test 2018-01-19 15:13:07 -08:00
Dale Hui 1cf095c51d Update more references to github.com/mattes/migrate 2018-01-19 14:59:27 -08:00
Dale Hui 7c9a261dc0 Fix MySQL docker image connection testing 2018-01-19 14:50:31 -08:00
Dale Hui 31367fbc97 Add tip about dealing with docker client API version issues 2018-01-19 14:31:56 -08:00
Dale Hui 7159f40304 Update all *DockerContainer receivers to handle nil
- Use errors.New() instead of fmt.Errorf() if no formatter string is needed
    - Spacing changes due to `go fmt`
2018-01-19 14:22:55 -08:00
Dale Hui 042e307bef Ignore vendor dir 2018-01-19 10:57:17 -08:00
Dale Hui f584949470 Update imports to reference fork.
Thanks `make rewrite-import-paths`!
2018-01-19 10:56:55 -08:00
Dale Hui 303cd74e5f Fix Docker install on TravisCi by allowing the docker-ce package to be downgraded
- Started happening since TravisCI updated docker-ce https://docs.travis-ci.com/user/build-environment-updates/2017-12-12/
    - Can't use v17.09 since the git tag doesn't exist yet
2018-01-19 02:28:41 -08:00
Dale Hui 3b61abf20a Update some examples and badges in README to reference new fork 2018-01-19 02:07:34 -08:00
Matthias KadenbachandGitHub d23f71b03c Merge pull request #305 from zikes/master
use DELETE FROM to avoid schema changes within transaction
2017-12-08 13:48:26 -08:00
Jason Hutchinson c156f5f3a1 replace TRUNCATE with DELETE FROM to avoid schema changes within transaction 2017-11-02 16:59:26 -05:00
Teddy Schmitz 155a8b241c Use db.Conn to fix postgres lock/unlock 2017-10-27 10:56:10 +08:00
Matthias KadenbachandGitHub 69472d5f5c Merge pull request #301 from shaneodonnell/master
Issue 247: Removed '(always included)'
2017-10-24 11:00:00 -07:00
Shane O'Donnell 4ac3ee9d75 Removed '(always included)' 2017-10-23 13:27:24 -04:00
Matthias KadenbachandGitHub 5b98c13eff Merge pull request #281 from ashoda/support_multiple_statements
(Spanner) Add support for multiple statements in one migration step
2017-10-09 11:03:59 -07:00
Ashod Ayanyan 28cb030ffe Add support for multiple DDL/SQL statements in one migration file 2017-09-18 15:43:46 -07:00
Matthias KadenbachandGitHub be1b075605 add go 1.9 for tests 2017-09-18 14:55:49 -07:00
Matthias KadenbachandGitHub 62cddb9e74 Merge pull request #278 from JensRantil/cassandra-auth
feat(cassandra): support for user/pw authentication
2017-09-18 10:08:58 -07:00
Matthias KadenbachandGitHub eabba5b316 Merge pull request #277 from JensRantil/cleanup
Various `gofmt`ing
2017-09-18 10:08:34 -07:00
Jens Rantil a5e584ce22 feat(cassandra): support for user/pw authentication 2017-09-17 13:32:00 +02:00
Jens Rantil d25732a53a sty: avoid unnecessary parenthesis
Golang best-practise. Cleanup made by `gofmt`.
2017-09-17 13:30:08 +02:00
Jens Rantil 701bd68731 style(cassandra): whitespace fixes
According to `gofmt`.
2017-09-17 13:26:30 +02:00
Jens Rantil 51acd3f3ad style: order imports lexicographically
Best-practise according to `gofmt`.
2017-09-17 13:26:23 +02:00
Matthias KadenbachandGitHub 804f9c2d4f Merge pull request #273 from maknahar/master
Fixed a couple of typos
2017-08-14 14:01:06 -07:00
maknahar 81c0a17f20 Fixed a couple of typos 2017-08-14 17:55:33 +05:30
Matthias KadenbachandGitHub 2570d5866d Merge pull request #272 from twrobel3/bugfix/cockroachTxnSig
Update cockroachDB Txn signatures
2017-08-11 17:11:41 -07:00
Taylor Wrobel 2c0f4f588e Update cockroachDB Txn signatures
cockroach-go made a backwards-incompatible change to their transaction
function signatures, which was causing the cockroach instrumentation
to fail.

Updates the signature to match the cockraoch-go change.
2017-08-10 22:56:41 -07:00
Matthias KadenbachandGitHub 8b5365e62c Merge pull request #267 from nicolas33/FAQ-extension
Faq extension
2017-07-30 15:05:05 -07:00
Nicolas Sebrecht d837bcc60c FAQ: add required extension
Github-fix: #265
Signed-off-by: Nicolas Sebrecht <nicolas.sebrecht@1001pneus.fr>
2017-07-28 14:30:10 +02:00
Nicolas Sebrecht bfe502654c FAQ: update format from dos to unix
Signed-off-by: Nicolas Sebrecht <nicolas.sebrecht@1001pneus.fr>
2017-07-28 14:24:46 +02:00
Matthias KadenbachandGitHub 6632d110b6 Merge pull request #264 from twrobel3/feature/migrations-readme
Fill out MIGRATIONS.md with migration format and best practice notes
2017-07-25 23:05:42 -07:00
Taylor Wrobel 341c35b1b8 Fill out MIGRATIONS.md with migration format and best practice notes 2017-07-19 00:18:02 -07:00
Matthias KadenbachandGitHub 8e6084bc9c Merge pull request #263 from twrobel3/feature/crdb-readme
Add cockroachdb README and add force lock functionality
2017-07-12 08:48:03 -07:00
Taylor Wrobel 40a40613cb Add cockroachdb README and add ForceLock functionality
Adds a readme to the cockroachdb database package, following the
postgres readme style.

Also adds the ability to force acquisition of the migration lock
via a connect URL parameter/WithInstance config, to allow for fixing
cases where an implementation error causes the schema lock to not
be released.

Lastly, tweaks the CLI readme to include information on building a
CLI for databases other than postgres.
2017-07-11 21:52:02 -07:00
Watanabe MasaruandMatthias Kadenbach 23a2745a3a fix spanner ensure migration table 2017-07-09 10:57:10 -07:00
Matthias KadenbachandGitHub 88c6a0a752 Merge pull request #253 from kshvakov/master
Add ClickHouse driver
2017-07-08 10:11:23 -07:00
Kirill ShvakovandGitHub 1f62e3f142 Merge branch 'master' into master 2017-07-08 13:50:51 +03:00
Kirill Shvakov 8424e946e4 remove deprecated group from Travis file 2017-07-08 13:49:23 +03:00
Matthias KadenbachandGitHub 66a0448698 Merge pull request #255 from twrobel3/feature/cockroachdb
Add CockroachDB Support
2017-07-08 00:26:12 -07:00
Taylor Wrobel 2cab359222 Remove debugging output from cockroachDB tests 2017-07-07 23:13:55 -07:00
Taylor Wrobel aebc0c3cdd Merge port mapping helper functions into one 2017-07-07 23:12:07 -07:00
Taylor Wrobel c2925c40c7 Add tests for CockroachDB 2017-07-07 23:12:07 -07:00
Taylor Wrobel 88115dedbf Support running docker with commands and handling multiple exposed ports
Adds the ability to specify a series of commands to run as part
of the docker image execution, and allows for retrieving a mapping
of an exposed via the port bound within the container.
2017-07-07 23:12:07 -07:00
Taylor Wrobel 1d8a881941 Add CockroachDB Support
Adds support for CockroachDB.  Cockroach uses the postges wire
protocol and has a large amount of common SQL functionality shared
with Postgres, so much of the postgres code was able to be copied
and modified.

Since the protocol is used in determining the driver, and the
Postgres protocol is also used by Cockroach, new connect string
prefixes were added: cockroach:// cockroachdb:// and
crdb-postgres://.  These fake protocol strings are replaced in
the connect function with the correct `postgres://` protocol.

TODO: Tests needed (Cockroach has a docker image, so this shouldn't
be too hard)
2017-07-07 23:12:07 -07:00
Matthias KadenbachandGitHub 8439d7195f Merge pull request #262 from twrobel3/feature/fixTravisBuild
Update docker dependency version to fix Tavis builds
2017-07-07 22:45:46 -07:00
Matthias KadenbachandGitHub f0f8b2c85c Merge pull request #260 from martin-magakian/patch-1
Update README.md
2017-07-03 22:21:55 -07:00
Martin MagakianandGitHub 35af42a0dd Update README.md 2017-07-03 15:45:57 +02:00
Martin MagakianandGitHub 56a56527c6 Update README.md
Late edit I promise...
2017-07-03 15:45:41 +02:00
Martin MagakianandGitHub 1485e567f1 Update README.md
Some format :-p
2017-07-03 15:35:43 +02:00
Martin MagakianandGitHub 180c57747d Update README.md
Forget package
2017-07-03 15:34:43 +02:00
Martin MagakianandGitHub 2361a7aa24 Update README.md
Code snippet don't compile.
It's missing a step (see https://github.com/mattes/migrate/blob/master/source/go-bindata/go-bindata_test.go)
2017-07-03 15:29:15 +02:00
Taylor Wrobel 4f9422ec85 Update docker dependency version to fix Tavis builds
Tavis has updated its docker images, and as part of the update,
no longer includes support for the (deprevated) docker-engine
package, moving instead to docker-ce.

See https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch
for details of the package change.

This updates the travis configuration to pin the docker library
version to 17.05.0 and installs the corresponding version of
docker-ce in the install script.
2017-06-24 10:37:08 -07:00
kshvakov 9d17987b8c Travis CI: using the previous Ubuntu Trusty images (https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch) 2017-06-23 12:16:14 +03:00
kshvakov 94056e51cc postgres: fix SetVersion. All requests were executed in different connections so commit/rollback doesn't work correctly 2017-06-23 11:13:54 +03:00
Kirill Shvakov 8cd9761d29 fix: set current database 2017-06-21 23:10:43 +03:00
kshvakov e86f4a201c change include path from github.com/kshvakov to github.com/mattes 2017-06-21 17:24:55 +03:00
kshvakov 9341404de0 move ensureVersionTable to init 2017-06-21 17:14:43 +03:00
kshvakov b031b22c80 Add WithInstance 2017-06-21 17:12:05 +03:00
kshvakov 1440a070c3 Add WithInstance 2017-06-21 17:06:04 +03:00
kshvakov 14636b77fb Add ClickHouse driver 2017-06-21 15:58:46 +03:00
Matthias KadenbachandGitHub da4160ce04 Merge pull request #251 from christianklotz/spanner-driver
Add Spanner driver
2017-06-20 21:52:00 -07:00
Christian Klotz eccc3a26d2 Skip test if SPANNER_DATABASE isn’t set 2017-06-20 17:53:41 +01:00
Christian KlotzandGitHub 51afcccb6b Merge branch 'master' into spanner-driver 2017-06-14 21:02:59 +01:00
Stephanie NgandChristian Klotz 8f7d3efb1a add create command to cli 2017-06-14 21:59:11 +02:00
BenandChristian Klotz 13a818e3b5 Fix typo forgotton => forgotten 2017-06-14 21:59:11 +02:00
Matthias KadenbachandChristian Klotz 5e049b9fb7 enable cgo 2017-06-14 21:59:11 +02:00
maxvwandChristian Klotz 33d5038bd3 adapted the sqlite driver for v3 (#165) 2017-06-14 21:58:37 +02:00
Fuyuan BieandChristian Klotz 39bb76224d Fix parenthesis problem. 2017-06-14 21:57:37 +02:00
Fuyuan BieandChristian Klotz 9632882b69 Use both db name and migration table name for lock hash 2017-06-14 21:57:37 +02:00
Christian Klotz 803598293e Add basic Google Spanner test 2017-06-14 19:46:10 +02:00
Matthias KadenbachandGitHub 50068870a1 Merge pull request #249 from recurza/master
Add create command to CLI
2017-06-10 10:25:06 -07:00
Stephanie Ng 9a19439e31 add create command to cli 2017-06-10 20:20:35 +08:00
Matthias KadenbachandGitHub d8dd1bc427 Merge pull request #248 from biefy/master
Solve lock contention problem when upgrading multiple logical MySQL databases backed by a single physical database.
2017-06-09 17:08:54 -07:00
Matthias KadenbachandGitHub c1134be427 Merge pull request #242 from benmoss/master
Fix typo forgotton => forgotten
2017-06-05 13:11:58 -07:00
Matthias KadenbachandGitHub 2aab3394cf Merge pull request #243 from mattes/mattes-patch-1
enable cgo
2017-06-05 13:11:27 -07:00
Ben 39a5a37822 Fix typo forgotton => forgotten 2017-06-05 12:14:10 -04:00
Jolan MalassigneandChristian Klotz 89879968bb add cassandra driver and function to retrieve networkSettings to get port bound to 9042 2017-06-05 11:03:49 +01:00
Pavlina DrososandChristian Klotz 6ecd671cfc Add code sample for use with existing MySQL client with multistatements=true 2017-06-05 11:02:51 +01:00
MaxandChristian Klotz 2f50c9c52b added file cli/build_sqlite3.go 2017-06-05 11:02:51 +01:00
maxvwandChristian Klotz 386ce008d9 adapted the sqlite driver for v3 (#165) 2017-06-05 11:02:51 +01:00
Christian Klotz 3d6c78875f Add example migrations 2017-06-05 10:50:14 +01:00
Matthias KadenbachandGitHub a9bcd83ff0 enable cgo 2017-06-05 00:59:53 -07:00
Matthias KadenbachandGitHub bf3f0f133e Merge pull request #240 from tinysquare/master
Add cassandra driver
2017-06-04 21:27:48 -07:00
Matthias KadenbachandGitHub 2aa16f5f66 Merge branch 'master' into master 2017-06-04 21:27:38 -07:00
Matthias KadenbachandGitHub 63f788f07e Merge pull request #239 from pdrosos/update-mysql-documentation-for-use-with-existing-db-client
Add documentation for use with existing MySQL client with multiStatements=true
2017-06-04 21:23:57 -07:00
Matthias KadenbachandGitHub c3d61852ee Merge pull request #238 from maxvw/v3-sqlite
adapted the sqlite driver for v3 (#165)
2017-06-04 21:21:55 -07:00
Christian Klotz 2742b9c467 Provide WithInstance method
Also includes some refactor around package naming, treats own repo as first-class and uses alternative package names for third party imports, Google spanner libraries in this case.
2017-06-04 21:53:33 +01:00
Christian Klotz 748ae8f06a Add Spanner driver
Support for Google Cloud Spanner, closes #172. Includes example migrations ported over from the PostgreSQL driver.

In all examples provided by Google I’ve come across, camel-case is used for table names, columns, etc. Hence the examples use this naming convention.
2017-06-04 21:13:49 +01:00
Jolan Malassigne 28524d977c add cassandra driver and function to retrieve networkSettings to get port bound to 9042 2017-05-31 10:05:56 +02:00
Pavlina Drosos b5c2f0f2bb Add code sample for use with existing MySQL client with multistatements=true 2017-05-30 18:52:26 +03:00
Max decee4abb2 added file cli/build_sqlite3.go 2017-05-30 14:06:59 +02:00
maxvw 8b4ce58a1d adapted the sqlite driver for v3 (#165) 2017-05-27 08:58:01 +02:00
Matthias KadenbachandGitHub 3682bcf9d2 Update README.md 2017-05-18 19:47:23 -07:00
Matthias KadenbachandGitHub 857dc4b862 Update README.md 2017-05-18 19:45:53 -07:00
Matthias KadenbachandGitHub 247ca90b33 Merge pull request #234 from augustzf/master
Add command to make CLI example with Go toolchain work
2017-05-18 10:29:15 -07:00
Matthias KadenbachandGitHub 00b22e527d one line 2017-05-18 10:28:39 -07:00
August Z. Flatby 83035ec925 Add command to make CLI example with Go toolchain work 2017-05-18 09:27:39 +02:00
Matthias KadenbachandGitHub 035c07716c Merge pull request #231 from mattes/cli-updates
build cli with all source and database drivers
2017-05-17 19:37:49 -07:00
Matthias Kadenbach 0d9689ab9f go get update 2017-05-17 19:37:36 -07:00
Matthias Kadenbach e696a861ba fix makefile 2017-05-17 19:26:24 -07:00
Matthias Kadenbach 801e84d4d2 fix make deps 2017-05-17 19:21:48 -07:00
Matthias Kadenbach ff7e060269 remove ql 2017-05-17 19:16:08 -07:00
Matthias Kadenbach d4d20346f2 build cli with all source and database drivers 2017-05-17 19:10:45 -07:00
Matthias KadenbachandGitHub 1b48b97aa4 Update README.md 2017-05-16 13:27:30 -07:00
Matthias KadenbachandGitHub d1e665bac7 Merge pull request #227 from fsouza/gcs-source
source/google-cloud-storage: implement the driver
2017-05-16 13:14:34 -07:00
Matthias KadenbachandGitHub dc109fc118 Merge pull request #228 from fsouza/s3-source
source/aws-s3: implement the driver
2017-05-16 13:13:00 -07:00
Matthias KadenbachandGitHub fb8698d89a Merge pull request #223 from Menta2L/master
ql database support
2017-05-16 13:09:22 -07:00
vesco 2821341551 make ExecuteQuery private 2017-05-16 12:07:40 +03:00
Fuyuan BieandGitHub bcdbe0f018 Fix parenthesis problem. 2017-05-15 22:01:45 -07:00
Fuyuan BieandGitHub 5f2a8b7f30 Use both db name and migration table name for lock hash 2017-05-15 21:50:41 -07:00
Matthias KadenbachandGitHub 2831b4c0d8 Merge pull request #222 from OrbotixInc/master
Added support for Redshift.
2017-05-15 18:31:13 -07:00
Francisco Souza c95cbb5931 source/aws-s3: implement the driver 2017-05-14 14:31:40 -04:00
Francisco Souza da89656fdc source/google-cloud-storage: implement the driver 2017-05-14 13:47:01 -04:00
root 4e5dc68b4e ql support 2017-05-12 11:11:16 +03:00
Jonathan Bryant 91ecd3cb6a Added support for Redshift.
* Added driver that wraps the standard PostgreSQL driver
* Added build.
* Added driver to documentation.
2017-05-11 15:47:40 -06:00
Matthias KadenbachandGitHub 448a36c5cb Merge pull request #210 from divan/master
Add missing import to example in README
2017-05-04 11:05:03 -07:00
Ivan DanilukandGitHub ed5a887579 Add import to example.
The example in README with file source doesn't work without importing "github.com/mattes/migrate/source/github" package.
2017-05-04 19:42:12 +02:00
Matthias KadenbachandGitHub e42058f15a Merge pull request #209 from pgmcgee/patch-1
Fix a typo in go-bindata/README.md
2017-05-04 09:33:40 -07:00
Parker McGeeandGitHub a7e1cacd36 Fix a typo
I was browsing this and noticed the typo that is causing syntax highlighting to be off.
2017-05-04 07:44:37 -04:00
Matthias KadenbachandGitHub 734803549d Merge pull request #204 from fln/master
Fix appending multiStatements=true parameter for mysql driver
2017-04-25 13:37:09 -07:00
Julius Kriukas 14f784961f Fix appending multiStatements=true parameter for mysql driver 2017-04-25 22:54:16 +03:00
Matthias Kadenbach 5c8da71dea update travis branch 2017-04-25 12:01:21 -07:00
Matthias Kadenbach a6fe4b73b3 update makefile 2017-04-25 11:59:52 -07:00
Matthias KadenbachandGitHub 256e8f045b Update README.md 2017-04-25 11:13:59 -07:00
Matthias KadenbachandGitHub 2e54bda0e1 Update README.md 2017-04-25 11:06:06 -07:00
Matthias Kadenbach 2ae8e6291c remove v1 migrate compat 2017-04-25 10:37:55 -07:00
Matthias KadenbachandGitHub 651f78cba8 Update README.md 2017-04-21 13:36:03 -07:00
Matthias KadenbachandGitHub d5674ed411 Create README.md 2017-04-21 13:30:59 -07:00
Matthias KadenbachandGitHub 4a2a810c20 Update README.md 2017-04-21 13:29:13 -07:00
Matthias KadenbachandGitHub 365b0565de Update README.md 2017-04-21 13:07:46 -07:00
Matthias KadenbachandGitHub 65d785d560 Update README.md 2017-04-21 13:02:10 -07:00
Matthias KadenbachandGitHub fff688c019 Create README.md 2017-04-21 13:00:19 -07:00
Matthias Kadenbach 6f07ac731b fix go toolchain build 2017-04-21 12:23:01 -07:00
Matthias Kadenbach f00c15d7b0 update go toolchain install 2017-04-20 16:03:43 -07:00
Matthias Kadenbach 1676a902f2 update readme 2017-04-20 15:22:06 -07:00
Matthias KadenbachandGitHub 576d323c52 Merge pull request #200 from Gobonoid/fix/ReadmeUpdate
Updates import path in README example
2017-04-19 10:04:37 -07:00
Gobonoid b4adf71696 Updates import path in README examplee 2017-04-19 11:48:58 +01:00
Matthias KadenbachandGitHub c0703e7a92 Merge pull request #199 from guzart/date-versions
Date-based versions
2017-04-16 12:38:48 -07:00
Arturo Guzman d789d50bd6 Support date-based versions 2017-04-13 20:57:22 -04:00
Matthias KadenbachandGitHub b83285bd59 Merge pull request #194 from Arkq/v3.0-prev
Support for alternative postgres URI scheme
2017-04-10 11:22:50 -07:00
Arkadiusz Bokowy bc93ed027f Support for alternative postgres URI scheme
According to the PostgreSQL documentation (section 32.1.1.2), postgres
library supports two URI schemes: postgresql:// and postgres://

Reference:
https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
2017-04-04 08:29:33 +02:00
Matthias Kadenbach 9f974b8996 Update Readme 2017-03-12 18:00:22 -07:00
Matthias Kadenbach c03b03c909 return status 0 for ErrNoChange #182 2017-03-12 12:33:13 -07:00
Matthias Kadenbach cecce18c51 fix source/file relative paths, ref #178 thanks @matyunin 2017-03-03 12:02:32 -08:00
Matthias Kadenbach 1f37f41ee8 remove cleanpath, fix mysql ensureVersionTable 2017-03-01 11:48:38 -08:00
Matthias Kadenbach 150ac7d708 add mysql custom TLS config
closes https://github.com/mattes/migrate/pull/117
2017-02-28 16:23:29 -08:00
Matthias Kadenbach be1ba9204a add mysql driver, add ENV to docker containers 2017-02-28 15:10:56 -08:00
Matthias Kadenbach 760bc3eb2d add note in readme 2017-02-23 14:50:48 -08:00
Matthias Kadenbach 706f23115f add v1compat 2017-02-23 14:48:44 -08:00
Matthias KadenbachandGitHub 171f9a256d Update FAQ.md 2017-02-23 14:27:04 -08:00
Matthias KadenbachandGitHub e7b9b0d0ed Update README.md 2017-02-23 13:36:34 -08:00
Matthias Kadenbach 5645143baa add example dockerfile 2017-02-23 13:13:02 -08:00
Matthias Kadenbach ec5c0ac3e0 fix version string 2017-02-23 10:36:33 -08:00
Matthias Kadenbach 511f9c1ffe only deploy on version go 1.8 2017-02-23 10:30:18 -08:00
Matthias Kadenbach 6325327ba1 fix dir 2017-02-22 17:58:25 -08:00
Matthias Kadenbach d60d590381 remove escaped new line 2017-02-22 17:48:50 -08:00
Matthias Kadenbach 5aac011e49 remove dups 2017-02-22 17:38:40 -08:00
Matthias Kadenbach 320d573ccd fix sudo whitespace 2017-02-22 17:27:18 -08:00
Matthias Kadenbach c51bef38b6 sudo 2017-02-22 17:15:40 -08:00
Matthias Kadenbach 28e0548cac fix github
see https://github.com/google/go-github/commit/23d6cb9cacb5aa314e93d600fe20a48496a718d4
2017-02-22 16:56:14 -08:00
Matthias Kadenbach d0acf6ecfe build deb package and push to packagecloud 2017-02-22 16:41:11 -08:00
Matthias Kadenbach 4b91f503eb make Force doc clear 2017-02-19 16:22:48 -08:00
Matthias Kadenbach eb02bc77fb allow to force NilVersion 2017-02-19 16:06:11 -08:00
Matthias KadenbachandGitHub 55f54251eb Update FAQ.md 2017-02-19 15:56:45 -08:00
Matthias Kadenbach e64b7cc233 explain dirty version error 2017-02-19 15:51:12 -08:00
Matthias Kadenbach 4bb46a98c6 fix github
see https://github.com/google/go-github/commit/2a4b92075e5c5e42ee0e55819b70f49b02c1ac64
2017-02-19 15:31:57 -08:00
Matthias Kadenbach 53b55288b3 bring back dirty state from v2 2017-02-19 15:15:00 -08:00
Matthias KadenbachandGitHub f576a64f11 Update README.md 2017-02-18 10:41:17 -08:00
Matthias Kadenbach d574676702 drop custom query params 2017-02-17 16:59:47 -08:00
Matthias Kadenbach f45821581c add multi statement test 2017-02-16 22:10:01 -08:00
Matthias KadenbachandGitHub bb6ae0898d Update README.md 2017-02-16 21:13:12 -08:00
Matthias KadenbachandGitHub ea7fc641ba Update README.md 2017-02-16 21:12:33 -08:00
Matthias KadenbachandGitHub 87e6482497 travis build with go 1.8 2017-02-16 20:46:12 -08:00
Matthias KadenbachandGitHub 1e08e21470 update with issues 2017-02-16 20:41:17 -08:00
Matthias Kadenbach ccb40ff3ff Update Readme 2017-02-16 11:29:18 -08:00
Matthias Kadenbach 3291f3f891 add comment 2017-02-16 11:18:02 -08:00
Matthias Kadenbach 2031939bfc allow postgres migrations_table config 2017-02-16 11:06:11 -08:00
Matthias Kadenbach 6394299937 clean cli -path 2017-02-15 11:07:28 -08:00
Matthias Kadenbach 316f6c0cf3 switch to binary search 2017-02-15 10:48:43 -08:00
Matthias Kadenbach 53630f5ff1 add NewWithDatabaseInstance example 2017-02-14 23:25:19 -08:00
Matthias Kadenbach 3feee5958a updated postgres driver, add database.Error struct 2017-02-14 23:12:16 -08:00
Matthias Kadenbach 209c056258 add database lock timeouts in migrate and in tests 2017-02-11 19:15:54 -08:00
Matthias Kadenbach cd6e62049c Update documentation 2017-02-11 11:13:27 -08:00
Matthias Kadenbach 526c0a918e add packagecloud.io badge 2017-02-09 20:09:11 -08:00
Matthias Kadenbach de7ffa60cc update readme 2017-02-09 20:07:04 -08:00
Matthias Kadenbach 81b9791178 move contributing in own file 2017-02-09 20:03:56 -08:00
Matthias Kadenbach c4ec6ac0f7 move CLI usage in own README 2017-02-09 20:00:38 -08:00
Matthias Kadenbach cfe3d2a83e add example for parallelTest 2017-02-09 19:52:50 -08:00
Matthias Kadenbach b6bd4fe02d add comments for source 2017-02-09 19:42:48 -08:00
Matthias Kadenbach 2da976155a add comments for database 2017-02-09 19:04:54 -08:00
Matthias Kadenbach e69e0de9c9 fix old ref to LogString 2017-02-09 18:26:58 -08:00
Matthias Kadenbach cfc9f10a6e add some more comments 2017-02-09 18:23:08 -08:00
Matthias Kadenbach 8ba71b0754 rename nameFromUrl to schemeFromUrl 2017-02-09 18:19:13 -08:00
Matthias Kadenbach 46c16bb8f1 add test for suint 2017-02-09 18:16:16 -08:00
Matthias Kadenbach 1b079a3cda rename Migration.LongString to LogString 2017-02-09 18:06:38 -08:00
Matthias Kadenbach c1b23fee45 add Migrate.Run 2017-02-09 18:04:36 -08:00
Matthias Kadenbach 199678e1bc add code comments for package migrate 2017-02-09 16:59:30 -08:00
Matthias Kadenbach d5daf6f6ae add .godoc.pid to gitignore 2017-02-09 13:47:28 -08:00
Matthias Kadenbach 03d8d3233f fix line break in makefile 2017-02-09 12:03:03 -08:00
Matthias Kadenbach 5c6a21b014 add make docs 2017-02-09 11:59:49 -08:00
Matthias Kadenbach 3965df69d5 fix indent in makefile 2017-02-08 20:13:17 -08:00
Matthias Kadenbach f47436e158 add make release 2017-02-08 19:54:50 -08:00
Matthias Kadenbach f474e4cae1 travis builds dep tree info 2017-02-08 19:44:12 -08:00
Matthias Kadenbach 7979fdb832 add list-external-deps 2017-02-08 19:34:53 -08:00
Matthias Kadenbach b3f6bc47bb update readme with dev, testing, and contribution section 2017-02-08 18:27:12 -08:00
Matthias Kadenbach e6163082d7 add restore-import-paths and rewrite-import-paths 2017-02-08 18:01:01 -08:00
Matthias Kadenbach fe1b5c06eb add go coverage 2017-02-08 16:17:04 -08:00
Matthias Kadenbach 297a4227ef fixing travis 2017-02-08 01:49:57 -08:00
Matthias Kadenbach 9f774d3e6a fix typo in go-bindata README 2017-02-07 23:00:25 -08:00
Matthias Kadenbach f7699f1dec update go-bindata with latest source.Migration helper 2017-02-07 22:59:12 -08:00
Matthias Kadenbach 3889c93b63 add Readme draft 2017-02-07 22:40:45 -08:00
Matthias Kadenbach ee384ed8cc initial version 3.0.0 preview
hard work
2017-02-07 22:01:29 -08:00
Matthias Kadenbach e92f038f40 skip neo4j tests 2017-01-23 13:54:25 -08:00
Matthias KadenbachandGitHub 0b3426bf5d Merge pull request #25 from daddykotex/master
Ensure that interrupt signals are not relayed to the interrupt channel when the migration is over
2017-01-23 13:37:42 -08:00
Matthias Kadenbach 0861ff957d Update Readme 2017-01-23 13:36:22 -08:00
Matthias Kadenbach 32607ef715 Merge branch 'carlosforero-neo4j-driver' 2017-01-23 13:32:56 -08:00
Matthias Kadenbach c643ffd6d6 fix merge conflicts 2017-01-23 13:32:48 -08:00
Matthias KadenbachandGitHub b8b3883950 Merge pull request #101 from bfosberry/fixing-unsafe-cast
Fixing unsafe typecast
2017-01-23 13:26:59 -08:00
Matthias KadenbachandGitHub 3547fc54cb Merge pull request #144 from benma/patch-1
Update version.go
2017-01-19 09:32:29 -08:00
benmaandGitHub 64c41a8af0 Update version.go 2017-01-19 14:11:19 +01:00
Matthias KadenbachandGitHub 4191ac9cfe Merge pull request #142 from goenning/master
allow use of big version numbers
2017-01-18 15:48:54 -08:00
goenning f1738e78db allow use of big version numbers 2017-01-18 22:19:02 +00:00
Matthias KadenbachandGitHub de494e4b98 Merge pull request #128 from oliverkofoed/master
Add support for ScyllaDB (remove cql counter use)
2017-01-10 09:26:17 -08:00
Matthias KadenbachandGitHub 585509f3bc Merge pull request #138 from mattes/add-mongo-to-readme
add mongodb to readme
2017-01-09 10:43:35 -08:00
Matthias KadenbachandGitHub f630dc5971 Update README.md 2017-01-09 10:43:22 -08:00
Matthias KadenbachandGitHub 9a8f78ba3b Merge pull request #137 from drboyer/updown-colors
Change colors for up and down migration arrows
2017-01-09 10:39:55 -08:00
Matthias KadenbachandGitHub 63c6168c7f Merge pull request #136 from dimag-jfrog/prepare_pr_2_rebase
Driver for MongoDB (New)
2017-01-09 10:38:56 -08:00
dimag 1993f0783b Merge remote-tracking branch 'fork/prepare_pr_2_rebase' into prepare_pr_2_rebase
# Conflicts:
#	docker-compose.yml
2017-01-07 16:44:36 +02:00
dimag 70238f448b - Reverted repository from dimag-jfrog to mattes to prepare for pull request 2017-01-07 16:36:25 +02:00
dimag 914c38e379 - Fixed test with not-exported method identified as missing: unified the 2 errors into one
- Fixed error message formatting
- Ensuring uniqueness of the version field (added unique index)
2017-01-07 16:34:06 +02:00
dimag 0f28b01383 specifying version of mongo 2017-01-07 16:34:06 +02:00
dimag 6909594c0b Checking non-exported migration method 2017-01-07 16:33:10 +02:00
dimag 583921e8bf Added README.md 2017-01-07 16:33:10 +02:00
dimag 61ecf2c096 - Refactored project structure to be consistent with original project
- Added docker support
2017-01-07 16:33:10 +02:00
dimag 0eecb74230 Added explicit migration method signature validation 2017-01-07 16:30:40 +02:00
dimag b7b75f3c59 - registering method receiver directly for the driver
- currently each driver contains only a single method receiver:
   - enforcing method receiver pre-registration on go methods driver initialization
   - Method receiver name can be removed from files format
- passing the DbName parameter inside the method receiver for the go methods driver
2017-01-07 16:30:40 +02:00
dimag fe48ddfdcc - redefined mongo db driver to be autonomous and not depending on specific methods
- defined methods receivers registration by name and change the migration files format to include them
- added extensive testing
2017-01-07 16:30:40 +02:00
dimag 70b23c3f84 Relaxed constratint that method receiver has to be of Driver interface. 2017-01-07 16:30:40 +02:00
dimag 223908f9d2 - Reset all changes to the upstream branch
- Changed logic not to use custom filename parser:
  -Supporting up and down files only, no both direction files
  -Using method names as is
- Added complete test to the mongo db migration scenarios
2017-01-07 16:30:40 +02:00
dimag 1838852d4d Changes that interfere with the Migrate open source code:
- Added go methods migrator, mongo db template: different from the usual driver
model.
- Added support for bidirectional files (for go methods), appending _up or _down upon context
- Added DriverWithFilnameParser for providing custom filename parser functionality that knows to parse bi-directional file names.
2017-01-07 16:29:19 +02:00
dimag 175550643c changed import paths to point to dimag-jfrog 2017-01-07 16:24:14 +02:00
Matthias KadenbachandGitHub eb0eec4471 Merge pull request #135 from ansel1/master
Avoid DDL when checking for versions table #134
2017-01-06 14:24:59 -08:00
Russ Egan da6c505f41 Avoid DDL when checking for versions table #134 2017-01-06 15:52:54 -05:00
Matthias KadenbachandGitHub 966ba843e0 Merge pull request #133 from wlcx/add-ql-driver
Add Ql driver
2017-01-05 14:04:34 -08:00
Sam Willcocks 5773f75698 Add Ql driver
Adds a driver for the embedded Ql database (https://github.com/cznic/ql)
2017-01-05 15:09:41 +00:00
163 changed files with 2050 additions and 5737 deletions
+1 -1
View File
@@ -25,7 +25,7 @@ e.g. s3, github, go-bindata, gcs, file
Obtained by running: `migrate -help`
**Loaded Database Drivers**
e.g. spanner, stub, datastore, cockroachdb, crdb-postgres, postgres, postgresql, pgx, redshift, cassandra, cockroach, mysql
e.g. spanner, stub, clickhouse, cockroachdb, crdb-postgres, postgres, postgresql, pgx, redshift, cassandra, cockroach, mysql
Obtained by running: `migrate -help`
**Go Version**
-9
View File
@@ -1,9 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1280" height="640" viewBox="0 0 1280 640" role="img" aria-label="migrate">
<rect width="1280" height="640" fill="#0A0A0A"/>
<svg x="96" y="215" width="210" height="210" viewBox="0 0 67 67"><path d="M22.21 67V44.6369H0V67H22.21Z" fill="#fff"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z" fill="#fff"/><path d="M22.21 0H0V22.3184H22.21V0Z" fill="#fff"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z" fill="#fff"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z" fill="#fff"/></svg>
<text x="378" y="276" font-family="Inter,system-ui,-apple-system,sans-serif" font-size="78" font-weight="800" letter-spacing="-2" fill="#ffffff">migrate</text>
<text x="378" y="322" font-family="Inter,system-ui,sans-serif" font-size="30" fill="#ffffff" opacity=".66">Database migrations. CLI and Golang library.</text>
<rect x="378" y="338" width="806" height="3" rx="1.5" fill="#ffffff" opacity=".9"/>
<text x="378" y="390" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">github.com/hanzoai</text>
<text x="1184" y="390" text-anchor="end" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">hanzo.ai</text>
</svg>

Before

Width:  |  Height:  |  Size: 1.3 KiB

+97
View File
@@ -0,0 +1,97 @@
name: CI
on:
push:
pull_request:
jobs:
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
test:
runs-on: ubuntu-latest
strategy:
matrix:
go: ["1.16.x", "1.17.x"]
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Run test
run: make test COVERAGE_DIR=/tmp/coverage
- name: Send goveralls coverage
uses: shogo82148/actions-goveralls@v1
with:
path-to-profile: /tmp/coverage/combined.txt
flag-name: Go-${{ matrix.go }}
parallel: true
check-coverage:
name: Check coverage
needs: [test]
runs-on: ubuntu-latest
steps:
- uses: shogo82148/actions-goveralls@v1
with:
parallel-finished: true
goreleaser:
name: Release a new version
needs: [lint, test]
runs-on: ubuntu-latest
environment: GoReleaser
# This job only runs when
# 1. When the previous `lint` and `test` jobs has completed successfully
# 2. When the repository is not a fork, i.e. it will only run on the official golang-migrate/migrate
# 3. When the workflow is triggered by a tag with `v` prefix
if: ${{ success() && github.repository == 'golang-migrate/migrate' && startsWith(github.ref, 'refs/tags/v') }}
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0
- uses: ruby/setup-ruby@v1
with:
ruby-version: 2.7
- uses: actions/setup-go@v2
with:
go-version: "1.17.x"
- uses: docker/setup-qemu-action@v1
- uses: docker/setup-buildx-action@v1
- uses: docker/login-action@v1
with:
username: golangmigrate
password: ${{ secrets.DOCKERHUB_TOKEN }}
- run: echo "SOURCE=$(make echo-source)" >> $GITHUB_ENV
- run: echo "DATABASE=$(make echo-database)" >> $GITHUB_ENV
- uses: goreleaser/goreleaser-action@v2
with:
version: latest
args: release --rm-dist
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: gem install package_cloud
- run: package_cloud push golang-migrate/migrate/ubuntu/bionic dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/ubuntu/focal dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/buster dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
- run: package_cloud push golang-migrate/migrate/debian/bullseye dist/migrate.linux-amd64.deb
env:
PACKAGECLOUD_TOKEN: ${{ secrets.PACKAGECLOUD_TOKEN }}
-1
View File
@@ -8,4 +8,3 @@ vendor/
.vscode/
.idea
dist/
migrate
+18 -29
View File
@@ -1,37 +1,26 @@
version: "2"
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
timeout: 5m
linters:
enable:
#- golint
- interfacer
- unconvert
#- dupl
- goconst
- gofmt
- misspell
- unparam
- nakedret
- prealloc
- revive
- unconvert
- unparam
settings:
misspell:
locale: US
revive:
rules:
- name: redundant-build-tag
exclusions:
generated: lax
rules:
- path: (.+)\.go$
text: G104
paths:
- third_party$
- builtin$
- examples$
#- gosec
linters-settings:
misspell:
locale: US
issues:
max-issues-per-linter: 0
max-same-issues: 0
formatters:
enable:
- gofmt
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
max-issues-per-linter: 0
exclude-use-default: false
exclude:
# gosec: Duplicated errcheck checks
- G104
-1
View File
@@ -1 +0,0 @@
LLM.md
-1
View File
@@ -1 +0,0 @@
LLM.md
+2 -2
View File
@@ -1,4 +1,4 @@
FROM golang:1.26.4-alpine3.21 AS builder
FROM golang:1.16-alpine3.13 AS builder
ARG VERSION
RUN apk add --no-cache git gcc musl-dev make
@@ -15,7 +15,7 @@ COPY . ./
RUN make build-docker
FROM alpine:3.21
FROM alpine:3.13
RUN apk add --no-cache ca-certificates
+2 -2
View File
@@ -1,4 +1,4 @@
FROM alpine:3.19
FROM alpine:3.13
RUN apk add --no-cache ca-certificates
@@ -8,4 +8,4 @@ RUN ln -s /usr/local/bin/migrate /usr/bin/migrate
RUN ln -s /usr/local/bin/migrate /migrate
ENTRYPOINT ["migrate"]
CMD ["--help"]
CMD ["--help"]
+2 -2
View File
@@ -50,7 +50,7 @@
and whenever we want, not just once at the beginning of all tests.
#### Can I maintain my driver in my own repository?
Yes, technically thats possible. We want to encourage you to contribute your driver to this repository though.
Yes, technically thats possible. We want to encourage you to contribute your driver to this respository though.
The driver's functionality is dictated by migrate's interfaces. That means there should really
just be one driver for a database/ source. We want to prevent a future where several drivers doing the exact same thing,
just implemented a bit differently, co-exist somewhere on GitHub. If users have to do research first to find the
@@ -65,7 +65,7 @@
and then "force" the expected version.
#### What happens if two programs try and update the database at the same time?
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations on
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations at the same time
the same database at the same time. For example, the MySQL driver uses the `GET_LOCK` function, while the Postgres driver uses
the `pg_advisory_lock` function.
+4 -4
View File
@@ -1,7 +1,7 @@
# Getting started
Before you start, you should understand the concept of forward/up and reverse/down database migrations.
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases).
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases)
## Create migrations
Create some migrations using migrate CLI. Here is an example:
@@ -10,7 +10,7 @@ migrate create -ext sql -dir db/migrations -seq create_users_table
```
Once you create your files, you should fill them.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created their migration later gets it merged to the repository first.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created his migration later gets it merged to the repository first.
Developers and Teams should keep an eye on such cases (especially during code review).
[Here](https://github.com/golang-migrate/migrate/issues/179#issuecomment-475821264) is the issue summary if you would like to read more.
@@ -28,9 +28,9 @@ migrate -database YOUR_DATABASE_URL -path PATH_TO_YOUR_MIGRATIONS up
Just add the code to your app and you're ready to go!
Before committing your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
Before commiting your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
(e.g. if you created a table in a migration but reverse migration did not delete it, you will encounter an error when running the forward migration again)
It's also worth checking your migrations in a separate, containerized environment. You can find some tools at the [end of this document](#further-reading).
It's also worth checking your migrations in a separate, containerized environment. You can find some tools in the end of this document.
**IMPORTANT:** If you would like to run multiple instances of your app on different machines be sure to use a database that supports locking when running migrations. Otherwise you may encounter issues.
-39
View File
@@ -1,39 +0,0 @@
# Hanzo Golang Migrate
## Overview
Go module: github.com/golang-migrate/migrate/v4
## Tech Stack
- **Language**: Go
## Build & Run
```bash
go build ./...
go test ./...
```
## Structure
```
golang-migrate/
CONTRIBUTING.md
Dockerfile
Dockerfile.circleci
Dockerfile.github-actions
FAQ.md
GETTING_STARTED.md
LICENSE
MIGRATIONS.md
Makefile
README.md
SECURITY.md
cli/
cmd/
database/
dktesting/
```
## Key Files
- `README.md` -- Project documentation
- `go.mod` -- Go module definition
- `Makefile` -- Build automation
- `Dockerfile` -- Container build
+1 -1
View File
@@ -1,5 +1,5 @@
SOURCE ?= file go_bindata github github_ee bitbucket aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb yugabytedb datastore mongodb sqlserver firebird neo4j pgx pgx5 rqlite
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver firebird neo4j pgx
DATABASE_TEST ?= $(DATABASE) sqlite sqlite3 sqlcipher
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
+10 -22
View File
@@ -1,13 +1,11 @@
<p align="center"><img src=".github/hero.svg" alt="migrate" width="880"></p>
[![GitHub Workflow Status (branch)](https://img.shields.io/github/actions/workflow/status/golang-migrate/migrate/ci.yaml?branch=master)](https://github.com/golang-migrate/migrate/actions/workflows/ci.yaml?query=branch%3Amaster)
[![GitHub Workflow Status (branch)](https://img.shields.io/github/workflow/status/golang-migrate/migrate/CI/master)](https://github.com/golang-migrate/migrate/actions/workflows/ci.yaml?query=branch%3Amaster)
[![GoDoc](https://pkg.go.dev/badge/github.com/golang-migrate/migrate)](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)
[![Coverage Status](https://img.shields.io/coveralls/github/golang-migrate/migrate/master.svg)](https://coveralls.io/github/golang-migrate/migrate?branch=master)
[![packagecloud.io](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/golang-migrate/migrate?filter=debs)
[![Docker Pulls](https://img.shields.io/docker/pulls/migrate/migrate.svg)](https://hub.docker.com/r/migrate/migrate/)
![Supported Go Versions](https://img.shields.io/badge/Go-1.24%2C%201.25-lightgrey.svg)
![Supported Go Versions](https://img.shields.io/badge/Go-1.16%2C%201.17-lightgrey.svg)
[![GitHub Release](https://img.shields.io/github/release/golang-migrate/migrate.svg)](https://github.com/golang-migrate/migrate/releases)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate/v4)](https://goreportcard.com/report/github.com/golang-migrate/migrate/v4)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate)](https://goreportcard.com/report/github.com/golang-migrate/migrate)
# migrate
@@ -26,26 +24,23 @@ Forked from [mattes/migrate](https://github.com/mattes/migrate)
Database drivers run migrations. [Add a new database?](database/driver.go)
* [PostgreSQL](database/postgres)
* [PGX v4](database/pgx)
* [PGX v5](database/pgx/v5)
* [PGX](database/pgx)
* [Redshift](database/redshift)
* [Ql](database/ql)
* [Cassandra / ScyllaDB](database/cassandra)
* [Cassandra](database/cassandra)
* [SQLite](database/sqlite)
* [SQLite3](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [SQLCipher](database/sqlcipher)
* [MySQL / MariaDB](database/mysql)
* [MySQL/ MariaDB](database/mysql)
* [Neo4j](database/neo4j)
* [MongoDB](database/mongodb)
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
* [Google Cloud Spanner](database/spanner)
* [CockroachDB](database/cockroachdb)
* [YugabyteDB](database/yugabytedb)
* [Datastore](database/datastore)
* [ClickHouse](database/clickhouse)
* [Firebird](database/firebird)
* [MS SQL Server](database/sqlserver)
* [rqlite](database/rqlite)
### Database URLs
@@ -73,9 +68,7 @@ $
Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go)
* [Filesystem](source/file) - read from filesystem
* [io/fs](source/iofs) - read from a Go [io/fs](https://pkg.go.dev/io/fs#FS)
* [Go-Bindata](source/go_bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata))
* [pkger](source/pkger) - read from embedded binary data ([markbates/pkger](https://github.com/markbates/pkger))
* [GitHub](source/github) - read from remote GitHub repositories
* [GitHub Enterprise](source/github_ee) - read from remote GitHub Enterprise repositories
* [Bitbucket](source/bitbucket) - read from remote Bitbucket repositories
@@ -89,7 +82,7 @@ Source drivers read migrations from local or remote sources. [Add a new source?]
* Handles ctrl+c (SIGINT) gracefully.
* No config search paths, no config files, no magic ENV var injections.
[CLI Documentation](cmd/migrate) (includes CLI install instructions)
__[CLI Documentation](cmd/migrate)__
### Basic usage
@@ -113,7 +106,7 @@ $ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
* Uses `io.Reader` streams internally for low memory overhead.
* Thread-safe and no goroutine leaks.
__[Go Documentation](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)__
__[Go Documentation](https://godoc.org/github.com/golang-migrate/migrate)__
```go
import (
@@ -147,7 +140,7 @@ func main() {
m, err := migrate.NewWithDatabaseInstance(
"file:///migrations",
"postgres", driver)
m.Up() // or m.Steps(2) if you want to explicitly set the number of migrations to run
m.Steps(2)
}
```
@@ -173,11 +166,6 @@ Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-file
[Best practices: How to write migrations.](MIGRATIONS.md)
## Coming from another db migration tool?
Check out [migradaptor](https://github.com/musinit/migradaptor/).
*Note: migradaptor is not affiliated or supported by this project*
## Versions
Version | Supported? | Import | Notes
-16
View File
@@ -1,16 +0,0 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| master | :white_check_mark: |
| 4.x | :white_check_mark: |
| 3.x | :x: |
| < 3.0 | :x: |
## Reporting a Vulnerability
We prefer [coordinated disclosures](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure). To start one, create a GitHub security advisory following [these instructions](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability)
Please suggest potential impact and urgency in your reports.
+11 -15
View File
@@ -7,7 +7,7 @@
[Release Downloads](https://github.com/golang-migrate/migrate/releases)
```bash
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$os-$arch.tar.gz | tar xvz
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
```
### MacOS
@@ -27,8 +27,8 @@ $ scoop install migrate
### Linux (*.deb package)
```bash
$ curl -fsSL https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo gpg --dearmor -o /etc/apt/keyrings/migrate.gpg
$ echo "deb [signed-by=/etc/apt/keyrings/migrate.gpg] https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
$ echo "deb https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ apt-get update
$ apt-get install -y migrate
```
@@ -63,7 +63,7 @@ $ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@l
to build the cli for use with other databases, replace the `postgres` build tag
with the appropriate database tag(s) for the databases desired. The tags
correspond to the names of the sub-packages underneath the
[`database`](../../database) package.
[`database`](../database) package.
1. Similarly to the database build tags, if you need to support other sources, use the appropriate build tag(s).
1. Support for build constraints will be removed in the future: https://github.com/golang-migrate/migrate/issues/60
1. For versions of Go 1.15 and lower, [make sure](https://github.com/golang-migrate/migrate/pull/257#issuecomment-705249902) you're not installing the `migrate` CLI from a module. e.g. there should not be any `go.mod` files in your current directory or any directory from your current directory to the root
@@ -86,18 +86,14 @@ Options:
-help Print usage
Commands:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] [-tz] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string. Note: migrations with the same time cause "duplicate migration version" error.
Use -tz option to specify the timezone that will be used when generating non-sequential migrations (defaults: UTC).
create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string.
goto V Migrate to version V
up [N] Apply all or N up migrations
down [N] [-all] Apply all or N down migrations
Use -all to apply all down migrations
drop [-f] Drop everything inside database
Use -f to bypass confirmation
down [N] Apply all or N down migrations
drop Drop everything inside database
force V Set version V but don't run migration (ignores dirty state)
version Print current migration version
```
@@ -131,7 +127,7 @@ $ migrate -database "$MY_MIGRATE_DATABASE"
Check out https://stedolan.github.io/jq/
```bash
$ migrate -database "$(cat config.json | jq -r '.database')"
$ migrate -database "$(cat config.json | jq '.database')"
```
### YAML files
+3 -9
View File
@@ -1,17 +1,12 @@
# Cassandra / ScyllaDB
# Cassandra
* `Drop()` method will not work on Cassandra 2.X because it rely on
* Drop command will not work on Cassandra 2.X because it rely on
system_schema table which comes with 3.X
* Other methods should work properly but are **not tested**
* Other commands should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
**ScyllaDB**
* No additional configuration is required since it is a drop-in replacement for Cassandra.
* The `Drop()` method` works for ScyllaDB 5.1
## Usage
`cassandra://host:port/keyspace?param1=value&param2=value2`
@@ -25,7 +20,6 @@ system_schema table which comes with 3.X
| `consistency` | ALL | Migration consistency
| `protocol` | | Cassandra protocol version (3 or 4)
| `timeout` | 1 minute | Migration timeout
| `connect-timeout` | 600ms | Initial connection timeout to the cluster |
| `username` | nil | Username to use when authenticating. |
| `password` | nil | Password to use when authenticating. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
+12 -14
View File
@@ -3,16 +3,18 @@ package cassandra
import (
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/gocql/gocql"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/hashicorp/go-multierror"
)
func init() {
@@ -131,14 +133,6 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
}
cluster.Timeout = timeout
}
if len(u.Query().Get("connect-timeout")) > 0 {
var connectTimeout time.Duration
connectTimeout, err = time.ParseDuration(u.Query().Get("connect-timeout"))
if err != nil {
return nil, err
}
cluster.ConnectTimeout = connectTimeout
}
if len(u.Query().Get("sslmode")) > 0 {
if u.Query().Get("sslmode") != "disable" {
@@ -197,14 +191,14 @@ func (c *Cassandra) Close() error {
}
func (c *Cassandra) Lock() error {
if !c.isLocked.CompareAndSwap(false, true) {
if !c.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (c *Cassandra) Unlock() error {
if !c.isLocked.CompareAndSwap(true, false) {
if !c.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
@@ -229,7 +223,7 @@ func (c *Cassandra) Run(migration io.Reader) error {
return err
}
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -314,7 +308,11 @@ func (c *Cassandra) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -336,7 +334,7 @@ func parseConsistency(consistencyStr string) (consistency gocql.Consistency, err
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("failed to parse consistency \"%s\": %v", consistencyStr, r)
err = fmt.Errorf("Failed to parse consistency \"%s\": %v", consistencyStr, r)
}
}
}()
+1 -17
View File
@@ -25,11 +25,9 @@ var (
// Although Cassandra 2.x is supported by the Apache Foundation,
// the migrate db driver only supports Cassandra 3.x since it uses
// the system_schema keyspace.
// last ScyllaDB version tested is 5.1.11
specs = []dktesting.ContainerSpec{
{ImageName: "cassandra:3.0", Options: opts},
{ImageName: "cassandra:3.11", Options: opts},
{ImageName: "scylladb/scylla:5.1.11", Options: opts},
}
)
@@ -61,20 +59,6 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
@@ -95,7 +79,7 @@ func test(t *testing.T) {
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
+25
View File
@@ -0,0 +1,25 @@
# ClickHouse
`clickhouse://host:port?username=user&password=qwerty&database=clicks&x-multi-statement=true`
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `x-migrations-table-engine`| Engine to use for the migrations table, defaults to TinyLog |
| `x-cluster-name` | Name of cluster for creating `schema_migrations` table cluster wide |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
| `x-multi-statement` | false | Enable multiple statements to be ran in a single migration (See note below) |
## Notes
* The Clickhouse driver does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
* Using the default TinyLog table engine for the schema_versions table prevents backing up the table if using the [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup) tool. If backing up the database with make sure the migrations are run with `x-migrations-table-engine=MergeTree`.
* Clickhouse cluster mode is not officially supported, since it's not tested right now, but you can try enabling `schema_migrations` table replication by specifying a `x-cluster-name`:
* When `x-cluster-name` is specified, `x-migrations-table-engine` also should be specified. See the docs regarding [replicated table engines](https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/replication/#table_engines-replication).
* When `x-cluster-name` is specified, only the `schema_migrations` table is replicated across the cluster. You still need to write your migrations so that the application tables are replicated within the cluster.
@@ -1,43 +1,26 @@
// Package datastore implements the golang-migrate database.Driver interface for
// Hanzo Datastore (a ClickHouse-wire-compatible database engine). It registers
// the "datastore://" URL scheme and connects through the native
// github.com/hanzo-ds/go driver, which registers the database/sql driver name
// "datastore".
//
// Usage:
//
// import _ "github.com/golang-migrate/migrate/v4/database/datastore"
//
// Then use a URL like:
//
// datastore://host:9000?username=default&database=default&x-multi-statement=true
package datastore
package clickhouse
import (
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
"net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"go.uber.org/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
"github.com/hashicorp/go-multierror"
)
var (
multiStmtDelimiter = []byte(";")
// DriverName is the database/sql driver registered by github.com/hanzo-ds/go.
DriverName = "datastore"
DefaultMigrationsTable = "schema_migrations"
DefaultMigrationsTableEngine = "TinyLog"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
@@ -55,11 +38,9 @@ type Config struct {
}
func init() {
database.Register("datastore", &Datastore{})
database.Register("clickhouse", &ClickHouse{})
}
// WithInstance creates a Datastore driver from an existing *sql.DB opened
// against the "datastore" driver.
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
@@ -69,34 +50,32 @@ func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
return nil, err
}
ds := &Datastore{
ch := &ClickHouse{
conn: conn,
config: config,
}
if err := ds.init(); err != nil {
if err := ch.init(); err != nil {
return nil, err
}
return ds, nil
return ch, nil
}
// Datastore is a golang-migrate database.Driver backed by the native
// github.com/hanzo-ds/go ClickHouse-wire driver.
type Datastore struct {
type ClickHouse struct {
conn *sql.DB
config *Config
isLocked atomic.Bool
}
func (ds *Datastore) Open(dsn string) (database.Driver, error) {
func (ch *ClickHouse) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = DriverName
conn, err := sql.Open(DriverName, q.String())
q.Scheme = "tcp"
conn, err := sql.Open("clickhouse", q.String())
if err != nil {
return nil, err
}
@@ -114,7 +93,7 @@ func (ds *Datastore) Open(dsn string) (database.Driver, error) {
migrationsTableEngine = s
}
ds = &Datastore{
ch = &ClickHouse{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
@@ -126,44 +105,44 @@ func (ds *Datastore) Open(dsn string) (database.Driver, error) {
},
}
if err := ds.init(); err != nil {
if err := ch.init(); err != nil {
return nil, err
}
return ds, nil
return ch, nil
}
func (ds *Datastore) init() error {
if len(ds.config.DatabaseName) == 0 {
if err := ds.conn.QueryRow("SELECT currentDatabase()").Scan(&ds.config.DatabaseName); err != nil {
func (ch *ClickHouse) init() error {
if len(ch.config.DatabaseName) == 0 {
if err := ch.conn.QueryRow("SELECT currentDatabase()").Scan(&ch.config.DatabaseName); err != nil {
return err
}
}
if len(ds.config.MigrationsTable) == 0 {
ds.config.MigrationsTable = DefaultMigrationsTable
if len(ch.config.MigrationsTable) == 0 {
ch.config.MigrationsTable = DefaultMigrationsTable
}
if ds.config.MultiStatementMaxSize <= 0 {
ds.config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
if ch.config.MultiStatementMaxSize <= 0 {
ch.config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
if len(ds.config.MigrationsTableEngine) == 0 {
ds.config.MigrationsTableEngine = DefaultMigrationsTableEngine
if len(ch.config.MigrationsTableEngine) == 0 {
ch.config.MigrationsTableEngine = DefaultMigrationsTableEngine
}
return ds.ensureVersionTable()
return ch.ensureVersionTable()
}
func (ds *Datastore) Run(r io.Reader) error {
if ds.config.MultiStatementEnabled {
func (ch *ClickHouse) Run(r io.Reader) error {
if ch.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(r, multiStmtDelimiter, ds.config.MultiStatementMaxSize, func(m []byte) bool {
if e := multistmt.Parse(r, multiStmtDelimiter, ch.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if _, e := ds.conn.Exec(string(m)); e != nil {
if _, e := ch.conn.Exec(string(m)); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
@@ -174,25 +153,24 @@ func (ds *Datastore) Run(r io.Reader) error {
return err
}
migration, err := io.ReadAll(r)
migration, err := ioutil.ReadAll(r)
if err != nil {
return err
}
if _, err := ds.conn.Exec(string(migration)); err != nil {
if _, err := ch.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ds *Datastore) Version() (int, bool, error) {
func (ch *ClickHouse) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ds.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
query = "SELECT version, dirty FROM `" + ch.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ds.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err := ch.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
@@ -201,7 +179,7 @@ func (ds *Datastore) Version() (int, bool, error) {
return version, dirty == 1, nil
}
func (ds *Datastore) SetVersion(version int, dirty bool) error {
func (ch *ClickHouse) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
@@ -209,13 +187,13 @@ func (ds *Datastore) SetVersion(version int, dirty bool) error {
}
return 0
}
tx, err = ds.conn.Begin()
tx, err = ch.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ds.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
query := "INSERT INTO " + ch.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -225,24 +203,28 @@ func (ds *Datastore) SetVersion(version int, dirty bool) error {
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Datastore type.
func (ds *Datastore) ensureVersionTable() (err error) {
if err = ds.Lock(); err != nil {
// convention of "caller locks" in the ClickHouse type.
func (ch *ClickHouse) ensureVersionTable() (err error) {
if err = ch.Lock(); err != nil {
return err
}
defer func() {
if e := ds.Unlock(); e != nil {
err = errors.Join(err, e)
if e := ch.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
var (
table string
query = "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName) + " LIKE '" + ds.config.MigrationsTable + "'"
query = "SHOW TABLES FROM " + ch.config.DatabaseName + " LIKE '" + ch.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ds.conn.QueryRow(query).Scan(&table); err != nil {
if err := ch.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -251,42 +233,42 @@ func (ds *Datastore) ensureVersionTable() (err error) {
}
// if not, create the empty migration table
if len(ds.config.ClusterName) > 0 {
if len(ch.config.ClusterName) > 0 {
query = fmt.Sprintf(`
CREATE TABLE %s ON CLUSTER %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.ClusterName, ds.config.MigrationsTableEngine)
) Engine=%s`, ch.config.MigrationsTable, ch.config.ClusterName, ch.config.MigrationsTableEngine)
} else {
query = fmt.Sprintf(`
CREATE TABLE %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.MigrationsTableEngine)
) Engine=%s`, ch.config.MigrationsTable, ch.config.MigrationsTableEngine)
}
if strings.HasSuffix(ds.config.MigrationsTableEngine, "Tree") {
if strings.HasSuffix(ch.config.MigrationsTableEngine, "Tree") {
query = fmt.Sprintf(`%s ORDER BY sequence`, query)
}
if _, err := ds.conn.Exec(query); err != nil {
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Drop() (err error) {
query := "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName)
tables, err := ds.conn.Query(query)
func (ch *ClickHouse) Drop() (err error) {
query := "SHOW TABLES FROM " + ch.config.DatabaseName
tables, err := ch.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -296,9 +278,9 @@ func (ds *Datastore) Drop() (err error) {
return err
}
query = "DROP TABLE IF EXISTS " + quoteIdentifier(ds.config.DatabaseName) + "." + quoteIdentifier(table)
query = "DROP TABLE IF EXISTS " + ch.config.DatabaseName + "." + table
if _, err := ds.conn.Exec(query); err != nil {
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
@@ -309,29 +291,18 @@ func (ds *Datastore) Drop() (err error) {
return nil
}
func (ds *Datastore) Lock() error {
if !ds.isLocked.CompareAndSwap(false, true) {
func (ch *ClickHouse) Lock() error {
if !ch.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (ds *Datastore) Unlock() error {
if !ds.isLocked.CompareAndSwap(true, false) {
func (ch *ClickHouse) Unlock() error {
if !ch.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (ds *Datastore) Close() error { return ds.conn.Close() }
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
func (ch *ClickHouse) Close() error { return ch.conn.Close() }
@@ -1,4 +1,4 @@
package datastore_test
package clickhouse_test
import (
"context"
@@ -8,15 +8,13 @@ import (
"log"
"testing"
_ "github.com/ClickHouse/clickhouse-go"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/datastore"
"github.com/golang-migrate/migrate/v4/database/clickhouse"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
const defaultPort = 9000
@@ -28,19 +26,19 @@ var (
PortRequired: true, ReadyFunc: isReady,
}
specs = []dktesting.ContainerSpec{
{ImageName: "clickhouse:24.8", Options: opts},
{ImageName: "yandex/clickhouse-server:21.3", Options: opts},
}
)
func datastoreConnectionString(host, port, engine string) string {
func clickhouseConnectionString(host, port, engine string) string {
if engine != "" {
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&x-migrations-table-engine=%v&debug=false",
"clickhouse://%v:%v?username=user&password=password&database=db&x-multi-statement=true&x-migrations-table-engine=%v&debug=false",
host, port, engine)
}
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&debug=false",
"clickhouse://%v:%v?username=user&password=password&database=db&x-multi-statement=true&debug=false",
host, port)
}
@@ -50,7 +48,7 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
return false
}
db, err := sql.Open("datastore", datastoreConnectionString(ip, port, ""))
db, err := sql.Open("clickhouse", clickhouseConnectionString(ip, port, ""))
if err != nil {
log.Println("open error", err)
@@ -92,8 +90,8 @@ func testSimple(t *testing.T, engine string) {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
addr := clickhouseConnectionString(ip, port, engine)
p := &clickhouse.ClickHouse{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
@@ -115,12 +113,12 @@ func testSimpleWithInstanceDefaultConfigValues(t *testing.T) {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, "")
conn, err := sql.Open("datastore", addr)
addr := clickhouseConnectionString(ip, port, "")
conn, err := sql.Open("clickhouse", addr)
if err != nil {
t.Fatal(err)
}
d, err := datastore.WithInstance(conn, &datastore.Config{})
d, err := clickhouse.WithInstance(conn, &clickhouse.Config{})
if err != nil {
_ = conn.Close()
t.Fatal(err)
@@ -142,8 +140,8 @@ func testMigrate(t *testing.T, engine string) {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
addr := clickhouseConnectionString(ip, port, engine)
p := &clickhouse.ClickHouse{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
@@ -171,8 +169,8 @@ func testVersion(t *testing.T, engine string) {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
addr := clickhouseConnectionString(ip, port, engine)
p := &clickhouse.ClickHouse{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
@@ -206,8 +204,8 @@ func testDrop(t *testing.T, engine string) {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
addr := clickhouseConnectionString(ip, port, engine)
p := &clickhouse.ClickHouse{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
@@ -1,3 +1,3 @@
CREATE TABLE test_1 (
Date Date
Date Date
) Engine=Memory;
@@ -1,3 +1,3 @@
CREATE TABLE test_2 (
Date Date
Date Date
) Engine=Memory;
+3 -3
View File
@@ -15,7 +15,7 @@ CREATE USER IF NOT EXISTS cockroach;
GRANT ALL ON DATABASE example TO cockroach;
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convienience:
```
export COCKROACHDB_URL='cockroachdb://cockroach:@localhost:26257/example?sslmode=disable'
```
@@ -50,7 +50,7 @@ And in the `.down.sql` let's delete it:
```
DROP TABLE IF EXISTS example.users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](/GETTING_STARTED.md#create-migrations)
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)
## Run migrations
```
@@ -139,4 +139,4 @@ func main() {
}
}
```
You can find details [here](README.md#use-in-your-go-project)
You can find details [here](README.md#use-in-your-go-project)
+17 -7
View File
@@ -3,18 +3,24 @@ package cockroachdb
import (
"context"
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"sync/atomic"
)
import (
"github.com/cockroachdb/cockroach-go/v2/crdb"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
import (
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/lib/pq"
)
func init() {
@@ -161,7 +167,7 @@ func (c *CockroachDb) Lock() error {
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -211,7 +217,7 @@ func (c *CockroachDb) Unlock() error {
}
func (c *CockroachDb) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -276,7 +282,7 @@ func (c *CockroachDb) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -318,7 +324,11 @@ func (c *CockroachDb) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+6 -6
View File
@@ -26,13 +26,13 @@ import (
const defaultPort = 26257
var (
opts = dktest.Options{Cmd: []string{"start-single-node", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.cockroachlabs.com/docs/releases/release-support-policy#supported-versions
opts = dktest.Options{Cmd: []string{"start", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Released versions: https://www.cockroachlabs.com/docs/releases/
specs = []dktesting.ContainerSpec{
{ImageName: "cockroachdb/cockroach:latest-v24.3", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v24.1", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.2", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.1", Options: opts},
{ImageName: "cockroachdb/cockroach:v1.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v1.1.9", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.1.3", Options: opts},
}
)
-31
View File
@@ -1,31 +0,0 @@
# Datastore
`datastore://host:port?username=user&password=password&database=clicks&x-multi-statement=true`
Hanzo Datastore is a ClickHouse-wire-compatible database engine. This driver
connects through the native `github.com/hanzo-ds/go` driver, which registers the
`database/sql` driver name `datastore`.
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `x-migrations-table-engine`| Engine to use for the migrations table, defaults to TinyLog |
| `x-cluster-name` | Name of cluster for creating `schema_migrations` table cluster wide |
| `x-multi-statement` | Enable multiple statements to be ran in a single migration (See note below) |
| `x-multi-statement-max-size` | Maximum size of a single migration in bytes when `x-multi-statement` is set (defaults to 10 MB) |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
## Notes
* The Datastore driver does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
* Using the default TinyLog table engine for the `schema_migrations` table prevents backing up the table. If you need to back up the database, run the migrations with `x-migrations-table-engine=MergeTree`.
* Datastore cluster mode is not officially supported, but you can try enabling `schema_migrations` table replication by specifying a `x-cluster-name`:
* When `x-cluster-name` is specified, `x-migrations-table-engine` also should be specified.
* When `x-cluster-name` is specified, only the `schema_migrations` table is replicated across the cluster. You still need to write your migrations so that the application tables are replicated within the cluster.
* If you want to create a database inside a migration, note that the `schema_migrations` table will live in the `default` database, so you cannot use `USE <database_name>` inside a migration. In this case you may omit the database in the connection string (example [here](examples/migrations/003_create_database.up.sql)).
@@ -1,10 +0,0 @@
DROP TABLE IF EXISTS driver_ratings;
DROP TABLE IF EXISTS user_ratings;
DROP TABLE IF EXISTS orders;
DROP TABLE IF EXISTS driver_ratings_queue;
DROP TABLE IF EXISTS user_ratings_queue;
DROP TABLE IF EXISTS orders_queue;
DROP VIEW IF EXISTS user_ratings_queue_mv;
DROP VIEW IF EXISTS driver_ratings_queue_mv;
DROP VIEW IF EXISTS orders_queue_mv;
DROP DATABASE IF EXISTS analytics;
@@ -1,81 +0,0 @@
CREATE DATABASE IF NOT EXISTS analytics;
CREATE TABLE IF NOT EXISTS analytics.driver_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.driver_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'driver-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.driver_ratings_queue_mv TO analytics.driver_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.driver_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.user_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY userID
ORDER BY (inserted_time);
CREATE TABLE analytics.user_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'user-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'JSON',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.user_ratings_queue_mv TO analytics.user_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.user_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.orders(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.orders_queue(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'orders',
kafka_group_name = 'order_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.orders_queue_mv TO orders AS
SELECT from_place, to_place, userID, driverID, orderID
FROM analytics.orders_queue;
+14 -14
View File
@@ -1,4 +1,4 @@
// Package database provides the Driver interface.
// Package database provides the Database interface.
// All database drivers must implement this interface, register themselves,
// optionally provide a `WithInstance` function and pass the tests
// in package database/testing.
@@ -25,22 +25,22 @@ var drivers = make(map[string]Driver)
// Driver is the interface every database driver must implement.
//
// How to implement a database driver?
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
//
// Guidelines:
// - Don't try to correct user input. Don't assume things.
// * Don't try to correct user input. Don't assume things.
// When in doubt, return an error and explain the situation to the user.
// - All configuration input must come from the URL string in func Open()
// * All configuration input must come from the URL string in func Open()
// or the Config{} struct in WithInstance. Don't os.Getenv().
type Driver interface {
// Open returns a new driver instance configured with parameters
+15 -10
View File
@@ -1,19 +1,20 @@
//go:build go1.9
// +build go1.9
package firebird
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/nakagami/firebirdsql"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
)
func init() {
@@ -107,21 +108,21 @@ func (f *Firebird) Close() error {
}
func (f *Firebird) Lock() error {
if !f.isLocked.CompareAndSwap(false, true) {
if !f.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (f *Firebird) Unlock() error {
if !f.isLocked.CompareAndSwap(true, false) {
if !f.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (f *Firebird) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -180,7 +181,7 @@ func (f *Firebird) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -223,7 +224,11 @@ func (f *Firebird) ensureVersionTable() (err error) {
defer func() {
if e := f.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+2 -3
View File
@@ -38,9 +38,8 @@ var (
},
}
specs = []dktesting.ContainerSpec{
{ImageName: "jacobalberty/firebird:v3.0", Options: opts},
{ImageName: "jacobalberty/firebird:v4.0", Options: opts},
{ImageName: "jacobalberty/firebird:v5.0", Options: opts},
{ImageName: "jacobalberty/firebird:2.5-ss", Options: opts},
{ImageName: "jacobalberty/firebird:3.0", Options: opts},
}
)
+2 -2
View File
@@ -15,8 +15,8 @@
| `x-transaction-mode` | `TransactionMode` | If set to `true` wrap commands in [transaction](https://docs.mongodb.com/manual/core/transactions). Available only for replica set. Driver is using [strconv.ParseBool](https://golang.org/pkg/strconv/#ParseBool) for parsing|
| `x-advisory-locking` | `true` | Feature flag for advisory locking, if set to false, disable advisory locking |
| `x-advisory-lock-collection` | `migrate_advisory_lock` | The name of the collection to use for advisory locking.|
| `x-advisory-lock-timeout` | `15` | The max time in seconds that migrate will wait to acquire a lock before failing. |
| `x-advisory-lock-timeout-interval` | `10` | The max time in seconds between attempts to acquire the advisory lock, the lock is attempted to be acquired using an exponential backoff algorithm. |
| `x-advisory-lock-timout` | `15` | The max time in seconds that the advisory lock will wait if the db is already locked. |
| `x-advisory-lock-timout-interval` | `10` | The max timeout in seconds interval that the advisory lock will wait if the db is already locked. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as. Can be omitted |
| `password` | | The user's password. Can be omitted |
+23 -35
View File
@@ -2,21 +2,21 @@ package mongodb
import (
"context"
"errors"
"fmt"
"io"
"net/url"
"os"
"strconv"
"sync/atomic"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/x/mongo/driver/connstring"
"go.uber.org/atomic"
"io"
"io/ioutil"
"net/url"
os "os"
"strconv"
"time"
)
func init() {
@@ -36,9 +36,8 @@ const LockIndexName = "lock_unique_key" // the name of the inde
const contextWaitTimeout = 5 * time.Second // how long to wait for the request to mongo to block/wait for.
var (
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
ErrLockTimeoutConfigConflict = fmt.Errorf("both x-advisory-lock-timeout-interval and x-advisory-lock-timout-interval were specified")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
)
type Mongo struct {
@@ -114,7 +113,7 @@ func WithInstance(instance *mongo.Client, config *Config) (database.Driver, erro
}
func (m *Mongo) Open(dsn string) (database.Driver, error) {
// connstring is experimental package, but it used for parse connection string in mongo.Connect function
//connstring is experimental package, but it used for parse connection string in mongo.Connect function
uri, err := connstring.Parse(dsn)
if err != nil {
return nil, err
@@ -138,22 +137,7 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
if err != nil {
return nil, err
}
lockTimeoutIntervalValue := unknown.Get("x-advisory-lock-timeout-interval")
// The initial release had a typo for this argument but for backwards compatibility sake, we will keep supporting it
// and we will error out if both values are set.
lockTimeoutIntervalValueFromTypo := unknown.Get("x-advisory-lock-timout-interval")
lockTimeout := lockTimeoutIntervalValue
if lockTimeoutIntervalValue != "" && lockTimeoutIntervalValueFromTypo != "" {
return nil, ErrLockTimeoutConfigConflict
} else if lockTimeoutIntervalValueFromTypo != "" {
lockTimeout = lockTimeoutIntervalValueFromTypo
}
maxLockCheckInterval, err := parseInt(lockTimeout, DefaultLockTimeoutInterval)
maxLockingIntervals, err := parseInt(unknown.Get("x-advisory-lock-timout-interval"), DefaultLockTimeoutInterval)
if err != nil {
return nil, err
}
@@ -173,7 +157,7 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
CollectionName: lockCollection,
Timeout: lockingTimout,
Enabled: advisoryLockingFlag,
Interval: maxLockCheckInterval,
Interval: maxLockingIntervals,
},
})
if err != nil {
@@ -182,7 +166,7 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
return mc, nil
}
// Parse the url param, convert it to boolean
//Parse the url param, convert it to boolean
// returns error if param invalid. returns defaultValue if param not present
func parseBoolean(urlParam string, defaultValue bool) (bool, error) {
@@ -199,7 +183,7 @@ func parseBoolean(urlParam string, defaultValue bool) (bool, error) {
return defaultValue, nil
}
// Parse the url param, convert it to int
//Parse the url param, convert it to int
// returns error if param invalid. returns defaultValue if param not present
func parseInt(urlParam string, defaultValue int) (int, error) {
@@ -241,7 +225,7 @@ func (m *Mongo) Version() (version int, dirty bool, err error) {
}
func (m *Mongo) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -268,8 +252,8 @@ func (m *Mongo) executeCommandsWithTransaction(ctx context.Context, cmds []bson.
return &database.Error{OrigErr: err, Err: "failed to start transaction"}
}
if err := m.executeCommands(sessionContext, cmds); err != nil {
// When command execution is failed, it's aborting transaction
// If you tried to call abortTransaction, it`s return error that transaction already aborted
//When command execution is failed, it's aborting transaction
//If you tried to call abortTransaction, it`s return error that transaction already aborted
return err
}
if err := sessionContext.CommitTransaction(sessionContext); err != nil {
@@ -325,7 +309,11 @@ func (m *Mongo) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+7 -32
View File
@@ -32,10 +32,10 @@ var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.mongodb.com/support-policy
specs = []dktesting.ContainerSpec{
{ImageName: "mongo:5.0", Options: opts},
{ImageName: "mongo:6.0", Options: opts},
{ImageName: "mongo:7.0", Options: opts},
{ImageName: "mongo:8.0", Options: opts},
{ImageName: "mongo:3.4", Options: opts},
{ImageName: "mongo:3.6", Options: opts},
{ImageName: "mongo:4.0", Options: opts},
{ImageName: "mongo:4.2", Options: opts},
}
)
@@ -74,22 +74,6 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testWithAuth", testWithAuth)
t.Run("testLockWorks", testLockWorks)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -115,7 +99,7 @@ func test(t *testing.T) {
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -141,7 +125,7 @@ func testMigrate(t *testing.T) {
})
}
func testWithAuth(t *testing.T) {
func TestWithAuth(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -196,7 +180,7 @@ func testWithAuth(t *testing.T) {
})
}
func testLockWorks(t *testing.T) {
func TestLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -257,15 +241,6 @@ func TestTransaction(t *testing.T) {
{ImageName: "mongo:4", Options: dktest.Options{PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"mongod", "--bind_ip_all", "--replSet", "rs0"}}},
}
t.Cleanup(func() {
for _, spec := range transactionSpecs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
dktesting.ParallelTest(t, transactionSpecs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
+4 -5
View File
@@ -6,7 +6,6 @@
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-no-lock` | `NoLock` | Set to `true` to skip `GET_LOCK`/`RELEASE_LOCK` statements. Useful for [multi-master MySQL flavors](https://www.percona.com/doc/percona-xtradb-cluster/LATEST/features/pxc-strict-mode.html#explicit-table-locking). Only run migrations from one host when this is enabled. |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds, functionally similar to [Server-side SELECT statement timeouts](https://dev.mysql.com/blog-archive/server-side-select-statement-timeouts/) but enforced by the client. Available for all versions of MySQL, not just >=5.7. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
@@ -14,7 +13,7 @@
| `port` | | The port to bind to. |
| `tls` | | TLS / SSL encrypted connection parameter; see [go-sql-driver](https://github.com/go-sql-driver/mysql#tls). Use any name (e.g. `migrate`) if you want to use a custom TLS config (`x-tls-` queries). |
| `x-tls-ca` | | The location of the CA (certificate authority) file. |
| `x-tls-cert` | | The location of the client certificate file. Must be used with `x-tls-key`. |
| `x-tls-cert` | | The location of the client certicicate file. Must be used with `x-tls-key`. |
| `x-tls-key` | | The location of the private key file. Must be used with `x-tls-cert`. |
| `x-tls-insecure-skip-verify` | | Whether or not to use SSL (true\|false) |
@@ -29,9 +28,9 @@ import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/mysql"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/golang-migrate/migrate"
"github.com/golang-migrate/migrate/database/mysql"
_ "github.com/golang-migrate/migrate/source/file"
)
func main() {
+24 -38
View File
@@ -1,4 +1,5 @@
//go:build go1.9
// +build go1.9
package mysql
@@ -7,18 +8,17 @@ import (
"crypto/tls"
"crypto/x509"
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"os"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
)
var _ database.Driver = (*Mysql)(nil) // explicit compile time type check
@@ -34,14 +34,13 @@ var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrAppendPEM = fmt.Errorf("failed to append PEM")
ErrTLSCertKeyConfig = fmt.Errorf("to use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
ErrTLSCertKeyConfig = fmt.Errorf("To use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
)
type Config struct {
MigrationsTable string
DatabaseName string
NoLock bool
StatementTimeout time.Duration
MigrationsTable string
DatabaseName string
NoLock bool
}
type Mysql struct {
@@ -155,7 +154,7 @@ func urlToMySQLConfig(url string) (*mysql.Config, error) {
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := os.ReadFile(parsedParams.Get("x-tls-ca"))
pem, err := ioutil.ReadFile(parsedParams.Get("x-tls-ca"))
if err != nil {
return nil, err
}
@@ -242,25 +241,15 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
}
}
statementTimeoutParam := customParams["x-statement-timeout"]
statementTimeout := 0
if statementTimeoutParam != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutParam)
if err != nil {
return nil, fmt.Errorf("could not parse x-statement-timeout as float: %w", err)
}
}
db, err := sql.Open("mysql", config.FormatDSN())
if err != nil {
return nil, err
}
mx, err := WithInstance(db, &Config{
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
NoLock: noLock,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
NoLock: noLock,
})
if err != nil {
return nil, err
@@ -333,20 +322,13 @@ func (m *Mysql) Unlock() error {
}
func (m *Mysql) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
ctx := context.Background()
if m.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, m.config.StatementTimeout)
defer cancel()
}
query := string(migr[:])
if _, err := m.conn.ExecContext(ctx, query); err != nil {
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
@@ -354,15 +336,15 @@ func (m *Mysql) Run(migration io.Reader) error {
}
func (m *Mysql) SetVersion(version int, dirty bool) error {
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable})
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "DELETE FROM `" + m.config.MigrationsTable + "` LIMIT 1"
query := "TRUNCATE `" + m.config.MigrationsTable + "`"
if _, err := tx.ExecContext(context.Background(), query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -374,7 +356,7 @@ func (m *Mysql) SetVersion(version int, dirty bool) error {
query := "INSERT INTO `" + m.config.MigrationsTable + "` (version, dirty) VALUES (?, ?)"
if _, err := tx.ExecContext(context.Background(), query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -416,7 +398,7 @@ func (m *Mysql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -469,7 +451,11 @@ func (m *Mysql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+20 -12
View File
@@ -9,6 +9,7 @@ import (
"encoding/pem"
"errors"
"fmt"
"io/ioutil"
"log"
"math/big"
"math/rand"
@@ -16,14 +17,19 @@ import (
"os"
"strconv"
"testing"
)
import (
"github.com/dhui/dktest"
"github.com/go-sql-driver/mysql"
"github.com/stretchr/testify/assert"
)
import (
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/stretchr/testify/assert"
)
const defaultPort = 3306
@@ -40,14 +46,16 @@ var (
}
// Supported versions: https://www.mysql.com/support/supportedplatforms/database.html
specs = []dktesting.ContainerSpec{
{ImageName: "mysql:8.0", Options: opts},
{ImageName: "mysql:8.4", Options: opts},
{ImageName: "mysql:9.0", Options: opts},
{ImageName: "mysql:5.5", Options: opts},
{ImageName: "mysql:5.6", Options: opts},
{ImageName: "mysql:5.7", Options: opts},
{ImageName: "mysql:8", Options: opts},
}
specsAnsiQuotes = []dktesting.ContainerSpec{
{ImageName: "mysql:8.0", Options: optsAnsiQuotes},
{ImageName: "mysql:8.4", Options: optsAnsiQuotes},
{ImageName: "mysql:9.0", Options: optsAnsiQuotes},
{ImageName: "mysql:5.5", Options: optsAnsiQuotes},
{ImageName: "mysql:5.6", Options: optsAnsiQuotes},
{ImageName: "mysql:5.7", Options: optsAnsiQuotes},
{ImageName: "mysql:8", Options: optsAnsiQuotes},
}
)
@@ -80,7 +88,7 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -113,7 +121,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -151,7 +159,7 @@ func TestMigrate(t *testing.T) {
}
func TestMigrateAnsiQuotes(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specsAnsiQuotes, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -334,7 +342,7 @@ func TestExtractCustomQueryParams(t *testing.T) {
}
func createTmpCert(t *testing.T) string {
tmpCertFile, err := os.CreateTemp("", "migrate_test_cert")
tmpCertFile, err := ioutil.TempFile("", "migrate_test_cert")
if err != nil {
t.Fatal("Failed to create temp cert file:", err)
}
@@ -389,7 +397,7 @@ func TestURLToMySQLConfig(t *testing.T) {
// Not supported yet: https://github.com/go-sql-driver/mysql/issues/591
// {name: "user/password - user with encoded :",
// urlStr: "mysql://username%3A:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
// expectedDSN: "username::password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
// expectedDSN: "username::pasword@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - user with encoded @",
urlStr: "mysql://username%40:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
+8 -7
View File
@@ -2,15 +2,16 @@ package neo4j
import (
"bytes"
"errors"
"fmt"
"io"
"io/ioutil"
neturl "net/url"
"strconv"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/hashicorp/go-multierror"
"github.com/neo4j/neo4j-go-driver/neo4j"
)
@@ -141,7 +142,7 @@ func (n *Neo4j) Run(migration io.Reader) (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
err = multierror.Append(err, cerr)
}
}()
@@ -172,7 +173,7 @@ func (n *Neo4j) Run(migration io.Reader) (err error) {
return err
}
body, err := io.ReadAll(migration)
body, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -188,7 +189,7 @@ func (n *Neo4j) SetVersion(version int, dirty bool) (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
err = multierror.Append(err, cerr)
}
}()
@@ -213,7 +214,7 @@ func (n *Neo4j) Version() (version int, dirty bool, err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
err = multierror.Append(err, cerr)
}
}()
@@ -261,7 +262,7 @@ func (n *Neo4j) Drop() (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
err = multierror.Append(err, cerr)
}
}()
@@ -278,7 +279,7 @@ func (n *Neo4j) ensureVersionConstraint() (err error) {
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
err = multierror.Append(err, cerr)
}
}()
+8 -9
View File
@@ -8,15 +8,13 @@ import (
)
const reservedChars = "!#$%&'()*+,/:;=?@[]"
const reservedCharTestNamePrefix = "reserved char "
const baseUsername = "username"
const scheme = "database://"
// TestUserUnencodedReservedURLChars documents the behavior of using unencoded reserved characters in usernames with
// net/url Parse()
func TestUserUnencodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
@@ -66,7 +64,7 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
t.Run("reserved char "+tc.char, func(t *testing.T) {
s := scheme + baseUsername + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -100,12 +98,13 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
}
func TestUserEncodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
for _, c := range reservedChars {
c := string(c)
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
t.Run("reserved char "+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := scheme + baseUsername + encodedChar + urlSuffixAndSep
expectedUsername := baseUsername + c
@@ -127,7 +126,7 @@ func TestUserEncodedReservedURLChars(t *testing.T) {
// with net/url Parse()
func TestPasswordUnencodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := scheme + username + ":"
schemeAndUsernameAndSep := "database://" + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
@@ -175,7 +174,7 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
t.Run("reserved char "+tc.char, func(t *testing.T) {
s := schemeAndUsernameAndSep + basePassword + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -214,13 +213,13 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
func TestPasswordEncodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := scheme + username + ":"
schemeAndUsernameAndSep := "database://" + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
for _, c := range reservedChars {
c := string(c)
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
t.Run("reserved char "+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := schemeAndUsernameAndSep + basePassword + encodedChar + urlSuffixAndSep
expectedPassword := basePassword + c
-4
View File
@@ -1,7 +1,5 @@
# pgx
This package is for [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). A backend for the newer [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5) is [also available](v5).
`pgx://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
@@ -11,8 +9,6 @@ This package is for [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). A back
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `x-lock-strategy` | `LockStrategy` | Strategy used for locking during migration (default: advisory) |
| `x-lock-table` | `LockTable` | Name of the table which maintains the migration lock (default: schema_lock) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
+38 -168
View File
@@ -1,38 +1,33 @@
//go:build go1.9
// +build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
multierror "github.com/hashicorp/go-multierror"
"github.com/jackc/pgconn"
"github.com/jackc/pgerrcode"
_ "github.com/jackc/pgx/v4/stdlib"
"github.com/lib/pq"
)
const (
LockStrategyAdvisory = "advisory"
LockStrategyTable = "table"
)
func init() {
db := Postgres{}
database.Register("pgx", &db)
database.Register("pgx4", &db)
}
var (
@@ -40,8 +35,6 @@ var (
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
DefaultLockTable = "schema_lock"
DefaultLockStrategy = LockStrategyAdvisory
)
var (
@@ -55,8 +48,6 @@ type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
LockTable string
LockStrategy string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
@@ -116,14 +107,6 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
config.MigrationsTable = DefaultMigrationsTable
}
if len(config.LockTable) == 0 {
config.LockTable = DefaultLockTable
}
if len(config.LockStrategy) == 0 {
config.LockStrategy = DefaultLockStrategy
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
@@ -149,10 +132,6 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
config: config,
}
if err := px.ensureLockTable(); err != nil {
return nil, err
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
@@ -171,7 +150,7 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v4", migrate.FilterCustomQuery(purl).String())
db, err := sql.Open("pgx", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
@@ -181,7 +160,7 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
return nil, fmt.Errorf("Unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
@@ -212,13 +191,10 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
return nil, fmt.Errorf("Unable to parse option x-multi-statement: %w", err)
}
}
lockStrategy := purl.Query().Get("x-lock-strategy")
lockTable := purl.Query().Get("x-lock-table")
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
@@ -226,8 +202,6 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
LockStrategy: lockStrategy,
LockTable: lockTable,
})
if err != nil {
@@ -246,118 +220,38 @@ func (p *Postgres) Close() error {
return nil
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.applyAdvisoryLock()
case LockStrategyTable:
return p.applyTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.releaseAdvisoryLock()
case LockStrategyTable:
return p.releaseTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
})
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) applyAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
}
func (p *Postgres) applyTableLock() error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
defer func() {
errRollback := tx.Rollback()
if errRollback != nil {
err = errors.Join(err, errRollback)
}
}()
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked {
return database.ErrLocked
}
query = "INSERT INTO " + pq.QuoteIdentifier(p.config.LockTable) + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return tx.Commit()
}
func (p *Postgres) releaseAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) releaseTableLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "DELETE FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
if _, err := p.db.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
@@ -371,7 +265,7 @@ func (p *Postgres) Run(migration io.Reader) error {
}
return err
}
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -412,7 +306,7 @@ func (p *Postgres) runStatement(statement []byte) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -454,7 +348,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -466,7 +360,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -508,7 +402,7 @@ func (p *Postgres) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -519,12 +413,6 @@ func (p *Postgres) Drop() (err error) {
if err := tables.Scan(&tableName); err != nil {
return err
}
// do not drop lock table
if tableName == p.config.LockTable && p.config.LockStrategy == LockStrategyTable {
continue
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
@@ -556,7 +444,11 @@ func (p *Postgres) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -585,33 +477,11 @@ func (p *Postgres) ensureVersionTable() (err error) {
return nil
}
func (p *Postgres) ensureLockTable() error {
if p.config.LockStrategy != LockStrategyTable {
return nil
}
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := p.db.QueryRow(query, p.config.LockTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE ` + pq.QuoteIdentifier(p.config.LockTable) + ` (lock_id BIGINT NOT NULL PRIMARY KEY)`
if _, err := p.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
return `"` + strings.Replace(name, `"`, `""`, -1) + `"`
}
+12 -36
View File
@@ -8,17 +8,19 @@ import (
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"io"
"strconv"
"strings"
"sync"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
@@ -34,11 +36,11 @@ var (
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
{ImageName: "postgres:9.5", Options: opts},
{ImageName: "postgres:9.6", Options: opts},
{ImageName: "postgres:10", Options: opts},
{ImageName: "postgres:11", Options: opts},
{ImageName: "postgres:12", Options: opts},
}
)
@@ -131,32 +133,6 @@ func TestMigrate(t *testing.T) {
})
}
func TestMigrateLockTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-lock-strategy=table", "x-lock-table=lock_table")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
@@ -755,10 +731,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
input = strings.Replace(input, "\n", "\r\n", -1)
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
input = strings.Replace(input, "FROM", "FRÖM", -1)
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
-41
View File
@@ -1,41 +0,0 @@
# pgx
This package is for [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5). A backend for the older [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). is [also available](..).
`pgx5://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
-480
View File
@@ -1,480 +0,0 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgerrcode"
"github.com/jackc/pgx/v5/pgconn"
_ "github.com/jackc/pgx/v5/stdlib"
)
func init() {
db := Postgres{}
database.Register("pgx5", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v5", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
-761
View File
@@ -1,761 +0,0 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://../examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
+5 -5
View File
@@ -27,7 +27,7 @@ If there were no errors, we should have two files available under `db/migrations
Note the `sql` extension that we provided.
In the `.up.sql` file let's create the table:
```sql
```
CREATE TABLE IF NOT EXISTS users(
user_id serial PRIMARY KEY,
username VARCHAR (50) UNIQUE NOT NULL,
@@ -36,7 +36,7 @@ CREATE TABLE IF NOT EXISTS users(
);
```
And in the `.down.sql` let's delete it:
```sql
```
DROP TABLE IF EXISTS users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](../../GETTING_STARTED.md#create-migrations)
@@ -79,7 +79,7 @@ Again, it should create for us two migrations files:
In Postgres, when we want our queries to be done in a transaction, we need to wrap it with `BEGIN` and `COMMIT` commands.
In our example, we are going to add a column to our database that can only accept enumerable values or NULL.
Migration up:
```sql
```
BEGIN;
CREATE TYPE enum_mood AS ENUM (
@@ -92,7 +92,7 @@ ALTER TABLE users ADD COLUMN mood enum_mood;
COMMIT;
```
Migration down:
```sql
```
BEGIN;
ALTER TABLE users DROP COLUMN mood;
@@ -124,7 +124,7 @@ Indexes:
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```go
```
import (
"log"
+31 -42
View File
@@ -1,23 +1,25 @@
//go:build go1.9
// +build go1.9
package postgres
import (
"context"
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
multierror "github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
@@ -63,19 +65,19 @@ type Postgres struct {
config *Config
}
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postgres, error) {
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.PingContext(ctx); err != nil {
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -88,16 +90,16 @@ func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postg
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName sql.NullString
if err := conn.QueryRowContext(ctx, query).Scan(&schemaName); err != nil {
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if !schemaName.Valid {
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName.String
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
@@ -117,8 +119,15 @@ func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postg
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
@@ -129,26 +138,6 @@ func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postg
return px, nil
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
px, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
px.db = instance
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
@@ -165,7 +154,7 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
return nil, fmt.Errorf("Unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
@@ -196,7 +185,7 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
return nil, fmt.Errorf("Unable to parse option x-multi-statement: %w", err)
}
}
@@ -218,11 +207,7 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
func (p *Postgres) Close() error {
connErr := p.conn.Close()
var dbErr error
if p.db != nil {
dbErr = p.db.Close()
}
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -275,7 +260,7 @@ func (p *Postgres) Run(migration io.Reader) error {
}
return err
}
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -319,7 +304,7 @@ func (p *Postgres) runStatement(statement []byte) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -361,7 +346,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -373,7 +358,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -415,7 +400,7 @@ func (p *Postgres) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -457,7 +442,11 @@ func (p *Postgres) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+19 -83
View File
@@ -35,11 +35,11 @@ var (
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
{ImageName: "postgres:9.5", Options: opts},
{ImageName: "postgres:9.6", Options: opts},
{ImageName: "postgres:10", Options: opts},
{ImageName: "postgres:11", Options: opts},
{ImageName: "postgres:12", Options: opts},
}
)
@@ -85,32 +85,6 @@ func mustRun(t *testing.T, d database.Driver, statements []string) {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testMultipleStatements", testMultipleStatements)
t.Run("testMultipleStatementsInMultiStatementMode", testMultipleStatementsInMultiStatementMode)
t.Run("testErrorParsing", testErrorParsing)
t.Run("testFilterCustomQuery", testFilterCustomQuery)
t.Run("testWithSchema", testWithSchema)
t.Run("testMigrationTableOption", testMigrationTableOption)
t.Run("testFailToCreateTableWithoutPermissions", testFailToCreateTableWithoutPermissions)
t.Run("testCheckBeforeCreateTable", testCheckBeforeCreateTable)
t.Run("testParallelSchema", testParallelSchema)
t.Run("testPostgresLock", testPostgresLock)
t.Run("testWithInstanceConcurrent", testWithInstanceConcurrent)
t.Run("testWithConnection", testWithConnection)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -132,7 +106,7 @@ func test(t *testing.T) {
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -158,7 +132,7 @@ func testMigrate(t *testing.T) {
})
}
func testMultipleStatements(t *testing.T) {
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -191,7 +165,7 @@ func testMultipleStatements(t *testing.T) {
})
}
func testMultipleStatementsInMultiStatementMode(t *testing.T) {
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -224,7 +198,7 @@ func testMultipleStatementsInMultiStatementMode(t *testing.T) {
})
}
func testErrorParsing(t *testing.T) {
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -253,7 +227,7 @@ func testErrorParsing(t *testing.T) {
})
}
func testFilterCustomQuery(t *testing.T) {
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -275,7 +249,7 @@ func testFilterCustomQuery(t *testing.T) {
})
}
func testWithSchema(t *testing.T) {
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -345,7 +319,7 @@ func testWithSchema(t *testing.T) {
})
}
func testMigrationTableOption(t *testing.T) {
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -413,7 +387,7 @@ func testMigrationTableOption(t *testing.T) {
})
}
func testFailToCreateTableWithoutPermissions(t *testing.T) {
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -483,7 +457,7 @@ func testFailToCreateTableWithoutPermissions(t *testing.T) {
})
}
func testCheckBeforeCreateTable(t *testing.T) {
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -560,7 +534,7 @@ func testCheckBeforeCreateTable(t *testing.T) {
})
}
func testParallelSchema(t *testing.T) {
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -628,7 +602,7 @@ func testParallelSchema(t *testing.T) {
})
}
func testPostgresLock(t *testing.T) {
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -668,7 +642,7 @@ func testPostgresLock(t *testing.T) {
})
}
func testWithInstanceConcurrent(t *testing.T) {
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -710,44 +684,6 @@ func testWithInstanceConcurrent(t *testing.T) {
}
})
}
func testWithConnection(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
db, err := sql.Open("postgres", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
ctx := context.Background()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatal(err)
}
p, err := WithConnection(ctx, conn, &Config{})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := p.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, p, []byte("SELECT 1"))
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
@@ -798,10 +734,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
input = strings.Replace(input, "\n", "\r\n", -1)
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
input = strings.Replace(input, "FROM", "FRÖM", -1)
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
+16 -10
View File
@@ -2,12 +2,14 @@ package ql
import (
"database/sql"
"errors"
"fmt"
"github.com/hashicorp/go-multierror"
"go.uber.org/atomic"
"io"
nurl "net/url"
"io/ioutil"
"strings"
"sync/atomic"
nurl "net/url"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
@@ -71,7 +73,11 @@ func (m *Ql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -128,7 +134,7 @@ func (m *Ql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -161,19 +167,19 @@ func (m *Ql) Drop() (err error) {
return nil
}
func (m *Ql) Lock() error {
if !m.isLocked.CompareAndSwap(false, true) {
if !m.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (m *Ql) Unlock() error {
if !m.isLocked.CompareAndSwap(true, false) {
if !m.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (m *Ql) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -188,7 +194,7 @@ func (m *Ql) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -216,7 +222,7 @@ func (m *Ql) SetVersion(version int, dirty bool) error {
m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+20 -2
View File
@@ -3,6 +3,8 @@ package ql
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -13,7 +15,15 @@ import (
)
func Test(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
p := &Ql{}
addr := fmt.Sprintf("ql://%s", filepath.Join(dir, "ql.db"))
@@ -35,7 +45,15 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
db, err := sql.Open("ql", filepath.Join(dir, "ql.db"))
+16 -10
View File
@@ -1,20 +1,22 @@
//go:build go1.9
// +build go1.9
package redshift
import (
"context"
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
@@ -126,21 +128,21 @@ func (p *Redshift) Close() error {
// Redshift does not support advisory lock functions: https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html
func (p *Redshift) Lock() error {
if !p.isLocked.CompareAndSwap(false, true) {
if !p.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (p *Redshift) Unlock() error {
if !p.isLocked.CompareAndSwap(true, false) {
if !p.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (p *Redshift) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -174,7 +176,7 @@ func (p *Redshift) Run(migration io.Reader) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -216,7 +218,7 @@ func (p *Redshift) SetVersion(version int, dirty bool) error {
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -228,7 +230,7 @@ func (p *Redshift) SetVersion(version int, dirty bool) error {
query = `INSERT INTO "` + p.config.MigrationsTable + `" (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -270,7 +272,7 @@ func (p *Redshift) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -312,7 +314,11 @@ func (p *Redshift) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+7 -16
View File
@@ -28,19 +28,10 @@ import (
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "redshift"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true,
ReadyFunc: isReady,
}
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
specs = []dktesting.ContainerSpec{
{ImageName: "migrate/postgres8:8", Options: opts},
{ImageName: "postgres:8", Options: opts},
}
)
@@ -53,7 +44,7 @@ func pgConnectionString(host, port string) string {
}
func connectionString(schema, host, port string) string {
return fmt.Sprintf("%s://postgres:%s@%s:%s/postgres?sslmode=disable", schema, pgPassword, host, port)
return fmt.Sprintf("%s://postgres@%s:%s/postgres?sslmode=disable", schema, host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -201,7 +192,7 @@ func TestFilterCustomQuery(t *testing.T) {
t.Fatal(err)
}
addr := fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-custom=foobar", pgPassword, ip, port)
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", ip, port)
p := &Redshift{}
d, err := p.Open(addr)
if err != nil {
@@ -243,7 +234,7 @@ func TestWithSchema(t *testing.T) {
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foobar", pgPassword, ip, port))
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", ip, port))
if err != nil {
t.Fatal(err)
}
@@ -378,10 +369,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
input = strings.Replace(input, "\n", "\r\n", -1)
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
input = strings.Replace(input, "FROM", "FRÖM", -1)
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
-18
View File
@@ -1,18 +0,0 @@
# rqlite
`rqlite://admin:secret@server1.example.com:4001/?level=strong&timeout=5`
The `rqlite` url scheme is used for both secure and insecure connections. If connecting to an insecure database, pass `x-connect-insecure` in your URL query, or use `WithInstance` to pass an established connection.
The migrations table name is configurable through the `x-migrations-table` URL query parameter, or by using `WithInstance` and passing `MigrationsTable` through `Config`.
Other connect parameters are directly passed through to the database driver. For examples of connection strings, see https://github.com/rqlite/gorqlite#examples.
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-connect-insecure` | n/a: set on instance | Boolean to indicate whether to use an insecure connection. Defaults to `false`. |
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table. Defaults to `schema_migrations`. |
## Notes
* Uses the https://github.com/rqlite/gorqlite driver
@@ -1 +0,0 @@
DROP TABLE IF EXISTS pets;
@@ -1,3 +0,0 @@
CREATE TABLE pets (
name string
);
@@ -1 +0,0 @@
DROP TABLE IF EXISTS pets;
@@ -1 +0,0 @@
ALTER TABLE pets ADD predator bool;
-328
View File
@@ -1,328 +0,0 @@
package rqlite
import (
"errors"
"fmt"
"io"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/rqlite/gorqlite"
)
func init() {
database.Register("rqlite", &Rqlite{})
}
const (
// DefaultMigrationsTable defines the default rqlite migrations table
DefaultMigrationsTable = "schema_migrations"
// DefaultConnectInsecure defines the default setting for connect insecure
DefaultConnectInsecure = false
)
// ErrNilConfig is returned if no configuration was passed to WithInstance
var ErrNilConfig = fmt.Errorf("no config")
// ErrBadConfig is returned if configuration was invalid
var ErrBadConfig = fmt.Errorf("bad parameter")
// Config defines the driver configuration
type Config struct {
// ConnectInsecure sets whether the connection uses TLS. Ineffectual when using WithInstance
ConnectInsecure bool
// MigrationsTable configures the migrations table name
MigrationsTable string
}
type Rqlite struct {
db *gorqlite.Connection
isLocked atomic.Bool
config *Config
}
// WithInstance creates a rqlite database driver with an existing gorqlite database connection
// and a Config struct
func WithInstance(instance *gorqlite.Connection, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
// we use the consistency level check as a database ping
if _, err := instance.ConsistencyLevel(); err != nil {
return nil, err
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
driver := &Rqlite{
db: instance,
config: config,
}
if err := driver.ensureVersionTable(); err != nil {
return nil, err
}
return driver, nil
}
// OpenURL creates a rqlite database driver from a connect URL
func OpenURL(url string) (database.Driver, error) {
d := &Rqlite{}
return d.Open(url)
}
func (r *Rqlite) ensureVersionTable() (err error) {
if err = r.Lock(); err != nil {
return err
}
defer func() {
if e := r.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
stmts := []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (version uint64, dirty bool)`, r.config.MigrationsTable),
fmt.Sprintf(`CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version)`, r.config.MigrationsTable),
}
if _, err := r.db.Write(stmts); err != nil {
return err
}
return nil
}
// Open returns a new driver instance configured with parameters
// coming from the URL string. Migrate will call this function
// only once per instance.
func (r *Rqlite) Open(url string) (database.Driver, error) {
dburl, config, err := parseUrl(url)
if err != nil {
return nil, err
}
r.config = config
r.db, err = gorqlite.Open(dburl.String())
if err != nil {
return nil, err
}
if err := r.ensureVersionTable(); err != nil {
return nil, err
}
return r, nil
}
// Close closes the underlying database instance managed by the driver.
// Migrate will call this function only once per instance.
func (r *Rqlite) Close() error {
r.db.Close()
return nil
}
// Lock should acquire a database lock so that only one migration process
// can run at a time. Migrate will call this function before Run is called.
// If the implementation can't provide this functionality, return nil.
// Return database.ErrLocked if database is already locked.
func (r *Rqlite) Lock() error {
if !r.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
// Unlock should release the lock. Migrate will call this function after
// all migrations have been run.
func (r *Rqlite) Unlock() error {
if !r.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
// Run applies a migration to the database. migration is guaranteed to be not nil.
func (r *Rqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if _, err := r.db.WriteOne(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// SetVersion saves version and dirty state.
// Migrate will call this function before and after each call to Run.
// version must be >= -1. -1 means NilVersion.
func (r *Rqlite) SetVersion(version int, dirty bool) error {
deleteQuery := fmt.Sprintf(`DELETE FROM %s`, r.config.MigrationsTable)
statements := []gorqlite.ParameterizedStatement{
{
Query: deleteQuery,
},
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
insertQuery := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, r.config.MigrationsTable)
if version >= 0 || (version == database.NilVersion && dirty) {
statements = append(statements, gorqlite.ParameterizedStatement{
Query: insertQuery,
Arguments: []interface{}{
version,
dirty,
},
})
}
wr, err := r.db.WriteParameterized(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i].Query)}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(deleteQuery + "\n" + insertQuery)}
}
return nil
}
// Version returns the currently active version and if the database is dirty.
// When no migration has been applied, it must return version -1.
// Dirty means, a previous migration failed and user interaction is required.
func (r *Rqlite) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM " + r.config.MigrationsTable + " LIMIT 1"
qr, err := r.db.QueryOne(query)
if err != nil {
return database.NilVersion, false, nil
}
if !qr.Next() {
return database.NilVersion, false, nil
}
if err := qr.Scan(&version, &dirty); err != nil {
return database.NilVersion, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty, nil
}
// Drop deletes everything in the database.
// Note that this is a breaking action, a new call to Open() is necessary to
// ensure subsequent calls work as expected.
func (r *Rqlite) Drop() error {
query := `SELECT name FROM sqlite_master WHERE type = 'table'`
tables, err := r.db.QueryOne(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
statements := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
statement := fmt.Sprintf(`DROP TABLE %s`, tableName)
statements = append(statements, statement)
}
}
// return if nothing to do
if len(statements) <= 0 {
return nil
}
wr, err := r.db.Write(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i])}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(strings.Join(statements, "\n"))}
}
return nil
}
func parseUrl(url string) (*nurl.URL, *Config, error) {
parsedUrl, err := nurl.Parse(url)
if err != nil {
return nil, nil, err
}
config, err := parseConfigFromQuery(parsedUrl.Query())
if err != nil {
return nil, nil, err
}
if parsedUrl.Scheme != "rqlite" {
return nil, nil, fmt.Errorf("bad scheme: %w", ErrBadConfig)
}
// adapt from rqlite to http/https schemes
if config.ConnectInsecure {
parsedUrl.Scheme = "http"
} else {
parsedUrl.Scheme = "https"
}
filteredUrl := migrate.FilterCustomQuery(parsedUrl)
return filteredUrl, config, nil
}
func parseConfigFromQuery(queryVals nurl.Values) (*Config, error) {
c := Config{
ConnectInsecure: DefaultConnectInsecure,
MigrationsTable: DefaultMigrationsTable,
}
migrationsTable := queryVals.Get("x-migrations-table")
if migrationsTable != "" {
if strings.HasPrefix(migrationsTable, "sqlite_") {
return nil, fmt.Errorf("invalid value for x-migrations-table: %w", ErrBadConfig)
}
c.MigrationsTable = migrationsTable
}
connectInsecureStr := queryVals.Get("x-connect-insecure")
if connectInsecureStr != "" {
connectInsecure, err := strconv.ParseBool(connectInsecureStr)
if err != nil {
return nil, fmt.Errorf("invalid value for x-connect-insecure: %w", ErrBadConfig)
}
c.ConnectInsecure = connectInsecure
}
return &c, nil
}
-324
View File
@@ -1,324 +0,0 @@
package rqlite
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"testing"
"github.com/dhui/dktest"
"github.com/rqlite/gorqlite"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
var defaultPort uint16 = 4001
var opts = dktest.Options{
Env: map[string]string{"NODE_ID": "1"},
PortRequired: true,
ReadyFunc: isReady,
}
var specs = []dktesting.ContainerSpec{
{ImageName: "rqlite/rqlite:7.21.4", Options: opts},
{ImageName: "rqlite/rqlite:8.0.6", Options: opts},
{ImageName: "rqlite/rqlite:8.11.1", Options: opts},
{ImageName: "rqlite/rqlite:8.12.3", Options: opts},
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
fmt.Println("error getting port")
return false
}
statusString := fmt.Sprintf("http://%s:%s/status", ip, port)
fmt.Println(statusString)
var readyResp struct {
Store struct {
Ready bool `json:"ready"`
} `json:"store"`
}
resp, err := http.Get(statusString)
if err != nil {
fmt.Println("error getting status")
return false
}
if resp.StatusCode != 200 {
fmt.Println("statusCode != 200")
return false
}
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("error reading body")
return false
}
if err := json.Unmarshal(body, &readyResp); err != nil {
fmt.Println("error unmarshaling body")
return false
}
return readyResp.Store.Ready
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
r := &Rqlite{}
d, err := r.Open(connectString)
assert.NoError(t, err)
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
driver, err := OpenURL(connectString)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
return
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
dt.TestMigrate(t, m)
})
}
func TestBadConnectInsecureParam(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?x-connect-insecure=foo", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestBadProtocol(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("postgres://%s:%s/database", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestNoConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
_, err = WithInstance(db, nil)
assert.ErrorIs(t, err, ErrNilConfig)
})
}
func TestWithInstanceEmptyConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
driver, err := WithInstance(db, &Config{})
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", DefaultMigrationsTable))
assert.NoError(t, err)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
})
}
func TestMigrationTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
config := Config{MigrationsTable: "my_migration_table"}
driver, err := WithInstance(db, &config)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
_, err = db.WriteOne(`INSERT INTO pets (name, predator) VALUES ("franklin", true)`)
assert.NoError(t, err)
res, err := db.QueryOne(`SELECT name, predator FROM pets LIMIT 1`)
assert.NoError(t, err)
_ = res.Next()
// make sure we can use the migrated table
var petName string
var petPredator int
err = res.Scan(&petName, &petPredator)
assert.NoError(t, err)
assert.Equal(t, petName, "franklin")
assert.Equal(t, petPredator, 1)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
})
}
func TestParseUrl(t *testing.T) {
tests := []struct {
name string
passedUrl string
expectedUrl string
expectedConfig *Config
expectedErr string
}{
{
"defaults",
"rqlite://localhost:4001",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"configure migration table",
"rqlite://localhost:4001?x-migrations-table=foo",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: "foo"},
"",
},
{
"configure connect insecure",
"rqlite://localhost:4001?x-connect-insecure=true",
"http://localhost:4001",
&Config{ConnectInsecure: true, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"invalid migration table",
"rqlite://localhost:4001?x-migrations-table=sqlite_bar",
"",
nil,
"invalid value for x-migrations-table: bad parameter",
},
{
"invalid connect insecure",
"rqlite://localhost:4001?x-connect-insecure=baz",
"",
nil,
"invalid value for x-connect-insecure: bad parameter",
},
{
"invalid url",
string([]byte{0x7f}),
"",
nil,
"parse \"\\x7f\": net/url: invalid control character in URL",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
actualUrl, actualConfig, actualErr := parseUrl(tt.passedUrl)
if tt.expectedUrl != "" {
assert.Equal(t, tt.expectedUrl, actualUrl.String())
} else {
assert.Nil(t, actualUrl)
}
assert.Equal(t, tt.expectedConfig, actualConfig)
if tt.expectedErr == "" {
assert.NoError(t, actualErr)
} else {
assert.EqualError(t, actualErr, tt.expectedErr)
}
})
}
}
+15 -10
View File
@@ -3,15 +3,16 @@ package snowflake
import (
"context"
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
sf "github.com/snowflakedb/gosnowflake"
)
@@ -158,21 +159,21 @@ func (p *Snowflake) Close() error {
}
func (p *Snowflake) Lock() error {
if !p.isLocked.CompareAndSwap(false, true) {
if !p.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (p *Snowflake) Unlock() error {
if !p.isLocked.CompareAndSwap(true, false) {
if !p.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (p *Snowflake) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -206,7 +207,7 @@ func (p *Snowflake) Run(migration io.Reader) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -248,7 +249,7 @@ func (p *Snowflake) SetVersion(version int, dirty bool) error {
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -262,7 +263,7 @@ func (p *Snowflake) SetVersion(version int, dirty bool) error {
` + strconv.FormatBool(dirty) + `)`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -304,7 +305,7 @@ func (p *Snowflake) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -346,7 +347,11 @@ func (p *Snowflake) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+1 -1
View File
@@ -33,7 +33,7 @@ as described in [README.md#database-urls](../../README.md#database-urls)
## DDL with comments
At the moment the GCP Spanner backed does not seem to allow for comments (See https://issuetracker.google.com/issues/159730604)
so in order to be able to use migration with DDL containing comments `x-clean-statements` is required
so in order to be able to use migration with DDL containing comments `x-clean-stamements` is required
## Multiple statements
+21 -8
View File
@@ -1,16 +1,17 @@
package spanner
import (
"context"
"errors"
"fmt"
"io"
"io/ioutil"
"log"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"context"
"cloud.google.com/go/spanner"
sdb "cloud.google.com/go/spanner/admin/database/apiv1"
@@ -19,8 +20,10 @@ import (
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
adminpb "cloud.google.com/go/spanner/admin/database/apiv1/databasepb"
"github.com/hashicorp/go-multierror"
uatomic "go.uber.org/atomic"
"google.golang.org/api/iterator"
adminpb "google.golang.org/genproto/googleapis/spanner/admin/database/v1"
)
func init() {
@@ -31,6 +34,11 @@ func init() {
// DefaultMigrationsTable is used if no custom table is specified
const DefaultMigrationsTable = "SchemaMigrations"
const (
unlockedVal = 0
lockedVal = 1
)
// Driver errors
var (
ErrNilConfig = errors.New("no config")
@@ -58,7 +66,7 @@ type Spanner struct {
config *Config
lock atomic.Bool
lock *uatomic.Uint32
}
type DB struct {
@@ -90,6 +98,7 @@ func WithInstance(instance *DB, config *Config) (database.Driver, error) {
sx := &Spanner{
db: instance,
config: config,
lock: uatomic.NewUint32(unlockedVal),
}
if err := sx.ensureVersionTable(); err != nil {
@@ -146,7 +155,7 @@ func (s *Spanner) Close() error {
// Lock implements database.Driver but doesn't do anything because Spanner only
// enqueues the UpdateDatabaseDdlRequest.
func (s *Spanner) Lock() error {
if swapped := s.lock.CompareAndSwap(false, true); swapped {
if swapped := s.lock.CAS(unlockedVal, lockedVal); swapped {
return nil
}
return ErrLockHeld
@@ -154,7 +163,7 @@ func (s *Spanner) Lock() error {
// Unlock implements database.Driver but no action required, see Lock.
func (s *Spanner) Unlock() error {
if swapped := s.lock.CompareAndSwap(true, false); swapped {
if swapped := s.lock.CAS(lockedVal, unlockedVal); swapped {
return nil
}
return ErrLockNotHeld
@@ -162,7 +171,7 @@ func (s *Spanner) Unlock() error {
// Run implements database.Driver
func (s *Spanner) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -298,7 +307,11 @@ func (s *Spanner) ensureVersionTable() (err error) {
defer func() {
if e := s.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+14 -9
View File
@@ -2,16 +2,17 @@ package sqlcipher
import (
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/mutecomm/go-sqlcipher/v4"
)
@@ -72,7 +73,11 @@ func (m *Sqlite) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -136,7 +141,7 @@ func (m *Sqlite) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -173,21 +178,21 @@ func (m *Sqlite) Drop() (err error) {
}
func (m *Sqlite) Lock() error {
if !m.isLocked.CompareAndSwap(false, true) {
if !m.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (m *Sqlite) Unlock() error {
if !m.isLocked.CompareAndSwap(true, false) {
if !m.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (m *Sqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -206,7 +211,7 @@ func (m *Sqlite) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -241,7 +246,7 @@ func (m *Sqlite) SetVersion(version int, dirty bool) error {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+48 -6
View File
@@ -3,6 +3,8 @@ package sqlcipher
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -15,7 +17,15 @@ import (
)
func Test(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s", filepath.Join(dir, "sqlite3.db"))
@@ -27,7 +37,15 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
db, err := sql.Open("sqlite3", filepath.Join(dir, "sqlite3.db"))
@@ -54,7 +72,15 @@ func TestMigrate(t *testing.T) {
}
func TestMigrationTable(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test-migration-table")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
@@ -94,7 +120,15 @@ func TestMigrationTable(t *testing.T) {
}
func TestNoTxWrap(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s?x-no-tx-wrap=true", filepath.Join(dir, "sqlite3.db"))
@@ -108,11 +142,19 @@ func TestNoTxWrap(t *testing.T) {
}
func TestNoTxWrapInvalidValue(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s?x-no-tx-wrap=yeppers", filepath.Join(dir, "sqlite3.db"))
_, err := p.Open(addr)
_, err = p.Open(addr)
if assert.Error(t, err) {
assert.Contains(t, err.Error(), "x-no-tx-wrap")
assert.Contains(t, err.Error(), "invalid syntax")
+14 -9
View File
@@ -2,16 +2,17 @@ package sqlite
import (
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "modernc.org/sqlite"
)
@@ -72,7 +73,11 @@ func (m *Sqlite) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -136,7 +141,7 @@ func (m *Sqlite) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -173,21 +178,21 @@ func (m *Sqlite) Drop() (err error) {
}
func (m *Sqlite) Lock() error {
if !m.isLocked.CompareAndSwap(false, true) {
if !m.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (m *Sqlite) Unlock() error {
if !m.isLocked.CompareAndSwap(true, false) {
if !m.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (m *Sqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -206,7 +211,7 @@ func (m *Sqlite) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -241,7 +246,7 @@ func (m *Sqlite) SetVersion(version int, dirty bool) error {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+57 -7
View File
@@ -3,6 +3,8 @@ package sqlite
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -15,7 +17,15 @@ import (
)
func Test(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://%s", filepath.Join(dir, "sqlite.db"))
@@ -27,7 +37,15 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
db, err := sql.Open("sqlite", filepath.Join(dir, "sqlite.db"))
@@ -54,7 +72,15 @@ func TestMigrate(t *testing.T) {
}
func TestMigrationTable(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite-driver-test-migration-table")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
@@ -94,7 +120,15 @@ func TestMigrationTable(t *testing.T) {
}
func TestNoTxWrap(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://%s?x-no-tx-wrap=true", filepath.Join(dir, "sqlite.db"))
@@ -108,11 +142,19 @@ func TestNoTxWrap(t *testing.T) {
}
func TestNoTxWrapInvalidValue(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite://%s?x-no-tx-wrap=yeppers", filepath.Join(dir, "sqlite.db"))
_, err := p.Open(addr)
_, err = p.Open(addr)
if assert.Error(t, err) {
assert.Contains(t, err.Error(), "x-no-tx-wrap")
assert.Contains(t, err.Error(), "invalid syntax")
@@ -120,7 +162,15 @@ func TestNoTxWrapInvalidValue(t *testing.T) {
}
func TestMigrateWithDirectoryNameContainsWhitespaces(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "directory name contains whitespaces")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dbPath := filepath.Join(dir, "sqlite.db")
t.Logf("DB path : %s\n", dbPath)
p := &Sqlite{}
+14 -9
View File
@@ -2,16 +2,17 @@ package sqlite3
import (
"database/sql"
"errors"
"fmt"
"go.uber.org/atomic"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/mattn/go-sqlite3"
)
@@ -72,7 +73,11 @@ func (m *Sqlite) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -136,7 +141,7 @@ func (m *Sqlite) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -173,21 +178,21 @@ func (m *Sqlite) Drop() (err error) {
}
func (m *Sqlite) Lock() error {
if !m.isLocked.CompareAndSwap(false, true) {
if !m.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (m *Sqlite) Unlock() error {
if !m.isLocked.CompareAndSwap(true, false) {
if !m.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (m *Sqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -206,7 +211,7 @@ func (m *Sqlite) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -241,7 +246,7 @@ func (m *Sqlite) SetVersion(version int, dirty bool) error {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+57 -7
View File
@@ -3,6 +3,8 @@ package sqlite3
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
@@ -15,7 +17,15 @@ import (
)
func Test(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s", filepath.Join(dir, "sqlite3.db"))
@@ -27,7 +37,15 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
db, err := sql.Open("sqlite3", filepath.Join(dir, "sqlite3.db"))
@@ -54,7 +72,15 @@ func TestMigrate(t *testing.T) {
}
func TestMigrationTable(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test-migration-table")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
@@ -94,7 +120,15 @@ func TestMigrationTable(t *testing.T) {
}
func TestNoTxWrap(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s?x-no-tx-wrap=true", filepath.Join(dir, "sqlite3.db"))
@@ -108,11 +142,19 @@ func TestNoTxWrap(t *testing.T) {
}
func TestNoTxWrapInvalidValue(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "sqlite3-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "sqlite3.db"))
p := &Sqlite{}
addr := fmt.Sprintf("sqlite3://%s?x-no-tx-wrap=yeppers", filepath.Join(dir, "sqlite3.db"))
_, err := p.Open(addr)
_, err = p.Open(addr)
if assert.Error(t, err) {
assert.Contains(t, err.Error(), "x-no-tx-wrap")
assert.Contains(t, err.Error(), "invalid syntax")
@@ -120,7 +162,15 @@ func TestNoTxWrapInvalidValue(t *testing.T) {
}
func TestMigrateWithDirectoryNameContainsWhitespaces(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "directory name contains whitespaces")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
dbPath := filepath.Join(dir, "sqlite3.db")
t.Logf("DB path : %s\n", dbPath)
p := &Sqlite{}
+2 -2
View File
@@ -18,14 +18,14 @@
| `app+name` || The application name (default is go-mssqldb). |
| `useMsi` | | `true` - Use Azure MSI Authentication for connecting to Sql Server. Must be running from an Azure VM/an instance with MSI enabled. `false` - Use password authentication (Default). See [here for Azure MSI Auth details](https://docs.microsoft.com/en-us/azure/app-service/app-service-web-tutorial-connect-msi). NOTE: Since this cannot be tested locally, this is not officially supported.
See https://github.com/microsoft/go-mssqldb for full parameter list.
See https://github.com/denisenkom/go-mssqldb for full parameter list.
## Driver Support
### Which go-mssqldb driver to us?
Please note that the deprecated `mssql` driver is not supported. Please use the newer `sqlserver` driver.
See https://github.com/microsoft/go-mssqldb#deprecated for more information.
See https://github.com/denisenkom/go-mssqldb#deprecated for more information.
### Official Support by migrate
+26 -30
View File
@@ -3,18 +3,20 @@ package sqlserver
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"go.uber.org/atomic"
"github.com/Azure/go-autorest/autorest/adal"
mssql "github.com/denisenkom/go-mssqldb" // mssql support
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
mssql "github.com/microsoft/go-mssqldb" // mssql support
"github.com/hashicorp/go-multierror"
)
func init() {
@@ -29,10 +31,10 @@ var (
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
ErrDatabaseDirty = fmt.Errorf("database is dirty")
ErrMultipleAuthOptionsPassed = fmt.Errorf("both password and useMsi=true were passed")
ErrMultipleAuthOptionsPassed = fmt.Errorf("both password and useMsi=true were passed.")
)
var lockErrorMap = map[int]string{
var lockErrorMap = map[mssql.ReturnStatus]string{
-1: "The lock request timed out.",
-2: "The lock request was canceled.",
-3: "The lock request was chosen as a deadlock victim.",
@@ -197,24 +199,18 @@ func (ss *SQLServer) Lock() error {
return err
}
// This will block until the lock is acquired.
// This will either obtain the lock immediately and return true,
// or return false if the lock cannot be acquired immediately.
// MS Docs: sp_getapplock: https://docs.microsoft.com/en-us/sql/relational-databases/system-stored-procedures/sp-getapplock-transact-sql?view=sql-server-2017
query := `
DECLARE @lockResult int;
EXEC @lockResult = sp_getapplock @Resource = @p1, @LockMode = 'Exclusive', @LockOwner = 'Session', @LockTimeout = -1;
SELECT @lockResult;`
query := `EXEC sp_getapplock @Resource = @p1, @LockMode = 'Update', @LockOwner = 'Session', @LockTimeout = 0`
var status int
if err = ss.conn.QueryRowContext(context.Background(), query, aid).Scan(&status); err == nil && status > -1 {
var status mssql.ReturnStatus
if _, err = ss.conn.ExecContext(context.Background(), query, aid, &status); err == nil && status > -1 {
return nil
} else if err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
} else {
errorDescription, ok := lockErrorMap[status]
if !ok {
errorDescription = "Unknown error"
}
return &database.Error{Err: fmt.Sprintf("try lock failed with error %v: %v", status, errorDescription), Query: []byte(query)}
return &database.Error{Err: fmt.Sprintf("try lock failed with error %v: %v", status, lockErrorMap[status]), Query: []byte(query)}
}
})
}
@@ -239,7 +235,7 @@ func (ss *SQLServer) Unlock() error {
// Run the migrations for the database
func (ss *SQLServer) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -268,10 +264,10 @@ func (ss *SQLServer) SetVersion(version int, dirty bool) error {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE TABLE ` + ss.getMigrationTable()
query := `TRUNCATE TABLE "` + ss.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -284,10 +280,10 @@ func (ss *SQLServer) SetVersion(version int, dirty bool) error {
if dirty {
dirtyBit = 1
}
query = `INSERT INTO ` + ss.getMigrationTable() + ` (version, dirty) VALUES (@p1, @p2)`
query = `INSERT INTO "` + ss.config.MigrationsTable + `" (version, dirty) VALUES (@p1, @p2)`
if _, err := tx.Exec(query, version, dirtyBit); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -302,7 +298,7 @@ func (ss *SQLServer) SetVersion(version int, dirty bool) error {
// Version of the current database state
func (ss *SQLServer) Version() (version int, dirty bool, err error) {
query := `SELECT TOP 1 version, dirty FROM ` + ss.getMigrationTable()
query := `SELECT TOP 1 version, dirty FROM "` + ss.config.MigrationsTable + `"`
err = ss.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
@@ -359,17 +355,21 @@ func (ss *SQLServer) ensureVersionTable() (err error) {
defer func() {
if e := ss.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
query := `IF NOT EXISTS
(SELECT *
FROM sysobjects
WHERE id = object_id(N'` + ss.getMigrationTable() + `')
WHERE id = object_id(N'[dbo].[` + ss.config.MigrationsTable + `]')
AND OBJECTPROPERTY(id, N'IsUserTable') = 1
)
CREATE TABLE ` + ss.getMigrationTable() + ` ( version BIGINT PRIMARY KEY NOT NULL, dirty BIT NOT NULL );`
CREATE TABLE ` + ss.config.MigrationsTable + ` ( version BIGINT PRIMARY KEY NOT NULL, dirty BIT NOT NULL );`
if _, err = ss.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
@@ -378,10 +378,6 @@ func (ss *SQLServer) ensureVersionTable() (err error) {
return nil
}
func (ss *SQLServer) getMigrationTable() string {
return fmt.Sprintf("[%s].[%s]", ss.config.SchemaName, ss.config.MigrationsTable)
}
func getMSITokenProvider(resource string) (func() (string, error), error) {
msi, err := adal.NewServicePrincipalTokenFromManagedIdentity(resource, nil)
if err != nil {
+13 -51
View File
@@ -6,7 +6,6 @@ import (
sqldriver "database/sql/driver"
"fmt"
"log"
"runtime"
"strings"
"testing"
"time"
@@ -24,16 +23,14 @@ const defaultPort = 1433
const saPassword = "Root1234"
var (
sqlServerOpts = dktest.Options{
Env: map[string]string{"ACCEPT_EULA": "Y", "MSSQL_SA_PASSWORD": saPassword, "MSSQL_PID": "Express"},
opts = dktest.Options{
Env: map[string]string{"ACCEPT_EULA": "Y", "SA_PASSWORD": saPassword, "MSSQL_PID": "Express"},
PortRequired: true, ReadyFunc: isReady, PullTimeout: 2 * time.Minute,
}
// Container versions: https://mcr.microsoft.com/v2/mssql/server/tags/list
specs = []dktesting.ContainerSpec{
{ImageName: "mcr.microsoft.com/mssql/server:2022-latest", Options: sqlServerOpts},
{ImageName: "mcr.microsoft.com/mssql/server:2019-latest", Options: sqlServerOpts},
// Add back support for 2017 version once the image is fixed: https://github.com/microsoft/mssql-docker/issues/899
// {ImageName: "mcr.microsoft.com/mssql/server:2017-latest", Options: sqlServerOpts},
{ImageName: "mcr.microsoft.com/mssql/server:2017-latest", Options: opts},
{ImageName: "mcr.microsoft.com/mssql/server:2019-latest", Options: opts},
}
)
@@ -77,35 +74,8 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
return true
}
func SkipIfUnsupportedArch(t *testing.T, c dktest.ContainerInfo) {
if strings.Contains(c.ImageName, "mssql") && !strings.HasPrefix(runtime.GOARCH, "amd") {
t.Skipf("Image %s is not supported on arch %s", c.ImageName, runtime.GOARCH)
}
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testMultiStatement", testMultiStatement)
t.Run("testErrorParsing", testErrorParsing)
t.Run("testLockWorks", testLockWorks)
t.Run("testMsiTrue", testMsiTrue)
t.Run("testOpenWithPasswordAndMSI", testOpenWithPasswordAndMSI)
t.Run("testMsiFalse", testMsiFalse)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
@@ -128,9 +98,8 @@ func test(t *testing.T) {
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
@@ -157,10 +126,9 @@ func testMigrate(t *testing.T) {
})
}
func testMultiStatement(t *testing.T) {
func TestMultiStatement(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
@@ -191,16 +159,14 @@ func testMultiStatement(t *testing.T) {
})
}
func testErrorParsing(t *testing.T) {
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := msConnectionString(ip, port)
p := &SQLServer{}
d, err := p.Open(addr)
if err != nil {
@@ -223,9 +189,8 @@ func testErrorParsing(t *testing.T) {
})
}
func testLockWorks(t *testing.T) {
func TestLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
@@ -262,9 +227,8 @@ func testLockWorks(t *testing.T) {
})
}
func testMsiTrue(t *testing.T) {
func TestMsiTrue(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
@@ -279,9 +243,8 @@ func testMsiTrue(t *testing.T) {
})
}
func testOpenWithPasswordAndMSI(t *testing.T) {
func TestOpenWithPasswordAndMSI(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
@@ -311,9 +274,8 @@ func testOpenWithPasswordAndMSI(t *testing.T) {
})
}
func testMsiFalse(t *testing.T) {
func TestMsiFalse(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
SkipIfUnsupportedArch(t, c)
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
+5 -4
View File
@@ -1,9 +1,10 @@
package stub
import (
"go.uber.org/atomic"
"io"
"io/ioutil"
"reflect"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
)
@@ -49,21 +50,21 @@ func (s *Stub) Close() error {
}
func (s *Stub) Lock() error {
if !s.isLocked.CompareAndSwap(false, true) {
if !s.isLocked.CAS(false, true) {
return database.ErrLocked
}
return nil
}
func (s *Stub) Unlock() error {
if !s.isLocked.CompareAndSwap(true, false) {
if !s.isLocked.CAS(true, false) {
return database.ErrNotLocked
}
return nil
}
func (s *Stub) Run(migration io.Reader) error {
m, err := io.ReadAll(migration)
m, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
+2 -1
View File
@@ -50,7 +50,7 @@ func TestLockAndUnlock(t *testing.T, d database.Driver) {
case <-done:
return
case <-timeout:
errs <- fmt.Errorf("timeout after 15 seconds, looks like a deadlock in Lock/UnLock\n%#v", d)
errs <- fmt.Errorf("Timeout after 15 seconds. Looks like a deadlock in Lock/UnLock.\n%#v", d)
return
}
}
@@ -116,6 +116,7 @@ func TestDrop(t *testing.T, d database.Driver) {
}
func TestSetVersion(t *testing.T, d database.Driver) {
// nolint:maligned
testCases := []struct {
name string
version int
+2 -2
View File
@@ -2,9 +2,9 @@ package database
import (
"fmt"
"go.uber.org/atomic"
"hash/crc32"
"strings"
"sync/atomic"
)
const advisoryLockIDSalt uint = 1486364155
@@ -21,7 +21,7 @@ func GenerateAdvisoryLockId(databaseName string, additionalNames ...string) (str
// CasRestoreOnErr CAS wrapper to automatically restore the lock state on error
func CasRestoreOnErr(lock *atomic.Bool, o, n bool, casErr error, f func() error) error {
if !lock.CompareAndSwap(o, n) {
if !lock.CAS(o, n) {
return casErr
}
if err := f(); err != nil {
+7 -9
View File
@@ -2,7 +2,7 @@ package database
import (
"errors"
"sync/atomic"
"go.uber.org/atomic"
"testing"
)
@@ -54,7 +54,7 @@ func TestCasRestoreOnErr(t *testing.T) {
testcases := []struct {
name string
lock bool
lock *atomic.Bool
from bool
to bool
expectLock bool
@@ -63,7 +63,7 @@ func TestCasRestoreOnErr(t *testing.T) {
}{
{
name: "Test positive CAS lock",
lock: false,
lock: atomic.NewBool(false),
from: false,
to: true,
expectLock: true,
@@ -72,7 +72,7 @@ func TestCasRestoreOnErr(t *testing.T) {
},
{
name: "Test negative CAS lock",
lock: true,
lock: atomic.NewBool(true),
from: false,
to: true,
expectLock: true,
@@ -81,7 +81,7 @@ func TestCasRestoreOnErr(t *testing.T) {
},
{
name: "Test negative with callback lock",
lock: false,
lock: atomic.NewBool(false),
from: false,
to: true,
expectLock: false,
@@ -92,15 +92,13 @@ func TestCasRestoreOnErr(t *testing.T) {
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
var lock atomic.Bool
lock.Store(tc.lock)
if err := CasRestoreOnErr(&lock, tc.from, tc.to, casErr, func() error {
if err := CasRestoreOnErr(tc.lock, tc.from, tc.to, casErr, func() error {
return tc.fErr
}); err != tc.expectError {
t.Error("Incorrect error value returned")
}
if lock.Load() != tc.expectLock {
if tc.lock.Load() != tc.expectLock {
t.Error("Incorrect state of lock")
}
})
-22
View File
@@ -1,22 +0,0 @@
# yugabytedb
`yugabytedb://user:password@host:port/dbname?query` (`yugabyte://`, and `ysql://` work, too)
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-lock-table` | `LockTable` | Name of the table which maintains the migration lock |
| `x-force-lock` | `ForceLock` | Force lock acquisition to fix faulty migrations which may not have released the schema lock (Boolean, default is `false`) |
| `x-max-retries` | `MaxRetries` | How many times retry queries on retryable errors (40001, 40P01, 08006, XX000). Default is 10 |
| `x-max-retry-interval` | `MaxRetryInterval` | Interval between retries increases exponentially. This option specifies maximum duration between retries. Default is 15s |
| `x-max-retry-elapsed-time` | `MaxRetryElapsedTime` | Total retries timeout. Default is 30s |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
@@ -1 +0,0 @@
DROP TABLE IF EXISTS users;
@@ -1,5 +0,0 @@
CREATE TABLE users (
user_id integer unique,
name varchar(40),
email varchar(40)
);
@@ -1 +0,0 @@
ALTER TABLE users DROP COLUMN IF EXISTS city;
@@ -1,3 +0,0 @@
ALTER TABLE users ADD COLUMN city varchar(100);
@@ -1 +0,0 @@
DROP INDEX IF EXISTS users_email_index;
@@ -1,3 +0,0 @@
CREATE UNIQUE INDEX users_email_index ON users (email);
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
DROP TABLE IF EXISTS books;
@@ -1,5 +0,0 @@
CREATE TABLE books (
user_id integer,
name varchar(40),
author varchar(40)
);
@@ -1 +0,0 @@
DROP TABLE IF EXISTS movies;
@@ -1,5 +0,0 @@
CREATE TABLE movies (
user_id integer,
name varchar(40),
director varchar(40)
);
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
-475
View File
@@ -1,475 +0,0 @@
package yugabytedb
import (
"context"
"database/sql"
"errors"
"io"
"net/url"
"regexp"
"strconv"
"sync/atomic"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/jackc/pgconn"
"github.com/jackc/pgerrcode"
"github.com/lib/pq"
)
const (
DefaultMaxRetryInterval = time.Second * 15
DefaultMaxRetryElapsedTime = time.Second * 30
DefaultMaxRetries = 10
DefaultMigrationsTable = "migrations"
DefaultLockTable = "migrations_locks"
)
var (
ErrNilConfig = errors.New("no config")
ErrNoDatabaseName = errors.New("no database name")
ErrMaxRetriesExceeded = errors.New("max retries exceeded")
)
func init() {
db := YugabyteDB{}
database.Register("yugabyte", &db)
database.Register("yugabytedb", &db)
database.Register("ysql", &db)
}
type Config struct {
MigrationsTable string
LockTable string
ForceLock bool
DatabaseName string
MaxRetryInterval time.Duration
MaxRetryElapsedTime time.Duration
MaxRetries int
}
type YugabyteDB struct {
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT current_database()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if len(config.LockTable) == 0 {
config.LockTable = DefaultLockTable
}
if config.MaxRetryInterval == 0 {
config.MaxRetryInterval = DefaultMaxRetryInterval
}
if config.MaxRetryElapsedTime == 0 {
config.MaxRetryElapsedTime = DefaultMaxRetryElapsedTime
}
if config.MaxRetries == 0 {
config.MaxRetries = DefaultMaxRetries
}
px := &YugabyteDB{
db: instance,
config: config,
}
// ensureVersionTable is a locking operation, so we need to ensureLockTable before we ensureVersionTable.
if err := px.ensureLockTable(); err != nil {
return nil, err
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (c *YugabyteDB) Open(dbURL string) (database.Driver, error) {
purl, err := url.Parse(dbURL)
if err != nil {
return nil, err
}
// As YugabyteDB uses the postgres protocol, and 'postgres' is already a registered database, we need to replace the
// connect prefix, with the actual protocol, so that the library can differentiate between the implementations
re := regexp.MustCompile("^(yugabyte(db)?|ysql)")
connectString := re.ReplaceAllString(migrate.FilterCustomQuery(purl).String(), "postgres")
db, err := sql.Open("postgres", connectString)
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
if len(migrationsTable) == 0 {
migrationsTable = DefaultMigrationsTable
}
lockTable := purl.Query().Get("x-lock-table")
if len(lockTable) == 0 {
lockTable = DefaultLockTable
}
forceLockQuery := purl.Query().Get("x-force-lock")
forceLock, err := strconv.ParseBool(forceLockQuery)
if err != nil {
forceLock = false
}
maxIntervalStr := purl.Query().Get("x-max-retry-interval")
maxInterval, err := time.ParseDuration(maxIntervalStr)
if err != nil {
maxInterval = DefaultMaxRetryInterval
}
maxElapsedTimeStr := purl.Query().Get("x-max-retry-elapsed-time")
maxElapsedTime, err := time.ParseDuration(maxElapsedTimeStr)
if err != nil {
maxElapsedTime = DefaultMaxRetryElapsedTime
}
maxRetriesStr := purl.Query().Get("x-max-retries")
maxRetries, err := strconv.Atoi(maxRetriesStr)
if err != nil {
maxRetries = DefaultMaxRetries
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
LockTable: lockTable,
ForceLock: forceLock,
MaxRetryInterval: maxInterval,
MaxRetryElapsedTime: maxElapsedTime,
MaxRetries: maxRetries,
})
if err != nil {
return nil, err
}
return px, nil
}
func (c *YugabyteDB) Close() error {
return c.db.Close()
}
// Locking is done manually with a separate lock table. Implementing advisory locks in YugabyteDB is being discussed
// See: https://github.com/yugabyte/yugabyte-db/issues/3642
func (c *YugabyteDB) Lock() error {
return database.CasRestoreOnErr(&c.isLocked, false, true, database.ErrLocked, func() (err error) {
return c.doTxWithRetry(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable}, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return nil
})
})
}
// Locking is done manually with a separate lock table. Implementing advisory locks in YugabyteDB is being discussed
// See: https://github.com/yugabyte/yugabyte-db/issues/3642
func (c *YugabyteDB) Unlock() error {
return database.CasRestoreOnErr(&c.isLocked, true, false, database.ErrNotLocked, func() (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in YugabyteDB
// https://github.com/yugabyte/yugabyte-db/blob/9c6b8e6beb56eed8eeb357178c0c6b837eb49896/src/postgres/src/backend/utils/errcodes.txt#L366
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
return nil
}
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
})
}
func (c *YugabyteDB) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if _, err := c.db.Exec(query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func (c *YugabyteDB) SetVersion(version int, dirty bool) error {
return c.doTxWithRetry(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable}, func(tx *sql.Tx) error {
if _, err := tx.Exec(`DELETE FROM "` + c.config.MigrationsTable + `"`); err != nil {
return err
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
if _, err := tx.Exec(`INSERT INTO "`+c.config.MigrationsTable+`" (version, dirty) VALUES ($1, $2)`, version, dirty); err != nil {
return err
}
}
return nil
})
}
func (c *YugabyteDB) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM "` + c.config.MigrationsTable + `" LIMIT 1`
err = c.db.QueryRow(query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in YugabyteDB
// https://github.com/yugabyte/yugabyte-db/blob/9c6b8e6beb56eed8eeb357178c0c6b837eb49896/src/postgres/src/backend/utils/errcodes.txt#L366
if e.Code == "42P01" {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (c *YugabyteDB) Drop() (err error) {
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := c.db.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + t + ` CASCADE`
if _, err := c.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database
func (c *YugabyteDB) ensureVersionTable() (err error) {
if err = c.Lock(); err != nil {
return err
}
defer func() {
if e := c.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// check if migration table exists
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := c.db.QueryRow(query, c.config.MigrationsTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
// if not, create the empty migration table
query = `CREATE TABLE "` + c.config.MigrationsTable + `" (version INT NOT NULL PRIMARY KEY, dirty BOOL NOT NULL)`
if _, err := c.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (c *YugabyteDB) ensureLockTable() error {
// check if lock table exists
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := c.db.QueryRow(query, c.config.LockTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
// if not, create the empty lock table
query = `CREATE TABLE "` + c.config.LockTable + `" (lock_id TEXT NOT NULL PRIMARY KEY)`
if _, err := c.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (c *YugabyteDB) doTxWithRetry(
ctx context.Context,
txOpts *sql.TxOptions,
fn func(tx *sql.Tx) error,
) error {
backOff := c.newBackoff(ctx)
return backoff.Retry(func() error {
tx, err := c.db.BeginTx(ctx, txOpts)
if err != nil {
return backoff.Permanent(err)
}
// If we've tried to commit the transaction Rollback just returns sql.ErrTxDone.
defer func() {
_ = tx.Rollback()
}()
if err := fn(tx); err != nil {
if errIsRetryable(err) {
return err
}
return backoff.Permanent(err)
}
if err := tx.Commit(); err != nil {
if errIsRetryable(err) {
return err
}
return backoff.Permanent(err)
}
return nil
}, backOff)
}
func (c *YugabyteDB) newBackoff(ctx context.Context) backoff.BackOff {
if ctx == nil {
ctx = context.Background()
}
retrier := backoff.WithMaxRetries(backoff.WithContext(&backoff.ExponentialBackOff{
InitialInterval: backoff.DefaultInitialInterval,
RandomizationFactor: backoff.DefaultRandomizationFactor,
Multiplier: backoff.DefaultMultiplier,
MaxInterval: c.config.MaxRetryInterval,
MaxElapsedTime: c.config.MaxRetryElapsedTime,
Stop: backoff.Stop,
Clock: backoff.SystemClock,
}, ctx), uint64(c.config.MaxRetries))
retrier.Reset()
return retrier
}
func errIsRetryable(err error) bool {
var pgErr *pgconn.PgError
if !errors.As(err, &pgErr) {
return false
}
// Assume that it's safe to retry 08006 and XX000 because we check for lock existence
// before creating and lock ID is primary key. Version field in migrations table is primary key too
// and delete all versions is an idempotent operation.
return pgErr.Code == pgerrcode.SerializationFailure || // optimistic locking conflict
pgErr.Code == pgerrcode.DeadlockDetected ||
pgErr.Code == pgerrcode.ConnectionFailure || // node down, need to reconnect
pgErr.Code == pgerrcode.InternalError // may happen during HA
}
-199
View File
@@ -1,199 +0,0 @@
package yugabytedb
// error codes https://github.com/lib/pq/blob/master/error.go
import (
"context"
"database/sql"
"fmt"
"log"
"strings"
"testing"
"time"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
_ "github.com/lib/pq"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const defaultPort = 5433
var (
opts = dktest.Options{
Cmd: []string{"bin/yugabyted", "start", "--daemon=false"},
PortRequired: true,
ReadyFunc: isReady,
Timeout: time.Duration(60) * time.Second,
}
// Released versions: https://docs.yugabyte.com/preview/releases/release-notes/
specs = []dktesting.ContainerSpec{
{ImageName: "yugabytedb/yugabyte:2.14.15.0-b57", Options: opts},
{ImageName: "yugabytedb/yugabyte:2.20.2.1-b3", Options: opts},
}
)
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
log.Println("port error:", err)
return false
}
db, err := sql.Open("postgres", fmt.Sprintf("postgres://yugabyte:yugabyte@%v:%v?sslmode=disable", ip, port))
if err != nil {
log.Println("open error:", err)
return false
}
if err := db.PingContext(ctx); err != nil {
log.Println("ping error:", err)
return false
}
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
return true
}
func createDB(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
db, err := sql.Open("postgres", fmt.Sprintf("postgres://yugabyte:yugabyte@%v:%v?sslmode=disable", ip, port))
if err != nil {
t.Fatal(err)
}
if err = db.Ping(); err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
if _, err = db.Exec("CREATE DATABASE migrate"); err != nil {
t.Fatal(err)
}
}
func getConnectionString(ip, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("yugabyte://yugabyte:yugabyte@%v:%v/migrate?%s", ip, port, strings.Join(options, "&"))
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testMultiStatement", testMultiStatement)
t.Run("testFilterCustomQuery", testFilterCustomQuery)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, ci dktest.ContainerInfo) {
createDB(t, ci)
ip, port, err := ci.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := getConnectionString(ip, port)
c := &YugabyteDB{}
d, err := c.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, ci dktest.ContainerInfo) {
createDB(t, ci)
ip, port, err := ci.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := getConnectionString(ip, port)
c := &YugabyteDB{}
d, err := c.Open(addr)
if err != nil {
t.Fatal(err)
}
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "migrate", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func testMultiStatement(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, ci dktest.ContainerInfo) {
createDB(t, ci)
ip, port, err := ci.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := getConnectionString(ip, port)
c := &YugabyteDB{}
d, err := c.Open(addr)
if err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*YugabyteDB).db.QueryRow("SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatal("expected table bar to exist")
}
})
}
func testFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, ci dktest.ContainerInfo) {
createDB(t, ci)
ip, port, err := ci.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := getConnectionString(ip, port, "x-custom=foobar")
c := &YugabyteDB{}
d, err := c.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
})
}
+2 -28
View File
@@ -1,13 +1,11 @@
package dktesting
import (
"context"
"fmt"
"testing"
)
import (
"github.com/dhui/dktest"
"github.com/docker/docker/api/types/image"
"github.com/docker/docker/client"
)
// ContainerSpec holds Docker testing setup specifications
@@ -16,30 +14,6 @@ type ContainerSpec struct {
Options dktest.Options
}
// Cleanup cleanups the ContainerSpec after a test run by removing the ContainerSpec's image
func (s *ContainerSpec) Cleanup() (retErr error) {
// copied from dktest.RunContext()
dc, err := client.NewClientWithOpts(client.FromEnv, client.WithVersion("1.41"))
if err != nil {
return err
}
defer func() {
if err := dc.Close(); err != nil && retErr == nil {
retErr = fmt.Errorf("error closing Docker client: %w", err)
}
}()
cleanupTimeout := s.Options.CleanupTimeout
if cleanupTimeout <= 0 {
cleanupTimeout = dktest.DefaultCleanupTimeout
}
ctx, timeoutCancelFunc := context.WithTimeout(context.Background(), cleanupTimeout)
defer timeoutCancelFunc()
if _, err := dc.ImageRemove(ctx, s.ImageName, image.RemoveOptions{Force: true, PruneChildren: true}); err != nil {
return err
}
return nil
}
// ParallelTest runs Docker tests in parallel
func ParallelTest(t *testing.T, specs []ContainerSpec,
testFunc func(*testing.T, dktest.ContainerInfo)) {
+45 -207
View File
@@ -1,231 +1,69 @@
module github.com/golang-migrate/migrate/v4
go 1.26.4
require (
cloud.google.com/go/spanner v1.85.0
cloud.google.com/go/storage v1.56.0
github.com/Azure/go-autorest/autorest/adal v0.9.16
github.com/aws/aws-sdk-go v1.49.6
github.com/cenkalti/backoff/v4 v4.3.0
cloud.google.com/go/spanner v1.24.0
cloud.google.com/go/storage v1.10.0
github.com/Azure/go-autorest/autorest/adal v0.9.14
github.com/ClickHouse/clickhouse-go v1.4.3
github.com/Microsoft/go-winio v0.5.0 // indirect
github.com/apache/arrow/go/arrow v0.0.0-20211013220434-5962184e7a30 // indirect
github.com/aws/aws-sdk-go v1.17.7
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.5.4 // indirect
github.com/cenkalti/backoff/v4 v4.0.2
github.com/cockroachdb/cockroach-go/v2 v2.1.1
github.com/dhui/dktest v0.4.6
github.com/docker/docker v28.3.3+incompatible
github.com/cznic/mathutil v0.0.0-20180504122225-ca4c9f2c1369 // indirect
github.com/denisenkom/go-mssqldb v0.10.0
github.com/dhui/dktest v0.3.7
github.com/docker/docker v20.10.9+incompatible
github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712 // indirect
github.com/fsouza/fake-gcs-server v1.17.0
github.com/gabriel-vasile/mimetype v1.4.0 // indirect
github.com/go-sql-driver/mysql v1.5.0
github.com/gobuffalo/here v0.6.0
github.com/gocql/gocql v0.0.0-20210515062232-b7ef815b4556
github.com/google/go-github/v39 v39.2.0
github.com/hanzo-ds/go v1.0.1
github.com/jackc/pgconn v1.14.3
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa
github.com/jackc/pgx/v4 v4.18.2
github.com/jackc/pgx/v5 v5.7.6
github.com/gofrs/uuid v4.0.0+incompatible // indirect
github.com/google/go-github/v35 v35.2.0
github.com/gorilla/mux v1.7.4 // indirect
github.com/hashicorp/go-multierror v1.1.0
github.com/jackc/pgconn v1.8.0
github.com/jackc/pgerrcode v0.0.0-20201024163028-a0d42d470451
github.com/jackc/pgproto3/v2 v2.0.7 // indirect
github.com/jackc/pgx/v4 v4.10.1
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 // indirect
github.com/klauspost/compress v1.13.6 // indirect
github.com/ktrysmt/go-bitbucket v0.6.4
github.com/lib/pq v1.10.9
github.com/lib/pq v1.10.0
github.com/markbates/pkger v0.15.1
github.com/mattn/go-sqlite3 v1.14.22
github.com/microsoft/go-mssqldb v1.0.0
github.com/mattn/go-sqlite3 v1.14.6
github.com/moby/term v0.0.0-20210619224110-3f7ff695adc6 // indirect
github.com/mutecomm/go-sqlcipher/v4 v4.4.0
github.com/nakagami/firebirdsql v0.0.0-20190310045651-3c02a58cfed8
github.com/neo4j/neo4j-go-driver v1.8.1-0.20200803113522-b626aa943eba
github.com/snowflakedb/gosnowflake v1.19.1
github.com/stretchr/testify v1.11.1
github.com/xanzy/go-gitlab v0.15.0
go.mongodb.org/mongo-driver v1.11.4
golang.org/x/oauth2 v0.30.0
golang.org/x/tools/godoc v0.1.0-deprecated
google.golang.org/api v0.247.0
modernc.org/ql v1.0.0
modernc.org/sqlite v1.29.6
)
require (
cel.dev/expr v0.24.0 // indirect
cloud.google.com/go/auth v0.16.4 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/monitoring v1.24.2 // indirect
github.com/BurntSushi/toml v1.4.0 // indirect
github.com/GoogleCloudPlatform/grpc-gcp-go/grpcgcp v1.5.3 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.27.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.53.0 // indirect
github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.53.0 // indirect
github.com/apache/arrow-go/v18 v18.4.0 // indirect
github.com/aws/aws-sdk-go-v2/config v1.32.31 // indirect
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.31 // indirect
github.com/aws/aws-sdk-go-v2/service/signin v1.5.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.33.0 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.38.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.45.0 // indirect
github.com/containerd/errdefs v1.0.0 // indirect
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/docker/go-connections v0.5.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.32.4 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-faster/city v1.0.1 // indirect
github.com/go-faster/errors v0.7.1 // indirect
github.com/go-jose/go-jose/v4 v4.0.5 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang-jwt/jwt/v5 v5.2.2 // indirect
github.com/hanzo-ds/native v0.71.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/montanaflynn/stats v0.6.6 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/paulmach/orb v0.12.0 // indirect
github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
github.com/segmentio/asm v1.2.1 // indirect
github.com/spiffe/go-spiffe/v2 v2.5.0 // indirect
github.com/zeebo/errs v1.4.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.36.0 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.61.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0 // indirect
go.opentelemetry.io/otel v1.44.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.29.0 // indirect
go.opentelemetry.io/otel/metric v1.44.0 // indirect
go.opentelemetry.io/otel/sdk v1.40.0 // indirect
go.opentelemetry.io/otel/sdk/metric v1.40.0 // indirect
go.opentelemetry.io/otel/trace v1.44.0 // indirect
go.opentelemetry.io/proto/otlp v1.3.1 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/telemetry v0.0.0-20251203150158-8fff8a5912fc // indirect
golang.org/x/tools v0.40.0 // indirect
modernc.org/gc/v3 v3.0.0-20240107210532-573471604cb6 // indirect
)
require (
cloud.google.com/go v0.121.6 // indirect
cloud.google.com/go/compute/metadata v0.8.0 // indirect
cloud.google.com/go/iam v1.5.2 // indirect
cloud.google.com/go/longrunning v0.6.7 // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/99designs/keyring v1.2.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.4.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/internal v1.1.2 // indirect
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v1.0.0 // indirect
github.com/Azure/go-ansiterm v0.0.0-20230124172434-306776ec8161 // indirect
github.com/Azure/go-autorest v14.2.0+incompatible // indirect
github.com/Azure/go-autorest/autorest/date v0.3.0 // indirect
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/Microsoft/go-winio v0.6.2 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/apache/thrift v0.22.0 // indirect
github.com/aws/aws-sdk-go-v2 v1.43.0 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 // indirect
github.com/aws/aws-sdk-go-v2/credentials v1.19.30 // indirect
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.22.35 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.31 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.31 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.32 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.24 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.31 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.32 // indirect
github.com/aws/aws-sdk-go-v2/service/s3 v1.106.0 // indirect
github.com/aws/smithy-go v1.27.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20250501225837-2ac532fd4443 // indirect
github.com/cznic/mathutil v0.0.0-20180504122225-ca4c9f2c1369 // indirect
github.com/danieljoos/wincred v1.2.2 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/dvsekhvalnov/jose2go v1.7.0 // indirect
github.com/edsrzf/mmap-go v0.0.0-20170320065105-0bce6a688712 // indirect
github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
github.com/gabriel-vasile/mimetype v1.4.7 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/godbus/dbus v0.0.0-20190726142602-4481cbc300e2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/golang-sql/civil v0.0.0-20190719163853-cb61b32ac6fe // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/snappy v1.0.0 // indirect
github.com/google/flatbuffers v25.2.10+incompatible // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/s2a-go v0.1.9 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.6 // indirect
github.com/googleapis/gax-go/v2 v2.15.0 // indirect
github.com/gorilla/handlers v1.4.2 // indirect
github.com/gorilla/mux v1.7.4 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed // indirect
github.com/jackc/chunkreader/v2 v2.0.1 // indirect
github.com/jackc/pgio v1.0.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgproto3/v2 v2.3.3 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgtype v1.14.0 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/k0kubun/pp v2.3.0+incompatible // indirect
github.com/kardianos/osext v0.0.0-20190222173326-2bc1f35cddc0 // indirect
github.com/klauspost/asmfmt v1.3.2 // indirect
github.com/klauspost/compress v1.18.6 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/mattn/go-colorable v0.1.6 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/minio/asm2plan9s v0.0.0-20200509001527-cdd76441f9d8 // indirect
github.com/minio/c2goasm v0.0.0-20190812172519-36a3d3bbc4f3 // indirect
github.com/mitchellh/mapstructure v1.1.2 // indirect
github.com/moby/term v0.5.0 // indirect
github.com/morikuni/aec v1.0.0 // indirect
github.com/mtibben/percent v0.2.1 // indirect
github.com/onsi/ginkgo v1.16.4 // indirect
github.com/onsi/gomega v1.15.0 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.0 // indirect
github.com/pierrec/lz4/v4 v4.1.27 // indirect
github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rqlite/gorqlite v0.0.0-20230708021416-2acd02b70b79
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.3 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/xdg-go/scram v1.1.1 // indirect
github.com/xdg-go/stringprep v1.0.3 // indirect
github.com/youmark/pkcs8 v0.0.0-20181117223130-1be2e3e5546d // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
github.com/snowflakedb/gosnowflake v1.6.3
github.com/stretchr/testify v1.7.0
github.com/xanzy/go-gitlab v0.15.0
gitlab.com/nyarla/go-crypt v0.0.0-20160106005555-d9a5dc2b789b // indirect
go.opencensus.io v0.24.0 // indirect
golang.org/x/crypto v0.47.0 // indirect
golang.org/x/exp v0.0.0-20250408133849-7e4ce0ab07d0 // indirect
golang.org/x/mod v0.31.0 // indirect
golang.org/x/net v0.48.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/sys v0.45.0 // indirect
golang.org/x/term v0.39.0 // indirect
golang.org/x/text v0.33.0 // indirect
golang.org/x/time v0.12.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20250603155806-513f23925822 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250818200422-3122310a409c // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250818200422-3122310a409c // indirect
google.golang.org/grpc v1.74.2 // indirect
google.golang.org/protobuf v1.36.7 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
go.mongodb.org/mongo-driver v1.7.0
go.uber.org/atomic v1.6.0
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519 // indirect
golang.org/x/net v0.0.0-20211013171255-e13a2654a71e // indirect
golang.org/x/sys v0.0.0-20211013075003-97ac67df715c // indirect
golang.org/x/tools v0.1.5
google.golang.org/api v0.51.0
google.golang.org/genproto v0.0.0-20211013025323-ce878158c4d4
google.golang.org/grpc v1.41.0 // indirect
modernc.org/b v1.0.0 // indirect
modernc.org/db v1.0.0 // indirect
modernc.org/file v1.0.0 // indirect
modernc.org/fileutil v1.3.0 // indirect
modernc.org/fileutil v1.0.0 // indirect
modernc.org/golex v1.0.0 // indirect
modernc.org/internal v1.0.0 // indirect
modernc.org/libc v1.41.0 // indirect
modernc.org/lldb v1.0.0 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.7.2 // indirect
modernc.org/ql v1.0.0
modernc.org/sortutil v1.1.0 // indirect
modernc.org/strutil v1.2.0 // indirect
modernc.org/token v1.1.0 // indirect
modernc.org/sqlite v1.10.6
modernc.org/zappy v1.0.0 // indirect
)
go 1.16
+793 -1394
View File
File diff suppressed because it is too large Load Diff
+1 -1
View File
@@ -1,4 +1,4 @@
//go:build aws_s3
// +build aws_s3
package cli

Some files were not shown because too many files have changed in this diff Show More