Compare commits

..
608 Commits
Author SHA1 Message Date
Erik DubbelboerandDale Hui e5b4be7771 Let database.Open() use schemeFromURL as well (#271)
* Let database.Open() use schemeFromURL as well

Otherwise it will fail on MySQL DSNs.

Moved schemeFromURL into the database package. Also removed databaseSchemeFromURL
and sourceSchemeFromURL as they were just calling schemeFromURL.

Fixes https://github.com/golang-migrate/migrate/pull/265#issuecomment-522301237

* Moved url functions into internal/url

Also merged the test cases.

* Add some database tests to improve coverage

* Fix suggestions
2019-08-20 09:59:15 -07:00
Sebastiaan van StijnandDale Hui d5960ade4a Docker: restore "from env" behavior, and perform API version negotiation (#270)
* NewDockerContainer: restore "from env" behavior

commit c31948caeb replaced the deprecated
`dockerclient.NewEnvClient` with `dockerclient.NewClientWithOpts`, but
did not add the `FromEnv` option to keep  the old behavior.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

* NewDockerContainer: use API version negotiation

This helps in situations where the daemon is older than
the API client. API version negotiation checks the maximum
supported API version by the daemon, and downgrades to that
API version if needed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2019-08-20 08:42:07 -07:00
Dale Hui d068374972 Test against MS SQL Server 2019-CTP3.1 (Ubuntu) 2019-08-18 23:37:33 -07:00
Dale Hui 947936279a Update firebird README to reflect lack of official support 2019-08-18 23:34:07 -07:00
Dale Hui 42492c82ca Update golangci-lint from v1.16.0 to v1.17.1 2019-08-17 10:50:42 -07:00
Dale HuiandGitHub 97cfb4d2a6 Merge pull request #267 from zikaeroh/fix-mongodb-dep
Update MongoDB driver to 1.1.0, use correct import path
2019-08-17 10:20:45 -07:00
zikaeroh d80e0e2f7f Merge branch 'master' into fix-mongodb-dep 2019-08-17 08:13:16 -07:00
Erik DubbelboerandDale Hui eb7d0dd6a1 Fix in the URL parser with go 1.12.8 and github.com/go-sql-driver/mysql (#265)
* Fix in the URL parser with go 1.12.8 and github.com/go-sql-driver/mysql

Change schemeFromURL to just split the url by :// to find the scheme.
It's not required to parse the whole URL. MySQL DSNs aren't valid URLs.

Fixes #264

* The mysql driver itself also used net/url.Parse

* Also fix TestPasswordUnencodedReservedURLChars

* Keep backwards compatibility with url encoded username and passwords

* Fix suggestions

* Reuse old function names
2019-08-16 23:09:12 -07:00
zikaeroh 9e406c40b5 Update MongoDB driver to 1.1.0, use correct import path 2019-08-14 17:40:52 -07:00
Dale HuiandGitHub b071731cc2 Merge pull request #258 from mknycha/add-tutorial-for-getting-started
Add getting started docs and Postgres tutorial
2019-08-13 08:31:01 -07:00
Marcin K d3ec78d996 Use different word 2019-08-13 16:40:03 +02:00
Marcin K 98c383cc42 Update info regarding migrations inconsistency 2019-08-13 16:38:16 +02:00
Marcin K 12f7e22e36 Put tutorial specific info into a separate file 2019-08-11 22:06:04 +02:00
Dale HuiandGitHub 6c7d3123c2 Merge pull request #260 from groyoh/exit_status
Change exit status to 2 on command error
2019-08-10 02:16:59 -07:00
Marcin K 86e4e4965d FAQ: Add question regarding non-Go project usage 2019-08-09 18:50:24 +02:00
Marcin K d0c2e446b7 Remove hint regarding Makefile
As it is too much environment-specific
2019-08-09 18:46:57 +02:00
Marcin K bb1e9dd3ae Describe database transactions 2019-08-09 18:46:04 +02:00
Yohan Robert dc5b8a46c6 Add documentation for exit 2 2019-08-09 15:15:29 +02:00
Marcin K d90f7d51aa Cosmetics 2019-08-08 21:03:54 +02:00
Marcin K d6fa9be6e6 Add a caveat regarding DB lock when running migrations 2019-08-08 21:03:00 +02:00
Marcin K fb4fb978e1 Use idempotent queries 2019-08-08 20:34:21 +02:00
Marcin K 604f1254ba Use sequential migrations naming in example
And add migration conflicts description
2019-08-08 20:11:05 +02:00
Marcin K b49854bde5 TUTORIAL: Use password explicitly 2019-08-07 22:36:20 +02:00
Marcin K fb771c8dd8 Rephrase, list created files 2019-08-07 22:35:22 +02:00
Marcin K 44b07c5f7b Use reverse/down migration instead of rollback 2019-08-07 22:01:04 +02:00
Marcin K e98f9f6a24 FAQ: Remove driver specific info 2019-08-07 21:52:15 +02:00
Yohan Robert 95623b991f Change exit status to 2 on command error
The default exit status when a command cannot be found was 0.
This can cause migrate to fail "silently" in case of typos.
Given that providing an invalid flag causes the program to exit
with a exit status set to 2, the default exit status was also set to 2.
2019-08-07 20:13:16 +02:00
Marcin K 1d402e46a8 Tutorial: Remove info about migration version table 2019-08-07 01:09:26 +02:00
Marcin K d7e0e40e0a FAQ: Add one more question 2019-08-07 01:07:29 +02:00
Marcin K 74b9fdb3cb Add reference to README 2019-08-07 01:01:15 +02:00
Marcin K 01107cb677 Add tutorial file 2019-08-07 01:00:58 +02:00
Dale Hui e93eaeb3fe Another attempt at fixing builds
Addresses: https://github.com/golang-migrate/migrate/issues/254
2019-07-30 00:06:35 -07:00
Dale Hui b55570703d Fix broken builds
Addresses: https://github.com/golang-migrate/migrate/issues/254
2019-07-29 23:59:39 -07:00
Dale HuiandGitHub fd16003059 Add Go Report Card badge 2019-07-24 22:58:07 -07:00
Dale Hui 71c6afaf98 Reduce docker binary size by stripping symbols 2019-07-12 10:54:42 -07:00
Dale Hui bb99610517 Update Docker image to use Alpine 3.10 2019-07-11 23:22:30 -07:00
Dale HuiandGitHub ff1977cbd4 Update migration docs
Correct and clarify information around empty migrations
Addresses: https://github.com/golang-migrate/migrate/issues/244
2019-07-11 23:05:33 -07:00
Dale HuiandGitHub 14a3bcdaa8 Merge pull request #236 from KlotzAndrew/migrate-down-less-destructive
migrate down less destructive
2019-06-26 10:55:06 -07:00
Andrew Klotz ffea024e7c migrate down less destructive
default behaviour for down is to apply all down migrations, which is
comparable to dropping a database - and usually not the desired default
action

proposed changes:

 * `down` prompts for a confirm `y` before applying all down migrations, defaulting to doing nothing
 * `down --all` does the current behaviour, applying all down migrations
 * `down N` is unchanged

 * `down N --all` errors
 * `down --all N` errors
2019-06-26 00:00:54 -04:00
Dale Hui d4e8e9ac07 Deprecate migrate's MultiError in favor of github.com/hashicorp/go-multierror 2019-06-24 08:58:28 -07:00
Dale HuiandGitHub 5399892051 Merge pull request #240 from k-yomo/fix/typo
Fix typo in README for CLI
2019-06-23 22:08:45 -07:00
k-yomo 8d9d92b7b1 Fix typo in README for CLI 2019-06-23 18:46:05 +09:00
Dale Hui cde3cfc30d Fix broken sqlite tests
Since v4.3.0, Migrate.Drop() no longer recreates the migrate schema version table.
See: https://github.com/golang-migrate/migrate/releases/tag/v4.3.0
2019-06-21 00:15:01 -07:00
Dale Hui 481bf0fb03 Run tests so that coverage profiles don't need to be manually concatenated
Addresses: https://github.com/golang-migrate/migrate/issues/2
2019-06-20 23:07:47 -07:00
Dale Hui 9f6c7e5fc9 Properly clean/normalize user specified directory for create command
Addresses: https://github.com/golang-migrate/migrate/issues/238
2019-06-20 21:35:51 -07:00
Kasparas GaldikasandDale Hui 7c76166697 Github Enterprise support (#234)
* exported Github struct fields and ReadDirectory method

* github ee implementation, tests and docs

* build fixes

* Github Enterprise API endpoint based on docs

* addressing PR comments

* code review

* make linter happy

* parseBool() takes fallback

* pr comments

* tweaks to Config{}
2019-06-16 12:48:01 -07:00
Dale Hui 0d13e794e4 Clean up CLI docs
- Move main CLI docs from cli to cmd/migrate
    - Clean up markdown
    - Update example CLI usage in Docker
2019-06-13 01:30:58 -07:00
Dale HuiandGitHub 8437fe6dc6 Merge pull request #227 from zhevron/mssql-params
Fix incorrect syntax errors for MSSQL driver
2019-05-27 10:25:36 -07:00
Thomas Lokshall dd0ead011e update Dockerfile referencing renamed mssql driver 2019-05-27 08:20:47 +02:00
Thomas Lokshall ffbd6893f6 sqlserver: update comment referencing mssql 2019-05-27 08:20:35 +02:00
Thomas Lokshall bad962cf21 Merge branch 'mssql-params' of https://github.com/zhevron/migrate into mssql-params 2019-05-24 15:16:24 +02:00
Thomas Lokshall 293bfec844 rename mssql driver to sqlserver 2019-05-24 15:16:12 +02:00
Thomas LokshallandGitHub bc0b686fc0 Merge branch 'master' into mssql-params 2019-05-23 11:12:50 +02:00
Thomas Lokshall e08ae0e996 mssql: remove mssql registration, document only support for sqlserver driver 2019-05-23 09:43:13 +02:00
Alex PliutauandDale Hui e877644ed1 Fix MongoDB docs: x-migrations-collection (#225) 2019-05-22 18:29:44 -07:00
Dale HuiandGitHub 5ed8a441d4 Merge pull request #229 from aryzing/patch-1
Typo
2019-05-22 18:25:41 -07:00
Eduard Bardají PuigandGitHub 6ff5b4ed4c Typo 2019-05-22 20:25:18 +01:00
Thomas Lokshall 3d4dad5655 mssql: use sqlserver driver instead of deprecated mssql driver 2019-05-22 14:10:37 +02:00
Thomas Lokshall ec3ae16f2e mssql: fix sp_MSforeachtable parameter 2019-05-22 14:09:50 +02:00
Thomas Lokshall 9f282f4ee3 use correct mssql parameter syntax 2019-05-22 10:34:53 +02:00
Ferdy PruisandDale Hui e85c5f51b9 Disable foreign_key_checks in Drop command for MySQL (#224)
* Disable FOREIGN_KEY_CHECKS in MySQL when dropping all tables.

* Lowercased system variable

* Discard error enabling foreign_key_checks, dropping is already successful at this point

* Explicitly discard error
2019-05-21 10:56:37 -07:00
Dale HuiandGitHub 2327ddb52d Fix reference to MS SQL Server support in README 2019-05-21 10:54:54 -07:00
Dale HuiandGitHub 7b3cd164d7 Merge pull request #222 from nathan-c/master
Add MS SQL Server Support
2019-05-21 10:52:41 -07:00
Nathan Collard aeb7f633f3 mssql: fix code review comments 2019-05-21 08:44:17 +01:00
Nathan Collard 7817573893 Merge remote-tracking branch 'root/master' 2019-05-21 08:21:33 +01:00
Dale Hui 923901c81f Cleanup regex usage in spanner DB driver
- Only compile a regex once for re-use
    - Don't use regex unnecessarily (removing trailing semicolon)
    - Add test for parsing multiple statements in a migration
2019-05-20 08:28:07 -07:00
Dale Hui bd81e32d1a Mark testing package as deprecated 2019-05-19 16:52:41 -07:00
nathan-c e211c5b5f5 mssql: increase pull timeout 2019-05-19 21:36:50 +01:00
nathan-c 98e5f88b9f mssql: fix error parsing and add tests 2019-05-19 16:08:15 +01:00
nathan-c 5ac583ba7b mssql: fix linter errors 2019-05-19 15:21:25 +01:00
nathan-c 3bd91a774e mssql: reorder readme 2019-05-19 15:15:07 +01:00
nathan-c 95290f2fbd mssql: change migration table name and add readme 2019-05-19 15:09:56 +01:00
nathan-c 14e6c50d2e fix mssql build flag 2019-05-19 14:47:09 +01:00
nathan-c c273636efe parse mssql errors 2019-05-19 14:08:10 +01:00
nathan-c 2fbec2f760 update mssql driver module 2019-05-19 13:43:47 +01:00
nathan-c ae500dbe45 finish implementing mssql db and add tests 2019-05-19 13:37:40 +01:00
nathan-c 50e972912d Merge remote-tracking branch 'evosure/mssql' 2019-05-18 14:39:55 +01:00
Dale HuiandGitHub e364488761 Merge pull request #220 from Flaque/patch-1
Fix Broken Markdown in cli/README.md
2019-05-16 12:15:54 -07:00
Evan ConradandGitHub c7e9414bfe Fix Broken Markdown in cli/README.md 2019-05-16 11:30:01 -07:00
Dale HuiandGitHub f2e5e31837 Remove offical Firebird support from README 2019-04-26 22:52:56 -07:00
Dale Hui 10e2545ceb Update dependencies to resolve: https://github.com/golang-migrate/migrate/issues/202 2019-04-26 19:09:13 -07:00
Dale Hui 89970b8951 Bring built-in and supported sources and databases into parity 2019-04-26 19:04:37 -07:00
Dale Hui 66510b82c4 Remove official support for Firebird
- In the future, support for Firebird should also include v2.5
2019-04-26 19:04:05 -07:00
Kirill MuratovandDale Hui 6c96ef02df Enabled maligned, staticcheck and errcheck linters (#206)
* Enabled maligned check

* Enabled staticcheck

* Fixes for golint

* Enabled errcheck linter

* Added fixes for error check

* Added errcheck for tests

* Fixed test

* Increased golangci-lint deadline for travis

* Increased golangci-lint deadline for travis

* Decreased golangci-lint deadline for travis

* Revert for backward compatibility

* Using log.Println() instead of fmt.Println()

* Handling os.RemoveAll() errors

* Using t.Error(error) instead of t.Errorf("%v", err)

* Using t.Fatal(error) instead of t.Fatalf("%v", err)

* Using fmt.Sprint(sum) instead of t.Srintf("%v", sum)

* Refactoring

* Revert for backward compatibility

* Revert

* go mod tidy

* Added error logging

* Added error logging

* Added error handling

* Added error handling

* Added error logging

* Fix error logging

* Added error handling

* Fix

* Added logging for migr.Buffer()

* Fixes

* Firebird test disabled

* Fixed nolint comment

* Updated firebird docker image version

* Disabled test for firebird 2.5

* Fixed // nolint
2019-04-26 15:47:16 -07:00
Dale Hui ddc72468fc Update packages that depend on the wrong golint import path
Addresses: https://github.com/golang-migrate/migrate/issues/202
2019-04-09 00:31:17 -07:00
Dale Hui 16197b67e3 Update to golangci-lint v1.16.0
- Fix newly raised issues
2019-04-07 00:51:27 -07:00
Dale Hui 6e2056f5b4 Include firebird db driver in default builds 2019-04-05 23:25:57 -07:00
Dale Hui 8d63b3f95b Remove spurious list item 2019-03-28 21:35:54 -07:00
Dale Hui 43e4503343 Update golangci-lint config and implore contributors to use it
- Run golangci-lint on tests and fix found issues
2019-03-28 18:55:12 -07:00
Dale HuiandGitHub c1404f0f98 Merge pull request #194 from kmuratov/master
Enabled golangci-lint
2019-03-28 11:57:10 -07:00
Kirill Muratov 46ee425bac Updated golangci.yml 2019-03-28 19:30:02 +03:00
Kirill Muratov 81f457f379 Added newline 2019-03-26 22:05:47 +03:00
Kirill Muratov 07c594d988 Added scopelint check 2019-03-26 22:01:12 +03:00
Kirill Muratov be4c129168 Added interfacer, unconvert, goconst, nakedret, prealloc 2019-03-23 00:40:35 +03:00
Kirill Muratov ef4a805b02 Added misspell linter 2019-03-22 23:57:00 +03:00
Kirill Muratov 5059f0bee6 Fixes for govet 2019-03-22 23:05:19 +03:00
Kirill Muratov c31948caeb Fixed dockerclient.NewEnvClient is depricated 2019-03-22 22:47:24 +03:00
Kirill Muratov 2758eb210f Fixes for unused linter 2019-03-22 22:08:29 +03:00
Kirill Muratov 465060c9a7 Fixes for gosimple linter 2019-03-22 22:02:08 +03:00
Kirill Muratov 8f6bfb0295 Added golangci-lint config 2019-03-22 21:11:31 +03:00
Dale HuiandGitHub cac03318eb List Gitlab as a supported source 2019-03-21 01:54:59 -07:00
Dale HuiandGitHub 7bf5fd7859 Add missing backtick 2019-03-21 01:54:41 -07:00
Kirill MuratovandDale Hui 2c9773ed89 Added Firebird support (#191)
* Added Firebird support

* Fixed typo

* Refactoring

* Schema migrations table name don't have to be upper case

* Fixed readme

* Added Firebird 2.5 support

* Removed SchemaName

* Refactoring
2019-03-21 01:51:49 -07:00
Cyrille HemidyandDale Hui 46fb82ce3d Fmt (#189)
* fix gofmt -s

* fix gofmt -s + fix mispelling
2019-03-13 12:15:21 -07:00
Bence PatyiandDale Hui 144aaaa986 Gitlab source (#184)
* New Gitlab source

* fix: use net package to build url string

* add: gitlab source to Makefile and Dockerfile
2019-03-10 18:38:23 -07:00
Dale HuiandGitHub 14f26d8818 Update supported Go versions badge
Drop 1.10 and add 1.12
2019-03-07 11:42:22 -08:00
Dale Hui f8f7beea54 Deploy using Go 1.12 2019-03-02 02:04:53 -08:00
Dale HuiandGitHub c6c22b34c8 Fix broken "versions" table formatting in README 2019-03-01 18:52:20 -08:00
Dale Hui 752cdc3b11 Update "versions" table in the README
- Reflect supported versions
    - With Go 1.12 released, module support is no longer an issue
2019-03-01 18:49:12 -08:00
Dale Hui f5c4ccff1f Update dependencies 2019-03-01 17:45:37 -08:00
Dale Hui 5e004dbef4 Fix issues with new version of MongoDB driver:
- "connect=single" option is no longer supported
        - "connect=direct" is the equivalent
        - https://jira.mongodb.org/browse/GODRIVER-665
    - Count() needs to be passed a non-nil filter/document
        - https://jira.mongodb.org/browse/GODRIVER-572
2019-03-01 17:44:40 -08:00
Dale Hui 4fba554b47 Drop support for go1.10
- Remove dep package pinnings
2019-03-01 14:37:54 -08:00
Dale Hui 6593dc8dbf Support Go 1.12 2019-02-27 23:44:23 -08:00
Lukas JoergensenandDale Hui 480a5a634a postgres: Move lock out of ensureVersionTable, for consistency with other SQL operations (#173)
* Consistently lock in ensureVersionTable and do not call ensureVersionTable from Drop across all database implementations

* Add test for dropping postgres databases

* Fix failing database tests

* Fix CockroachDb test, lock table should be created before versionTable

* Add Initialize() to Driver interface, and add integration tests for Drop() between database implementations and migrate

* Remove Initialize, document breaking behaviour of Drop

* Revert introduction of Initialize method

* Removed Initialize in Stub as well

* Remove call to non-existent Initialize and make sure to close re-initialized database connections

* Revert changes to TestDrop in database/testing

* Split Test and TestMigrate into different test entrypoints

* Remove unused import in migrate_testing

* Remove erroneous code to fix tests

* Add stub source imports to database tests

* Add Stub source to migrate tests

* Use example migrations for tests

* Add file driver to database tests

* Align database directory layout

* Add file source driver to Cassandra

* Review changes

* Minor syntactic change for cleaner diff
2019-02-26 15:56:57 -08:00
Dale Hui f213007d4d Use alpine 3.9 for building Docker images 2019-02-19 03:25:47 -08:00
Dale Hui 0f8263de2e Remove unnecessary panics from core implementation 2019-02-19 02:07:07 -08:00
Dale HuiandGitHub 9b449be538 Merge pull request #176 from jszwedko/quote-postgres-identifiers
Quote postgres table identifiers when `DROP`ing
2019-02-18 03:14:11 -08:00
Jesse Szwedko 3db0395e0a Replace " quoting with pq.QuoteIdentifier for postgres driver 2019-02-18 11:44:38 +01:00
Jesse Szwedko fe66cd7734 Quote postgres table identifiers when DROPing
Handles table identifiers that are invalid without quotes (such as
uppercase letters).
2019-02-18 11:29:10 +01:00
Dale HuiandGitHub 4190b85018 Merge pull request #175 from plutov/master
Add mongodb driver to Dockerfile
2019-02-18 02:05:12 -08:00
Alex Pliutau 0a355bbee0 Add mongodb driver to Dockerfile 2019-02-18 10:15:23 +01:00
Brian Wigginton 407d1e3982 bump go.sum 2019-02-15 16:12:57 -06:00
Brian Wigginton 9b551a1c10 initial mssql support 2019-02-15 16:12:57 -06:00
Dale HuiandGitHub c0f025d2f6 Merge pull request #167 from arpando/fix/DocumentNilError
Fix document nil error
2019-02-05 03:48:34 -08:00
Arcadio Pando 56620dc3dd Fix document nil error 2019-02-04 17:37:22 +01:00
Dale HuiandGitHub 598addc6d7 Merge pull request #166 from bcho/fix/sqlite3-migration-table
sqlite3: fix wrong schema table usage
2019-02-02 11:19:58 -08:00
hbc 95e701da6a sqlite3: fix wrong schema table usage
fixes: https://github.com/golang-migrate/migrate/issues/165
2019-02-02 13:22:40 +08:00
Dale HuiandGitHub f6d624c729 Merge pull request #150 from tsenart/atomic-migrations-table-creation
postgres: Make `ensureVersionTable` atomic
2019-01-15 11:55:50 -08:00
Dale Hui 5ad19c5071 Run dep ensure
- Updates Gopkg.lock after github.com/hashicorp/go-multierror package was manually added to Gopkg.toml
2019-01-15 11:33:17 -08:00
Tomás Senart 41a595f62a fixup! Add github.com/hashicorp/go-multierror@v1.0.0 to Gopkg.yaml 2019-01-15 11:28:49 +01:00
Tomás Senart f58da1d4dc fixup! More explicit conditionals 2019-01-15 11:28:49 +01:00
Tomás Senart b57dbceec3 postgres: Revert back to use CREATE IF NOT EXISTS 2019-01-15 11:28:49 +01:00
Tomás Senart 06c80ae2da postgres: Use multierror.Error in ensureVersionTable 2019-01-15 11:28:49 +01:00
Tomás Senart 0beddcd2d0 postgres: Be explicit about max idle and open conns 2019-01-15 11:28:48 +01:00
Tomás Senart 94f5aa00fc postgres: Query table existance in ensureVersionTable
This commit reverts back to querying the existance of the table instead
of using CREATE IF NOT EXISTS because we want to support versions of
Postgres older than 9.1 which is when this feature was introduced.
2019-01-15 11:28:48 +01:00
Tomás Senart 5bf80d7ef2 postgres: Add comment about connection pool in TestWithInstance_Concurrent 2019-01-15 11:28:48 +01:00
Tomás Senart 09a1959309 postgres: Preserve Unlock error when outer err isn't nil 2019-01-15 11:28:48 +01:00
Tomás Senart 15713e64cb postgres: Use sync.WaitGroup in TestWithInstance_Concurrent 2019-01-15 11:28:48 +01:00
Tomás Senart 519dae2639 postgres: Make ensureVersionTable atomic
Fixes https://github.com/golang-migrate/migrate/issues/55
2019-01-15 11:28:48 +01:00
Tomás Senart ce7a2234ee fixup! Pass config to WithInstance 2019-01-15 11:28:48 +01:00
Tomás Senart a06a92d620 postgres: TestWithInstance_Concurrent 2019-01-15 11:28:47 +01:00
Dale Hui 1ca4da9a33 Run go mod tidy 2019-01-14 12:56:51 -08:00
Dale Hui 52fc08fdf5 Update supported DB versions 2019-01-12 16:13:26 -08:00
Dale Hui 2feaaaddda Update dktest from v0.2.0 to v0.3.0
- Update ReadyFuncs accordingly
2019-01-08 22:39:26 -08:00
Dale Hui 9870a84b99 Update Cassandra DB driver (gocql) 2019-01-08 11:25:23 -08:00
Dale Hui bbec63fda7 Use one of the 2 ports exposed by MySQL Docker images
- MySQL driver tests should pass more consistently now
2019-01-08 05:11:07 -08:00
Dale Hui 809c7f0ac8 Use dktest to run docker tests
- Leaving migrate/testing in case there are unknown consumers
    - Add migrate/dktesting package
    - Update tests to use migrate/dktesting instead of migrate/testing
2019-01-08 05:11:07 -08:00
Dale Hui fd47ba9a9a Remove TODO for MongoDB database driver 2019-01-08 05:01:48 -08:00
Dale HuiandGitHub b1a199a272 Merge pull request #146 from bobrovde/mongodb-driver
added mongodb driver
2019-01-08 02:34:59 -08:00
Dale HuiandGitHub fc8d57d128 Merge branch 'master' into mongodb-driver 2019-01-07 22:50:51 -08:00
Dale HuiandGitHub f24ea4d6c9 Update TravisCI badge after migration from .org to .com 2019-01-07 22:45:35 -08:00
DBobrov 12b93a3a07 Change checking of mongo replica set initialization 2019-01-07 15:17:35 +03:00
DBobrov 3fa0df7553 Refactor mongodb transactions test 2019-01-07 11:47:37 +03:00
DBobrov 13978a108a Add test for migrations in transaction mode 2019-01-06 12:45:22 +03:00
Dale Hui d549fd5265 Fix memory leak in tests
- Stop leaking tickers. As the docs state, using time.Tick() will leak tickers,
      so use time.NewTicker() with time.Ticker.Stop().
    - Be more efficient by using time.NewTimer() with time.Timer.Stop() instead of time.After().
2019-01-05 13:52:25 -08:00
DBobrov 30d2d94a22 Fix review comments 2019-01-05 11:10:25 +03:00
Dale Hui caa5719a86 Update pinned (via dep) MySQL driver from v1.4.0 to v1.4.0 2019-01-04 17:47:53 -08:00
DBobrov 436b81b357 Replaced bsonx.Doc with bson.D 2019-01-04 22:01:21 +03:00
Dale Hui 600f2eda93 Fix broken build
- `gem install` no longer recognizes the `--no-ri` and `--no-rdoc` options. Use `--no-document` instead.
    - References:
        - https://github.com/rubygems/rubygems/pull/2354
        - https://github.com/bundler/bundler/pull/6624
2019-01-03 16:35:22 -08:00
DBobrov 309edb6da6 - Add dependency for dep
- Add transaction mode for migrations
2019-01-03 18:00:23 +03:00
DBobrov bcd4f6e7dd added mongodb driver 2019-01-01 11:14:27 +03:00
Dale HuiandGitHub fd50054781 Merge pull request #145 from bobrovde/fix-clickhouse
Changed clickhouse migrations table struct
2018-12-30 13:42:39 -08:00
DBobrov 26064905d4 Changed migrations table struct of field version to fix support database.NilVersion and default version format. 2018-12-28 15:53:55 +03:00
Dale Hui 24176463f4 Update docker/docker checksum for Go 1.11.4
Addresses: https://github.com/golang-migrate/migrate/issues/138
2018-12-21 12:53:53 -08:00
Dale HuiandGitHub 418d41c2a1 Merge pull request #143 from coolaj86/url-opaque
use URL.Opaque when available
2018-12-20 21:37:54 -08:00
AJ ONeal 01291f7cb5 use URL.Opaque when available 2018-12-20 15:08:49 -07:00
Dale HuiandGitHub e8b8975ec3 Merge pull request #141 from fsouza/upgrade-mysql-driver
Upgrade go-sql-driver/mysql to v1.4.1
2018-12-19 11:31:31 -08:00
Francisco Souza 6a53d7d10d Upgrade go-sql-driver/mysql to v1.4.1
Fixes #139.
2018-12-18 18:01:52 -05:00
Dale HuiandGitHub 65870ff3e8 Merge pull request #126 from a-h/patch-1
Include details about migration locking in the FAQ
2018-12-14 17:36:07 -08:00
6f1479fa41 Update FAQ.md
Co-Authored-By: a-h <a-h@users.noreply.github.com>
2018-12-14 16:28:04 +00:00
Dale Hui f694b616aa Add supported major versions table to README
- Consolidates info about migrate major versions
    - https://github.com/golang-migrate/migrate/issues/134#issuecomment-446664565
2018-12-13 22:53:07 -08:00
Dale Hui 1bd41c2a0b README copy tweak
- helpers -> snippets
2018-12-13 22:41:26 -08:00
Dale HuiandGitHub f1d88ac6c1 Add guide for installing a specific version of migrate using the Go toolchain 2018-11-28 01:00:46 -08:00
Dale HuiandGitHub 0d5d918563 Fix CLI install with Go toolchain docs
`-d` option shouldn't be specified with `go get` when installing
2018-11-28 00:44:59 -08:00
Dale Hui 547bc41026 Merge branch 'deb' 2018-11-27 13:54:07 -08:00
Dale Hui d27dc63026 Move before_deploy steps to after_success to avoid re-building the binary before every deploy
- https://docs.travis-ci.com/user/job-lifecycle/#deploying-your-code
2018-11-27 13:48:57 -08:00
Dale Hui 83f5b9f644 Don't need to move and remove files before caching build data in Travis 2018-11-27 13:48:57 -08:00
Dale Hui addb3c1a79 .deb package changes
- Install binary to /usr/local/bin instead of /usr/bin
    - Support recent/stable Debian and Ubuntu distros that work with .deb packages
2018-11-27 13:48:57 -08:00
Dale HuiandGitHub 88e7eafcf9 Merge pull request #131 from vearutop/cli-layout
Refactor cli package to allow `go get` one-liner with a proper binary name
2018-11-27 12:15:59 -08:00
Viacheslav Poturaev 5f88d4bf22 Copy cli main package under migrate directory, keep deprecated original main package for backwards compatibility, fixes #70 2018-11-21 09:01:35 +01:00
Dale HuiandGitHub 7f00868584 Merge pull request #128 from andrei-m/fix-redshift
Fix Redshift migrations driver
2018-11-14 10:32:07 -08:00
Dale Hui 6ebc6c7dae Unlock in postgres db driver should also use the schema name 2018-11-14 02:18:57 -08:00
Dale Hui f38fe38d45 Use git describe --tags to set the version number when building the CLI 2018-11-12 02:23:08 -08:00
Dale Hui 323427e951 Update copyright in LICENSE 2018-11-08 01:45:19 -08:00
Dale Hui 06d3ebde1c Use strings.NewReader() instead of bytes.NewReader() when source is a string
- Group imports
2018-11-05 22:50:08 -08:00
Dale HuiandGitHub 1df8057f97 Merge pull request #127 from vporoshok/master
Add support for multi-schema migrations in Postgres
2018-11-05 22:34:54 -08:00
Evgeniy Bastrykov d2d449ad78 Review comments 2018-11-05 16:03:54 +04:00
Evgeniy Bastrykov 16d63e3a76 Add support for multi-schema migrations in Postgres
There is lock conflict on parallel migrations in different postgres
schemas. To avoid this conflicts function GenerateAdvisoryLockId added
variadic params to change lock id with schema name. Schema name taked
with postgres CURRENT_SCHEMA function. Null byte used as separator
between database and schema name, because any other symbol may be used
in both of it.

Closes #118
2018-11-05 12:56:38 +04:00
Andrei Mackenzie cc573a74dc Remove unused errors 2018-11-03 19:27:55 -04:00
Andrei Mackenzie 1d54cf5f18 Replace 'redshift' with 'redshift2'
This addresses https://github.com/golang-migrate/migrate/issues/90 . The
exported Redshift object no longer exports an embedde 'Driver' however,
so some more work is needed to make this backwards compatible.
2018-11-03 19:01:37 -04:00
Andrei Mackenzie 4e098f74cd Use 'DELETE FROM' over 'TRUNCATE' in Redshift
TRUNCATE commits the current transaction, which breaks the expection of
the 'Commit()' that follows.

See:
https://github.com/golang-migrate/migrate/issues/90
https://docs.aws.amazon.com/redshift/latest/dg/r_TRUNCATE.html
2018-11-03 18:52:09 -04:00
Andrei Mackenzie b0300df5e7 Use the most Redshift-like Postgres version available on Dockerhub
Redshift is based on Postgres version 8.0.2.
2018-11-03 18:49:31 -04:00
Andrei Mackenzie 9c132fe622 Update README with Redshift-specific info 2018-11-03 18:47:12 -04:00
Andrei Mackenzie 22334834ac Support the 'redshift' URL scheme
This brings the 'redshift2' package in alignment with the existing
'redshift' package.
2018-11-03 18:44:11 -04:00
Andrei Mackenzie 3373343f23 Remove advisory locking from the Redshift implementation
Redshift does not support advisory lock functions. The closest
capability is in-transaction table locks, which aren't quite right here
because the transaction scope is established within SetVersion, not
higher up above the Lock-before/Unlock-after SetVersion.

Local locking is left intact to satisfy expected "can't Lock twice
before Unlocking" behavior asserted in shared tests.
2018-11-03 18:31:41 -04:00
Andrei Mackenzie 1d0bedc454 Rename 'Postgres' to 'Redshift' in 'redshift2' 2018-11-03 18:09:53 -04:00
Andrei Mackenzie 8381ea0307 Clone the 'postgres' driver as 'redshift2'
Avoid stepping on the 'redshift' driver for the time being. Driver
registration is also modified to identify the clone as 'redshift2'
rather than 'postgres'.
2018-11-03 18:04:22 -04:00
Adrian HeskethandGitHub 2e0b820637 Include details about migration locking in the FAQ 2018-11-02 15:41:41 +00:00
Dale HuiandGitHub 9f5e1bd505 Merge pull request #124 from lopezator/master
cli: Update README.md with MacOS install info
2018-10-29 11:02:56 -07:00
Dale Hui 7db5c9a274 Remove references to dep from docs 2018-10-29 10:56:55 -07:00
David López 4fad212a25 cli: Update README.md with MacOS install info
Document in MacOS install section of README.md that brew packag is now available. References mattes/migrate#156 https://github.com/Homebrew/homebrew-core/pull/30032
2018-10-29 17:27:48 +01:00
Dale HuiandGitHub f6b3bf548d Merge pull request #121 from wmetaw/master
Add NewDB to spanner.go
2018-10-28 16:58:22 -07:00
Ryo Takashima ca888b96e3 Add NewDB to spanner.go 2018-10-28 22:00:05 +09:00
Dale HuiandGitHub 2fc1610248 Merge pull request #115 from weszeloos/add_support_for_tls_in_cassandra
Add support for TLS on Cassandra.
2018-10-16 15:09:30 -07:00
Wessel Oosthuizen bc28cee1dc Add support for TLS on Cassandra. 2018-10-16 11:18:38 +02:00
Dale Hui ea1928c904 Update dependencies 2018-10-15 13:26:18 -07:00
Dale HuiandGitHub 8c3f34e0c9 Merge pull request #114 from albenik/master
github.com/docker/docker package updated to api v1.37
2018-10-15 13:09:43 -07:00
Veniamin Albaev 956441176f github.com/docker/docker package updated to api v1.37
To avoid github.com/sirupsen/logrus package name case collision with go mod
2018-10-15 17:39:43 +03:00
Dale Hui 83d01b7e1c Run go mod tidy 2018-10-10 16:16:23 -07:00
Dale Hui fc2b2e9c64 Use v4 for Go module support 2018-10-10 16:16:23 -07:00
Dale Hui 60b45ba428 Revert "Docker builds need to use dep to manage dependencies"
This reverts commit 4937cd088f.
2018-10-10 16:16:23 -07:00
Dale Hui 107db3f407 Revert "Remove support for Go modules"
This reverts commit 5e96539f55.
2018-10-10 16:15:33 -07:00
Dale Hui 4937cd088f Docker builds need to use dep to manage dependencies 2018-10-10 16:11:44 -07:00
Dale Hui 5e96539f55 Remove support for Go modules 2018-10-10 13:34:03 -07:00
Dale HuiandGitHub 708d67fa02 Merge pull request #112 from xiamengyu/master
Fix typo in comments
2018-10-10 13:26:36 -07:00
xiamengyu1 a95b13d804 Fix typo 2018-10-10 16:26:27 +08:00
Dale HuiandGitHub 1e60261d3f Merge pull request #110 from MikeFitzgerald/master
Fix postgres drop
2018-10-04 00:30:33 -07:00
Mike Fitzgerald 0d4f1c0212 Fix postgres drop 2018-10-04 01:22:39 -04:00
Dale HuiandGitHub 4d30c4c87f Add Go version to but report template 2018-09-19 03:10:08 -07:00
Dale Hui 6c02d65563 Update commented out imports to use v3 2018-09-14 14:12:07 -07:00
Dale Hui 16dea48c82 Fix module support by opting in to modules 2018-09-14 01:14:42 -07:00
Dale Hui f517029483 Test latest version of Go as well, but allow failures 2018-09-13 16:14:43 -07:00
Dale Hui 6f233b3b58 Update dependencies 2018-09-13 16:14:43 -07:00
Dale HuiandGitHub f8c85735f4 Merge pull request #98 from Kay-Zee/Kay-Zee/close-sql-db-connections-on-driver-close
Add sql.DB to Postgres and Mysql structs to allow closing in Close()
2018-09-07 01:26:43 -07:00
Kay-Zee 1e4c50b70d Add sql.DB to Postgres and Mysql structs to allow closing in Close() 2018-09-06 17:16:02 -07:00
Dale Hui 93d53a5ae8 Use Go 1.11 (with modules enabled on alpine linux 3.8) for docker builds 2018-09-04 22:18:49 -07:00
Dale Hui 16f2b1736e Support Go 1.11 and drop support for Go 1.9
- Support modules in Go 1.11
2018-09-04 19:11:19 -07:00
Dale Hui 4d60607b0b Update issue templates 2018-08-29 12:14:54 -07:00
Dale Hui beb09d85fa Upgrade dep from v0.4.1 to v0.5.0 2018-08-15 12:08:25 -07:00
Dale HuiandGitHub 8ed902364c Merge pull request #91 from stabacov/master
Fixes MySQL custom TLS connection failure.
2018-08-15 00:03:14 -07:00
Tatsuhito KATO 02c83fb38b more descriptive. 2018-08-15 02:42:18 +00:00
Tatsuhito KATO 9049e49f9e x-tls-cert and x-tls-key must set both. 2018-08-14 07:43:14 +00:00
Tatsuhito KATO 9d36dd9841 Add detail of tls query. 2018-08-14 07:42:10 +00:00
Tatsuhito KATO 9905791932 Fixes MySQL custom TLS connection failure. 2018-08-13 11:43:19 +00:00
Dale Hui dd00ca926e Add support for postgres 10 and drop support for postgres 9.2
https://www.postgresql.org/support/versioning/
2018-08-12 15:54:21 -07:00
Dale HuiandGitHub 4092e160a4 Merge pull request #85 from HaraldNordgren/improve_tests
Update tests for 'Migrate', 'Steps' and 'UpAndDown' to assert exact migration sequence
2018-08-08 15:57:04 -07:00
Dale HuiandGitHub 6092802b3a Merge pull request #88 from HaraldNordgren/error_output
Improve error output for missing or malformed '-database' and '-source' parameters
2018-08-08 15:24:39 -07:00
Harald Nordgren 42f074062b Update tests for 'Migrate', 'Steps' and 'UpAndDown' to assert exact migration sequence 2018-08-08 22:24:37 +02:00
Harald Nordgren 112191042b Improve error output for missing or malformed '-database' and '-source' parameters 2018-08-08 21:27:12 +02:00
Dale Hui 1566dd06cd Pass the error through instead of overriding it 2018-08-07 16:22:05 -07:00
Dale HuiandGitHub 251bf76be4 Merge pull request #89 from HaraldNordgren/improve_documentation
Document that '-source' is a required parameter
2018-08-07 15:59:20 -07:00
Harald Nordgren a6eddccfe5 Document that '-source' is a required parameter 2018-08-07 09:53:36 +02:00
Dale Hui 3bd43bedad Reference mattes/migrate now that golang-migrate/migrate has been detached
https://github.com/golang-migrate/migrate/issues/81
2018-08-07 00:17:39 -07:00
Dale HuiandGitHub 4750e57ce1 Merge pull request #82 from UnitedTraders/clickhouse-multiline
Add support for the multiline clickhouse queries
2018-08-06 04:09:10 -06:00
Anton Markelov df7ce674e9 add support for the multiline clickhouse queries 2018-08-03 16:08:47 +10:00
Dale Hui e968e8a5f0 Merge branch 'update_pkgs' 2018-07-24 18:30:53 -07:00
Dale HuiandGitHub c8705dd876 Merge pull request #78 from dhui/mysql_dsn
Allow DB connection URLs to contain encoded reserved URL characters
2018-07-24 18:22:04 -07:00
Dale Hui c7a29e331f Another round of package updates couldn't hurt... 2018-07-24 18:19:22 -07:00
Dale Hui ffcbbd5fea Override dependency on docker/distribution to fix build
https://github.com/golang/dep/issues/1890#issuecomment-404036309
2018-07-24 18:13:11 -07:00
Dale Hui ebf9c0bf91 Update package dependencies 2018-07-24 18:13:11 -07:00
Dale Hui df658e8fa8 Don't urlencode the username and password for MySQL
Addresses: https://github.com/golang-migrate/migrate/pull/69
2018-07-24 17:56:17 -07:00
Dale Hui b5edb8e50c Improve error messaging when URL parsing fails
- Improve docs around escaping/encoding reserved URL characters
    - Add tests for net/url Parse() and reserved characters to document understanding
    - Addresses: https://github.com/golang-migrate/migrate/issues/77
2018-07-24 17:55:58 -07:00
Dale Hui 97eb0d191b Fix broken GenerateAdvisoryLockId() test
- Add it to the test suite to ensure that it doesn't break again
2018-07-24 17:55:58 -07:00
Dale Hui 857d7a620d Revert "Merge pull request #69 from bcho/fix/mysql-url-password-encode"
This reverts commit 78d696c1e5, reversing
changes made to 18583d5a91.
2018-07-24 17:55:58 -07:00
Dale HuiandGitHub fad64ed381 Merge pull request #76 from agquick/master
Support for multiple statements in Cassandra cql migration files
2018-07-24 16:02:44 -07:00
Alan Graham 4244ebea6a Add detail to README for Cassandra flag, improve if block 2018-07-24 19:30:44 +00:00
Alan Graham 327822c29a Added Cassandra URL flag to enable multi statement migrations 2018-07-20 13:27:54 +00:00
Dale HuiandGitHub feea45c205 Merge pull request #75 from LUSHDigital/master
Lock go-sql-driver/mysql to v1.4.0
2018-07-17 16:42:01 -07:00
Alan Graham c4b81d1210 Support for multiple statements in Cassandra cql migration files based on semi-colon and newline. 2018-07-17 15:19:54 +00:00
Dan Richards 3988074cae Lock digest. 2018-07-16 12:09:29 +01:00
Dan Richards ad2b69e115 Typo 2018-07-16 12:08:24 +01:00
Dan Richards f9eb960848 Just lock go-sql-driver/mysql. 2018-07-16 12:06:27 +01:00
Dan Richards 5927d6e5f4 Locked go-sql-driver/mysql to v1.4.0 2018-07-16 12:01:52 +01:00
Dale Hui 6bb3e494e5 Disable CGO to build binaries statically on the building platform
- Previously, only cross-compiled binaries were built statically
    - Addresses: https://github.com/golang-migrate/migrate/issues/45
2018-07-14 13:31:59 -07:00
Dale Hui 5d5d15e7a3 Link to comparison of transactional DDLs 2018-07-09 19:25:29 -07:00
Dale Hui b742ec3de4 Use unix line endings 2018-07-09 19:21:40 -07:00
Dale Hui 75f0aade0d Reduce docker image size by removing unnecessary chown
https://github.com/moby/moby/issues/5505
2018-07-09 14:52:59 -07:00
Dale HuiandGitHub 3872c8f41d Merge pull request #71 from qdentity/postgres-parse-error
Add postgres lib/pq error parsing
2018-07-09 12:26:24 -07:00
Damir Vandic ae2fd4fa7b Support postgres non-ascii error parsing 2018-07-09 11:39:29 +02:00
Damir Vandic 3eb26a65d3 Add postgres lib/pq error parsing 2018-07-07 12:17:38 +02:00
Dale Hui 157433893c Update CLI installation docs to reduce confusion
https://github.com/golang-migrate/migrate/issues/64#issuecomment-402300146
2018-07-05 10:55:44 -07:00
Dale Hui 236dac0c96 Add latest tag to docker image during build 2018-07-03 14:09:26 -07:00
Dale HuiandGitHub 78d696c1e5 Merge pull request #69 from bcho/fix/mysql-url-password-encode
prevent net/url encoding the user password
2018-06-29 14:17:45 -07:00
Chao Dengandhbc 48a4062f75 prevent net/url encoding the user password 2018-06-29 21:07:10 +08:00
Dale Hui 18583d5a91 Update docs
- Addresses: https://github.com/golang-migrate/migrate/issues/67
    - Add badge for supported Go versions
    - Suggest dep for dependency management
2018-06-27 11:45:54 -07:00
Dale HuiandGitHub 642913dbf4 Merge pull request #66 from dadlerj/cli-readme
Update CLI usage in readme
2018-06-26 13:25:18 -07:00
dadlerj 2ca5473e52 Update CLI usage in readme 2018-06-26 09:16:04 -07:00
Dale Hui 6070aa8281 Push latest image to docker hub 2018-06-19 01:42:37 -07:00
Dale HuiandGitHub 78dc468555 Merge pull request #47 from dhui/docker_badge
Add badge for num docker pulls
2018-06-19 01:33:06 -07:00
Dale Hui bcd996f3df Use environment variables since there's no way to pass secure data to deploy script provider 2018-06-19 00:39:54 -07:00
Dale Hui a1d959d0e9 Use latest version of docker 2018-06-18 23:07:54 -07:00
Dale Hui 71e97bccc7 Fix TravisCI conditional release 2018-06-18 21:42:31 -07:00
Dale Hui 3239b18671 Fix time format flag to be backwards compatible
https://github.com/golang-migrate/migrate/pull/17#issuecomment-397960867
2018-06-18 00:06:17 -07:00
Dale Hui c3794da4ed Make -ext flag required for the create command 2018-06-17 23:18:23 -07:00
Dale Hui 872a561495 Fix broken CLI build 2018-06-17 23:16:38 -07:00
Dale Hui d1adc8b15f Add golang/x/tools (godoc/vfs) as a dependency 2018-06-17 19:59:39 -07:00
Dale HuiandGitHub e7d20b35ee Merge pull request #57 from SamWhited/vfs
source/vfs: add virtual file system source
2018-06-17 19:22:06 -07:00
Dale Hui 21f566a60a Add docker hub credentials for uploading docker image 2018-06-17 17:42:03 -07:00
Sam Whited 7562e9ad5d source/vfs: add virtual file system source
Fixes #56
2018-06-17 19:35:13 -05:00
Dale Hui 0f16f7f6a2 Rename sources to reflect updated build constraint names 2018-06-12 16:45:56 +08:00
Dale Hui dfa9fc05cb Use correct build contraints (must be alphanumeric)
Fixes: https://github.com/golang-migrate/migrate/issues/52
2018-06-12 16:18:27 +08:00
Dale Hui c4e855b58c List source and database drivers in help text 2018-06-12 16:17:52 +08:00
Dale Hui 0d9f636356 Use dep to build docker image 2018-06-11 19:18:05 +08:00
Dale HuiandGitHub 387f79be13 Merge pull request #48 from dhui/static_build
Build binaries statically
2018-06-04 09:24:35 -07:00
Dale Hui 87f3e4a910 Build binaries statically
Addresses: https://github.com/golang-migrate/migrate/issues/45
2018-06-04 19:27:37 +08:00
Dale Hui 01ea635328 Add badge for num docker pulls 2018-06-04 18:54:43 +08:00
Dale HuiandGitHub 3d1bb8a4b4 Merge pull request #13 from Teddy-Schmitz/postgres-wait
postgres driver should wait for lock
2018-05-31 19:04:30 -07:00
Teddy Schmitz 046d1cb767 postgres driver should wait for lock 2018-06-01 09:11:24 +08:00
Dale Hui 3d7b06e24c Override google cloud package version (was pinned to v0.19.0 by github.com/fsouza/fake-gcs-server) 2018-05-31 15:26:18 -07:00
Dale Hui 063b69db2f Update OpenCensus dependency 2018-05-31 15:05:31 -07:00
Dale Hui c0b0829d5a Use "vanity" import to allow forks to build
https://docs.travis-ci.com/user/languages/go/#Go-Import-Path
2018-05-31 14:57:38 -07:00
Dale Hui 3cf42dd36d Don't need to re-solve package dependencies every time. Only use Gopkg.lock
Reference: https://github.com/golang/dep/blob/master/docs/ensure-mechanics.md#-no-vendor-and--vendor-only
2018-05-31 14:57:38 -07:00
Dale Hui 89e558ee4f Don't cache go source since dependencies are managed by dep now 2018-05-31 14:57:38 -07:00
Dale Hui f277ef7428 Update github.com/go-sql-driver/mysql and github.com/gocql/gocql 2018-05-31 14:57:38 -07:00
Dale Hui b7647a749e Use the master branch for go-sql-driver/mysql since there hasn't been a release in a yr and a half 2018-05-31 14:57:38 -07:00
Dale Hui 4cc6aeb274 Update gocql package 2018-05-31 14:57:38 -07:00
Dale Hui 3de903f51f Use latest minor revision of Go 1.10 in TravisCI 2018-05-31 14:57:38 -07:00
Dale Hui 4b4f49b1eb Update docs regarding usage of dep 2018-05-31 14:57:38 -07:00
Dale Hui 4407720b95 Remove "deps" target from Makefile 2018-05-31 14:57:38 -07:00
Dale Hui 36dc939f91 Use dep for TravisCI builds 2018-05-31 14:57:38 -07:00
Dale Hui 40cd326ed7 Use dep to manage dependencies 2018-05-31 14:57:38 -07:00
Dale Hui a890d44d2f CockroachDB driver improvements
- Correctly return database.ErrLocked when the DB is locked
    - Group imports
    - gofmt goodness
2018-05-31 14:44:48 -07:00
Dale HuiandGitHub c98426505e Merge pull request #43 from wmetaw/master
Fixed unkeyed fields in file.go for GoogleAppEngine
2018-05-31 14:23:14 -07:00
Ryo Takashima 6fa8039b55 Fixed unkeyed fields for GoogleAppEngine 2018-05-31 15:40:15 +09:00
Dale HuiandGitHub faf4307fcc Merge pull request #32 from kenjones-cisco/feature/cassandra-withinstance
Minor tweaks to remove duplication
2018-05-17 01:01:47 -07:00
kenjones 1512e41e41 Minor tweaks to remove duplication
Adds missing connection close for cassandra tests

Revert to default timeout of 600ms
2018-05-16 20:09:12 -04:00
Dale Hui 55a25c5e0e Cassandra DB driver test improvements
- shorten connect timeout to allow more connection retry attempts
    - close session after it's created
    - group imports
2018-05-16 15:12:36 -07:00
Dale HuiandGitHub f815731412 Merge pull request #30 from Decemberlabs/master
Add WithSession helper for Cassandra driver
2018-05-16 14:26:12 -07:00
Andrés Rodríguez d607064fd2 Switch Cassandra's package lock to instance lock 2018-05-16 16:26:23 -03:00
Andrés Rodríguez 8b0f876416 Rename Cassandra's WithSession to WithInstance 2018-05-16 16:23:21 -03:00
Andrés Rodríguez bbd8f410d9 Correct method name in comment 2018-05-16 10:54:56 -03:00
Dale Hui fb3d809e81 Cleanup CLI docs 2018-05-12 15:25:19 -07:00
Dale HuiandGitHub 95db494b30 Merge pull request #28 from tayhalla/patch-1
Update README.md
2018-05-12 14:58:12 -07:00
Taylor HallidayandGitHub 5ebdb74ba8 Update README.md
There's no homebrew support. Even the referenced TODO says so, so not sure why it's in the Readme. Let's encourage others to support the development of it rather than think the support for it is broken.
2018-05-12 07:22:21 -07:00
Dale HuiandGitHub 07c20c5357 Merge pull request #27 from Its-Alex/master
Fix dockerfile version and deps
2018-05-10 01:13:05 -07:00
ItsAlex 31daa8dad3 fix dockerfile version and deps
Signed-off-by: Alexandre MARRE <me@itsalex.fr>
2018-05-10 09:12:23 +02:00
Dale HuiandGitHub a0b03c3a68 Merge pull request #24 from Its-Alex/master
Dockerization
2018-05-09 11:22:34 -07:00
ItsAlex fb7bb4f87a add dockerization Fix #23
Signed-off-by: Alexandre MARRE <me@itsalex.fr>
2018-05-09 09:48:20 +02:00
Dale HuiandGitHub 81e9fff605 Merge pull request #22 from thurt/add-ref-to-github
add ref (SHA, branch, or tag) to the github migration source
2018-05-04 10:13:18 +02:00
thurt e6baca2ffb add ref (SHA, branch, or tag) to the github migration source 2018-05-04 03:16:12 +00:00
Dale HuiandGitHub 2b28a0bc0a Merge pull request #17 from bobmeister/cli_create_yyyymmddhhmmss_migration_files
-datetime option to create files in yyyymmddhhmmss format
2018-04-17 10:50:48 -07:00
Robert Haupt 1f389f1422 Removed commented code 2018-04-17 11:17:10 -06:00
Robert Haupt 8cdee40695 Use Go time format string if version other than timestamp is desired. 2018-04-17 11:09:08 -06:00
Robert Haupt c6c6874d72 Added -datetime option to create up/down files with version in yyyymmddhhmmss format. 2018-04-16 11:12:06 -06:00
Andrés Rodríguez 78c47074a3 Add WithSession helper for Cassandra driver
Also changes method receivers names for the Cassandra struct to "c"
since "p" makes no sense in this context.
2018-03-13 15:49:08 -03:00
Dale Hui 22f249514d Merge branch 'postgres-lock-fix'
Addresses: https://github.com/golang-migrate/migrate/pull/4
2018-02-20 16:12:54 -08:00
Dale HuiandGitHub 8f6826c9e3 Merge pull request #8 from rdallman/fix-lockyloo-golang
Fix lockyloo golang
2018-02-20 15:51:18 -08:00
Dale Hui 1519c59188 Rename Postgres.db to Postgres.conn 2018-02-20 15:40:20 -08:00
Dale HuiandGitHub a9176e9969 Merge branch 'master' into fix-lockyloo-golang 2018-02-20 15:37:20 -08:00
Dale Hui ab7dfb38ac Merge remote-tracking branch 'origin/master' into postgres-lock-fix 2018-02-20 15:32:34 -08:00
Dale Hui aa21a21647 Support Go 1.10 and drop support for 1.8
- Test using latest minor versions of the last 2 major (supported) releases
        - TravisCI's gimme doesn't support Go 1.10.x until a minor version is released
    - Explicitly use YAML strings so Go versions aren't interpreted as numbers
2018-02-20 15:20:43 -08:00
Reed Allman 094bad431e s/db/conn/ 2018-02-09 15:14:05 -08:00
Reed Allman a8ef2b8cfe update CI and build tags to go1.9 2018-02-09 00:32:47 -08:00
Reed Allman 636c911d33 fixes mysql lock failure
I believe this closes #297 as well.

I have been working on adding testing of migrations and it requires acquiring
the lock in mysql multiple times to go up and down. After nailing this down to
GET_LOCK returning a failure for every subsequent GET_LOCK call after the
first, I decided it was time to rtfm and lo and behold:

https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_release-lock

RELEASE_LOCK will not work if called from a different thread than GET_LOCK.
The simplest solution using the golang database/sql pkg appears to be to just
get a single conn to use for every operation. since migrations are happening
sequentially, I don't think this will be a performance hit (possible
improvement). now calling Lock() and Unlock() multiple times will work;
prior to this patch, every call to RELEASE_LOCK would fail. this required
minimal changes to use the *sql.Conn methods instead of the *sql.DB methods.

other changes:

* upped time out to 10s on GET_LOCK, 1s timeout can too easily leave us in a
state where we think we have the lock but it has timed out (during the
operation).
* fixes SetVersion which was not using the tx it was intending to, and fixed a
bug where the transaction could have been left open since Rollback or Commit
may never have been called.

I added a test but it does not seem to come up in the previous patch, at least
easily, I tried some shenanigans. At least, this behavior should be fixed with
this patch and hopefully the test / comment is advisory enough.

thank you for maintaining this library, it has been relatively easy to
integrate with and most stuff works (pg works great :)
2018-02-09 00:27:01 -08:00
Dale Hui 53dd06057f Add release badge 2018-01-21 02:07:25 -08:00
Dale Hui 1b840d9c79 Remove unused invalid formatter 2018-01-21 02:04:36 -08:00
Dale Hui 012f051f2e Use cached badges when possible and make badges reflect the state of the master branch 2018-01-21 01:40:41 -08:00
Dale Hui 8684cad808 Update README to reflect the current status of the sqlite driver 2018-01-21 00:20:10 -08:00
Dale Hui 15611d4559 Fix cli builds
- Removed sqlite3 support from binaries
        - It was never working anyways: https://github.com/mattes/migrate/issues/244
    - Don't use cgo as is requires a cross-compiler to build for other platforms
        - cgo was originally added for this reason: https://github.com/mattes/migrate/pull/238#issuecomment-306291988
2018-01-20 23:59:36 -08:00
Dale Hui d542e144dc Manually fetch github.com/kshvakov/clickhouse 2018-01-20 10:30:57 -08:00
Dale Hui f8fd159fbc Fork packagecloud releases 2018-01-20 02:10:22 -08:00
Dale Hui 2eb9570b52 Build release binaries (using G 1.9) and upload to Github using new key 2018-01-20 01:49:02 -08:00
Dale Hui f5ae4b42db Support creating migrations using sequences in addition to timestamps 2018-01-20 00:29:06 -08:00
Dale Hui 24dd870559 MySQL and Postgres db driver test improvements
- Log unexpected connection errors while waiting for the docker image to start
    - Don't leave connections open
2018-01-20 00:11:35 -08:00
Dale Hui 97ef1b8393 Remove redundant delay since the readyFn is sufficient 2018-01-20 00:11:35 -08:00
Dale Hui bdb332a5f8 Remove unused slowReader 2018-01-19 22:38:32 -08:00
Dale Hui 8cbdc28e66 Source code needs to be kept for goveralls to use but removed before caching in travis-ci 2018-01-19 20:53:20 -08:00
Dale Hui 4ebf080da8 Speed up travis-ci builds by caching dependencies 2018-01-19 20:32:15 -08:00
Dale Hui 53606261bf Make coverage directory configurable 2018-01-19 20:32:15 -08:00
Dale Hui 50d049d5f5 Only use coverage profiles from test runs to create combined coverage profile.
- Prevent issue where the combined coverage profile is combined with itself
2018-01-19 18:04:46 -08:00
Daniel CormierandDale Hui 395391525b Changed source.Up from untyped to a typed const
Changed source.Up from being an untyped string const to being a typed source.Direction const.
2018-01-19 16:24:25 -08:00
Dale Hui 59f3f6d6c2 Increase test timeout from default 10m to 20m since some tests are slow as molasses in January
- Example build being killed by Go's test binary: https://travis-ci.org/golang-migrate/migrate/jobs/331027558
2018-01-19 15:42:02 -08:00
Dale Hui 12cc536541 Drop support for Go 1.7
- Only the latest 2 major release version is supported. https://golang.org/doc/devel/release.html#policy
2018-01-19 15:36:13 -08:00
Dale Hui 51ec421ad8 Remove debugging print from mysql driver test 2018-01-19 15:13:07 -08:00
Dale Hui 1cf095c51d Update more references to github.com/mattes/migrate 2018-01-19 14:59:27 -08:00
Dale Hui 7c9a261dc0 Fix MySQL docker image connection testing 2018-01-19 14:50:31 -08:00
Dale Hui 31367fbc97 Add tip about dealing with docker client API version issues 2018-01-19 14:31:56 -08:00
Dale Hui 7159f40304 Update all *DockerContainer receivers to handle nil
- Use errors.New() instead of fmt.Errorf() if no formatter string is needed
    - Spacing changes due to `go fmt`
2018-01-19 14:22:55 -08:00
Dale Hui 042e307bef Ignore vendor dir 2018-01-19 10:57:17 -08:00
Dale Hui f584949470 Update imports to reference fork.
Thanks `make rewrite-import-paths`!
2018-01-19 10:56:55 -08:00
Dale Hui 303cd74e5f Fix Docker install on TravisCi by allowing the docker-ce package to be downgraded
- Started happening since TravisCI updated docker-ce https://docs.travis-ci.com/user/build-environment-updates/2017-12-12/
    - Can't use v17.09 since the git tag doesn't exist yet
2018-01-19 02:28:41 -08:00
Dale Hui 3b61abf20a Update some examples and badges in README to reference new fork 2018-01-19 02:07:34 -08:00
Matthias KadenbachandGitHub d23f71b03c Merge pull request #305 from zikes/master
use DELETE FROM to avoid schema changes within transaction
2017-12-08 13:48:26 -08:00
Jason Hutchinson c156f5f3a1 replace TRUNCATE with DELETE FROM to avoid schema changes within transaction 2017-11-02 16:59:26 -05:00
Teddy Schmitz 155a8b241c Use db.Conn to fix postgres lock/unlock 2017-10-27 10:56:10 +08:00
Matthias KadenbachandGitHub 69472d5f5c Merge pull request #301 from shaneodonnell/master
Issue 247: Removed '(always included)'
2017-10-24 11:00:00 -07:00
Shane O'Donnell 4ac3ee9d75 Removed '(always included)' 2017-10-23 13:27:24 -04:00
Matthias KadenbachandGitHub 5b98c13eff Merge pull request #281 from ashoda/support_multiple_statements
(Spanner) Add support for multiple statements in one migration step
2017-10-09 11:03:59 -07:00
Ashod Ayanyan 28cb030ffe Add support for multiple DDL/SQL statements in one migration file 2017-09-18 15:43:46 -07:00
Matthias KadenbachandGitHub be1b075605 add go 1.9 for tests 2017-09-18 14:55:49 -07:00
Matthias KadenbachandGitHub 62cddb9e74 Merge pull request #278 from JensRantil/cassandra-auth
feat(cassandra): support for user/pw authentication
2017-09-18 10:08:58 -07:00
Matthias KadenbachandGitHub eabba5b316 Merge pull request #277 from JensRantil/cleanup
Various `gofmt`ing
2017-09-18 10:08:34 -07:00
Jens Rantil a5e584ce22 feat(cassandra): support for user/pw authentication 2017-09-17 13:32:00 +02:00
Jens Rantil d25732a53a sty: avoid unnecessary parenthesis
Golang best-practise. Cleanup made by `gofmt`.
2017-09-17 13:30:08 +02:00
Jens Rantil 701bd68731 style(cassandra): whitespace fixes
According to `gofmt`.
2017-09-17 13:26:30 +02:00
Jens Rantil 51acd3f3ad style: order imports lexicographically
Best-practise according to `gofmt`.
2017-09-17 13:26:23 +02:00
Matthias KadenbachandGitHub 804f9c2d4f Merge pull request #273 from maknahar/master
Fixed a couple of typos
2017-08-14 14:01:06 -07:00
maknahar 81c0a17f20 Fixed a couple of typos 2017-08-14 17:55:33 +05:30
Matthias KadenbachandGitHub 2570d5866d Merge pull request #272 from twrobel3/bugfix/cockroachTxnSig
Update cockroachDB Txn signatures
2017-08-11 17:11:41 -07:00
Taylor Wrobel 2c0f4f588e Update cockroachDB Txn signatures
cockroach-go made a backwards-incompatible change to their transaction
function signatures, which was causing the cockroach instrumentation
to fail.

Updates the signature to match the cockraoch-go change.
2017-08-10 22:56:41 -07:00
Matthias KadenbachandGitHub 8b5365e62c Merge pull request #267 from nicolas33/FAQ-extension
Faq extension
2017-07-30 15:05:05 -07:00
Nicolas Sebrecht d837bcc60c FAQ: add required extension
Github-fix: #265
Signed-off-by: Nicolas Sebrecht <nicolas.sebrecht@1001pneus.fr>
2017-07-28 14:30:10 +02:00
Nicolas Sebrecht bfe502654c FAQ: update format from dos to unix
Signed-off-by: Nicolas Sebrecht <nicolas.sebrecht@1001pneus.fr>
2017-07-28 14:24:46 +02:00
Matthias KadenbachandGitHub 6632d110b6 Merge pull request #264 from twrobel3/feature/migrations-readme
Fill out MIGRATIONS.md with migration format and best practice notes
2017-07-25 23:05:42 -07:00
Taylor Wrobel 341c35b1b8 Fill out MIGRATIONS.md with migration format and best practice notes 2017-07-19 00:18:02 -07:00
Matthias KadenbachandGitHub 8e6084bc9c Merge pull request #263 from twrobel3/feature/crdb-readme
Add cockroachdb README and add force lock functionality
2017-07-12 08:48:03 -07:00
Taylor Wrobel 40a40613cb Add cockroachdb README and add ForceLock functionality
Adds a readme to the cockroachdb database package, following the
postgres readme style.

Also adds the ability to force acquisition of the migration lock
via a connect URL parameter/WithInstance config, to allow for fixing
cases where an implementation error causes the schema lock to not
be released.

Lastly, tweaks the CLI readme to include information on building a
CLI for databases other than postgres.
2017-07-11 21:52:02 -07:00
Watanabe MasaruandMatthias Kadenbach 23a2745a3a fix spanner ensure migration table 2017-07-09 10:57:10 -07:00
Matthias KadenbachandGitHub 88c6a0a752 Merge pull request #253 from kshvakov/master
Add ClickHouse driver
2017-07-08 10:11:23 -07:00
Kirill ShvakovandGitHub 1f62e3f142 Merge branch 'master' into master 2017-07-08 13:50:51 +03:00
Kirill Shvakov 8424e946e4 remove deprecated group from Travis file 2017-07-08 13:49:23 +03:00
Matthias KadenbachandGitHub 66a0448698 Merge pull request #255 from twrobel3/feature/cockroachdb
Add CockroachDB Support
2017-07-08 00:26:12 -07:00
Taylor Wrobel 2cab359222 Remove debugging output from cockroachDB tests 2017-07-07 23:13:55 -07:00
Taylor Wrobel aebc0c3cdd Merge port mapping helper functions into one 2017-07-07 23:12:07 -07:00
Taylor Wrobel c2925c40c7 Add tests for CockroachDB 2017-07-07 23:12:07 -07:00
Taylor Wrobel 88115dedbf Support running docker with commands and handling multiple exposed ports
Adds the ability to specify a series of commands to run as part
of the docker image execution, and allows for retrieving a mapping
of an exposed via the port bound within the container.
2017-07-07 23:12:07 -07:00
Taylor Wrobel 1d8a881941 Add CockroachDB Support
Adds support for CockroachDB.  Cockroach uses the postges wire
protocol and has a large amount of common SQL functionality shared
with Postgres, so much of the postgres code was able to be copied
and modified.

Since the protocol is used in determining the driver, and the
Postgres protocol is also used by Cockroach, new connect string
prefixes were added: cockroach:// cockroachdb:// and
crdb-postgres://.  These fake protocol strings are replaced in
the connect function with the correct `postgres://` protocol.

TODO: Tests needed (Cockroach has a docker image, so this shouldn't
be too hard)
2017-07-07 23:12:07 -07:00
Matthias KadenbachandGitHub 8439d7195f Merge pull request #262 from twrobel3/feature/fixTravisBuild
Update docker dependency version to fix Tavis builds
2017-07-07 22:45:46 -07:00
Matthias KadenbachandGitHub f0f8b2c85c Merge pull request #260 from martin-magakian/patch-1
Update README.md
2017-07-03 22:21:55 -07:00
Martin MagakianandGitHub 35af42a0dd Update README.md 2017-07-03 15:45:57 +02:00
Martin MagakianandGitHub 56a56527c6 Update README.md
Late edit I promise...
2017-07-03 15:45:41 +02:00
Martin MagakianandGitHub 1485e567f1 Update README.md
Some format :-p
2017-07-03 15:35:43 +02:00
Martin MagakianandGitHub 180c57747d Update README.md
Forget package
2017-07-03 15:34:43 +02:00
Martin MagakianandGitHub 2361a7aa24 Update README.md
Code snippet don't compile.
It's missing a step (see https://github.com/mattes/migrate/blob/master/source/go-bindata/go-bindata_test.go)
2017-07-03 15:29:15 +02:00
Taylor Wrobel 4f9422ec85 Update docker dependency version to fix Tavis builds
Tavis has updated its docker images, and as part of the update,
no longer includes support for the (deprevated) docker-engine
package, moving instead to docker-ce.

See https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch
for details of the package change.

This updates the travis configuration to pin the docker library
version to 17.05.0 and installs the corresponding version of
docker-ce in the install script.
2017-06-24 10:37:08 -07:00
kshvakov 9d17987b8c Travis CI: using the previous Ubuntu Trusty images (https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch) 2017-06-23 12:16:14 +03:00
kshvakov 94056e51cc postgres: fix SetVersion. All requests were executed in different connections so commit/rollback doesn't work correctly 2017-06-23 11:13:54 +03:00
Kirill Shvakov 8cd9761d29 fix: set current database 2017-06-21 23:10:43 +03:00
kshvakov e86f4a201c change include path from github.com/kshvakov to github.com/mattes 2017-06-21 17:24:55 +03:00
kshvakov 9341404de0 move ensureVersionTable to init 2017-06-21 17:14:43 +03:00
kshvakov b031b22c80 Add WithInstance 2017-06-21 17:12:05 +03:00
kshvakov 1440a070c3 Add WithInstance 2017-06-21 17:06:04 +03:00
kshvakov 14636b77fb Add ClickHouse driver 2017-06-21 15:58:46 +03:00
Matthias KadenbachandGitHub da4160ce04 Merge pull request #251 from christianklotz/spanner-driver
Add Spanner driver
2017-06-20 21:52:00 -07:00
Christian Klotz eccc3a26d2 Skip test if SPANNER_DATABASE isn’t set 2017-06-20 17:53:41 +01:00
Christian KlotzandGitHub 51afcccb6b Merge branch 'master' into spanner-driver 2017-06-14 21:02:59 +01:00
Stephanie NgandChristian Klotz 8f7d3efb1a add create command to cli 2017-06-14 21:59:11 +02:00
BenandChristian Klotz 13a818e3b5 Fix typo forgotton => forgotten 2017-06-14 21:59:11 +02:00
Matthias KadenbachandChristian Klotz 5e049b9fb7 enable cgo 2017-06-14 21:59:11 +02:00
maxvwandChristian Klotz 33d5038bd3 adapted the sqlite driver for v3 (#165) 2017-06-14 21:58:37 +02:00
Fuyuan BieandChristian Klotz 39bb76224d Fix parenthesis problem. 2017-06-14 21:57:37 +02:00
Fuyuan BieandChristian Klotz 9632882b69 Use both db name and migration table name for lock hash 2017-06-14 21:57:37 +02:00
Christian Klotz 803598293e Add basic Google Spanner test 2017-06-14 19:46:10 +02:00
Matthias KadenbachandGitHub 50068870a1 Merge pull request #249 from recurza/master
Add create command to CLI
2017-06-10 10:25:06 -07:00
Stephanie Ng 9a19439e31 add create command to cli 2017-06-10 20:20:35 +08:00
Matthias KadenbachandGitHub d8dd1bc427 Merge pull request #248 from biefy/master
Solve lock contention problem when upgrading multiple logical MySQL databases backed by a single physical database.
2017-06-09 17:08:54 -07:00
Matthias KadenbachandGitHub c1134be427 Merge pull request #242 from benmoss/master
Fix typo forgotton => forgotten
2017-06-05 13:11:58 -07:00
Matthias KadenbachandGitHub 2aab3394cf Merge pull request #243 from mattes/mattes-patch-1
enable cgo
2017-06-05 13:11:27 -07:00
Ben 39a5a37822 Fix typo forgotton => forgotten 2017-06-05 12:14:10 -04:00
Jolan MalassigneandChristian Klotz 89879968bb add cassandra driver and function to retrieve networkSettings to get port bound to 9042 2017-06-05 11:03:49 +01:00
Pavlina DrososandChristian Klotz 6ecd671cfc Add code sample for use with existing MySQL client with multistatements=true 2017-06-05 11:02:51 +01:00
MaxandChristian Klotz 2f50c9c52b added file cli/build_sqlite3.go 2017-06-05 11:02:51 +01:00
maxvwandChristian Klotz 386ce008d9 adapted the sqlite driver for v3 (#165) 2017-06-05 11:02:51 +01:00
Christian Klotz 3d6c78875f Add example migrations 2017-06-05 10:50:14 +01:00
Matthias KadenbachandGitHub a9bcd83ff0 enable cgo 2017-06-05 00:59:53 -07:00
Matthias KadenbachandGitHub bf3f0f133e Merge pull request #240 from tinysquare/master
Add cassandra driver
2017-06-04 21:27:48 -07:00
Matthias KadenbachandGitHub 2aa16f5f66 Merge branch 'master' into master 2017-06-04 21:27:38 -07:00
Matthias KadenbachandGitHub 63f788f07e Merge pull request #239 from pdrosos/update-mysql-documentation-for-use-with-existing-db-client
Add documentation for use with existing MySQL client with multiStatements=true
2017-06-04 21:23:57 -07:00
Matthias KadenbachandGitHub c3d61852ee Merge pull request #238 from maxvw/v3-sqlite
adapted the sqlite driver for v3 (#165)
2017-06-04 21:21:55 -07:00
Christian Klotz 2742b9c467 Provide WithInstance method
Also includes some refactor around package naming, treats own repo as first-class and uses alternative package names for third party imports, Google spanner libraries in this case.
2017-06-04 21:53:33 +01:00
Christian Klotz 748ae8f06a Add Spanner driver
Support for Google Cloud Spanner, closes #172. Includes example migrations ported over from the PostgreSQL driver.

In all examples provided by Google I’ve come across, camel-case is used for table names, columns, etc. Hence the examples use this naming convention.
2017-06-04 21:13:49 +01:00
Jolan Malassigne 28524d977c add cassandra driver and function to retrieve networkSettings to get port bound to 9042 2017-05-31 10:05:56 +02:00
Pavlina Drosos b5c2f0f2bb Add code sample for use with existing MySQL client with multistatements=true 2017-05-30 18:52:26 +03:00
Max decee4abb2 added file cli/build_sqlite3.go 2017-05-30 14:06:59 +02:00
maxvw 8b4ce58a1d adapted the sqlite driver for v3 (#165) 2017-05-27 08:58:01 +02:00
Matthias KadenbachandGitHub 3682bcf9d2 Update README.md 2017-05-18 19:47:23 -07:00
Matthias KadenbachandGitHub 857dc4b862 Update README.md 2017-05-18 19:45:53 -07:00
Matthias KadenbachandGitHub 247ca90b33 Merge pull request #234 from augustzf/master
Add command to make CLI example with Go toolchain work
2017-05-18 10:29:15 -07:00
Matthias KadenbachandGitHub 00b22e527d one line 2017-05-18 10:28:39 -07:00
August Z. Flatby 83035ec925 Add command to make CLI example with Go toolchain work 2017-05-18 09:27:39 +02:00
Matthias KadenbachandGitHub 035c07716c Merge pull request #231 from mattes/cli-updates
build cli with all source and database drivers
2017-05-17 19:37:49 -07:00
Matthias Kadenbach 0d9689ab9f go get update 2017-05-17 19:37:36 -07:00
Matthias Kadenbach e696a861ba fix makefile 2017-05-17 19:26:24 -07:00
Matthias Kadenbach 801e84d4d2 fix make deps 2017-05-17 19:21:48 -07:00
Matthias Kadenbach ff7e060269 remove ql 2017-05-17 19:16:08 -07:00
Matthias Kadenbach d4d20346f2 build cli with all source and database drivers 2017-05-17 19:10:45 -07:00
Matthias KadenbachandGitHub 1b48b97aa4 Update README.md 2017-05-16 13:27:30 -07:00
Matthias KadenbachandGitHub d1e665bac7 Merge pull request #227 from fsouza/gcs-source
source/google-cloud-storage: implement the driver
2017-05-16 13:14:34 -07:00
Matthias KadenbachandGitHub dc109fc118 Merge pull request #228 from fsouza/s3-source
source/aws-s3: implement the driver
2017-05-16 13:13:00 -07:00
Matthias KadenbachandGitHub fb8698d89a Merge pull request #223 from Menta2L/master
ql database support
2017-05-16 13:09:22 -07:00
vesco 2821341551 make ExecuteQuery private 2017-05-16 12:07:40 +03:00
Fuyuan BieandGitHub bcdbe0f018 Fix parenthesis problem. 2017-05-15 22:01:45 -07:00
Fuyuan BieandGitHub 5f2a8b7f30 Use both db name and migration table name for lock hash 2017-05-15 21:50:41 -07:00
Matthias KadenbachandGitHub 2831b4c0d8 Merge pull request #222 from OrbotixInc/master
Added support for Redshift.
2017-05-15 18:31:13 -07:00
Francisco Souza c95cbb5931 source/aws-s3: implement the driver 2017-05-14 14:31:40 -04:00
Francisco Souza da89656fdc source/google-cloud-storage: implement the driver 2017-05-14 13:47:01 -04:00
root 4e5dc68b4e ql support 2017-05-12 11:11:16 +03:00
Jonathan Bryant 91ecd3cb6a Added support for Redshift.
* Added driver that wraps the standard PostgreSQL driver
* Added build.
* Added driver to documentation.
2017-05-11 15:47:40 -06:00
Matthias KadenbachandGitHub 448a36c5cb Merge pull request #210 from divan/master
Add missing import to example in README
2017-05-04 11:05:03 -07:00
Ivan DanilukandGitHub ed5a887579 Add import to example.
The example in README with file source doesn't work without importing "github.com/mattes/migrate/source/github" package.
2017-05-04 19:42:12 +02:00
Matthias KadenbachandGitHub e42058f15a Merge pull request #209 from pgmcgee/patch-1
Fix a typo in go-bindata/README.md
2017-05-04 09:33:40 -07:00
Parker McGeeandGitHub a7e1cacd36 Fix a typo
I was browsing this and noticed the typo that is causing syntax highlighting to be off.
2017-05-04 07:44:37 -04:00
Matthias KadenbachandGitHub 734803549d Merge pull request #204 from fln/master
Fix appending multiStatements=true parameter for mysql driver
2017-04-25 13:37:09 -07:00
Julius Kriukas 14f784961f Fix appending multiStatements=true parameter for mysql driver 2017-04-25 22:54:16 +03:00
Matthias Kadenbach 5c8da71dea update travis branch 2017-04-25 12:01:21 -07:00
Matthias Kadenbach a6fe4b73b3 update makefile 2017-04-25 11:59:52 -07:00
Matthias KadenbachandGitHub 256e8f045b Update README.md 2017-04-25 11:13:59 -07:00
Matthias KadenbachandGitHub 2e54bda0e1 Update README.md 2017-04-25 11:06:06 -07:00
Matthias Kadenbach 2ae8e6291c remove v1 migrate compat 2017-04-25 10:37:55 -07:00
Matthias KadenbachandGitHub 651f78cba8 Update README.md 2017-04-21 13:36:03 -07:00
Matthias KadenbachandGitHub d5674ed411 Create README.md 2017-04-21 13:30:59 -07:00
Matthias KadenbachandGitHub 4a2a810c20 Update README.md 2017-04-21 13:29:13 -07:00
Matthias KadenbachandGitHub 365b0565de Update README.md 2017-04-21 13:07:46 -07:00
Matthias KadenbachandGitHub 65d785d560 Update README.md 2017-04-21 13:02:10 -07:00
Matthias KadenbachandGitHub fff688c019 Create README.md 2017-04-21 13:00:19 -07:00
Matthias Kadenbach 6f07ac731b fix go toolchain build 2017-04-21 12:23:01 -07:00
Matthias Kadenbach f00c15d7b0 update go toolchain install 2017-04-20 16:03:43 -07:00
Matthias Kadenbach 1676a902f2 update readme 2017-04-20 15:22:06 -07:00
Matthias KadenbachandGitHub 576d323c52 Merge pull request #200 from Gobonoid/fix/ReadmeUpdate
Updates import path in README example
2017-04-19 10:04:37 -07:00
Gobonoid b4adf71696 Updates import path in README examplee 2017-04-19 11:48:58 +01:00
Matthias KadenbachandGitHub c0703e7a92 Merge pull request #199 from guzart/date-versions
Date-based versions
2017-04-16 12:38:48 -07:00
Arturo Guzman d789d50bd6 Support date-based versions 2017-04-13 20:57:22 -04:00
Matthias KadenbachandGitHub b83285bd59 Merge pull request #194 from Arkq/v3.0-prev
Support for alternative postgres URI scheme
2017-04-10 11:22:50 -07:00
Arkadiusz Bokowy bc93ed027f Support for alternative postgres URI scheme
According to the PostgreSQL documentation (section 32.1.1.2), postgres
library supports two URI schemes: postgresql:// and postgres://

Reference:
https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
2017-04-04 08:29:33 +02:00
Matthias Kadenbach 9f974b8996 Update Readme 2017-03-12 18:00:22 -07:00
Matthias Kadenbach c03b03c909 return status 0 for ErrNoChange #182 2017-03-12 12:33:13 -07:00
Matthias Kadenbach cecce18c51 fix source/file relative paths, ref #178 thanks @matyunin 2017-03-03 12:02:32 -08:00
Matthias Kadenbach 1f37f41ee8 remove cleanpath, fix mysql ensureVersionTable 2017-03-01 11:48:38 -08:00
Matthias Kadenbach 150ac7d708 add mysql custom TLS config
closes https://github.com/mattes/migrate/pull/117
2017-02-28 16:23:29 -08:00
Matthias Kadenbach be1ba9204a add mysql driver, add ENV to docker containers 2017-02-28 15:10:56 -08:00
Matthias Kadenbach 760bc3eb2d add note in readme 2017-02-23 14:50:48 -08:00
Matthias Kadenbach 706f23115f add v1compat 2017-02-23 14:48:44 -08:00
Matthias KadenbachandGitHub 171f9a256d Update FAQ.md 2017-02-23 14:27:04 -08:00
Matthias KadenbachandGitHub e7b9b0d0ed Update README.md 2017-02-23 13:36:34 -08:00
Matthias Kadenbach 5645143baa add example dockerfile 2017-02-23 13:13:02 -08:00
Matthias Kadenbach ec5c0ac3e0 fix version string 2017-02-23 10:36:33 -08:00
Matthias Kadenbach 511f9c1ffe only deploy on version go 1.8 2017-02-23 10:30:18 -08:00
Matthias Kadenbach 6325327ba1 fix dir 2017-02-22 17:58:25 -08:00
Matthias Kadenbach d60d590381 remove escaped new line 2017-02-22 17:48:50 -08:00
Matthias Kadenbach 5aac011e49 remove dups 2017-02-22 17:38:40 -08:00
Matthias Kadenbach 320d573ccd fix sudo whitespace 2017-02-22 17:27:18 -08:00
Matthias Kadenbach c51bef38b6 sudo 2017-02-22 17:15:40 -08:00
Matthias Kadenbach 28e0548cac fix github
see https://github.com/google/go-github/commit/23d6cb9cacb5aa314e93d600fe20a48496a718d4
2017-02-22 16:56:14 -08:00
Matthias Kadenbach d0acf6ecfe build deb package and push to packagecloud 2017-02-22 16:41:11 -08:00
Matthias Kadenbach 4b91f503eb make Force doc clear 2017-02-19 16:22:48 -08:00
Matthias Kadenbach eb02bc77fb allow to force NilVersion 2017-02-19 16:06:11 -08:00
Matthias KadenbachandGitHub 55f54251eb Update FAQ.md 2017-02-19 15:56:45 -08:00
Matthias Kadenbach e64b7cc233 explain dirty version error 2017-02-19 15:51:12 -08:00
Matthias Kadenbach 4bb46a98c6 fix github
see https://github.com/google/go-github/commit/2a4b92075e5c5e42ee0e55819b70f49b02c1ac64
2017-02-19 15:31:57 -08:00
Matthias Kadenbach 53b55288b3 bring back dirty state from v2 2017-02-19 15:15:00 -08:00
Matthias KadenbachandGitHub f576a64f11 Update README.md 2017-02-18 10:41:17 -08:00
Matthias Kadenbach d574676702 drop custom query params 2017-02-17 16:59:47 -08:00
Matthias Kadenbach f45821581c add multi statement test 2017-02-16 22:10:01 -08:00
Matthias KadenbachandGitHub bb6ae0898d Update README.md 2017-02-16 21:13:12 -08:00
Matthias KadenbachandGitHub ea7fc641ba Update README.md 2017-02-16 21:12:33 -08:00
Matthias KadenbachandGitHub 87e6482497 travis build with go 1.8 2017-02-16 20:46:12 -08:00
Matthias KadenbachandGitHub 1e08e21470 update with issues 2017-02-16 20:41:17 -08:00
Matthias Kadenbach ccb40ff3ff Update Readme 2017-02-16 11:29:18 -08:00
Matthias Kadenbach 3291f3f891 add comment 2017-02-16 11:18:02 -08:00
Matthias Kadenbach 2031939bfc allow postgres migrations_table config 2017-02-16 11:06:11 -08:00
Matthias Kadenbach 6394299937 clean cli -path 2017-02-15 11:07:28 -08:00
Matthias Kadenbach 316f6c0cf3 switch to binary search 2017-02-15 10:48:43 -08:00
Matthias Kadenbach 53630f5ff1 add NewWithDatabaseInstance example 2017-02-14 23:25:19 -08:00
Matthias Kadenbach 3feee5958a updated postgres driver, add database.Error struct 2017-02-14 23:12:16 -08:00
Matthias Kadenbach 209c056258 add database lock timeouts in migrate and in tests 2017-02-11 19:15:54 -08:00
Matthias Kadenbach cd6e62049c Update documentation 2017-02-11 11:13:27 -08:00
Matthias Kadenbach 526c0a918e add packagecloud.io badge 2017-02-09 20:09:11 -08:00
Matthias Kadenbach de7ffa60cc update readme 2017-02-09 20:07:04 -08:00
Matthias Kadenbach 81b9791178 move contributing in own file 2017-02-09 20:03:56 -08:00
Matthias Kadenbach c4ec6ac0f7 move CLI usage in own README 2017-02-09 20:00:38 -08:00
Matthias Kadenbach cfe3d2a83e add example for parallelTest 2017-02-09 19:52:50 -08:00
Matthias Kadenbach b6bd4fe02d add comments for source 2017-02-09 19:42:48 -08:00
Matthias Kadenbach 2da976155a add comments for database 2017-02-09 19:04:54 -08:00
Matthias Kadenbach e69e0de9c9 fix old ref to LogString 2017-02-09 18:26:58 -08:00
Matthias Kadenbach cfc9f10a6e add some more comments 2017-02-09 18:23:08 -08:00
Matthias Kadenbach 8ba71b0754 rename nameFromUrl to schemeFromUrl 2017-02-09 18:19:13 -08:00
Matthias Kadenbach 46c16bb8f1 add test for suint 2017-02-09 18:16:16 -08:00
Matthias Kadenbach 1b079a3cda rename Migration.LongString to LogString 2017-02-09 18:06:38 -08:00
Matthias Kadenbach c1b23fee45 add Migrate.Run 2017-02-09 18:04:36 -08:00
Matthias Kadenbach 199678e1bc add code comments for package migrate 2017-02-09 16:59:30 -08:00
Matthias Kadenbach d5daf6f6ae add .godoc.pid to gitignore 2017-02-09 13:47:28 -08:00
Matthias Kadenbach 03d8d3233f fix line break in makefile 2017-02-09 12:03:03 -08:00
Matthias Kadenbach 5c6a21b014 add make docs 2017-02-09 11:59:49 -08:00
Matthias Kadenbach 3965df69d5 fix indent in makefile 2017-02-08 20:13:17 -08:00
Matthias Kadenbach f47436e158 add make release 2017-02-08 19:54:50 -08:00
Matthias Kadenbach f474e4cae1 travis builds dep tree info 2017-02-08 19:44:12 -08:00
Matthias Kadenbach 7979fdb832 add list-external-deps 2017-02-08 19:34:53 -08:00
Matthias Kadenbach b3f6bc47bb update readme with dev, testing, and contribution section 2017-02-08 18:27:12 -08:00
Matthias Kadenbach e6163082d7 add restore-import-paths and rewrite-import-paths 2017-02-08 18:01:01 -08:00
Matthias Kadenbach fe1b5c06eb add go coverage 2017-02-08 16:17:04 -08:00
Matthias Kadenbach 297a4227ef fixing travis 2017-02-08 01:49:57 -08:00
Matthias Kadenbach 9f774d3e6a fix typo in go-bindata README 2017-02-07 23:00:25 -08:00
Matthias Kadenbach f7699f1dec update go-bindata with latest source.Migration helper 2017-02-07 22:59:12 -08:00
Matthias Kadenbach 3889c93b63 add Readme draft 2017-02-07 22:40:45 -08:00
Matthias Kadenbach ee384ed8cc initial version 3.0.0 preview
hard work
2017-02-07 22:01:29 -08:00
Matthias Kadenbach e92f038f40 skip neo4j tests 2017-01-23 13:54:25 -08:00
Matthias KadenbachandGitHub 0b3426bf5d Merge pull request #25 from daddykotex/master
Ensure that interrupt signals are not relayed to the interrupt channel when the migration is over
2017-01-23 13:37:42 -08:00
Matthias Kadenbach 0861ff957d Update Readme 2017-01-23 13:36:22 -08:00
Matthias Kadenbach 32607ef715 Merge branch 'carlosforero-neo4j-driver' 2017-01-23 13:32:56 -08:00
Matthias Kadenbach c643ffd6d6 fix merge conflicts 2017-01-23 13:32:48 -08:00
Matthias KadenbachandGitHub b8b3883950 Merge pull request #101 from bfosberry/fixing-unsafe-cast
Fixing unsafe typecast
2017-01-23 13:26:59 -08:00
Matthias KadenbachandGitHub 3547fc54cb Merge pull request #144 from benma/patch-1
Update version.go
2017-01-19 09:32:29 -08:00
benmaandGitHub 64c41a8af0 Update version.go 2017-01-19 14:11:19 +01:00
Matthias KadenbachandGitHub 4191ac9cfe Merge pull request #142 from goenning/master
allow use of big version numbers
2017-01-18 15:48:54 -08:00
goenning f1738e78db allow use of big version numbers 2017-01-18 22:19:02 +00:00
Matthias KadenbachandGitHub de494e4b98 Merge pull request #128 from oliverkofoed/master
Add support for ScyllaDB (remove cql counter use)
2017-01-10 09:26:17 -08:00
Matthias KadenbachandGitHub 585509f3bc Merge pull request #138 from mattes/add-mongo-to-readme
add mongodb to readme
2017-01-09 10:43:35 -08:00
Matthias KadenbachandGitHub f630dc5971 Update README.md 2017-01-09 10:43:22 -08:00
Matthias KadenbachandGitHub 9a8f78ba3b Merge pull request #137 from drboyer/updown-colors
Change colors for up and down migration arrows
2017-01-09 10:39:55 -08:00
Matthias KadenbachandGitHub 63c6168c7f Merge pull request #136 from dimag-jfrog/prepare_pr_2_rebase
Driver for MongoDB (New)
2017-01-09 10:38:56 -08:00
dimag 1993f0783b Merge remote-tracking branch 'fork/prepare_pr_2_rebase' into prepare_pr_2_rebase
# Conflicts:
#	docker-compose.yml
2017-01-07 16:44:36 +02:00
dimag 70238f448b - Reverted repository from dimag-jfrog to mattes to prepare for pull request 2017-01-07 16:36:25 +02:00
dimag 914c38e379 - Fixed test with not-exported method identified as missing: unified the 2 errors into one
- Fixed error message formatting
- Ensuring uniqueness of the version field (added unique index)
2017-01-07 16:34:06 +02:00
dimag 0f28b01383 specifying version of mongo 2017-01-07 16:34:06 +02:00
dimag 6909594c0b Checking non-exported migration method 2017-01-07 16:33:10 +02:00
dimag 583921e8bf Added README.md 2017-01-07 16:33:10 +02:00
dimag 61ecf2c096 - Refactored project structure to be consistent with original project
- Added docker support
2017-01-07 16:33:10 +02:00
dimag 0eecb74230 Added explicit migration method signature validation 2017-01-07 16:30:40 +02:00
dimag b7b75f3c59 - registering method receiver directly for the driver
- currently each driver contains only a single method receiver:
   - enforcing method receiver pre-registration on go methods driver initialization
   - Method receiver name can be removed from files format
- passing the DbName parameter inside the method receiver for the go methods driver
2017-01-07 16:30:40 +02:00
dimag fe48ddfdcc - redefined mongo db driver to be autonomous and not depending on specific methods
- defined methods receivers registration by name and change the migration files format to include them
- added extensive testing
2017-01-07 16:30:40 +02:00
dimag 70b23c3f84 Relaxed constratint that method receiver has to be of Driver interface. 2017-01-07 16:30:40 +02:00
dimag 223908f9d2 - Reset all changes to the upstream branch
- Changed logic not to use custom filename parser:
  -Supporting up and down files only, no both direction files
  -Using method names as is
- Added complete test to the mongo db migration scenarios
2017-01-07 16:30:40 +02:00
dimag 1838852d4d Changes that interfere with the Migrate open source code:
- Added go methods migrator, mongo db template: different from the usual driver
model.
- Added support for bidirectional files (for go methods), appending _up or _down upon context
- Added DriverWithFilnameParser for providing custom filename parser functionality that knows to parse bi-directional file names.
2017-01-07 16:29:19 +02:00
dimag 175550643c changed import paths to point to dimag-jfrog 2017-01-07 16:24:14 +02:00
Matthias KadenbachandGitHub eb0eec4471 Merge pull request #135 from ansel1/master
Avoid DDL when checking for versions table #134
2017-01-06 14:24:59 -08:00
Russ Egan da6c505f41 Avoid DDL when checking for versions table #134 2017-01-06 15:52:54 -05:00
Matthias KadenbachandGitHub 966ba843e0 Merge pull request #133 from wlcx/add-ql-driver
Add Ql driver
2017-01-05 14:04:34 -08:00
Sam Willcocks 5773f75698 Add Ql driver
Adds a driver for the embedded Ql database (https://github.com/cznic/ql)
2017-01-05 15:09:41 +00:00
254 changed files with 2204 additions and 14406 deletions
-56
View File
@@ -1,56 +0,0 @@
# Golang CircleCI 2.0 configuration file
#
# Check https://circleci.com/docs/2.0/language-go/ for more details
version: 2.1
jobs:
"golang-1_15": &template
machine:
# https://circleci.com/docs/2.0/configuration-reference/#available-machine-images
image: ubuntu-2004:202010-01
# docker_layer_caching: true
# https://circleci.com/docs/2.0/configuration-reference/#resource_class
resource_class: medium
# Leave working directory unspecified and use defaults:
# https://circleci.com/blog/go-v1.11-modules-and-circleci/
# working_directory: /go/src/github.com/golang-migrate/migrate
environment:
GO111MODULE: "on"
GO_VERSION: "1.15.x"
steps:
# - setup_remote_docker:
# version: 19.03.13
# docker_layer_caching: true
- run: curl -sL -o ~/bin/gimme https://raw.githubusercontent.com/travis-ci/gimme/master/gimme
- run: curl -sfL -o ~/bin/golangci-lint.sh https://install.goreleaser.com/github.com/golangci/golangci-lint.sh
- run: chmod +x ~/bin/gimme ~/bin/golangci-lint.sh
- run: eval "$(gimme $GO_VERSION)"
- run: golangci-lint.sh -b ~/bin v1.37.0
- checkout
- restore_cache:
keys:
- go-mod-v1-{{ arch }}-{{ checksum "go.sum" }}
- run: golangci-lint run
- run: make test COVERAGE_DIR=/tmp/coverage
- save_cache:
key: go-mod-v1-{{ arch }}-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod"
- run: go get github.com/mattn/goveralls
- run: goveralls -service=circle-ci -coverprofile /tmp/coverage/combined.txt
"golang-1_16":
<<: *template
environment:
GO_VERSION: "1.16.x"
workflows:
version: 2
build:
jobs:
- "golang-1_15"
- "golang-1_16"
+1
View File
@@ -2,6 +2,7 @@
FAQ.md
README.md
LICENSE
Makefile
.gitignore
.travis.yml
CONTRIBUTING.md
+1 -1
View File
@@ -25,7 +25,7 @@ e.g. s3, github, go-bindata, gcs, file
Obtained by running: `migrate -help`
**Loaded Database Drivers**
e.g. spanner, stub, datastore, cockroachdb, crdb-postgres, postgres, postgresql, pgx, redshift, cassandra, cockroach, mysql
e.g. spanner, stub, clickhouse, cockroachdb, crdb-postgres, postgres, postgresql, redshift, cassandra, cockroach, mysql
Obtained by running: `migrate -help`
**Go Version**
-9
View File
@@ -1,9 +0,0 @@
<svg xmlns="http://www.w3.org/2000/svg" width="1280" height="640" viewBox="0 0 1280 640" role="img" aria-label="migrate">
<rect width="1280" height="640" fill="#0A0A0A"/>
<svg x="96" y="215" width="210" height="210" viewBox="0 0 67 67"><path d="M22.21 67V44.6369H0V67H22.21Z" fill="#fff"/><path d="M66.7038 22.3184H22.2534L0.0878906 44.6367H44.4634L66.7038 22.3184Z" fill="#fff"/><path d="M22.21 0H0V22.3184H22.21V0Z" fill="#fff"/><path d="M66.7198 0H44.5098V22.3184H66.7198V0Z" fill="#fff"/><path d="M66.7198 67V44.6369H44.5098V67H66.7198Z" fill="#fff"/></svg>
<text x="378" y="276" font-family="Inter,system-ui,-apple-system,sans-serif" font-size="78" font-weight="800" letter-spacing="-2" fill="#ffffff">migrate</text>
<text x="378" y="322" font-family="Inter,system-ui,sans-serif" font-size="30" fill="#ffffff" opacity=".66">Database migrations. CLI and Golang library.</text>
<rect x="378" y="338" width="806" height="3" rx="1.5" fill="#ffffff" opacity=".9"/>
<text x="378" y="390" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">github.com/hanzoai</text>
<text x="1184" y="390" text-anchor="end" font-family="Inter,system-ui,sans-serif" font-size="24" font-weight="600" fill="#ffffff" opacity=".5">hanzo.ai</text>
</svg>

Before

Width:  |  Height:  |  Size: 1.3 KiB

-3
View File
@@ -6,6 +6,3 @@ cli/migrate
.godoc.pid
vendor/
.vscode/
.idea
dist/
migrate
+20 -30
View File
@@ -1,37 +1,27 @@
version: "2"
run:
# timeout for analysis, e.g. 30s, 5m, default is 1m
deadline: 2m
linters:
enable:
#- golint
- interfacer
- unconvert
#- dupl
- goconst
- gofmt
- misspell
- maligned
- unparam
- nakedret
- prealloc
- revive
- unconvert
- unparam
settings:
misspell:
locale: US
revive:
rules:
- name: redundant-build-tag
exclusions:
generated: lax
rules:
- path: (.+)\.go$
text: G104
paths:
- third_party$
- builtin$
- examples$
#- gosec
linters-settings:
misspell:
locale: US
issues:
max-issues-per-linter: 0
max-same-issues: 0
formatters:
enable:
- gofmt
exclusions:
generated: lax
paths:
- third_party$
- builtin$
- examples$
max-same: 0
max-per-linter: 0
exclude-use-default: false
exclude:
# gosec: Duplicated errcheck checks
- G104
-102
View File
@@ -1,102 +0,0 @@
project_name: migrate
before:
hooks:
- go mod tidy
builds:
- env:
- CGO_ENABLED=0
goos:
- linux
- windows
- darwin
goarch:
- amd64
- arm
- arm64
- 386
goarm:
- 7
main: ./cmd/migrate
ldflags:
- '-w -s -X main.Version={{ .Version }} -extldflags "static"'
flags:
- "-tags={{ .Env.DATABASE }} {{ .Env.SOURCE }}"
- "-trimpath"
nfpms:
- homepage: "https://github.com/golang-migrate/migrate"
maintainer: "dhui@users.noreply.github.com"
license: MIT
description: "Database migrations"
formats:
- deb
file_name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
dockers:
- goos: linux
goarch: amd64
dockerfile: Dockerfile.github-actions
use: buildx
ids:
- migrate
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
build_flag_templates:
- '--label=org.opencontainers.image.created={{ .Date }}'
- '--label=org.opencontainers.image.title={{ .ProjectName }}'
- '--label=org.opencontainers.image.revision={{ .FullCommit }}'
- '--label=org.opencontainers.image.version={{ .Version }}'
- "--label=org.opencontainers.image.source={{ .GitURL }}"
- "--platform=linux/amd64"
- goos: linux
goarch: arm64
dockerfile: Dockerfile.github-actions
use: buildx
ids:
- migrate
image_templates:
- 'migrate/migrate:{{ .Tag }}-arm64'
build_flag_templates:
- '--label=org.opencontainers.image.created={{ .Date }}'
- '--label=org.opencontainers.image.title={{ .ProjectName }}'
- '--label=org.opencontainers.image.revision={{ .FullCommit }}'
- '--label=org.opencontainers.image.version={{ .Version }}'
- "--label=org.opencontainers.image.source={{ .GitURL }}"
- "--platform=linux/arm64"
docker_manifests:
- name_template: 'migrate/migrate:{{ .Tag }}'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
- name_template: 'migrate/migrate:{{ .Major }}'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
- name_template: 'migrate/migrate:latest'
image_templates:
- 'migrate/migrate:{{ .Tag }}-amd64'
- 'migrate/migrate:{{ .Tag }}-arm64'
archives:
- name_template: "{{ .ProjectName }}.{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
format_overrides:
- goos: windows
format: zip
checksum:
name_template: 'sha256sum.txt'
release:
draft: true
prerelease: auto
source:
enabled: true
format: zip
changelog:
skip: false
sort: asc
filters:
exclude:
- '^docs:'
- '^test:'
- Merge pull request
- Merge branch
- go mod tidy
snapshot:
name_template: "{{ .Tag }}-next"
+11 -14
View File
@@ -6,8 +6,8 @@ matrix:
- go: master
include:
# Supported versions of Go: https://golang.org/dl/
- go: "1.14.x"
- go: "1.15.x"
- go: "1.11.x"
- go: "1.12.x"
- go: master
go_import_path: github.com/golang-migrate/migrate
@@ -34,7 +34,7 @@ before_install:
- sudo apt-get update
- sudo apt-get -y -o Dpkg::Options::="--force-confnew" install docker-ce
# Install golangci-lint
- curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.30.0
- curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.17.1
- echo "TRAVIS_GO_VERSION=${TRAVIS_GO_VERSION}"
install:
@@ -57,16 +57,13 @@ deploy:
secure: hWH1HLPpzpfA8pXQ93T1qKQVFSpQp0as/JLQ7D91jHuJ8p+RxVeqblDrR6HQY/95R/nyiE9GJmvUolSuw5h449LSrGxPtVWhdh6EnkxlQHlen5XeMhVjRjFV0sE9qGe8v7uAkiTfRO61ktTWHrEAvw5qpyqnNISodmZS78XIasPODQbNlzwINhWhDTHIjXGb4FpizYaL3OGCanrxfR9fQyCaqKGGBjRq3Mfq8U6Yd4mApmsE+uJxgaZV8K5zBqpkSzQRWhcVGNL5DuLsU3gfSJOo7kZeA2G71SHffH577dBoqtCZ4VFv169CoUZehLWCb+7XKJZmHXVujCURATSySLGUOPc6EoLFAn3YtsCA04mS4bZVo5FZPWVwfhjmkhtDR4f6wscKp7r1HsFHSOgm59QfETQdrn4MnZ44H2Jd39axqndn5DvK9EcZVjPHynOPnueXP2u6mTuUgh2VyyWBCDO3CNo0fGlo7VJI69IkIWNSD87K9cHZWYMClyKZkUzS+PmRAhHRYbVd+9ZjKOmnU36kUHNDG/ft1D4ogsY+rhVtXB4lgWDM5adri+EIScYdYnB1/pQexLBigcJY9uE7nQTR0U6QgVNYvun7uRNs40E0c4voSfmPdFO0FlOD2y1oQhnaXfWLbu9nMcTcs4RFGrcC7NzkUN4/WjG8s285V6w=
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
file:
- cli/build/migrate.linux-amd64.tar.gz
- cli/build/migrate.linux-armv7.tar.gz
- cli/build/migrate.linux-arm64.tar.gz
- cli/build/migrate.darwin-amd64.tar.gz
- cli/build/migrate.windows-amd64.exe.tar.gz
- cli/build/migrate.windows-386.exe.tar.gz
- cli/build/sha256sum.txt
- dependency_tree.txt
- provider: packagecloud
@@ -78,7 +75,7 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -90,7 +87,7 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -98,11 +95,11 @@ deploy:
username: golang-migrate
token:
secure: aICwu3gJ1sJ1QVCD3elpg+Jxzt4P+Zj1uoh5f0sOwnjDNIZ4FwUT1cMrWloP8P2KD0iyCOawuZER27o/kQ21oX2OxHvQbYPReA2znLm7lHzCmypAAOHPxpgnQ4rMGHHJXd+OsxtdclGs67c+EbdBfoRRbK400Qz/vjPJEDeH4mh02ZHC2nw4Nk/wV4jjBIkIt9dGEx6NgOA17FCMa3MaPHlHeFIzU7IfTlDHbS0mCCYbg/wafWBWcbGqtZLWAYtJDmfjrAStmDLdAX5J5PsB7taGSGPZHmPmpGoVgrKt/tb9Xz1rFBGslTpGROOiO4CiMAvkEKFn8mxrBGjfSBqp7Dp3eeSalKXB1DJAbEXx2sEbMcvmnoR9o43meaAn+ZRts8lRL8S/skBloe6Nk8bx3NlJCGB9WPK1G56b7c/fZnJxQbrCw6hxDfbZwm8S2YPviFTo/z1BfZDhRsL74reKsN2kgnGo2W/k38vvzIpsssQ9DHN1b0TLCxolCNPtQ7oHcQ1ohcjP2UgYXk0FhqDoL+9LQva/DU4N9sKH0UbAaqsMVSErLeG8A4aauuFcVrWRBaDYyTag4dQqzTulEy7iru2kDDIBgSQ1gMW/yoBOIPK4oi6MtbTf1X39fzXFLS1cDd3LW61yAu3YrbjAetpfx2frIvrRAiL9TxWA1gnrs5o=
dist: ubuntu/focal
dist: ubuntu/cosmic
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -114,7 +111,7 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
- provider: packagecloud
@@ -126,13 +123,13 @@ deploy:
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
- provider: script
script: ./docker-deploy.sh
skip_cleanup: true
on:
go: "1.15.x"
go: "1.12.x"
repo: golang-migrate/migrate
tags: true
-1
View File
@@ -1 +0,0 @@
LLM.md
-1
View File
@@ -1 +0,0 @@
LLM.md
+10 -13
View File
@@ -1,26 +1,23 @@
FROM golang:1.26.4-alpine3.21 AS builder
FROM golang:1.12-alpine3.10 AS downloader
ARG VERSION
RUN apk add --no-cache git gcc musl-dev make
RUN apk add --no-cache git gcc musl-dev
WORKDIR /go/src/github.com/golang-migrate/migrate
ENV GO111MODULE=on
COPY go.mod go.sum ./
RUN go mod download
COPY . ./
RUN make build-docker
ENV GO111MODULE=on
ENV DATABASES="postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver"
ENV SOURCES="file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab"
FROM alpine:3.21
RUN go build -a -o build/migrate.linux-386 -ldflags="-s -w -X main.Version=${VERSION}" -tags "$DATABASES $SOURCES" ./cmd/migrate
FROM alpine:3.10
RUN apk add --no-cache ca-certificates
COPY --from=builder /go/src/github.com/golang-migrate/migrate/build/migrate.linux-386 /usr/local/bin/migrate
RUN ln -s /usr/local/bin/migrate /migrate
COPY --from=downloader /go/src/github.com/golang-migrate/migrate/build/migrate.linux-386 /migrate
ENTRYPOINT ["migrate"]
ENTRYPOINT ["/migrate"]
CMD ["--help"]
-17
View File
@@ -1,17 +0,0 @@
ARG DOCKER_IMAGE
FROM $DOCKER_IMAGE
RUN apk add --no-cache git gcc musl-dev make
WORKDIR /go/src/github.com/golang-migrate/migrate
ENV GO111MODULE=on
ENV COVERAGE_DIR=/tmp/coverage
COPY go.mod go.sum ./
RUN go mod download
COPY . ./
CMD ["make", "test"]
-11
View File
@@ -1,11 +0,0 @@
FROM alpine:3.19
RUN apk add --no-cache ca-certificates
COPY migrate /usr/local/bin/migrate
RUN ln -s /usr/local/bin/migrate /usr/bin/migrate
RUN ln -s /usr/local/bin/migrate /migrate
ENTRYPOINT ["migrate"]
CMD ["--help"]
+5 -8
View File
@@ -16,7 +16,7 @@
NilMigration defines a migration without a body. NilVersion is defined as const -1.
#### What is the difference between uint(version) and int(targetVersion)?
version refers to an existing migration version coming from a source and therefore can never be negative.
version refers to an existing migration version coming from a source and therefor can never be negative.
targetVersion can either be a version OR represent a NilVersion, which equals -1.
#### What's the difference between Next/Previous and Up/Down?
@@ -50,10 +50,10 @@
and whenever we want, not just once at the beginning of all tests.
#### Can I maintain my driver in my own repository?
Yes, technically thats possible. We want to encourage you to contribute your driver to this repository though.
Yes, technically thats possible. We want to encourage you to contribute your driver to this respository though.
The driver's functionality is dictated by migrate's interfaces. That means there should really
just be one driver for a database/ source. We want to prevent a future where several drivers doing the exact same thing,
just implemented a bit differently, co-exist somewhere on GitHub. If users have to do research first to find the
just implemented a bit differently, co-exist somewhere on Github. If users have to do research first to find the
"best" available driver for a database in order to get started, we would have failed as an open source community.
#### Can I mix multiple sources during a batch of migrations?
@@ -65,7 +65,7 @@
and then "force" the expected version.
#### What happens if two programs try and update the database at the same time?
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations on
Database-specific locking features are used by *some* database drivers to prevent multiple instances of migrate from running migrations at the same time
the same database at the same time. For example, the MySQL driver uses the `GET_LOCK` function, while the Postgres driver uses
the `pg_advisory_lock` function.
@@ -73,7 +73,4 @@
No, it is done automatically.
#### Can I use migrate with a non-Go project?
Yes, you can use the migrate CLI in a non-Go project, but there are probably other libraries/frameworks available that offer better test and deploy integrations in that language/framework.
#### I have got an error `Dirty database version 1. Fix and force version`. What should I do?
Keep calm and refer to [the getting started docs](GETTING_STARTED.md#forcing-your-database-version).
Yes, you can use the migrate CLI in a non-Go project, but there are probably other libraries/frameworks available that offer better test and deploy integrations in that language/framework.
+5 -15
View File
@@ -1,7 +1,7 @@
# Getting started
Before you start, you should understand the concept of forward/up and reverse/down database migrations.
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases).
Configure a database for your application. Make sure that your database driver is supported [here](README.md#databases)
## Create migrations
Create some migrations using migrate CLI. Here is an example:
@@ -10,7 +10,7 @@ migrate create -ext sql -dir db/migrations -seq create_users_table
```
Once you create your files, you should fill them.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created their migration later gets it merged to the repository first.
**IMPORTANT:** In a project developed by more than one person there is a chance of migrations inconsistency - e.g. two developers can create conflicting migrations, and the developer that created his migration later gets it merged to the repository first.
Developers and Teams should keep an eye on such cases (especially during code review).
[Here](https://github.com/golang-migrate/migrate/issues/179#issuecomment-475821264) is the issue summary if you would like to read more.
@@ -23,27 +23,17 @@ This way if one of commands fails, our database will remain unchanged.
Run your migrations through the CLI or your app and check if they applied expected changes.
Just to give you an idea:
```
migrate -database YOUR_DATABASE_URL -path PATH_TO_YOUR_MIGRATIONS up
migrate -database YOUR_DATBASE_URL -path PATH_TO_YOUR_MIGRATIONS up
```
Just add the code to your app and you're ready to go!
Before committing your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
Before commiting your migrations you should run your migrations up, down, and then up again to see if migrations are working properly both ways.
(e.g. if you created a table in a migration but reverse migration did not delete it, you will encounter an error when running the forward migration again)
It's also worth checking your migrations in a separate, containerized environment. You can find some tools at the [end of this document](#further-reading).
It's also worth checking your migrations in a separate, containerized environment. You can find some tools in the end of this document.
**IMPORTANT:** If you would like to run multiple instances of your app on different machines be sure to use a database that supports locking when running migrations. Otherwise you may encounter issues.
## Forcing your database version
In case you run a migration that contained an error, migrate will not let you run other migrations on the same database. You will see an error like `Dirty database version 1. Fix and force version`, even when you fix the erred migration. This means your database was marked as 'dirty'.
You need to investigate the migration error - was your migration applied partially, or was it not applied at all? Once you know, you should force your database to a version reflecting it's real state. You can do so with `force` command:
```
migrate -path PATH_TO_YOUR_MIGRATIONS -database YOUR_DATABASE_URL force VERSION
```
Once you force the version and your migration was fixed, your database is 'clean' again and you can proceed with your migrations.
For details and example of usage see [this comment](https://github.com/golang-migrate/migrate/issues/282#issuecomment-530743258).
## Further reading:
- [PostgreSQL tutorial](database/postgres/TUTORIAL.md)
- [Best practices](MIGRATIONS.md)
-39
View File
@@ -1,39 +0,0 @@
# Hanzo Golang Migrate
## Overview
Go module: github.com/golang-migrate/migrate/v4
## Tech Stack
- **Language**: Go
## Build & Run
```bash
go build ./...
go test ./...
```
## Structure
```
golang-migrate/
CONTRIBUTING.md
Dockerfile
Dockerfile.circleci
Dockerfile.github-actions
FAQ.md
GETTING_STARTED.md
LICENSE
MIGRATIONS.md
Makefile
README.md
SECURITY.md
cli/
cmd/
database/
dktesting/
```
## Key Files
- `README.md` -- Project documentation
- `go.mod` -- Go module definition
- `Makefile` -- Build automation
- `Dockerfile` -- Container build
+6 -21
View File
@@ -1,24 +1,15 @@
SOURCE ?= file go_bindata github github_ee bitbucket aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb yugabytedb datastore mongodb sqlserver firebird neo4j pgx pgx5 rqlite
DATABASE_TEST ?= $(DATABASE) sqlite sqlite3 sqlcipher
SOURCE ?= file go_bindata github github_ee aws_s3 google_cloud_storage godoc_vfs gitlab
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb clickhouse mongodb sqlserver
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
REPO_OWNER ?= $(shell cd .. && basename "$$(pwd)")
COVERAGE_DIR ?= .coverage
build:
CGO_ENABLED=0 go build -ldflags='-X main.Version=$(VERSION)' -tags '$(DATABASE) $(SOURCE)' ./cmd/migrate
build-docker:
CGO_ENABLED=0 go build -a -o build/migrate.linux-386 -ldflags="-s -w -X main.Version=${VERSION}" -tags "$(DATABASE) $(SOURCE)" ./cmd/migrate
build-cli: clean
-mkdir ./cli/build
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o ../../cli/build/migrate.linux-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm GOARM=7 go build -a -o ../../cli/build/migrate.linux-armv7 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -a -o ../../cli/build/migrate.linux-arm64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -a -o ../../cli/build/migrate.darwin-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=windows GOARCH=386 go build -a -o ../../cli/build/migrate.windows-386.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cmd/migrate && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -a -o ../../cli/build/migrate.windows-amd64.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cli/build && find . -name 'migrate*' | xargs -I{} tar czf {}.tar.gz {}
cd ./cli/build && shasum -a 256 * > sha256sum.txt
@@ -41,7 +32,7 @@ test:
test-with-flags:
@echo SOURCE: $(SOURCE)
@echo DATABASE_TEST: $(DATABASE_TEST)
@echo DATABASE: $(DATABASE)
@go test $(TEST_FLAGS) ./...
@@ -98,12 +89,6 @@ release:
git tag v$(V)
@read -p "Press enter to confirm and push to origin ..." && git push origin v$(V)
echo-source:
@echo "$(SOURCE)"
echo-database:
@echo "$(DATABASE)"
define external_deps
@echo '-- $(1)'; go list -f '{{join .Deps "\n"}}' $(1) | grep -v github.com/$(REPO_OWNER)/migrate | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'
@@ -111,10 +96,10 @@ define external_deps
endef
.PHONY: build build-docker build-cli clean test-short test test-with-flags html-coverage \
.PHONY: build-cli clean test-short test test-with-flags html-coverage \
restore-import-paths rewrite-import-paths list-external-deps release \
docs kill-docs open-docs kill-orphaned-docker-containers echo-source echo-database
docs kill-docs open-docs kill-orphaned-docker-containers
SHELL = /bin/sh
SHELL = /bin/bash
RAND = $(shell echo $$RANDOM)
+17 -34
View File
@@ -1,13 +1,11 @@
<p align="center"><img src=".github/hero.svg" alt="migrate" width="880"></p>
[![GitHub Workflow Status (branch)](https://img.shields.io/github/actions/workflow/status/golang-migrate/migrate/ci.yaml?branch=master)](https://github.com/golang-migrate/migrate/actions/workflows/ci.yaml?query=branch%3Amaster)
[![GoDoc](https://pkg.go.dev/badge/github.com/golang-migrate/migrate)](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)
[![Build Status](https://img.shields.io/travis/com/golang-migrate/migrate/master.svg)](https://travis-ci.com/golang-migrate/migrate)
[![GoDoc](https://godoc.org/github.com/golang-migrate/migrate?status.svg)](https://godoc.org/github.com/golang-migrate/migrate)
[![Coverage Status](https://img.shields.io/coveralls/github/golang-migrate/migrate/master.svg)](https://coveralls.io/github/golang-migrate/migrate?branch=master)
[![packagecloud.io](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/golang-migrate/migrate?filter=debs)
[![Docker Pulls](https://img.shields.io/docker/pulls/migrate/migrate.svg)](https://hub.docker.com/r/migrate/migrate/)
![Supported Go Versions](https://img.shields.io/badge/Go-1.24%2C%201.25-lightgrey.svg)
![Supported Go Versions](https://img.shields.io/badge/Go-1.11%2C%201.12-lightgrey.svg)
[![GitHub Release](https://img.shields.io/github/release/golang-migrate/migrate.svg)](https://github.com/golang-migrate/migrate/releases)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate/v4)](https://goreportcard.com/report/github.com/golang-migrate/migrate/v4)
[![Go Report Card](https://goreportcard.com/badge/github.com/golang-migrate/migrate)](https://goreportcard.com/report/github.com/golang-migrate/migrate)
# migrate
@@ -26,30 +24,24 @@ Forked from [mattes/migrate](https://github.com/mattes/migrate)
Database drivers run migrations. [Add a new database?](database/driver.go)
* [PostgreSQL](database/postgres)
* [PGX v4](database/pgx)
* [PGX v5](database/pgx/v5)
* [Redshift](database/redshift)
* [Ql](database/ql)
* [Cassandra / ScyllaDB](database/cassandra)
* [SQLite](database/sqlite)
* [SQLite3](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [SQLCipher](database/sqlcipher)
* [MySQL / MariaDB](database/mysql)
* [Neo4j](database/neo4j)
* [Cassandra](database/cassandra)
* [SQLite](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [MySQL/ MariaDB](database/mysql)
* [Neo4j](database/neo4j) ([todo #167](https://github.com/mattes/migrate/issues/167))
* [MongoDB](database/mongodb)
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
* [Google Cloud Spanner](database/spanner)
* [CockroachDB](database/cockroachdb)
* [YugabyteDB](database/yugabytedb)
* [Datastore](database/datastore)
* [Firebird](database/firebird)
* [ClickHouse](database/clickhouse)
* [Firebird](database/firebird) ([todo #49](https://github.com/golang-migrate/migrate/issues/49))
* [MS SQL Server](database/sqlserver)
* [rqlite](database/rqlite)
### Database URLs
Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: `dbdriver://username:password@host:port/dbname?param1=true&param2=false`
Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: `dbdriver://username:password@host:port/dbname?option1=true&option2=false`
Any [reserved URL characters](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters) need to be escaped. Note, the `%` character also [needs to be escaped](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_the_percent_character)
@@ -73,12 +65,9 @@ $
Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go)
* [Filesystem](source/file) - read from filesystem
* [io/fs](source/iofs) - read from a Go [io/fs](https://pkg.go.dev/io/fs#FS)
* [Go-Bindata](source/go_bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata))
* [pkger](source/pkger) - read from embedded binary data ([markbates/pkger](https://github.com/markbates/pkger))
* [GitHub](source/github) - read from remote GitHub repositories
* [GitHub Enterprise](source/github_ee) - read from remote GitHub Enterprise repositories
* [Bitbucket](source/bitbucket) - read from remote Bitbucket repositories
* [Github](source/github) - read from remote Github repositories
* [Github Enterprise](source/github_ee) - read from remote Github Enterprise repositories
* [Gitlab](source/gitlab) - read from remote Gitlab repositories
* [AWS S3](source/aws_s3) - read from Amazon Web Services S3
* [Google Cloud Storage](source/google_cloud_storage) - read from Google Cloud Platform Storage
@@ -89,7 +78,7 @@ Source drivers read migrations from local or remote sources. [Add a new source?]
* Handles ctrl+c (SIGINT) gracefully.
* No config search paths, no config files, no magic ENV var injections.
[CLI Documentation](cmd/migrate) (includes CLI install instructions)
__[CLI Documentation](cmd/migrate)__
### Basic usage
@@ -113,7 +102,7 @@ $ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
* Uses `io.Reader` streams internally for low memory overhead.
* Thread-safe and no goroutine leaks.
__[Go Documentation](https://pkg.go.dev/github.com/golang-migrate/migrate/v4)__
__[Go Documentation](https://godoc.org/github.com/golang-migrate/migrate)__
```go
import (
@@ -147,7 +136,7 @@ func main() {
m, err := migrate.NewWithDatabaseInstance(
"file:///migrations",
"postgres", driver)
m.Up() // or m.Steps(2) if you want to explicitly set the number of migrations to run
m.Steps(2)
}
```
@@ -157,8 +146,7 @@ Go to [getting started](GETTING_STARTED.md)
## Tutorials
* [CockroachDB](database/cockroachdb/TUTORIAL.md)
* [PostgreSQL](database/postgres/TUTORIAL.md)
- [PostgreSQL](database/postgres/TUTORIAL.md)
(more tutorials to come)
@@ -173,11 +161,6 @@ Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-file
[Best practices: How to write migrations.](MIGRATIONS.md)
## Coming from another db migration tool?
Check out [migradaptor](https://github.com/musinit/migradaptor/).
*Note: migradaptor is not affiliated or supported by this project*
## Versions
Version | Supported? | Import | Notes
-16
View File
@@ -1,16 +0,0 @@
# Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| master | :white_check_mark: |
| 4.x | :white_check_mark: |
| 3.x | :x: |
| < 3.0 | :x: |
## Reporting a Vulnerability
We prefer [coordinated disclosures](https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure). To start one, create a GitHub security advisory following [these instructions](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing/privately-reporting-a-security-vulnerability)
Please suggest potential impact and urgency in your reports.
+12 -31
View File
@@ -7,7 +7,7 @@
[Release Downloads](https://github.com/golang-migrate/migrate/releases)
```bash
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$os-$arch.tar.gz | tar xvz
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
```
### MacOS
@@ -16,19 +16,11 @@ $ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/m
$ brew install golang-migrate
```
### Windows
Using [scoop](https://scoop.sh/)
```bash
$ scoop install migrate
```
### Linux (*.deb package)
```bash
$ curl -fsSL https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo gpg --dearmor -o /etc/apt/keyrings/migrate.gpg
$ echo "deb [signed-by=/etc/apt/keyrings/migrate.gpg] https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
$ echo "deb https://packagecloud.io/golang-migrate/migrate/ubuntu/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/migrate.list
$ apt-get update
$ apt-get install -y migrate
```
@@ -41,19 +33,13 @@ $ apt-get install -y migrate
$ go get -u -d github.com/golang-migrate/migrate/cmd/migrate
$ cd $GOPATH/src/github.com/golang-migrate/migrate/cmd/migrate
$ git checkout $TAG # e.g. v4.1.0
$ # Go 1.15 and below
$ go build -tags 'postgres' -ldflags="-X main.Version=$(git describe --tags)" -o $GOPATH/bin/migrate $GOPATH/src/github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.16+
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@$TAG
$ go build -tags 'postgres' -ldflags="-X main.Version=$(git describe --tags)" -o $GOPATH/bin/migrate github.com/golang-migrate/migrate/cmd/migrate
```
#### Unversioned
```bash
$ # Go 1.15 and below
$ go get -tags 'postgres' -u github.com/golang-migrate/migrate/cmd/migrate
$ # Go 1.16+
$ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
```
#### Notes
@@ -63,10 +49,9 @@ $ go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@l
to build the cli for use with other databases, replace the `postgres` build tag
with the appropriate database tag(s) for the databases desired. The tags
correspond to the names of the sub-packages underneath the
[`database`](../../database) package.
[`database`](../database) package.
1. Similarly to the database build tags, if you need to support other sources, use the appropriate build tag(s).
1. Support for build constraints will be removed in the future: https://github.com/golang-migrate/migrate/issues/60
1. For versions of Go 1.15 and lower, [make sure](https://github.com/golang-migrate/migrate/pull/257#issuecomment-705249902) you're not installing the `migrate` CLI from a module. e.g. there should not be any `go.mod` files in your current directory or any directory from your current directory to the root
## Usage
@@ -86,18 +71,14 @@ Options:
-help Print usage
Commands:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] [-tz] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string. Note: migrations with the same time cause "duplicate migration version" error.
Use -tz option to specify the timezone that will be used when generating non-sequential migrations (defaults: UTC).
create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string.
goto V Migrate to version V
up [N] Apply all or N up migrations
down [N] [-all] Apply all or N down migrations
Use -all to apply all down migrations
drop [-f] Drop everything inside database
Use -f to bypass confirmation
down [N] Apply all or N down migrations
drop Drop everything inside database
force V Set version V but don't run migration (ignores dirty state)
version Print current migration version
```
@@ -131,7 +112,7 @@ $ migrate -database "$MY_MIGRATE_DATABASE"
Check out https://stedolan.github.io/jq/
```bash
$ migrate -database "$(cat config.json | jq -r '.database')"
$ migrate -database "$(cat config.json | jq '.database')"
```
### YAML files
+4 -11
View File
@@ -1,17 +1,12 @@
# Cassandra / ScyllaDB
# Cassandra
* `Drop()` method will not work on Cassandra 2.X because it rely on
* Drop command will not work on Cassandra 2.X because it rely on
system_schema table which comes with 3.X
* Other methods should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* Other commands should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
**ScyllaDB**
* No additional configuration is required since it is a drop-in replacement for Cassandra.
* The `Drop()` method` works for ScyllaDB 5.1
## Usage
`cassandra://host:port/keyspace?param1=value&param2=value2`
@@ -25,14 +20,12 @@ system_schema table which comes with 3.X
| `consistency` | ALL | Migration consistency
| `protocol` | | Cassandra protocol version (3 or 4)
| `timeout` | 1 minute | Migration timeout
| `connect-timeout` | 600ms | Initial connection timeout to the cluster |
| `username` | nil | Username to use when authenticating. |
| `password` | nil | Password to use when authenticating. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
| `disable-host-lookup`| false | Disable initial host lookup. |
`timeout` is parsed using [time.ParseDuration(s string)](https://golang.org/pkg/time/#ParseDuration)
+44 -99
View File
@@ -4,15 +4,15 @@ import (
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/gocql/gocql"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/hashicorp/go-multierror"
)
func init() {
@@ -20,12 +20,6 @@ func init() {
database.Register("cassandra", db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var DefaultMigrationsTable = "schema_migrations"
var (
@@ -39,12 +33,11 @@ type Config struct {
MigrationsTable string
KeyspaceName string
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Cassandra struct {
session *gocql.Session
isLocked atomic.Bool
isLocked bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -65,10 +58,6 @@ func WithInstance(session *gocql.Session, config *Config) (database.Driver, erro
config.MigrationsTable = DefaultMigrationsTable
}
if config.MultiStatementMaxSize <= 0 {
config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
c := &Cassandra{
session: session,
config: config,
@@ -131,42 +120,17 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
}
cluster.Timeout = timeout
}
if len(u.Query().Get("connect-timeout")) > 0 {
var connectTimeout time.Duration
connectTimeout, err = time.ParseDuration(u.Query().Get("connect-timeout"))
if err != nil {
return nil, err
}
cluster.ConnectTimeout = connectTimeout
}
if len(u.Query().Get("sslmode")) > 0 {
if len(u.Query().Get("sslmode")) > 0 && len(u.Query().Get("sslrootcert")) > 0 && len(u.Query().Get("sslcert")) > 0 && len(u.Query().Get("sslkey")) > 0 {
if u.Query().Get("sslmode") != "disable" {
sslOpts := &gocql.SslOptions{}
if len(u.Query().Get("sslrootcert")) > 0 {
sslOpts.CaPath = u.Query().Get("sslrootcert")
cluster.SslOpts = &gocql.SslOptions{
CaPath: u.Query().Get("sslrootcert"),
CertPath: u.Query().Get("sslcert"),
KeyPath: u.Query().Get("sslkey"),
}
if len(u.Query().Get("sslcert")) > 0 {
sslOpts.CertPath = u.Query().Get("sslcert")
}
if len(u.Query().Get("sslkey")) > 0 {
sslOpts.KeyPath = u.Query().Get("sslkey")
}
if u.Query().Get("sslmode") == "verify-full" {
sslOpts.EnableHostVerification = true
cluster.SslOpts.EnableHostVerification = true
}
cluster.SslOpts = sslOpts
}
}
if len(u.Query().Get("disable-host-lookup")) > 0 {
if flag, err := strconv.ParseBool(u.Query().Get("disable-host-lookup")); err != nil && flag {
cluster.DisableInitialHostLookup = true
} else if err != nil {
return nil, err
}
}
@@ -175,19 +139,10 @@ func (c *Cassandra) Open(url string) (database.Driver, error) {
return nil, err
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := u.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
return WithInstance(session, &Config{
KeyspaceName: strings.TrimPrefix(u.Path, "/"),
MigrationsTable: u.Query().Get("x-migrations-table"),
MultiStatementEnabled: u.Query().Get("x-multi-statement") == "true",
MultiStatementMaxSize: multiStatementMaxSize,
})
}
@@ -197,44 +152,44 @@ func (c *Cassandra) Close() error {
}
func (c *Cassandra) Lock() error {
if !c.isLocked.CompareAndSwap(false, true) {
if c.isLocked {
return database.ErrLocked
}
c.isLocked = true
return nil
}
func (c *Cassandra) Unlock() error {
if !c.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
c.isLocked = false
return nil
}
func (c *Cassandra) Run(migration io.Reader) error {
if c.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, c.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if e := c.session.Query(tq).Exec(); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
// run migration
if err := c.session.Query(string(migr)).Exec(); err != nil {
query := string(migr[:])
if c.config.MultiStatementEnabled {
// split query by semi-colon
queries := strings.Split(query, ";")
for _, q := range queries {
tq := strings.TrimSpace(q)
if tq == "" {
continue
}
if err := c.session.Query(tq).Exec(); err != nil {
// TODO: cast to Cassandra error and get line number
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
}
return nil
}
if err := c.session.Query(query).Exec(); err != nil {
// TODO: cast to Cassandra error and get line number
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
@@ -242,26 +197,12 @@ func (c *Cassandra) Run(migration io.Reader) error {
}
func (c *Cassandra) SetVersion(version int, dirty bool) error {
// DELETE instead of TRUNCATE because AWS Keyspaces does not support it
// see: https://docs.aws.amazon.com/keyspaces/latest/devguide/cassandra-apis.html
squery := `SELECT version FROM "` + c.config.MigrationsTable + `"`
dquery := `DELETE FROM "` + c.config.MigrationsTable + `" WHERE version = ?`
iter := c.session.Query(squery).Iter()
var previous int
for iter.Scan(&previous) {
if err := c.session.Query(dquery, previous).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(dquery)}
}
query := `TRUNCATE "` + c.config.MigrationsTable + `"`
if err := c.session.Query(query).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if err := iter.Close(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(squery)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query := `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
if version >= 0 {
query = `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
if err := c.session.Query(query, version, dirty).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -314,7 +255,11 @@ func (c *Cassandra) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -336,7 +281,7 @@ func parseConsistency(consistencyStr string) (consistency gocql.Consistency, err
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("failed to parse consistency \"%s\": %v", consistencyStr, r)
err = fmt.Errorf("Failed to parse consistency \"%s\": %v", consistencyStr, r)
}
}
}()
+2 -18
View File
@@ -25,11 +25,9 @@ var (
// Although Cassandra 2.x is supported by the Apache Foundation,
// the migrate db driver only supports Cassandra 3.x since it uses
// the system_schema keyspace.
// last ScyllaDB version tested is 5.1.11
specs = []dktesting.ContainerSpec{
{ImageName: "cassandra:3.0", Options: opts},
{ImageName: "cassandra:3.11", Options: opts},
{ImageName: "scylladb/scylla:5.1.11", Options: opts},
}
)
@@ -61,20 +59,6 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
@@ -95,7 +79,7 @@ func test(t *testing.T) {
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(9042)
if err != nil {
@@ -117,6 +101,6 @@ func testMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("SELECT table_name from system_schema.tables"))
})
}
+19
View File
@@ -0,0 +1,19 @@
# ClickHouse
`clickhouse://host:port?username=user&password=qwerty&database=clicks&x-multi-statement=true`
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
| `x-multi-statement` | false | Enable multiple statements to be ran in a single migration (See note below) |
## Notes
* The Clickhouse driver does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
+236
View File
@@ -0,0 +1,236 @@
package clickhouse
import (
"database/sql"
"fmt"
"io"
"io/ioutil"
"net/url"
"strings"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
)
var DefaultMigrationsTable = "schema_migrations"
var ErrNilConfig = fmt.Errorf("no config")
type Config struct {
DatabaseName string
MigrationsTable string
MultiStatementEnabled bool
}
func init() {
database.Register("clickhouse", &ClickHouse{})
}
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ch := &ClickHouse{
conn: conn,
config: config,
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
type ClickHouse struct {
conn *sql.DB
config *Config
}
func (ch *ClickHouse) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = "tcp"
conn, err := sql.Open("clickhouse", q.String())
if err != nil {
return nil, err
}
ch = &ClickHouse{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
DatabaseName: purl.Query().Get("database"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
},
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
func (ch *ClickHouse) init() error {
if len(ch.config.DatabaseName) == 0 {
if err := ch.conn.QueryRow("SELECT currentDatabase()").Scan(&ch.config.DatabaseName); err != nil {
return err
}
}
if len(ch.config.MigrationsTable) == 0 {
ch.config.MigrationsTable = DefaultMigrationsTable
}
return ch.ensureVersionTable()
}
func (ch *ClickHouse) Run(r io.Reader) error {
migration, err := ioutil.ReadAll(r)
if err != nil {
return err
}
if ch.config.MultiStatementEnabled {
// split query by semi-colon
queries := strings.Split(string(migration), ";")
for _, q := range queries {
tq := strings.TrimSpace(q)
if tq == "" {
continue
}
if _, err := ch.conn.Exec(q); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: []byte(q)}
}
}
return nil
}
if _, err := ch.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ch *ClickHouse) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ch.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ch.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ch *ClickHouse) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ch.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ch.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the ClickHouse type.
func (ch *ClickHouse) ensureVersionTable() (err error) {
if err = ch.Lock(); err != nil {
return err
}
defer func() {
if e := ch.Unlock(); e != nil {
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
var (
table string
query = "SHOW TABLES FROM " + ch.config.DatabaseName + " LIKE '" + ch.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ch.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
query = `
CREATE TABLE ` + ch.config.MigrationsTable + ` (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=TinyLog
`
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ch *ClickHouse) Drop() (err error) {
query := "SHOW TABLES FROM " + ch.config.DatabaseName
tables, err := ch.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = multierror.Append(err, errClose)
}
}()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + ch.config.DatabaseName + "." + table
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
return nil
}
func (ch *ClickHouse) Lock() error { return nil }
func (ch *ClickHouse) Unlock() error { return nil }
func (ch *ClickHouse) Close() error { return ch.conn.Close() }
@@ -1,3 +1,3 @@
CREATE TABLE test_1 (
Date Date
Date Date
) Engine=Memory;
@@ -1,3 +1,3 @@
CREATE TABLE test_2 (
Date Date
Date Date
) Engine=Memory;
-142
View File
@@ -1,142 +0,0 @@
# CockroachDB tutorial for beginners (insecure cluster)
## Create/configure database
First, let's start a local cluster - follow step 1. and 2. from [the docs](https://www.cockroachlabs.com/docs/stable/start-a-local-cluster.html#step-1-start-the-first-node).
Once you have it, create a database. Here I am going to create a database called `example`.
Our user here is `cockroach`. We are not going to use a password, since it's not supported for insecure cluster.
```
cockroach sql --insecure --host=localhost:26257
```
```
CREATE DATABASE example;
CREATE USER IF NOT EXISTS cockroach;
GRANT ALL ON DATABASE example TO cockroach;
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
```
export COCKROACHDB_URL='cockroachdb://cockroach:@localhost:26257/example?sslmode=disable'
```
`sslmode=disable` means that the connection with our database will not be encrypted. This is needed to connect to an insecure node.
**NOTE:** Do not use COCKROACH_URL as a variable name here, it's already in use for discrete parameters and you may run into connection problems. For more info check out [docs](https://www.cockroachlabs.com/docs/stable/connection-parameters.html#connect-using-discrete-parameters).
You can find further description of database URLs [here](README.md#database-urls).
## Create migrations
Let's create a table called `users`:
```
migrate create -ext sql -dir db/migrations -seq create_users_table
```
If there were no errors, we should have two files available under `db/migrations` folder:
- 000001_create_users_table.down.sql
- 000001_create_users_table.up.sql
Note the `sql` extension that we provided.
In the `.up.sql` file let's create the table:
```
CREATE TABLE IF NOT EXISTS example.users
(
user_id INT PRIMARY KEY,
username VARCHAR (50) UNIQUE NOT NULL,
password VARCHAR (50) NOT NULL,
email VARCHAR (300) UNIQUE NOT NULL
);
```
And in the `.down.sql` let's delete it:
```
DROP TABLE IF EXISTS example.users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](/GETTING_STARTED.md#create-migrations)
## Run migrations
```
migrate -database ${COCKROACHDB_URL} -path db/migrations up
```
Let's check if the table was created properly by running `cockroach sql --insecure --host=localhost:26257 -e "show columns from example.users;"`.
The output you are supposed to see:
```
column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden
+-------------+--------------+-------------+----------------+-----------------------+----------------------------------------------+-----------+
user_id | INT8 | false | NULL | | {primary,users_username_key,users_email_key} | false
username | VARCHAR(50) | false | NULL | | {users_username_key} | false
password | VARCHAR(50) | false | NULL | | {} | false
email | VARCHAR(300) | false | NULL | | {users_email_key} | false
(4 rows)
```
Now let's check if running reverse migration also works:
```
migrate -database ${COCKROACHDB_URL} -path db/migrations down
```
Make sure to check if your database changed as expected in this case as well.
## Database transactions
To show database transactions usage, let's create another set of migrations by running:
```
migrate create -ext sql -dir db/migrations -seq add_mood_to_users
```
Again, it should create for us two migrations files:
- 000002_add_mood_to_users.down.sql
- 000002_add_mood_to_users.up.sql
In Cockroach, when we want our queries to be done in a transaction, we need to wrap it with `BEGIN` and `COMMIT` commands, similar to PostgreSQL.
In our example, we are going to add a column to our database that can only accept enumerable values or NULL.
Migration up:
```
BEGIN;
ALTER TABLE example.users ADD COLUMN mood STRING;
ALTER TABLE example.users ADD CONSTRAINT check_mood CHECK (mood IN ('happy', 'sad', 'neutral'));
COMMIT;
```
Migration down:
```
ALTER TABLE example.users DROP COLUMN mood;
```
Now we can run our new migration and check the database:
```
migrate -database ${COCKROACHDB_URL} -path db/migrations up
cockroach sql --insecure --host=localhost:26257 -e "show columns from example.users;"
```
Expected output:
```
column_name | data_type | is_nullable | column_default | generation_expression | indices | is_hidden
+-------------+--------------+-------------+----------------+-----------------------+----------------------------------------------+-----------+
user_id | INT8 | false | NULL | | {primary,users_username_key,users_email_key} | false
username | VARCHAR(50) | false | NULL | | {users_username_key} | false
password | VARCHAR(50) | false | NULL | | {} | false
email | VARCHAR(300) | false | NULL | | {users_email_key} | false
mood | STRING | true | NULL | | {} | false
(5 rows)
```
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
import (
"log"
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/cockroachdb"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
m, err := migrate.New(
"file://db/migrations",
"cockroachdb://cockroach:@localhost:26257/example?sslmode=disable")
if err != nil {
log.Fatal(err)
}
if err := m.Up(); err != nil {
log.Fatal(err)
}
}
```
You can find details [here](README.md#use-in-your-go-project)
+82 -77
View File
@@ -3,18 +3,23 @@ package cockroachdb
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"sync/atomic"
)
"github.com/cockroachdb/cockroach-go/v2/crdb"
import (
"github.com/cockroachdb/cockroach-go/crdb"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
import (
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/lib/pq"
)
func init() {
@@ -41,7 +46,7 @@ type Config struct {
type CockroachDb struct {
db *sql.DB
isLocked atomic.Bool
isLocked bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -56,20 +61,18 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT current_database()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
query := `SELECT current_database()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
@@ -147,71 +150,75 @@ func (c *CockroachDb) Close() error {
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Lock() error {
return database.CasRestoreOnErr(&c.isLocked, false, true, database.ErrLocked, func() (err error) {
return crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return nil
})
})
}
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Unlock() error {
return database.CasRestoreOnErr(&c.isLocked, true, false, database.ErrNotLocked, func() (err error) {
err := crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) (err error) {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
return nil
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = multierror.Append(err, errClose)
}
}()
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return nil
})
if err != nil {
return err
} else {
c.isLocked = true
return nil
}
}
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Unlock() error {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
c.isLocked = false
return nil
}
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
c.isLocked = false
return nil
}
func (c *CockroachDb) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -231,10 +238,7 @@ func (c *CockroachDb) SetVersion(version int, dirty bool) error {
return err
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
if version >= 0 {
if _, err := tx.Exec(`INSERT INTO "`+c.config.MigrationsTable+`" (version, dirty) VALUES ($1, $2)`, version, dirty); err != nil {
return err
}
@@ -276,7 +280,7 @@ func (c *CockroachDb) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -291,9 +295,6 @@ func (c *CockroachDb) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -318,7 +319,11 @@ func (c *CockroachDb) ensureVersionTable() (err error) {
defer func() {
if e := c.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+7 -7
View File
@@ -26,13 +26,13 @@ import (
const defaultPort = 26257
var (
opts = dktest.Options{Cmd: []string{"start-single-node", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.cockroachlabs.com/docs/releases/release-support-policy#supported-versions
opts = dktest.Options{Cmd: []string{"start", "--insecure"}, PortRequired: true, ReadyFunc: isReady}
// Released versions: https://www.cockroachlabs.com/docs/releases/
specs = []dktesting.ContainerSpec{
{ImageName: "cockroachdb/cockroach:latest-v24.3", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v24.1", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.2", Options: opts},
{ImageName: "cockroachdb/cockroach:latest-v23.1", Options: opts},
{ImageName: "cockroachdb/cockroach:v1.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v1.1.9", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.0.7", Options: opts},
{ImageName: "cockroachdb/cockroach:v2.1.3", Options: opts},
}
)
@@ -121,7 +121,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("SELECT 1"))
})
}
-31
View File
@@ -1,31 +0,0 @@
# Datastore
`datastore://host:port?username=user&password=password&database=clicks&x-multi-statement=true`
Hanzo Datastore is a ClickHouse-wire-compatible database engine. This driver
connects through the native `github.com/hanzo-ds/go` driver, which registers the
`database/sql` driver name `datastore`.
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `x-migrations-table-engine`| Engine to use for the migrations table, defaults to TinyLog |
| `x-cluster-name` | Name of cluster for creating `schema_migrations` table cluster wide |
| `x-multi-statement` | Enable multiple statements to be ran in a single migration (See note below) |
| `x-multi-statement-max-size` | Maximum size of a single migration in bytes when `x-multi-statement` is set (defaults to 10 MB) |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
## Notes
* The Datastore driver does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
* Using the default TinyLog table engine for the `schema_migrations` table prevents backing up the table. If you need to back up the database, run the migrations with `x-migrations-table-engine=MergeTree`.
* Datastore cluster mode is not officially supported, but you can try enabling `schema_migrations` table replication by specifying a `x-cluster-name`:
* When `x-cluster-name` is specified, `x-migrations-table-engine` also should be specified.
* When `x-cluster-name` is specified, only the `schema_migrations` table is replicated across the cluster. You still need to write your migrations so that the application tables are replicated within the cluster.
* If you want to create a database inside a migration, note that the `schema_migrations` table will live in the `default` database, so you cannot use `USE <database_name>` inside a migration. In this case you may omit the database in the connection string (example [here](examples/migrations/003_create_database.up.sql)).
-337
View File
@@ -1,337 +0,0 @@
// Package datastore implements the golang-migrate database.Driver interface for
// Hanzo Datastore (a ClickHouse-wire-compatible database engine). It registers
// the "datastore://" URL scheme and connects through the native
// github.com/hanzo-ds/go driver, which registers the database/sql driver name
// "datastore".
//
// Usage:
//
// import _ "github.com/golang-migrate/migrate/v4/database/datastore"
//
// Then use a URL like:
//
// datastore://host:9000?username=default&database=default&x-multi-statement=true
package datastore
import (
"database/sql"
"errors"
"fmt"
"io"
"net/url"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
var (
multiStmtDelimiter = []byte(";")
// DriverName is the database/sql driver registered by github.com/hanzo-ds/go.
DriverName = "datastore"
DefaultMigrationsTable = "schema_migrations"
DefaultMigrationsTableEngine = "TinyLog"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
ErrNilConfig = fmt.Errorf("no config")
)
type Config struct {
DatabaseName string
ClusterName string
MigrationsTable string
MigrationsTableEngine string
MultiStatementEnabled bool
MultiStatementMaxSize int
}
func init() {
database.Register("datastore", &Datastore{})
}
// WithInstance creates a Datastore driver from an existing *sql.DB opened
// against the "datastore" driver.
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ds := &Datastore{
conn: conn,
config: config,
}
if err := ds.init(); err != nil {
return nil, err
}
return ds, nil
}
// Datastore is a golang-migrate database.Driver backed by the native
// github.com/hanzo-ds/go ClickHouse-wire driver.
type Datastore struct {
conn *sql.DB
config *Config
isLocked atomic.Bool
}
func (ds *Datastore) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = DriverName
conn, err := sql.Open(DriverName, q.String())
if err != nil {
return nil, err
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
migrationsTableEngine := DefaultMigrationsTableEngine
if s := purl.Query().Get("x-migrations-table-engine"); len(s) > 0 {
migrationsTableEngine = s
}
ds = &Datastore{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
MigrationsTableEngine: migrationsTableEngine,
DatabaseName: purl.Query().Get("database"),
ClusterName: purl.Query().Get("x-cluster-name"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
MultiStatementMaxSize: multiStatementMaxSize,
},
}
if err := ds.init(); err != nil {
return nil, err
}
return ds, nil
}
func (ds *Datastore) init() error {
if len(ds.config.DatabaseName) == 0 {
if err := ds.conn.QueryRow("SELECT currentDatabase()").Scan(&ds.config.DatabaseName); err != nil {
return err
}
}
if len(ds.config.MigrationsTable) == 0 {
ds.config.MigrationsTable = DefaultMigrationsTable
}
if ds.config.MultiStatementMaxSize <= 0 {
ds.config.MultiStatementMaxSize = DefaultMultiStatementMaxSize
}
if len(ds.config.MigrationsTableEngine) == 0 {
ds.config.MigrationsTableEngine = DefaultMigrationsTableEngine
}
return ds.ensureVersionTable()
}
func (ds *Datastore) Run(r io.Reader) error {
if ds.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(r, multiStmtDelimiter, ds.config.MultiStatementMaxSize, func(m []byte) bool {
tq := strings.TrimSpace(string(m))
if tq == "" {
return true
}
if _, e := ds.conn.Exec(string(m)); e != nil {
err = database.Error{OrigErr: e, Err: "migration failed", Query: m}
return false
}
return true
}); e != nil {
return e
}
return err
}
migration, err := io.ReadAll(r)
if err != nil {
return err
}
if _, err := ds.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ds *Datastore) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ds.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ds.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ds *Datastore) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ds.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ds.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Datastore type.
func (ds *Datastore) ensureVersionTable() (err error) {
if err = ds.Lock(); err != nil {
return err
}
defer func() {
if e := ds.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
var (
table string
query = "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName) + " LIKE '" + ds.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ds.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
if len(ds.config.ClusterName) > 0 {
query = fmt.Sprintf(`
CREATE TABLE %s ON CLUSTER %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.ClusterName, ds.config.MigrationsTableEngine)
} else {
query = fmt.Sprintf(`
CREATE TABLE %s (
version Int64,
dirty UInt8,
sequence UInt64
) Engine=%s`, ds.config.MigrationsTable, ds.config.MigrationsTableEngine)
}
if strings.HasSuffix(ds.config.MigrationsTableEngine, "Tree") {
query = fmt.Sprintf(`%s ORDER BY sequence`, query)
}
if _, err := ds.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Drop() (err error) {
query := "SHOW TABLES FROM " + quoteIdentifier(ds.config.DatabaseName)
tables, err := ds.conn.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + quoteIdentifier(ds.config.DatabaseName) + "." + quoteIdentifier(table)
if _, err := ds.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ds *Datastore) Lock() error {
if !ds.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
func (ds *Datastore) Unlock() error {
if !ds.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (ds *Datastore) Close() error { return ds.conn.Close() }
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
-226
View File
@@ -1,226 +0,0 @@
package datastore_test
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"fmt"
"log"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/datastore"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
// Registers the "datastore" database/sql driver (ClickHouse-wire native).
_ "github.com/hanzo-ds/go"
)
const defaultPort = 9000
var (
tableEngines = []string{"TinyLog", "MergeTree"}
opts = dktest.Options{
Env: map[string]string{"CLICKHOUSE_USER": "user", "CLICKHOUSE_PASSWORD": "password", "CLICKHOUSE_DB": "db"},
PortRequired: true, ReadyFunc: isReady,
}
specs = []dktesting.ContainerSpec{
{ImageName: "clickhouse:24.8", Options: opts},
}
)
func datastoreConnectionString(host, port, engine string) string {
if engine != "" {
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&x-migrations-table-engine=%v&debug=false",
host, port, engine)
}
return fmt.Sprintf(
"datastore://%v:%v?username=user&password=password&database=db&x-multi-statement=true&debug=false",
host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
return false
}
db, err := sql.Open("datastore", datastoreConnectionString(ip, port, ""))
if err != nil {
log.Println("open error", err)
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn:
return false
default:
fmt.Println(err)
}
return false
}
return true
}
func TestCases(t *testing.T) {
for _, engine := range tableEngines {
t.Run("Test_"+engine, func(t *testing.T) { testSimple(t, engine) })
t.Run("Migrate_"+engine, func(t *testing.T) { testMigrate(t, engine) })
t.Run("Version_"+engine, func(t *testing.T) { testVersion(t, engine) })
t.Run("Drop_"+engine, func(t *testing.T) { testDrop(t, engine) })
}
t.Run("WithInstanceDefaultConfigValues", func(t *testing.T) { testSimpleWithInstanceDefaultConfigValues(t) })
}
func testSimple(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testSimpleWithInstanceDefaultConfigValues(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, "")
conn, err := sql.Open("datastore", addr)
if err != nil {
t.Fatal(err)
}
d, err := datastore.WithInstance(conn, &datastore.Config{})
if err != nil {
_ = conn.Close()
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func testMigrate(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "db", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func testVersion(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
expectedVersion := 1
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
err = d.SetVersion(expectedVersion, false)
if err != nil {
t.Fatal(err)
}
version, _, err := d.Version()
if err != nil {
t.Fatal(err)
}
if version != expectedVersion {
t.Fatal("Version mismatch")
}
})
}
func testDrop(t *testing.T, engine string) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := datastoreConnectionString(ip, port, engine)
p := &datastore.Datastore{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
err = d.Drop()
if err != nil {
t.Fatal(err)
}
})
}
@@ -1,10 +0,0 @@
DROP TABLE IF EXISTS driver_ratings;
DROP TABLE IF EXISTS user_ratings;
DROP TABLE IF EXISTS orders;
DROP TABLE IF EXISTS driver_ratings_queue;
DROP TABLE IF EXISTS user_ratings_queue;
DROP TABLE IF EXISTS orders_queue;
DROP VIEW IF EXISTS user_ratings_queue_mv;
DROP VIEW IF EXISTS driver_ratings_queue_mv;
DROP VIEW IF EXISTS orders_queue_mv;
DROP DATABASE IF EXISTS analytics;
@@ -1,81 +0,0 @@
CREATE DATABASE IF NOT EXISTS analytics;
CREATE TABLE IF NOT EXISTS analytics.driver_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.driver_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'driver-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.driver_ratings_queue_mv TO analytics.driver_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.driver_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.user_ratings(
rate UInt8,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY userID
ORDER BY (inserted_time);
CREATE TABLE analytics.user_ratings_queue(
rate UInt8,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'user-ratings',
kafka_group_name = 'rating_readers',
kafka_format = 'JSON',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.user_ratings_queue_mv TO analytics.user_ratings AS
SELECT rate, userID, driverID, orderID
FROM analytics.user_ratings_queue;
CREATE TABLE IF NOT EXISTS analytics.orders(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String,
inserted_time DateTime DEFAULT now()
) ENGINE = MergeTree
PARTITION BY driverID
ORDER BY (inserted_time);
CREATE TABLE analytics.orders_queue(
from_place String,
to_place String,
userID Int64,
driverID String,
orderID String
) ENGINE = Kafka
SETTINGS kafka_broker_list = 'broker:9092',
kafka_topic_list = 'orders',
kafka_group_name = 'order_readers',
kafka_format = 'Avro',
kafka_max_block_size = 1048576;
CREATE MATERIALIZED VIEW analytics.orders_queue_mv TO orders AS
SELECT from_place, to_place, userID, driverID, orderID
FROM analytics.orders_queue;
+16 -17
View File
@@ -1,4 +1,4 @@
// Package database provides the Driver interface.
// Package database provides the Database interface.
// All database drivers must implement this interface, register themselves,
// optionally provide a `WithInstance` function and pass the tests
// in package database/testing.
@@ -13,8 +13,7 @@ import (
)
var (
ErrLocked = fmt.Errorf("can't acquire lock")
ErrNotLocked = fmt.Errorf("can't unlock, as not currently locked")
ErrLocked = fmt.Errorf("can't acquire lock")
)
const NilVersion int = -1
@@ -25,22 +24,22 @@ var drivers = make(map[string]Driver)
// Driver is the interface every database driver must implement.
//
// How to implement a database driver?
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a internal/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a migrate/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
//
// Guidelines:
// - Don't try to correct user input. Don't assume things.
// * Don't try to correct user input. Don't assume things.
// When in doubt, return an error and explain the situation to the user.
// - All configuration input must come from the URL string in func Open()
// * All configuration input must come from the URL string in func Open()
// or the Config{} struct in WithInstance. Don't os.Getenv().
type Driver interface {
// Open returns a new driver instance configured with parameters
@@ -62,7 +61,7 @@ type Driver interface {
// all migrations have been run.
Unlock() error
// Run applies a migration to the database. migration is guaranteed to be not nil.
// Run applies a migration to the database. migration is garantueed to be not nil.
Run(migration io.Reader) error
// SetVersion saves version and dirty state.
+6
View File
@@ -10,3 +10,9 @@
| `role` | | Role name |
| `tzname` | | Time Zone name. (For Firebird 4.0+) |
| `wire_crypt` | | Enable wire data encryption or not. For Firebird 3.0+ (default is true) |
## Not officially supported
If you need a `migrate` CLI that supports Firebird, you'll need need to build it yourself. There are also no guarantees that the integration will work since we're not testing with Firebird.
For more info, see: https://github.com/golang-migrate/migrate/issues/49#issuecomment-487193672
+20 -24
View File
@@ -1,19 +1,18 @@
//go:build go1.9
// +build go1.9
package firebird
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
_ "github.com/nakagami/firebirdsql"
"io"
"io/ioutil"
nurl "net/url"
)
func init() {
@@ -37,7 +36,7 @@ type Firebird struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
isLocked bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
@@ -107,21 +106,20 @@ func (f *Firebird) Close() error {
}
func (f *Firebird) Lock() error {
if !f.isLocked.CompareAndSwap(false, true) {
if f.isLocked {
return database.ErrLocked
}
f.isLocked = true
return nil
}
func (f *Firebird) Unlock() error {
if !f.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
f.isLocked = false
return nil
}
func (f *Firebird) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -136,20 +134,17 @@ func (f *Firebird) Run(migration io.Reader) error {
}
func (f *Firebird) SetVersion(version int, dirty bool) error {
// Always re-write the schema version to prevent empty schema version
// for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version < 0 {
return nil
}
// TODO: parameterize this SQL statement
// https://firebirdsql.org/refdocs/langrefupd20-execblock.html
// VALUES (?, ?) doesn't work
query := fmt.Sprintf(`EXECUTE BLOCK AS BEGIN
DELETE FROM "%v";
INSERT INTO "%v" (version, dirty) VALUES (%v, %v);
END;`,
f.config.MigrationsTable, f.config.MigrationsTable, version, btoi(dirty))
if _, err := f.conn.ExecContext(context.Background(), query); err != nil {
if _, err := f.conn.ExecContext(context.Background(), query, version, btoi(dirty)); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -180,7 +175,7 @@ func (f *Firebird) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -195,9 +190,6 @@ func (f *Firebird) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// delete one by one ...
for _, t := range tableNames {
@@ -223,7 +215,11 @@ func (f *Firebird) ensureVersionTable() (err error) {
defer func() {
if e := f.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+3 -4
View File
@@ -38,9 +38,8 @@ var (
},
}
specs = []dktesting.ContainerSpec{
{ImageName: "jacobalberty/firebird:v3.0", Options: opts},
{ImageName: "jacobalberty/firebird:v4.0", Options: opts},
{ImageName: "jacobalberty/firebird:v5.0", Options: opts},
{ImageName: "jacobalberty/firebird:2.5-ss", Options: opts},
{ImageName: "jacobalberty/firebird:3.0", Options: opts},
}
)
@@ -122,7 +121,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("SELECT Count(*) FROM rdb$relations"))
})
}
+1 -5
View File
@@ -7,16 +7,12 @@
# Usage
`mongodb://user:password@host:port/dbname?query` (`mongodb+srv://` also works, but behaves a bit differently. See [docs](https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format) for more information)
`mongodb://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-collection` | `MigrationsCollection` | Name of the migrations collection |
| `x-transaction-mode` | `TransactionMode` | If set to `true` wrap commands in [transaction](https://docs.mongodb.com/manual/core/transactions). Available only for replica set. Driver is using [strconv.ParseBool](https://golang.org/pkg/strconv/#ParseBool) for parsing|
| `x-advisory-locking` | `true` | Feature flag for advisory locking, if set to false, disable advisory locking |
| `x-advisory-lock-collection` | `migrate_advisory_lock` | The name of the collection to use for advisory locking.|
| `x-advisory-lock-timeout` | `15` | The max time in seconds that migrate will wait to acquire a lock before failing. |
| `x-advisory-lock-timeout-interval` | `10` | The max time in seconds between attempts to acquire the advisory lock, the lock is attempted to be acquired using an exponential backoff algorithm. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as. Can be omitted |
| `password` | | The user's password. Can be omitted |
@@ -1,16 +0,0 @@
[
{
"update": "users",
"updates": [
{
"q": {},
"u": {
"$unset": {
"status": ""
}
},
"multi": true
}
]
}
]
@@ -1,16 +0,0 @@
[
{
"update": "users",
"updates": [
{
"q": {},
"u": {
"$set": {
"status": "active"
}
},
"multi": true
}
]
}
]
@@ -1,14 +0,0 @@
[
{
"update": "users",
"updates": [
{
"q": {},
"u": {
"fullname": ""
},
"multi": true
}
]
}
]
@@ -1,23 +0,0 @@
[
{
"aggregate": "users",
"pipeline": [
{
"$project": {
"_id": 1,
"firstname": 1,
"lastname": 1,
"username": 1,
"password": 1,
"email": 1,
"active": 1,
"fullname": { "$concat": ["$firstname", " ", "$lastname"] }
}
},
{
"$out": "users"
}
],
"cursor": {}
}
]
+17 -226
View File
@@ -2,16 +2,12 @@ package mongodb
import (
"context"
"errors"
"fmt"
"io"
"io/ioutil"
"net/url"
"os"
"strconv"
"sync/atomic"
"time"
"github.com/cenkalti/backoff/v4"
"github.com/golang-migrate/migrate/v4/database"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
@@ -20,61 +16,34 @@ import (
)
func init() {
db := Mongo{}
database.Register("mongodb", &db)
database.Register("mongodb+srv", &db)
database.Register("mongodb", &Mongo{})
}
var DefaultMigrationsCollection = "schema_migrations"
const DefaultLockingCollection = "migrate_advisory_lock" // the collection to use for advisory locking by default.
const lockKeyUniqueValue = 0 // the unique value to lock on. If multiple clients try to insert the same key, it will fail (locked).
const DefaultLockTimeout = 15 // the default maximum time to wait for a lock to be released.
const DefaultLockTimeoutInterval = 10 // the default maximum intervals time for the locking timout.
const DefaultAdvisoryLockingFlag = true // the default value for the advisory locking feature flag. Default is true.
const LockIndexName = "lock_unique_key" // the name of the index which adds unique constraint to the locking_key field.
const contextWaitTimeout = 5 * time.Second // how long to wait for the request to mongo to block/wait for.
var (
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
ErrLockTimeoutConfigConflict = fmt.Errorf("both x-advisory-lock-timeout-interval and x-advisory-lock-timout-interval were specified")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNilConfig = fmt.Errorf("no config")
)
type Mongo struct {
client *mongo.Client
db *mongo.Database
config *Config
isLocked atomic.Bool
client *mongo.Client
db *mongo.Database
config *Config
}
type Locking struct {
CollectionName string
Timeout int
Enabled bool
Interval int
}
type Config struct {
DatabaseName string
MigrationsCollection string
TransactionMode bool
Locking Locking
}
type versionInfo struct {
Version int `bson:"version"`
Dirty bool `bson:"dirty"`
}
type lockObj struct {
Key int `bson:"locking_key"`
Pid int `bson:"pid"`
Hostname string `bson:"hostname"`
CreatedAt time.Time `bson:"created_at"`
}
type findFilter struct {
Key int `bson:"locking_key"`
}
func WithInstance(instance *mongo.Client, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
@@ -85,36 +54,17 @@ func WithInstance(instance *mongo.Client, config *Config) (database.Driver, erro
if len(config.MigrationsCollection) == 0 {
config.MigrationsCollection = DefaultMigrationsCollection
}
if len(config.Locking.CollectionName) == 0 {
config.Locking.CollectionName = DefaultLockingCollection
}
if config.Locking.Timeout <= 0 {
config.Locking.Timeout = DefaultLockTimeout
}
if config.Locking.Interval <= 0 {
config.Locking.Interval = DefaultLockTimeoutInterval
}
mc := &Mongo{
client: instance,
db: instance.Database(config.DatabaseName),
config: config,
}
if mc.config.Locking.Enabled {
if err := mc.ensureLockTable(); err != nil {
return nil, err
}
}
if err := mc.ensureVersionTable(); err != nil {
return nil, err
}
return mc, nil
}
func (m *Mongo) Open(dsn string) (database.Driver, error) {
// connstring is experimental package, but it used for parse connection string in mongo.Connect function
//connsting is experimental package, but it used for parse connection string in mongo.Connect function
uri, err := connstring.Parse(dsn)
if err != nil {
return nil, err
@@ -122,46 +72,16 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
if len(uri.Database) == 0 {
return nil, ErrNoDatabaseName
}
unknown := url.Values(uri.UnknownOptions)
migrationsCollection := unknown.Get("x-migrations-collection")
lockCollection := unknown.Get("x-advisory-lock-collection")
transactionMode, err := parseBoolean(unknown.Get("x-transaction-mode"), false)
if err != nil {
return nil, err
}
advisoryLockingFlag, err := parseBoolean(unknown.Get("x-advisory-locking"), DefaultAdvisoryLockingFlag)
if err != nil {
return nil, err
}
lockingTimout, err := parseInt(unknown.Get("x-advisory-lock-timeout"), DefaultLockTimeout)
if err != nil {
return nil, err
}
transactionMode, _ := strconv.ParseBool(unknown.Get("x-transaction-mode"))
lockTimeoutIntervalValue := unknown.Get("x-advisory-lock-timeout-interval")
// The initial release had a typo for this argument but for backwards compatibility sake, we will keep supporting it
// and we will error out if both values are set.
lockTimeoutIntervalValueFromTypo := unknown.Get("x-advisory-lock-timout-interval")
lockTimeout := lockTimeoutIntervalValue
if lockTimeoutIntervalValue != "" && lockTimeoutIntervalValueFromTypo != "" {
return nil, ErrLockTimeoutConfigConflict
} else if lockTimeoutIntervalValueFromTypo != "" {
lockTimeout = lockTimeoutIntervalValueFromTypo
}
maxLockCheckInterval, err := parseInt(lockTimeout, DefaultLockTimeoutInterval)
if err != nil {
return nil, err
}
client, err := mongo.Connect(context.TODO(), options.Client().ApplyURI(dsn))
if err != nil {
return nil, err
}
if err = client.Ping(context.TODO(), nil); err != nil {
return nil, err
}
@@ -169,12 +89,6 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
DatabaseName: uri.Database,
MigrationsCollection: migrationsCollection,
TransactionMode: transactionMode,
Locking: Locking{
CollectionName: lockCollection,
Timeout: lockingTimout,
Enabled: advisoryLockingFlag,
Interval: maxLockCheckInterval,
},
})
if err != nil {
return nil, err
@@ -182,39 +96,6 @@ func (m *Mongo) Open(dsn string) (database.Driver, error) {
return mc, nil
}
// Parse the url param, convert it to boolean
// returns error if param invalid. returns defaultValue if param not present
func parseBoolean(urlParam string, defaultValue bool) (bool, error) {
// if parameter passed, parse it (otherwise return default value)
if urlParam != "" {
result, err := strconv.ParseBool(urlParam)
if err != nil {
return false, err
}
return result, nil
}
// if no url Param passed, return default value
return defaultValue, nil
}
// Parse the url param, convert it to int
// returns error if param invalid. returns defaultValue if param not present
func parseInt(urlParam string, defaultValue int) (int, error) {
// if parameter passed, parse it (otherwise return default value)
if urlParam != "" {
result, err := strconv.Atoi(urlParam)
if err != nil {
return -1, err
}
return result, nil
}
// if no url Param passed, return default value
return defaultValue, nil
}
func (m *Mongo) SetVersion(version int, dirty bool) error {
migrationsCollection := m.db.Collection(m.config.MigrationsCollection)
if err := migrationsCollection.Drop(context.TODO()); err != nil {
@@ -241,7 +122,7 @@ func (m *Mongo) Version() (version int, dirty bool, err error) {
}
func (m *Mongo) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -268,8 +149,8 @@ func (m *Mongo) executeCommandsWithTransaction(ctx context.Context, cmds []bson.
return &database.Error{OrigErr: err, Err: "failed to start transaction"}
}
if err := m.executeCommands(sessionContext, cmds); err != nil {
// When command execution is failed, it's aborting transaction
// If you tried to call abortTransaction, it`s return error that transaction already aborted
//When command execution is failed, it's aborting transaction
//If you tried to call abortTransaction, it`s return error that transaction already aborted
return err
}
if err := sessionContext.CommitTransaction(sessionContext); err != nil {
@@ -301,100 +182,10 @@ func (m *Mongo) Drop() error {
return m.db.Drop(context.TODO())
}
func (m *Mongo) ensureLockTable() error {
indexes := m.db.Collection(m.config.Locking.CollectionName).Indexes()
indexOptions := options.Index().SetUnique(true).SetName(LockIndexName)
_, err := indexes.CreateOne(context.TODO(), mongo.IndexModel{
Options: indexOptions,
Keys: findFilter{Key: -1},
})
if err != nil {
return err
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the MongoDb type.
func (m *Mongo) ensureVersionTable() (err error) {
if err = m.Lock(); err != nil {
return err
}
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
if err != nil {
return err
}
if _, _, err = m.Version(); err != nil {
return err
}
return nil
}
// Utilizes advisory locking on the config.LockingCollection collection
// This uses a unique index on the `locking_key` field.
func (m *Mongo) Lock() error {
return database.CasRestoreOnErr(&m.isLocked, false, true, database.ErrLocked, func() error {
if !m.config.Locking.Enabled {
return nil
}
pid := os.Getpid()
hostname, err := os.Hostname()
if err != nil {
hostname = fmt.Sprintf("Could not determine hostname. Error: %s", err.Error())
}
newLockObj := lockObj{
Key: lockKeyUniqueValue,
Pid: pid,
Hostname: hostname,
CreatedAt: time.Now(),
}
operation := func() error {
timeout, cancelFunc := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).InsertOne(timeout, newLockObj)
defer cancelFunc()
return err
}
exponentialBackOff := backoff.NewExponentialBackOff()
duration := time.Duration(m.config.Locking.Timeout) * time.Second
exponentialBackOff.MaxElapsedTime = duration
exponentialBackOff.MaxInterval = time.Duration(m.config.Locking.Interval) * time.Second
err = backoff.Retry(operation, exponentialBackOff)
if err != nil {
return database.ErrLocked
}
return nil
})
return nil
}
func (m *Mongo) Unlock() error {
return database.CasRestoreOnErr(&m.isLocked, true, false, database.ErrNotLocked, func() error {
if !m.config.Locking.Enabled {
return nil
}
filter := findFilter{
Key: lockKeyUniqueValue,
}
ctx, cancel := context.WithTimeout(context.Background(), contextWaitTimeout)
_, err := m.db.Collection(m.config.Locking.CollectionName).DeleteMany(ctx, filter)
defer cancel()
if err != nil {
return err
}
return nil
})
return nil
}
+8 -89
View File
@@ -32,10 +32,10 @@ var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.mongodb.com/support-policy
specs = []dktesting.ContainerSpec{
{ImageName: "mongo:5.0", Options: opts},
{ImageName: "mongo:6.0", Options: opts},
{ImageName: "mongo:7.0", Options: opts},
{ImageName: "mongo:8.0", Options: opts},
{ImageName: "mongo:3.4", Options: opts},
{ImageName: "mongo:3.6", Options: opts},
{ImageName: "mongo:4.0", Options: opts},
{ImageName: "mongo:4.2", Options: opts},
}
)
@@ -74,22 +74,6 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testWithAuth", testWithAuth)
t.Run("testLockWorks", testLockWorks)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -108,14 +92,14 @@ func test(t *testing.T) {
}
}()
dt.TestNilVersion(t, d)
dt.TestLockAndUnlock(t, d)
//TestLockAndUnlock(t, d) driver doesn't support lock on database level
dt.TestRun(t, d, bytes.NewReader([]byte(`[{"insert":"hello","documents":[{"wild":"world"}]}]`)))
dt.TestSetVersion(t, d)
dt.TestDrop(t, d)
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -137,11 +121,11 @@ func testMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte(`[{"insert":"hello","documents":[{"wild":"world"}]}]`))
})
}
func testWithAuth(t *testing.T) {
func TestWithAuth(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -196,76 +180,11 @@ func testWithAuth(t *testing.T) {
})
}
func testLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := mongoConnectionString(ip, port)
p := &Mongo{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.TestRun(t, d, bytes.NewReader([]byte(`[{"insert":"hello","documents":[{"wild":"world"}]}]`)))
mc := d.(*Mongo)
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Unlock()
if err != nil {
t.Fatal(err)
}
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Unlock()
if err != nil {
t.Fatal(err)
}
// enable locking,
//try to hit a lock conflict
mc.config.Locking.Enabled = true
mc.config.Locking.Timeout = 1
err = mc.Lock()
if err != nil {
t.Fatal(err)
}
err = mc.Lock()
if err == nil {
t.Fatal("should have failed, mongo should be locked already")
}
})
}
func TestTransaction(t *testing.T) {
transactionSpecs := []dktesting.ContainerSpec{
{ImageName: "mongo:4", Options: dktest.Options{PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"mongod", "--bind_ip_all", "--replSet", "rs0"}}},
}
t.Cleanup(func() {
for _, spec := range transactionSpecs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
dktesting.ParallelTest(t, transactionSpecs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
-46
View File
@@ -1,46 +0,0 @@
// Package multistmt provides methods for parsing multi-statement database migrations
package multistmt
import (
"bufio"
"bytes"
"io"
)
// StartBufSize is the default starting size of the buffer used to scan and parse multi-statement migrations
var StartBufSize = 4096
// Handler handles a single migration parsed from a multi-statement migration.
// It's given the single migration to handle and returns whether or not further statements
// from the multi-statement migration should be parsed and handled.
type Handler func(migration []byte) bool
func splitWithDelimiter(delimiter []byte) func(d []byte, atEOF bool) (int, []byte, error) {
return func(d []byte, atEOF bool) (int, []byte, error) {
// SplitFunc inspired by bufio.ScanLines() implementation
if atEOF {
if len(d) == 0 {
return 0, nil, nil
}
return len(d), d, nil
}
if i := bytes.Index(d, delimiter); i >= 0 {
return i + len(delimiter), d[:i+len(delimiter)], nil
}
return 0, nil, nil
}
}
// Parse parses the given multi-statement migration
func Parse(reader io.Reader, delimiter []byte, maxMigrationSize int, h Handler) error {
scanner := bufio.NewScanner(reader)
scanner.Buffer(make([]byte, 0, StartBufSize), maxMigrationSize)
scanner.Split(splitWithDelimiter(delimiter))
for scanner.Scan() {
cont := h(scanner.Bytes())
if !cont {
break
}
}
return scanner.Err()
}
-57
View File
@@ -1,57 +0,0 @@
package multistmt_test
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4/database/multistmt"
)
const maxMigrationSize = 1024
func TestParse(t *testing.T) {
testCases := []struct {
name string
multiStmt string
delimiter string
expected []string
expectedErr error
}{
{name: "single statement, no delimiter", multiStmt: "single statement, no delimiter", delimiter: ";",
expected: []string{"single statement, no delimiter"}, expectedErr: nil},
{name: "single statement, one delimiter", multiStmt: "single statement, one delimiter;", delimiter: ";",
expected: []string{"single statement, one delimiter;"}, expectedErr: nil},
{name: "two statements, no trailing delimiter", multiStmt: "statement one; statement two", delimiter: ";",
expected: []string{"statement one;", " statement two"}, expectedErr: nil},
{name: "two statements, with trailing delimiter", multiStmt: "statement one; statement two;", delimiter: ";",
expected: []string{"statement one;", " statement two;"}, expectedErr: nil},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
stmts := make([]string, 0, len(tc.expected))
err := multistmt.Parse(strings.NewReader(tc.multiStmt), []byte(tc.delimiter), maxMigrationSize, func(b []byte) bool {
stmts = append(stmts, string(b))
return true
})
assert.Equal(t, tc.expectedErr, err)
assert.Equal(t, tc.expected, stmts)
})
}
}
func TestParseDiscontinue(t *testing.T) {
multiStmt := "statement one; statement two"
delimiter := ";"
expected := []string{"statement one;"}
stmts := make([]string, 0, len(expected))
err := multistmt.Parse(strings.NewReader(multiStmt), []byte(delimiter), maxMigrationSize, func(b []byte) bool {
stmts = append(stmts, string(b))
return false
})
assert.Nil(t, err)
assert.Equal(t, expected, stmts)
}
+4 -6
View File
@@ -5,8 +5,6 @@
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-no-lock` | `NoLock` | Set to `true` to skip `GET_LOCK`/`RELEASE_LOCK` statements. Useful for [multi-master MySQL flavors](https://www.percona.com/doc/percona-xtradb-cluster/LATEST/features/pxc-strict-mode.html#explicit-table-locking). Only run migrations from one host when this is enabled. |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds, functionally similar to [Server-side SELECT statement timeouts](https://dev.mysql.com/blog-archive/server-side-select-statement-timeouts/) but enforced by the client. Available for all versions of MySQL, not just >=5.7. |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
@@ -14,7 +12,7 @@
| `port` | | The port to bind to. |
| `tls` | | TLS / SSL encrypted connection parameter; see [go-sql-driver](https://github.com/go-sql-driver/mysql#tls). Use any name (e.g. `migrate`) if you want to use a custom TLS config (`x-tls-` queries). |
| `x-tls-ca` | | The location of the CA (certificate authority) file. |
| `x-tls-cert` | | The location of the client certificate file. Must be used with `x-tls-key`. |
| `x-tls-cert` | | The location of the client certicicate file. Must be used with `x-tls-key`. |
| `x-tls-key` | | The location of the private key file. Must be used with `x-tls-cert`. |
| `x-tls-insecure-skip-verify` | | Whether or not to use SSL (true\|false) |
@@ -29,9 +27,9 @@ import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/mysql"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/golang-migrate/migrate"
"github.com/golang-migrate/migrate/database/mysql"
_ "github.com/golang-migrate/migrate/source/file"
)
func main() {
+148 -240
View File
@@ -1,4 +1,4 @@
//go:build go1.9
// +build go1.9
package mysql
@@ -7,21 +7,22 @@ import (
"crypto/tls"
"crypto/x509"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"os"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate/v4/database"
)
var _ database.Driver = (*Mysql)(nil) // explicit compile time type check
import (
"github.com/go-sql-driver/mysql"
"github.com/hashicorp/go-multierror"
)
import (
"github.com/golang-migrate/migrate/v4/database"
)
func init() {
database.Register("mysql", &Mysql{})
@@ -34,14 +35,12 @@ var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrAppendPEM = fmt.Errorf("failed to append PEM")
ErrTLSCertKeyConfig = fmt.Errorf("to use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
ErrTLSCertKeyConfig = fmt.Errorf("To use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
)
type Config struct {
MigrationsTable string
DatabaseName string
NoLock bool
StatementTimeout time.Duration
MigrationsTable string
DatabaseName string
}
type Mysql struct {
@@ -49,45 +48,48 @@ type Mysql struct {
// just do everything over a single conn anyway.
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
isLocked bool
config *Config
}
// connection instance must have `multiStatements` set to true
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Mysql, error) {
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.PingContext(ctx); err != nil {
if err := instance.Ping(); err != nil {
return nil, err
}
query := `SELECT DATABASE()`
var databaseName sql.NullString
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName.String) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName.String
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
mx := &Mysql{
conn: conn,
db: nil,
db: instance,
config: config,
}
if config.DatabaseName == "" {
query := `SELECT DATABASE()`
var databaseName sql.NullString
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName.String) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName.String
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if err := mx.ensureVersionTable(); err != nil {
return nil, err
}
@@ -95,109 +97,7 @@ func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Mysql
return mx, nil
}
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
mx, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
mx.db = instance
return mx, nil
}
// extractCustomQueryParams extracts the custom query params (ones that start with "x-") from
// mysql.Config.Params (connection parameters) as to not interfere with connecting to MySQL
func extractCustomQueryParams(c *mysql.Config) (map[string]string, error) {
if c == nil {
return nil, ErrNilConfig
}
customQueryParams := map[string]string{}
for k, v := range c.Params {
if strings.HasPrefix(k, "x-") {
customQueryParams[k] = v
delete(c.Params, k)
}
}
return customQueryParams, nil
}
func urlToMySQLConfig(url string) (*mysql.Config, error) {
// Need to parse out custom TLS parameters and call
// mysql.RegisterTLSConfig() before mysql.ParseDSN() is called
// which consumes the registered tls.Config
// Fixes: https://github.com/golang-migrate/migrate/issues/411
//
// Can't use url.Parse() since it fails to parse MySQL DSNs
// mysql.ParseDSN() also searches for "?" to find query parameters:
// https://github.com/go-sql-driver/mysql/blob/46351a8/dsn.go#L344
if idx := strings.LastIndex(url, "?"); idx > 0 {
rawParams := url[idx+1:]
parsedParams, err := nurl.ParseQuery(rawParams)
if err != nil {
return nil, err
}
ctls := parsedParams.Get("tls")
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := os.ReadFile(parsedParams.Get("x-tls-ca"))
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := parsedParams.Get("x-tls-cert"), parsedParams.Get("x-tls-key"); ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
insecureSkipVerifyStr := parsedParams.Get("x-tls-insecure-skip-verify")
if len(insecureSkipVerifyStr) > 0 {
x, err := strconv.ParseBool(insecureSkipVerifyStr)
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
err = mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
if err != nil {
return nil, err
}
}
}
}
config, err := mysql.ParseDSN(strings.TrimPrefix(url, "mysql://"))
if err != nil {
return nil, err
@@ -220,6 +120,52 @@ func urlToMySQLConfig(url string) (*mysql.Config, error) {
}
config.Passwd = password
// use custom TLS?
ctls := config.TLSConfig
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := ioutil.ReadFile(config.Params["x-tls-ca"])
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := config.Params["x-tls-cert"], config.Params["x-tls-key"]; ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
if len(config.Params["x-tls-insecure-skip-verify"]) > 0 {
x, err := strconv.ParseBool(config.Params["x-tls-insecure-skip-verify"])
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
err = mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
if err != nil {
return nil, err
}
}
}
return config, nil
}
@@ -229,38 +175,14 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
return nil, err
}
customParams, err := extractCustomQueryParams(config)
if err != nil {
return nil, err
}
noLockParam, noLock := customParams["x-no-lock"], false
if noLockParam != "" {
noLock, err = strconv.ParseBool(noLockParam)
if err != nil {
return nil, fmt.Errorf("could not parse x-no-lock as bool: %w", err)
}
}
statementTimeoutParam := customParams["x-statement-timeout"]
statementTimeout := 0
if statementTimeoutParam != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutParam)
if err != nil {
return nil, fmt.Errorf("could not parse x-statement-timeout as float: %w", err)
}
}
db, err := sql.Open("mysql", config.FormatDSN())
if err != nil {
return nil, err
}
mx, err := WithInstance(db, &Config{
DatabaseName: config.DBName,
MigrationsTable: customParams["x-migrations-table"],
NoLock: noLock,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
DatabaseName: config.DBName,
MigrationsTable: config.Params["x-migrations-table"],
})
if err != nil {
return nil, err
@@ -271,11 +193,7 @@ func (m *Mysql) Open(url string) (database.Driver, error) {
func (m *Mysql) Close() error {
connErr := m.conn.Close()
var dbErr error
if m.db != nil {
dbErr = m.db.Close()
}
dbErr := m.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -283,70 +201,62 @@ func (m *Mysql) Close() error {
}
func (m *Mysql) Lock() error {
return database.CasRestoreOnErr(&m.isLocked, false, true, database.ErrLocked, func() error {
if m.config.NoLock {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
if m.isLocked {
return database.ErrLocked
}
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
if !success {
return database.ErrLocked
}
return nil
})
}
func (m *Mysql) Unlock() error {
return database.CasRestoreOnErr(&m.isLocked, true, false, database.ErrNotLocked, func() error {
if m.config.NoLock {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
return nil
})
}
func (m *Mysql) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
ctx := context.Background()
if m.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, m.config.StatementTimeout)
defer cancel()
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
if success {
m.isLocked = true
return nil
}
return database.ErrLocked
}
func (m *Mysql) Unlock() error {
if !m.isLocked {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
m.isLocked = false
return nil
}
func (m *Mysql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if _, err := m.conn.ExecContext(ctx, query); err != nil {
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
@@ -354,27 +264,24 @@ func (m *Mysql) Run(migration io.Reader) error {
}
func (m *Mysql) SetVersion(version int, dirty bool) error {
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{Isolation: sql.LevelSerializable})
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "DELETE FROM `" + m.config.MigrationsTable + "` LIMIT 1"
query := "TRUNCATE `" + m.config.MigrationsTable + "`"
if _, err := tx.ExecContext(context.Background(), query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
if version >= 0 {
query := "INSERT INTO `" + m.config.MigrationsTable + "` (version, dirty) VALUES (?, ?)"
if _, err := tx.ExecContext(context.Background(), query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -416,7 +323,7 @@ func (m *Mysql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -431,9 +338,6 @@ func (m *Mysql) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// disable checking foreign key constraints until finished
@@ -469,13 +373,17 @@ func (m *Mysql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
// check if migration table exists
var result string
query := `SHOW TABLES LIKE '` + m.config.MigrationsTable + `'`
query := `SHOW TABLES LIKE "` + m.config.MigrationsTable + `"`
if err := m.conn.QueryRowContext(context.Background(), query).Scan(&result); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
+16 -212
View File
@@ -2,28 +2,24 @@ package mysql
import (
"context"
"crypto/ed25519"
"crypto/x509"
"database/sql"
sqldriver "database/sql/driver"
"encoding/pem"
"errors"
"fmt"
"log"
"math/big"
"math/rand"
"net/url"
"os"
"strconv"
"testing"
"github.com/golang-migrate/migrate/v4"
"testing"
)
import (
"github.com/dhui/dktest"
"github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate/v4"
)
import (
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
"github.com/stretchr/testify/assert"
)
const defaultPort = 3306
@@ -33,21 +29,12 @@ var (
Env: map[string]string{"MYSQL_ROOT_PASSWORD": "root", "MYSQL_DATABASE": "public"},
PortRequired: true, ReadyFunc: isReady,
}
optsAnsiQuotes = dktest.Options{
Env: map[string]string{"MYSQL_ROOT_PASSWORD": "root", "MYSQL_DATABASE": "public"},
PortRequired: true, ReadyFunc: isReady,
Cmd: []string{"--sql-mode=ANSI_QUOTES"},
}
// Supported versions: https://www.mysql.com/support/supportedplatforms/database.html
specs = []dktesting.ContainerSpec{
{ImageName: "mysql:8.0", Options: opts},
{ImageName: "mysql:8.4", Options: opts},
{ImageName: "mysql:9.0", Options: opts},
}
specsAnsiQuotes = []dktesting.ContainerSpec{
{ImageName: "mysql:8.0", Options: optsAnsiQuotes},
{ImageName: "mysql:8.4", Options: optsAnsiQuotes},
{ImageName: "mysql:9.0", Options: optsAnsiQuotes},
{ImageName: "mysql:5.5", Options: opts},
{ImageName: "mysql:5.6", Options: opts},
{ImageName: "mysql:5.7", Options: opts},
{ImageName: "mysql:8", Options: opts},
}
)
@@ -80,7 +67,7 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
}
func Test(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -113,7 +100,7 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
@@ -137,45 +124,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
// check ensureVersionTable
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
// check again
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
})
}
func TestMigrateAnsiQuotes(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(io.Discard, "", log.Ltime)))
dktesting.ParallelTest(t, specsAnsiQuotes, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "public", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("SELECT 1"))
// check ensureVersionTable
if err := d.(*Mysql).ensureVersionTable(); err != nil {
@@ -226,149 +175,7 @@ func TestLockWorks(t *testing.T) {
})
}
func TestNoLockParamValidation(t *testing.T) {
ip := "127.0.0.1"
port := 3306
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
_, err := p.Open(addr + "?x-no-lock=not-a-bool")
if !errors.Is(err, strconv.ErrSyntax) {
t.Fatal("Expected syntax error when passing a non-bool as x-no-lock parameter")
}
}
func TestNoLockWorks(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", ip, port)
p := &Mysql{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
lock := d.(*Mysql)
p = &Mysql{}
d, err = p.Open(addr + "?x-no-lock=true")
if err != nil {
t.Fatal(err)
}
noLock := d.(*Mysql)
// Should be possible to take real lock and no-lock at the same time
if err = lock.Lock(); err != nil {
t.Fatal(err)
}
if err = noLock.Lock(); err != nil {
t.Fatal(err)
}
if err = lock.Unlock(); err != nil {
t.Fatal(err)
}
if err = noLock.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestExtractCustomQueryParams(t *testing.T) {
testcases := []struct {
name string
config *mysql.Config
expectedParams map[string]string
expectedCustomParams map[string]string
expectedErr error
}{
{name: "nil config", expectedErr: ErrNilConfig},
{
name: "no params",
config: mysql.NewConfig(),
expectedCustomParams: map[string]string{},
},
{
name: "no custom params",
config: &mysql.Config{Params: map[string]string{"hello": "world"}},
expectedParams: map[string]string{"hello": "world"},
expectedCustomParams: map[string]string{},
},
{
name: "one param, one custom param",
config: &mysql.Config{
Params: map[string]string{"hello": "world", "x-foo": "bar"},
},
expectedParams: map[string]string{"hello": "world"},
expectedCustomParams: map[string]string{"x-foo": "bar"},
},
{
name: "multiple params, multiple custom params",
config: &mysql.Config{
Params: map[string]string{
"hello": "world",
"x-foo": "bar",
"dead": "beef",
"x-cat": "hat",
},
},
expectedParams: map[string]string{"hello": "world", "dead": "beef"},
expectedCustomParams: map[string]string{"x-foo": "bar", "x-cat": "hat"},
},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
customParams, err := extractCustomQueryParams(tc.config)
if tc.config != nil {
assert.Equal(t, tc.expectedParams, tc.config.Params,
"Expected config params have custom params properly removed")
}
assert.Equal(t, tc.expectedErr, err, "Expected errors to match")
assert.Equal(t, tc.expectedCustomParams, customParams,
"Expected custom params to be properly extracted")
})
}
}
func createTmpCert(t *testing.T) string {
tmpCertFile, err := os.CreateTemp("", "migrate_test_cert")
if err != nil {
t.Fatal("Failed to create temp cert file:", err)
}
t.Cleanup(func() {
if err := os.Remove(tmpCertFile.Name()); err != nil {
t.Log("Failed to cleanup temp cert file:", err)
}
})
r := rand.New(rand.NewSource(0))
pub, priv, err := ed25519.GenerateKey(r)
if err != nil {
t.Fatal("Failed to generate ed25519 key for temp cert file:", err)
}
tmpl := x509.Certificate{
SerialNumber: big.NewInt(0),
}
derBytes, err := x509.CreateCertificate(r, &tmpl, &tmpl, pub, priv)
if err != nil {
t.Fatal("Failed to generate temp cert file:", err)
}
if err := pem.Encode(tmpCertFile, &pem.Block{Type: "CERTIFICATE", Bytes: derBytes}); err != nil {
t.Fatal("Failed to encode ")
}
if err := tmpCertFile.Close(); err != nil {
t.Fatal("Failed to close temp cert file:", err)
}
return tmpCertFile.Name()
}
func TestURLToMySQLConfig(t *testing.T) {
tmpCertFilename := createTmpCert(t)
tmpCertFilenameEscaped := url.PathEscape(tmpCertFilename)
testcases := []struct {
name string
urlStr string
@@ -389,7 +196,7 @@ func TestURLToMySQLConfig(t *testing.T) {
// Not supported yet: https://github.com/go-sql-driver/mysql/issues/591
// {name: "user/password - user with encoded :",
// urlStr: "mysql://username%3A:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
// expectedDSN: "username::password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
// expectedDSN: "username::pasword@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - user with encoded @",
urlStr: "mysql://username%40:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
@@ -399,9 +206,6 @@ func TestURLToMySQLConfig(t *testing.T) {
{name: "user/password - password with encoded @",
urlStr: "mysql://username:password%40@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:password@@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "custom tls",
urlStr: "mysql://username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true&tls=custom&x-tls-ca=" + tmpCertFilenameEscaped,
expectedDSN: "username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true&tls=custom&x-tls-ca=" + tmpCertFilenameEscaped},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
-20
View File
@@ -1,20 +0,0 @@
# neo4j
The Neo4j driver (bolt) does not natively support executing multiple statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param.
This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
The queries **should** run in a single transaction, so partial migrations should not be a concern, but this is untested.
`neo4j://user:password@host:port/`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-multi-statement` | `MultiStatement` | Enable multiple statements to be ran in a single migration (See note above) |
| `user` | Contained within `AuthConfig` | The user to sign in as |
| `password` | Contained within `AuthConfig` | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 7687) |
| | `MigrationsLabel` | Name of the migrations node label |
## Supported versions
Only Neo4j v3.5+ is [supported](https://github.com/neo4j/neo4j-go-driver/issues/64#issuecomment-625133600)
-97
View File
@@ -1,97 +0,0 @@
## Create migrations
Let's create nodes called `Users`:
```
migrate create -ext cypher -dir db/migrations -seq create_user_nodes
```
If there were no errors, we should have two files available under `db/migrations` folder:
- 000001_create_user_nodes.down.cypher
- 000001_create_user_nodes.up.cypher
Note the `cypher` extension that we provided.
In the `.up.cypher` file let's create the table:
```
CREATE (u1:User {name: "Peter"})
CREATE (u2:User {name: "Paul"})
CREATE (u3:User {name: "Mary"})
```
And in the `.down.sql` let's delete it:
```
MATCH (u:User) WHERE u.name IN ["Peter", "Paul", "Mary"] DELETE u
```
Ideally your migrations should be idempotent. You can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)
## Run migrations
```
migrate -database ${NEO4J_URL} -path db/migrations up
```
Let's check if the table was created properly by running `bin/cypher-shell -u neo4j -p password`, then `neo4j> MATCH (u:User)`
The output you are supposed to see:
```
+-----------------------------------------------------------------+
| u |
+-----------------------------------------------------------------+
| (:User {name: "Peter") |
| (:User {name: "Paul") |
| (:User {name: "Mary") |
+-----------------------------------------------------------------+
```
Great! Now let's check if running reverse migration also works:
```
migrate -database ${NEO4J_URL} -path db/migrations down
```
Make sure to check if your database changed as expected in this case as well.
## Database transactions
To show database transactions usage, let's create another set of migrations by running:
```
migrate create -ext cypher -dir db/migrations -seq add_mood_to_users
```
Again, it should create for us two migrations files:
- 000002_add_mood_to_users.down.cypher
- 000002_add_mood_to_users.up.cypher
In Neo4j, when we want our queries to be done in a transaction, we need to wrap it with `:BEGIN` and `:COMMIT` commands.
Migration up:
```
:BEGIN
MATCH (u:User)
SET u.mood = "Cheery"
:COMMIT
```
Migration down:
```
:BEGIN
MATCH (u:User)
SET u.mood = null
:COMMIT
```
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```
import (
"log"
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/neo4j"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
m, err := migrate.New(
"file://db/migrations",
"neo4j://neo4j:password@localhost:7687/")
if err != nil {
log.Fatal(err)
}
if err := m.Up(); err != nil {
log.Fatal(err)
}
}
```
@@ -1 +0,0 @@
DROP CONSTRAINT ON (m:Movie) ASSERT m.Name IS UNIQUE
@@ -1 +0,0 @@
CREATE CONSTRAINT ON (m:Movie) ASSERT m.Name IS UNIQUE
@@ -1,2 +0,0 @@
MATCH (m:Movie)
DELETE m
@@ -1,2 +0,0 @@
CREATE (:Movie {name: "Footloose"})
CREATE (:Movie {name: "Ghost"})
@@ -1,3 +0,0 @@
CREATE (:Movie {name: "Hollow Man"});
CREATE (:Movie {name: "Mystic River"});
;;;
-303
View File
@@ -1,303 +0,0 @@
package neo4j
import (
"bytes"
"errors"
"fmt"
"io"
neturl "net/url"
"strconv"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/neo4j/neo4j-go-driver/neo4j"
)
func init() {
db := Neo4j{}
database.Register("neo4j", &db)
}
const DefaultMigrationsLabel = "SchemaMigration"
var (
StatementSeparator = []byte(";")
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
)
type Config struct {
MigrationsLabel string
MultiStatement bool
MultiStatementMaxSize int
}
type Neo4j struct {
driver neo4j.Driver
lock uint32
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(driver neo4j.Driver, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
nDriver := &Neo4j{
driver: driver,
config: config,
}
if err := nDriver.ensureVersionConstraint(); err != nil {
return nil, err
}
return nDriver, nil
}
func (n *Neo4j) Open(url string) (database.Driver, error) {
uri, err := neturl.Parse(url)
if err != nil {
return nil, err
}
password, _ := uri.User.Password()
authToken := neo4j.BasicAuth(uri.User.Username(), password, "")
uri.User = nil
uri.Scheme = "bolt"
msQuery := uri.Query().Get("x-multi-statement")
// Whether to turn on/off TLS encryption.
tlsEncrypted := uri.Query().Get("x-tls-encrypted")
multi := false
encrypted := false
if msQuery != "" {
multi, err = strconv.ParseBool(uri.Query().Get("x-multi-statement"))
if err != nil {
return nil, err
}
}
if tlsEncrypted != "" {
encrypted, err = strconv.ParseBool(tlsEncrypted)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := uri.Query().Get("x-multi-statement-max-size"); s != "" {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
}
uri.RawQuery = ""
driver, err := neo4j.NewDriver(uri.String(), authToken, func(config *neo4j.Config) {
config.Encrypted = encrypted
})
if err != nil {
return nil, err
}
return WithInstance(driver, &Config{
MigrationsLabel: DefaultMigrationsLabel,
MultiStatement: multi,
MultiStatementMaxSize: multiStatementMaxSize,
})
}
func (n *Neo4j) Close() error {
return n.driver.Close()
}
// local locking in order to pass tests, Neo doesn't support database locking
func (n *Neo4j) Lock() error {
if !atomic.CompareAndSwapUint32(&n.lock, 0, 1) {
return database.ErrLocked
}
return nil
}
func (n *Neo4j) Unlock() error {
if !atomic.CompareAndSwapUint32(&n.lock, 1, 0) {
return database.ErrNotLocked
}
return nil
}
func (n *Neo4j) Run(migration io.Reader) (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
if n.config.MultiStatement {
_, err = session.WriteTransaction(func(transaction neo4j.Transaction) (interface{}, error) {
var stmtRunErr error
if err := multistmt.Parse(migration, StatementSeparator, n.config.MultiStatementMaxSize, func(stmt []byte) bool {
trimStmt := bytes.TrimSpace(stmt)
if len(trimStmt) == 0 {
return true
}
trimStmt = bytes.TrimSuffix(trimStmt, StatementSeparator)
if len(trimStmt) == 0 {
return true
}
result, err := transaction.Run(string(trimStmt), nil)
if _, err := neo4j.Collect(result, err); err != nil {
stmtRunErr = err
return false
}
return true
}); err != nil {
return nil, err
}
return nil, stmtRunErr
})
return err
}
body, err := io.ReadAll(migration)
if err != nil {
return err
}
_, err = neo4j.Collect(session.Run(string(body[:]), nil))
return err
}
func (n *Neo4j) SetVersion(version int, dirty bool) (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
query := fmt.Sprintf("MERGE (sm:%s {version: $version}) SET sm.dirty = $dirty, sm.ts = datetime()",
n.config.MigrationsLabel)
_, err = neo4j.Collect(session.Run(query, map[string]interface{}{"version": version, "dirty": dirty}))
if err != nil {
return err
}
return nil
}
type MigrationRecord struct {
Version int
Dirty bool
}
func (n *Neo4j) Version() (version int, dirty bool, err error) {
session, err := n.driver.Session(neo4j.AccessModeRead)
if err != nil {
return database.NilVersion, false, err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
query := fmt.Sprintf(`MATCH (sm:%s) RETURN sm.version AS version, sm.dirty AS dirty
ORDER BY COALESCE(sm.ts, datetime({year: 0})) DESC, sm.version DESC LIMIT 1`,
n.config.MigrationsLabel)
result, err := session.ReadTransaction(func(transaction neo4j.Transaction) (interface{}, error) {
result, err := transaction.Run(query, nil)
if err != nil {
return nil, err
}
if result.Next() {
record := result.Record()
mr := MigrationRecord{}
versionResult, ok := record.Get("version")
if !ok {
mr.Version = database.NilVersion
} else {
mr.Version = int(versionResult.(int64))
}
dirtyResult, ok := record.Get("dirty")
if ok {
mr.Dirty = dirtyResult.(bool)
}
return mr, nil
}
return nil, result.Err()
})
if err != nil {
return database.NilVersion, false, err
}
if result == nil {
return database.NilVersion, false, err
}
mr := result.(MigrationRecord)
return mr.Version, mr.Dirty, err
}
func (n *Neo4j) Drop() (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
if _, err := neo4j.Collect(session.Run("MATCH (n) DETACH DELETE n", nil)); err != nil {
return err
}
return nil
}
func (n *Neo4j) ensureVersionConstraint() (err error) {
session, err := n.driver.Session(neo4j.AccessModeWrite)
if err != nil {
return err
}
defer func() {
if cerr := session.Close(); cerr != nil {
err = errors.Join(err, cerr)
}
}()
/**
Get constraint and check to avoid error duplicate
using db.labels() to support Neo4j 3 and 4.
Neo4J 3 doesn't support db.constraints() YIELD name
*/
res, err := neo4j.Collect(session.Run(fmt.Sprintf("CALL db.labels() YIELD label WHERE label=\"%s\" RETURN label", n.config.MigrationsLabel), nil))
if err != nil {
return err
}
if len(res) == 1 {
return nil
}
query := fmt.Sprintf("CREATE CONSTRAINT ON (a:%s) ASSERT a.version IS UNIQUE", n.config.MigrationsLabel)
if _, err := neo4j.Collect(session.Run(query, nil)); err != nil {
return err
}
return nil
}
-138
View File
@@ -1,138 +0,0 @@
package neo4j
import (
"bytes"
"context"
"fmt"
"log"
"testing"
"github.com/dhui/dktest"
"github.com/neo4j/neo4j-go-driver/neo4j"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
var (
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady,
Env: map[string]string{"NEO4J_AUTH": "neo4j/migratetest", "NEO4J_ACCEPT_LICENSE_AGREEMENT": "yes"}}
specs = []dktesting.ContainerSpec{
{ImageName: "neo4j:4.0", Options: opts},
{ImageName: "neo4j:4.0-enterprise", Options: opts},
{ImageName: "neo4j:3.5", Options: opts},
{ImageName: "neo4j:3.5-enterprise", Options: opts},
}
)
func neoConnectionString(host, port string) string {
return fmt.Sprintf("bolt://neo4j:migratetest@%s:%s", host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(7687)
if err != nil {
return false
}
driver, err := neo4j.NewDriver(
neoConnectionString(ip, port),
neo4j.BasicAuth("neo4j", "migratetest", ""),
func(config *neo4j.Config) {
config.Encrypted = false
})
if err != nil {
return false
}
defer func() {
if err := driver.Close(); err != nil {
log.Println("close error:", err)
}
}()
session, err := driver.Session(neo4j.AccessModeRead)
if err != nil {
return false
}
result, err := session.Run("RETURN 1", nil)
if err != nil {
return false
} else if result.Err() != nil {
return false
}
return true
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(7687)
if err != nil {
t.Fatal(err)
}
n := &Neo4j{}
d, err := n.Open(neoConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("MATCH (a) RETURN a"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(7687)
if err != nil {
t.Fatal(err)
}
n := &Neo4j{}
neoUrl := neoConnectionString(ip, port) + "/?x-multi-statement=true"
d, err := n.Open(neoUrl)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "neo4j", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMalformed(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(7687)
if err != nil {
t.Fatal(err)
}
n := &Neo4j{}
d, err := n.Open(neoConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
migration := bytes.NewReader([]byte("CREATE (a {qid: 1) RETURN a"))
if err := d.Run(migration); err == nil {
t.Fatal("expected failure for malformed migration")
}
})
}
+8 -9
View File
@@ -8,15 +8,13 @@ import (
)
const reservedChars = "!#$%&'()*+,/:;=?@[]"
const reservedCharTestNamePrefix = "reserved char "
const baseUsername = "username"
const scheme = "database://"
// TestUserUnencodedReservedURLChars documents the behavior of using unencoded reserved characters in usernames with
// net/url Parse()
func TestUserUnencodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
@@ -66,7 +64,7 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
t.Run("reserved char "+tc.char, func(t *testing.T) {
s := scheme + baseUsername + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -100,12 +98,13 @@ func TestUserUnencodedReservedURLChars(t *testing.T) {
}
func TestUserEncodedReservedURLChars(t *testing.T) {
scheme := "database://"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
for _, c := range reservedChars {
c := string(c)
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
t.Run("reserved char "+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := scheme + baseUsername + encodedChar + urlSuffixAndSep
expectedUsername := baseUsername + c
@@ -127,7 +126,7 @@ func TestUserEncodedReservedURLChars(t *testing.T) {
// with net/url Parse()
func TestPasswordUnencodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := scheme + username + ":"
schemeAndUsernameAndSep := "database://" + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
@@ -175,7 +174,7 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run(reservedCharTestNamePrefix+tc.char, func(t *testing.T) {
t.Run("reserved char "+tc.char, func(t *testing.T) {
s := schemeAndUsernameAndSep + basePassword + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
@@ -214,13 +213,13 @@ func TestPasswordUnencodedReservedURLChars(t *testing.T) {
func TestPasswordEncodedReservedURLChars(t *testing.T) {
username := baseUsername
schemeAndUsernameAndSep := scheme + username + ":"
schemeAndUsernameAndSep := "database://" + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
for _, c := range reservedChars {
c := string(c)
t.Run(reservedCharTestNamePrefix+c, func(t *testing.T) {
t.Run("reserved char "+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := schemeAndUsernameAndSep + basePassword + encodedChar + urlSuffixAndSep
expectedPassword := basePassword + c
-43
View File
@@ -1,43 +0,0 @@
# pgx
This package is for [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). A backend for the newer [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5) is [also available](v5).
`pgx://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `x-lock-strategy` | `LockStrategy` | Strategy used for locking during migration (default: advisory) |
| `x-lock-table` | `LockTable` | Name of the table which maintains the migration lock (default: schema_lock) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
@@ -1 +0,0 @@
DROP TABLE IF EXISTS users;
@@ -1,5 +0,0 @@
CREATE TABLE users (
user_id integer unique,
name varchar(40),
email varchar(40)
);
@@ -1 +0,0 @@
ALTER TABLE users DROP COLUMN IF EXISTS city;
@@ -1,3 +0,0 @@
ALTER TABLE users ADD COLUMN city varchar(100);
@@ -1 +0,0 @@
DROP INDEX IF EXISTS users_email_index;
@@ -1,3 +0,0 @@
CREATE UNIQUE INDEX CONCURRENTLY users_email_index ON users (email);
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
DROP TABLE IF EXISTS books;
@@ -1,5 +0,0 @@
CREATE TABLE books (
user_id integer,
name varchar(40),
author varchar(40)
);
@@ -1 +0,0 @@
DROP TABLE IF EXISTS movies;
@@ -1,5 +0,0 @@
CREATE TABLE movies (
user_id integer,
name varchar(40),
director varchar(40)
);
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -1 +0,0 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
-617
View File
@@ -1,617 +0,0 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgconn"
"github.com/jackc/pgerrcode"
_ "github.com/jackc/pgx/v4/stdlib"
"github.com/lib/pq"
)
const (
LockStrategyAdvisory = "advisory"
LockStrategyTable = "table"
)
func init() {
db := Postgres{}
database.Register("pgx", &db)
database.Register("pgx4", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
DefaultLockTable = "schema_lock"
DefaultLockStrategy = LockStrategyAdvisory
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
ErrDatabaseDirty = fmt.Errorf("database is dirty")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
LockTable string
LockStrategy string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if len(config.LockTable) == 0 {
config.LockTable = DefaultLockTable
}
if len(config.LockStrategy) == 0 {
config.LockStrategy = DefaultLockStrategy
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureLockTable(); err != nil {
return nil, err
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v4", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
lockStrategy := purl.Query().Get("x-lock-strategy")
lockTable := purl.Query().Get("x-lock-table")
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
LockStrategy: lockStrategy,
LockTable: lockTable,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.applyAdvisoryLock()
case LockStrategyTable:
return p.applyTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
}
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
switch p.config.LockStrategy {
case LockStrategyAdvisory:
return p.releaseAdvisoryLock()
case LockStrategyTable:
return p.releaseTableLock()
default:
return fmt.Errorf("unknown lock strategy \"%s\"", p.config.LockStrategy)
}
})
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) applyAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
}
func (p *Postgres) applyTableLock() error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
defer func() {
errRollback := tx.Rollback()
if errRollback != nil {
err = errors.Join(err, errRollback)
}
}()
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer func() {
if errClose := rows.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// If row exists at all, lock is present
locked := rows.Next()
if locked {
return database.ErrLocked
}
query = "INSERT INTO " + pq.QuoteIdentifier(p.config.LockTable) + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return tx.Commit()
}
func (p *Postgres) releaseAdvisoryLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) releaseTableLock() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := "DELETE FROM " + pq.QuoteIdentifier(p.config.LockTable) + " WHERE lock_id = $1"
if _, err := p.db.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
return nil
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
// do not drop lock table
if tableName == p.config.LockTable && p.config.LockStrategy == LockStrategyTable {
continue
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (p *Postgres) ensureLockTable() error {
if p.config.LockStrategy != LockStrategyTable {
return nil
}
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := p.db.QueryRow(query, p.config.LockTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE ` + pq.QuoteIdentifier(p.config.LockTable) + ` (lock_id BIGINT NOT NULL PRIMARY KEY)`
if _, err := p.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
-786
View File
@@ -1,786 +0,0 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMigrateLockTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-lock-strategy=table", "x-lock-table=lock_table")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://./examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
-41
View File
@@ -1,41 +0,0 @@
# pgx
This package is for [pgx/v5](https://pkg.go.dev/github.com/jackc/pgx/v5). A backend for the older [pgx/v4](https://pkg.go.dev/github.com/jackc/pgx/v4). is [also available](..).
`pgx5://user:password@host:port/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
-480
View File
@@ -1,480 +0,0 @@
//go:build go1.9
package pgx
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
"github.com/jackc/pgerrcode"
"github.com/jackc/pgx/v5/pgconn"
_ "github.com/jackc/pgx/v5/stdlib"
)
func init() {
db := Postgres{}
database.Register("pgx5", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
)
type Config struct {
MigrationsTable string
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MigrationsTableQuoted bool
MultiStatementEnabled bool
MultiStatementMaxSize int
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Driver is registered as pgx, but connection string must use postgres schema
// when making actual connection
// i.e. pgx://user:password@host:port/db => postgres://user:password@host:port/db
purl.Scheme = "postgres"
db, err := sql.Open("pgx/v5", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
}
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
return p.runStatement(migr)
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
if pgErr, ok := err.(*pgconn.PgError); ok {
var line uint
var col uint
var lineColOK bool
line, col, lineColOK = computeLineFromPos(query, int(pgErr.Position))
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pgconn.PgError); ok {
if e.SQLState() == pgerrcode.UndefinedTable {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + quoteIdentifier(t) + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Postgres type.
func (p *Postgres) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + quoteIdentifier(p.config.migrationsSchemaName) + `.` + quoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Copied from lib/pq implementation: https://github.com/lib/pq/blob/v1.9.0/conn.go#L1611
func quoteIdentifier(name string) string {
end := strings.IndexRune(name, 0)
if end > -1 {
name = name[:end]
}
return `"` + strings.ReplaceAll(name, `"`, `""`) + `"`
}
-761
View File
@@ -1,761 +0,0 @@
package pgx
// error codes https://github.com/jackc/pgerrcode/blob/master/errcode.go
import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.FirstPort()
if err != nil {
return false
}
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
return false
}
defer func() {
if err := db.Close(); err != nil {
log.Println("close error:", err)
}
}()
if err = db.PingContext(ctx); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
log.Println(err)
}
return false
}
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
m, err := migrate.NewWithDatabaseInstance("file://../examples/migrations", "pgx", d)
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
})
}
func TestMultipleStatements(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: ERROR: syntax error at or near "TABLEE" (SQLSTATE 42601))`
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);")); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-custom=foobar")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
})
}
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create foobar schema
if err := d.Run(strings.NewReader("CREATE SCHEMA foobar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(pgConnectionString(ip, port, "search_path=foobar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func TestFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func TestCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create foo and bar schemas
if err := d.Run(strings.NewReader("CREATE SCHEMA foo AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
if err := d.Run(strings.NewReader("CREATE SCHEMA bar AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// re-connect using that schemas
dfoo, err := p.Open(pgConnectionString(ip, port, "search_path=foo"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dfoo.Close(); err != nil {
t.Error(err)
}
}()
dbar, err := p.Open(pgConnectionString(ip, port, "search_path=bar"))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := dbar.Close(); err != nil {
t.Error(err)
}
}()
if err := dfoo.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Lock(); err != nil {
t.Fatal(err)
}
if err := dbar.Unlock(); err != nil {
t.Fatal(err)
}
if err := dfoo.Unlock(); err != nil {
t.Fatal(err)
}
})
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
// The number of concurrent processes running WithInstance
const concurrency = 30
// We can instantiate a single database handle because it is
// actually a connection pool, and so, each of the below go
// routines will have a high probability of using a separate
// connection, which is something we want to exercise.
db, err := sql.Open("pgx", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
db.SetMaxIdleConns(concurrency)
db.SetMaxOpenConns(concurrency)
var wg sync.WaitGroup
defer wg.Wait()
wg.Add(concurrency)
for i := 0; i < concurrency; i++ {
go func(i int) {
defer wg.Done()
_, err := WithInstance(db, &Config{})
if err != nil {
t.Errorf("process %d error: %s", i, err)
}
}(i)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
-11
View File
@@ -5,10 +5,6 @@
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-migrations-table-quoted` | `MigrationsTableQuoted` | By default, migrate quotes the migration table for SQL injection safety reasons. This option disable quoting and naively checks that you have quoted the migration table name. e.g. `"my_schema"."schema_migrations"` |
| `x-statement-timeout` | `StatementTimeout` | Abort any statement that takes more than the specified number of milliseconds |
| `x-multi-statement` | `MultiStatementEnabled` | Enable multi-statement execution (default: false) |
| `x-multi-statement-max-size` | `MultiStatementMaxSize` | Maximum size of single statement in bytes (default: 10MB) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
@@ -30,10 +26,3 @@
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
## Multi-statement mode
In PostgreSQL running multiple SQL statements in one `Exec` executes them inside a transaction. Sometimes this
behavior is not desirable because some statements can be only run outside of transaction (e.g.
`CREATE INDEX CONCURRENTLY`). If you want to use `CREATE INDEX CONCURRENTLY` without activating multi-statement mode
you have to put such statements in a separate migration files.
+9 -28
View File
@@ -7,9 +7,9 @@ Our user here is `postgres`, password `password`, and host is `localhost`.
```
psql -h localhost -U postgres -w -c "create database example;"
```
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convenience:
When using Migrate CLI we need to pass to database URL. Let's export it to a variable for convienience:
```
export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/example?sslmode=disable'
export POSTGRESQL_URL=postgres://postgres:password@localhost:5432/example?sslmode=disable
```
`sslmode=disable` means that the connection with our database will not be encrypted. Enabling it is left as an exercise.
@@ -27,7 +27,7 @@ If there were no errors, we should have two files available under `db/migrations
Note the `sql` extension that we provided.
In the `.up.sql` file let's create the table:
```sql
```
CREATE TABLE IF NOT EXISTS users(
user_id serial PRIMARY KEY,
username VARCHAR (50) UNIQUE NOT NULL,
@@ -36,10 +36,10 @@ CREATE TABLE IF NOT EXISTS users(
);
```
And in the `.down.sql` let's delete it:
```sql
```
DROP TABLE IF EXISTS users;
```
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](../../GETTING_STARTED.md#create-migrations)
By adding `IF EXISTS/IF NOT EXISTS` we are making migrations idempotent - you can read more about idempotency in [getting started](GETTING_STARTED.md#create-migrations)
## Run migrations
```
@@ -79,7 +79,7 @@ Again, it should create for us two migrations files:
In Postgres, when we want our queries to be done in a transaction, we need to wrap it with `BEGIN` and `COMMIT` commands.
In our example, we are going to add a column to our database that can only accept enumerable values or NULL.
Migration up:
```sql
```
BEGIN;
CREATE TYPE enum_mood AS ENUM (
@@ -92,7 +92,7 @@ ALTER TABLE users ADD COLUMN mood enum_mood;
COMMIT;
```
Migration down:
```sql
```
BEGIN;
ALTER TABLE users DROP COLUMN mood;
@@ -124,7 +124,7 @@ Indexes:
## Optional: Run migrations within your Go app
Here is a very simple app running migrations for the above configuration:
```go
```
import (
"log"
@@ -145,23 +145,4 @@ func main() {
}
}
```
You can find details [here](README.md#use-in-your-go-project)
## Fix issue where migrations run twice
When the schema and role names are the same, you might run into issues if you create this schema using migrations.
This is caused by the fact that the [default `search_path`](https://www.postgresql.org/docs/current/ddl-schemas.html#DDL-SCHEMAS-PATH) is `"$user", public`.
In the first run (with an empty database) the migrate table is created in `public`.
When the migrations create the `$user` schema, the next run will store (a new) migrate table in this schema (due to order of schemas in `search_path`) and tries to apply all migrations again (most likely failing).
To solve this you need to change the default `search_path` by removing the `$user` component, so the migrate table is always stored in the (available) `public` schema.
This can be done using the [`search_path` query parameter in the URL](https://github.com/jexia/migrate/blob/fix-postgres-version-table/database/postgres/README.md#postgres).
For example to force the migrations table in the public schema you can use:
```
export POSTGRESQL_URL='postgres://postgres:password@localhost:5432/example?sslmode=disable&search_path=public'
```
Note that you need to explicitly add the schema names to the table names in your migrations when you to modify the tables of the non-public schema.
Alternatively you can add the non-public schema manually (before applying the migrations) if that is possible in your case and let the tool store the migrations table in this schema as well.
You can find details [here](README.md#use-in-your-go-project)
+94 -218
View File
@@ -1,23 +1,20 @@
//go:build go1.9
// +build go1.9
package postgres
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/multistmt"
multierror "github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
@@ -27,12 +24,7 @@ func init() {
database.Register("postgresql", &db)
}
var (
multiStmtDelimiter = []byte(";")
DefaultMigrationsTable = "schema_migrations"
DefaultMultiStatementMaxSize = 10 * 1 << 20 // 10 MB
)
var DefaultMigrationsTable = "schema_migrations"
var (
ErrNilConfig = fmt.Errorf("no config")
@@ -42,83 +34,67 @@ var (
)
type Config struct {
MigrationsTable string
MigrationsTableQuoted bool
MultiStatementEnabled bool
DatabaseName string
SchemaName string
migrationsSchemaName string
migrationsTableName string
StatementTimeout time.Duration
MultiStatementMaxSize int
MigrationsTable string
DatabaseName string
SchemaName string
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked atomic.Bool
isLocked bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postgres, error) {
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.PingContext(ctx); err != nil {
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := conn.QueryRowContext(ctx, query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if config.SchemaName == "" {
query := `SELECT CURRENT_SCHEMA()`
var schemaName sql.NullString
if err := conn.QueryRowContext(ctx, query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if !schemaName.Valid {
return nil, ErrNoSchema
}
config.SchemaName = schemaName.String
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
query = `SELECT CURRENT_SCHEMA()`
var schemaName string
if err := instance.QueryRow(query).Scan(&schemaName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(schemaName) == 0 {
return nil, ErrNoSchema
}
config.SchemaName = schemaName
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
config.migrationsSchemaName = config.SchemaName
config.migrationsTableName = config.MigrationsTable
if config.MigrationsTableQuoted {
re := regexp.MustCompile(`"(.*?)"`)
result := re.FindAllStringSubmatch(config.MigrationsTable, -1)
config.migrationsTableName = result[len(result)-1][1]
if len(result) == 2 {
config.migrationsSchemaName = result[0][1]
} else if len(result) > 2 {
return nil, fmt.Errorf("\"%s\" MigrationsTable contains too many dot characters", config.MigrationsTable)
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
@@ -129,26 +105,6 @@ func WithConnection(ctx context.Context, conn *sql.Conn, config *Config) (*Postg
return px, nil
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
ctx := context.Background()
if err := instance.Ping(); err != nil {
return nil, err
}
conn, err := instance.Conn(ctx)
if err != nil {
return nil, err
}
px, err := WithConnection(ctx, conn, config)
if err != nil {
return nil, err
}
px.db = instance
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
@@ -161,52 +117,10 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
}
migrationsTable := purl.Query().Get("x-migrations-table")
migrationsTableQuoted := false
if s := purl.Query().Get("x-migrations-table-quoted"); len(s) > 0 {
migrationsTableQuoted, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-migrations-table-quoted: %w", err)
}
}
if (len(migrationsTable) > 0) && (migrationsTableQuoted) && ((migrationsTable[0] != '"') || (migrationsTable[len(migrationsTable)-1] != '"')) {
return nil, fmt.Errorf("x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: %s", migrationsTable)
}
statementTimeoutString := purl.Query().Get("x-statement-timeout")
statementTimeout := 0
if statementTimeoutString != "" {
statementTimeout, err = strconv.Atoi(statementTimeoutString)
if err != nil {
return nil, err
}
}
multiStatementMaxSize := DefaultMultiStatementMaxSize
if s := purl.Query().Get("x-multi-statement-max-size"); len(s) > 0 {
multiStatementMaxSize, err = strconv.Atoi(s)
if err != nil {
return nil, err
}
if multiStatementMaxSize <= 0 {
multiStatementMaxSize = DefaultMultiStatementMaxSize
}
}
multiStatementEnabled := false
if s := purl.Query().Get("x-multi-statement"); len(s) > 0 {
multiStatementEnabled, err = strconv.ParseBool(s)
if err != nil {
return nil, fmt.Errorf("unable to parse option x-multi-statement: %w", err)
}
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
MigrationsTableQuoted: migrationsTableQuoted,
StatementTimeout: time.Duration(statementTimeout) * time.Millisecond,
MultiStatementEnabled: multiStatementEnabled,
MultiStatementMaxSize: multiStatementMaxSize,
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
})
if err != nil {
@@ -218,11 +132,7 @@ func (p *Postgres) Open(url string) (database.Driver, error) {
func (p *Postgres) Close() error {
connErr := p.conn.Close()
var dbErr error
if p.db != nil {
dbErr = p.db.Close()
}
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
@@ -231,69 +141,53 @@ func (p *Postgres) Close() error {
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
return database.CasRestoreOnErr(&p.isLocked, false, true, database.ErrLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
// This will wait indefinitely until the lock can be acquired.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Unlock() error {
return database.CasRestoreOnErr(&p.isLocked, true, false, database.ErrNotLocked, func() error {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.migrationsSchemaName, p.config.migrationsTableName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
})
}
func (p *Postgres) Run(migration io.Reader) error {
if p.config.MultiStatementEnabled {
var err error
if e := multistmt.Parse(migration, multiStmtDelimiter, p.config.MultiStatementMaxSize, func(m []byte) bool {
if err = p.runStatement(m); err != nil {
return false
}
return true
}); e != nil {
return e
}
return err
if p.isLocked {
return database.ErrLocked
}
migr, err := io.ReadAll(migration)
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.SchemaName)
if err != nil {
return err
}
return p.runStatement(migr)
// This will either obtain the lock immediately and return true,
// or return false if the lock cannot be acquired immediately.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
p.isLocked = true
return nil
}
func (p *Postgres) runStatement(statement []byte) error {
ctx := context.Background()
if p.config.StatementTimeout != 0 {
var cancel context.CancelFunc
ctx, cancel = context.WithTimeout(ctx, p.config.StatementTimeout)
defer cancel()
}
query := string(statement)
if strings.TrimSpace(query) == "" {
func (p *Postgres) Unlock() error {
if !p.isLocked {
return nil
}
if _, err := p.conn.ExecContext(ctx, query); err != nil {
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName, p.config.SchemaName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
p.isLocked = false
return nil
}
func (p *Postgres) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
if pgErr, ok := err.(*pq.Error); ok {
var line uint
var col uint
@@ -310,16 +204,17 @@ func (p *Postgres) runStatement(statement []byte) error {
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: statement, Line: line}
return database.Error{OrigErr: err, Err: message, Query: migr, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: statement}
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -358,22 +253,19 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName)
query := `TRUNCATE ` + pq.QuoteIdentifier(p.config.MigrationsTable)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version, dirty) VALUES ($1, $2)`
if version >= 0 {
query = `INSERT INTO ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -387,7 +279,7 @@ func (p *Postgres) SetVersion(version int, dirty bool) error {
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` LIMIT 1`
query := `SELECT version, dirty FROM ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
@@ -415,7 +307,7 @@ func (p *Postgres) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -430,9 +322,6 @@ func (p *Postgres) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -457,28 +346,15 @@ func (p *Postgres) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
// This block checks whether the `MigrationsTable` already exists. This is useful because it allows read only postgres
// users to also check the current version of the schema. Previously, even if `MigrationsTable` existed, the
// `CREATE TABLE IF NOT EXISTS...` query would fail because the user does not have the CREATE permission.
// Taken from https://github.com/mattes/migrate/blob/master/database/postgres/postgres.go#L258
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_schema = $1 AND table_name = $2 LIMIT 1`
row := p.conn.QueryRowContext(context.Background(), query, p.config.migrationsSchemaName, p.config.migrationsTableName)
var count int
err = row.Scan(&count)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
query = `CREATE TABLE IF NOT EXISTS ` + pq.QuoteIdentifier(p.config.migrationsSchemaName) + `.` + pq.QuoteIdentifier(p.config.migrationsTableName) + ` (version bigint not null primary key, dirty boolean not null)`
query := `CREATE TABLE IF NOT EXISTS ` + pq.QuoteIdentifier(p.config.MigrationsTable) + ` (version bigint not null primary key, dirty boolean not null)`
if _, err = p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+32 -360
View File
@@ -6,46 +6,37 @@ import (
"context"
"database/sql"
sqldriver "database/sql/driver"
"errors"
"fmt"
"io"
"log"
"github.com/golang-migrate/migrate/v4"
"io"
"strconv"
"strings"
"sync"
"testing"
"github.com/golang-migrate/migrate/v4"
"github.com/dhui/dktest"
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "postgres"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true, ReadyFunc: isReady}
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
// Supported versions: https://www.postgresql.org/support/versioning/
specs = []dktesting.ContainerSpec{
{ImageName: "postgres:13", Options: opts},
{ImageName: "postgres:14", Options: opts},
{ImageName: "postgres:15", Options: opts},
{ImageName: "postgres:16", Options: opts},
{ImageName: "postgres:17", Options: opts},
{ImageName: "postgres:9.4", Options: opts},
{ImageName: "postgres:9.5", Options: opts},
{ImageName: "postgres:9.6", Options: opts},
{ImageName: "postgres:10", Options: opts},
{ImageName: "postgres:11", Options: opts},
}
)
func pgConnectionString(host, port string, options ...string) string {
options = append(options, "sslmode=disable")
return fmt.Sprintf("postgres://postgres:%s@%s:%s/postgres?%s", pgPassword, host, port, strings.Join(options, "&"))
func pgConnectionString(host, port string) string {
return fmt.Sprintf("postgres://postgres@%s:%s/postgres?sslmode=disable", host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -76,41 +67,7 @@ func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
return true
}
func mustRun(t *testing.T, d database.Driver, statements []string) {
for _, statement := range statements {
if err := d.Run(strings.NewReader(statement)); err != nil {
t.Fatal(err)
}
}
}
func Test(t *testing.T) {
t.Run("test", test)
t.Run("testMigrate", testMigrate)
t.Run("testMultipleStatements", testMultipleStatements)
t.Run("testMultipleStatementsInMultiStatementMode", testMultipleStatementsInMultiStatementMode)
t.Run("testErrorParsing", testErrorParsing)
t.Run("testFilterCustomQuery", testFilterCustomQuery)
t.Run("testWithSchema", testWithSchema)
t.Run("testMigrationTableOption", testMigrationTableOption)
t.Run("testFailToCreateTableWithoutPermissions", testFailToCreateTableWithoutPermissions)
t.Run("testCheckBeforeCreateTable", testCheckBeforeCreateTable)
t.Run("testParallelSchema", testParallelSchema)
t.Run("testPostgresLock", testPostgresLock)
t.Run("testWithInstanceConcurrent", testWithInstanceConcurrent)
t.Run("testWithConnection", testWithConnection)
t.Cleanup(func() {
for _, spec := range specs {
t.Log("Cleaning up ", spec.ImageName)
if err := spec.Cleanup(); err != nil {
t.Error("Error removing ", spec.ImageName, "error:", err)
}
}
})
}
func test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -132,7 +89,7 @@ func test(t *testing.T) {
})
}
func testMigrate(t *testing.T) {
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -154,11 +111,11 @@ func testMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("SELECT 1"))
})
}
func testMultipleStatements(t *testing.T) {
func TestMultiStatement(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -191,40 +148,7 @@ func testMultipleStatements(t *testing.T) {
})
}
func testMultipleStatementsInMultiStatementMode(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port, "x-multi-statement=true")
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
if err := d.Run(strings.NewReader("CREATE TABLE foo (foo text); CREATE INDEX CONCURRENTLY idx_foo ON foo (foo);")); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure created index exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM pg_indexes WHERE schemaname = (SELECT current_schema()) AND indexname = 'idx_foo')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func testErrorParsing(t *testing.T) {
func TestErrorParsing(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -253,15 +177,14 @@ func testErrorParsing(t *testing.T) {
})
}
func testFilterCustomQuery(t *testing.T) {
func TestFilterCustomQuery(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-custom=foobar",
pgPassword, ip, port)
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", ip, port)
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
@@ -275,7 +198,7 @@ func testFilterCustomQuery(t *testing.T) {
})
}
func testWithSchema(t *testing.T) {
func TestWithSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -303,8 +226,7 @@ func testWithSchema(t *testing.T) {
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foobar",
pgPassword, ip, port))
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", ip, port))
if err != nil {
t.Fatal(err)
}
@@ -318,7 +240,7 @@ func testWithSchema(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
if version != -1 {
t.Fatal("expected NilVersion")
}
@@ -345,222 +267,7 @@ func testWithSchema(t *testing.T) {
})
}
func testMigrationTableOption(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
p := &Postgres{}
d, _ := p.Open(addr)
defer func() {
if err := d.Close(); err != nil {
t.Fatal(err)
}
}()
// create migrate schema
if err := d.Run(strings.NewReader("CREATE SCHEMA migrate AUTHORIZATION postgres")); err != nil {
t.Fatal(err)
}
// bad unquoted x-migrations-table parameter
wantErr := "x-migrations-table must be quoted (for instance '\"migrate\".\"schema_migrations\"') when x-migrations-table-quoted is enabled, current value is: migrate.schema_migrations"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// too many quoted x-migrations-table parameters
wantErr = "\"\"migrate\".\"schema_migrations\".\"toomany\"\" MigrationsTable contains too many dot characters"
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\".\"toomany\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if (err != nil) && (err.Error() != wantErr) {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
// good quoted x-migrations-table parameter
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"migrate\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
// make sure migrate.schema_migrations table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'schema_migrations' AND table_schema = 'migrate')").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table migrate.schema_migrations to exist")
}
d, err = p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=migrate.schema_migrations",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'migrate.schema_migrations' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table 'migrate.schema_migrations' to exist")
}
})
}
func testFailToCreateTableWithoutPermissions(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
defer func() {
if d2 == nil {
return
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
}()
var e *database.Error
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
// re-connect using that x-migrations-table and x-migrations-table-quoted
d2, err = p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&x-migrations-table=\"barfoo\".\"schema_migrations\"&x-migrations-table-quoted=1",
pgPassword, ip, port))
if !errors.As(err, &e) || err == nil {
t.Fatal("Unexpected error, want permission denied error. Got: ", err)
}
if !strings.Contains(e.OrigErr.Error(), "permission denied for schema barfoo") {
t.Fatal(e)
}
})
}
func testCheckBeforeCreateTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
addr := pgConnectionString(ip, port)
// Check that opening the postgres connection returns NilVersion
p := &Postgres{}
d, err := p.Open(addr)
if err != nil {
t.Fatal(err)
}
defer func() {
if err := d.Close(); err != nil {
t.Error(err)
}
}()
// create user who is not the owner. Although we're concatenating strings in an sql statement it should be fine
// since this is a test environment and we're not expecting to the pgPassword to be malicious
mustRun(t, d, []string{
"CREATE USER not_owner WITH ENCRYPTED PASSWORD '" + pgPassword + "'",
"CREATE SCHEMA barfoo AUTHORIZATION postgres",
"GRANT USAGE ON SCHEMA barfoo TO not_owner",
"GRANT CREATE ON SCHEMA barfoo TO not_owner",
})
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
if err := d2.Close(); err != nil {
t.Fatal(err)
}
// revoke privileges
mustRun(t, d, []string{
"REVOKE CREATE ON SCHEMA barfoo FROM PUBLIC",
"REVOKE CREATE ON SCHEMA barfoo FROM not_owner",
})
// re-connect using that schema
d3, err := p.Open(fmt.Sprintf("postgres://not_owner:%s@%v:%v/postgres?sslmode=disable&search_path=barfoo",
pgPassword, ip, port))
if err != nil {
t.Fatal(err)
}
version, _, err := d3.Version()
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
t.Fatal("Unexpected version, want database.NilVersion. Got: ", version)
}
defer func() {
if err := d3.Close(); err != nil {
t.Fatal(err)
}
}()
})
}
func testParallelSchema(t *testing.T) {
func TestParallelSchema(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -588,8 +295,7 @@ func testParallelSchema(t *testing.T) {
}
// re-connect using that schemas
dfoo, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foo",
pgPassword, ip, port))
dfoo, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foo", ip, port))
if err != nil {
t.Fatal(err)
}
@@ -599,8 +305,7 @@ func testParallelSchema(t *testing.T) {
}
}()
dbar, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=bar",
pgPassword, ip, port))
dbar, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=bar", ip, port))
if err != nil {
t.Fatal(err)
}
@@ -628,7 +333,11 @@ func testParallelSchema(t *testing.T) {
})
}
func testPostgresLock(t *testing.T) {
func TestWithInstance(t *testing.T) {
}
func TestPostgres_Lock(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -668,7 +377,7 @@ func testPostgresLock(t *testing.T) {
})
}
func testWithInstanceConcurrent(t *testing.T) {
func TestWithInstance_Concurrent(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
@@ -710,44 +419,6 @@ func testWithInstanceConcurrent(t *testing.T) {
}
})
}
func testWithConnection(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.FirstPort()
if err != nil {
t.Fatal(err)
}
db, err := sql.Open("postgres", pgConnectionString(ip, port))
if err != nil {
t.Fatal(err)
}
defer func() {
if err := db.Close(); err != nil {
t.Error(err)
}
}()
ctx := context.Background()
conn, err := db.Conn(ctx)
if err != nil {
t.Fatal(err)
}
p, err := WithConnection(ctx, conn, &Config{})
if err != nil {
t.Fatal(err)
}
defer func() {
if err := p.Close(); err != nil {
t.Error(err)
}
}()
dt.Test(t, p, []byte("SELECT 1"))
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
@@ -798,10 +469,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
input = strings.Replace(input, "\n", "\r\n", -1)
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
input = strings.Replace(input, "FROM", "FRÖM", -1)
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
@@ -826,4 +497,5 @@ func Test_computeLineFromPos(t *testing.T) {
run(true, true)
})
}
}
+24 -26
View File
@@ -2,16 +2,17 @@ package ql
import (
"database/sql"
"errors"
"fmt"
"github.com/hashicorp/go-multierror"
"io"
nurl "net/url"
"io/ioutil"
"strings"
"sync/atomic"
nurl "net/url"
_ "github.com/cznic/ql/driver"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
_ "modernc.org/ql/driver"
)
func init() {
@@ -33,7 +34,7 @@ type Config struct {
type Ql struct {
db *sql.DB
isLocked atomic.Bool
isLocked bool
config *Config
}
@@ -71,7 +72,11 @@ func (m *Ql) ensureVersionTable() (err error) {
defer func() {
if e := m.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
@@ -80,7 +85,7 @@ func (m *Ql) ensureVersionTable() (err error) {
return err
}
if _, err := tx.Exec(fmt.Sprintf(`
CREATE TABLE IF NOT EXISTS %s (version uint64, dirty bool);
CREATE TABLE IF NOT EXISTS %s (version uint64,dirty bool);
CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version);
`, m.config.MigrationsTable, m.config.MigrationsTable)); err != nil {
if err := tx.Rollback(); err != nil {
@@ -128,10 +133,9 @@ func (m *Ql) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
@@ -144,10 +148,6 @@ func (m *Ql) Drop() (err error) {
}
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
@@ -161,19 +161,21 @@ func (m *Ql) Drop() (err error) {
return nil
}
func (m *Ql) Lock() error {
if !m.isLocked.CompareAndSwap(false, true) {
if m.isLocked {
return database.ErrLocked
}
m.isLocked = true
return nil
}
func (m *Ql) Unlock() error {
if !m.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
if !m.isLocked {
return nil
}
m.isLocked = false
return nil
}
func (m *Ql) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -188,7 +190,7 @@ func (m *Ql) executeQuery(query string) error {
}
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -208,15 +210,11 @@ func (m *Ql) SetVersion(version int, dirty bool) error {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (uint64(?1), ?2)`,
m.config.MigrationsTable)
if _, err := tx.Exec(query, version, dirty); err != nil {
if version >= 0 {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (%d, %t)`, m.config.MigrationsTable, version, dirty)
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
+22 -4
View File
@@ -3,17 +3,27 @@ package ql
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
_ "github.com/cznic/ql/driver"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
_ "github.com/golang-migrate/migrate/v4/source/file"
_ "modernc.org/ql/driver"
)
func Test(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
p := &Ql{}
addr := fmt.Sprintf("ql://%s", filepath.Join(dir, "ql.db"))
@@ -35,7 +45,15 @@ func Test(t *testing.T) {
}
func TestMigrate(t *testing.T) {
dir := t.TempDir()
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
if err := os.RemoveAll(dir); err != nil {
t.Error(err)
}
}()
t.Logf("DB path : %s\n", filepath.Join(dir, "ql.db"))
db, err := sql.Open("ql", filepath.Join(dir, "ql.db"))
@@ -59,5 +77,5 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("CREATE TABLE t (Qty int, Name string);"))
}
+28 -33
View File
@@ -1,20 +1,20 @@
//go:build go1.9
// +build go1.9
package redshift
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/hashicorp/go-multierror"
"github.com/lib/pq"
)
@@ -36,7 +36,7 @@ type Config struct {
}
type Redshift struct {
isLocked atomic.Bool
isLocked bool
conn *sql.Conn
db *sql.DB
@@ -53,20 +53,18 @@ func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
@@ -126,21 +124,20 @@ func (p *Redshift) Close() error {
// Redshift does not support advisory lock functions: https://docs.aws.amazon.com/redshift/latest/dg/c_unsupported-postgresql-functions.html
func (p *Redshift) Lock() error {
if !p.isLocked.CompareAndSwap(false, true) {
if p.isLocked {
return database.ErrLocked
}
p.isLocked = true
return nil
}
func (p *Redshift) Unlock() error {
if !p.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
p.isLocked = false
return nil
}
func (p *Redshift) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
@@ -174,7 +171,7 @@ func (p *Redshift) Run(migration io.Reader) error {
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
@@ -216,19 +213,16 @@ func (p *Redshift) SetVersion(version int, dirty bool) error {
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
if version >= 0 {
query = `INSERT INTO "` + p.config.MigrationsTable + `" (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
err = multierror.Append(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
@@ -270,7 +264,7 @@ func (p *Redshift) Drop() (err error) {
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
err = multierror.Append(err, errClose)
}
}()
@@ -285,9 +279,6 @@ func (p *Redshift) Drop() (err error) {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
@@ -312,7 +303,11 @@ func (p *Redshift) ensureVersionTable() (err error) {
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
if err == nil {
err = e
} else {
err = multierror.Append(err, e)
}
}
}()
+9 -19
View File
@@ -22,25 +22,15 @@ import (
)
import (
"github.com/golang-migrate/migrate/v4/database"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
const (
pgPassword = "redshift"
)
var (
opts = dktest.Options{
Env: map[string]string{"POSTGRES_PASSWORD": pgPassword},
PortRequired: true,
ReadyFunc: isReady,
}
opts = dktest.Options{PortRequired: true, ReadyFunc: isReady}
specs = []dktesting.ContainerSpec{
{ImageName: "migrate/postgres8:8", Options: opts},
{ImageName: "postgres:8", Options: opts},
}
)
@@ -53,7 +43,7 @@ func pgConnectionString(host, port string) string {
}
func connectionString(schema, host, port string) string {
return fmt.Sprintf("%s://postgres:%s@%s:%s/postgres?sslmode=disable", schema, pgPassword, host, port)
return fmt.Sprintf("%s://postgres@%s:%s/postgres?sslmode=disable", schema, host, port)
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
@@ -128,7 +118,7 @@ func TestMigrate(t *testing.T) {
if err != nil {
t.Fatal(err)
}
dt.TestMigrate(t, m)
dt.TestMigrate(t, m, []byte("SELECT 1"))
})
}
@@ -201,7 +191,7 @@ func TestFilterCustomQuery(t *testing.T) {
t.Fatal(err)
}
addr := fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&x-custom=foobar", pgPassword, ip, port)
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", ip, port)
p := &Redshift{}
d, err := p.Open(addr)
if err != nil {
@@ -243,7 +233,7 @@ func TestWithSchema(t *testing.T) {
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres:%s@%v:%v/postgres?sslmode=disable&search_path=foobar", pgPassword, ip, port))
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", ip, port))
if err != nil {
t.Fatal(err)
}
@@ -257,7 +247,7 @@ func TestWithSchema(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if version != database.NilVersion {
if version != -1 {
t.Fatal("expected NilVersion")
}
@@ -378,10 +368,10 @@ func Test_computeLineFromPos(t *testing.T) {
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.ReplaceAll(input, "\n", "\r\n")
input = strings.Replace(input, "\n", "\r\n", -1)
}
if nonASCII {
input = strings.ReplaceAll(input, "FROM", "FRÖM")
input = strings.Replace(input, "FROM", "FRÖM", -1)
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
-18
View File
@@ -1,18 +0,0 @@
# rqlite
`rqlite://admin:secret@server1.example.com:4001/?level=strong&timeout=5`
The `rqlite` url scheme is used for both secure and insecure connections. If connecting to an insecure database, pass `x-connect-insecure` in your URL query, or use `WithInstance` to pass an established connection.
The migrations table name is configurable through the `x-migrations-table` URL query parameter, or by using `WithInstance` and passing `MigrationsTable` through `Config`.
Other connect parameters are directly passed through to the database driver. For examples of connection strings, see https://github.com/rqlite/gorqlite#examples.
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-connect-insecure` | n/a: set on instance | Boolean to indicate whether to use an insecure connection. Defaults to `false`. |
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table. Defaults to `schema_migrations`. |
## Notes
* Uses the https://github.com/rqlite/gorqlite driver
@@ -1 +0,0 @@
DROP TABLE IF EXISTS pets;
@@ -1,3 +0,0 @@
CREATE TABLE pets (
name string
);
@@ -1 +0,0 @@
DROP TABLE IF EXISTS pets;
@@ -1 +0,0 @@
ALTER TABLE pets ADD predator bool;
-328
View File
@@ -1,328 +0,0 @@
package rqlite
import (
"errors"
"fmt"
"io"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/rqlite/gorqlite"
)
func init() {
database.Register("rqlite", &Rqlite{})
}
const (
// DefaultMigrationsTable defines the default rqlite migrations table
DefaultMigrationsTable = "schema_migrations"
// DefaultConnectInsecure defines the default setting for connect insecure
DefaultConnectInsecure = false
)
// ErrNilConfig is returned if no configuration was passed to WithInstance
var ErrNilConfig = fmt.Errorf("no config")
// ErrBadConfig is returned if configuration was invalid
var ErrBadConfig = fmt.Errorf("bad parameter")
// Config defines the driver configuration
type Config struct {
// ConnectInsecure sets whether the connection uses TLS. Ineffectual when using WithInstance
ConnectInsecure bool
// MigrationsTable configures the migrations table name
MigrationsTable string
}
type Rqlite struct {
db *gorqlite.Connection
isLocked atomic.Bool
config *Config
}
// WithInstance creates a rqlite database driver with an existing gorqlite database connection
// and a Config struct
func WithInstance(instance *gorqlite.Connection, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
// we use the consistency level check as a database ping
if _, err := instance.ConsistencyLevel(); err != nil {
return nil, err
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
driver := &Rqlite{
db: instance,
config: config,
}
if err := driver.ensureVersionTable(); err != nil {
return nil, err
}
return driver, nil
}
// OpenURL creates a rqlite database driver from a connect URL
func OpenURL(url string) (database.Driver, error) {
d := &Rqlite{}
return d.Open(url)
}
func (r *Rqlite) ensureVersionTable() (err error) {
if err = r.Lock(); err != nil {
return err
}
defer func() {
if e := r.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
stmts := []string{
fmt.Sprintf(`CREATE TABLE IF NOT EXISTS %s (version uint64, dirty bool)`, r.config.MigrationsTable),
fmt.Sprintf(`CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version)`, r.config.MigrationsTable),
}
if _, err := r.db.Write(stmts); err != nil {
return err
}
return nil
}
// Open returns a new driver instance configured with parameters
// coming from the URL string. Migrate will call this function
// only once per instance.
func (r *Rqlite) Open(url string) (database.Driver, error) {
dburl, config, err := parseUrl(url)
if err != nil {
return nil, err
}
r.config = config
r.db, err = gorqlite.Open(dburl.String())
if err != nil {
return nil, err
}
if err := r.ensureVersionTable(); err != nil {
return nil, err
}
return r, nil
}
// Close closes the underlying database instance managed by the driver.
// Migrate will call this function only once per instance.
func (r *Rqlite) Close() error {
r.db.Close()
return nil
}
// Lock should acquire a database lock so that only one migration process
// can run at a time. Migrate will call this function before Run is called.
// If the implementation can't provide this functionality, return nil.
// Return database.ErrLocked if database is already locked.
func (r *Rqlite) Lock() error {
if !r.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
// Unlock should release the lock. Migrate will call this function after
// all migrations have been run.
func (r *Rqlite) Unlock() error {
if !r.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
// Run applies a migration to the database. migration is guaranteed to be not nil.
func (r *Rqlite) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if _, err := r.db.WriteOne(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// SetVersion saves version and dirty state.
// Migrate will call this function before and after each call to Run.
// version must be >= -1. -1 means NilVersion.
func (r *Rqlite) SetVersion(version int, dirty bool) error {
deleteQuery := fmt.Sprintf(`DELETE FROM %s`, r.config.MigrationsTable)
statements := []gorqlite.ParameterizedStatement{
{
Query: deleteQuery,
},
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
insertQuery := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (?, ?)`, r.config.MigrationsTable)
if version >= 0 || (version == database.NilVersion && dirty) {
statements = append(statements, gorqlite.ParameterizedStatement{
Query: insertQuery,
Arguments: []interface{}{
version,
dirty,
},
})
}
wr, err := r.db.WriteParameterized(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i].Query)}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(deleteQuery + "\n" + insertQuery)}
}
return nil
}
// Version returns the currently active version and if the database is dirty.
// When no migration has been applied, it must return version -1.
// Dirty means, a previous migration failed and user interaction is required.
func (r *Rqlite) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM " + r.config.MigrationsTable + " LIMIT 1"
qr, err := r.db.QueryOne(query)
if err != nil {
return database.NilVersion, false, nil
}
if !qr.Next() {
return database.NilVersion, false, nil
}
if err := qr.Scan(&version, &dirty); err != nil {
return database.NilVersion, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty, nil
}
// Drop deletes everything in the database.
// Note that this is a breaking action, a new call to Open() is necessary to
// ensure subsequent calls work as expected.
func (r *Rqlite) Drop() error {
query := `SELECT name FROM sqlite_master WHERE type = 'table'`
tables, err := r.db.QueryOne(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
statements := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
statement := fmt.Sprintf(`DROP TABLE %s`, tableName)
statements = append(statements, statement)
}
}
// return if nothing to do
if len(statements) <= 0 {
return nil
}
wr, err := r.db.Write(statements)
if err != nil {
for i, res := range wr {
if res.Err != nil {
return &database.Error{OrigErr: err, Query: []byte(statements[i])}
}
}
// if somehow we're still here, return the original error with combined queries
return &database.Error{OrigErr: err, Query: []byte(strings.Join(statements, "\n"))}
}
return nil
}
func parseUrl(url string) (*nurl.URL, *Config, error) {
parsedUrl, err := nurl.Parse(url)
if err != nil {
return nil, nil, err
}
config, err := parseConfigFromQuery(parsedUrl.Query())
if err != nil {
return nil, nil, err
}
if parsedUrl.Scheme != "rqlite" {
return nil, nil, fmt.Errorf("bad scheme: %w", ErrBadConfig)
}
// adapt from rqlite to http/https schemes
if config.ConnectInsecure {
parsedUrl.Scheme = "http"
} else {
parsedUrl.Scheme = "https"
}
filteredUrl := migrate.FilterCustomQuery(parsedUrl)
return filteredUrl, config, nil
}
func parseConfigFromQuery(queryVals nurl.Values) (*Config, error) {
c := Config{
ConnectInsecure: DefaultConnectInsecure,
MigrationsTable: DefaultMigrationsTable,
}
migrationsTable := queryVals.Get("x-migrations-table")
if migrationsTable != "" {
if strings.HasPrefix(migrationsTable, "sqlite_") {
return nil, fmt.Errorf("invalid value for x-migrations-table: %w", ErrBadConfig)
}
c.MigrationsTable = migrationsTable
}
connectInsecureStr := queryVals.Get("x-connect-insecure")
if connectInsecureStr != "" {
connectInsecure, err := strconv.ParseBool(connectInsecureStr)
if err != nil {
return nil, fmt.Errorf("invalid value for x-connect-insecure: %w", ErrBadConfig)
}
c.ConnectInsecure = connectInsecure
}
return &c, nil
}
-324
View File
@@ -1,324 +0,0 @@
package rqlite
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"testing"
"github.com/dhui/dktest"
"github.com/rqlite/gorqlite"
"github.com/stretchr/testify/assert"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
"github.com/golang-migrate/migrate/v4/dktesting"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
var defaultPort uint16 = 4001
var opts = dktest.Options{
Env: map[string]string{"NODE_ID": "1"},
PortRequired: true,
ReadyFunc: isReady,
}
var specs = []dktesting.ContainerSpec{
{ImageName: "rqlite/rqlite:7.21.4", Options: opts},
{ImageName: "rqlite/rqlite:8.0.6", Options: opts},
{ImageName: "rqlite/rqlite:8.11.1", Options: opts},
{ImageName: "rqlite/rqlite:8.12.3", Options: opts},
}
func isReady(ctx context.Context, c dktest.ContainerInfo) bool {
ip, port, err := c.Port(defaultPort)
if err != nil {
fmt.Println("error getting port")
return false
}
statusString := fmt.Sprintf("http://%s:%s/status", ip, port)
fmt.Println(statusString)
var readyResp struct {
Store struct {
Ready bool `json:"ready"`
} `json:"store"`
}
resp, err := http.Get(statusString)
if err != nil {
fmt.Println("error getting status")
return false
}
if resp.StatusCode != 200 {
fmt.Println("statusCode != 200")
return false
}
body, err := io.ReadAll(resp.Body)
if err != nil {
fmt.Println("error reading body")
return false
}
if err := json.Unmarshal(body, &readyResp); err != nil {
fmt.Println("error unmarshaling body")
return false
}
return readyResp.Store.Ready
}
func Test(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
r := &Rqlite{}
d, err := r.Open(connectString)
assert.NoError(t, err)
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
})
}
func TestMigrate(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?level=strong&disableClusterDiscovery=true&x-connect-insecure=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
driver, err := OpenURL(connectString)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
return
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
dt.TestMigrate(t, m)
})
}
func TestBadConnectInsecureParam(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("rqlite://%s:%s?x-connect-insecure=foo", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestBadProtocol(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
connectString := fmt.Sprintf("postgres://%s:%s/database", ip, port)
t.Logf("DB connect string : %s\n", connectString)
_, err = OpenURL(connectString)
assert.ErrorIs(t, err, ErrBadConfig)
})
}
func TestNoConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
_, err = WithInstance(db, nil)
assert.ErrorIs(t, err, ErrNilConfig)
})
}
func TestWithInstanceEmptyConfig(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
driver, err := WithInstance(db, &Config{})
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", DefaultMigrationsTable))
assert.NoError(t, err)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
})
}
func TestMigrationTable(t *testing.T) {
dktesting.ParallelTest(t, specs, func(t *testing.T, c dktest.ContainerInfo) {
ip, port, err := c.Port(defaultPort)
assert.NoError(t, err)
// gorqlite expects http(s) schemes
connectString := fmt.Sprintf("http://%s:%s?level=strong&disableClusterDiscovery=true", ip, port)
t.Logf("DB connect string : %s\n", connectString)
db, err := gorqlite.Open(connectString)
assert.NoError(t, err)
config := Config{MigrationsTable: "my_migration_table"}
driver, err := WithInstance(db, &config)
assert.NoError(t, err)
defer func() {
if err := driver.Close(); err != nil {
t.Fatal(err)
}
}()
m, err := migrate.NewWithDatabaseInstance(
"file://./examples/migrations",
"ql", driver)
assert.NoError(t, err)
t.Log("UP")
err = m.Up()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
_, err = db.WriteOne(`INSERT INTO pets (name, predator) VALUES ("franklin", true)`)
assert.NoError(t, err)
res, err := db.QueryOne(`SELECT name, predator FROM pets LIMIT 1`)
assert.NoError(t, err)
_ = res.Next()
// make sure we can use the migrated table
var petName string
var petPredator int
err = res.Scan(&petName, &petPredator)
assert.NoError(t, err)
assert.Equal(t, petName, "franklin")
assert.Equal(t, petPredator, 1)
t.Log("DOWN")
err = m.Down()
assert.NoError(t, err)
_, err = db.QueryOne(fmt.Sprintf("SELECT * FROM %s", config.MigrationsTable))
assert.NoError(t, err)
})
}
func TestParseUrl(t *testing.T) {
tests := []struct {
name string
passedUrl string
expectedUrl string
expectedConfig *Config
expectedErr string
}{
{
"defaults",
"rqlite://localhost:4001",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"configure migration table",
"rqlite://localhost:4001?x-migrations-table=foo",
"https://localhost:4001",
&Config{ConnectInsecure: DefaultConnectInsecure, MigrationsTable: "foo"},
"",
},
{
"configure connect insecure",
"rqlite://localhost:4001?x-connect-insecure=true",
"http://localhost:4001",
&Config{ConnectInsecure: true, MigrationsTable: DefaultMigrationsTable},
"",
},
{
"invalid migration table",
"rqlite://localhost:4001?x-migrations-table=sqlite_bar",
"",
nil,
"invalid value for x-migrations-table: bad parameter",
},
{
"invalid connect insecure",
"rqlite://localhost:4001?x-connect-insecure=baz",
"",
nil,
"invalid value for x-connect-insecure: bad parameter",
},
{
"invalid url",
string([]byte{0x7f}),
"",
nil,
"parse \"\\x7f\": net/url: invalid control character in URL",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
actualUrl, actualConfig, actualErr := parseUrl(tt.passedUrl)
if tt.expectedUrl != "" {
assert.Equal(t, tt.expectedUrl, actualUrl.String())
} else {
assert.Nil(t, actualUrl)
}
assert.Equal(t, tt.expectedConfig, actualConfig)
if tt.expectedErr == "" {
assert.NoError(t, actualErr)
} else {
assert.EqualError(t, actualErr, tt.expectedErr)
}
})
}
}
-12
View File
@@ -1,12 +0,0 @@
# Snowflake
`snowflake://user:password@accountname/schema/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
Snowflake is PostgreSQL compatible but has some specific features (or lack thereof) that require slightly different behavior.
## Status
This driver is not officially supported as there are no tests for it.
-371
View File
@@ -1,371 +0,0 @@
package snowflake
import (
"context"
"database/sql"
"errors"
"fmt"
"io"
nurl "net/url"
"strconv"
"strings"
"sync/atomic"
"github.com/golang-migrate/migrate/v4/database"
"github.com/lib/pq"
sf "github.com/snowflakedb/gosnowflake"
)
func init() {
db := Snowflake{}
database.Register("snowflake", &db)
}
var DefaultMigrationsTable = "schema_migrations"
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoPassword = fmt.Errorf("no password")
ErrNoSchema = fmt.Errorf("no schema")
ErrNoSchemaOrDatabase = fmt.Errorf("no schema/database name")
)
type Config struct {
MigrationsTable string
DatabaseName string
}
type Snowflake struct {
isLocked atomic.Bool
conn *sql.Conn
db *sql.DB
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if config.DatabaseName == "" {
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Snowflake{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Snowflake) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
password, isPasswordSet := purl.User.Password()
if !isPasswordSet {
return nil, ErrNoPassword
}
splitPath := strings.Split(purl.Path, "/")
if len(splitPath) < 3 {
return nil, ErrNoSchemaOrDatabase
}
database := splitPath[2]
if len(database) == 0 {
return nil, ErrNoDatabaseName
}
schema := splitPath[1]
if len(schema) == 0 {
return nil, ErrNoSchema
}
cfg := &sf.Config{
Account: purl.Host,
User: purl.User.Username(),
Password: password,
Database: database,
Schema: schema,
}
dsn, err := sf.DSN(cfg)
if err != nil {
return nil, err
}
db, err := sql.Open("snowflake", dsn)
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
px, err := WithInstance(db, &Config{
DatabaseName: database,
MigrationsTable: migrationsTable,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Snowflake) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
func (p *Snowflake) Lock() error {
if !p.isLocked.CompareAndSwap(false, true) {
return database.ErrLocked
}
return nil
}
func (p *Snowflake) Unlock() error {
if !p.isLocked.CompareAndSwap(true, false) {
return database.ErrNotLocked
}
return nil
}
func (p *Snowflake) Run(migration io.Reader) error {
migr, err := io.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
if pgErr, ok := err.(*pq.Error); ok {
var line uint
var col uint
var lineColOK bool
if pgErr.Position != "" {
if pos, err := strconv.ParseUint(pgErr.Position, 10, 64); err == nil {
line, col, lineColOK = computeLineFromPos(query, int(pos))
}
}
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: migr, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.ReplaceAll(s, "\r\n", "\n")
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Snowflake) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `DELETE FROM "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// Also re-write the schema version for nil dirty versions to prevent
// empty schema version for failed down migration on the first migration
// See: https://github.com/golang-migrate/migrate/issues/330
if version >= 0 || (version == database.NilVersion && dirty) {
query = `INSERT INTO "` + p.config.MigrationsTable + `" (version,
dirty) VALUES (` + strconv.FormatInt(int64(version), 10) + `,
` + strconv.FormatBool(dirty) + `)`
if _, err := tx.Exec(query); err != nil {
if errRollback := tx.Rollback(); errRollback != nil {
err = errors.Join(err, errRollback)
}
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Snowflake) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM "` + p.config.MigrationsTable + `" LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pq.Error); ok {
if e.Code.Name() == "undefined_table" {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Snowflake) Drop() (err error) {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer func() {
if errClose := tables.Close(); errClose != nil {
err = errors.Join(err, errClose)
}
}()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if err := tables.Err(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + t + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
}
return nil
}
// ensureVersionTable checks if versions table exists and, if not, creates it.
// Note that this function locks the database, which deviates from the usual
// convention of "caller locks" in the Snowflake type.
func (p *Snowflake) ensureVersionTable() (err error) {
if err = p.Lock(); err != nil {
return err
}
defer func() {
if e := p.Unlock(); e != nil {
err = errors.Join(err, e)
}
}()
// check if migration table exists
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := p.conn.QueryRowContext(context.Background(), query, p.config.MigrationsTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
// if not, create the empty migration table
query = `CREATE TABLE if not exists "` + p.config.MigrationsTable + `" (
version bigint not null primary key, dirty boolean not null)`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}

Some files were not shown because too many files have changed in this diff Show More