Compare commits

...
419 Commits
Author SHA1 Message Date
Dale Hui 83d01b7e1c Run go mod tidy 2018-10-10 16:16:23 -07:00
Dale Hui fc2b2e9c64 Use v4 for Go module support 2018-10-10 16:16:23 -07:00
Dale Hui 60b45ba428 Revert "Docker builds need to use dep to manage dependencies"
This reverts commit 4937cd088f.
2018-10-10 16:16:23 -07:00
Dale Hui 107db3f407 Revert "Remove support for Go modules"
This reverts commit 5e96539f55.
2018-10-10 16:15:33 -07:00
Dale Hui 4937cd088f Docker builds need to use dep to manage dependencies 2018-10-10 16:11:44 -07:00
Dale Hui 5e96539f55 Remove support for Go modules 2018-10-10 13:34:03 -07:00
Dale HuiandGitHub 708d67fa02 Merge pull request #112 from xiamengyu/master
Fix typo in comments
2018-10-10 13:26:36 -07:00
xiamengyu1 a95b13d804 Fix typo 2018-10-10 16:26:27 +08:00
Dale HuiandGitHub 1e60261d3f Merge pull request #110 from MikeFitzgerald/master
Fix postgres drop
2018-10-04 00:30:33 -07:00
Mike Fitzgerald 0d4f1c0212 Fix postgres drop 2018-10-04 01:22:39 -04:00
Dale HuiandGitHub 4d30c4c87f Add Go version to but report template 2018-09-19 03:10:08 -07:00
Dale Hui 6c02d65563 Update commented out imports to use v3 2018-09-14 14:12:07 -07:00
Dale Hui 16dea48c82 Fix module support by opting in to modules 2018-09-14 01:14:42 -07:00
Dale Hui f517029483 Test latest version of Go as well, but allow failures 2018-09-13 16:14:43 -07:00
Dale Hui 6f233b3b58 Update dependencies 2018-09-13 16:14:43 -07:00
Dale HuiandGitHub f8c85735f4 Merge pull request #98 from Kay-Zee/Kay-Zee/close-sql-db-connections-on-driver-close
Add sql.DB to Postgres and Mysql structs to allow closing in Close()
2018-09-07 01:26:43 -07:00
Kay-Zee 1e4c50b70d Add sql.DB to Postgres and Mysql structs to allow closing in Close() 2018-09-06 17:16:02 -07:00
Dale Hui 93d53a5ae8 Use Go 1.11 (with modules enabled on alpine linux 3.8) for docker builds 2018-09-04 22:18:49 -07:00
Dale Hui 16f2b1736e Support Go 1.11 and drop support for Go 1.9
- Support modules in Go 1.11
2018-09-04 19:11:19 -07:00
Dale Hui 4d60607b0b Update issue templates 2018-08-29 12:14:54 -07:00
Dale Hui beb09d85fa Upgrade dep from v0.4.1 to v0.5.0 2018-08-15 12:08:25 -07:00
Dale HuiandGitHub 8ed902364c Merge pull request #91 from stabacov/master
Fixes MySQL custom TLS connection failure.
2018-08-15 00:03:14 -07:00
Tatsuhito KATO 02c83fb38b more descriptive. 2018-08-15 02:42:18 +00:00
Tatsuhito KATO 9049e49f9e x-tls-cert and x-tls-key must set both. 2018-08-14 07:43:14 +00:00
Tatsuhito KATO 9d36dd9841 Add detail of tls query. 2018-08-14 07:42:10 +00:00
Tatsuhito KATO 9905791932 Fixes MySQL custom TLS connection failure. 2018-08-13 11:43:19 +00:00
Dale Hui dd00ca926e Add support for postgres 10 and drop support for postgres 9.2
https://www.postgresql.org/support/versioning/
2018-08-12 15:54:21 -07:00
Dale HuiandGitHub 4092e160a4 Merge pull request #85 from HaraldNordgren/improve_tests
Update tests for 'Migrate', 'Steps' and 'UpAndDown' to assert exact migration sequence
2018-08-08 15:57:04 -07:00
Dale HuiandGitHub 6092802b3a Merge pull request #88 from HaraldNordgren/error_output
Improve error output for missing or malformed '-database' and '-source' parameters
2018-08-08 15:24:39 -07:00
Harald Nordgren 42f074062b Update tests for 'Migrate', 'Steps' and 'UpAndDown' to assert exact migration sequence 2018-08-08 22:24:37 +02:00
Harald Nordgren 112191042b Improve error output for missing or malformed '-database' and '-source' parameters 2018-08-08 21:27:12 +02:00
Dale Hui 1566dd06cd Pass the error through instead of overriding it 2018-08-07 16:22:05 -07:00
Dale HuiandGitHub 251bf76be4 Merge pull request #89 from HaraldNordgren/improve_documentation
Document that '-source' is a required parameter
2018-08-07 15:59:20 -07:00
Harald Nordgren a6eddccfe5 Document that '-source' is a required parameter 2018-08-07 09:53:36 +02:00
Dale Hui 3bd43bedad Reference mattes/migrate now that golang-migrate/migrate has been detached
https://github.com/golang-migrate/migrate/issues/81
2018-08-07 00:17:39 -07:00
Dale HuiandGitHub 4750e57ce1 Merge pull request #82 from UnitedTraders/clickhouse-multiline
Add support for the multiline clickhouse queries
2018-08-06 04:09:10 -06:00
Anton Markelov df7ce674e9 add support for the multiline clickhouse queries 2018-08-03 16:08:47 +10:00
Dale Hui e968e8a5f0 Merge branch 'update_pkgs' 2018-07-24 18:30:53 -07:00
Dale HuiandGitHub c8705dd876 Merge pull request #78 from dhui/mysql_dsn
Allow DB connection URLs to contain encoded reserved URL characters
2018-07-24 18:22:04 -07:00
Dale Hui c7a29e331f Another round of package updates couldn't hurt... 2018-07-24 18:19:22 -07:00
Dale Hui ffcbbd5fea Override dependency on docker/distribution to fix build
https://github.com/golang/dep/issues/1890#issuecomment-404036309
2018-07-24 18:13:11 -07:00
Dale Hui ebf9c0bf91 Update package dependencies 2018-07-24 18:13:11 -07:00
Dale Hui df658e8fa8 Don't urlencode the username and password for MySQL
Addresses: https://github.com/golang-migrate/migrate/pull/69
2018-07-24 17:56:17 -07:00
Dale Hui b5edb8e50c Improve error messaging when URL parsing fails
- Improve docs around escaping/encoding reserved URL characters
    - Add tests for net/url Parse() and reserved characters to document understanding
    - Addresses: https://github.com/golang-migrate/migrate/issues/77
2018-07-24 17:55:58 -07:00
Dale Hui 97eb0d191b Fix broken GenerateAdvisoryLockId() test
- Add it to the test suite to ensure that it doesn't break again
2018-07-24 17:55:58 -07:00
Dale Hui 857d7a620d Revert "Merge pull request #69 from bcho/fix/mysql-url-password-encode"
This reverts commit 78d696c1e5, reversing
changes made to 18583d5a91.
2018-07-24 17:55:58 -07:00
Dale HuiandGitHub fad64ed381 Merge pull request #76 from agquick/master
Support for multiple statements in Cassandra cql migration files
2018-07-24 16:02:44 -07:00
Alan Graham 4244ebea6a Add detail to README for Cassandra flag, improve if block 2018-07-24 19:30:44 +00:00
Alan Graham 327822c29a Added Cassandra URL flag to enable multi statement migrations 2018-07-20 13:27:54 +00:00
Dale HuiandGitHub feea45c205 Merge pull request #75 from LUSHDigital/master
Lock go-sql-driver/mysql to v1.4.0
2018-07-17 16:42:01 -07:00
Alan Graham c4b81d1210 Support for multiple statements in Cassandra cql migration files based on semi-colon and newline. 2018-07-17 15:19:54 +00:00
Dan Richards 3988074cae Lock digest. 2018-07-16 12:09:29 +01:00
Dan Richards ad2b69e115 Typo 2018-07-16 12:08:24 +01:00
Dan Richards f9eb960848 Just lock go-sql-driver/mysql. 2018-07-16 12:06:27 +01:00
Dan Richards 5927d6e5f4 Locked go-sql-driver/mysql to v1.4.0 2018-07-16 12:01:52 +01:00
Dale Hui 6bb3e494e5 Disable CGO to build binaries statically on the building platform
- Previously, only cross-compiled binaries were built statically
    - Addresses: https://github.com/golang-migrate/migrate/issues/45
2018-07-14 13:31:59 -07:00
Dale Hui 5d5d15e7a3 Link to comparison of transactional DDLs 2018-07-09 19:25:29 -07:00
Dale Hui b742ec3de4 Use unix line endings 2018-07-09 19:21:40 -07:00
Dale Hui 75f0aade0d Reduce docker image size by removing unnecessary chown
https://github.com/moby/moby/issues/5505
2018-07-09 14:52:59 -07:00
Dale HuiandGitHub 3872c8f41d Merge pull request #71 from qdentity/postgres-parse-error
Add postgres lib/pq error parsing
2018-07-09 12:26:24 -07:00
Damir Vandic ae2fd4fa7b Support postgres non-ascii error parsing 2018-07-09 11:39:29 +02:00
Damir Vandic 3eb26a65d3 Add postgres lib/pq error parsing 2018-07-07 12:17:38 +02:00
Dale Hui 157433893c Update CLI installation docs to reduce confusion
https://github.com/golang-migrate/migrate/issues/64#issuecomment-402300146
2018-07-05 10:55:44 -07:00
Dale Hui 236dac0c96 Add latest tag to docker image during build 2018-07-03 14:09:26 -07:00
Dale HuiandGitHub 78d696c1e5 Merge pull request #69 from bcho/fix/mysql-url-password-encode
prevent net/url encoding the user password
2018-06-29 14:17:45 -07:00
Chao Dengandhbc 48a4062f75 prevent net/url encoding the user password 2018-06-29 21:07:10 +08:00
Dale Hui 18583d5a91 Update docs
- Addresses: https://github.com/golang-migrate/migrate/issues/67
    - Add badge for supported Go versions
    - Suggest dep for dependency management
2018-06-27 11:45:54 -07:00
Dale HuiandGitHub 642913dbf4 Merge pull request #66 from dadlerj/cli-readme
Update CLI usage in readme
2018-06-26 13:25:18 -07:00
dadlerj 2ca5473e52 Update CLI usage in readme 2018-06-26 09:16:04 -07:00
Dale Hui 6070aa8281 Push latest image to docker hub 2018-06-19 01:42:37 -07:00
Dale HuiandGitHub 78dc468555 Merge pull request #47 from dhui/docker_badge
Add badge for num docker pulls
2018-06-19 01:33:06 -07:00
Dale Hui bcd996f3df Use environment variables since there's no way to pass secure data to deploy script provider 2018-06-19 00:39:54 -07:00
Dale Hui a1d959d0e9 Use latest version of docker 2018-06-18 23:07:54 -07:00
Dale Hui 71e97bccc7 Fix TravisCI conditional release 2018-06-18 21:42:31 -07:00
Dale Hui 3239b18671 Fix time format flag to be backwards compatible
https://github.com/golang-migrate/migrate/pull/17#issuecomment-397960867
2018-06-18 00:06:17 -07:00
Dale Hui c3794da4ed Make -ext flag required for the create command 2018-06-17 23:18:23 -07:00
Dale Hui 872a561495 Fix broken CLI build 2018-06-17 23:16:38 -07:00
Dale Hui d1adc8b15f Add golang/x/tools (godoc/vfs) as a dependency 2018-06-17 19:59:39 -07:00
Dale HuiandGitHub e7d20b35ee Merge pull request #57 from SamWhited/vfs
source/vfs: add virtual file system source
2018-06-17 19:22:06 -07:00
Dale Hui 21f566a60a Add docker hub credentials for uploading docker image 2018-06-17 17:42:03 -07:00
Sam Whited 7562e9ad5d source/vfs: add virtual file system source
Fixes #56
2018-06-17 19:35:13 -05:00
Dale Hui 0f16f7f6a2 Rename sources to reflect updated build constraint names 2018-06-12 16:45:56 +08:00
Dale Hui dfa9fc05cb Use correct build contraints (must be alphanumeric)
Fixes: https://github.com/golang-migrate/migrate/issues/52
2018-06-12 16:18:27 +08:00
Dale Hui c4e855b58c List source and database drivers in help text 2018-06-12 16:17:52 +08:00
Dale Hui 0d9f636356 Use dep to build docker image 2018-06-11 19:18:05 +08:00
Dale HuiandGitHub 387f79be13 Merge pull request #48 from dhui/static_build
Build binaries statically
2018-06-04 09:24:35 -07:00
Dale Hui 87f3e4a910 Build binaries statically
Addresses: https://github.com/golang-migrate/migrate/issues/45
2018-06-04 19:27:37 +08:00
Dale Hui 01ea635328 Add badge for num docker pulls 2018-06-04 18:54:43 +08:00
Dale HuiandGitHub 3d1bb8a4b4 Merge pull request #13 from Teddy-Schmitz/postgres-wait
postgres driver should wait for lock
2018-05-31 19:04:30 -07:00
Teddy Schmitz 046d1cb767 postgres driver should wait for lock 2018-06-01 09:11:24 +08:00
Dale Hui 3d7b06e24c Override google cloud package version (was pinned to v0.19.0 by github.com/fsouza/fake-gcs-server) 2018-05-31 15:26:18 -07:00
Dale Hui 063b69db2f Update OpenCensus dependency 2018-05-31 15:05:31 -07:00
Dale Hui c0b0829d5a Use "vanity" import to allow forks to build
https://docs.travis-ci.com/user/languages/go/#Go-Import-Path
2018-05-31 14:57:38 -07:00
Dale Hui 3cf42dd36d Don't need to re-solve package dependencies every time. Only use Gopkg.lock
Reference: https://github.com/golang/dep/blob/master/docs/ensure-mechanics.md#-no-vendor-and--vendor-only
2018-05-31 14:57:38 -07:00
Dale Hui 89e558ee4f Don't cache go source since dependencies are managed by dep now 2018-05-31 14:57:38 -07:00
Dale Hui f277ef7428 Update github.com/go-sql-driver/mysql and github.com/gocql/gocql 2018-05-31 14:57:38 -07:00
Dale Hui b7647a749e Use the master branch for go-sql-driver/mysql since there hasn't been a release in a yr and a half 2018-05-31 14:57:38 -07:00
Dale Hui 4cc6aeb274 Update gocql package 2018-05-31 14:57:38 -07:00
Dale Hui 3de903f51f Use latest minor revision of Go 1.10 in TravisCI 2018-05-31 14:57:38 -07:00
Dale Hui 4b4f49b1eb Update docs regarding usage of dep 2018-05-31 14:57:38 -07:00
Dale Hui 4407720b95 Remove "deps" target from Makefile 2018-05-31 14:57:38 -07:00
Dale Hui 36dc939f91 Use dep for TravisCI builds 2018-05-31 14:57:38 -07:00
Dale Hui 40cd326ed7 Use dep to manage dependencies 2018-05-31 14:57:38 -07:00
Dale Hui a890d44d2f CockroachDB driver improvements
- Correctly return database.ErrLocked when the DB is locked
    - Group imports
    - gofmt goodness
2018-05-31 14:44:48 -07:00
Dale HuiandGitHub c98426505e Merge pull request #43 from wmetaw/master
Fixed unkeyed fields in file.go for GoogleAppEngine
2018-05-31 14:23:14 -07:00
Ryo Takashima 6fa8039b55 Fixed unkeyed fields for GoogleAppEngine 2018-05-31 15:40:15 +09:00
Dale HuiandGitHub faf4307fcc Merge pull request #32 from kenjones-cisco/feature/cassandra-withinstance
Minor tweaks to remove duplication
2018-05-17 01:01:47 -07:00
kenjones 1512e41e41 Minor tweaks to remove duplication
Adds missing connection close for cassandra tests

Revert to default timeout of 600ms
2018-05-16 20:09:12 -04:00
Dale Hui 55a25c5e0e Cassandra DB driver test improvements
- shorten connect timeout to allow more connection retry attempts
    - close session after it's created
    - group imports
2018-05-16 15:12:36 -07:00
Dale HuiandGitHub f815731412 Merge pull request #30 from Decemberlabs/master
Add WithSession helper for Cassandra driver
2018-05-16 14:26:12 -07:00
Andrés Rodríguez d607064fd2 Switch Cassandra's package lock to instance lock 2018-05-16 16:26:23 -03:00
Andrés Rodríguez 8b0f876416 Rename Cassandra's WithSession to WithInstance 2018-05-16 16:23:21 -03:00
Andrés Rodríguez bbd8f410d9 Correct method name in comment 2018-05-16 10:54:56 -03:00
Dale Hui fb3d809e81 Cleanup CLI docs 2018-05-12 15:25:19 -07:00
Dale HuiandGitHub 95db494b30 Merge pull request #28 from tayhalla/patch-1
Update README.md
2018-05-12 14:58:12 -07:00
Taylor HallidayandGitHub 5ebdb74ba8 Update README.md
There's no homebrew support. Even the referenced TODO says so, so not sure why it's in the Readme. Let's encourage others to support the development of it rather than think the support for it is broken.
2018-05-12 07:22:21 -07:00
Dale HuiandGitHub 07c20c5357 Merge pull request #27 from Its-Alex/master
Fix dockerfile version and deps
2018-05-10 01:13:05 -07:00
ItsAlex 31daa8dad3 fix dockerfile version and deps
Signed-off-by: Alexandre MARRE <me@itsalex.fr>
2018-05-10 09:12:23 +02:00
Dale HuiandGitHub a0b03c3a68 Merge pull request #24 from Its-Alex/master
Dockerization
2018-05-09 11:22:34 -07:00
ItsAlex fb7bb4f87a add dockerization Fix #23
Signed-off-by: Alexandre MARRE <me@itsalex.fr>
2018-05-09 09:48:20 +02:00
Dale HuiandGitHub 81e9fff605 Merge pull request #22 from thurt/add-ref-to-github
add ref (SHA, branch, or tag) to the github migration source
2018-05-04 10:13:18 +02:00
thurt e6baca2ffb add ref (SHA, branch, or tag) to the github migration source 2018-05-04 03:16:12 +00:00
Dale HuiandGitHub 2b28a0bc0a Merge pull request #17 from bobmeister/cli_create_yyyymmddhhmmss_migration_files
-datetime option to create files in yyyymmddhhmmss format
2018-04-17 10:50:48 -07:00
Robert Haupt 1f389f1422 Removed commented code 2018-04-17 11:17:10 -06:00
Robert Haupt 8cdee40695 Use Go time format string if version other than timestamp is desired. 2018-04-17 11:09:08 -06:00
Robert Haupt c6c6874d72 Added -datetime option to create up/down files with version in yyyymmddhhmmss format. 2018-04-16 11:12:06 -06:00
Andrés Rodríguez 78c47074a3 Add WithSession helper for Cassandra driver
Also changes method receivers names for the Cassandra struct to "c"
since "p" makes no sense in this context.
2018-03-13 15:49:08 -03:00
Dale Hui 22f249514d Merge branch 'postgres-lock-fix'
Addresses: https://github.com/golang-migrate/migrate/pull/4
2018-02-20 16:12:54 -08:00
Dale HuiandGitHub 8f6826c9e3 Merge pull request #8 from rdallman/fix-lockyloo-golang
Fix lockyloo golang
2018-02-20 15:51:18 -08:00
Dale Hui 1519c59188 Rename Postgres.db to Postgres.conn 2018-02-20 15:40:20 -08:00
Dale HuiandGitHub a9176e9969 Merge branch 'master' into fix-lockyloo-golang 2018-02-20 15:37:20 -08:00
Dale Hui ab7dfb38ac Merge remote-tracking branch 'origin/master' into postgres-lock-fix 2018-02-20 15:32:34 -08:00
Dale Hui aa21a21647 Support Go 1.10 and drop support for 1.8
- Test using latest minor versions of the last 2 major (supported) releases
        - TravisCI's gimme doesn't support Go 1.10.x until a minor version is released
    - Explicitly use YAML strings so Go versions aren't interpreted as numbers
2018-02-20 15:20:43 -08:00
Reed Allman 094bad431e s/db/conn/ 2018-02-09 15:14:05 -08:00
Reed Allman a8ef2b8cfe update CI and build tags to go1.9 2018-02-09 00:32:47 -08:00
Reed Allman 636c911d33 fixes mysql lock failure
I believe this closes #297 as well.

I have been working on adding testing of migrations and it requires acquiring
the lock in mysql multiple times to go up and down. After nailing this down to
GET_LOCK returning a failure for every subsequent GET_LOCK call after the
first, I decided it was time to rtfm and lo and behold:

https://dev.mysql.com/doc/refman/5.7/en/miscellaneous-functions.html#function_release-lock

RELEASE_LOCK will not work if called from a different thread than GET_LOCK.
The simplest solution using the golang database/sql pkg appears to be to just
get a single conn to use for every operation. since migrations are happening
sequentially, I don't think this will be a performance hit (possible
improvement). now calling Lock() and Unlock() multiple times will work;
prior to this patch, every call to RELEASE_LOCK would fail. this required
minimal changes to use the *sql.Conn methods instead of the *sql.DB methods.

other changes:

* upped time out to 10s on GET_LOCK, 1s timeout can too easily leave us in a
state where we think we have the lock but it has timed out (during the
operation).
* fixes SetVersion which was not using the tx it was intending to, and fixed a
bug where the transaction could have been left open since Rollback or Commit
may never have been called.

I added a test but it does not seem to come up in the previous patch, at least
easily, I tried some shenanigans. At least, this behavior should be fixed with
this patch and hopefully the test / comment is advisory enough.

thank you for maintaining this library, it has been relatively easy to
integrate with and most stuff works (pg works great :)
2018-02-09 00:27:01 -08:00
Dale Hui 53dd06057f Add release badge 2018-01-21 02:07:25 -08:00
Dale Hui 1b840d9c79 Remove unused invalid formatter 2018-01-21 02:04:36 -08:00
Dale Hui 012f051f2e Use cached badges when possible and make badges reflect the state of the master branch 2018-01-21 01:40:41 -08:00
Dale Hui 8684cad808 Update README to reflect the current status of the sqlite driver 2018-01-21 00:20:10 -08:00
Dale Hui 15611d4559 Fix cli builds
- Removed sqlite3 support from binaries
        - It was never working anyways: https://github.com/mattes/migrate/issues/244
    - Don't use cgo as is requires a cross-compiler to build for other platforms
        - cgo was originally added for this reason: https://github.com/mattes/migrate/pull/238#issuecomment-306291988
2018-01-20 23:59:36 -08:00
Dale Hui d542e144dc Manually fetch github.com/kshvakov/clickhouse 2018-01-20 10:30:57 -08:00
Dale Hui f8fd159fbc Fork packagecloud releases 2018-01-20 02:10:22 -08:00
Dale Hui 2eb9570b52 Build release binaries (using G 1.9) and upload to Github using new key 2018-01-20 01:49:02 -08:00
Dale Hui f5ae4b42db Support creating migrations using sequences in addition to timestamps 2018-01-20 00:29:06 -08:00
Dale Hui 24dd870559 MySQL and Postgres db driver test improvements
- Log unexpected connection errors while waiting for the docker image to start
    - Don't leave connections open
2018-01-20 00:11:35 -08:00
Dale Hui 97ef1b8393 Remove redundant delay since the readyFn is sufficient 2018-01-20 00:11:35 -08:00
Dale Hui bdb332a5f8 Remove unused slowReader 2018-01-19 22:38:32 -08:00
Dale Hui 8cbdc28e66 Source code needs to be kept for goveralls to use but removed before caching in travis-ci 2018-01-19 20:53:20 -08:00
Dale Hui 4ebf080da8 Speed up travis-ci builds by caching dependencies 2018-01-19 20:32:15 -08:00
Dale Hui 53606261bf Make coverage directory configurable 2018-01-19 20:32:15 -08:00
Dale Hui 50d049d5f5 Only use coverage profiles from test runs to create combined coverage profile.
- Prevent issue where the combined coverage profile is combined with itself
2018-01-19 18:04:46 -08:00
Daniel CormierandDale Hui 395391525b Changed source.Up from untyped to a typed const
Changed source.Up from being an untyped string const to being a typed source.Direction const.
2018-01-19 16:24:25 -08:00
Dale Hui 59f3f6d6c2 Increase test timeout from default 10m to 20m since some tests are slow as molasses in January
- Example build being killed by Go's test binary: https://travis-ci.org/golang-migrate/migrate/jobs/331027558
2018-01-19 15:42:02 -08:00
Dale Hui 12cc536541 Drop support for Go 1.7
- Only the latest 2 major release version is supported. https://golang.org/doc/devel/release.html#policy
2018-01-19 15:36:13 -08:00
Dale Hui 51ec421ad8 Remove debugging print from mysql driver test 2018-01-19 15:13:07 -08:00
Dale Hui 1cf095c51d Update more references to github.com/mattes/migrate 2018-01-19 14:59:27 -08:00
Dale Hui 7c9a261dc0 Fix MySQL docker image connection testing 2018-01-19 14:50:31 -08:00
Dale Hui 31367fbc97 Add tip about dealing with docker client API version issues 2018-01-19 14:31:56 -08:00
Dale Hui 7159f40304 Update all *DockerContainer receivers to handle nil
- Use errors.New() instead of fmt.Errorf() if no formatter string is needed
    - Spacing changes due to `go fmt`
2018-01-19 14:22:55 -08:00
Dale Hui 042e307bef Ignore vendor dir 2018-01-19 10:57:17 -08:00
Dale Hui f584949470 Update imports to reference fork.
Thanks `make rewrite-import-paths`!
2018-01-19 10:56:55 -08:00
Dale Hui 303cd74e5f Fix Docker install on TravisCi by allowing the docker-ce package to be downgraded
- Started happening since TravisCI updated docker-ce https://docs.travis-ci.com/user/build-environment-updates/2017-12-12/
    - Can't use v17.09 since the git tag doesn't exist yet
2018-01-19 02:28:41 -08:00
Dale Hui 3b61abf20a Update some examples and badges in README to reference new fork 2018-01-19 02:07:34 -08:00
Matthias KadenbachandGitHub d23f71b03c Merge pull request #305 from zikes/master
use DELETE FROM to avoid schema changes within transaction
2017-12-08 13:48:26 -08:00
Jason Hutchinson c156f5f3a1 replace TRUNCATE with DELETE FROM to avoid schema changes within transaction 2017-11-02 16:59:26 -05:00
Teddy Schmitz 155a8b241c Use db.Conn to fix postgres lock/unlock 2017-10-27 10:56:10 +08:00
Matthias KadenbachandGitHub 69472d5f5c Merge pull request #301 from shaneodonnell/master
Issue 247: Removed '(always included)'
2017-10-24 11:00:00 -07:00
Shane O'Donnell 4ac3ee9d75 Removed '(always included)' 2017-10-23 13:27:24 -04:00
Matthias KadenbachandGitHub 5b98c13eff Merge pull request #281 from ashoda/support_multiple_statements
(Spanner) Add support for multiple statements in one migration step
2017-10-09 11:03:59 -07:00
Ashod Ayanyan 28cb030ffe Add support for multiple DDL/SQL statements in one migration file 2017-09-18 15:43:46 -07:00
Matthias KadenbachandGitHub be1b075605 add go 1.9 for tests 2017-09-18 14:55:49 -07:00
Matthias KadenbachandGitHub 62cddb9e74 Merge pull request #278 from JensRantil/cassandra-auth
feat(cassandra): support for user/pw authentication
2017-09-18 10:08:58 -07:00
Matthias KadenbachandGitHub eabba5b316 Merge pull request #277 from JensRantil/cleanup
Various `gofmt`ing
2017-09-18 10:08:34 -07:00
Jens Rantil a5e584ce22 feat(cassandra): support for user/pw authentication 2017-09-17 13:32:00 +02:00
Jens Rantil d25732a53a sty: avoid unnecessary parenthesis
Golang best-practise. Cleanup made by `gofmt`.
2017-09-17 13:30:08 +02:00
Jens Rantil 701bd68731 style(cassandra): whitespace fixes
According to `gofmt`.
2017-09-17 13:26:30 +02:00
Jens Rantil 51acd3f3ad style: order imports lexicographically
Best-practise according to `gofmt`.
2017-09-17 13:26:23 +02:00
Matthias KadenbachandGitHub 804f9c2d4f Merge pull request #273 from maknahar/master
Fixed a couple of typos
2017-08-14 14:01:06 -07:00
maknahar 81c0a17f20 Fixed a couple of typos 2017-08-14 17:55:33 +05:30
Matthias KadenbachandGitHub 2570d5866d Merge pull request #272 from twrobel3/bugfix/cockroachTxnSig
Update cockroachDB Txn signatures
2017-08-11 17:11:41 -07:00
Taylor Wrobel 2c0f4f588e Update cockroachDB Txn signatures
cockroach-go made a backwards-incompatible change to their transaction
function signatures, which was causing the cockroach instrumentation
to fail.

Updates the signature to match the cockraoch-go change.
2017-08-10 22:56:41 -07:00
Matthias KadenbachandGitHub 8b5365e62c Merge pull request #267 from nicolas33/FAQ-extension
Faq extension
2017-07-30 15:05:05 -07:00
Nicolas Sebrecht d837bcc60c FAQ: add required extension
Github-fix: #265
Signed-off-by: Nicolas Sebrecht <nicolas.sebrecht@1001pneus.fr>
2017-07-28 14:30:10 +02:00
Nicolas Sebrecht bfe502654c FAQ: update format from dos to unix
Signed-off-by: Nicolas Sebrecht <nicolas.sebrecht@1001pneus.fr>
2017-07-28 14:24:46 +02:00
Matthias KadenbachandGitHub 6632d110b6 Merge pull request #264 from twrobel3/feature/migrations-readme
Fill out MIGRATIONS.md with migration format and best practice notes
2017-07-25 23:05:42 -07:00
Taylor Wrobel 341c35b1b8 Fill out MIGRATIONS.md with migration format and best practice notes 2017-07-19 00:18:02 -07:00
Matthias KadenbachandGitHub 8e6084bc9c Merge pull request #263 from twrobel3/feature/crdb-readme
Add cockroachdb README and add force lock functionality
2017-07-12 08:48:03 -07:00
Taylor Wrobel 40a40613cb Add cockroachdb README and add ForceLock functionality
Adds a readme to the cockroachdb database package, following the
postgres readme style.

Also adds the ability to force acquisition of the migration lock
via a connect URL parameter/WithInstance config, to allow for fixing
cases where an implementation error causes the schema lock to not
be released.

Lastly, tweaks the CLI readme to include information on building a
CLI for databases other than postgres.
2017-07-11 21:52:02 -07:00
Watanabe MasaruandMatthias Kadenbach 23a2745a3a fix spanner ensure migration table 2017-07-09 10:57:10 -07:00
Matthias KadenbachandGitHub 88c6a0a752 Merge pull request #253 from kshvakov/master
Add ClickHouse driver
2017-07-08 10:11:23 -07:00
Kirill ShvakovandGitHub 1f62e3f142 Merge branch 'master' into master 2017-07-08 13:50:51 +03:00
Kirill Shvakov 8424e946e4 remove deprecated group from Travis file 2017-07-08 13:49:23 +03:00
Matthias KadenbachandGitHub 66a0448698 Merge pull request #255 from twrobel3/feature/cockroachdb
Add CockroachDB Support
2017-07-08 00:26:12 -07:00
Taylor Wrobel 2cab359222 Remove debugging output from cockroachDB tests 2017-07-07 23:13:55 -07:00
Taylor Wrobel aebc0c3cdd Merge port mapping helper functions into one 2017-07-07 23:12:07 -07:00
Taylor Wrobel c2925c40c7 Add tests for CockroachDB 2017-07-07 23:12:07 -07:00
Taylor Wrobel 88115dedbf Support running docker with commands and handling multiple exposed ports
Adds the ability to specify a series of commands to run as part
of the docker image execution, and allows for retrieving a mapping
of an exposed via the port bound within the container.
2017-07-07 23:12:07 -07:00
Taylor Wrobel 1d8a881941 Add CockroachDB Support
Adds support for CockroachDB.  Cockroach uses the postges wire
protocol and has a large amount of common SQL functionality shared
with Postgres, so much of the postgres code was able to be copied
and modified.

Since the protocol is used in determining the driver, and the
Postgres protocol is also used by Cockroach, new connect string
prefixes were added: cockroach:// cockroachdb:// and
crdb-postgres://.  These fake protocol strings are replaced in
the connect function with the correct `postgres://` protocol.

TODO: Tests needed (Cockroach has a docker image, so this shouldn't
be too hard)
2017-07-07 23:12:07 -07:00
Matthias KadenbachandGitHub 8439d7195f Merge pull request #262 from twrobel3/feature/fixTravisBuild
Update docker dependency version to fix Tavis builds
2017-07-07 22:45:46 -07:00
Matthias KadenbachandGitHub f0f8b2c85c Merge pull request #260 from martin-magakian/patch-1
Update README.md
2017-07-03 22:21:55 -07:00
Martin MagakianandGitHub 35af42a0dd Update README.md 2017-07-03 15:45:57 +02:00
Martin MagakianandGitHub 56a56527c6 Update README.md
Late edit I promise...
2017-07-03 15:45:41 +02:00
Martin MagakianandGitHub 1485e567f1 Update README.md
Some format :-p
2017-07-03 15:35:43 +02:00
Martin MagakianandGitHub 180c57747d Update README.md
Forget package
2017-07-03 15:34:43 +02:00
Martin MagakianandGitHub 2361a7aa24 Update README.md
Code snippet don't compile.
It's missing a step (see https://github.com/mattes/migrate/blob/master/source/go-bindata/go-bindata_test.go)
2017-07-03 15:29:15 +02:00
Taylor Wrobel 4f9422ec85 Update docker dependency version to fix Tavis builds
Tavis has updated its docker images, and as part of the update,
no longer includes support for the (deprevated) docker-engine
package, moving instead to docker-ce.

See https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch
for details of the package change.

This updates the travis configuration to pin the docker library
version to 17.05.0 and installs the corresponding version of
docker-ce in the install script.
2017-06-24 10:37:08 -07:00
kshvakov 9d17987b8c Travis CI: using the previous Ubuntu Trusty images (https://blog.travis-ci.com/2017-06-21-trusty-updates-2017-Q2-launch) 2017-06-23 12:16:14 +03:00
kshvakov 94056e51cc postgres: fix SetVersion. All requests were executed in different connections so commit/rollback doesn't work correctly 2017-06-23 11:13:54 +03:00
Kirill Shvakov 8cd9761d29 fix: set current database 2017-06-21 23:10:43 +03:00
kshvakov e86f4a201c change include path from github.com/kshvakov to github.com/mattes 2017-06-21 17:24:55 +03:00
kshvakov 9341404de0 move ensureVersionTable to init 2017-06-21 17:14:43 +03:00
kshvakov b031b22c80 Add WithInstance 2017-06-21 17:12:05 +03:00
kshvakov 1440a070c3 Add WithInstance 2017-06-21 17:06:04 +03:00
kshvakov 14636b77fb Add ClickHouse driver 2017-06-21 15:58:46 +03:00
Matthias KadenbachandGitHub da4160ce04 Merge pull request #251 from christianklotz/spanner-driver
Add Spanner driver
2017-06-20 21:52:00 -07:00
Christian Klotz eccc3a26d2 Skip test if SPANNER_DATABASE isn’t set 2017-06-20 17:53:41 +01:00
Christian KlotzandGitHub 51afcccb6b Merge branch 'master' into spanner-driver 2017-06-14 21:02:59 +01:00
Stephanie NgandChristian Klotz 8f7d3efb1a add create command to cli 2017-06-14 21:59:11 +02:00
BenandChristian Klotz 13a818e3b5 Fix typo forgotton => forgotten 2017-06-14 21:59:11 +02:00
Matthias KadenbachandChristian Klotz 5e049b9fb7 enable cgo 2017-06-14 21:59:11 +02:00
maxvwandChristian Klotz 33d5038bd3 adapted the sqlite driver for v3 (#165) 2017-06-14 21:58:37 +02:00
Fuyuan BieandChristian Klotz 39bb76224d Fix parenthesis problem. 2017-06-14 21:57:37 +02:00
Fuyuan BieandChristian Klotz 9632882b69 Use both db name and migration table name for lock hash 2017-06-14 21:57:37 +02:00
Christian Klotz 803598293e Add basic Google Spanner test 2017-06-14 19:46:10 +02:00
Matthias KadenbachandGitHub 50068870a1 Merge pull request #249 from recurza/master
Add create command to CLI
2017-06-10 10:25:06 -07:00
Stephanie Ng 9a19439e31 add create command to cli 2017-06-10 20:20:35 +08:00
Matthias KadenbachandGitHub d8dd1bc427 Merge pull request #248 from biefy/master
Solve lock contention problem when upgrading multiple logical MySQL databases backed by a single physical database.
2017-06-09 17:08:54 -07:00
Matthias KadenbachandGitHub c1134be427 Merge pull request #242 from benmoss/master
Fix typo forgotton => forgotten
2017-06-05 13:11:58 -07:00
Matthias KadenbachandGitHub 2aab3394cf Merge pull request #243 from mattes/mattes-patch-1
enable cgo
2017-06-05 13:11:27 -07:00
Ben 39a5a37822 Fix typo forgotton => forgotten 2017-06-05 12:14:10 -04:00
Jolan MalassigneandChristian Klotz 89879968bb add cassandra driver and function to retrieve networkSettings to get port bound to 9042 2017-06-05 11:03:49 +01:00
Pavlina DrososandChristian Klotz 6ecd671cfc Add code sample for use with existing MySQL client with multistatements=true 2017-06-05 11:02:51 +01:00
MaxandChristian Klotz 2f50c9c52b added file cli/build_sqlite3.go 2017-06-05 11:02:51 +01:00
maxvwandChristian Klotz 386ce008d9 adapted the sqlite driver for v3 (#165) 2017-06-05 11:02:51 +01:00
Christian Klotz 3d6c78875f Add example migrations 2017-06-05 10:50:14 +01:00
Matthias KadenbachandGitHub a9bcd83ff0 enable cgo 2017-06-05 00:59:53 -07:00
Matthias KadenbachandGitHub bf3f0f133e Merge pull request #240 from tinysquare/master
Add cassandra driver
2017-06-04 21:27:48 -07:00
Matthias KadenbachandGitHub 2aa16f5f66 Merge branch 'master' into master 2017-06-04 21:27:38 -07:00
Matthias KadenbachandGitHub 63f788f07e Merge pull request #239 from pdrosos/update-mysql-documentation-for-use-with-existing-db-client
Add documentation for use with existing MySQL client with multiStatements=true
2017-06-04 21:23:57 -07:00
Matthias KadenbachandGitHub c3d61852ee Merge pull request #238 from maxvw/v3-sqlite
adapted the sqlite driver for v3 (#165)
2017-06-04 21:21:55 -07:00
Christian Klotz 2742b9c467 Provide WithInstance method
Also includes some refactor around package naming, treats own repo as first-class and uses alternative package names for third party imports, Google spanner libraries in this case.
2017-06-04 21:53:33 +01:00
Christian Klotz 748ae8f06a Add Spanner driver
Support for Google Cloud Spanner, closes #172. Includes example migrations ported over from the PostgreSQL driver.

In all examples provided by Google I’ve come across, camel-case is used for table names, columns, etc. Hence the examples use this naming convention.
2017-06-04 21:13:49 +01:00
Jolan Malassigne 28524d977c add cassandra driver and function to retrieve networkSettings to get port bound to 9042 2017-05-31 10:05:56 +02:00
Pavlina Drosos b5c2f0f2bb Add code sample for use with existing MySQL client with multistatements=true 2017-05-30 18:52:26 +03:00
Max decee4abb2 added file cli/build_sqlite3.go 2017-05-30 14:06:59 +02:00
maxvw 8b4ce58a1d adapted the sqlite driver for v3 (#165) 2017-05-27 08:58:01 +02:00
Matthias KadenbachandGitHub 3682bcf9d2 Update README.md 2017-05-18 19:47:23 -07:00
Matthias KadenbachandGitHub 857dc4b862 Update README.md 2017-05-18 19:45:53 -07:00
Matthias KadenbachandGitHub 247ca90b33 Merge pull request #234 from augustzf/master
Add command to make CLI example with Go toolchain work
2017-05-18 10:29:15 -07:00
Matthias KadenbachandGitHub 00b22e527d one line 2017-05-18 10:28:39 -07:00
August Z. Flatby 83035ec925 Add command to make CLI example with Go toolchain work 2017-05-18 09:27:39 +02:00
Matthias KadenbachandGitHub 035c07716c Merge pull request #231 from mattes/cli-updates
build cli with all source and database drivers
2017-05-17 19:37:49 -07:00
Matthias Kadenbach 0d9689ab9f go get update 2017-05-17 19:37:36 -07:00
Matthias Kadenbach e696a861ba fix makefile 2017-05-17 19:26:24 -07:00
Matthias Kadenbach 801e84d4d2 fix make deps 2017-05-17 19:21:48 -07:00
Matthias Kadenbach ff7e060269 remove ql 2017-05-17 19:16:08 -07:00
Matthias Kadenbach d4d20346f2 build cli with all source and database drivers 2017-05-17 19:10:45 -07:00
Matthias KadenbachandGitHub 1b48b97aa4 Update README.md 2017-05-16 13:27:30 -07:00
Matthias KadenbachandGitHub d1e665bac7 Merge pull request #227 from fsouza/gcs-source
source/google-cloud-storage: implement the driver
2017-05-16 13:14:34 -07:00
Matthias KadenbachandGitHub dc109fc118 Merge pull request #228 from fsouza/s3-source
source/aws-s3: implement the driver
2017-05-16 13:13:00 -07:00
Matthias KadenbachandGitHub fb8698d89a Merge pull request #223 from Menta2L/master
ql database support
2017-05-16 13:09:22 -07:00
vesco 2821341551 make ExecuteQuery private 2017-05-16 12:07:40 +03:00
Fuyuan BieandGitHub bcdbe0f018 Fix parenthesis problem. 2017-05-15 22:01:45 -07:00
Fuyuan BieandGitHub 5f2a8b7f30 Use both db name and migration table name for lock hash 2017-05-15 21:50:41 -07:00
Matthias KadenbachandGitHub 2831b4c0d8 Merge pull request #222 from OrbotixInc/master
Added support for Redshift.
2017-05-15 18:31:13 -07:00
Francisco Souza c95cbb5931 source/aws-s3: implement the driver 2017-05-14 14:31:40 -04:00
Francisco Souza da89656fdc source/google-cloud-storage: implement the driver 2017-05-14 13:47:01 -04:00
root 4e5dc68b4e ql support 2017-05-12 11:11:16 +03:00
Jonathan Bryant 91ecd3cb6a Added support for Redshift.
* Added driver that wraps the standard PostgreSQL driver
* Added build.
* Added driver to documentation.
2017-05-11 15:47:40 -06:00
Matthias KadenbachandGitHub 448a36c5cb Merge pull request #210 from divan/master
Add missing import to example in README
2017-05-04 11:05:03 -07:00
Ivan DanilukandGitHub ed5a887579 Add import to example.
The example in README with file source doesn't work without importing "github.com/mattes/migrate/source/github" package.
2017-05-04 19:42:12 +02:00
Matthias KadenbachandGitHub e42058f15a Merge pull request #209 from pgmcgee/patch-1
Fix a typo in go-bindata/README.md
2017-05-04 09:33:40 -07:00
Parker McGeeandGitHub a7e1cacd36 Fix a typo
I was browsing this and noticed the typo that is causing syntax highlighting to be off.
2017-05-04 07:44:37 -04:00
Matthias KadenbachandGitHub 734803549d Merge pull request #204 from fln/master
Fix appending multiStatements=true parameter for mysql driver
2017-04-25 13:37:09 -07:00
Julius Kriukas 14f784961f Fix appending multiStatements=true parameter for mysql driver 2017-04-25 22:54:16 +03:00
Matthias Kadenbach 5c8da71dea update travis branch 2017-04-25 12:01:21 -07:00
Matthias Kadenbach a6fe4b73b3 update makefile 2017-04-25 11:59:52 -07:00
Matthias KadenbachandGitHub 256e8f045b Update README.md 2017-04-25 11:13:59 -07:00
Matthias KadenbachandGitHub 2e54bda0e1 Update README.md 2017-04-25 11:06:06 -07:00
Matthias Kadenbach 2ae8e6291c remove v1 migrate compat 2017-04-25 10:37:55 -07:00
Matthias KadenbachandGitHub 651f78cba8 Update README.md 2017-04-21 13:36:03 -07:00
Matthias KadenbachandGitHub d5674ed411 Create README.md 2017-04-21 13:30:59 -07:00
Matthias KadenbachandGitHub 4a2a810c20 Update README.md 2017-04-21 13:29:13 -07:00
Matthias KadenbachandGitHub 365b0565de Update README.md 2017-04-21 13:07:46 -07:00
Matthias KadenbachandGitHub 65d785d560 Update README.md 2017-04-21 13:02:10 -07:00
Matthias KadenbachandGitHub fff688c019 Create README.md 2017-04-21 13:00:19 -07:00
Matthias Kadenbach 6f07ac731b fix go toolchain build 2017-04-21 12:23:01 -07:00
Matthias Kadenbach f00c15d7b0 update go toolchain install 2017-04-20 16:03:43 -07:00
Matthias Kadenbach 1676a902f2 update readme 2017-04-20 15:22:06 -07:00
Matthias KadenbachandGitHub 576d323c52 Merge pull request #200 from Gobonoid/fix/ReadmeUpdate
Updates import path in README example
2017-04-19 10:04:37 -07:00
Gobonoid b4adf71696 Updates import path in README examplee 2017-04-19 11:48:58 +01:00
Matthias KadenbachandGitHub c0703e7a92 Merge pull request #199 from guzart/date-versions
Date-based versions
2017-04-16 12:38:48 -07:00
Arturo Guzman d789d50bd6 Support date-based versions 2017-04-13 20:57:22 -04:00
Matthias KadenbachandGitHub b83285bd59 Merge pull request #194 from Arkq/v3.0-prev
Support for alternative postgres URI scheme
2017-04-10 11:22:50 -07:00
Arkadiusz Bokowy bc93ed027f Support for alternative postgres URI scheme
According to the PostgreSQL documentation (section 32.1.1.2), postgres
library supports two URI schemes: postgresql:// and postgres://

Reference:
https://www.postgresql.org/docs/current/static/libpq-connect.html#LIBPQ-CONNSTRING
2017-04-04 08:29:33 +02:00
Matthias Kadenbach 9f974b8996 Update Readme 2017-03-12 18:00:22 -07:00
Matthias Kadenbach c03b03c909 return status 0 for ErrNoChange #182 2017-03-12 12:33:13 -07:00
Matthias Kadenbach cecce18c51 fix source/file relative paths, ref #178 thanks @matyunin 2017-03-03 12:02:32 -08:00
Matthias Kadenbach 1f37f41ee8 remove cleanpath, fix mysql ensureVersionTable 2017-03-01 11:48:38 -08:00
Matthias Kadenbach 150ac7d708 add mysql custom TLS config
closes https://github.com/mattes/migrate/pull/117
2017-02-28 16:23:29 -08:00
Matthias Kadenbach be1ba9204a add mysql driver, add ENV to docker containers 2017-02-28 15:10:56 -08:00
Matthias Kadenbach 760bc3eb2d add note in readme 2017-02-23 14:50:48 -08:00
Matthias Kadenbach 706f23115f add v1compat 2017-02-23 14:48:44 -08:00
Matthias KadenbachandGitHub 171f9a256d Update FAQ.md 2017-02-23 14:27:04 -08:00
Matthias KadenbachandGitHub e7b9b0d0ed Update README.md 2017-02-23 13:36:34 -08:00
Matthias Kadenbach 5645143baa add example dockerfile 2017-02-23 13:13:02 -08:00
Matthias Kadenbach ec5c0ac3e0 fix version string 2017-02-23 10:36:33 -08:00
Matthias Kadenbach 511f9c1ffe only deploy on version go 1.8 2017-02-23 10:30:18 -08:00
Matthias Kadenbach 6325327ba1 fix dir 2017-02-22 17:58:25 -08:00
Matthias Kadenbach d60d590381 remove escaped new line 2017-02-22 17:48:50 -08:00
Matthias Kadenbach 5aac011e49 remove dups 2017-02-22 17:38:40 -08:00
Matthias Kadenbach 320d573ccd fix sudo whitespace 2017-02-22 17:27:18 -08:00
Matthias Kadenbach c51bef38b6 sudo 2017-02-22 17:15:40 -08:00
Matthias Kadenbach 28e0548cac fix github
see https://github.com/google/go-github/commit/23d6cb9cacb5aa314e93d600fe20a48496a718d4
2017-02-22 16:56:14 -08:00
Matthias Kadenbach d0acf6ecfe build deb package and push to packagecloud 2017-02-22 16:41:11 -08:00
Matthias Kadenbach 4b91f503eb make Force doc clear 2017-02-19 16:22:48 -08:00
Matthias Kadenbach eb02bc77fb allow to force NilVersion 2017-02-19 16:06:11 -08:00
Matthias KadenbachandGitHub 55f54251eb Update FAQ.md 2017-02-19 15:56:45 -08:00
Matthias Kadenbach e64b7cc233 explain dirty version error 2017-02-19 15:51:12 -08:00
Matthias Kadenbach 4bb46a98c6 fix github
see https://github.com/google/go-github/commit/2a4b92075e5c5e42ee0e55819b70f49b02c1ac64
2017-02-19 15:31:57 -08:00
Matthias Kadenbach 53b55288b3 bring back dirty state from v2 2017-02-19 15:15:00 -08:00
Matthias KadenbachandGitHub f576a64f11 Update README.md 2017-02-18 10:41:17 -08:00
Matthias Kadenbach d574676702 drop custom query params 2017-02-17 16:59:47 -08:00
Matthias Kadenbach f45821581c add multi statement test 2017-02-16 22:10:01 -08:00
Matthias KadenbachandGitHub bb6ae0898d Update README.md 2017-02-16 21:13:12 -08:00
Matthias KadenbachandGitHub ea7fc641ba Update README.md 2017-02-16 21:12:33 -08:00
Matthias KadenbachandGitHub 87e6482497 travis build with go 1.8 2017-02-16 20:46:12 -08:00
Matthias KadenbachandGitHub 1e08e21470 update with issues 2017-02-16 20:41:17 -08:00
Matthias Kadenbach ccb40ff3ff Update Readme 2017-02-16 11:29:18 -08:00
Matthias Kadenbach 3291f3f891 add comment 2017-02-16 11:18:02 -08:00
Matthias Kadenbach 2031939bfc allow postgres migrations_table config 2017-02-16 11:06:11 -08:00
Matthias Kadenbach 6394299937 clean cli -path 2017-02-15 11:07:28 -08:00
Matthias Kadenbach 316f6c0cf3 switch to binary search 2017-02-15 10:48:43 -08:00
Matthias Kadenbach 53630f5ff1 add NewWithDatabaseInstance example 2017-02-14 23:25:19 -08:00
Matthias Kadenbach 3feee5958a updated postgres driver, add database.Error struct 2017-02-14 23:12:16 -08:00
Matthias Kadenbach 209c056258 add database lock timeouts in migrate and in tests 2017-02-11 19:15:54 -08:00
Matthias Kadenbach cd6e62049c Update documentation 2017-02-11 11:13:27 -08:00
Matthias Kadenbach 526c0a918e add packagecloud.io badge 2017-02-09 20:09:11 -08:00
Matthias Kadenbach de7ffa60cc update readme 2017-02-09 20:07:04 -08:00
Matthias Kadenbach 81b9791178 move contributing in own file 2017-02-09 20:03:56 -08:00
Matthias Kadenbach c4ec6ac0f7 move CLI usage in own README 2017-02-09 20:00:38 -08:00
Matthias Kadenbach cfe3d2a83e add example for parallelTest 2017-02-09 19:52:50 -08:00
Matthias Kadenbach b6bd4fe02d add comments for source 2017-02-09 19:42:48 -08:00
Matthias Kadenbach 2da976155a add comments for database 2017-02-09 19:04:54 -08:00
Matthias Kadenbach e69e0de9c9 fix old ref to LogString 2017-02-09 18:26:58 -08:00
Matthias Kadenbach cfc9f10a6e add some more comments 2017-02-09 18:23:08 -08:00
Matthias Kadenbach 8ba71b0754 rename nameFromUrl to schemeFromUrl 2017-02-09 18:19:13 -08:00
Matthias Kadenbach 46c16bb8f1 add test for suint 2017-02-09 18:16:16 -08:00
Matthias Kadenbach 1b079a3cda rename Migration.LongString to LogString 2017-02-09 18:06:38 -08:00
Matthias Kadenbach c1b23fee45 add Migrate.Run 2017-02-09 18:04:36 -08:00
Matthias Kadenbach 199678e1bc add code comments for package migrate 2017-02-09 16:59:30 -08:00
Matthias Kadenbach d5daf6f6ae add .godoc.pid to gitignore 2017-02-09 13:47:28 -08:00
Matthias Kadenbach 03d8d3233f fix line break in makefile 2017-02-09 12:03:03 -08:00
Matthias Kadenbach 5c6a21b014 add make docs 2017-02-09 11:59:49 -08:00
Matthias Kadenbach 3965df69d5 fix indent in makefile 2017-02-08 20:13:17 -08:00
Matthias Kadenbach f47436e158 add make release 2017-02-08 19:54:50 -08:00
Matthias Kadenbach f474e4cae1 travis builds dep tree info 2017-02-08 19:44:12 -08:00
Matthias Kadenbach 7979fdb832 add list-external-deps 2017-02-08 19:34:53 -08:00
Matthias Kadenbach b3f6bc47bb update readme with dev, testing, and contribution section 2017-02-08 18:27:12 -08:00
Matthias Kadenbach e6163082d7 add restore-import-paths and rewrite-import-paths 2017-02-08 18:01:01 -08:00
Matthias Kadenbach fe1b5c06eb add go coverage 2017-02-08 16:17:04 -08:00
Matthias Kadenbach 297a4227ef fixing travis 2017-02-08 01:49:57 -08:00
Matthias Kadenbach 9f774d3e6a fix typo in go-bindata README 2017-02-07 23:00:25 -08:00
Matthias Kadenbach f7699f1dec update go-bindata with latest source.Migration helper 2017-02-07 22:59:12 -08:00
Matthias Kadenbach 3889c93b63 add Readme draft 2017-02-07 22:40:45 -08:00
Matthias Kadenbach ee384ed8cc initial version 3.0.0 preview
hard work
2017-02-07 22:01:29 -08:00
Matthias Kadenbach e92f038f40 skip neo4j tests 2017-01-23 13:54:25 -08:00
Matthias KadenbachandGitHub 0b3426bf5d Merge pull request #25 from daddykotex/master
Ensure that interrupt signals are not relayed to the interrupt channel when the migration is over
2017-01-23 13:37:42 -08:00
Matthias Kadenbach 0861ff957d Update Readme 2017-01-23 13:36:22 -08:00
Matthias Kadenbach 32607ef715 Merge branch 'carlosforero-neo4j-driver' 2017-01-23 13:32:56 -08:00
Matthias Kadenbach c643ffd6d6 fix merge conflicts 2017-01-23 13:32:48 -08:00
Matthias KadenbachandGitHub b8b3883950 Merge pull request #101 from bfosberry/fixing-unsafe-cast
Fixing unsafe typecast
2017-01-23 13:26:59 -08:00
Matthias KadenbachandGitHub 3547fc54cb Merge pull request #144 from benma/patch-1
Update version.go
2017-01-19 09:32:29 -08:00
benmaandGitHub 64c41a8af0 Update version.go 2017-01-19 14:11:19 +01:00
Matthias KadenbachandGitHub 4191ac9cfe Merge pull request #142 from goenning/master
allow use of big version numbers
2017-01-18 15:48:54 -08:00
goenning f1738e78db allow use of big version numbers 2017-01-18 22:19:02 +00:00
Matthias KadenbachandGitHub de494e4b98 Merge pull request #128 from oliverkofoed/master
Add support for ScyllaDB (remove cql counter use)
2017-01-10 09:26:17 -08:00
Matthias KadenbachandGitHub 585509f3bc Merge pull request #138 from mattes/add-mongo-to-readme
add mongodb to readme
2017-01-09 10:43:35 -08:00
Matthias KadenbachandGitHub f630dc5971 Update README.md 2017-01-09 10:43:22 -08:00
Matthias KadenbachandGitHub 9a8f78ba3b Merge pull request #137 from drboyer/updown-colors
Change colors for up and down migration arrows
2017-01-09 10:39:55 -08:00
Matthias KadenbachandGitHub 63c6168c7f Merge pull request #136 from dimag-jfrog/prepare_pr_2_rebase
Driver for MongoDB (New)
2017-01-09 10:38:56 -08:00
dimag 1993f0783b Merge remote-tracking branch 'fork/prepare_pr_2_rebase' into prepare_pr_2_rebase
# Conflicts:
#	docker-compose.yml
2017-01-07 16:44:36 +02:00
dimag 70238f448b - Reverted repository from dimag-jfrog to mattes to prepare for pull request 2017-01-07 16:36:25 +02:00
dimag 914c38e379 - Fixed test with not-exported method identified as missing: unified the 2 errors into one
- Fixed error message formatting
- Ensuring uniqueness of the version field (added unique index)
2017-01-07 16:34:06 +02:00
dimag 0f28b01383 specifying version of mongo 2017-01-07 16:34:06 +02:00
dimag 6909594c0b Checking non-exported migration method 2017-01-07 16:33:10 +02:00
dimag 583921e8bf Added README.md 2017-01-07 16:33:10 +02:00
dimag 61ecf2c096 - Refactored project structure to be consistent with original project
- Added docker support
2017-01-07 16:33:10 +02:00
dimag 0eecb74230 Added explicit migration method signature validation 2017-01-07 16:30:40 +02:00
dimag b7b75f3c59 - registering method receiver directly for the driver
- currently each driver contains only a single method receiver:
   - enforcing method receiver pre-registration on go methods driver initialization
   - Method receiver name can be removed from files format
- passing the DbName parameter inside the method receiver for the go methods driver
2017-01-07 16:30:40 +02:00
dimag fe48ddfdcc - redefined mongo db driver to be autonomous and not depending on specific methods
- defined methods receivers registration by name and change the migration files format to include them
- added extensive testing
2017-01-07 16:30:40 +02:00
dimag 70b23c3f84 Relaxed constratint that method receiver has to be of Driver interface. 2017-01-07 16:30:40 +02:00
dimag 223908f9d2 - Reset all changes to the upstream branch
- Changed logic not to use custom filename parser:
  -Supporting up and down files only, no both direction files
  -Using method names as is
- Added complete test to the mongo db migration scenarios
2017-01-07 16:30:40 +02:00
dimag 1838852d4d Changes that interfere with the Migrate open source code:
- Added go methods migrator, mongo db template: different from the usual driver
model.
- Added support for bidirectional files (for go methods), appending _up or _down upon context
- Added DriverWithFilnameParser for providing custom filename parser functionality that knows to parse bi-directional file names.
2017-01-07 16:29:19 +02:00
dimag 175550643c changed import paths to point to dimag-jfrog 2017-01-07 16:24:14 +02:00
dimag 34a3e58ab9 - Reverted repository from dimag-jfrog to mattes to prepare for pull request 2017-01-07 16:16:00 +02:00
dimag dec52a778f - Using double quotes for parameter in docker-compose - to be more aligned with master repository 2017-01-07 16:12:48 +02:00
dimag 8865262a03 - Fixed test with not-exported method identified as missing: unified the 2 errors into one
- Fixed error message formatting
- Ensuring uniqueness of the version field (added unique index)
2017-01-07 16:08:09 +02:00
Matthias KadenbachandGitHub eb0eec4471 Merge pull request #135 from ansel1/master
Avoid DDL when checking for versions table #134
2017-01-06 14:24:59 -08:00
Russ Egan da6c505f41 Avoid DDL when checking for versions table #134 2017-01-06 15:52:54 -05:00
Matthias KadenbachandGitHub 966ba843e0 Merge pull request #133 from wlcx/add-ql-driver
Add Ql driver
2017-01-05 14:04:34 -08:00
Sam Willcocks 5773f75698 Add Ql driver
Adds a driver for the embedded Ql database (https://github.com/cznic/ql)
2017-01-05 15:09:41 +00:00
Oliver Kofoed 9e11d704a4 Add support for ScyllaDB (remove cql counter use) 2016-12-18 19:25:52 -08:00
dimag 32cb825110 specifying version of mongo 2016-08-10 21:33:36 +03:00
dimag 9bd4c8c83c Checking non-exported migration method 2016-08-10 21:25:36 +03:00
dimag e9ede5506a Added README.md 2016-08-10 20:39:41 +03:00
dimag e27b9c2a7b - Refactored project structure to be consistent with original project
- Added docker support
2016-08-10 17:15:17 +03:00
dimag 834a096bf8 Added explicit migration method signature validation 2016-08-10 14:13:53 +03:00
dimag 866ff82ffe - registering method receiver directly for the driver
- currently each driver contains only a single method receiver:
   - enforcing method receiver pre-registration on go methods driver initialization
   - Method receiver name can be removed from files format
- passing the DbName parameter inside the method receiver for the go methods driver
2016-08-09 21:16:21 +03:00
dimag 570e4b42be - redefined mongo db driver to be autonomous and not depending on specific methods
- defined methods receivers registration by name and change the migration files format to include them
- added extensive testing
2016-08-09 16:47:34 +03:00
dimag c71fea90d6 Relaxed constratint that method receiver has to be of Driver interface. 2016-08-08 21:34:52 +03:00
dimag 5f7b2caee8 - Reset all changes to the upstream branch
- Changed logic not to use custom filename parser:
  -Supporting up and down files only, no both direction files
  -Using method names as is
- Added complete test to the mongo db migration scenarios
2016-08-08 21:07:37 +03:00
dimag 64101a13e9 Changes that interfere with the Migrate open source code:
- Added go methods migrator, mongo db template: different from the usual driver
model.
- Added support for bidirectional files (for go methods), appending _up or _down upon context
- Added DriverWithFilnameParser for providing custom filename parser functionality that knows to parse bi-directional file names.
2016-08-08 16:29:25 +03:00
dimag 8630ea78b7 changed import paths to point to dimag-jfrog 2016-08-04 15:52:31 +03:00
Brendan Fosberry b68c58c136 Fixing unsafe typecast
If a non pq error is returned, we rollback and return the error
2016-06-03 13:41:05 -05:00
Carlos Forero e303d64469 Add driver for Neo4j 2015-12-08 19:39:22 -05:00
David Francoeur 820c58b7f2 Merge branch 'master' of github.com:mattes/migrate 2014-12-28 15:38:33 -05:00
David Francoeur ab520d7b6b Ensure that os signals are not relayed to the interrupt channel when the
function returns
2014-12-28 15:36:46 -05:00
207 changed files with 12363 additions and 3304 deletions
+13
View File
@@ -0,0 +1,13 @@
# Project
FAQ.md
README.md
LICENSE
Makefile
.gitignore
.travis.yml
CONTRIBUTING.md
MIGRATIONS.md
docker-deploy.sh
# Golang
testing
+39
View File
@@ -0,0 +1,39 @@
---
name: Bug report
about: Create a report to help us improve
---
**Describe the Bug**
A clear and concise description of what the bug is.
**Steps to Reproduce**
Steps to reproduce the behavior:
1. My migrations look like '...'
2. I ran migrate with the following options '....'
3. See error
**Expected Behavior**
A clear and concise description of what you expected to happen.
**Migrate Version**
e.g. v3.4.0
Obtained by running: `migrate -version`
**Loaded Source Drivers**
e.g. s3, github, go-bindata, gcs, file
Obtained by running: `migrate -help`
**Loaded Database Drivers**
e.g. spanner, stub, clickhouse, cockroachdb, crdb-postgres, postgres, postgresql, redshift, cassandra, cockroach, mysql
Obtained by running: `migrate -help`
**Go Version**
e.g. go version go1.11 linux/amd64
Obtained by running: `go version`
**Stacktrace**
Please provide if available
**Additional context**
Add any other context about the problem here.
+17
View File
@@ -0,0 +1,17 @@
---
name: Feature request
about: Suggest an idea for this project
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+6 -1
View File
@@ -1,2 +1,7 @@
.DS_Store
test.db
cli/build
cli/cli
cli/migrate
.coverage
.godoc.pid
vendor/
+81 -11
View File
@@ -1,19 +1,89 @@
language: go
sudo: required
go:
- 1.4
- 1.5
- 1.6
- 1.7
matrix:
allow_failures:
- go: master
include:
# Supported versions of Go: https://golang.org/dl/
- go: "1.10.x"
- go: "1.11.x"
- go: master
go_import_path: github.com/golang-migrate/migrate
env:
global:
- GO111MODULE=on
- MIGRATE_TEST_CONTAINER_BOOT_TIMEOUT=60
- DOCKER_USERNAME=golangmigrate
- secure: "oSOznzUrgr5h45qW4PONkREpisPAt40tnM+KFWtS/Ggu5UI2Ie0CmyYXWuBjbt7B97a4yN9Qzmn8FxJHJ7kk+ABOi3muhkxeIhr6esXbzHhX/Jhv0mj1xkzX7KoVN9oHBz3cOI/QeRyEAO68xjDHNE2kby4RTT9VBt6TQUakKVkqI5qkqLBTADepCjVC+9XhxVxUNyeWKU8ormaUfJBjoNVoDlwXekUPnJenfmfZqXxUInvBCfUyp7Pq+kurBORmg4yc6qOlRYuK67Xw+i5xpjbZouNlXPk0rq7pPy5zjhmZQ3kImoFPvNMeKViDcI6kSIJKtjdhms9/g/6MgXS9HlL5kFy8tYKbsyiHnHB1BsvaLAKXctbUZFDPstgMPADfnad2kZXPrNqIhfWKZrGRWidawCYJ1sKKwYxLMKrtA0umqgMoL90MmBOELhuGmvMV0cFJB+zo+K2YWjEiMGd8xRb5mC5aAy0ZcCehO46jGtpr217EJmMF8Ywr7cFqM2Shg5U2jev9qUpYiXwmPnJKDuoT2ZHuHmPgFIkYiWC5yeJnnmG5bed1sKBp93AFrJX+1Rx5oC4BpNegewmBZKpOSwls/D1uMAeQK3dPmQHLsT6o2VBLfeDGr+zY0R85ywwPZCv00vGol02zYoTqN7eFqr6Qhjr/qx5K1nnxJdFK3Ts="
services:
- docker
before_install:
- curl -L https://github.com/docker/compose/releases/download/1.4.2/docker-compose-`uname -s`-`uname -m` > docker-compose
- chmod +x docker-compose
- sudo mv docker-compose /usr/local/bin
- sed -i -e 's/golang/golang:'"$TRAVIS_GO_VERSION"'/' docker-compose.yml
before_cache:
- mv $GOPATH/src/github.com/golang-migrate /tmp/golang-migrate
- rm -rf $GOPATH/pkg/**/github.com/golang-migrate
script: make test
cache:
directories:
- $GOPATH/pkg
before_install:
- echo "TRAVIS_GO_VERSION=${TRAVIS_GO_VERSION}"
# Download the binary to bin folder in $GOPATH
- if [[ "${TRAVIS_GO_VERSION}" == 1.10* ]]; then curl -L -s https://github.com/golang/dep/releases/download/v0.5.0/dep-linux-amd64 -o $GOPATH/bin/dep; fi
# Make the binary executable
- if [[ "${TRAVIS_GO_VERSION}" == 1.10* ]]; then chmod +x $GOPATH/bin/dep; fi
install:
- if [[ "${TRAVIS_GO_VERSION}" == 1.10* ]]; then dep ensure -vendor-only; fi
- go get github.com/mattn/goveralls
script:
- make test COVERAGE_DIR=/tmp/coverage
after_success:
- mv /tmp/golang-migrate $GOPATH/src/github.com/golang-migrate
- goveralls -service=travis-ci -coverprofile /tmp/coverage/combined.txt
- make list-external-deps > dependency_tree.txt && cat dependency_tree.txt
before_deploy:
- make build-cli
- gem install --no-ri --no-rdoc fpm
- fpm -s dir -t deb -n migrate -v "$(git describe --tags 2>/dev/null | cut -c 2-)" --license MIT -m dhui@users.noreply.github.com --url https://github.com/golang-migrate/migrate --description='Database migrations' -a amd64 -p migrate.$(git describe --tags 2>/dev/null | cut -c 2-).deb --deb-no-default-config-files -f -C cli/build migrate.linux-amd64=/usr/bin/migrate
deploy:
- provider: releases
api_key:
secure: hWH1HLPpzpfA8pXQ93T1qKQVFSpQp0as/JLQ7D91jHuJ8p+RxVeqblDrR6HQY/95R/nyiE9GJmvUolSuw5h449LSrGxPtVWhdh6EnkxlQHlen5XeMhVjRjFV0sE9qGe8v7uAkiTfRO61ktTWHrEAvw5qpyqnNISodmZS78XIasPODQbNlzwINhWhDTHIjXGb4FpizYaL3OGCanrxfR9fQyCaqKGGBjRq3Mfq8U6Yd4mApmsE+uJxgaZV8K5zBqpkSzQRWhcVGNL5DuLsU3gfSJOo7kZeA2G71SHffH577dBoqtCZ4VFv169CoUZehLWCb+7XKJZmHXVujCURATSySLGUOPc6EoLFAn3YtsCA04mS4bZVo5FZPWVwfhjmkhtDR4f6wscKp7r1HsFHSOgm59QfETQdrn4MnZ44H2Jd39axqndn5DvK9EcZVjPHynOPnueXP2u6mTuUgh2VyyWBCDO3CNo0fGlo7VJI69IkIWNSD87K9cHZWYMClyKZkUzS+PmRAhHRYbVd+9ZjKOmnU36kUHNDG/ft1D4ogsY+rhVtXB4lgWDM5adri+EIScYdYnB1/pQexLBigcJY9uE7nQTR0U6QgVNYvun7uRNs40E0c4voSfmPdFO0FlOD2y1oQhnaXfWLbu9nMcTcs4RFGrcC7NzkUN4/WjG8s285V6w=
skip_cleanup: true
on:
go: "1.11.x"
repo: golang-migrate/migrate
tags: true
file:
- cli/build/migrate.linux-amd64.tar.gz
- cli/build/migrate.darwin-amd64.tar.gz
- cli/build/migrate.windows-amd64.exe.tar.gz
- cli/build/sha256sum.txt
- dependency_tree.txt
- provider: packagecloud
repository: migrate
username: golang-migrate
token:
secure: aICwu3gJ1sJ1QVCD3elpg+Jxzt4P+Zj1uoh5f0sOwnjDNIZ4FwUT1cMrWloP8P2KD0iyCOawuZER27o/kQ21oX2OxHvQbYPReA2znLm7lHzCmypAAOHPxpgnQ4rMGHHJXd+OsxtdclGs67c+EbdBfoRRbK400Qz/vjPJEDeH4mh02ZHC2nw4Nk/wV4jjBIkIt9dGEx6NgOA17FCMa3MaPHlHeFIzU7IfTlDHbS0mCCYbg/wafWBWcbGqtZLWAYtJDmfjrAStmDLdAX5J5PsB7taGSGPZHmPmpGoVgrKt/tb9Xz1rFBGslTpGROOiO4CiMAvkEKFn8mxrBGjfSBqp7Dp3eeSalKXB1DJAbEXx2sEbMcvmnoR9o43meaAn+ZRts8lRL8S/skBloe6Nk8bx3NlJCGB9WPK1G56b7c/fZnJxQbrCw6hxDfbZwm8S2YPviFTo/z1BfZDhRsL74reKsN2kgnGo2W/k38vvzIpsssQ9DHN1b0TLCxolCNPtQ7oHcQ1ohcjP2UgYXk0FhqDoL+9LQva/DU4N9sKH0UbAaqsMVSErLeG8A4aauuFcVrWRBaDYyTag4dQqzTulEy7iru2kDDIBgSQ1gMW/yoBOIPK4oi6MtbTf1X39fzXFLS1cDd3LW61yAu3YrbjAetpfx2frIvrRAiL9TxWA1gnrs5o=
dist: ubuntu/xenial
package_glob: '*.deb'
skip_cleanup: true
on:
go: "1.11.x"
repo: golang-migrate/migrate
tags: true
- provider: script
script: ./docker-deploy.sh
on:
go: "1.11.x"
repo: golang-migrate/migrate
tags: true
+21
View File
@@ -0,0 +1,21 @@
# Development, Testing and Contributing
1. Make sure you have a running Docker daemon
(Install for [MacOS](https://docs.docker.com/docker-for-mac/))
1. Fork this repo and `git clone` somewhere to `$GOPATH/src/github.com/golang-migrate/migrate`
1. Install [dep](https://github.com/golang/dep) and run `dep ensure` to pull dependencies
1. Confirm tests are working: `make test-short`
1. Write awesome code ...
1. `make test` to run all tests against all database versions
1. Push code and open Pull Request
Some more helpful commands:
* You can specify which database/ source tests to run:
`make test-short SOURCE='file go_bindata' DATABASE='postgres cassandra'`
* After `make test`, run `make html-coverage` which opens a shiny test coverage overview.
* `make build-cli` builds the CLI in directory `cli/build/`.
* `make list-external-deps` lists all external dependencies for each package
* `make docs && make open-docs` opens godoc in your browser, `make kill-docs` kills the godoc server.
Repeatedly call `make docs` to refresh the server.
* Set the `DOCKER_API_VERSION` environment variable to the latest supported version if you get errors regarding the docker client API version being too new.
+22 -2
View File
@@ -1,3 +1,23 @@
FROM scratch
ADD migrate /migrate
FROM golang:1.11-alpine3.8 AS downloader
ARG VERSION
RUN apk add --no-cache git gcc musl-dev
WORKDIR /go/src/github.com/golang-migrate/migrate
COPY . ./
ENV GO111MODULE=on
ENV DATABASES="postgres mysql redshift cassandra spanner cockroachdb clickhouse"
ENV SOURCES="file go_bindata github aws_s3 google_cloud_storage"
RUN go build -a -o build/migrate.linux-386 -ldflags="-X main.Version=${VERSION}" -tags "$DATABASES $SOURCES" ./cli
FROM alpine:3.8
RUN apk add --no-cache ca-certificates
COPY --from=downloader /go/src/github.com/golang-migrate/migrate/build/migrate.linux-386 /migrate
ENTRYPOINT ["/migrate"]
CMD ["--help"]
+67
View File
@@ -0,0 +1,67 @@
# FAQ
#### How is the code base structured?
```
/ package migrate (the heart of everything)
/cli the CLI wrapper
/database database driver and sub directories have the actual driver implementations
/source source driver and sub directories have the actual driver implementations
```
#### Why is there no `source/driver.go:Last()`?
It's not needed. And unless the source has a "native" way to read a directory in reversed order,
it might be expensive to do a full directory scan in order to get the last element.
#### What is a NilMigration? NilVersion?
NilMigration defines a migration without a body. NilVersion is defined as const -1.
#### What is the difference between uint(version) and int(targetVersion)?
version refers to an existing migration version coming from a source and therefor can never be negative.
targetVersion can either be a version OR represent a NilVersion, which equals -1.
#### What's the difference between Next/Previous and Up/Down?
```
1_first_migration.up.extension next -> 2_second_migration.up.extension ...
1_first_migration.down.extension <- previous 2_second_migration.down.extension ...
```
#### Why two separate files (up and down) for a migration?
It makes all of our lives easier. No new markup/syntax to learn for users
and existing database utility tools continue to work as expected.
#### How many migrations can migrate handle?
Whatever the maximum positive signed integer value is for your platform.
For 32bit it would be 2,147,483,647 migrations. Migrate only keeps references to
the currently run and pre-fetched migrations in memory. Please note that some
source drivers need to do build a full "directory" tree first, which puts some
heat on the memory consumption.
#### Are the table tests in migrate_test.go bloated?
Yes and no. There are duplicate test cases for sure but they don't hurt here. In fact
the tests are very visual now and might help new users understand expected behaviors quickly.
Migrate from version x to y and y is the last migration? Just check out the test for
that particular case and know what's going on instantly.
#### What is Docker being used for?
Only for testing. See [testing/docker.go](testing/docker.go)
#### Why not just use docker-compose?
It doesn't give us enough runtime control for testing. We want to be able to bring up containers fast
and whenever we want, not just once at the beginning of all tests.
#### Can I maintain my driver in my own repository?
Yes, technically thats possible. We want to encourage you to contribute your driver to this respository though.
The driver's functionality is dictated by migrate's interfaces. That means there should really
just be one driver for a database/ source. We want to prevent a future where several drivers doing the exact same thing,
just implemented a bit differently, co-exist somewhere on Github. If users have to do research first to find the
"best" available driver for a database in order to get started, we would have failed as an open source community.
#### Can I mix multiple sources during a batch of migrations?
No.
#### What does "dirty" database mean?
Before a migration runs, each database sets a dirty flag. Execution stops if a migration fails and the dirty state persists,
which prevents attempts to run more migrations on top of a failed migration. You need to manually fix the error
and then "force" the expected version.
Generated
+659
View File
@@ -0,0 +1,659 @@
# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
[[projects]]
digest = "1:098b27970336695d686ef8a701328cfb231445e2d54564d09c3b0c7d43ce129a"
name = "cloud.google.com/go"
packages = [
"civil",
"compute/metadata",
"iam",
"internal",
"internal/atomiccache",
"internal/fields",
"internal/optional",
"internal/protostruct",
"internal/trace",
"internal/version",
"longrunning",
"longrunning/autogen",
"spanner",
"spanner/admin/database/apiv1",
"storage",
]
pruneopts = "UT"
revision = "0fd7230b2a7505833d5f69b75cbd6c9582401479"
version = "v0.23.0"
[[projects]]
digest = "1:adbfc2db750d6fbe0c165ad801ba2d730883a3bc40bc8fe829516eb42ed17771"
name = "github.com/Microsoft/go-winio"
packages = ["."]
pruneopts = "UT"
revision = "a6d595ae73cf27a1b8fc32930668708f45ce1c85"
version = "v0.4.9"
[[projects]]
digest = "1:d867dfa6751c8d7a435821ad3b736310c2ed68945d05b50fb9d23aee0540c8cc"
name = "github.com/Sirupsen/logrus"
packages = ["."]
pruneopts = "UT"
revision = "3e01752db0189b9157070a0e1668a620f9a85da2"
version = "v1.0.6"
[[projects]]
digest = "1:1a42d6d0e8fe8cc1024ed542c5f8a0bc7346a292701b3bd94a07996f29fcbd38"
name = "github.com/aws/aws-sdk-go"
packages = [
"aws",
"aws/awserr",
"aws/awsutil",
"aws/client",
"aws/client/metadata",
"aws/corehandlers",
"aws/credentials",
"aws/credentials/ec2rolecreds",
"aws/credentials/endpointcreds",
"aws/credentials/stscreds",
"aws/csm",
"aws/defaults",
"aws/ec2metadata",
"aws/endpoints",
"aws/request",
"aws/session",
"aws/signer/v4",
"internal/sdkio",
"internal/sdkrand",
"internal/sdkuri",
"internal/shareddefaults",
"private/protocol",
"private/protocol/eventstream",
"private/protocol/eventstream/eventstreamapi",
"private/protocol/query",
"private/protocol/query/queryutil",
"private/protocol/rest",
"private/protocol/restxml",
"private/protocol/xml/xmlutil",
"service/s3",
"service/s3/s3iface",
"service/sts",
]
pruneopts = "UT"
revision = "468b9714c11f10b22e533253b35eb9c28f4be691"
version = "v1.14.32"
[[projects]]
branch = "master"
digest = "1:568184e644ca0114e16fa472037e18bb23a8c0668f9da12f3d2b059e0c548637"
name = "github.com/cockroachdb/cockroach-go"
packages = ["crdb"]
pruneopts = "UT"
revision = "59c0560478b705bf9bd12f9252224a0fad7c87df"
[[projects]]
branch = "master"
digest = "1:3ba3d94d45f2d5c4b3411452474d96996b7e0dce730b689920f7c307511e5655"
name = "github.com/cznic/b"
packages = ["."]
pruneopts = "UT"
revision = "35e9bbe41f07452a183c517a5fc5f3c9f45eaa0f"
[[projects]]
branch = "master"
digest = "1:4688e1d4b22367c5bd24fe65f44b3e53a443a93fd19cf34169f1c8337d93dcf7"
name = "github.com/cznic/fileutil"
packages = ["."]
pruneopts = "UT"
revision = "6a051e75936f623600b67c2b1116b6b6c0ffb936"
[[projects]]
digest = "1:a9e2943ed681d4758a43d1bd9e80d7e993b66ce5adc7aa5dcc083ba48a47a4eb"
name = "github.com/cznic/internal"
packages = [
"buffer",
"file",
"slice",
]
pruneopts = "UT"
revision = "cef02a853c3a93623c42eacd574e7ea05f55531b"
version = "1.0.0"
[[projects]]
digest = "1:ec7b58207ac8eee1c554b7947b3f5124f711238a80edb7224a53bae7263c95ae"
name = "github.com/cznic/lldb"
packages = ["."]
pruneopts = "UT"
revision = "bea8611dd5c407f3c5eab9f9c68e887a27dc6f0e"
version = "v1.1.0"
[[projects]]
branch = "master"
digest = "1:42951d361293c2f7068ab0411afeb57856afba843fe681cea1e2731b47021222"
name = "github.com/cznic/mathutil"
packages = ["."]
pruneopts = "UT"
revision = "ca4c9f2c136954238c3158b92de72078c7672ecc"
[[projects]]
digest = "1:905947603b7f1066a7149576b2cbad2e6f890729f892c827cdbcee046d07db9c"
name = "github.com/cznic/ql"
packages = [
".",
"driver",
"vendored/github.com/camlistore/go4/lock",
]
pruneopts = "UT"
revision = "7a63cd7aa46ecd9f549b96983029576af2178f60"
version = "v1.1.0"
[[projects]]
branch = "master"
digest = "1:06877d57543857a87186f43b6195bbdf5e8f4a0f51412fdf4baef504cead7973"
name = "github.com/cznic/sortutil"
packages = ["."]
pruneopts = "UT"
revision = "4c7342852e65c2088c981288f2c5610d10b9f7f4"
[[projects]]
branch = "master"
digest = "1:8a26e11e94f1c9f03ab9fc1e816b003629d41b26ab41eefcb7444fad5b24fc4b"
name = "github.com/cznic/strutil"
packages = ["."]
pruneopts = "UT"
revision = "529a34b1c186b483642a7a230c67521d9aa4b0fb"
[[projects]]
branch = "master"
digest = "1:bb8236408f88f411f565dffcf1e648fe34c7456820b707f96ccbfb89e1cd8eeb"
name = "github.com/cznic/zappy"
packages = ["."]
pruneopts = "UT"
revision = "2533cb5b45cc6c07421468ce262899ddc9d53fb7"
[[projects]]
branch = "master"
digest = "1:4189ee6a3844f555124d9d2656fe7af02fca961c2a9bad9074789df13a0c62e0"
name = "github.com/docker/distribution"
packages = [
"digestset",
"reference",
]
pruneopts = "UT"
revision = "0dae0957e5fe3156c265d22bef4cba9efbf388e2"
[[projects]]
digest = "1:ec821dda59d7dd340498d74f798aa218b2c782bba54a690e866dc4f520d900d5"
name = "github.com/docker/docker"
packages = [
"api",
"api/types",
"api/types/blkiodev",
"api/types/container",
"api/types/events",
"api/types/filters",
"api/types/image",
"api/types/mount",
"api/types/network",
"api/types/registry",
"api/types/strslice",
"api/types/swarm",
"api/types/time",
"api/types/versions",
"api/types/volume",
"client",
"pkg/ioutils",
"pkg/longpath",
"pkg/system",
"pkg/tlsconfig",
]
pruneopts = "UT"
revision = "90d35abf7b3535c1c319c872900fbd76374e521c"
version = "v17.05.0-ce-rc3"
[[projects]]
digest = "1:b6b5c3e8da0fb8073cd2886ba249a40f4402b4391ca6eba905a142cceea97a12"
name = "github.com/docker/go-connections"
packages = [
"nat",
"sockets",
"tlsconfig",
]
pruneopts = "UT"
revision = "3ede32e2033de7505e6500d6c868c2b9ed9f169d"
version = "v0.3.0"
[[projects]]
digest = "1:6f82cacd0af5921e99bf3f46748705239b36489464f4529a1589bc895764fb18"
name = "github.com/docker/go-units"
packages = ["."]
pruneopts = "UT"
revision = "47565b4f722fb6ceae66b95f853feed578a4a51c"
version = "v0.3.3"
[[projects]]
branch = "master"
digest = "1:4841e14252a2cecf11840bd05230412ad469709bbacfc12467e2ce5ad07f339b"
name = "github.com/docker/libtrust"
packages = ["."]
pruneopts = "UT"
revision = "aabc10ec26b754e797f9028f4589c5b7bd90dc20"
[[projects]]
branch = "master"
digest = "1:67d0b50be0549e610017cb91e0b0b745ec0cad7c613bc8e18ff2d1c1fc8825a7"
name = "github.com/edsrzf/mmap-go"
packages = ["."]
pruneopts = "UT"
revision = "0bce6a6887123b67a60366d2c9fe2dfb74289d2e"
[[projects]]
digest = "1:3a1c71661e5d956a3c3ff639dba511871757b5f37cb2107abf717d631ece8b1a"
name = "github.com/fsouza/fake-gcs-server"
packages = ["fakestorage"]
pruneopts = "UT"
revision = "9162fe06459e3d9859987fc2802fa729ddb6dc53"
version = "v1.0.0"
[[projects]]
digest = "1:fe8a03a8222d5b913f256972933d26d24ad7c8286692a42943bc01633cc8fce3"
name = "github.com/go-ini/ini"
packages = ["."]
pruneopts = "UT"
revision = "358ee7663966325963d4e8b2e1fbd570c5195153"
version = "v1.38.1"
[[projects]]
digest = "1:adea5a94903eb4384abef30f3d878dc9ff6b6b5b0722da25b82e5169216dfb61"
name = "github.com/go-sql-driver/mysql"
packages = ["."]
pruneopts = "UT"
revision = "d523deb1b23d913de5bdada721a6071e71283618"
version = "v1.4.0"
[[projects]]
branch = "master"
digest = "1:2b43ef0cd185eb036ccd68b629c33aeb10d4ee3264a3912ca25ef1a5ade9c439"
name = "github.com/gocql/gocql"
packages = [
".",
"internal/lru",
"internal/murmur",
"internal/streams",
]
pruneopts = "UT"
revision = "e06f8c1bcd787e6bf0608288b314522f08cc7848"
[[projects]]
digest = "1:d6d8ef66dbf92848883e228323ca4ee7404eb4fe3681c0e0395f515f128e0d29"
name = "github.com/golang/protobuf"
packages = [
"proto",
"protoc-gen-go/descriptor",
"ptypes",
"ptypes/any",
"ptypes/duration",
"ptypes/empty",
"ptypes/struct",
"ptypes/timestamp",
]
pruneopts = "UT"
revision = "b4deda0973fb4c70b50d226b1af49f3da59f5265"
version = "v1.1.0"
[[projects]]
branch = "master"
digest = "1:4a0c6bb4805508a6287675fac876be2ac1182539ca8a32468d8128882e9d5009"
name = "github.com/golang/snappy"
packages = ["."]
pruneopts = "UT"
revision = "2e65f85255dbc3072edf28d6b5b8efc472979f5a"
[[projects]]
digest = "1:51bee9f1987dcdb9f9a1b4c20745d78f6bf6f5f14ad4e64ca883eb64df4c0045"
name = "github.com/google/go-github"
packages = ["github"]
pruneopts = "UT"
revision = "e48060a28fac52d0f1cb758bc8b87c07bac4a87d"
version = "v15.0.0"
[[projects]]
branch = "master"
digest = "1:a63cff6b5d8b95638bfe300385d93b2a6d9d687734b863da8e09dc834510a690"
name = "github.com/google/go-querystring"
packages = ["query"]
pruneopts = "UT"
revision = "53e6ce116135b80d037921a7fdd5138cf32d7a8a"
[[projects]]
digest = "1:e145e9710a10bc114a6d3e2738aadf8de146adaa031854ffdf7bbfe15da85e63"
name = "github.com/googleapis/gax-go"
packages = ["."]
pruneopts = "UT"
revision = "317e0006254c44a0ac427cc52a0e083ff0b9622f"
version = "v2.0.0"
[[projects]]
digest = "1:c79fb010be38a59d657c48c6ba1d003a8aa651fa56b579d959d74573b7dff8e1"
name = "github.com/gorilla/context"
packages = ["."]
pruneopts = "UT"
revision = "08b5f424b9271eedf6f9f0ce86cb9396ed337a42"
version = "v1.1.1"
[[projects]]
digest = "1:e73f5b0152105f18bc131fba127d9949305c8693f8a762588a82a48f61756f5f"
name = "github.com/gorilla/mux"
packages = ["."]
pruneopts = "UT"
revision = "e3702bed27f0d39777b0b37b664b6280e8ef8fbf"
version = "v1.6.2"
[[projects]]
branch = "master"
digest = "1:364b908b9b27b97ab838f2f6f1b1f46281fa29b978a037d72a9b1d4f6d940190"
name = "github.com/hailocab/go-hostpool"
packages = ["."]
pruneopts = "UT"
revision = "e80d13ce29ede4452c43dea11e79b9bc8a15b478"
[[projects]]
digest = "1:e22af8c7518e1eab6f2eab2b7d7558927f816262586cd6ed9f349c97a6c285c4"
name = "github.com/jmespath/go-jmespath"
packages = ["."]
pruneopts = "UT"
revision = "0b12d6b5"
[[projects]]
digest = "1:729cb96e4ff32f992aab19a58384fc4304c05f414a9169ef46b8c6ebb5515f77"
name = "github.com/kshvakov/clickhouse"
packages = [
".",
"lib/binary",
"lib/column",
"lib/data",
"lib/protocol",
"lib/types",
"lib/writebuffer",
]
pruneopts = "UT"
revision = "8a2dd1e831a6c7381c11d40bcaf80ae98023c605"
version = "v1.3.3"
[[projects]]
branch = "master"
digest = "1:37ce7d7d80531b227023331002c0d42b4b4b291a96798c82a049d03a54ba79e4"
name = "github.com/lib/pq"
packages = [
".",
"oid",
]
pruneopts = "UT"
revision = "90697d60dd844d5ef6ff15135d0203f65d2f53b8"
[[projects]]
digest = "1:3cafc6a5a1b8269605d9df4c6956d43d8011fc57f266ca6b9d04da6c09dee548"
name = "github.com/mattn/go-sqlite3"
packages = ["."]
pruneopts = "UT"
revision = "25ecb14adfc7543176f7d85291ec7dba82c6f7e4"
version = "v1.9.0"
[[projects]]
digest = "1:ee4d4af67d93cc7644157882329023ce9a7bcfce956a079069a9405521c7cc8d"
name = "github.com/opencontainers/go-digest"
packages = ["."]
pruneopts = "UT"
revision = "279bed98673dd5bef374d3b6e4b09e2af76183bf"
version = "v1.0.0-rc1"
[[projects]]
digest = "1:40e195917a951a8bf867cd05de2a46aaf1806c50cf92eebf4c16f78cd196f747"
name = "github.com/pkg/errors"
packages = ["."]
pruneopts = "UT"
revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
version = "v0.8.0"
[[projects]]
digest = "1:4ed20c76a0526d584d895c0a21c518c1d5a84ff97bd3c5828f3682016445ed39"
name = "go.opencensus.io"
packages = [
".",
"exporter/stackdriver/propagation",
"internal",
"internal/tagencoding",
"plugin/ocgrpc",
"plugin/ochttp",
"plugin/ochttp/propagation/b3",
"stats",
"stats/internal",
"stats/view",
"tag",
"trace",
"trace/internal",
"trace/propagation",
]
pruneopts = "UT"
revision = "e262766cd0d230a1bb7c37281e345e465f19b41b"
version = "v0.14.0"
[[projects]]
branch = "master"
digest = "1:3f3a05ae0b95893d90b9b3b5afdb79a9b3d96e4e36e099d841ae602e4aca0da8"
name = "golang.org/x/crypto"
packages = ["ssh/terminal"]
pruneopts = "UT"
revision = "c126467f60eb25f8f27e5a981f32a87e3965053f"
[[projects]]
branch = "master"
digest = "1:4d337ebc84e0ed0205f84f919934e70d830df9c86fc429ca8ce7ac286e29d247"
name = "golang.org/x/net"
packages = [
"context",
"context/ctxhttp",
"http/httpguts",
"http2",
"http2/hpack",
"idna",
"internal/socks",
"internal/timeseries",
"proxy",
"trace",
]
pruneopts = "UT"
revision = "3673e40ba22529d22c3fd7c93e97b0ce50fa7bdd"
[[projects]]
branch = "master"
digest = "1:bea0314c10bd362ab623af4880d853b5bad3b63d0ab9945c47e461b8d04203ed"
name = "golang.org/x/oauth2"
packages = [
".",
"google",
"internal",
"jws",
"jwt",
]
pruneopts = "UT"
revision = "3d292e4d0cdc3a0113e6d207bb137145ef1de42f"
[[projects]]
branch = "master"
digest = "1:be95d758fc4d9216e5d41ff5b98b46938fba85ca5bc2ddc45a1b03e2bb10fe7c"
name = "golang.org/x/sys"
packages = [
"unix",
"windows",
]
pruneopts = "UT"
revision = "e072cadbbdc8dd3d3ffa82b8b4b9304c261d9311"
[[projects]]
digest = "1:a2ab62866c75542dd18d2b069fec854577a20211d7c0ea6ae746072a1dccdd18"
name = "golang.org/x/text"
packages = [
"collate",
"collate/build",
"internal/colltab",
"internal/gen",
"internal/tag",
"internal/triegen",
"internal/ucd",
"language",
"secure/bidirule",
"transform",
"unicode/bidi",
"unicode/cldr",
"unicode/norm",
"unicode/rangetable",
]
pruneopts = "UT"
revision = "f21a4dfb5e38f5895301dc265a8def02365cc3d0"
version = "v0.3.0"
[[projects]]
branch = "master"
digest = "1:7db86209b441978ca91b574ec2ec5a12a6c39bc89b659ca667d99eb96521d587"
name = "golang.org/x/tools"
packages = [
"godoc/vfs",
"godoc/vfs/mapfs",
]
pruneopts = "UT"
revision = "526516e9c4bf5a3f76fe506dec01ca51d04cbe50"
[[projects]]
branch = "master"
digest = "1:2892dda36e78635b8ef9e64be5af673150ff9a6c5e3946e9102353e9f01d8b81"
name = "google.golang.org/api"
packages = [
"gensupport",
"googleapi",
"googleapi/internal/uritemplates",
"googleapi/transport",
"internal",
"iterator",
"option",
"storage/v1",
"transport",
"transport/grpc",
"transport/http",
]
pruneopts = "UT"
revision = "2c45710c7f3fb0ab63506810a1ba84325ab90ab8"
[[projects]]
digest = "1:0e781d9592d20f30c4280da30f27b448b4815215f9b43b5258b763a79fb98e98"
name = "google.golang.org/appengine"
packages = [
".",
"cloudsql",
"internal",
"internal/app_identity",
"internal/base",
"internal/datastore",
"internal/log",
"internal/modules",
"internal/remote_api",
"internal/socket",
"internal/urlfetch",
"socket",
"urlfetch",
]
pruneopts = "UT"
revision = "b1f26356af11148e710935ed1ac8a7f5702c7612"
version = "v1.1.0"
[[projects]]
branch = "master"
digest = "1:7381a470b84bba1d4ded5a158604b04bd002dbb265a63f416a9329bfeee095c7"
name = "google.golang.org/genproto"
packages = [
"googleapis/api/annotations",
"googleapis/iam/v1",
"googleapis/longrunning",
"googleapis/rpc/code",
"googleapis/rpc/errdetails",
"googleapis/rpc/status",
"googleapis/spanner/admin/database/v1",
"googleapis/spanner/v1",
]
pruneopts = "UT"
revision = "02b4e95473316948020af0b7a4f0f22c73929b0e"
[[projects]]
digest = "1:686525281321f81747cfb0bba0bc384511bbdc5cdc92a6aa1afe5f1808f82c36"
name = "google.golang.org/grpc"
packages = [
".",
"balancer",
"balancer/base",
"balancer/roundrobin",
"codes",
"connectivity",
"credentials",
"credentials/oauth",
"encoding",
"encoding/proto",
"grpclog",
"internal",
"internal/backoff",
"internal/channelz",
"internal/grpcrand",
"keepalive",
"metadata",
"naming",
"peer",
"resolver",
"resolver/dns",
"resolver/passthrough",
"stats",
"status",
"tap",
"transport",
]
pruneopts = "UT"
revision = "168a6198bcb0ef175f7dacec0b8691fc141dc9b8"
version = "v1.13.0"
[[projects]]
digest = "1:2d1fbdc6777e5408cabeb02bf336305e724b925ff4546ded0fa8715a7267922a"
name = "gopkg.in/inf.v0"
packages = ["."]
pruneopts = "UT"
revision = "d2d2541c53f18d2a059457998ce2876cc8e67cbf"
version = "v0.9.1"
[solve-meta]
analyzer-name = "dep"
analyzer-version = 1
input-imports = [
"cloud.google.com/go/spanner",
"cloud.google.com/go/spanner/admin/database/apiv1",
"cloud.google.com/go/storage",
"github.com/aws/aws-sdk-go/aws",
"github.com/aws/aws-sdk-go/aws/session",
"github.com/aws/aws-sdk-go/service/s3",
"github.com/aws/aws-sdk-go/service/s3/s3iface",
"github.com/cockroachdb/cockroach-go/crdb",
"github.com/cznic/ql/driver",
"github.com/docker/docker/api/types",
"github.com/docker/docker/api/types/container",
"github.com/docker/docker/api/types/network",
"github.com/docker/docker/client",
"github.com/fsouza/fake-gcs-server/fakestorage",
"github.com/go-sql-driver/mysql",
"github.com/gocql/gocql",
"github.com/google/go-github/github",
"github.com/kshvakov/clickhouse",
"github.com/lib/pq",
"github.com/mattn/go-sqlite3",
"golang.org/x/net/context",
"golang.org/x/tools/godoc/vfs",
"golang.org/x/tools/godoc/vfs/mapfs",
"google.golang.org/api/iterator",
"google.golang.org/genproto/googleapis/spanner/admin/database/v1",
]
solver-name = "gps-cdcl"
solver-version = 1
+98
View File
@@ -0,0 +1,98 @@
# Gopkg.toml example
#
# Refer to https://golang.github.io/dep/docs/Gopkg.toml.html
# for detailed Gopkg.toml documentation.
#
# required = ["github.com/user/thing/cmd/thing"]
# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
#
# [[constraint]]
# name = "github.com/user/project"
# version = "1.0.0"
#
# [[constraint]]
# name = "github.com/user/project2"
# branch = "dev"
# source = "github.com/myfork/project2"
#
# [[override]]
# name = "github.com/x/y"
# version = "2.4.0"
#
# [prune]
# non-go = false
# go-tests = true
# unused-packages = true
[[constraint]]
name = "github.com/aws/aws-sdk-go"
version = "1.15.34"
[[constraint]]
branch = "master"
name = "github.com/cockroachdb/cockroach-go"
[[constraint]]
name = "github.com/cznic/ql"
version = "1.2.0"
[[constraint]]
name = "github.com/docker/docker"
version = "v17.05.0-ce"
[[constraint]]
name = "github.com/fsouza/fake-gcs-server"
version = "1.2.0"
[[constraint]]
name = "github.com/go-sql-driver/mysql"
version = "v1.4.0"
[[constraint]]
branch = "master"
name = "github.com/gocql/gocql"
[[constraint]]
name = "github.com/google/go-github"
version = "17.0.0"
[[constraint]]
name = "github.com/kshvakov/clickhouse"
version = "1.3.4"
[[constraint]]
name = "github.com/lib/pq"
version = "1.0.0"
[[constraint]]
name = "github.com/mattn/go-sqlite3"
version = "1.9.0"
[[constraint]]
branch = "master"
name = "golang.org/x/net"
[[constraint]]
branch = "master"
name = "google.golang.org/api"
[[constraint]]
branch = "master"
name = "google.golang.org/genproto"
[[override]]
name = "cloud.google.com/go"
version = "0.27.0"
[[override]]
branch = "master"
name = "golang.org/x/tools"
[[override]]
branch = "master"
name = "github.com/docker/distribution"
[prune]
go-tests = true
unused-packages = true
+4 -2
View File
@@ -1,6 +1,8 @@
The MIT License (MIT)
Copyright (c) 2014 Matthias Kadenbach
Copyright (c) 2016 Matthias Kadenbach
https://github.com/mattes/migrate
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
@@ -18,4 +20,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
THE SOFTWARE.
+81
View File
@@ -0,0 +1,81 @@
# Migrations
## Migration Filename Format
A single logical migration is represented as two separate migration files, one
to migrate "up" to the specified version from the previous version, and a second
to migrate back "down" to the previous version. These migrations can be provided
by any one of the supported [migration sources](./README.md#migration-sources).
The ordering and direction of the migration files is determined by the filenames
used for them. `migrate` expects the filenames of migrations to have the format:
{version}_{title}.up.{extension}
{version}_{title}.down.{extension}
The `title` of each migration is unused, and is only for readability. Similarly,
the `extension` of the migration files is not checked by the library, and should
be an appropriate format for the database in use (`.sql` for SQL variants, for
instance).
Versions of migrations may be represented as any 64 bit unsigned integer.
All migrations are applied upward in order of increasing version number, and
downward by decreasing version number.
Common versioning schemes include incrementing integers:
1_initialize_schema.down.sql
1_initialize_schema.up.sql
2_add_table.down.sql
2_add_table.up.sql
...
Or timestamps at an appropriate resolution:
1500360784_initialize_schema.down.sql
1500360784_initialize_schema.up.sql
1500445949_add_table.down.sql
1500445949_add_table.up.sql
...
But any scheme resulting in distinct, incrementing integers as versions is valid.
It is suggested that the version number of corresponding `up` and `down` migration
files be equivalent for clarity, but they are allowed to differ so long as the
relative ordering of the migrations is preserved.
The migration files are permitted to be empty, so in the event that a migration
is a no-op or is irreversible, it is recommended to still include both migration
files, and either leaving them empty or adding a comment as appropriate.
## Migration Content Format
The format of the migration files themselves varies between database systems.
Different databases have different semantics around schema changes and when and
how they are allowed to occur
(for instance, [if schema changes can occur within a transaction](https://wiki.postgresql.org/wiki/Transactional_DDL_in_PostgreSQL:_A_Competitive_Analysis)).
As such, the `migrate` library has little to no checking around the format of
migration sources. The migration files are generally processed directly by the
drivers as raw operations.
## Reversibility of Migrations
Best practice for writing schema migration is that all migrations should be
reversible. It should in theory be possible for run migrations down and back up
through any and all versions with the state being fully cleaned and recreated
by doing so.
By adhering to this recommended practice, development and deployment of new code
is cleaner and easier (cleaning database state for a new feature should be as
easy as migrating down to a prior version, and back up to the latest).
As opposed to some other migration libraries, `migrate` represents up and down
migrations as separate files. This prevents any non-standard file syntax from
being introduced which may result in unintended behavior or errors, depending
on what database is processing the file.
While it is technically possible for an up or down migration to exist on its own
without an equivalently versioned counterpart, it is strongly recommended to
always include a down migration which cleans up the state of the corresponding
up migration.
+108 -20
View File
@@ -1,30 +1,118 @@
TESTFLAGS?=
IMAGE=mattes/migrate
DCR=docker-compose run --rm
GOTEST=go test $(TESTFLAGS) `go list ./... | grep -v "/vendor/"`
SOURCE ?= file go_bindata github aws_s3 google_cloud_storage godoc_vfs
DATABASE ?= postgres mysql redshift cassandra spanner cockroachdb clickhouse
VERSION ?= $(shell git describe --tags 2>/dev/null | cut -c 2-)
TEST_FLAGS ?=
REPO_OWNER ?= $(shell cd .. && basename "$$(pwd)")
COVERAGE_DIR ?= .coverage
build-cli: clean
-mkdir ./cli/build
cd ./cli && CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o build/migrate.linux-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cli && CGO_ENABLED=0 GOOS=darwin GOARCH=amd64 go build -a -o build/migrate.darwin-amd64 -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cli && CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build -a -o build/migrate.windows-amd64.exe -ldflags='-X main.Version=$(VERSION) -extldflags "-static"' -tags '$(DATABASE) $(SOURCE)' .
cd ./cli/build && find . -name 'migrate*' | xargs -I{} tar czf {}.tar.gz {}
cd ./cli/build && shasum -a 256 * > sha256sum.txt
cat ./cli/build/sha256sum.txt
.PHONY: clean test build release docker-build docker-push run
all: release
clean:
rm -f migrate
-rm -r ./cli/build
fmt:
@gofmt -s -w `go list -f {{.Dir}} ./... | grep -v "/vendor/"`
test: fmt
$(DCR) go-test
test-short:
make test-with-flags --ignore-errors TEST_FLAGS='-short'
go-test: fmt
@$(GOTEST)
build:
$(DCR) go-build
test:
@-rm -r $(COVERAGE_DIR)
@mkdir $(COVERAGE_DIR)
make test-with-flags TEST_FLAGS='-v -race -covermode atomic -coverprofile $$(COVERAGE_DIR)/_$$(RAND).txt -bench=. -benchmem -timeout 20m'
@echo 'mode: atomic' > $(COVERAGE_DIR)/combined.txt
@cat $(COVERAGE_DIR)/_*.txt | grep -v 'mode: atomic' >> $(COVERAGE_DIR)/combined.txt
release: test build docker-build docker-push
docker-build:
docker build --rm -t $(IMAGE) .
test-with-flags:
@echo SOURCE: $(SOURCE)
@echo DATABASE: $(DATABASE)
@go test $(TEST_FLAGS) .
@go test $(TEST_FLAGS) ./cli/...
@go test $(TEST_FLAGS) ./database
@go test $(TEST_FLAGS) ./testing/...
@echo -n '$(SOURCE)' | tr -s ' ' '\n' | xargs -I{} go test $(TEST_FLAGS) ./source/{}
@go test $(TEST_FLAGS) ./source/testing/...
@go test $(TEST_FLAGS) ./source/stub/...
@echo -n '$(DATABASE)' | tr -s ' ' '\n' | xargs -I{} go test $(TEST_FLAGS) ./database/{}
@go test $(TEST_FLAGS) ./database/testing/...
@go test $(TEST_FLAGS) ./database/stub/...
kill-orphaned-docker-containers:
docker rm -f $(shell docker ps -aq --filter label=migrate_test)
html-coverage:
go tool cover -html=$(COVERAGE_DIR)/combined.txt
list-external-deps:
$(call external_deps,'.')
$(call external_deps,'./cli/...')
$(call external_deps,'./testing/...')
$(foreach v, $(SOURCE), $(call external_deps,'./source/$(v)/...'))
$(call external_deps,'./source/testing/...')
$(call external_deps,'./source/stub/...')
$(foreach v, $(DATABASE), $(call external_deps,'./database/$(v)/...'))
$(call external_deps,'./database/testing/...')
$(call external_deps,'./database/stub/...')
restore-import-paths:
find . -name '*.go' -type f -execdir sed -i '' s%\"github.com/$(REPO_OWNER)/migrate%\"github.com/mattes/migrate%g '{}' \;
rewrite-import-paths:
find . -name '*.go' -type f -execdir sed -i '' s%\"github.com/mattes/migrate%\"github.com/$(REPO_OWNER)/migrate%g '{}' \;
# example: fswatch -0 --exclude .godoc.pid --event Updated . | xargs -0 -n1 -I{} make docs
docs:
-make kill-docs
nohup godoc -play -http=127.0.0.1:6064 </dev/null >/dev/null 2>&1 & echo $$! > .godoc.pid
cat .godoc.pid
kill-docs:
@cat .godoc.pid
kill -9 $$(cat .godoc.pid)
rm .godoc.pid
open-docs:
open http://localhost:6064/pkg/github.com/$(REPO_OWNER)/migrate
# example: make release V=0.0.0
release:
git tag v$(V)
@read -p "Press enter to confirm and push to origin ..." && git push origin v$(V)
define external_deps
@echo '-- $(1)'; go list -f '{{join .Deps "\n"}}' $(1) | grep -v github.com/$(REPO_OWNER)/migrate | xargs go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}'
endef
.PHONY: build-cli clean test-short test test-with-flags html-coverage \
restore-import-paths rewrite-import-paths list-external-deps release \
docs kill-docs open-docs kill-orphaned-docker-containers
SHELL = /bin/bash
RAND = $(shell echo $$RANDOM)
docker-push:
docker push $(IMAGE)
+136 -92
View File
@@ -1,124 +1,168 @@
[![Build Status](https://img.shields.io/travis/golang-migrate/migrate/master.svg)](https://travis-ci.org/golang-migrate/migrate)
[![GoDoc](https://godoc.org/github.com/golang-migrate/migrate?status.svg)](https://godoc.org/github.com/golang-migrate/migrate)
[![Coverage Status](https://img.shields.io/coveralls/github/golang-migrate/migrate/master.svg)](https://coveralls.io/github/golang-migrate/migrate?branch=master)
[![packagecloud.io](https://img.shields.io/badge/deb-packagecloud.io-844fec.svg)](https://packagecloud.io/golang-migrate/migrate?filter=debs)
[![Docker Pulls](https://img.shields.io/docker/pulls/migrate/migrate.svg)](https://hub.docker.com/r/migrate/migrate/)
![Supported Go Versions](https://img.shields.io/badge/Go-1.10%2C%201.11-lightgrey.svg)
[![GitHub Release](https://img.shields.io/github/release/golang-migrate/migrate.svg)](https://github.com/golang-migrate/migrate/releases)
# migrate
[![Build Status](https://travis-ci.org/mattes/migrate.svg?branch=master)](https://travis-ci.org/mattes/migrate)
[![GoDoc](https://godoc.org/github.com/mattes/migrate?status.svg)](https://godoc.org/github.com/mattes/migrate)
__Database migrations written in Go. Use as [CLI](#cli-usage) or import as [library](#use-in-your-go-project).__
A migration helper written in Go. Use it in your existing Golang code
or run commands via the CLI.
```
GoCode import github.com/mattes/migrate/migrate
CLI go get -u github.com/mattes/migrate
```
__Features__
* Super easy to implement [Driver interface](http://godoc.org/github.com/mattes/migrate/driver#Driver).
* Gracefully quit running migrations on ``^C``.
* No magic search paths routines, no hard-coded config files.
* CLI is build on top of the ``migrate package``.
* Migrate reads migrations from [sources](#migration-sources)
and applies them in correct order to a [database](#databases).
* Drivers are "dumb", migrate glues everything together and makes sure the logic is bulletproof.
(Keeps the drivers lightweight, too.)
* Database drivers don't assume things or try to correct user input. When in doubt, fail.
## Available Drivers
Looking for [v1](https://github.com/golang-migrate/migrate/tree/v1)?
* [PostgreSQL](https://github.com/mattes/migrate/tree/master/driver/postgres)
* [Cassandra](https://github.com/mattes/migrate/tree/master/driver/cassandra)
* [SQLite](https://github.com/mattes/migrate/tree/master/driver/sqlite3)
* [MySQL](https://github.com/mattes/migrate/tree/master/driver/mysql) ([experimental](https://github.com/mattes/migrate/issues/1#issuecomment-58728186))
* Bash (planned)
Need another driver? Just implement the [Driver interface](http://godoc.org/github.com/mattes/migrate/driver#Driver) and open a PR.
Forked from [mattes/migrate](https://github.com/mattes/migrate)
## Usage from Terminal
## Databases
Database drivers run migrations. [Add a new database?](database/driver.go)
* [PostgreSQL](database/postgres)
* [Redshift](database/redshift)
* [Ql](database/ql)
* [Cassandra](database/cassandra)
* [SQLite](database/sqlite3) ([todo #165](https://github.com/mattes/migrate/issues/165))
* [MySQL/ MariaDB](database/mysql)
* [Neo4j](database/neo4j) ([todo #167](https://github.com/mattes/migrate/issues/167))
* [MongoDB](database/mongodb) ([todo #169](https://github.com/mattes/migrate/issues/169))
* [CrateDB](database/crate) ([todo #170](https://github.com/mattes/migrate/issues/170))
* [Shell](database/shell) ([todo #171](https://github.com/mattes/migrate/issues/171))
* [Google Cloud Spanner](database/spanner)
* [CockroachDB](database/cockroachdb)
* [ClickHouse](database/clickhouse)
### Database URLs
Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: `dbdriver://username:password@host:port/dbname?option1=true&option2=false`
Any [reserved URL characters](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_reserved_characters) need to be escaped. Note, the `%` character also [needs to be escaped](https://en.wikipedia.org/wiki/Percent-encoding#Percent-encoding_the_percent_character)
Explicitly, the following characters need to be escaped:
`!`, `#`, `$`, `%`, `&`, `'`, `(`, `)`, `*`, `+`, `,`, `/`, `:`, `;`, `=`, `?`, `@`, `[`, `]`
It's easiest to always run the URL parts of your DB connection URL (e.g. username, password, etc) through an URL encoder. See the example Python helpers below:
```bash
# install
go get github.com/mattes/migrate
# create new migration file in path
migrate -url driver://url -path ./migrations create migration_file_xyz
# apply all available migrations
migrate -url driver://url -path ./migrations up
# roll back all migrations
migrate -url driver://url -path ./migrations down
# roll back the most recently applied migration, then run it again.
migrate -url driver://url -path ./migrations redo
# run down and then up command
migrate -url driver://url -path ./migrations reset
# show the current migration version
migrate -url driver://url -path ./migrations version
# apply the next n migrations
migrate -url driver://url -path ./migrations migrate +1
migrate -url driver://url -path ./migrations migrate +2
migrate -url driver://url -path ./migrations migrate +n
# roll back the previous n migrations
migrate -url driver://url -path ./migrations migrate -1
migrate -url driver://url -path ./migrations migrate -2
migrate -url driver://url -path ./migrations migrate -n
# go to specific migration
migrate -url driver://url -path ./migrations goto 1
migrate -url driver://url -path ./migrations goto 10
migrate -url driver://url -path ./migrations goto v
$ python3 -c 'import urllib.parse; print(urllib.parse.quote(input("String to encode: "), ""))'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$ python2 -c 'import urllib; print urllib.quote(raw_input("String to encode: "), "")'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$
```
## Migration Sources
## Usage in Go
Source drivers read migrations from local or remote sources. [Add a new source?](source/driver.go)
See GoDoc here: http://godoc.org/github.com/mattes/migrate/migrate
* [Filesystem](source/file) - read from fileystem
* [Go-Bindata](source/go_bindata) - read from embedded binary data ([jteeuwen/go-bindata](https://github.com/jteeuwen/go-bindata))
* [Github](source/github) - read from remote Github repositories
* [AWS S3](source/aws_s3) - read from Amazon Web Services S3
* [Google Cloud Storage](source/google_cloud_storage) - read from Google Cloud Platform Storage
## CLI usage
* Simple wrapper around this library.
* Handles ctrl+c (SIGINT) gracefully.
* No config search paths, no config files, no magic ENV var injections.
__[CLI Documentation](cli)__
### Basic usage:
```
$ migrate -source file://path/to/migrations -database postgres://localhost:5432/database up 2
```
### Docker usage
```
$ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
-path=/migrations/ -database postgres://localhost:5432/database up 2
```
## Use in your Go project
* API is stable and frozen for this release (v3.x).
* Uses [dep](https://github.com/golang/dep) to manage dependencies
* To help prevent database corruptions, it supports graceful stops via `GracefulStop chan bool`.
* Bring your own logger.
* Uses `io.Reader` streams internally for low memory overhead.
* Thread-safe and no goroutine leaks.
__[Go Documentation](https://godoc.org/github.com/golang-migrate/migrate)__
```go
import "github.com/mattes/migrate/migrate"
import (
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/postgres"
_ "github.com/golang-migrate/migrate/v4/source/github"
)
// Import any required drivers so that they are registered and available
import _ "github.com/mattes/migrate/driver/mysql"
// use synchronous versions of migration functions ...
allErrors, ok := migrate.UpSync("driver://url", "./path")
if !ok {
fmt.Println("Oh no ...")
// do sth with allErrors slice
func main() {
m, err := migrate.New(
"github://mattes:personal-access-token@mattes/migrate_test",
"postgres://localhost:5432/database?sslmode=enable")
m.Steps(2)
}
```
// use the asynchronous version of migration functions ...
pipe := migrate.NewPipe()
go migrate.Up(pipe, "driver://url", "./path")
// pipe is basically just a channel
// write your own channel listener. see writePipe() in main.go as an example.
Want to use an existing database client?
```go
import (
"database/sql"
_ "github.com/lib/pq"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database/postgres"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func main() {
db, err := sql.Open("postgres", "postgres://localhost:5432/database?sslmode=enable")
driver, err := postgres.WithInstance(db, &postgres.Config{})
m, err := migrate.NewWithDatabaseInstance(
"file:///migrations",
"postgres", driver)
m.Steps(2)
}
```
## Migration files
The format of migration files looks like this:
Each migration has an up and down migration. [Why?](FAQ.md#why-two-separate-files-up-and-down-for-a-migration)
```
1481574547_initial_plan_to_do_sth.up.sql # up migration instructions
1481574547_initial_plan_to_do_sth.down.sql # down migration instructions
1482438365_xxx.up.sql
1482438365_xxx.down.sql
...
1481574547_create_users_table.up.sql
1481574547_create_users_table.down.sql
```
Why two files? This way you could still do sth like
``psql -f ./db/migrations/1481574547_initial_plan_to_do_sth.up.sql`` and there is no
need for any custom markup language to divide up and down migrations. Please note
that the filename extension depends on the driver.
[Best practices: How to write migrations.](MIGRATIONS.md)
## Alternatives
* https://bitbucket.org/liamstask/goose
* https://github.com/tanel/dbmigrate
* https://github.com/BurntSushi/migration
* https://github.com/DavidHuie/gomigrate
* https://github.com/rubenv/sql-migrate
## Development and Contributing
Yes, please! [`Makefile`](Makefile) is your friend,
read the [development guide](CONTRIBUTING.md).
Also have a look at the [FAQ](FAQ.md).
---
Looking for alternatives? [https://awesome-go.com/#database](https://awesome-go.com/#database).
+117
View File
@@ -0,0 +1,117 @@
# migrate CLI
## Installation
#### Download pre-build binary (Windows, MacOS, or Linux)
[Release Downloads](https://github.com/golang-migrate/migrate/releases)
```
$ curl -L https://github.com/golang-migrate/migrate/releases/download/$version/migrate.$platform-amd64.tar.gz | tar xvz
```
#### MacOS
We have not released support for homebrew yet, but there is a live issue here: [todo #156](https://github.com/mattes/migrate/issues/156)
Any help to make this happen would be appreciated!
#### Linux (*.deb package)
```
$ curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
$ echo "deb https://packagecloud.io/golang-migrate/migrate/ubuntu/ xenial main" > /etc/apt/sources.list.d/migrate.list
$ apt-get update
$ apt-get install -y migrate
```
#### With Go toolchain
```
$ go get -u -d github.com/golang-migrate/migrate/cli
$ cd $GOPATH/src/github.com/golang-migrate/migrate/cli
$ dep ensure
$ go build -tags 'postgres' -o /usr/local/bin/migrate github.com/golang-migrate/migrate/cli
```
##### Notes:
1. This example builds the cli which will only work with postgres. In order
to build the cli for use with other databases, replace the `postgres` build tag
with the appropriate database tag(s) for the databases desired. The tags
correspond to the names of the sub-packages underneath the
[`database`](../database) package.
1. Similarly to the database build tags, if you need to support other sources, use the appropriate build tag(s).
1. Support for build constraints will be removed in the future: https://github.com/golang-migrate/migrate/issues/60
## Usage
```
$ migrate -help
Usage: migrate OPTIONS COMMAND [arg...]
migrate [ -version | -help ]
Options:
-source Location of the migrations (driver://url)
-path Shorthand for -source=file://path
-database Run migrations against this database (driver://url)
-prefetch N Number of migrations to load in advance before executing (default 10)
-lock-timeout N Allow N seconds to acquire database lock (default 15)
-verbose Print verbose logging
-version Print version
-help Print usage
Commands:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string.
goto V Migrate to version V
up [N] Apply all or N up migrations
down [N] Apply all or N down migrations
drop Drop everyting inside database
force V Set version V but don't run migration (ignores dirty state)
version Print current migration version
```
So let's say you want to run the first two migrations
```
$ migrate -source file://path/to/migrations -database postgres://localhost:5432/database up 2
```
If your migrations are hosted on github
```
$ migrate -source github://mattes:personal-access-token@mattes/migrate_test \
-database postgres://localhost:5432/database down 2
```
The CLI will gracefully stop at a safe point when SIGINT (ctrl+c) is received.
Send SIGKILL for immediate halt.
## Reading CLI arguments from somewhere else
##### ENV variables
```
$ migrate -database "$MY_MIGRATE_DATABASE"
```
##### JSON files
Check out https://stedolan.github.io/jq/
```
$ migrate -database "$(cat config.json | jq '.database')"
```
##### YAML files
````
$ migrate -database "$(cat config/database.yml | ruby -ryaml -e "print YAML.load(STDIN.read)['database']")"
$ migrate -database "$(cat config/database.yml | python -c 'import yaml,sys;print yaml.safe_load(sys.stdin)["database"]')"
```
+7
View File
@@ -0,0 +1,7 @@
// +build aws_s3
package main
import (
_ "github.com/golang-migrate/migrate/v4/source/aws_s3"
)
+7
View File
@@ -0,0 +1,7 @@
// +build cassandra
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/cassandra"
)
+8
View File
@@ -0,0 +1,8 @@
// +build clickhouse
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/clickhouse"
_ "github.com/kshvakov/clickhouse"
)
+7
View File
@@ -0,0 +1,7 @@
// +build cockroachdb
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/cockroachdb"
)
+7
View File
@@ -0,0 +1,7 @@
// +build github
package main
import (
_ "github.com/golang-migrate/migrate/v4/source/github"
)
+7
View File
@@ -0,0 +1,7 @@
// +build go_bindata
package main
import (
_ "github.com/golang-migrate/migrate/v4/source/go_bindata"
)
+7
View File
@@ -0,0 +1,7 @@
// +build godoc_vfs
package main
import (
_ "github.com/golang-migrate/migrate/v4/source/godoc_vfs"
)
+7
View File
@@ -0,0 +1,7 @@
// +build google_cloud_storage
package main
import (
_ "github.com/golang-migrate/migrate/v4/source/google_cloud_storage"
)
+7
View File
@@ -0,0 +1,7 @@
// +build mysql
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/mysql"
)
+7
View File
@@ -0,0 +1,7 @@
// +build postgres
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/postgres"
)
+7
View File
@@ -0,0 +1,7 @@
// +build ql
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/ql"
)
+7
View File
@@ -0,0 +1,7 @@
// +build redshift
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/redshift"
)
+7
View File
@@ -0,0 +1,7 @@
// +build spanner
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/spanner"
)
+7
View File
@@ -0,0 +1,7 @@
// +build sqlite3
package main
import (
_ "github.com/golang-migrate/migrate/v4/database/sqlite3"
)
+166
View File
@@ -0,0 +1,166 @@
package main
import (
"errors"
"fmt"
"github.com/golang-migrate/migrate/v4"
_ "github.com/golang-migrate/migrate/v4/database/stub" // TODO remove again
_ "github.com/golang-migrate/migrate/v4/source/file"
"os"
"path/filepath"
"strconv"
"strings"
"time"
)
func nextSeq(matches []string, dir string, seqDigits int) (string, error) {
if seqDigits <= 0 {
return "", errors.New("Digits must be positive")
}
nextSeq := 1
if len(matches) > 0 {
filename := matches[len(matches)-1]
matchSeqStr := strings.TrimPrefix(filename, dir)
idx := strings.Index(matchSeqStr, "_")
if idx < 1 { // Using 1 instead of 0 since there should be at least 1 digit
return "", errors.New("Malformed migration filename: " + filename)
}
matchSeqStr = matchSeqStr[0:idx]
var err error
nextSeq, err = strconv.Atoi(matchSeqStr)
if err != nil {
return "", err
}
nextSeq++
}
if nextSeq <= 0 {
return "", errors.New("Next sequence number must be positive")
}
nextSeqStr := strconv.Itoa(nextSeq)
if len(nextSeqStr) > seqDigits {
return "", fmt.Errorf("Next sequence number %s too large. At most %d digits are allowed", nextSeqStr, seqDigits)
}
padding := seqDigits - len(nextSeqStr)
if padding > 0 {
nextSeqStr = strings.Repeat("0", padding) + nextSeqStr
}
return nextSeqStr, nil
}
func createCmd(dir string, startTime time.Time, format string, name string, ext string, seq bool, seqDigits int) {
var base string
if seq && format != defaultTimeFormat {
log.fatalErr(errors.New("The seq and format options are mutually exclusive"))
}
if seq {
if seqDigits <= 0 {
log.fatalErr(errors.New("Digits must be positive"))
}
matches, err := filepath.Glob(dir + "*" + ext)
if err != nil {
log.fatalErr(err)
}
nextSeqStr, err := nextSeq(matches, dir, seqDigits)
if err != nil {
log.fatalErr(err)
}
base = fmt.Sprintf("%v%v_%v.", dir, nextSeqStr, name)
} else {
switch format {
case "":
log.fatal("Time format may not be empty")
case "unix":
base = fmt.Sprintf("%v%v_%v.", dir, startTime.Unix(), name)
case "unixNano":
base = fmt.Sprintf("%v%v_%v.", dir, startTime.UnixNano(), name)
default:
base = fmt.Sprintf("%v%v_%v.", dir, startTime.Format(format), name)
}
}
os.MkdirAll(dir, os.ModePerm)
createFile(base + "up" + ext)
createFile(base + "down" + ext)
}
func createFile(fname string) {
if _, err := os.Create(fname); err != nil {
log.fatalErr(err)
}
}
func gotoCmd(m *migrate.Migrate, v uint) {
if err := m.Migrate(v); err != nil {
if err != migrate.ErrNoChange {
log.fatalErr(err)
} else {
log.Println(err)
}
}
}
func upCmd(m *migrate.Migrate, limit int) {
if limit >= 0 {
if err := m.Steps(limit); err != nil {
if err != migrate.ErrNoChange {
log.fatalErr(err)
} else {
log.Println(err)
}
}
} else {
if err := m.Up(); err != nil {
if err != migrate.ErrNoChange {
log.fatalErr(err)
} else {
log.Println(err)
}
}
}
}
func downCmd(m *migrate.Migrate, limit int) {
if limit >= 0 {
if err := m.Steps(-limit); err != nil {
if err != migrate.ErrNoChange {
log.fatalErr(err)
} else {
log.Println(err)
}
}
} else {
if err := m.Down(); err != nil {
if err != migrate.ErrNoChange {
log.fatalErr(err)
} else {
log.Println(err)
}
}
}
}
func dropCmd(m *migrate.Migrate) {
if err := m.Drop(); err != nil {
log.fatalErr(err)
}
}
func forceCmd(m *migrate.Migrate, v int) {
if err := m.Force(v); err != nil {
log.fatalErr(err)
}
}
func versionCmd(m *migrate.Migrate) {
v, dirty, err := m.Version()
if err != nil {
log.fatalErr(err)
}
if dirty {
log.Printf("%v (dirty)\n", v)
} else {
log.Println(v)
}
}
+45
View File
@@ -0,0 +1,45 @@
package main
import (
"testing"
)
func TestNextSeq(t *testing.T) {
cases := []struct {
name string
matches []string
dir string
seqDigits int
expected string
expectedErrStr string
}{
{"Bad digits", []string{}, "migrationDir", 0, "", "Digits must be positive"},
{"Single digit initialize", []string{}, "migrationDir", 1, "1", ""},
{"Single digit malformed", []string{"bad"}, "migrationDir", 1, "", "Malformed migration filename: bad"},
{"Single digit no int", []string{"bad_bad"}, "migrationDir", 1, "", "strconv.Atoi: parsing \"bad\": invalid syntax"},
{"Single digit negative seq", []string{"-5_test"}, "migrationDir", 1, "", "Next sequence number must be positive"},
{"Single digit increment", []string{"3_test", "4_test"}, "migrationDir", 1, "5", ""},
{"Single digit overflow", []string{"9_test"}, "migrationDir", 1, "", "Next sequence number 10 too large. At most 1 digits are allowed"},
{"Zero-pad initialize", []string{}, "migrationDir", 6, "000001", ""},
{"Zero-pad malformed", []string{"bad"}, "migrationDir", 6, "", "Malformed migration filename: bad"},
{"Zero-pad no int", []string{"bad_bad"}, "migrationDir", 6, "", "strconv.Atoi: parsing \"bad\": invalid syntax"},
{"Zero-pad negative seq", []string{"-000005_test"}, "migrationDir", 6, "", "Next sequence number must be positive"},
{"Zero-pad increment", []string{"000003_test", "000004_test"}, "migrationDir", 6, "000005", ""},
{"Zero-pad overflow", []string{"999999_test"}, "migrationDir", 6, "", "Next sequence number 1000000 too large. At most 6 digits are allowed"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
nextSeq, err := nextSeq(c.matches, c.dir, c.seqDigits)
if nextSeq != c.expected {
t.Error("Incorrect nextSeq: " + nextSeq + " != " + c.expected)
}
if err != nil {
if err.Error() != c.expectedErrStr {
t.Error("Incorrect error: " + err.Error() + " != " + c.expectedErrStr)
}
} else if c.expectedErrStr != "" {
t.Error("Expected error: " + c.expectedErrStr + " but got nil instead")
}
})
}
}
+12
View File
@@ -0,0 +1,12 @@
FROM ubuntu:xenial
RUN apt-get update && \
apt-get install -y curl apt-transport-https
RUN curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add - && \
echo "deb https://packagecloud.io/golang-migrate/migrate/ubuntu/ xenial main" > /etc/apt/sources.list.d/migrate.list && \
apt-get update && \
apt-get install -y migrate
RUN migrate -version
+45
View File
@@ -0,0 +1,45 @@
package main
import (
"fmt"
logpkg "log"
"os"
)
type Log struct {
verbose bool
}
func (l *Log) Printf(format string, v ...interface{}) {
if l.verbose {
logpkg.Printf(format, v...)
} else {
fmt.Fprintf(os.Stderr, format, v...)
}
}
func (l *Log) Println(args ...interface{}) {
if l.verbose {
logpkg.Println(args...)
} else {
fmt.Fprintln(os.Stderr, args...)
}
}
func (l *Log) Verbose() bool {
return l.verbose
}
func (l *Log) fatalf(format string, v ...interface{}) {
l.Printf(format, v...)
os.Exit(1)
}
func (l *Log) fatal(args ...interface{}) {
l.Println(args...)
os.Exit(1)
}
func (l *Log) fatalErr(err error) {
l.fatal("error:", err)
}
+250
View File
@@ -0,0 +1,250 @@
package main
import (
"flag"
"fmt"
"os"
"os/signal"
"strconv"
"strings"
"syscall"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/source"
)
const defaultTimeFormat = "20060102150405"
// set main log
var log = &Log{}
func main() {
helpPtr := flag.Bool("help", false, "")
versionPtr := flag.Bool("version", false, "")
verbosePtr := flag.Bool("verbose", false, "")
prefetchPtr := flag.Uint("prefetch", 10, "")
lockTimeoutPtr := flag.Uint("lock-timeout", 15, "")
pathPtr := flag.String("path", "", "")
databasePtr := flag.String("database", "", "")
sourcePtr := flag.String("source", "", "")
flag.Usage = func() {
fmt.Fprint(os.Stderr,
`Usage: migrate OPTIONS COMMAND [arg...]
migrate [ -version | -help ]
Options:
-source Location of the migrations (driver://url)
-path Shorthand for -source=file://path
-database Run migrations against this database (driver://url)
-prefetch N Number of migrations to load in advance before executing (default 10)
-lock-timeout N Allow N seconds to acquire database lock (default 15)
-verbose Print verbose logging
-version Print version
-help Print usage
Commands:
create [-ext E] [-dir D] [-seq] [-digits N] [-format] NAME
Create a set of timestamped up/down migrations titled NAME, in directory D with extension E.
Use -seq option to generate sequential up/down migrations with N digits.
Use -format option to specify a Go time format string.
goto V Migrate to version V
up [N] Apply all or N up migrations
down [N] Apply all or N down migrations
drop Drop everyting inside database
force V Set version V but don't run migration (ignores dirty state)
version Print current migration version
Source drivers: `+strings.Join(source.List(), ", ")+`
Database drivers: `+strings.Join(database.List(), ", ")+"\n")
}
flag.Parse()
// initialize logger
log.verbose = *verbosePtr
// show cli version
if *versionPtr {
fmt.Fprintln(os.Stderr, Version)
os.Exit(0)
}
// show help
if *helpPtr {
flag.Usage()
os.Exit(0)
}
// translate -path into -source if given
if *sourcePtr == "" && *pathPtr != "" {
*sourcePtr = fmt.Sprintf("file://%v", *pathPtr)
}
// initialize migrate
// don't catch migraterErr here and let each command decide
// how it wants to handle the error
migrater, migraterErr := migrate.New(*sourcePtr, *databasePtr)
defer func() {
if migraterErr == nil {
migrater.Close()
}
}()
if migraterErr == nil {
migrater.Log = log
migrater.PrefetchMigrations = *prefetchPtr
migrater.LockTimeout = time.Duration(int64(*lockTimeoutPtr)) * time.Second
// handle Ctrl+c
signals := make(chan os.Signal, 1)
signal.Notify(signals, syscall.SIGINT)
go func() {
for range signals {
log.Println("Stopping after this running migration ...")
migrater.GracefulStop <- true
return
}
}()
}
startTime := time.Now()
switch flag.Arg(0) {
case "create":
args := flag.Args()[1:]
seq := false
seqDigits := 6
createFlagSet := flag.NewFlagSet("create", flag.ExitOnError)
extPtr := createFlagSet.String("ext", "", "File extension")
dirPtr := createFlagSet.String("dir", "", "Directory to place file in (default: current working directory)")
formatPtr := createFlagSet.String("format", defaultTimeFormat, `The Go time format string to use. If the string "unix" or "unixNano" is specified, then the seconds or nanoseconds since January 1, 1970 UTC respectively will be used. Caution, due to the behavior of time.Time.Format(), invalid format strings will not error`)
createFlagSet.BoolVar(&seq, "seq", seq, "Use sequential numbers instead of timestamps (default: false)")
createFlagSet.IntVar(&seqDigits, "digits", seqDigits, "The number of digits to use in sequences (default: 6)")
createFlagSet.Parse(args)
if createFlagSet.NArg() == 0 {
log.fatal("error: please specify name")
}
name := createFlagSet.Arg(0)
if *extPtr == "" {
log.fatal("error: -ext flag must be specified")
}
*extPtr = "." + strings.TrimPrefix(*extPtr, ".")
if *dirPtr != "" {
*dirPtr = strings.Trim(*dirPtr, "/") + "/"
}
createCmd(*dirPtr, startTime, *formatPtr, name, *extPtr, seq, seqDigits)
case "goto":
if migraterErr != nil {
log.fatalErr(migraterErr)
}
if flag.Arg(1) == "" {
log.fatal("error: please specify version argument V")
}
v, err := strconv.ParseUint(flag.Arg(1), 10, 64)
if err != nil {
log.fatal("error: can't read version argument V")
}
gotoCmd(migrater, uint(v))
if log.verbose {
log.Println("Finished after", time.Now().Sub(startTime))
}
case "up":
if migraterErr != nil {
log.fatalErr(migraterErr)
}
limit := -1
if flag.Arg(1) != "" {
n, err := strconv.ParseUint(flag.Arg(1), 10, 64)
if err != nil {
log.fatal("error: can't read limit argument N")
}
limit = int(n)
}
upCmd(migrater, limit)
if log.verbose {
log.Println("Finished after", time.Now().Sub(startTime))
}
case "down":
if migraterErr != nil {
log.fatalErr(migraterErr)
}
limit := -1
if flag.Arg(1) != "" {
n, err := strconv.ParseUint(flag.Arg(1), 10, 64)
if err != nil {
log.fatal("error: can't read limit argument N")
}
limit = int(n)
}
downCmd(migrater, limit)
if log.verbose {
log.Println("Finished after", time.Now().Sub(startTime))
}
case "drop":
if migraterErr != nil {
log.fatalErr(migraterErr)
}
dropCmd(migrater)
if log.verbose {
log.Println("Finished after", time.Now().Sub(startTime))
}
case "force":
if migraterErr != nil {
log.fatalErr(migraterErr)
}
if flag.Arg(1) == "" {
log.fatal("error: please specify version argument V")
}
v, err := strconv.ParseInt(flag.Arg(1), 10, 64)
if err != nil {
log.fatal("error: can't read version argument V")
}
if v < -1 {
log.fatal("error: argument V must be >= -1")
}
forceCmd(migrater, int(v))
if log.verbose {
log.Println("Finished after", time.Now().Sub(startTime))
}
case "version":
if migraterErr != nil {
log.fatalErr(migraterErr)
}
versionCmd(migrater)
default:
flag.Usage()
os.Exit(0)
}
}
+4
View File
@@ -0,0 +1,4 @@
package main
// Version is set in Makefile with build flags
var Version = "dev"
+35
View File
@@ -0,0 +1,35 @@
# Cassandra
* Drop command will not work on Cassandra 2.X because it rely on
system_schema table which comes with 3.X
* Other commands should work properly but are **not tested**
* The Cassandra driver (gocql) does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
## Usage
`cassandra://host:port/keyspace?param1=value&param2=value2`
| URL Query | Default value | Description |
|------------|-------------|-----------|
| `x-migrations-table` | schema_migrations | Name of the migrations table |
| `x-multi-statement` | false | Enable multiple statements to be ran in a single migration (See note above) |
| `port` | 9042 | The port to bind to |
| `consistency` | ALL | Migration consistency
| `protocol` | | Cassandra protocol version (3 or 4)
| `timeout` | 1 minute | Migration timeout
| `username` | nil | Username to use when authenticating. |
| `password` | nil | Password to use when authenticating. |
`timeout` is parsed using [time.ParseDuration(s string)](https://golang.org/pkg/time/#ParseDuration)
## Upgrading from v1
1. Write down the current migration version from schema_migrations
2. `DROP TABLE schema_migrations`
4. Download and install the latest migrate version.
5. Force the current migration version with `migrate force <current_version>`.
+261
View File
@@ -0,0 +1,261 @@
package cassandra
import (
"errors"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"time"
"github.com/gocql/gocql"
"github.com/golang-migrate/migrate/v4/database"
)
func init() {
db := new(Cassandra)
database.Register("cassandra", db)
}
var DefaultMigrationsTable = "schema_migrations"
var (
ErrNilConfig = errors.New("no config")
ErrNoKeyspace = errors.New("no keyspace provided")
ErrDatabaseDirty = errors.New("database is dirty")
ErrClosedSession = errors.New("session is closed")
)
type Config struct {
MigrationsTable string
KeyspaceName string
MultiStatementEnabled bool
}
type Cassandra struct {
session *gocql.Session
isLocked bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(session *gocql.Session, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
} else if len(config.KeyspaceName) == 0 {
return nil, ErrNoKeyspace
}
if session.Closed() {
return nil, ErrClosedSession
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
c := &Cassandra{
session: session,
config: config,
}
if err := c.ensureVersionTable(); err != nil {
return nil, err
}
return c, nil
}
func (c *Cassandra) Open(url string) (database.Driver, error) {
u, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// Check for missing mandatory attributes
if len(u.Path) == 0 {
return nil, ErrNoKeyspace
}
cluster := gocql.NewCluster(u.Host)
cluster.Keyspace = strings.TrimPrefix(u.Path, "/")
cluster.Consistency = gocql.All
cluster.Timeout = 1 * time.Minute
if len(u.Query().Get("username")) > 0 && len(u.Query().Get("password")) > 0 {
authenticator := gocql.PasswordAuthenticator{
Username: u.Query().Get("username"),
Password: u.Query().Get("password"),
}
cluster.Authenticator = authenticator
}
// Retrieve query string configuration
if len(u.Query().Get("consistency")) > 0 {
var consistency gocql.Consistency
consistency, err = parseConsistency(u.Query().Get("consistency"))
if err != nil {
return nil, err
}
cluster.Consistency = consistency
}
if len(u.Query().Get("protocol")) > 0 {
var protoversion int
protoversion, err = strconv.Atoi(u.Query().Get("protocol"))
if err != nil {
return nil, err
}
cluster.ProtoVersion = protoversion
}
if len(u.Query().Get("timeout")) > 0 {
var timeout time.Duration
timeout, err = time.ParseDuration(u.Query().Get("timeout"))
if err != nil {
return nil, err
}
cluster.Timeout = timeout
}
session, err := cluster.CreateSession()
if err != nil {
return nil, err
}
return WithInstance(session, &Config{
KeyspaceName: strings.TrimPrefix(u.Path, "/"),
MigrationsTable: u.Query().Get("x-migrations-table"),
MultiStatementEnabled: u.Query().Get("x-multi-statement") == "true",
})
}
func (c *Cassandra) Close() error {
c.session.Close()
return nil
}
func (c *Cassandra) Lock() error {
if c.isLocked {
return database.ErrLocked
}
c.isLocked = true
return nil
}
func (c *Cassandra) Unlock() error {
c.isLocked = false
return nil
}
func (c *Cassandra) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if c.config.MultiStatementEnabled {
// split query by semi-colon
queries := strings.Split(query, ";")
for _, q := range queries {
tq := strings.TrimSpace(q)
if tq == "" {
continue
}
if err := c.session.Query(tq).Exec(); err != nil {
// TODO: cast to Cassandra error and get line number
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
}
return nil
}
if err := c.session.Query(query).Exec(); err != nil {
// TODO: cast to Cassandra error and get line number
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func (c *Cassandra) SetVersion(version int, dirty bool) error {
query := `TRUNCATE "` + c.config.MigrationsTable + `"`
if err := c.session.Query(query).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
query = `INSERT INTO "` + c.config.MigrationsTable + `" (version, dirty) VALUES (?, ?)`
if err := c.session.Query(query, version, dirty).Exec(); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
return nil
}
// Return current keyspace version
func (c *Cassandra) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM "` + c.config.MigrationsTable + `" LIMIT 1`
err = c.session.Query(query).Scan(&version, &dirty)
switch {
case err == gocql.ErrNotFound:
return database.NilVersion, false, nil
case err != nil:
if _, ok := err.(*gocql.Error); ok {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (c *Cassandra) Drop() error {
// select all tables in current schema
query := fmt.Sprintf(`SELECT table_name from system_schema.tables WHERE keyspace_name='%s'`, c.config.KeyspaceName)
iter := c.session.Query(query).Iter()
var tableName string
for iter.Scan(&tableName) {
err := c.session.Query(fmt.Sprintf(`DROP TABLE %s`, tableName)).Exec()
if err != nil {
return err
}
}
// Re-create the version table
return c.ensureVersionTable()
}
// Ensure version table exists
func (c *Cassandra) ensureVersionTable() error {
err := c.session.Query(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s (version bigint, dirty boolean, PRIMARY KEY(version))", c.config.MigrationsTable)).Exec()
if err != nil {
return err
}
if _, _, err = c.Version(); err != nil {
return err
}
return nil
}
// ParseConsistency wraps gocql.ParseConsistency
// to return an error instead of a panicking.
func parseConsistency(consistencyStr string) (consistency gocql.Consistency, err error) {
defer func() {
if r := recover(); r != nil {
var ok bool
err, ok = r.(error)
if !ok {
err = fmt.Errorf("Failed to parse consistency \"%s\": %v", consistencyStr, r)
}
}
}()
consistency = gocql.ParseConsistency(consistencyStr)
return consistency, nil
}
+56
View File
@@ -0,0 +1,56 @@
package cassandra
import (
"fmt"
"strconv"
"testing"
"github.com/gocql/gocql"
dt "github.com/golang-migrate/migrate/v4/database/testing"
mt "github.com/golang-migrate/migrate/v4/testing"
)
var versions = []mt.Version{
{Image: "cassandra:3.0.10"},
{Image: "cassandra:3.0"},
}
func isReady(i mt.Instance) bool {
// Cassandra exposes 5 ports (7000, 7001, 7199, 9042 & 9160)
// We only need the port bound to 9042, but we can only access to the first one
// through 'i.Port()' (which calls DockerContainer.firstPortMapping())
// So we need to get port mapping to retrieve correct port number bound to 9042
portMap := i.NetworkSettings().Ports
port, _ := strconv.Atoi(portMap["9042/tcp"][0].HostPort)
cluster := gocql.NewCluster(i.Host())
cluster.Port = port
cluster.Consistency = gocql.All
p, err := cluster.CreateSession()
if err != nil {
return false
}
defer p.Close()
// Create keyspace for tests
if err = p.Query("CREATE KEYSPACE testks WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor':1}").Exec(); err != nil {
return false
}
return true
}
func Test(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Cassandra{}
portMap := i.NetworkSettings().Ports
port, _ := strconv.Atoi(portMap["9042/tcp"][0].HostPort)
addr := fmt.Sprintf("cassandra://%v:%v/testks", i.Host(), port)
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
dt.Test(t, d, []byte("SELECT table_name from system_schema.tables"))
})
}
+19
View File
@@ -0,0 +1,19 @@
# ClickHouse
`clickhouse://host:port?username=user&password=qwerty&database=clicks&x-multi-statement=true`
| URL Query | Description |
|------------|-------------|
| `x-migrations-table`| Name of the migrations table |
| `database` | The name of the database to connect to |
| `username` | The user to sign in as |
| `password` | The user's password |
| `host` | The host to connect to. |
| `port` | The port to bind to. |
| `x-multi-statement` | false | Enable multiple statements to be ran in a single migration (See note below) |
## Notes
* The Clickhouse driver does not natively support executing multipe statements in a single query. To allow for multiple statements in a single migration, you can use the `x-multi-statement` param. There are two important caveats:
* This mode splits the migration text into separately-executed statements by a semi-colon `;`. Thus `x-multi-statement` cannot be used when a statement in the migration contains a string with a semi-colon.
* The queries are not executed in any sort of transaction/batch, meaning you are responsible for fixing partial migrations.
+215
View File
@@ -0,0 +1,215 @@
package clickhouse
import (
"database/sql"
"fmt"
"io"
"io/ioutil"
"net/url"
"strings"
"time"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
)
var DefaultMigrationsTable = "schema_migrations"
var ErrNilConfig = fmt.Errorf("no config")
type Config struct {
DatabaseName string
MigrationsTable string
MultiStatementEnabled bool
}
func init() {
database.Register("clickhouse", &ClickHouse{})
}
func WithInstance(conn *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := conn.Ping(); err != nil {
return nil, err
}
ch := &ClickHouse{
conn: conn,
config: config,
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
type ClickHouse struct {
conn *sql.DB
config *Config
}
func (ch *ClickHouse) Open(dsn string) (database.Driver, error) {
purl, err := url.Parse(dsn)
if err != nil {
return nil, err
}
q := migrate.FilterCustomQuery(purl)
q.Scheme = "tcp"
conn, err := sql.Open("clickhouse", q.String())
if err != nil {
return nil, err
}
ch = &ClickHouse{
conn: conn,
config: &Config{
MigrationsTable: purl.Query().Get("x-migrations-table"),
DatabaseName: purl.Query().Get("database"),
MultiStatementEnabled: purl.Query().Get("x-multi-statement") == "true",
},
}
if err := ch.init(); err != nil {
return nil, err
}
return ch, nil
}
func (ch *ClickHouse) init() error {
if len(ch.config.DatabaseName) == 0 {
if err := ch.conn.QueryRow("SELECT currentDatabase()").Scan(&ch.config.DatabaseName); err != nil {
return err
}
}
if len(ch.config.MigrationsTable) == 0 {
ch.config.MigrationsTable = DefaultMigrationsTable
}
return ch.ensureVersionTable()
}
func (ch *ClickHouse) Run(r io.Reader) error {
migration, err := ioutil.ReadAll(r)
if err != nil {
return err
}
if ch.config.MultiStatementEnabled {
// split query by semi-colon
queries := strings.Split(string(migration), ";")
for _, q := range queries {
tq := strings.TrimSpace(q)
if tq == "" {
continue
}
if _, err := ch.conn.Exec(string(q)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: []byte(q)}
}
}
return nil
}
if _, err := ch.conn.Exec(string(migration)); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migration}
}
return nil
}
func (ch *ClickHouse) Version() (int, bool, error) {
var (
version int
dirty uint8
query = "SELECT version, dirty FROM `" + ch.config.MigrationsTable + "` ORDER BY sequence DESC LIMIT 1"
)
if err := ch.conn.QueryRow(query).Scan(&version, &dirty); err != nil {
if err == sql.ErrNoRows {
return database.NilVersion, false, nil
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
}
return version, dirty == 1, nil
}
func (ch *ClickHouse) SetVersion(version int, dirty bool) error {
var (
bool = func(v bool) uint8 {
if v {
return 1
}
return 0
}
tx, err = ch.conn.Begin()
)
if err != nil {
return err
}
query := "INSERT INTO " + ch.config.MigrationsTable + " (version, dirty, sequence) VALUES (?, ?, ?)"
if _, err := tx.Exec(query, version, bool(dirty), time.Now().UnixNano()); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return tx.Commit()
}
func (ch *ClickHouse) ensureVersionTable() error {
var (
table string
query = "SHOW TABLES FROM " + ch.config.DatabaseName + " LIKE '" + ch.config.MigrationsTable + "'"
)
// check if migration table exists
if err := ch.conn.QueryRow(query).Scan(&table); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
query = `
CREATE TABLE ` + ch.config.MigrationsTable + ` (
version UInt32,
dirty UInt8,
sequence UInt64
) Engine=TinyLog
`
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (ch *ClickHouse) Drop() error {
var (
query = "SHOW TABLES FROM " + ch.config.DatabaseName
tables, err = ch.conn.Query(query)
)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer tables.Close()
for tables.Next() {
var table string
if err := tables.Scan(&table); err != nil {
return err
}
query = "DROP TABLE IF EXISTS " + ch.config.DatabaseName + "." + table
if _, err := ch.conn.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
return ch.ensureVersionTable()
}
func (ch *ClickHouse) Lock() error { return nil }
func (ch *ClickHouse) Unlock() error { return nil }
func (ch *ClickHouse) Close() error { return ch.conn.Close() }
@@ -0,0 +1 @@
DROP TABLE IF EXISTS test_1;
@@ -0,0 +1,3 @@
CREATE TABLE test_1 (
Date Date
) Engine=Memory;
@@ -0,0 +1 @@
DROP TABLE IF EXISTS test_2;
@@ -0,0 +1,3 @@
CREATE TABLE test_2 (
Date Date
) Engine=Memory;
+19
View File
@@ -0,0 +1,19 @@
# cockroachdb
`cockroachdb://user:password@host:port/dbname?query` (`cockroach://`, and `crdb-postgres://` work, too)
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `x-lock-table` | `LockTable` | Name of the table which maintains the migration lock |
| `x-force-lock` | `ForceLock` | Force lock acquisition to fix faulty migrations which may not have released the schema lock (Boolean, default is `false`) |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
+342
View File
@@ -0,0 +1,342 @@
package cockroachdb
import (
"context"
"database/sql"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"regexp"
"strconv"
)
import (
"github.com/cockroachdb/cockroach-go/crdb"
"github.com/lib/pq"
)
import (
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
)
func init() {
db := CockroachDb{}
database.Register("cockroach", &db)
database.Register("cockroachdb", &db)
database.Register("crdb-postgres", &db)
}
var DefaultMigrationsTable = "schema_migrations"
var DefaultLockTable = "schema_lock"
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
)
type Config struct {
MigrationsTable string
LockTable string
ForceLock bool
DatabaseName string
}
type CockroachDb struct {
db *sql.DB
isLocked bool
// Open and WithInstance need to guarantee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
query := `SELECT current_database()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
if len(config.LockTable) == 0 {
config.LockTable = DefaultLockTable
}
px := &CockroachDb{
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
if err := px.ensureLockTable(); err != nil {
return nil, err
}
return px, nil
}
func (c *CockroachDb) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
// As Cockroach uses the postgres protocol, and 'postgres' is already a registered database, we need to replace the
// connect prefix, with the actual protocol, so that the library can differentiate between the implementations
re := regexp.MustCompile("^(cockroach(db)?|crdb-postgres)")
connectString := re.ReplaceAllString(migrate.FilterCustomQuery(purl).String(), "postgres")
db, err := sql.Open("postgres", connectString)
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
if len(migrationsTable) == 0 {
migrationsTable = DefaultMigrationsTable
}
lockTable := purl.Query().Get("x-lock-table")
if len(lockTable) == 0 {
lockTable = DefaultLockTable
}
forceLockQuery := purl.Query().Get("x-force-lock")
forceLock, err := strconv.ParseBool(forceLockQuery)
if err != nil {
forceLock = false
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
LockTable: lockTable,
ForceLock: forceLock,
})
if err != nil {
return nil, err
}
return px, nil
}
func (c *CockroachDb) Close() error {
return c.db.Close()
}
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Lock() error {
err := crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) error {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
query := "SELECT * FROM " + c.config.LockTable + " WHERE lock_id = $1"
rows, err := tx.Query(query, aid)
if err != nil {
return database.Error{OrigErr: err, Err: "failed to fetch migration lock", Query: []byte(query)}
}
defer rows.Close()
// If row exists at all, lock is present
locked := rows.Next()
if locked && !c.config.ForceLock {
return database.ErrLocked
}
query = "INSERT INTO " + c.config.LockTable + " (lock_id) VALUES ($1)"
if _, err := tx.Exec(query, aid); err != nil {
return database.Error{OrigErr: err, Err: "failed to set migration lock", Query: []byte(query)}
}
return nil
})
if err != nil {
return err
} else {
c.isLocked = true
return nil
}
}
// Locking is done manually with a separate lock table. Implementing advisory locks in CRDB is being discussed
// See: https://github.com/cockroachdb/cockroach/issues/13546
func (c *CockroachDb) Unlock() error {
aid, err := database.GenerateAdvisoryLockId(c.config.DatabaseName)
if err != nil {
return err
}
// In the event of an implementation (non-migration) error, it is possible for the lock to not be released. Until
// a better locking mechanism is added, a manual purging of the lock table may be required in such circumstances
query := "DELETE FROM " + c.config.LockTable + " WHERE lock_id = $1"
if _, err := c.db.Exec(query, aid); err != nil {
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
// On drops, the lock table is fully removed; This is fine, and is a valid "unlocked" state for the schema
c.isLocked = false
return nil
}
}
return database.Error{OrigErr: err, Err: "failed to release migration lock", Query: []byte(query)}
}
c.isLocked = false
return nil
}
func (c *CockroachDb) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if _, err := c.db.Exec(query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func (c *CockroachDb) SetVersion(version int, dirty bool) error {
return crdb.ExecuteTx(context.Background(), c.db, nil, func(tx *sql.Tx) error {
if _, err := tx.Exec(`DELETE FROM "` + c.config.MigrationsTable + `"`); err != nil {
return err
}
if version >= 0 {
if _, err := tx.Exec(`INSERT INTO "`+c.config.MigrationsTable+`" (version, dirty) VALUES ($1, $2)`, version, dirty); err != nil {
return err
}
}
return nil
})
}
func (c *CockroachDb) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM "` + c.config.MigrationsTable + `" LIMIT 1`
err = c.db.QueryRow(query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pq.Error); ok {
// 42P01 is "UndefinedTableError" in CockroachDB
// https://github.com/cockroachdb/cockroach/blob/master/pkg/sql/pgwire/pgerror/codes.go
if e.Code == "42P01" {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (c *CockroachDb) Drop() error {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema())`
tables, err := c.db.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer tables.Close()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + t + ` CASCADE`
if _, err := c.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := c.ensureVersionTable(); err != nil {
return err
}
}
return nil
}
func (c *CockroachDb) ensureVersionTable() error {
// check if migration table exists
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := c.db.QueryRow(query, c.config.MigrationsTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
// if not, create the empty migration table
query = `CREATE TABLE "` + c.config.MigrationsTable + `" (version INT NOT NULL PRIMARY KEY, dirty BOOL NOT NULL)`
if _, err := c.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
func (c *CockroachDb) ensureLockTable() error {
// check if lock table exists
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := c.db.QueryRow(query, c.config.LockTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
// if not, create the empty lock table
query = `CREATE TABLE "` + c.config.LockTable + `" (lock_id INT NOT NULL PRIMARY KEY)`
if _, err := c.db.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
+91
View File
@@ -0,0 +1,91 @@
package cockroachdb
// error codes https://github.com/lib/pq/blob/master/error.go
import (
//"bytes"
"database/sql"
"fmt"
"io"
"testing"
"bytes"
dt "github.com/golang-migrate/migrate/v4/database/testing"
mt "github.com/golang-migrate/migrate/v4/testing"
"github.com/lib/pq"
)
var versions = []mt.Version{
{Image: "cockroachdb/cockroach:v1.0.2", Cmd: []string{"start", "--insecure"}},
}
func isReady(i mt.Instance) bool {
db, err := sql.Open("postgres", fmt.Sprintf("postgres://root@%v:%v?sslmode=disable", i.Host(), i.PortFor(26257)))
if err != nil {
return false
}
defer db.Close()
err = db.Ping()
if err == io.EOF {
_, err = db.Exec("CREATE DATABASE migrate")
return err == nil
} else if e, ok := err.(*pq.Error); ok {
if e.Code.Name() == "cannot_connect_now" {
return false
}
}
_, err = db.Exec("CREATE DATABASE migrate")
return err == nil
return true
}
func Test(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
c := &CockroachDb{}
addr := fmt.Sprintf("cockroach://root@%v:%v/migrate?sslmode=disable", i.Host(), i.PortFor(26257))
d, err := c.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMultiStatement(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
c := &CockroachDb{}
addr := fmt.Sprintf("cockroach://root@%v:%v/migrate?sslmode=disable", i.Host(), i.Port())
d, err := c.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
if err := d.Run(bytes.NewReader([]byte("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);"))); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*CockroachDb).db.QueryRow("SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestFilterCustomQuery(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
c := &CockroachDb{}
addr := fmt.Sprintf("cockroach://root@%v:%v/migrate?sslmode=disable&x-custom=foobar", i.Host(), i.PortFor(26257))
_, err := c.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
})
}
@@ -0,0 +1 @@
DROP TABLE IF EXISTS users;
@@ -0,0 +1,5 @@
CREATE TABLE users (
user_id INT UNIQUE,
name STRING(40),
email STRING(40)
);
@@ -0,0 +1 @@
ALTER TABLE users DROP COLUMN IF EXISTS city;
@@ -0,0 +1 @@
ALTER TABLE users ADD COLUMN city TEXT;
@@ -0,0 +1 @@
DROP INDEX IF EXISTS users_email_index;
@@ -0,0 +1,3 @@
CREATE UNIQUE INDEX IF NOT EXISTS users_email_index ON users (email);
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS books;
@@ -0,0 +1,5 @@
CREATE TABLE books (
user_id INT,
name STRING(40),
author STRING(40)
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS movies;
@@ -0,0 +1,5 @@
CREATE TABLE movies (
user_id INT,
name STRING(40),
director STRING(40)
);
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
View File
+124
View File
@@ -0,0 +1,124 @@
// Package database provides the Database interface.
// All database drivers must implement this interface, register themselves,
// optionally provide a `WithInstance` function and pass the tests
// in package database/testing.
package database
import (
"fmt"
"io"
nurl "net/url"
"sync"
)
var (
ErrLocked = fmt.Errorf("can't acquire lock")
)
const NilVersion int = -1
var driversMu sync.RWMutex
var drivers = make(map[string]Driver)
// Driver is the interface every database driver must implement.
//
// How to implement a database driver?
// 1. Implement this interface.
// 2. Optionally, add a function named `WithInstance`.
// This function should accept an existing DB instance and a Config{} struct
// and return a driver instance.
// 3. Add a test that calls database/testing.go:Test()
// 4. Add own tests for Open(), WithInstance() (when provided) and Close().
// All other functions are tested by tests in database/testing.
// Saves you some time and makes sure all database drivers behave the same way.
// 5. Call Register in init().
// 6. Create a migrate/cli/build_<driver-name>.go file
// 7. Add driver name in 'DATABASE' variable in Makefile
//
// Guidelines:
// * Don't try to correct user input. Don't assume things.
// When in doubt, return an error and explain the situation to the user.
// * All configuration input must come from the URL string in func Open()
// or the Config{} struct in WithInstance. Don't os.Getenv().
type Driver interface {
// Open returns a new driver instance configured with parameters
// coming from the URL string. Migrate will call this function
// only once per instance.
Open(url string) (Driver, error)
// Close closes the underlying database instance managed by the driver.
// Migrate will call this function only once per instance.
Close() error
// Lock should acquire a database lock so that only one migration process
// can run at a time. Migrate will call this function before Run is called.
// If the implementation can't provide this functionality, return nil.
// Return database.ErrLocked if database is already locked.
Lock() error
// Unlock should release the lock. Migrate will call this function after
// all migrations have been run.
Unlock() error
// Run applies a migration to the database. migration is garantueed to be not nil.
Run(migration io.Reader) error
// SetVersion saves version and dirty state.
// Migrate will call this function before and after each call to Run.
// version must be >= -1. -1 means NilVersion.
SetVersion(version int, dirty bool) error
// Version returns the currently active version and if the database is dirty.
// When no migration has been applied, it must return version -1.
// Dirty means, a previous migration failed and user interaction is required.
Version() (version int, dirty bool, err error)
// Drop deletes everything in the database.
Drop() error
}
// Open returns a new driver instance.
func Open(url string) (Driver, error) {
u, err := nurl.Parse(url)
if err != nil {
return nil, fmt.Errorf("Unable to parse URL. Did you escape all reserved URL characters? "+
"See: https://github.com/golang-migrate/migrate#database-urls Error: %v", err)
}
if u.Scheme == "" {
return nil, fmt.Errorf("database driver: invalid URL scheme")
}
driversMu.RLock()
d, ok := drivers[u.Scheme]
driversMu.RUnlock()
if !ok {
return nil, fmt.Errorf("database driver: unknown driver %v (forgotten import?)", u.Scheme)
}
return d.Open(url)
}
// Register globally registers a driver.
func Register(name string, driver Driver) {
driversMu.Lock()
defer driversMu.Unlock()
if driver == nil {
panic("Register driver is nil")
}
if _, dup := drivers[name]; dup {
panic("Register called twice for driver " + name)
}
drivers[name] = driver
}
// List lists the registered drivers
func List() []string {
driversMu.RLock()
defer driversMu.RUnlock()
names := make([]string, 0, len(drivers))
for n := range drivers {
names = append(names, n)
}
return names
}
+8
View File
@@ -0,0 +1,8 @@
package database
func ExampleDriver() {
// see database/stub for an example
// database/stub/stub.go has the driver implementation
// database/stub/stub_test.go runs database/testing/test.go:Test
}
+27
View File
@@ -0,0 +1,27 @@
package database
import (
"fmt"
)
// Error should be used for errors involving queries ran against the database
type Error struct {
// Optional: the line number
Line uint
// Query is a query excerpt
Query []byte
// Err is a useful/helping error message for humans
Err string
// OrigErr is the underlying error
OrigErr error
}
func (e Error) Error() string {
if len(e.Err) == 0 {
return fmt.Sprintf("%v in line %v: %s", e.OrigErr, e.Line, e.Query)
}
return fmt.Sprintf("%v in line %v: %s (details: %v)", e.Err, e.Line, e.Query, e.OrigErr)
}
View File
+54
View File
@@ -0,0 +1,54 @@
# MySQL
`mysql://user:password@tcp(host:port)/dbname?query`
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. |
| `port` | | The port to bind to. |
| `tls` | | TLS / SSL encrypted connection parameter; see [go-sql-driver](https://github.com/go-sql-driver/mysql#tls). Use any name (e.g. `migrate`) if you want to use a custom TLS config (`x-tls-` queries). |
| `x-tls-ca` | | The location of the CA (certificate authority) file. |
| `x-tls-cert` | | The location of the client certicicate file. Must be used with `x-tls-key`. |
| `x-tls-key` | | The location of the private key file. Must be used with `x-tls-cert`. |
| `x-tls-insecure-skip-verify` | | Whether or not to use SSL (true\|false) |
## Use with existing client
If you use the MySQL driver with existing database client, you must create the client with parameter `multiStatements=true`:
```go
package main
import (
"database/sql"
_ "github.com/go-sql-driver/mysql"
"github.com/golang-migrate/migrate"
"github.com/golang-migrate/migrate/database/mysql"
_ "github.com/golang-migrate/migrate/source/file"
)
func main() {
db, _ := sql.Open("mysql", "user:password@tcp(host:port)/dbname?multiStatements=true")
driver, _ := mysql.WithInstance(db, &mysql.Config{})
m, _ := migrate.NewWithDatabaseInstance(
"file:///migrations",
"mysql",
driver,
)
m.Steps(2)
}
```
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://dev.mysql.com/doc/refman/5.7/en/commit.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
+386
View File
@@ -0,0 +1,386 @@
// +build go1.9
package mysql
import (
"context"
"crypto/tls"
"crypto/x509"
"database/sql"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
)
import (
"github.com/go-sql-driver/mysql"
)
import (
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
)
func init() {
database.Register("mysql", &Mysql{})
}
var DefaultMigrationsTable = "schema_migrations"
var (
ErrDatabaseDirty = fmt.Errorf("database is dirty")
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrAppendPEM = fmt.Errorf("failed to append PEM")
ErrTLSCertKeyConfig = fmt.Errorf("To use TLS client authentication, both x-tls-cert and x-tls-key must not be empty")
)
type Config struct {
MigrationsTable string
DatabaseName string
}
type Mysql struct {
// mysql RELEASE_LOCK must be called from the same conn, so
// just do everything over a single conn anyway.
conn *sql.Conn
db *sql.DB
isLocked bool
config *Config
}
// instance must have `multiStatements` set to true
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
query := `SELECT DATABASE()`
var databaseName sql.NullString
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName.String) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName.String
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
mx := &Mysql{
conn: conn,
db: instance,
config: config,
}
if err := mx.ensureVersionTable(); err != nil {
return nil, err
}
return mx, nil
}
// urlToMySQLConfig takes a net/url URL and returns a go-sql-driver/mysql Config.
// Manually sets username and password to avoid net/url from url-encoding the reserved URL characters
func urlToMySQLConfig(u nurl.URL) (*mysql.Config, error) {
origUserInfo := u.User
u.User = nil
c, err := mysql.ParseDSN(strings.TrimPrefix(u.String(), "mysql://"))
if err != nil {
return nil, err
}
if origUserInfo != nil {
c.User = origUserInfo.Username()
if p, ok := origUserInfo.Password(); ok {
c.Passwd = p
}
}
return c, nil
}
func (m *Mysql) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
q := purl.Query()
q.Set("multiStatements", "true")
purl.RawQuery = q.Encode()
migrationsTable := purl.Query().Get("x-migrations-table")
if len(migrationsTable) == 0 {
migrationsTable = DefaultMigrationsTable
}
// use custom TLS?
ctls := purl.Query().Get("tls")
if len(ctls) > 0 {
if _, isBool := readBool(ctls); !isBool && strings.ToLower(ctls) != "skip-verify" {
rootCertPool := x509.NewCertPool()
pem, err := ioutil.ReadFile(purl.Query().Get("x-tls-ca"))
if err != nil {
return nil, err
}
if ok := rootCertPool.AppendCertsFromPEM(pem); !ok {
return nil, ErrAppendPEM
}
clientCert := make([]tls.Certificate, 0, 1)
if ccert, ckey := purl.Query().Get("x-tls-cert"), purl.Query().Get("x-tls-key"); ccert != "" || ckey != "" {
if ccert == "" || ckey == "" {
return nil, ErrTLSCertKeyConfig
}
certs, err := tls.LoadX509KeyPair(ccert, ckey)
if err != nil {
return nil, err
}
clientCert = append(clientCert, certs)
}
insecureSkipVerify := false
if len(purl.Query().Get("x-tls-insecure-skip-verify")) > 0 {
x, err := strconv.ParseBool(purl.Query().Get("x-tls-insecure-skip-verify"))
if err != nil {
return nil, err
}
insecureSkipVerify = x
}
mysql.RegisterTLSConfig(ctls, &tls.Config{
RootCAs: rootCertPool,
Certificates: clientCert,
InsecureSkipVerify: insecureSkipVerify,
})
}
}
c, err := urlToMySQLConfig(*migrate.FilterCustomQuery(purl))
if err != nil {
return nil, err
}
db, err := sql.Open("mysql", c.FormatDSN())
if err != nil {
return nil, err
}
mx, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
})
if err != nil {
return nil, err
}
return mx, nil
}
func (m *Mysql) Close() error {
connErr := m.conn.Close()
dbErr := m.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
func (m *Mysql) Lock() error {
if m.isLocked {
return database.ErrLocked
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := "SELECT GET_LOCK(?, 10)"
var success bool
if err := m.conn.QueryRowContext(context.Background(), query, aid).Scan(&success); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
if success {
m.isLocked = true
return nil
}
return database.ErrLocked
}
func (m *Mysql) Unlock() error {
if !m.isLocked {
return nil
}
aid, err := database.GenerateAdvisoryLockId(
fmt.Sprintf("%s:%s", m.config.DatabaseName, m.config.MigrationsTable))
if err != nil {
return err
}
query := `SELECT RELEASE_LOCK(?)`
if _, err := m.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
// NOTE: RELEASE_LOCK could return NULL or (or 0 if the code is changed),
// in which case isLocked should be true until the timeout expires -- synchronizing
// these states is likely not worth trying to do; reconsider the necessity of isLocked.
m.isLocked = false
return nil
}
func (m *Mysql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func (m *Mysql) SetVersion(version int, dirty bool) error {
tx, err := m.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "TRUNCATE `" + m.config.MigrationsTable + "`"
if _, err := tx.ExecContext(context.Background(), query); err != nil {
tx.Rollback()
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
query := "INSERT INTO `" + m.config.MigrationsTable + "` (version, dirty) VALUES (?, ?)"
if _, err := tx.ExecContext(context.Background(), query, version, dirty); err != nil {
tx.Rollback()
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (m *Mysql) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM `" + m.config.MigrationsTable + "` LIMIT 1"
err = m.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*mysql.MySQLError); ok {
if e.Number == 0 {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (m *Mysql) Drop() error {
// select all tables
query := `SHOW TABLES LIKE '%'`
tables, err := m.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer tables.Close()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = "DROP TABLE IF EXISTS `" + t + "` CASCADE"
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := m.ensureVersionTable(); err != nil {
return err
}
}
return nil
}
func (m *Mysql) ensureVersionTable() error {
// check if migration table exists
var result string
query := `SHOW TABLES LIKE "` + m.config.MigrationsTable + `"`
if err := m.conn.QueryRowContext(context.Background(), query).Scan(&result); err != nil {
if err != sql.ErrNoRows {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
} else {
return nil
}
// if not, create the empty migration table
query = "CREATE TABLE `" + m.config.MigrationsTable + "` (version bigint not null primary key, dirty boolean not null)"
if _, err := m.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
// Returns the bool value of the input.
// The 2nd return value indicates if the input was a valid bool value
// See https://github.com/go-sql-driver/mysql/blob/a059889267dc7170331388008528b3b44479bffb/utils.go#L71
func readBool(input string) (value bool, valid bool) {
switch input {
case "1", "true", "TRUE", "True":
return true, true
case "0", "false", "FALSE", "False":
return false, true
}
// Not a valid bool value
return
}
+155
View File
@@ -0,0 +1,155 @@
package mysql
import (
"database/sql"
sqldriver "database/sql/driver"
"fmt"
"net/url"
"testing"
)
import (
"github.com/go-sql-driver/mysql"
)
import (
dt "github.com/golang-migrate/migrate/v4/database/testing"
mt "github.com/golang-migrate/migrate/v4/testing"
)
var versions = []mt.Version{
{Image: "mysql:8", ENV: []string{"MYSQL_ROOT_PASSWORD=root", "MYSQL_DATABASE=public"}},
{Image: "mysql:5.7", ENV: []string{"MYSQL_ROOT_PASSWORD=root", "MYSQL_DATABASE=public"}},
{Image: "mysql:5.6", ENV: []string{"MYSQL_ROOT_PASSWORD=root", "MYSQL_DATABASE=public"}},
{Image: "mysql:5.5", ENV: []string{"MYSQL_ROOT_PASSWORD=root", "MYSQL_DATABASE=public"}},
}
func isReady(i mt.Instance) bool {
db, err := sql.Open("mysql", fmt.Sprintf("root:root@tcp(%v:%v)/public", i.Host(), i.Port()))
if err != nil {
return false
}
defer db.Close()
if err = db.Ping(); err != nil {
switch err {
case sqldriver.ErrBadConn, mysql.ErrInvalidConn:
return false
default:
fmt.Println(err)
}
return false
}
return true
}
func Test(t *testing.T) {
// mysql.SetLogger(mysql.Logger(log.New(ioutil.Discard, "", log.Ltime)))
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Mysql{}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
dt.Test(t, d, []byte("SELECT 1"))
// check ensureVersionTable
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
// check again
if err := d.(*Mysql).ensureVersionTable(); err != nil {
t.Fatal(err)
}
})
}
func TestLockWorks(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Mysql{}
addr := fmt.Sprintf("mysql://root:root@tcp(%v:%v)/public", i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
dt.Test(t, d, []byte("SELECT 1"))
ms := d.(*Mysql)
err = ms.Lock()
if err != nil {
t.Fatal(err)
}
err = ms.Unlock()
if err != nil {
t.Fatal(err)
}
// make sure the 2nd lock works (RELEASE_LOCK is very finicky)
err = ms.Lock()
if err != nil {
t.Fatal(err)
}
err = ms.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func TestURLToMySQLConfig(t *testing.T) {
testcases := []struct {
name string
urlStr string
expectedDSN string // empty string signifies that an error is expected
}{
{name: "no user/password", urlStr: "mysql://tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "only user", urlStr: "mysql://username@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "only user - with encoded :",
urlStr: "mysql://username%3A@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "only user - with encoded @",
urlStr: "mysql://username%40@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password", urlStr: "mysql://username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
// Not supported yet: https://github.com/go-sql-driver/mysql/issues/591
// {name: "user/password - user with encoded :",
// urlStr: "mysql://username%3A:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
// expectedDSN: "username::pasword@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - user with encoded @",
urlStr: "mysql://username%40:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username@:password@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - password with encoded :",
urlStr: "mysql://username:password%3A@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:password:@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
{name: "user/password - password with encoded @",
urlStr: "mysql://username:password%40@tcp(127.0.0.1:3306)/myDB?multiStatements=true",
expectedDSN: "username:password@@tcp(127.0.0.1:3306)/myDB?multiStatements=true"},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
u, err := url.Parse(tc.urlStr)
if err != nil {
t.Fatal("Failed to parse url string:", tc.urlStr, "error:", err)
}
if config, err := urlToMySQLConfig(*u); err == nil {
dsn := config.FormatDSN()
if dsn != tc.expectedDSN {
t.Error("Got unexpected DSN:", dsn, "!=", tc.expectedDSN)
}
} else {
if tc.expectedDSN != "" {
t.Error("Got unexpected error:", err, "urlStr:", tc.urlStr)
}
}
})
}
}
View File
+244
View File
@@ -0,0 +1,244 @@
package database_test
import (
"encoding/hex"
"net/url"
"strings"
"testing"
)
const reservedChars = "!#$%&'()*+,/:;=?@[]"
// TestUserUnencodedReservedURLChars documents the behavior of using unencoded reserved characters in usernames with
// net/url Parse()
func TestUserUnencodedReservedURLChars(t *testing.T) {
scheme := "database://"
baseUsername := "username"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
testcases := []struct {
char string
parses bool
expectedUsername string // empty string means that the username failed to parse
encodedURL string
}{
{char: "!", parses: true, expectedUsername: baseUsername + "!",
encodedURL: scheme + baseUsername + "%21" + urlSuffixAndSep},
{char: "#", parses: true, expectedUsername: "",
encodedURL: scheme + baseUsername + "#" + urlSuffixAndSep},
{char: "$", parses: true, expectedUsername: baseUsername + "$",
encodedURL: scheme + baseUsername + "$" + urlSuffixAndSep},
{char: "%", parses: false},
{char: "&", parses: true, expectedUsername: baseUsername + "&",
encodedURL: scheme + baseUsername + "&" + urlSuffixAndSep},
{char: "'", parses: true, expectedUsername: "username'",
encodedURL: scheme + baseUsername + "%27" + urlSuffixAndSep},
{char: "(", parses: true, expectedUsername: "username(",
encodedURL: scheme + baseUsername + "%28" + urlSuffixAndSep},
{char: ")", parses: true, expectedUsername: "username)",
encodedURL: scheme + baseUsername + "%29" + urlSuffixAndSep},
{char: "*", parses: true, expectedUsername: "username*",
encodedURL: scheme + baseUsername + "%2A" + urlSuffixAndSep},
{char: "+", parses: true, expectedUsername: "username+",
encodedURL: scheme + baseUsername + "+" + urlSuffixAndSep},
{char: ",", parses: true, expectedUsername: "username,",
encodedURL: scheme + baseUsername + "," + urlSuffixAndSep},
{char: "/", parses: true, expectedUsername: "",
encodedURL: scheme + baseUsername + "/" + urlSuffixAndSep},
{char: ":", parses: true, expectedUsername: "username",
encodedURL: scheme + baseUsername + ":%3A" + urlSuffix},
{char: ";", parses: true, expectedUsername: "username;",
encodedURL: scheme + baseUsername + ";" + urlSuffixAndSep},
{char: "=", parses: true, expectedUsername: "username=",
encodedURL: scheme + baseUsername + "=" + urlSuffixAndSep},
{char: "?", parses: true, expectedUsername: "",
encodedURL: scheme + baseUsername + "?" + urlSuffixAndSep},
{char: "@", parses: true, expectedUsername: "username@",
encodedURL: scheme + baseUsername + "%40" + urlSuffixAndSep},
{char: "[", parses: false},
{char: "]", parses: false},
}
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run("reserved char "+tc.char, func(t *testing.T) {
s := scheme + baseUsername + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
if !tc.parses {
t.Error("Unexpectedly parsed reserved character. url:", s)
return
}
var username string
if u.User != nil {
username = u.User.Username()
}
if username != tc.expectedUsername {
t.Error("Got unexpected username:", username, "!=", tc.expectedUsername)
}
if s := u.String(); s != tc.encodedURL {
t.Error("Got unexpected encoded URL:", s, "!=", tc.encodedURL)
}
} else {
if tc.parses {
t.Error("Failed to parse reserved character. url:", s)
}
}
})
}
t.Run("All reserved chars tested", func(t *testing.T) {
if s := strings.Join(testedChars, ""); s != reservedChars {
t.Error("Not all reserved URL characters were tested:", s, "!=", reservedChars)
}
})
}
func TestUserEncodedReservedURLChars(t *testing.T) {
scheme := "database://"
baseUsername := "username"
urlSuffix := "password@localhost:12345/myDB?someParam=true"
urlSuffixAndSep := ":" + urlSuffix
for _, c := range reservedChars {
c := string(c)
t.Run("reserved char "+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := scheme + baseUsername + encodedChar + urlSuffixAndSep
expectedUsername := baseUsername + c
u, err := url.Parse(s)
if err != nil {
t.Fatal("Failed to parse url with encoded reserved character. url:", s)
}
if u.User == nil {
t.Fatal("Failed to parse userinfo with encoded reserve character. url:", s)
}
if username := u.User.Username(); username != expectedUsername {
t.Fatal("Got unexpected username:", username, "!=", expectedUsername)
}
})
}
}
// TestPasswordUnencodedReservedURLChars documents the behavior of using unencoded reserved characters in passwords
// with net/url Parse()
func TestPasswordUnencodedReservedURLChars(t *testing.T) {
username := "username"
schemeAndUsernameAndSep := "database://" + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
testcases := []struct {
char string
parses bool
expectedUsername string // empty string means that the username failed to parse
expectedPassword string // empty string means that the password failed to parse
encodedURL string
}{
{char: "!", parses: true, expectedUsername: username, expectedPassword: basePassword + "!",
encodedURL: schemeAndUsernameAndSep + basePassword + "%21" + urlSuffixAndSep},
{char: "#", parses: true, expectedUsername: "", expectedPassword: "",
encodedURL: schemeAndUsernameAndSep + basePassword + "#" + urlSuffixAndSep},
{char: "$", parses: true, expectedUsername: username, expectedPassword: basePassword + "$",
encodedURL: schemeAndUsernameAndSep + basePassword + "$" + urlSuffixAndSep},
{char: "%", parses: false},
{char: "&", parses: true, expectedUsername: username, expectedPassword: basePassword + "&",
encodedURL: schemeAndUsernameAndSep + basePassword + "&" + urlSuffixAndSep},
{char: "'", parses: true, expectedUsername: username, expectedPassword: "password'",
encodedURL: schemeAndUsernameAndSep + basePassword + "%27" + urlSuffixAndSep},
{char: "(", parses: true, expectedUsername: username, expectedPassword: "password(",
encodedURL: schemeAndUsernameAndSep + basePassword + "%28" + urlSuffixAndSep},
{char: ")", parses: true, expectedUsername: username, expectedPassword: "password)",
encodedURL: schemeAndUsernameAndSep + basePassword + "%29" + urlSuffixAndSep},
{char: "*", parses: true, expectedUsername: username, expectedPassword: "password*",
encodedURL: schemeAndUsernameAndSep + basePassword + "%2A" + urlSuffixAndSep},
{char: "+", parses: true, expectedUsername: username, expectedPassword: "password+",
encodedURL: schemeAndUsernameAndSep + basePassword + "+" + urlSuffixAndSep},
{char: ",", parses: true, expectedUsername: username, expectedPassword: "password,",
encodedURL: schemeAndUsernameAndSep + basePassword + "," + urlSuffixAndSep},
{char: "/", parses: true, expectedUsername: "", expectedPassword: "",
encodedURL: schemeAndUsernameAndSep + basePassword + "/" + urlSuffixAndSep},
{char: ":", parses: true, expectedUsername: username, expectedPassword: "password:",
encodedURL: schemeAndUsernameAndSep + basePassword + "%3A" + urlSuffixAndSep},
{char: ";", parses: true, expectedUsername: username, expectedPassword: "password;",
encodedURL: schemeAndUsernameAndSep + basePassword + ";" + urlSuffixAndSep},
{char: "=", parses: true, expectedUsername: username, expectedPassword: "password=",
encodedURL: schemeAndUsernameAndSep + basePassword + "=" + urlSuffixAndSep},
{char: "?", parses: true, expectedUsername: "", expectedPassword: "",
encodedURL: schemeAndUsernameAndSep + basePassword + "?" + urlSuffixAndSep},
{char: "@", parses: true, expectedUsername: username, expectedPassword: "password@",
encodedURL: schemeAndUsernameAndSep + basePassword + "%40" + urlSuffixAndSep},
{char: "[", parses: false},
{char: "]", parses: false},
}
testedChars := make([]string, 0, len(reservedChars))
for _, tc := range testcases {
testedChars = append(testedChars, tc.char)
t.Run("reserved char "+tc.char, func(t *testing.T) {
s := schemeAndUsernameAndSep + basePassword + tc.char + urlSuffixAndSep
u, err := url.Parse(s)
if err == nil {
if !tc.parses {
t.Error("Unexpectedly parsed reserved character. url:", s)
return
}
var username, password string
if u.User != nil {
username = u.User.Username()
password, _ = u.User.Password()
}
if username != tc.expectedUsername {
t.Error("Got unexpected username:", username, "!=", tc.expectedUsername)
}
if password != tc.expectedPassword {
t.Error("Got unexpected password:", password, "!=", tc.expectedPassword)
}
if s := u.String(); s != tc.encodedURL {
t.Error("Got unexpected encoded URL:", s, "!=", tc.encodedURL)
}
} else {
if tc.parses {
t.Error("Failed to parse reserved character. url:", s)
}
}
})
}
t.Run("All reserved chars tested", func(t *testing.T) {
if s := strings.Join(testedChars, ""); s != reservedChars {
t.Error("Not all reserved URL characters were tested:", s, "!=", reservedChars)
}
})
}
func TestPasswordEncodedReservedURLChars(t *testing.T) {
username := "username"
schemeAndUsernameAndSep := "database://" + username + ":"
basePassword := "password"
urlSuffixAndSep := "@localhost:12345/myDB?someParam=true"
for _, c := range reservedChars {
c := string(c)
t.Run("reserved char "+c, func(t *testing.T) {
encodedChar := "%" + hex.EncodeToString([]byte(c))
s := schemeAndUsernameAndSep + basePassword + encodedChar + urlSuffixAndSep
expectedPassword := basePassword + c
u, err := url.Parse(s)
if err != nil {
t.Fatal("Failed to parse url with encoded reserved character. url:", s)
}
if u.User == nil {
t.Fatal("Failed to parse userinfo with encoded reserve character. url:", s)
}
if n := u.User.Username(); n != username {
t.Fatal("Got unexpected username:", n, "!=", username)
}
if p, _ := u.User.Password(); p != expectedPassword {
t.Fatal("Got unexpected password:", p, "!=", expectedPassword)
}
})
}
}
+28
View File
@@ -0,0 +1,28 @@
# postgres
`postgres://user:password@host:port/dbname?query` (`postgresql://` works, too)
| URL Query | WithInstance Config | Description |
|------------|---------------------|-------------|
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `dbname` | `DatabaseName` | The name of the database to connect to |
| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
| `user` | | The user to sign in as |
| `password` | | The user's password |
| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
| `port` | | The port to bind to. (default is 5432) |
| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
| `sslkey` | | Key file location. The file must contain PEM encoded data. |
| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. |
| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
## Upgrading from v1
1. Write down the current migration version from schema_migrations
1. `DROP TABLE schema_migrations`
2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
3. Download and install the latest migrate version.
4. Force the current migration version with `migrate force <current_version>`.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS users;
@@ -0,0 +1,5 @@
CREATE TABLE users (
user_id integer unique,
name varchar(40),
email varchar(40)
);
@@ -0,0 +1 @@
ALTER TABLE users DROP COLUMN IF EXISTS city;
@@ -0,0 +1,3 @@
ALTER TABLE users ADD COLUMN city varchar(100);
@@ -0,0 +1 @@
DROP INDEX IF EXISTS users_email_index;
@@ -0,0 +1,3 @@
CREATE UNIQUE INDEX CONCURRENTLY users_email_index ON users (email);
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
DROP TABLE IF EXISTS books;
@@ -0,0 +1,5 @@
CREATE TABLE books (
user_id integer,
name varchar(40),
author varchar(40)
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS movies;
@@ -0,0 +1,5 @@
CREATE TABLE movies (
user_id integer,
name varchar(40),
director varchar(40)
);
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
@@ -0,0 +1 @@
-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.
+339
View File
@@ -0,0 +1,339 @@
// +build go1.9
package postgres
import (
"context"
"database/sql"
"fmt"
"io"
"io/ioutil"
nurl "net/url"
"strconv"
"strings"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
"github.com/lib/pq"
)
func init() {
db := Postgres{}
database.Register("postgres", &db)
database.Register("postgresql", &db)
}
var DefaultMigrationsTable = "schema_migrations"
var (
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrNoSchema = fmt.Errorf("no schema")
ErrDatabaseDirty = fmt.Errorf("database is dirty")
)
type Config struct {
MigrationsTable string
DatabaseName string
}
type Postgres struct {
// Locking and unlocking need to use the same connection
conn *sql.Conn
db *sql.DB
isLocked bool
// Open and WithInstance need to garantuee that config is never nil
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
query := `SELECT CURRENT_DATABASE()`
var databaseName string
if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
return nil, &database.Error{OrigErr: err, Query: []byte(query)}
}
if len(databaseName) == 0 {
return nil, ErrNoDatabaseName
}
config.DatabaseName = databaseName
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
conn, err := instance.Conn(context.Background())
if err != nil {
return nil, err
}
px := &Postgres{
conn: conn,
db: instance,
config: config,
}
if err := px.ensureVersionTable(); err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
db, err := sql.Open("postgres", migrate.FilterCustomQuery(purl).String())
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
if len(migrationsTable) == 0 {
migrationsTable = DefaultMigrationsTable
}
px, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
})
if err != nil {
return nil, err
}
return px, nil
}
func (p *Postgres) Close() error {
connErr := p.conn.Close()
dbErr := p.db.Close()
if connErr != nil || dbErr != nil {
return fmt.Errorf("conn: %v, db: %v", connErr, dbErr)
}
return nil
}
// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
func (p *Postgres) Lock() error {
if p.isLocked {
return database.ErrLocked
}
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
// This will either obtain the lock immediately and return true,
// or return false if the lock cannot be acquired immediately.
query := `SELECT pg_advisory_lock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
}
p.isLocked = true
return nil
}
func (p *Postgres) Unlock() error {
if !p.isLocked {
return nil
}
aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
if err != nil {
return err
}
query := `SELECT pg_advisory_unlock($1)`
if _, err := p.conn.ExecContext(context.Background(), query, aid); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
p.isLocked = false
return nil
}
func (p *Postgres) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
// run migration
query := string(migr[:])
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
if pgErr, ok := err.(*pq.Error); ok {
var line uint
var col uint
var lineColOK bool
if pgErr.Position != "" {
if pos, err := strconv.ParseUint(pgErr.Position, 10, 64); err == nil {
line, col, lineColOK = computeLineFromPos(query, int(pos))
}
}
message := fmt.Sprintf("migration failed: %s", pgErr.Message)
if lineColOK {
message = fmt.Sprintf("%s (column %d)", message, col)
}
if pgErr.Detail != "" {
message = fmt.Sprintf("%s, %s", message, pgErr.Detail)
}
return database.Error{OrigErr: err, Err: message, Query: migr, Line: line}
}
return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
}
return nil
}
func computeLineFromPos(s string, pos int) (line uint, col uint, ok bool) {
// replace crlf with lf
s = strings.Replace(s, "\r\n", "\n", -1)
// pg docs: pos uses index 1 for the first character, and positions are measured in characters not bytes
runes := []rune(s)
if pos > len(runes) {
return 0, 0, false
}
sel := runes[:pos]
line = uint(runesCount(sel, newLine) + 1)
col = uint(pos - 1 - runesLastIndex(sel, newLine))
return line, col, true
}
const newLine = '\n'
func runesCount(input []rune, target rune) int {
var count int
for _, r := range input {
if r == target {
count++
}
}
return count
}
func runesLastIndex(input []rune, target rune) int {
for i := len(input) - 1; i >= 0; i-- {
if input[i] == target {
return i
}
}
return -1
}
func (p *Postgres) SetVersion(version int, dirty bool) error {
tx, err := p.conn.BeginTx(context.Background(), &sql.TxOptions{})
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := `TRUNCATE "` + p.config.MigrationsTable + `"`
if _, err := tx.Exec(query); err != nil {
tx.Rollback()
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
query = `INSERT INTO "` + p.config.MigrationsTable + `" (version, dirty) VALUES ($1, $2)`
if _, err := tx.Exec(query, version, dirty); err != nil {
tx.Rollback()
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (p *Postgres) Version() (version int, dirty bool, err error) {
query := `SELECT version, dirty FROM "` + p.config.MigrationsTable + `" LIMIT 1`
err = p.conn.QueryRowContext(context.Background(), query).Scan(&version, &dirty)
switch {
case err == sql.ErrNoRows:
return database.NilVersion, false, nil
case err != nil:
if e, ok := err.(*pq.Error); ok {
if e.Code.Name() == "undefined_table" {
return database.NilVersion, false, nil
}
}
return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
default:
return version, dirty, nil
}
}
func (p *Postgres) Drop() error {
// select all tables in current schema
query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema()) AND table_type='BASE TABLE'`
tables, err := p.conn.QueryContext(context.Background(), query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer tables.Close()
// delete one table after another
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
tableNames = append(tableNames, tableName)
}
}
if len(tableNames) > 0 {
// delete one by one ...
for _, t := range tableNames {
query = `DROP TABLE IF EXISTS ` + t + ` CASCADE`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := p.ensureVersionTable(); err != nil {
return err
}
}
return nil
}
func (p *Postgres) ensureVersionTable() error {
// check if migration table exists
var count int
query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
if err := p.conn.QueryRowContext(context.Background(), query, p.config.MigrationsTable).Scan(&count); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if count == 1 {
return nil
}
// if not, create the empty migration table
query = `CREATE TABLE "` + p.config.MigrationsTable + `" (version bigint not null primary key, dirty boolean not null)`
if _, err := p.conn.ExecContext(context.Background(), query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
return nil
}
+300
View File
@@ -0,0 +1,300 @@
package postgres
// error codes https://github.com/lib/pq/blob/master/error.go
import (
"bytes"
"context"
"database/sql"
sqldriver "database/sql/driver"
"fmt"
"io"
"strconv"
"strings"
"testing"
dt "github.com/golang-migrate/migrate/v4/database/testing"
mt "github.com/golang-migrate/migrate/v4/testing"
)
var versions = []mt.Version{
{Image: "postgres:10"},
{Image: "postgres:9.6"},
{Image: "postgres:9.5"},
{Image: "postgres:9.4"},
{Image: "postgres:9.3"},
}
func pgConnectionString(host string, port uint) string {
return fmt.Sprintf("postgres://postgres@%s:%v/postgres?sslmode=disable", host, port)
}
func isReady(i mt.Instance) bool {
db, err := sql.Open("postgres", pgConnectionString(i.Host(), i.Port()))
if err != nil {
return false
}
defer db.Close()
if err = db.Ping(); err != nil {
switch err {
case sqldriver.ErrBadConn, io.EOF:
return false
default:
fmt.Println(err)
}
return false
}
return true
}
func Test(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Postgres{}
addr := pgConnectionString(i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
dt.Test(t, d, []byte("SELECT 1"))
})
}
func TestMultiStatement(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Postgres{}
addr := pgConnectionString(i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
if err := d.Run(bytes.NewReader([]byte("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);"))); err != nil {
t.Fatalf("expected err to be nil, got %v", err)
}
// make sure second table exists
var exists bool
if err := d.(*Postgres).conn.QueryRowContext(context.Background(), "SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
t.Fatal(err)
}
if !exists {
t.Fatalf("expected table bar to exist")
}
})
}
func TestErrorParsing(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Postgres{}
addr := pgConnectionString(i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
wantErr := `migration failed: syntax error at or near "TABLEE" (column 37) in line 1: CREATE TABLE foo ` +
`(foo text); CREATE TABLEE bar (bar text); (details: pq: syntax error at or near "TABLEE")`
if err := d.Run(bytes.NewReader([]byte("CREATE TABLE foo (foo text); CREATE TABLEE bar (bar text);"))); err == nil {
t.Fatal("expected err but got nil")
} else if err.Error() != wantErr {
t.Fatalf("expected '%s' but got '%s'", wantErr, err.Error())
}
})
}
func TestFilterCustomQuery(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Postgres{}
addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
})
}
func TestWithSchema(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Postgres{}
addr := pgConnectionString(i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
defer d.Close()
// create foobar schema
if err := d.Run(bytes.NewReader([]byte("CREATE SCHEMA foobar AUTHORIZATION postgres"))); err != nil {
t.Fatal(err)
}
if err := d.SetVersion(1, false); err != nil {
t.Fatal(err)
}
// re-connect using that schema
d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", i.Host(), i.Port()))
if err != nil {
t.Fatalf("%v", err)
}
defer d2.Close()
version, _, err := d2.Version()
if err != nil {
t.Fatal(err)
}
if version != -1 {
t.Fatal("expected NilVersion")
}
// now update version and compare
if err := d2.SetVersion(2, false); err != nil {
t.Fatal(err)
}
version, _, err = d2.Version()
if err != nil {
t.Fatal(err)
}
if version != 2 {
t.Fatal("expected version 2")
}
// meanwhile, the public schema still has the other version
version, _, err = d.Version()
if err != nil {
t.Fatal(err)
}
if version != 1 {
t.Fatal("expected version 2")
}
})
}
func TestWithInstance(t *testing.T) {
}
func TestPostgres_Lock(t *testing.T) {
mt.ParallelTest(t, versions, isReady,
func(t *testing.T, i mt.Instance) {
p := &Postgres{}
addr := pgConnectionString(i.Host(), i.Port())
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
dt.Test(t, d, []byte("SELECT 1"))
ps := d.(*Postgres)
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
err = ps.Lock()
if err != nil {
t.Fatal(err)
}
err = ps.Unlock()
if err != nil {
t.Fatal(err)
}
})
}
func Test_computeLineFromPos(t *testing.T) {
testcases := []struct {
pos int
wantLine uint
wantCol uint
input string
wantOk bool
}{
{
15, 2, 6, "SELECT *\nFROM foo", true, // foo table does not exists
},
{
16, 3, 6, "SELECT *\n\nFROM foo", true, // foo table does not exists, empty line
},
{
25, 3, 7, "SELECT *\nFROM foo\nWHERE x", true, // x column error
},
{
27, 5, 7, "SELECT *\n\nFROM foo\n\nWHERE x", true, // x column error, empty lines
},
{
10, 2, 1, "SELECT *\nFROMM foo", true, // FROMM typo
},
{
11, 3, 1, "SELECT *\n\nFROMM foo", true, // FROMM typo, empty line
},
{
17, 2, 8, "SELECT *\nFROM foo", true, // last character
},
{
18, 0, 0, "SELECT *\nFROM foo", false, // invalid position
},
}
for i, tc := range testcases {
t.Run("tc"+strconv.Itoa(i), func(t *testing.T) {
run := func(crlf bool, nonASCII bool) {
var name string
if crlf {
name = "crlf"
} else {
name = "lf"
}
if nonASCII {
name += "-nonascii"
} else {
name += "-ascii"
}
t.Run(name, func(t *testing.T) {
input := tc.input
if crlf {
input = strings.Replace(input, "\n", "\r\n", -1)
}
if nonASCII {
input = strings.Replace(input, "FROM", "FRÖM", -1)
}
gotLine, gotCol, gotOK := computeLineFromPos(input, tc.pos)
if tc.wantOk {
t.Logf("pos %d, want %d:%d, %#v", tc.pos, tc.wantLine, tc.wantCol, input)
}
if gotOK != tc.wantOk {
t.Fatalf("expected ok %v but got %v", tc.wantOk, gotOK)
}
if gotLine != tc.wantLine {
t.Fatalf("expected line %d but got %d", tc.wantLine, gotLine)
}
if gotCol != tc.wantCol {
t.Fatalf("expected col %d but got %d", tc.wantCol, gotCol)
}
})
}
run(false, false)
run(true, false)
run(false, true)
run(true, true)
})
}
}
View File
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1,3 @@
CREATE TABLE pets (
name string
);
@@ -0,0 +1 @@
DROP TABLE IF EXISTS pets;
@@ -0,0 +1 @@
ALTER TABLE pets ADD predator bool;;
+212
View File
@@ -0,0 +1,212 @@
package ql
import (
"database/sql"
"fmt"
"io"
"io/ioutil"
"strings"
nurl "net/url"
_ "github.com/cznic/ql/driver"
"github.com/golang-migrate/migrate/v4"
"github.com/golang-migrate/migrate/v4/database"
)
func init() {
database.Register("ql", &Ql{})
}
var DefaultMigrationsTable = "schema_migrations"
var (
ErrDatabaseDirty = fmt.Errorf("database is dirty")
ErrNilConfig = fmt.Errorf("no config")
ErrNoDatabaseName = fmt.Errorf("no database name")
ErrAppendPEM = fmt.Errorf("failed to append PEM")
)
type Config struct {
MigrationsTable string
DatabaseName string
}
type Ql struct {
db *sql.DB
isLocked bool
config *Config
}
func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
if config == nil {
return nil, ErrNilConfig
}
if err := instance.Ping(); err != nil {
return nil, err
}
if len(config.MigrationsTable) == 0 {
config.MigrationsTable = DefaultMigrationsTable
}
mx := &Ql{
db: instance,
config: config,
}
if err := mx.ensureVersionTable(); err != nil {
return nil, err
}
return mx, nil
}
func (m *Ql) ensureVersionTable() error {
tx, err := m.db.Begin()
if err != nil {
return err
}
if _, err := tx.Exec(fmt.Sprintf(`
CREATE TABLE IF NOT EXISTS %s (version uint64,dirty bool);
CREATE UNIQUE INDEX IF NOT EXISTS version_unique ON %s (version);
`, m.config.MigrationsTable, m.config.MigrationsTable)); err != nil {
if err := tx.Rollback(); err != nil {
return err
}
return err
}
if err := tx.Commit(); err != nil {
return err
}
return nil
}
func (m *Ql) Open(url string) (database.Driver, error) {
purl, err := nurl.Parse(url)
if err != nil {
return nil, err
}
dbfile := strings.Replace(migrate.FilterCustomQuery(purl).String(), "ql://", "", 1)
db, err := sql.Open("ql", dbfile)
if err != nil {
return nil, err
}
migrationsTable := purl.Query().Get("x-migrations-table")
if len(migrationsTable) == 0 {
migrationsTable = DefaultMigrationsTable
}
mx, err := WithInstance(db, &Config{
DatabaseName: purl.Path,
MigrationsTable: migrationsTable,
})
if err != nil {
return nil, err
}
return mx, nil
}
func (m *Ql) Close() error {
return m.db.Close()
}
func (m *Ql) Drop() error {
query := `SELECT Name FROM __Table`
tables, err := m.db.Query(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
defer tables.Close()
tableNames := make([]string, 0)
for tables.Next() {
var tableName string
if err := tables.Scan(&tableName); err != nil {
return err
}
if len(tableName) > 0 {
if strings.HasPrefix(tableName, "__") == false {
tableNames = append(tableNames, tableName)
}
}
}
if len(tableNames) > 0 {
for _, t := range tableNames {
query := "DROP TABLE " + t
err = m.executeQuery(query)
if err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := m.ensureVersionTable(); err != nil {
return err
}
}
return nil
}
func (m *Ql) Lock() error {
if m.isLocked {
return database.ErrLocked
}
m.isLocked = true
return nil
}
func (m *Ql) Unlock() error {
if !m.isLocked {
return nil
}
m.isLocked = false
return nil
}
func (m *Ql) Run(migration io.Reader) error {
migr, err := ioutil.ReadAll(migration)
if err != nil {
return err
}
query := string(migr[:])
return m.executeQuery(query)
}
func (m *Ql) executeQuery(query string) error {
tx, err := m.db.Begin()
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
if _, err := tx.Exec(query); err != nil {
tx.Rollback()
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (m *Ql) SetVersion(version int, dirty bool) error {
tx, err := m.db.Begin()
if err != nil {
return &database.Error{OrigErr: err, Err: "transaction start failed"}
}
query := "TRUNCATE TABLE " + m.config.MigrationsTable
if _, err := tx.Exec(query); err != nil {
return &database.Error{OrigErr: err, Query: []byte(query)}
}
if version >= 0 {
query := fmt.Sprintf(`INSERT INTO %s (version, dirty) VALUES (%d, %t)`, m.config.MigrationsTable, version, dirty)
if _, err := tx.Exec(query); err != nil {
tx.Rollback()
return &database.Error{OrigErr: err, Query: []byte(query)}
}
}
if err := tx.Commit(); err != nil {
return &database.Error{OrigErr: err, Err: "transaction commit failed"}
}
return nil
}
func (m *Ql) Version() (version int, dirty bool, err error) {
query := "SELECT version, dirty FROM " + m.config.MigrationsTable + " LIMIT 1"
err = m.db.QueryRow(query).Scan(&version, &dirty)
if err != nil {
return database.NilVersion, false, nil
}
return version, dirty, nil
}
+62
View File
@@ -0,0 +1,62 @@
package ql
import (
"database/sql"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"testing"
_ "github.com/cznic/ql/driver"
"github.com/golang-migrate/migrate/v4"
dt "github.com/golang-migrate/migrate/v4/database/testing"
_ "github.com/golang-migrate/migrate/v4/source/file"
)
func Test(t *testing.T) {
dir, err := ioutil.TempDir("", "ql-driver-test")
if err != nil {
return
}
defer func() {
os.RemoveAll(dir)
}()
fmt.Printf("DB path : %s\n", filepath.Join(dir, "ql.db"))
p := &Ql{}
addr := fmt.Sprintf("ql://%s", filepath.Join(dir, "ql.db"))
d, err := p.Open(addr)
if err != nil {
t.Fatalf("%v", err)
}
db, err := sql.Open("ql", filepath.Join(dir, "ql.db"))
if err != nil {
return
}
defer func() {
if err := db.Close(); err != nil {
return
}
}()
dt.Test(t, d, []byte("CREATE TABLE t (Qty int, Name string);"))
driver, err := WithInstance(db, &Config{})
if err != nil {
t.Fatalf("%v", err)
}
if err := d.Drop(); err != nil {
t.Fatal(err)
}
m, err := migrate.NewWithDatabaseInstance(
"file://./migration",
"ql", driver)
if err != nil {
t.Fatalf("%v", err)
}
fmt.Println("UP")
err = m.Up()
if err != nil {
t.Fatalf("%v", err)
}
}
+6
View File
@@ -0,0 +1,6 @@
Redshift
===
This provides a Redshift driver for migrations. It is used whenever the URL of the database starts with `redshift://`.
Redshift is PostgreSQL compatible but has some specific features (or lack thereof) that require slightly different behavior.
+46
View File
@@ -0,0 +1,46 @@
package redshift
import (
"net/url"
"github.com/golang-migrate/migrate/v4/database"
"github.com/golang-migrate/migrate/v4/database/postgres"
)
// init registers the driver under the name 'redshift'
func init() {
db := new(Redshift)
db.Driver = new(postgres.Postgres)
database.Register("redshift", db)
}
// Redshift is a wrapper around the PostgreSQL driver which implements Redshift-specific behavior.
//
// Currently, the only different behaviour is the lack of locking in Redshift. The (Un)Lock() method(s) have been overridden from the PostgreSQL adapter to simply return nil.
type Redshift struct {
// The wrapped PostgreSQL driver.
database.Driver
}
// Open implements the database.Driver interface by parsing the URL, switching the scheme from "redshift" to "postgres", and delegating to the underlying PostgreSQL driver.
func (driver *Redshift) Open(dsn string) (database.Driver, error) {
parsed, err := url.Parse(dsn)
if err != nil {
return nil, err
}
parsed.Scheme = "postgres"
psql, err := driver.Driver.Open(parsed.String())
if err != nil {
return nil, err
}
return &Redshift{Driver: psql}, nil
}
// Lock implements the database.Driver interface by not locking and returning nil.
func (driver *Redshift) Lock() error { return nil }
// Unlock implements the database.Driver interface by not unlocking and returning nil.
func (driver *Redshift) Unlock() error { return nil }
View File
+35
View File
@@ -0,0 +1,35 @@
# Google Cloud Spanner
## Usage
The DSN must be given in the following format.
`spanner://projects/{projectId}/instances/{instanceId}/databases/{databaseName}`
See [Google Spanner Documentation](https://cloud.google.com/spanner/docs) for details.
| Param | WithInstance Config | Description |
| ----- | ------------------- | ----------- |
| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
| `url` | `DatabaseName` | The full path to the Spanner database resource. If provided as part of `Config` it must not contain a scheme or query string to match the format `projects/{projectId}/instances/{instanceId}/databases/{databaseName}`|
| `projectId` || The Google Cloud Platform project id
| `instanceId` || The id of the instance running Spanner
| `databaseName` || The name of the Spanner database
> **Note:** Google Cloud Spanner migrations can take a considerable amount of
> time. The migrations provided as part of the example take about 6 minutes to
> run on a small instance.
>
> ```log
> 1481574547/u create_users_table (21.354507597s)
> 1496539702/u add_city_to_users (41.647359754s)
> 1496601752/u add_index_on_user_emails (2m12.155787369s)
> 1496602638/u create_books_table (2m30.77299181s)
## Testing
To unit test the `spanner` driver, `SPANNER_DATABASE` needs to be set. You'll
need to sign-up to Google Cloud Platform (GCP) and have a running Spanner
instance since it is not possible to run Google Spanner outside GCP.
@@ -0,0 +1 @@
DROP TABLE Users

Some files were not shown because too many files have changed in this diff Show More