Greg Sabino Mullane [Thu, 10 Apr 2014 18:00:08 +0000 (14:00 -0400)]
Rewrite the delete_rows function.
Loop through and make sure we have the maximum number of async targets running at all times.
Moved much of the logic to per-target attribs.
Greg Sabino Mullane [Thu, 10 Apr 2014 13:43:28 +0000 (09:43 -0400)]
Debug the initdb call during testing
Greg Sabino Mullane [Thu, 10 Apr 2014 13:42:46 +0000 (09:42 -0400)]
Better tweaking for pgservice items
Greg Sabino Mullane [Wed, 9 Apr 2014 23:20:08 +0000 (19:20 -0400)]
Remove debugging.
Greg Sabino Mullane [Wed, 9 Apr 2014 23:18:20 +0000 (19:18 -0400)]
Better munging if we are using a service
Greg Sabino Mullane [Sun, 6 Apr 2014 20:07:03 +0000 (16:07 -0400)]
Ignore any local .bucardorc files.
Greg Sabino Mullane [Tue, 1 Apr 2014 11:05:29 +0000 (07:05 -0400)]
Remove useless comment
Greg Sabino Mullane [Mon, 31 Mar 2014 13:27:17 +0000 (09:27 -0400)]
Bump copyright to 2014
Joshua Tolley [Thu, 27 Mar 2014 21:01:09 +0000 (15:01 -0600)]
Add test for 'bucardo upgrade'
Joshua Tolley [Thu, 27 Mar 2014 20:51:47 +0000 (14:51 -0600)]
More bucardo.upgrade fixes
Now bucardo upgrade run against a database that doesn't need any changes
correctly reports that no changes are necessary
Greg Sabino Mullane [Thu, 20 Mar 2014 21:08:51 +0000 (17:08 -0400)]
If we are using a service file, make sure we don't default slurp in host and port env variables
Greg Sabino Mullane [Thu, 20 Mar 2014 18:24:35 +0000 (14:24 -0400)]
git should ignore service testing files
Joshua Tolley [Thu, 20 Mar 2014 17:36:27 +0000 (11:36 -0600)]
Modify syntax so the upgrade process can understand it
Greg Sabino Mullane [Mon, 17 Mar 2014 19:38:49 +0000 (15:38 -0400)]
Make the dbservice column have an empty default, so we don't have to start checking "defined or length..." everywhere
Joshua Tolley [Wed, 12 Mar 2014 18:29:44 +0000 (12:29 -0600)]
Revert comment, so it stays in sync with reverted code
Joshua Tolley [Wed, 12 Mar 2014 16:30:47 +0000 (10:30 -0600)]
Finish supporting non-ASCII table and column names
Joshua Tolley [Tue, 18 Feb 2014 20:06:37 +0000 (13:06 -0700)]
Bucardo now accepts pg_service for databases
Joshua Tolley [Fri, 14 Feb 2014 17:59:34 +0000 (10:59 -0700)]
service strings are working; still needs tests
Greg Sabino Mullane [Tue, 18 Feb 2014 14:54:46 +0000 (09:54 -0500)]
Always use async when doing deletes. The previous way had a thinko - this should ensure large deletes run concurrently on different hosts.
Joshua Tolley [Thu, 13 Feb 2014 23:25:17 +0000 (16:25 -0700)]
Make install quit ignoring dbname and dbuser options
... again.
Joshua Tolley [Fri, 14 Feb 2014 19:59:48 +0000 (12:59 -0700)]
Don't assume the bucardo database exists on install
Joshua Tolley [Fri, 14 Feb 2014 18:14:18 +0000 (11:14 -0700)]
Revert "Make install quit ignoring dbname and dbuser options"
I should probably have run the test suite before committing...
This reverts commit
39dcde241abcdfcc45415d2eb210d8b2c694f479.
Joshua Tolley [Thu, 13 Feb 2014 23:25:17 +0000 (16:25 -0700)]
Make install quit ignoring dbname and dbuser options
Joshua Tolley [Thu, 13 Feb 2014 20:55:15 +0000 (13:55 -0700)]
s/vaccuum/vacuum/
Greg Sabino Mullane [Fri, 17 Jan 2014 15:49:09 +0000 (10:49 -0500)]
Make our statement_chunk_size default match up.
Greg Sabino Mullane [Thu, 16 Jan 2014 15:58:40 +0000 (10:58 -0500)]
Debugging code
Greg Sabino Mullane [Thu, 16 Jan 2014 15:42:02 +0000 (10:42 -0500)]
Password should default to undef, not an empty string
Greg Sabino Mullane [Sun, 22 Dec 2013 16:12:52 +0000 (11:12 -0500)]
Gotta have log levels on all glog() calls.
Greg Sabino Mullane [Sun, 22 Dec 2013 16:12:15 +0000 (11:12 -0500)]
Version bump to 4.99.11
David E. Wheeler [Tue, 17 Dec 2013 19:46:21 +0000 (11:46 -0800)]
Eval everything.
David E. Wheeler [Tue, 17 Dec 2013 01:11:53 +0000 (17:11 -0800)]
Someone cannot count.
David E. Wheeler [Tue, 17 Dec 2013 01:00:14 +0000 (17:00 -0800)]
Fix typo in comment.
Greg Sabino Mullane [Tue, 3 Dec 2013 17:10:55 +0000 (12:10 -0500)]
Check @tableoids, not tablelist, as the latter may be empty (inside validate_sync)
Greg Sabino Mullane [Sat, 30 Nov 2013 16:43:45 +0000 (11:43 -0500)]
Meta things for the new dev/ directory
Greg Sabino Mullane [Sat, 30 Nov 2013 16:42:15 +0000 (11:42 -0500)]
Various tests for developers
Greg Sabino Mullane [Sat, 30 Nov 2013 16:41:06 +0000 (11:41 -0500)]
Better makedelta test. Need to move these somewhere else.
Greg Sabino Mullane [Sat, 30 Nov 2013 16:39:34 +0000 (11:39 -0500)]
Fix up some of the makedelta logic.
Greg Sabino Mullane [Fri, 22 Nov 2013 04:24:03 +0000 (23:24 -0500)]
Better handling of inactive databases.
Greg Sabino Mullane [Fri, 22 Nov 2013 03:20:52 +0000 (22:20 -0500)]
Add a fifth inactive database to ensure we handle inactive targets gracefully.
Michelle Sullivan [Sun, 17 Nov 2013 01:57:57 +0000 (20:57 -0500)]
Better cleanup of syncrun entries.
Wrap some of the kid cleanup in evals.
Greg Sabino Mullane [Sun, 17 Nov 2013 01:52:00 +0000 (20:52 -0500)]
Don't package up that MYMETA junk
Greg Sabino Mullane [Fri, 15 Nov 2013 23:06:59 +0000 (18:06 -0500)]
Version 4.99.10
Greg Sabino Mullane [Fri, 15 Nov 2013 23:04:14 +0000 (18:04 -0500)]
Add quick test of sequence-only sync. Needs to do some actual diff testing though.
Greg Sabino Mullane [Fri, 15 Nov 2013 22:45:24 +0000 (17:45 -0500)]
Reduce logging for VAC subprocess to warning - we don't want purging details flooding the logs.
Greg Sabino Mullane [Fri, 15 Nov 2013 22:19:33 +0000 (17:19 -0500)]
User-supplied messages should always be shown.
Greg Sabino Mullane [Thu, 14 Nov 2013 15:01:08 +0000 (10:01 -0500)]
Don't gather column information if there are no columns! Fix for validate_sync and a table-less sync.
Greg Sabino Mullane [Thu, 14 Nov 2013 03:13:47 +0000 (22:13 -0500)]
Don't populate bucardo_delta_names if no tables (e.g. sequence only sync)
Greg Sabino Mullane [Tue, 12 Nov 2013 18:14:36 +0000 (13:14 -0500)]
Signature for 4.99.9
Greg Sabino Mullane [Tue, 12 Nov 2013 18:13:01 +0000 (13:13 -0500)]
Version 4.99.9
Greg Sabino Mullane [Tue, 12 Nov 2013 17:59:27 +0000 (12:59 -0500)]
Fix quoting for inserts to the bucardo_deltanames table in validate_sync.
Greg Sabino Mullane [Tue, 12 Nov 2013 05:16:54 +0000 (00:16 -0500)]
Initialize var
Greg Sabino Mullane [Tue, 12 Nov 2013 05:15:23 +0000 (00:15 -0500)]
Fixes for the new bucardo_delta_names table.
Greg Sabino Mullane [Tue, 12 Nov 2013 05:02:33 +0000 (00:02 -0500)]
Make sure we specify schema
Greg Sabino Mullane [Tue, 12 Nov 2013 04:39:38 +0000 (23:39 -0500)]
Always check pg_async_status before calling pg_cancel
Greg Sabino Mullane [Tue, 12 Nov 2013 04:20:57 +0000 (23:20 -0500)]
Stupid tabs
Greg Sabino Mullane [Tue, 12 Nov 2013 04:19:59 +0000 (23:19 -0500)]
Make sure we only try to populate bucardo_delta_names for source databases.
Greg Sabino Mullane [Sat, 9 Nov 2013 19:31:24 +0000 (14:31 -0500)]
On FK warnings, show the contraint name, otherwise we often have duplicated we cannot distinguish.
Greg Sabino Mullane [Sat, 9 Nov 2013 18:18:46 +0000 (13:18 -0500)]
Show the host in our connection logging if we have one.
Greg Sabino Mullane [Sat, 9 Nov 2013 18:10:40 +0000 (13:10 -0500)]
Testing putting a pg_cancel at the end of the kid before the rollback.
Greg Sabino Mullane [Sat, 9 Nov 2013 18:06:57 +0000 (13:06 -0500)]
Don't show the initial DSN to the Bucardo database
Greg Sabino Mullane [Fri, 8 Nov 2013 19:41:35 +0000 (14:41 -0500)]
We don't need to check database wide when we already check database/table combo anyway.
Greg Sabino Mullane [Fri, 8 Nov 2013 04:53:09 +0000 (23:53 -0500)]
Introduce the concept of a "quick delta" check.
By default, this is turned on, but is left as a config variable in case this turns out to be a very bad idea. :)
This is a huge win on slow networks. Rather than getting a distinct pk count for every table in the sync, we ask the remote
database for a list of all tables and whether or not they have any changes. If they don't, we do not have to bother
with the normal "select distinct" query. This reduces the number of database calls drastically for syncs with a
large number of tables. In addition, the delta quick queries inside the function are extraorfinarily fast compared
to their distinct counterparts, as they do a select 1 ... limit 1.
All of this is accomplished by a new table on each source database called bucardo_delta_names, which stores a
list of all tables for a sync, along with their delta and track table names. This table's information is rewritten
on Bucardo startup each time. The new function uses this table to dynamically generate the quick delta queries
for each table, and then returns it in a simple text format.
This is especially noticeable on slow networks with a large number of tables in a sync. Testing in the case
that drove all this improved the sync run speed from 42 seconds to less than a second (for no rows - the
number of rows is a constant cost limited by how fast COPY goes, in most cases).
Greg Sabino Mullane [Wed, 6 Nov 2013 16:56:55 +0000 (11:56 -0500)]
Let's try this again...
Greg Sabino Mullane [Wed, 6 Nov 2013 16:55:15 +0000 (11:55 -0500)]
Typo.
Greg Sabino Mullane [Tue, 5 Nov 2013 23:20:31 +0000 (18:20 -0500)]
Show chunk when copying
Greg Sabino Mullane [Tue, 5 Nov 2013 20:31:13 +0000 (15:31 -0500)]
Change default statement_chunk_size from 10000 to 8000, based on real-world experience.
Greg Sabino Mullane [Tue, 5 Nov 2013 16:47:29 +0000 (11:47 -0500)]
Use prepare_cached, not prepare, inside of fetch1_sql
Greg Sabino Mullane [Tue, 5 Nov 2013 16:18:30 +0000 (11:18 -0500)]
Typo: compress_delta is a function, not a table
Greg Sabino Mullane [Tue, 29 Oct 2013 04:37:13 +0000 (00:37 -0400)]
Cache the column lookups inside of validate_sync.
Greg Sabino Mullane [Fri, 25 Oct 2013 03:12:12 +0000 (23:12 -0400)]
Make sure we do something sensible if we call for a log_level before we have loaded the config.
Greg Sabino Mullane [Thu, 24 Oct 2013 03:36:59 +0000 (23:36 -0400)]
Tabs cleanup
Greg Sabino Mullane [Thu, 24 Oct 2013 03:35:29 +0000 (23:35 -0400)]
Replace 'haztrig' with cached version inside of validate_sync.
Greg Sabino Mullane [Thu, 24 Oct 2013 02:38:56 +0000 (22:38 -0400)]
Show the DSN we are connecting with
Greg Sabino Mullane [Wed, 23 Oct 2013 14:58:49 +0000 (10:58 -0400)]
Report on total sync time in the logs, even if no rows were changed.
Greg Sabino Mullane [Wed, 23 Oct 2013 03:38:26 +0000 (23:38 -0400)]
Only try and create that index once.
Greg Sabino Mullane [Wed, 23 Oct 2013 03:29:22 +0000 (23:29 -0400)]
Speed up the initial validate sync quite a bit by pre-loading all the table, index, and function existence information.
Greg Sabino Mullane [Wed, 23 Oct 2013 02:16:06 +0000 (22:16 -0400)]
No longer use the whole prefix when building the listen/notify strings, as it may contain a syncname now.
Greg Sabino Mullane [Sun, 20 Oct 2013 02:10:32 +0000 (22:10 -0400)]
When doing `bucardo list databases`, don't show the psql port connection param unless it is set to something.
Greg Sabino Mullane [Sat, 19 Oct 2013 01:41:56 +0000 (21:41 -0400)]
Fix wrong search before inserting to bucardo_delta_targets.
This would explain the large number of rows seen in that table in the wild!
Greg Sabino Mullane [Fri, 18 Oct 2013 04:12:02 +0000 (00:12 -0400)]
Expand the arrays in the opening config dump in the logs.
Greg Sabino Mullane [Fri, 18 Oct 2013 03:41:20 +0000 (23:41 -0400)]
Enforce sane isolation levels
Greg Sabino Mullane [Fri, 18 Oct 2013 02:42:28 +0000 (22:42 -0400)]
Add new config 'log_showsyncname', defaulting to 1 (on), which will show the name of the current sync in parens after the 'KID' in the log output.
So rather than this:
(29985) [Thu Oct 17 22:40:12.885 2013] KID Database "A" backend PID: 29987
You will see this:
(29985) [Thu Oct 17 22:40:12.885 2013] KID (fctest) Database "A" backend PID: 29987
As suggested by Michelle Sullivan on the mailing list.
Greg Sabino Mullane [Fri, 18 Oct 2013 02:30:52 +0000 (22:30 -0400)]
Typo.
Greg Sabino Mullane [Thu, 17 Oct 2013 03:12:42 +0000 (23:12 -0400)]
Update versions in html docs
Greg Sabino Mullane [Thu, 17 Oct 2013 03:10:44 +0000 (23:10 -0400)]
Fix some pod
Greg Sabino Mullane [Thu, 17 Oct 2013 02:59:57 +0000 (22:59 -0400)]
Don't remember why this was here, but mark it as LOG_DEBUG
Greg Sabino Mullane [Thu, 17 Oct 2013 02:58:51 +0000 (22:58 -0400)]
Away with ye, tab
David E. Wheeler [Wed, 16 Oct 2013 21:25:07 +0000 (14:25 -0700)]
Set version to 4.99.8.
David E. Wheeler [Wed, 16 Oct 2013 20:39:09 +0000 (13:39 -0700)]
Try harder to collect and emit errors on validation failures.
There's no need to wait for a NOTIFY, as `validate_sync()` runs synchronously.
So just check the return value for errors, and use excpetion handling to catch
other kinds of errors. This allows us to catch and emit all errors and also
allow execution to finish validating all syncs (should it be validating
multiples). Closes #65.
David E. Wheeler [Tue, 15 Oct 2013 23:35:57 +0000 (16:35 -0700)]
Improve the auto-creation of the "bucardo" role.
When adding a database, that is. The old code led to confusing errors, where
the user would be told that Bucardo failed to connect as user "postgres" when
you had not, in fact, asked it to. So log more informat as it goes along,
noting failure to connect as "bucardo", trying to connect as "postgres", and
showing success or failure at doing so. This makes things much clearer to the
user what's going on -- and is less fussy, to boot.
Resolves #63.
Greg Sabino Mullane [Tue, 15 Oct 2013 00:56:56 +0000 (20:56 -0400)]
More bypassing of track_rates
Greg Sabino Mullane [Tue, 15 Oct 2013 00:55:05 +0000 (20:55 -0400)]
Skip the not-working-now track_rates section.
Joshua Tolley [Mon, 14 Oct 2013 14:52:23 +0000 (08:52 -0600)]
Merge in patch to ensure objects from different databases but having the same
names are treated distinctly.
David E. Wheeler [Mon, 7 Oct 2013 18:45:39 +0000 (11:45 -0700)]
Log verbose instead of debug.
For serialization and deadlock failures.
David E. Wheeler [Mon, 7 Oct 2013 16:56:06 +0000 (09:56 -0700)]
Log loglevel as a string.
David E. Wheeler [Mon, 7 Oct 2013 16:23:07 +0000 (09:23 -0700)]
Make it clearer which handles to log.
David E. Wheeler [Mon, 7 Oct 2013 16:21:55 +0000 (09:21 -0700)]
Topic, not lexical.
David E. Wheeler [Mon, 7 Oct 2013 16:20:29 +0000 (09:20 -0700)]
Log only errors.
Greg Sabino Mullane [Sat, 5 Oct 2013 01:19:27 +0000 (21:19 -0400)]
Try harder to find initdb when testing.
Greg Sabino Mullane [Sat, 5 Oct 2013 01:19:03 +0000 (21:19 -0400)]
Add some text fields to the dbgroup and herd tables for future use in "cloning"