skytools.git
14 years agoremoved next arg from generated dispatcher handler init
Egon Valdmees [Tue, 22 Mar 2011 11:03:15 +0000 (11:03 +0000)]
removed next arg from generated dispatcher handler init

14 years agofix --ssh-add-key with missing authorized_keys file
Martin Pihlak [Fri, 18 Mar 2011 11:47:58 +0000 (13:47 +0200)]
fix --ssh-add-key with missing authorized_keys file

14 years agoempty handler bug fix
Egon Valdmees [Thu, 17 Mar 2011 21:58:13 +0000 (21:58 +0000)]
empty handler bug fix

14 years agoCouple of bugfixes for stale backup lock handling.
Martin Pihlak [Thu, 17 Mar 2011 19:03:29 +0000 (21:03 +0200)]
Couple of bugfixes for stale backup lock handling.

14 years agoMove the pg_stop_backup() into a finally: block.
Steve Singer [Thu, 17 Mar 2011 18:18:45 +0000 (20:18 +0200)]
Move the pg_stop_backup() into a finally: block.
Some instances were reported where the base backup failed with some issue
but pg_stop_backup() hadn't been called and had to be called manually.
This should make that less likely

14 years agoAdd support for detecting stale locks and releasing them instead of aborting
Steve Singer [Thu, 17 Mar 2011 18:17:23 +0000 (20:17 +0200)]
Add support for detecting stale locks and releasing them instead of aborting

14 years agowalmgr.py: add "cleanup" command
Martin Pihlak [Wed, 16 Mar 2011 11:12:08 +0000 (13:12 +0200)]
walmgr.py: add "cleanup" command

14 years agomodernize wal-master.ini
Martin Pihlak [Wed, 16 Mar 2011 11:11:35 +0000 (13:11 +0200)]
modernize wal-master.ini

14 years agodoc/todo: expand some points
Marko Kreen [Wed, 23 Feb 2011 12:41:30 +0000 (14:41 +0200)]
doc/todo: expand some points

14 years agotodo cleanup
Marko Kreen [Wed, 23 Feb 2011 11:56:52 +0000 (13:56 +0200)]
todo cleanup

14 years agomake test: quickly test doctest-enabled modules
Marko Kreen [Wed, 23 Feb 2011 11:54:26 +0000 (13:54 +0200)]
make test: quickly test doctest-enabled modules

14 years agocascade status: avoid negative lag, sort by name
Marko Kreen [Wed, 23 Feb 2011 11:49:49 +0000 (13:49 +0200)]
cascade status: avoid negative lag, sort by name

negative lag can happen if info fetching takes too long.

14 years agoCoopConsumer: subconsumer_timeout config param
Marko Kreen [Wed, 23 Feb 2011 11:49:15 +0000 (13:49 +0200)]
CoopConsumer: subconsumer_timeout config param

14 years agoMerge remote-tracking branch 'intgit/master'
Marko Kreen [Wed, 16 Feb 2011 11:35:24 +0000 (13:35 +0200)]
Merge remote-tracking branch 'intgit/master'

14 years agobump db ver
Marko Kreen [Wed, 16 Feb 2011 11:29:29 +0000 (13:29 +0200)]
bump db ver

14 years agopgq_coop: add version() function
Marko Kreen [Wed, 16 Feb 2011 11:27:31 +0000 (13:27 +0200)]
pgq_coop: add version() function

14 years agopgq_coop: add possibility to take over dead subconsumer batch
Marko Kreen [Wed, 16 Feb 2011 11:26:57 +0000 (13:26 +0200)]
pgq_coop: add possibility to take over dead subconsumer batch

14 years agoDispatcher handler
Egon Valdmees [Fri, 11 Feb 2011 13:39:04 +0000 (15:39 +0200)]
Dispatcher handler

14 years agobulk-handler: sync with new api, add GP compat
Marko Kreen [Fri, 11 Feb 2011 13:36:20 +0000 (15:36 +0200)]
bulk-handler: sync with new api, add GP compat

14 years agolondiste: new qtable handler
Marko Kreen [Fri, 11 Feb 2011 13:34:44 +0000 (15:34 +0200)]
londiste: new qtable handler

14 years agolondiste; drop chained handler, use single handler per table
Marko Kreen [Fri, 11 Feb 2011 13:31:47 +0000 (15:31 +0200)]
londiste; drop chained handler, use single handler per table

Original idea was to let admins cascade different handlers,
but that seems too complicated.  Instead let programmer
use subclassing to create handlers with required functionality.

14 years agodbstruct: GP dist key support
Egon Valdmees [Fri, 11 Feb 2011 11:43:53 +0000 (13:43 +0200)]
dbstruct: GP dist key support

14 years agoAdminScript.display_table: allow field formatter function
Egon Valdmees [Fri, 11 Feb 2011 11:35:19 +0000 (13:35 +0200)]
AdminScript.display_table: allow field formatter function

14 years agoserialconsumer: fix batch_info usage
Marko Kreen [Fri, 11 Feb 2011 13:29:20 +0000 (15:29 +0200)]
serialconsumer: fix batch_info usage

14 years agogitignore: dont ignore too much
Marko Kreen [Fri, 11 Feb 2011 13:51:25 +0000 (15:51 +0200)]
gitignore: dont ignore too much

14 years agoSSH batch mode and no strict host checking for remote_walmgr
Martin Pihlak [Wed, 9 Feb 2011 13:12:47 +0000 (13:12 +0000)]
SSH batch mode and no strict host checking for remote_walmgr

14 years agoCreate log,pid,conf directories during --init
Martin Pihlak [Wed, 9 Feb 2011 13:08:33 +0000 (13:08 +0000)]
Create log,pid,conf directories during --init

14 years agodbscript: make db service-api errors noiseless
Marko Kreen [Wed, 9 Feb 2011 12:05:04 +0000 (14:05 +0200)]
dbscript: make db service-api errors noiseless

14 years agopython/pgq: batch_info cleanup
Marko Kreen [Fri, 4 Feb 2011 15:13:41 +0000 (17:13 +0200)]
python/pgq: batch_info cleanup

- move batch_info compat from sql to python
- get rid of get_batch_info(), self.batch_info is always defined
- make remoteconsumer use self.batch_info

14 years agosql/pgq: separate out the truncate trigger tests
Marko Kreen [Fri, 4 Feb 2011 15:08:37 +0000 (17:08 +0200)]
sql/pgq: separate out the truncate trigger tests

this makes success/failure on 8.3 clearer.

14 years agopgq.get_queue_info: report last tick in queue
Marko Kreen [Fri, 4 Feb 2011 14:59:34 +0000 (16:59 +0200)]
pgq.get_queue_info: report last tick in queue

14 years agopgq_node.get_worker_state: new function
Marko Kreen [Fri, 4 Feb 2011 14:44:30 +0000 (16:44 +0200)]
pgq_node.get_worker_state: new function

combines get_node_info & get_consumer_info

14 years agocascadedworker: fix tick tx handling on branch
Marko Kreen [Wed, 2 Feb 2011 13:51:49 +0000 (15:51 +0200)]
cascadedworker: fix tick tx handling on branch

14 years agopgqd: set defaults for pidfile & logfile
Marko Kreen [Tue, 1 Feb 2011 12:56:14 +0000 (14:56 +0200)]
pgqd: set defaults for pidfile & logfile

14 years agocascade tag-dead: spread info over all nodes
Marko Kreen [Tue, 1 Feb 2011 12:55:29 +0000 (14:55 +0200)]
cascade tag-dead: spread info over all nodes

14 years agorefresh libusual
Marko Kreen [Tue, 1 Feb 2011 12:55:18 +0000 (14:55 +0200)]
refresh libusual

14 years agopsycopgwrapper: export psycopg2.Error as DBError
Marko Kreen [Tue, 1 Feb 2011 12:54:04 +0000 (14:54 +0200)]
psycopgwrapper: export psycopg2.Error as DBError

14 years agostatus: info improvements
Marko Kreen [Tue, 1 Feb 2011 12:25:46 +0000 (14:25 +0200)]
status: info improvements

- show queue name
- show local node
- show lag in nicer way
- show last tick on node

14 years agocascade status: catch node errors, instead of aborting
Marko Kreen [Tue, 1 Feb 2011 12:25:18 +0000 (14:25 +0200)]
cascade status: catch node errors, instead of aborting

14 years agopgq_node.get_node_info: set worker_last_tick to last tick on root.
Marko Kreen [Tue, 1 Feb 2011 12:23:52 +0000 (14:23 +0200)]
pgq_node.get_node_info: set worker_last_tick to last tick on root.

14 years agorefresh libusual
Marko Kreen [Thu, 27 Jan 2011 11:05:54 +0000 (13:05 +0200)]
refresh libusual

14 years agobump db ver
Marko Kreen [Thu, 27 Jan 2011 10:54:23 +0000 (12:54 +0200)]
bump db ver

14 years agoCascadedWorker: call pgq_node.set_global_watermark() only on branch
Marko Kreen [Wed, 26 Jan 2011 14:54:26 +0000 (16:54 +0200)]
CascadedWorker: call pgq_node.set_global_watermark() only on branch

14 years agodbscript: debug log for new connections
Marko Kreen [Mon, 24 Jan 2011 15:28:35 +0000 (17:28 +0200)]
dbscript: debug log for new connections

14 years agotodo: drop done items
Marko Kreen [Mon, 24 Jan 2011 15:28:18 +0000 (17:28 +0200)]
todo: drop done items

14 years agopgq/cascade/admin: make tag-dead store into db
Marko Kreen [Mon, 24 Jan 2011 15:26:27 +0000 (17:26 +0200)]
pgq/cascade/admin: make tag-dead store into db

14 years agopgq_node.drop_node.sql: on local drop, delete all locations
Marko Kreen [Mon, 24 Jan 2011 15:25:33 +0000 (17:25 +0200)]
pgq_node.drop_node.sql: on local drop, delete all locations

14 years agopgq_node.register_location: NULL location keeps old value
Marko Kreen [Mon, 24 Jan 2011 15:25:03 +0000 (17:25 +0200)]
pgq_node.register_location: NULL location keeps old value

14 years agolondiste: change behaviour of --create* flags
Marko Kreen [Mon, 24 Jan 2011 13:28:21 +0000 (15:28 +0200)]
londiste: change behaviour of --create* flags

--create now does minimal pkey-only structure

--create-full does maximal structure

--create-only is dropped as too complex

14 years agoTweaking --init to work on custom postgres setup
Martin Pihlak [Sat, 22 Jan 2011 08:52:08 +0000 (10:52 +0200)]
Tweaking --init to work on custom postgres setup

14 years agoAdded --init-X commands to simplify walmgr setup.
Martin Pihlak [Fri, 21 Jan 2011 13:36:18 +0000 (15:36 +0200)]
Added --init-X commands to simplify walmgr setup.

- Both wal-master and slave now use [walmgr] configuration section.
- Added --init-master and --init-slave command to set up configuration files.
- Various path are guessed from PGDATA, optionally override with --set X=Y
- "slave" config key in wal-master.ini now only contains hostname
- "slave" config key in wal-slave.ini has been replaced with walmgr_data
- ssh-keygen option to init-master, ssh-add-key option to init-slave
- most of the cf.get() calls for directories have been replaced with cf.getfile()

14 years agorewrote netxt_batch for coop consumer to be more flexible, using next_batch_custom
Asko Tiidumaa [Tue, 18 Jan 2011 09:41:10 +0000 (09:41 +0000)]
rewrote netxt_batch for coop consumer to be more flexible, using next_batch_custom

14 years agoIgnore pgsql_tmp directory while creating base backup.
Martin Pihlak [Fri, 14 Jan 2011 14:11:02 +0000 (16:11 +0200)]
Ignore pgsql_tmp directory while creating base backup.

bug reported by Guillaume Smet

14 years agoDBScript: allow override config parameters from command line (--set).
Marko Kreen [Mon, 17 Jan 2011 14:47:56 +0000 (16:47 +0200)]
DBScript: allow override config parameters from command line (--set).

14 years agopgq/triggers: sanitize some dubious code
Marko Kreen [Fri, 14 Jan 2011 15:03:22 +0000 (17:03 +0200)]
pgq/triggers: sanitize some dubious code

14 years agobump db ver
Marko Kreen [Fri, 14 Jan 2011 13:19:50 +0000 (15:19 +0200)]
bump db ver

14 years agopgq.ticker(): rework the old tick sanity check
Marko Kreen [Fri, 14 Jan 2011 12:09:47 +0000 (14:09 +0200)]
pgq.ticker(): rework the old tick sanity check

due to good-enough timing, seq and xmax may appear reversed.

only error out when xmin is wrong.

otherwise throw warning, to debug the issue.

14 years agofixed case when event id is NULL on retry
Asko Tiidumaa [Fri, 14 Jan 2011 12:01:20 +0000 (12:01 +0000)]
fixed case when event id is NULL on retry

14 years agoMerge remote branch 'mpihlak/master'
Marko Kreen [Thu, 13 Jan 2011 13:58:56 +0000 (15:58 +0200)]
Merge remote branch 'mpihlak/master'

14 years agoFix "walmgr stop" not to disable archive_mode.
Martin Pihlak [Thu, 13 Jan 2011 13:31:03 +0000 (15:31 +0200)]
Fix "walmgr stop" not to disable archive_mode.

That is, unless master_restart_cmd is defined.

14 years agotests/londiste: test lagging takeover
Marko Kreen [Wed, 12 Jan 2011 13:59:34 +0000 (15:59 +0200)]
tests/londiste: test lagging takeover

14 years agonodeinfo status: show only first line of error
Marko Kreen [Wed, 12 Jan 2011 13:54:04 +0000 (15:54 +0200)]
nodeinfo status: show only first line of error

14 years agodoc/devnotes: point to new repo
Marko Kreen [Wed, 12 Jan 2011 13:53:38 +0000 (15:53 +0200)]
doc/devnotes: point to new repo

14 years agorefresh libusual
Marko Kreen [Mon, 10 Jan 2011 14:33:28 +0000 (16:33 +0200)]
refresh libusual

14 years agotests: default to tcp connection
Marko Kreen [Mon, 10 Jan 2011 14:32:24 +0000 (16:32 +0200)]
tests: default to tcp connection

14 years agocascadedworker: create ticks on branch in separate tx
Marko Kreen [Mon, 10 Jan 2011 14:30:30 +0000 (16:30 +0200)]
cascadedworker: create ticks on branch in separate tx

txid_visible_in_snapshot() does not consider same txid
as visible in current snapshot.

14 years agotests/londiste: generate some load when testing
Marko Kreen [Mon, 10 Jan 2011 14:28:04 +0000 (16:28 +0200)]
tests/londiste: generate some load when testing

14 years agopgq/triggers: disallow primary key updates
Marko Kreen [Mon, 10 Jan 2011 14:27:03 +0000 (16:27 +0200)]
pgq/triggers: disallow primary key updates

14 years agopgq.ticker: better error message when sanity check fails
Marko Kreen [Mon, 10 Jan 2011 14:26:05 +0000 (16:26 +0200)]
pgq.ticker: better error message when sanity check fails

14 years agoMerge branch 'master' of internal-git:/git/dba/skytools-3
Asko Tiidumaa [Wed, 22 Dec 2010 09:30:21 +0000 (09:30 +0000)]
Merge branch 'master' of internal-git:/git/dba/skytools-3

14 years agofixes to producer sql filtering
Asko Tiidumaa [Wed, 22 Dec 2010 09:29:51 +0000 (09:29 +0000)]
fixes to producer sql filtering

14 years agoupdate syslog settings in pgqd.ini
Martin Pihlak [Wed, 22 Dec 2010 09:06:50 +0000 (11:06 +0200)]
update syslog settings in pgqd.ini

14 years agochange default syslog=1 for pgqd.
Martin Pihlak [Wed, 22 Dec 2010 08:56:19 +0000 (10:56 +0200)]
change default syslog=1 for pgqd.

14 years agoMerge remote branch 'intgit/master'
Marko Kreen [Tue, 21 Dec 2010 12:39:47 +0000 (14:39 +0200)]
Merge remote branch 'intgit/master'

14 years agolambda vs quite_literal
Asko Tiidumaa [Tue, 21 Dec 2010 12:37:52 +0000 (12:37 +0000)]
lambda vs quite_literal

14 years agofixed local_only functionality, structural changes
Asko Tiidumaa [Mon, 20 Dec 2010 18:22:48 +0000 (18:22 +0000)]
fixed local_only functionality, structural changes

14 years agoadded local_only to londiste config parameter - accepts only events for local tables
Asko Tiidumaa [Mon, 20 Dec 2010 16:57:38 +0000 (16:57 +0000)]
added local_only to londiste config parameter - accepts only events for local tables

14 years agoqaadmin: unquote_literal fix
Marko Kreen [Thu, 16 Dec 2010 13:46:00 +0000 (15:46 +0200)]
qaadmin: unquote_literal fix

14 years agolondiste add: create_flags must be 0 by default
Marko Kreen [Tue, 14 Dec 2010 13:16:48 +0000 (15:16 +0200)]
londiste add: create_flags must be 0 by default

14 years agobump db ver
Marko Kreen [Tue, 14 Dec 2010 12:54:51 +0000 (14:54 +0200)]
bump db ver

14 years agopgq_node: workarounds for bandaid casts
Marko Kreen [Tue, 14 Dec 2010 12:51:21 +0000 (14:51 +0200)]
pgq_node: workarounds for bandaid casts

14 years agopgq: avoid text||int concet, it breaks if db has bandaid casts installed
Marko Kreen [Tue, 14 Dec 2010 12:48:19 +0000 (14:48 +0200)]
pgq: avoid text||int concet, it breaks if db has bandaid casts installed

14 years agolondiste add-table: dont set create_flags prematurely
Marko Kreen [Tue, 14 Dec 2010 12:24:08 +0000 (14:24 +0200)]
londiste add-table: dont set create_flags prematurely

14 years agobump db ver
Marko Kreen [Mon, 13 Dec 2010 08:34:20 +0000 (10:34 +0200)]
bump db ver

14 years agoMerge remote branch 'askotm/master'
Marko Kreen [Mon, 13 Dec 2010 08:02:35 +0000 (10:02 +0200)]
Merge remote branch 'askotm/master'

14 years agofix pg_autovacuum defaults in pgq.tune_storage()
Martin Pihlak [Sun, 12 Dec 2010 13:34:00 +0000 (15:34 +0200)]
fix pg_autovacuum defaults in pgq.tune_storage()

Previous version used zero value for freeze_max_age, which caused
autovacuum to constantly re-vacuum all of the event tables because
of the wraparound threat. Set this and other values to -1 to indicate
system default.

14 years agofixed two typos
Asko Tiidumaa [Thu, 9 Dec 2010 16:27:42 +0000 (16:27 +0000)]
fixed two typos

14 years agowrote handle_fkeys to more gp-friendlier version
Asko Tiidumaa [Thu, 9 Dec 2010 16:25:33 +0000 (16:25 +0000)]
wrote handle_fkeys to more gp-friendlier version

14 years agoCosmetics for wal-master.ini + hot_standby support.
Martin Pihlak [Thu, 9 Dec 2010 15:34:03 +0000 (17:34 +0200)]
Cosmetics for wal-ini + hot_standby support.

14 years agoAllow "primary_conninfo" to be specified in wal-slave.ini
Martin Pihlak [Thu, 9 Dec 2010 15:32:58 +0000 (17:32 +0200)]
Allow "primary_conninfo" to be specified in wal-slave.ini

Based on a patch by Mark Kirkwood.

14 years agoSome adjustments for 9.0, wal_level GUC is also modified during setup/stop.
Martin Pihlak [Thu, 9 Dec 2010 15:32:00 +0000 (17:32 +0200)]
Some adjustments for 9.0, wal_level GUC is also modified during setup/stop.

14 years agodbstruct: use new acl parser
Marko Kreen [Wed, 8 Dec 2010 14:39:37 +0000 (16:39 +0200)]
dbstruct: use new acl parser

14 years agoqadmin: remove exception test
Marko Kreen [Wed, 8 Dec 2010 14:37:28 +0000 (16:37 +0200)]
qadmin: remove exception test

14 years agoskytools: parse_acl() func for postgres ACL parsing
Marko Kreen [Wed, 8 Dec 2010 14:33:22 +0000 (16:33 +0200)]
skytools: parse_acl() func for postgres ACL parsing

14 years agolondiste tables: dont show non-local tables
Marko Kreen [Wed, 8 Dec 2010 14:11:50 +0000 (16:11 +0200)]
londiste tables: dont show non-local tables

those can be seen with 'missing'

14 years agoqadmin: catch and show python exceptions
Marko Kreen [Wed, 8 Dec 2010 13:59:04 +0000 (15:59 +0200)]
qadmin: catch and show python exceptions

14 years agolondiste do-sync: actually reset the pgq params
Marko Kreen [Wed, 8 Dec 2010 13:58:36 +0000 (15:58 +0200)]
londiste do-sync: actually reset the pgq params

14 years agotick logic now respects either lag or max tick count, not just both
Asko Tiidumaa [Wed, 8 Dec 2010 13:42:21 +0000 (13:42 +0000)]
tick logic now respects either lag or max tick count, not just both

14 years agopgq_node.version: 3.0.0.4
Marko Kreen [Wed, 8 Dec 2010 12:47:21 +0000 (14:47 +0200)]
pgq_node.version: 3.0.0.4

14 years agoskytools.unquote_ident: dont crash on empty string
Marko Kreen [Wed, 8 Dec 2010 12:45:58 +0000 (14:45 +0200)]
skytools.unquote_ident: dont crash on empty string