skytools.git
14 years agopgq_node.drop_node: simplify
Marko Kreen [Wed, 27 Oct 2010 15:08:23 +0000 (18:08 +0300)]
pgq_node.drop_node: simplify

- londiste data is removed by cascaded fkey
- use force when dropping queue

14 years agopgq.tune_storage: turn autovac off for queue tables
Marko Kreen [Wed, 27 Oct 2010 14:58:49 +0000 (17:58 +0300)]
pgq.tune_storage: turn autovac off for queue tables

14 years agopgq.drop_queue: add force flag
Marko Kreen [Wed, 27 Oct 2010 14:57:39 +0000 (17:57 +0300)]
pgq.drop_queue: add force flag

14 years agoFix pgq.next_batch_custom()
Marko Kreen [Mon, 25 Oct 2010 10:25:03 +0000 (13:25 +0300)]
Fix pgq.next_batch_custom()

Seems it was broken in some reorg.

14 years agoInstall queue_mover3
Marko Kreen [Mon, 25 Oct 2010 07:52:54 +0000 (10:52 +0300)]
Install queue_mover3

14 years agochecker: compare-repair-apply
Marko Kreen [Wed, 20 Oct 2010 11:35:02 +0000 (14:35 +0300)]
checker: compare-repair-apply

14 years agopgq: fix table reference
Marko Kreen [Wed, 20 Oct 2010 10:04:39 +0000 (13:04 +0300)]
pgq: fix table reference

14 years agocascadedconsumer: missing return in get_consumer_state
Marko Kreen [Tue, 19 Oct 2010 11:56:26 +0000 (14:56 +0300)]
cascadedconsumer: missing return in get_consumer_state

14 years agodbsctruct: fix reorg
Marko Kreen [Tue, 19 Oct 2010 08:54:50 +0000 (11:54 +0300)]
dbsctruct: fix reorg

15 years agolondiste: trigger-cleanup trigger on londiste.table_info
Marko Kreen [Wed, 13 Oct 2010 13:58:53 +0000 (16:58 +0300)]
londiste: trigger-cleanup trigger on londiste.table_info

15 years agoskytools.checker: fix, add repair-apply mode
Marko Kreen [Wed, 13 Oct 2010 13:57:37 +0000 (16:57 +0300)]
skytools.checker: fix, add repair-apply mode

15 years agolondiste: automatic db upgrade
Marko Kreen [Tue, 12 Oct 2010 13:26:43 +0000 (16:26 +0300)]
londiste: automatic db upgrade

this is an experiment...

15 years agolondiste: "on delete cascade" fkeys for tables
Marko Kreen [Tue, 12 Oct 2010 13:26:04 +0000 (16:26 +0300)]
londiste: "on delete cascade" fkeys for tables

this avoids need to write cleanup code

15 years agohack: cascadedconsumer send standby info
Marko Kreen [Tue, 12 Oct 2010 12:50:12 +0000 (15:50 +0300)]
hack: cascadedconsumer send standby info

15 years agolondiste.root_check_seqs: lock node row
Marko Kreen [Tue, 12 Oct 2010 12:32:23 +0000 (15:32 +0300)]
londiste.root_check_seqs: lock node row

15 years agorefresh libusual
Marko Kreen [Tue, 12 Oct 2010 12:16:39 +0000 (15:16 +0300)]
refresh libusual

15 years agodoc: fix make rules
Marko Kreen [Tue, 12 Oct 2010 12:15:57 +0000 (15:15 +0300)]
doc: fix make rules

15 years agotodo update, move existing stuff to NEWS
Marko Kreen [Tue, 12 Oct 2010 12:13:30 +0000 (15:13 +0300)]
todo update, move existing stuff to NEWS

15 years agocascadeadmin: remove obsolete entries from help text
Marko Kreen [Tue, 12 Oct 2010 12:09:20 +0000 (15:09 +0300)]
cascadeadmin: remove obsolete entries from help text

15 years agoskytools.checker: generalize more
Marko Kreen [Tue, 12 Oct 2010 12:08:18 +0000 (15:08 +0300)]
skytools.checker: generalize more

15 years agomove londiste & pgq_node maint into pgq.maint_operations
Marko Kreen [Tue, 12 Oct 2010 11:58:29 +0000 (14:58 +0300)]
move londiste & pgq_node maint into pgq.maint_operations

15 years agoUse different SQL for trigger detection on Postgres 9.0
Marko Kreen [Tue, 12 Oct 2010 11:46:37 +0000 (14:46 +0300)]
Use different SQL for trigger detection on Postgres 9.0

15 years agodrop_node: fix message
Marko Kreen [Tue, 12 Oct 2010 11:40:24 +0000 (14:40 +0300)]
drop_node: fix message

15 years agouse defa sklog
Marko Kreen [Tue, 12 Oct 2010 11:39:26 +0000 (14:39 +0300)]
use defa sklog

15 years agoconfigure: --with-skylog arg to set default value
Marko Kreen [Tue, 12 Oct 2010 11:37:29 +0000 (14:37 +0300)]
configure: --with-skylog arg to set default value

15 years agomore checker
Marko Kreen [Thu, 23 Sep 2010 00:52:16 +0000 (17:52 -0700)]
more checker

15 years agoduplicated repair and compart into checker for crossmover
Asko Oja [Sat, 18 Sep 2010 16:43:23 +0000 (16:43 +0000)]
duplicated repair and compart into checker for crossmover

15 years agonew dbservice module, for pl/python funcs
Marko Kreen [Tue, 28 Sep 2010 21:16:21 +0000 (14:16 -0700)]
new dbservice module, for pl/python funcs

15 years agoskytools: JSON wrapper
Marko Kreen [Thu, 23 Sep 2010 00:52:39 +0000 (17:52 -0700)]
skytools: JSON wrapper

- find working JSON module
- easier kwargs API

15 years agolondiste: helpstr reorg, add --reset and --rewind
Marko Kreen [Fri, 17 Sep 2010 03:12:00 +0000 (06:12 +0300)]
londiste: helpstr reorg, add --reset and --rewind

15 years agoCascadedConsumer: --reset and --rewind commands
Marko Kreen [Fri, 17 Sep 2010 03:11:37 +0000 (06:11 +0300)]
CascadedConsumer: --reset and --rewind commands

15 years agolondiste: use londiste.upgrade.2to3.sql for ugprade instead of hacking around
Marko Kreen [Wed, 15 Sep 2010 13:22:42 +0000 (16:22 +0300)]
londiste: use londiste.upgrade.2to3.sql for ugprade instead of hacking around

15 years agoRefresh libusual
Marko Kreen [Wed, 15 Sep 2010 13:01:35 +0000 (16:01 +0300)]
Refresh libusual

15 years agoskytools: Support psycopg 2.2
Marko Kreen [Wed, 15 Sep 2010 08:24:20 +0000 (11:24 +0300)]
skytools: Support psycopg 2.2

There fileno() has moved from cursor to connection.

15 years agoskytools.config: apply defaults to only main section
Marko Kreen [Tue, 14 Sep 2010 10:17:35 +0000 (13:17 +0300)]
skytools.config: apply defaults to only main section

Otherwise they appear in all sections.

Patch by Martin Otto

15 years agoskytools.config: always check main section
Marko Kreen [Tue, 14 Sep 2010 09:44:31 +0000 (12:44 +0300)]
skytools.config: always check main section

Patch by Martin Otto

15 years agopgqd: remove leftover _exit()
Marko Kreen [Tue, 14 Sep 2010 08:37:55 +0000 (11:37 +0300)]
pgqd: remove leftover _exit()

15 years agopgq/triggers: throw error if SKIP if used in AFTER trigger.
Marko Kreen [Mon, 6 Sep 2010 13:12:18 +0000 (16:12 +0300)]
pgq/triggers: throw error if SKIP if used in AFTER trigger.

15 years agolondiste,pgq: Support & use TRUNCATE trigger by default.
Marko Kreen [Mon, 6 Sep 2010 12:57:13 +0000 (15:57 +0300)]
londiste,pgq: Support & use TRUNCATE trigger by default.

Event format:
 ev_type   - R
 ev_extra1 - table name

Works only on Postgres 8.4+.

Patch by Hannu Krosing.

15 years agosql/ticker: make new logic actually work
Marko Kreen [Fri, 3 Sep 2010 14:08:38 +0000 (17:08 +0300)]
sql/ticker: make new logic actually work

15 years agopgq.maint_operations: don't do rotate_step2, if nothing to rotate
Marko Kreen [Fri, 3 Sep 2010 14:03:16 +0000 (17:03 +0300)]
pgq.maint_operations: don't do rotate_step2, if nothing to rotate

15 years agoconfigure: enable asciidoc/xmlto automatically, if building from .git
Marko Kreen [Fri, 3 Sep 2010 10:53:31 +0000 (13:53 +0300)]
configure: enable asciidoc/xmlto automatically, if building from .git

15 years agopgq/trigger: add when= argument to sqltriga/logutriga.
Marko Kreen [Thu, 2 Sep 2010 14:56:56 +0000 (17:56 +0300)]
pgq/trigger: add when= argument to sqltriga/logutriga.

Similar to WHEN argument for CREATE TRIGGER in 9.0.

15 years agoAdd more junk to gitignore
Marko Kreen [Wed, 1 Sep 2010 14:32:00 +0000 (17:32 +0300)]
Add more junk to gitignore

15 years agorefresh libusual
Marko Kreen [Wed, 1 Sep 2010 14:31:40 +0000 (17:31 +0300)]
refresh libusual

15 years agopgqd: allow customizable maintenance
Marko Kreen [Wed, 1 Sep 2010 14:31:12 +0000 (17:31 +0300)]
pgqd: allow customizable maintenance

15 years agonew function too
Marko Kreen [Wed, 1 Sep 2010 14:29:24 +0000 (17:29 +0300)]
new function too

15 years agopgq: customizable maintenance functions
Marko Kreen [Wed, 1 Sep 2010 14:28:55 +0000 (17:28 +0300)]
pgq: customizable maintenance functions

15 years agoscripts/: conform with new -ini standard.
Marko Kreen [Tue, 31 Aug 2010 08:55:43 +0000 (11:55 +0300)]
scripts/: conform with new -ini standard.

Each script should define only its own config vars.

15 years agopgqd: switches to signal running process
Marko Kreen [Tue, 31 Aug 2010 08:55:08 +0000 (11:55 +0300)]
pgqd: switches to signal running process

15 years agoBublin handler.
Marko Kreen [Fri, 25 Jun 2010 11:31:49 +0000 (14:31 +0300)]
Bublin handler.

  londiste3 cf.ini add-table foo --handler='bublin(key_column)'

15 years agoPluggable per-table handlers for event processing.
Marko Kreen [Fri, 7 May 2010 12:17:53 +0000 (15:17 +0300)]
Pluggable per-table handlers for event processing.

1. Make handler modules known for Londiste:

  [londiste3]
  handler_modules = londiste.bublin, some_other.module

Modules are imported and classes found in __londiste_handlers__
module variable are registered.

2. Generic table setup:

  londiste3 cf.ini add-table foo --handler='handler1' --handler='handler2(arg1, arg2)'

Londiste standard handler is default and always appended to custom plugins.

15 years agopgq.force_tick: work correctly if ticking is not allowed.
Marko Kreen [Mon, 30 Aug 2010 10:48:33 +0000 (13:48 +0300)]
pgq.force_tick: work correctly if ticking is not allowed.

Simply return last tick_id.

15 years agoFix typo.
Marko Kreen [Mon, 2 Aug 2010 14:10:12 +0000 (17:10 +0300)]
Fix typo.

15 years agoDisable use_skylog if non-daemon on console.
Marko Kreen [Mon, 2 Aug 2010 11:44:00 +0000 (14:44 +0300)]
Disable use_skylog if non-daemon on console.

This avoids console commands to send errors/warnings to central logging server.

Set use_skylog=2 to always enable.

15 years agoDBScript: Wake from sleep on NOTIFY.
Marko Kreen [Mon, 2 Aug 2010 11:11:43 +0000 (14:11 +0300)]
DBScript: Wake from sleep on NOTIFY.

Use DBScript .listen(dbname, chan)/.unlisten(dbname, chan) to register.

15 years agodoc: install error handling
Marko Kreen [Wed, 28 Jul 2010 13:03:38 +0000 (16:03 +0300)]
doc: install error handling

15 years agoFix manpage deps
Marko Kreen [Tue, 27 Jul 2010 11:11:20 +0000 (14:11 +0300)]
Fix manpage deps

15 years agoMerge remote branch 'mpihlak/walmgr-dev'
Marko Kreen [Tue, 27 Jul 2010 10:41:59 +0000 (13:41 +0300)]
Merge remote branch 'mpihlak/walmgr-dev'

15 years agoIntroduce a 'slave_pg_xlog' configuration variable. This
Martin Pihlak [Tue, 27 Jul 2010 08:38:36 +0000 (11:38 +0300)]
Introduce a 'slave_pg_xlog' configuration variable. This
allows master and slave pg_xlog files to be in different
locations. During restore this directory is symlinked to
slave pg_xlog. Patch from Steve Singer.

15 years agoIntroduce a 'backup_datadir' configuration variable to
Martin Pihlak [Tue, 27 Jul 2010 07:54:24 +0000 (10:54 +0300)]
Introduce a 'backup_datadir' configuration variable to
control whether the slave data directory is kept or
overwritten during restore. Patch from Steve Singer.

15 years agomore files to ignore
Martin Pihlak [Tue, 27 Jul 2010 07:54:01 +0000 (10:54 +0300)]
more files to ignore

15 years agowalmgr.py: add --version option
Martin Pihlak [Tue, 27 Jul 2010 07:33:20 +0000 (10:33 +0300)]
walmgr.py: add --version option

15 years agoSimplify the cleanup logic for partial wals.
Martin Pihlak [Tue, 27 Jul 2010 07:27:10 +0000 (10:27 +0300)]
Simplify the cleanup logic for partial wals.

15 years ago"walmgr.py sync" now omits unneeded WAL segments if the
Martin Pihlak [Tue, 27 Jul 2010 07:18:35 +0000 (10:18 +0300)]
"walmgr.py sync" now omits unneeded WAL segments if the
database has been cleanly shut down. This greatly reduces
sync time during planned swithcovers as usually there
is only a single WAL file to be synched to slave.

15 years agomore files to ignore
Martin Pihlak [Tue, 27 Jul 2010 07:18:26 +0000 (10:18 +0300)]
more files to ignore

15 years agoMerge branch 'master' of git://github.com/markokr/skytools-dev
Martin Pihlak [Tue, 27 Jul 2010 06:47:22 +0000 (09:47 +0300)]
Merge branch 'master' of git://github.com/markokr/skytools-dev

15 years agodnscript: avoid .rollback() when closing connection, its unnecessary
Marko Kreen [Mon, 26 Jul 2010 13:01:20 +0000 (16:01 +0300)]
dnscript: avoid .rollback() when closing connection, its unnecessary

15 years agoskytools: set_tcp_keepalive()
Marko Kreen [Wed, 21 Jul 2010 16:02:07 +0000 (19:02 +0300)]
skytools: set_tcp_keepalive()

Separate TCP keepalive code out from connect_database(),
to be usable separately.  (Eg. for HTTP connections.)

Add also support for Darwin.

15 years agoConfigurable suffix support.
Marko Kreen [Tue, 20 Jul 2010 14:37:25 +0000 (17:37 +0300)]
Configurable suffix support.

Apply suffix also to manpages, SQL and doc directories

15 years agodbstruct: save & restore clustering info
Marko Kreen [Tue, 20 Jul 2010 14:34:05 +0000 (17:34 +0300)]
dbstruct: save & restore clustering info

Original patch by AndrĂ© Malo, applied with minor modifications

15 years agoskytools: export parse_sqltriga_sql
Marko Kreen [Mon, 19 Jul 2010 11:20:33 +0000 (14:20 +0300)]
skytools: export parse_sqltriga_sql

15 years agolondiste: avoid zombie processes with P_WAIT
Marko Kreen [Mon, 19 Jul 2010 11:18:46 +0000 (14:18 +0300)]
londiste: avoid zombie processes with P_WAIT

os.spawn(P_NOWAIT) leaves zombie childs.  P_WAIT is safe
to use here, as child should daemonize itself anyway.

15 years agoqadmin: exit command
Marko Kreen [Tue, 13 Jul 2010 11:48:41 +0000 (14:48 +0300)]
qadmin: exit command

15 years agoMerge commit 'upstream/master'
Martin Pihlak [Mon, 19 Jul 2010 07:12:41 +0000 (10:12 +0300)]
Merge commit 'upstream/master'

15 years agodeb: accept modules-9.0
Marko Kreen [Fri, 9 Jul 2010 10:18:37 +0000 (13:18 +0300)]
deb: accept modules-9.0

15 years agodeb: Lets see what happens if package is named skytools3.
Marko Kreen [Fri, 9 Jul 2010 10:14:58 +0000 (13:14 +0300)]
deb: Lets see what happens if package is named skytools3.

15 years agoClean ./build before install to avoid pkgloader installing too much.
Marko Kreen [Fri, 9 Jul 2010 10:14:14 +0000 (13:14 +0300)]
Clean ./build before install to avoid pkgloader installing too much.

15 years agoFix tgargs initialization for pgq.logtriga()
Marko Kreen [Fri, 9 Jul 2010 10:06:38 +0000 (13:06 +0300)]
Fix tgargs initialization for pgq.logtriga()

15 years agopgq_node.drop_node(): drop queue
Marko Kreen [Tue, 6 Jul 2010 12:27:50 +0000 (15:27 +0300)]
pgq_node.drop_node(): drop queue

15 years agolondiste.global_add_table(): catch unique_violation
Marko Kreen [Tue, 6 Jul 2010 12:23:49 +0000 (15:23 +0300)]
londiste.global_add_table(): catch unique_violation

To avoid failure when both setup and replay add the table.

15 years agoskytools.parse_sqltriga_sql(): extract PK field names
Marko Kreen [Mon, 28 Jun 2010 11:49:07 +0000 (14:49 +0300)]
skytools.parse_sqltriga_sql(): extract PK field names

15 years agoMove querybuilder code into separate file and refresh with some new code.
Marko Kreen [Mon, 28 Jun 2010 08:10:29 +0000 (11:10 +0300)]
Move querybuilder code into separate file and refresh with some new code.

15 years agodeb: deb90, keep debugging info.
Marko Kreen [Fri, 25 Jun 2010 11:35:56 +0000 (14:35 +0300)]
deb: deb90, keep debugging info.

It does not make sense to strip server packages.

15 years agoTune pylint quieter, to be more useful.
Marko Kreen [Thu, 10 Jun 2010 14:30:27 +0000 (17:30 +0300)]
Tune pylint quieter, to be more useful.

15 years agoUpdate londiste test with new script names
Marko Kreen [Thu, 3 Jun 2010 15:11:19 +0000 (18:11 +0300)]
Update londiste test with new script names

15 years agopgqd: dont complain about 'no pgq' too loudly
Marko Kreen [Fri, 25 Jun 2010 11:19:37 +0000 (14:19 +0300)]
pgqd: dont complain about 'no pgq' too loudly

15 years agoqadmin: max_cnt->max_count as thats what is accepted by ALTER.
Marko Kreen [Fri, 25 Jun 2010 10:08:22 +0000 (13:08 +0300)]
qadmin: max_cnt->max_count as thats what is accepted by ALTER.

15 years agolondiste: add compare/repair params to --ini
Marko Kreen [Wed, 23 Jun 2010 14:12:58 +0000 (17:12 +0300)]
londiste: add compare/repair params to --ini

15 years agolondiste compare: calculate checksum
Marko Kreen [Wed, 23 Jun 2010 14:11:03 +0000 (17:11 +0300)]
londiste compare: calculate checksum

Also allow specify format string.

Patch by Hannu Krosing

15 years agoskytools.Config: .options() and .has_options() methods.
Marko Kreen [Tue, 15 Jun 2010 12:25:02 +0000 (15:25 +0300)]
skytools.Config: .options() and .has_options() methods.

15 years agoFix double install on debian.
Marko Kreen [Wed, 23 Jun 2010 13:35:33 +0000 (16:35 +0300)]
Fix double install on debian.

15 years agopgq.Consumer: Provide self.batch_info with batch details
Marko Kreen [Fri, 11 Jun 2010 11:25:39 +0000 (14:25 +0300)]
pgq.Consumer: Provide self.batch_info with batch details

15 years agosql/londiste: set client_min_messages = 'warning' for regtest
Marko Kreen [Thu, 10 Jun 2010 14:35:04 +0000 (17:35 +0300)]
sql/londiste: set client_min_messages = 'warning' for regtest

15 years agolondiste.local_add_table(): skip triggers on leaf node
Marko Kreen [Thu, 10 Jun 2010 14:28:37 +0000 (17:28 +0300)]
londiste.local_add_table(): skip triggers on leaf node

15 years agopgq_node.is_leaf_node()
Marko Kreen [Thu, 10 Jun 2010 14:26:06 +0000 (17:26 +0300)]
pgq_node.is_leaf_node()

helper function to detect leaf node.

15 years agodont install pgq.logtriga()
Marko Kreen [Wed, 9 Jun 2010 13:09:37 +0000 (16:09 +0300)]
dont install pgq.logtriga()

15 years agoinstaller_find_file: stop searching from cwd
Marko Kreen [Wed, 9 Jun 2010 10:33:33 +0000 (13:33 +0300)]
installer_find_file: stop searching from cwd

It was fine for some testing, but not for serious usage.

15 years agopgq/cascade: when reporting batch pos problem, allow None
Marko Kreen [Tue, 8 Jun 2010 11:20:30 +0000 (14:20 +0300)]
pgq/cascade: when reporting batch pos problem, allow None

15 years agopgqd: Detect and remove dropped databases.
Marko Kreen [Thu, 3 Jun 2010 15:01:52 +0000 (18:01 +0300)]
pgqd: Detect and remove dropped databases.