skytools.git
14 years agosql/pgq: sanity check against incorrect db state
Marko Kreen [Fri, 12 Nov 2010 14:18:13 +0000 (16:18 +0200)]
sql/pgq: sanity check against incorrect db state

14 years agoqadmin: advanced syntax for londiste cmds, fix more quoting problems
Marko Kreen [Fri, 26 Nov 2010 11:02:10 +0000 (13:02 +0200)]
qadmin: advanced syntax for londiste cmds, fix more quoting problems

14 years agolondiste add-table --create: fix flag handling
Marko Kreen [Fri, 26 Nov 2010 10:04:07 +0000 (12:04 +0200)]
londiste add-table --create: fix flag handling

14 years agofix var name in last commit
Marko Kreen [Fri, 26 Nov 2010 10:01:40 +0000 (12:01 +0200)]
fix var name in last commit

14 years agocascadedconsumer: dont use .finish_remote_batch() in --reset
Marko Kreen [Fri, 26 Nov 2010 09:59:45 +0000 (11:59 +0200)]
cascadedconsumer: dont use .finish_remote_batch() in --reset

14 years agolondiste: play with session_replication_role only on 8.3+
Marko Kreen [Thu, 25 Nov 2010 15:47:10 +0000 (17:47 +0200)]
londiste: play with session_replication_role only on 8.3+

14 years agolondiste worker --reset fixes
Marko Kreen [Thu, 25 Nov 2010 15:44:54 +0000 (17:44 +0200)]
londiste worker --reset fixes

14 years agopgq.version: 3.0.0.2
Marko Kreen [Thu, 25 Nov 2010 13:07:48 +0000 (15:07 +0200)]
pgq.version: 3.0.0.2

14 years agotests/merge: some improvements
Marko Kreen [Thu, 25 Nov 2010 13:07:05 +0000 (15:07 +0200)]
tests/merge: some improvements

14 years agopgq.drop_queue: make force drop old consumers
Marko Kreen [Thu, 25 Nov 2010 13:06:28 +0000 (15:06 +0200)]
pgq.drop_queue: make force drop old consumers

14 years agoqadmin; better quoting support
Marko Kreen [Thu, 25 Nov 2010 12:19:18 +0000 (14:19 +0200)]
qadmin; better quoting support

14 years agounquote_ident: lowercase the non-quoted value
Marko Kreen [Thu, 25 Nov 2010 12:10:13 +0000 (14:10 +0200)]
unquote_ident: lowercase the non-quoted value

Otherwise the value will later be quoted incorrectly.

14 years agoskytools.sql_tokenizer: optionally return token position
Marko Kreen [Thu, 25 Nov 2010 12:04:43 +0000 (14:04 +0200)]
skytools.sql_tokenizer: optionally return token position

14 years agoskytools.unquote_(fq)ident: tests, relax fq
Marko Kreen [Thu, 25 Nov 2010 12:03:52 +0000 (14:03 +0200)]
skytools.unquote_(fq)ident: tests, relax fq

unquote_fqident does not require both parts anymore

14 years agolondiste setup: only set replica role for EXECUTE transaction
Marko Kreen [Thu, 25 Nov 2010 11:58:06 +0000 (13:58 +0200)]
londiste setup: only set replica role for EXECUTE transaction

otherwise cleanup triggers wont launch

14 years agoqadmin: londiste commands
Marko Kreen [Thu, 25 Nov 2010 09:53:17 +0000 (11:53 +0200)]
qadmin: londiste commands

add_table, remove_table, tables,
add_seq, remove_seq, seqs,
missing

14 years agoskytools.sql_tokenizer: support fully-qualified names
Marko Kreen [Thu, 25 Nov 2010 07:24:25 +0000 (09:24 +0200)]
skytools.sql_tokenizer: support fully-qualified names

It's easier to merge name parts here than to do it in upper layers.

The merging does not support some crazier parts of sql syntax
like comments between name parts, so it defaults to off.

14 years agopgq_node.version() - 3.0.0.2
Marko Kreen [Wed, 24 Nov 2010 20:29:25 +0000 (22:29 +0200)]
pgq_node.version() - 3.0.0.2

14 years agotests/merge: make it work again
Marko Kreen [Wed, 24 Nov 2010 20:27:21 +0000 (22:27 +0200)]
tests/merge: make it work again

14 years agotestlib: nicer psql launcher
Marko Kreen [Wed, 24 Nov 2010 20:19:48 +0000 (22:19 +0200)]
testlib: nicer psql launcher

14 years agopgq_node.set_partition_watermark: finally sync with other pgq_node cleanups
Marko Kreen [Wed, 24 Nov 2010 20:19:02 +0000 (22:19 +0200)]
pgq_node.set_partition_watermark: finally sync with other pgq_node cleanups

14 years agopgq.BatchWalker: relaxed __len__(), ignore out-of-order calls
Marko Kreen [Wed, 24 Nov 2010 20:18:22 +0000 (22:18 +0200)]
pgq.BatchWalker: relaxed __len__(), ignore out-of-order calls

14 years agoCascadedWorker: override is_batch_done
Marko Kreen [Wed, 24 Nov 2010 20:16:51 +0000 (22:16 +0200)]
CascadedWorker: override is_batch_done

To allow combined-branch get more that one batch ahead.

14 years agorefresh libusual
Marko Kreen [Wed, 24 Nov 2010 09:48:51 +0000 (11:48 +0200)]
refresh libusual

14 years agopgqd: report database removal
Marko Kreen [Wed, 24 Nov 2010 09:48:09 +0000 (11:48 +0200)]
pgqd: report database removal

14 years agolondiste copy: survive remove-table
Marko Kreen [Wed, 24 Nov 2010 09:11:34 +0000 (11:11 +0200)]
londiste copy: survive remove-table

14 years agopgq_node: add version() function, bump ver in pgq and londiste schemas.
Marko Kreen [Tue, 23 Nov 2010 12:06:02 +0000 (14:06 +0200)]
pgq_node: add version() function, bump ver in pgq and londiste schemas.

All code has seen various fixes, this will help detect old versions.

14 years agolondiste.get_table_list: allow merge-leafs have different tables
Marko Kreen [Tue, 23 Nov 2010 10:28:57 +0000 (12:28 +0200)]
londiste.get_table_list: allow merge-leafs have different tables

try to account only nodes that have particular table

14 years agopgqd: sync with cfparser changes
Marko Kreen [Mon, 22 Nov 2010 23:03:14 +0000 (01:03 +0200)]
pgqd: sync with cfparser changes

14 years agotestlib cleanup fixes
Marko Kreen [Fri, 19 Nov 2010 13:07:07 +0000 (15:07 +0200)]
testlib  cleanup fixes

14 years agoCleanup for the new querybuilder functions
Marko Kreen [Thu, 11 Nov 2010 11:24:07 +0000 (13:24 +0200)]
Cleanup for the new querybuilder functions

14 years agohelper functions added into querybuilder
Asko Oja [Wed, 10 Nov 2010 23:02:10 +0000 (23:02 +0000)]
helper functions added into querybuilder

14 years agopgqd: tilde expansion for pidfile and logfile
Marko Kreen [Wed, 10 Nov 2010 15:46:46 +0000 (17:46 +0200)]
pgqd: tilde expansion for pidfile and logfile

Patch by Asko Tiidumaa

14 years agoset_tcp_keepalive: check socket.fromfd(), its missing on win32
Marko Kreen [Wed, 10 Nov 2010 12:32:46 +0000 (14:32 +0200)]
set_tcp_keepalive: check socket.fromfd(), its missing on win32

14 years agopgq.maint_rotate_tables: detect and report tick inconsistency
Marko Kreen [Tue, 9 Nov 2010 13:37:12 +0000 (15:37 +0200)]
pgq.maint_rotate_tables: detect and report tick inconsistency

14 years agochanged syslog handler to send service type
Aleksei Plotnikov [Thu, 4 Nov 2010 13:26:55 +0000 (13:26 +0000)]
changed syslog handler to send service type

14 years agorefresh libusual
Marko Kreen [Thu, 4 Nov 2010 12:06:40 +0000 (14:06 +0200)]
refresh libusual

14 years agoRestore old compat package location.
Marko Kreen [Wed, 3 Nov 2010 15:57:33 +0000 (17:57 +0200)]
Restore old compat package location.

Workaround for broken debian broke it, needs bit different owrkaround later.

14 years agopgqd: apply connection lifetime to ticker connection
Marko Kreen [Wed, 3 Nov 2010 15:50:10 +0000 (17:50 +0200)]
pgqd: apply connection lifetime to ticker connection

14 years agorefresh libusual
Marko Kreen [Wed, 3 Nov 2010 15:49:42 +0000 (17:49 +0200)]
refresh libusual

14 years agoadded syslog logging handler with hostname support
Aleksei Plotnikov [Tue, 2 Nov 2010 13:51:22 +0000 (15:51 +0200)]
added syslog logging handler with hostname support

14 years agodbservice: use txid_current() instead of old name
Marko Kreen [Tue, 2 Nov 2010 13:25:51 +0000 (15:25 +0200)]
dbservice: use txid_current() instead of old name

14 years agodbstruct: use adbin instead of adsrc when detecting default
Marko Kreen [Tue, 2 Nov 2010 13:25:06 +0000 (15:25 +0200)]
dbstruct: use adbin instead of adsrc when detecting default

14 years agodeb: fwd-port fix for braindamaged Debian's python
Marko Kreen [Mon, 1 Nov 2010 15:44:21 +0000 (17:44 +0200)]
deb: fwd-port fix for braindamaged Debian's python

They broke distutils install just for fun, thus each
python .deb needs their braindamaged hack to work.

14 years agoskytools.Config: fix bug in has_option()
Marko Kreen [Mon, 1 Nov 2010 15:18:49 +0000 (17:18 +0200)]
skytools.Config: fix bug in has_option()

14 years agopgqd: do reset_logging() in cleanup(), bit cleaner shutdown
Marko Kreen [Mon, 1 Nov 2010 15:16:48 +0000 (17:16 +0200)]
pgqd: do reset_logging() in cleanup(), bit cleaner shutdown

14 years agopgq_node.drop_node: simplify
Marko Kreen [Wed, 27 Oct 2010 15:08:23 +0000 (18:08 +0300)]
pgq_node.drop_node: simplify

- londiste data is removed by cascaded fkey
- use force when dropping queue

14 years agopgq.tune_storage: turn autovac off for queue tables
Marko Kreen [Wed, 27 Oct 2010 14:58:49 +0000 (17:58 +0300)]
pgq.tune_storage: turn autovac off for queue tables

14 years agopgq.drop_queue: add force flag
Marko Kreen [Wed, 27 Oct 2010 14:57:39 +0000 (17:57 +0300)]
pgq.drop_queue: add force flag

14 years agoFix pgq.next_batch_custom()
Marko Kreen [Mon, 25 Oct 2010 10:25:03 +0000 (13:25 +0300)]
Fix pgq.next_batch_custom()

Seems it was broken in some reorg.

14 years agoInstall queue_mover3
Marko Kreen [Mon, 25 Oct 2010 07:52:54 +0000 (10:52 +0300)]
Install queue_mover3

14 years agochecker: compare-repair-apply
Marko Kreen [Wed, 20 Oct 2010 11:35:02 +0000 (14:35 +0300)]
checker: compare-repair-apply

14 years agopgq: fix table reference
Marko Kreen [Wed, 20 Oct 2010 10:04:39 +0000 (13:04 +0300)]
pgq: fix table reference

14 years agocascadedconsumer: missing return in get_consumer_state
Marko Kreen [Tue, 19 Oct 2010 11:56:26 +0000 (14:56 +0300)]
cascadedconsumer: missing return in get_consumer_state

14 years agodbsctruct: fix reorg
Marko Kreen [Tue, 19 Oct 2010 08:54:50 +0000 (11:54 +0300)]
dbsctruct: fix reorg

14 years agolondiste: trigger-cleanup trigger on londiste.table_info
Marko Kreen [Wed, 13 Oct 2010 13:58:53 +0000 (16:58 +0300)]
londiste: trigger-cleanup trigger on londiste.table_info

14 years agoskytools.checker: fix, add repair-apply mode
Marko Kreen [Wed, 13 Oct 2010 13:57:37 +0000 (16:57 +0300)]
skytools.checker: fix, add repair-apply mode

14 years agolondiste: automatic db upgrade
Marko Kreen [Tue, 12 Oct 2010 13:26:43 +0000 (16:26 +0300)]
londiste: automatic db upgrade

this is an experiment...

14 years agolondiste: "on delete cascade" fkeys for tables
Marko Kreen [Tue, 12 Oct 2010 13:26:04 +0000 (16:26 +0300)]
londiste: "on delete cascade" fkeys for tables

this avoids need to write cleanup code

14 years agohack: cascadedconsumer send standby info
Marko Kreen [Tue, 12 Oct 2010 12:50:12 +0000 (15:50 +0300)]
hack: cascadedconsumer send standby info

14 years agolondiste.root_check_seqs: lock node row
Marko Kreen [Tue, 12 Oct 2010 12:32:23 +0000 (15:32 +0300)]
londiste.root_check_seqs: lock node row

14 years agorefresh libusual
Marko Kreen [Tue, 12 Oct 2010 12:16:39 +0000 (15:16 +0300)]
refresh libusual

14 years agodoc: fix make rules
Marko Kreen [Tue, 12 Oct 2010 12:15:57 +0000 (15:15 +0300)]
doc: fix make rules

14 years agotodo update, move existing stuff to NEWS
Marko Kreen [Tue, 12 Oct 2010 12:13:30 +0000 (15:13 +0300)]
todo update, move existing stuff to NEWS

14 years agocascadeadmin: remove obsolete entries from help text
Marko Kreen [Tue, 12 Oct 2010 12:09:20 +0000 (15:09 +0300)]
cascadeadmin: remove obsolete entries from help text

14 years agoskytools.checker: generalize more
Marko Kreen [Tue, 12 Oct 2010 12:08:18 +0000 (15:08 +0300)]
skytools.checker: generalize more

14 years agomove londiste & pgq_node maint into pgq.maint_operations
Marko Kreen [Tue, 12 Oct 2010 11:58:29 +0000 (14:58 +0300)]
move londiste & pgq_node maint into pgq.maint_operations

14 years agoUse different SQL for trigger detection on Postgres 9.0
Marko Kreen [Tue, 12 Oct 2010 11:46:37 +0000 (14:46 +0300)]
Use different SQL for trigger detection on Postgres 9.0

14 years agodrop_node: fix message
Marko Kreen [Tue, 12 Oct 2010 11:40:24 +0000 (14:40 +0300)]
drop_node: fix message

14 years agouse defa sklog
Marko Kreen [Tue, 12 Oct 2010 11:39:26 +0000 (14:39 +0300)]
use defa sklog

14 years agoconfigure: --with-skylog arg to set default value
Marko Kreen [Tue, 12 Oct 2010 11:37:29 +0000 (14:37 +0300)]
configure: --with-skylog arg to set default value

15 years agomore checker
Marko Kreen [Thu, 23 Sep 2010 00:52:16 +0000 (17:52 -0700)]
more checker

15 years agoduplicated repair and compart into checker for crossmover
Asko Oja [Sat, 18 Sep 2010 16:43:23 +0000 (16:43 +0000)]
duplicated repair and compart into checker for crossmover

15 years agonew dbservice module, for pl/python funcs
Marko Kreen [Tue, 28 Sep 2010 21:16:21 +0000 (14:16 -0700)]
new dbservice module, for pl/python funcs

15 years agoskytools: JSON wrapper
Marko Kreen [Thu, 23 Sep 2010 00:52:39 +0000 (17:52 -0700)]
skytools: JSON wrapper

- find working JSON module
- easier kwargs API

15 years agolondiste: helpstr reorg, add --reset and --rewind
Marko Kreen [Fri, 17 Sep 2010 03:12:00 +0000 (06:12 +0300)]
londiste: helpstr reorg, add --reset and --rewind

15 years agoCascadedConsumer: --reset and --rewind commands
Marko Kreen [Fri, 17 Sep 2010 03:11:37 +0000 (06:11 +0300)]
CascadedConsumer: --reset and --rewind commands

15 years agolondiste: use londiste.upgrade.2to3.sql for ugprade instead of hacking around
Marko Kreen [Wed, 15 Sep 2010 13:22:42 +0000 (16:22 +0300)]
londiste: use londiste.upgrade.2to3.sql for ugprade instead of hacking around

15 years agoRefresh libusual
Marko Kreen [Wed, 15 Sep 2010 13:01:35 +0000 (16:01 +0300)]
Refresh libusual

15 years agoskytools: Support psycopg 2.2
Marko Kreen [Wed, 15 Sep 2010 08:24:20 +0000 (11:24 +0300)]
skytools: Support psycopg 2.2

There fileno() has moved from cursor to connection.

15 years agoskytools.config: apply defaults to only main section
Marko Kreen [Tue, 14 Sep 2010 10:17:35 +0000 (13:17 +0300)]
skytools.config: apply defaults to only main section

Otherwise they appear in all sections.

Patch by Martin Otto

15 years agoskytools.config: always check main section
Marko Kreen [Tue, 14 Sep 2010 09:44:31 +0000 (12:44 +0300)]
skytools.config: always check main section

Patch by Martin Otto

15 years agopgqd: remove leftover _exit()
Marko Kreen [Tue, 14 Sep 2010 08:37:55 +0000 (11:37 +0300)]
pgqd: remove leftover _exit()

15 years agopgq/triggers: throw error if SKIP if used in AFTER trigger.
Marko Kreen [Mon, 6 Sep 2010 13:12:18 +0000 (16:12 +0300)]
pgq/triggers: throw error if SKIP if used in AFTER trigger.

15 years agolondiste,pgq: Support & use TRUNCATE trigger by default.
Marko Kreen [Mon, 6 Sep 2010 12:57:13 +0000 (15:57 +0300)]
londiste,pgq: Support & use TRUNCATE trigger by default.

Event format:
 ev_type   - R
 ev_extra1 - table name

Works only on Postgres 8.4+.

Patch by Hannu Krosing.

15 years agosql/ticker: make new logic actually work
Marko Kreen [Fri, 3 Sep 2010 14:08:38 +0000 (17:08 +0300)]
sql/ticker: make new logic actually work

15 years agopgq.maint_operations: don't do rotate_step2, if nothing to rotate
Marko Kreen [Fri, 3 Sep 2010 14:03:16 +0000 (17:03 +0300)]
pgq.maint_operations: don't do rotate_step2, if nothing to rotate

15 years agoconfigure: enable asciidoc/xmlto automatically, if building from .git
Marko Kreen [Fri, 3 Sep 2010 10:53:31 +0000 (13:53 +0300)]
configure: enable asciidoc/xmlto automatically, if building from .git

15 years agopgq/trigger: add when= argument to sqltriga/logutriga.
Marko Kreen [Thu, 2 Sep 2010 14:56:56 +0000 (17:56 +0300)]
pgq/trigger: add when= argument to sqltriga/logutriga.

Similar to WHEN argument for CREATE TRIGGER in 9.0.

15 years agoAdd more junk to gitignore
Marko Kreen [Wed, 1 Sep 2010 14:32:00 +0000 (17:32 +0300)]
Add more junk to gitignore

15 years agorefresh libusual
Marko Kreen [Wed, 1 Sep 2010 14:31:40 +0000 (17:31 +0300)]
refresh libusual

15 years agopgqd: allow customizable maintenance
Marko Kreen [Wed, 1 Sep 2010 14:31:12 +0000 (17:31 +0300)]
pgqd: allow customizable maintenance

15 years agonew function too
Marko Kreen [Wed, 1 Sep 2010 14:29:24 +0000 (17:29 +0300)]
new function too

15 years agopgq: customizable maintenance functions
Marko Kreen [Wed, 1 Sep 2010 14:28:55 +0000 (17:28 +0300)]
pgq: customizable maintenance functions

15 years agoscripts/: conform with new -ini standard.
Marko Kreen [Tue, 31 Aug 2010 08:55:43 +0000 (11:55 +0300)]
scripts/: conform with new -ini standard.

Each script should define only its own config vars.

15 years agopgqd: switches to signal running process
Marko Kreen [Tue, 31 Aug 2010 08:55:08 +0000 (11:55 +0300)]
pgqd: switches to signal running process

15 years agoBublin handler.
Marko Kreen [Fri, 25 Jun 2010 11:31:49 +0000 (14:31 +0300)]
Bublin handler.

  londiste3 cf.ini add-table foo --handler='bublin(key_column)'

15 years agoPluggable per-table handlers for event processing.
Marko Kreen [Fri, 7 May 2010 12:17:53 +0000 (15:17 +0300)]
Pluggable per-table handlers for event processing.

1. Make handler modules known for Londiste:

  [londiste3]
  handler_modules = londiste.bublin, some_other.module

Modules are imported and classes found in __londiste_handlers__
module variable are registered.

2. Generic table setup:

  londiste3 cf.ini add-table foo --handler='handler1' --handler='handler2(arg1, arg2)'

Londiste standard handler is default and always appended to custom plugins.

15 years agopgq.force_tick: work correctly if ticking is not allowed.
Marko Kreen [Mon, 30 Aug 2010 10:48:33 +0000 (13:48 +0300)]
pgq.force_tick: work correctly if ticking is not allowed.

Simply return last tick_id.

15 years agoFix typo.
Marko Kreen [Mon, 2 Aug 2010 14:10:12 +0000 (17:10 +0300)]
Fix typo.