Include howtos written by Hannu Krosing.
Clean up rest of the docs for release.
= SkyTools - tools for PostgreSQL =
-This is a package of tools in use in Skype for replication and
-failover. Also it includes a generic queuing mechanism PgQ and
-utility library for Python scripts.
-
-== Dependencies ==
+== Builing ==
Skytools modules use some other code to run, you need to install the
following dependencies on the system where you want to build and run
=== Runtime dependencies ===
- python
- psycopg2 or psycopg1
- rsync
+ python psycopg2 rsync
== Building from source tarball ==
== Building from GIT ==
- ## fetch git tree
+Fetch git tree:
+
$ git clone git://github.com/markokr/skytools.git
- ## fetch libusual submodule
+Fetch libusual submodule:
+
$ git submodule init
$ git submodule update
- ## generate ./configure script
+Generate ./configure script:
+
$ ./autogen.sh
- ## now build as usual (--with-asciidoc is required when building from GIT)
- $ ./configure --prefix=... --with-asciidoc
+Now build as usual:
+
+ $ ./configure --prefix=...
$ make
== Building a debian package ==
debuild -uc -us -b
tgz: config.mak clean
- $(MAKE) -C doc man html
+ $(MAKE) -C doc man
rm -f source.list
$(PYTHON) setup_skytools.py sdist -t source.cfg -m source.list
-2009-xx-xx - SkyTools 3.0 - ""
-
-* make python modules parallel installable with skytools 2.x:
- - new pkgloader module, (ala pygtk but universal)
- - installed under `skytools-3.0` top dir
- - python modules import skytools via pkgloader:
-
- import pkgloader
- pkgloader.require('skytools', '3.0')
- import skytools
-
-* newadm: rename to qadmin
-* pgq
- - drop failed event concept
- - lazy fetch is now default
- - relaxed event processing for python modules - .tag_done() is not needed
- - get_{queue|consumer}_info() now report quick stats
- - ability to request big batches (10000 rows, 5 min), which combine several actual batches.
- - ability to stay behind on queue.
- - pgq.Consumer new config vars: pgq_batch_collect_events, pgq_batch_collect_interval, pgq_keep_lag
- - pgq triggers: magic fields to set event fields directly:
- `_pgq_ev_type, _pgq_ev_data, _pgq_ev_extra[1-4]`
-* python/skytools:
- - config templates in docstr: `./script.py --ini` shows default conf
- - show connection name with error message
- - use config file name as default job name, `job_name` is thus optional now
- - new querybuilder class
- - .getdict() method for Config
-* pgqd:
- - specifying fixed list of databases in config works
- - periodic detection works
-* cascading
- - fix watermark publishing
- - several log message and level fixes
- - Replace 'switchover'/'failover' with 'takeover'
- - 'status' with dead nodes works
- - consumers last error is stored in db, showed by `londiste status`
-* londiste
- - resync
- - missing
- - --all
- - globbing
- - compare/repair
- - apply sql one-by-one if last batch got error
- - support WHERE for COPY: londiste add --copy-condition=XX
-
-* Simple Python consumer for pgq_coop, where each subconsumer
- is separate process.
-
-* psycopgwrapper: make .server_version always available
-
-* Sleeping while waiting notices from db. It seems good to speed up
- change-provider, switchover etc commands by using LISTEN/NOTIFY
- to wake up daemons.
-
-* londiste/pgq: truncate trigger / event
-
-
-2009-03-13 - SkyTools 2.1.9 - "Six Pack of Luck"
-
- = WalMgr improvements =
-
- * walmgr.py: WAL files purge procedure now pays attention to recovery
- restart points. (%r patch by Omar Kilani, Mark Kirkwood)
-
- * walmgr.py: archive_mode GUC is now set during setup. Followed by an
- optional restart if master_restart_cmd is specified (Mark Kirkwood)
-
- * walmgr.py: PostgreSQL configuration files are backed up to
- "config_backup" and restored to "slave_config_dir". (Mark Kirkwood)
-
- * walmgr.py: Backups taken from slave now generate backup_label and
- history files.
-
- * walmgr.py Configuration files now default to PostgreSQL 8.3
-
- = Fixes =
-
- * londiste copy: Add missing identifier quoting for
- TRUNCATE and ADD CONSTRAINT. (Luc Van Hoeylandt)
-
- * Quote identifiers starting with numeric. (Andrew Dunstan)
-
- * Fix crash with pgq.sqltrigq/pgq.logutriga, which automatically
- detect table structure. Problem was handling table invalidation
- event during internal query.
- (Götz Lange, André Malo)
-
- * Fix 'uninitialized variable' warning and potentially bad code
- in pgq.sqltriga(). (Götz Lange, André Malo)
-
- * skytools._cquoting: Fix crash with Decimal type.
- (May affects users who have compiled psycopg2 to use
- Decimal() instead of float for NUMERIC)
-
- * skytools.DBStruct: The DEFAULT keyword was missing when
- creating table columns with default value. Table creation
- functionality is unused in 2.1.x thus no users should be affected.
-
- * skytools.magic_insert: be more flexible about whats dict.
- In particular, now accept DictRow.
-
- * configure.ac: don't detect xmlto and asciidoc version
- if --with-asciidoc is not used. Otherwise unnecessary error
- message was printed.
-
- * Add few headers for Postgres 8.4 that are not included
- automatically anymore. (Devrim Gündüz)
-
-2008-10-12 - SkyTools 2.1.8 - "Perestroika For Your Data"
-
- = Fixes =
-
- * deb: Make debian package accept skytools-8.3 too.
- * add skylog.ini as installalble file (David Fetter)
- * Londiste:
- - Document the fkeys, triggers, restore-triggers commands.
- - Run ANALYZE after table is copied over.
- - Fix "provider add-seq --all"
- * pgq.SerialConsumer (used by Londiste) - fix position check,
- which got confused when sequence numbers did large jump.
- * PgQ database functions:
- - pgq.maint_rotate_step1() function removed tick #1
- even if consumer was registered on it.
- - pgq triggers: import cache invalidation from HEAD.
- Now pgq.sqltriga() and pgq.logutriga() should be preferable
- to pgq.logtriga().
- - uninstall_pgq.sql: Correct syntax for DROP SCHEMA CASCADE.
- * skytools.DBScript, used by all Python scripts:
- - Don't try to stay running if MemoryError was thrown.
- - Detect stale pidfile and ignore it.
- - Stop hooking SIGTERM anymore. Python signal handling and
- libpq polling did not interact well.
- * scriptmgr:
- - Don't crash on missing pidfile.
- - Fix few typos on scriptmgr (Martin Pihlak)
- * walmgr (Martin Pihlak):
- - improved error messages on startup if no config file specified.
- - walmgr.py stop now also stops syncdaemon
- - pg_auth file is copied to slave as part of archiving. restored during boot.
- - cleanup in remote_walmgr() to always pass config file to slave walmgr.
- - master_backup() now reports exceptions.
-
- = Features =
-
- * londiste -s/-k now kill "copy" process with SIGTERM.
- Previously the process was left running. On startup
- "replay" will check if "copy" process had died before
- finishing and launches one if needed. (Dimitri Fontaine)
- * New skytools.signal_pidfile() function.
- * Londiste: New lock_timeout parameter to limit time
- locks are held on provider. Applies to operations:
- provider add/remove, compare, repair.
- * Londiste: on psycopg2 v2.0.6+ use new .copy_expert() API,
- which does not crash on tables with large number of columns.
-
-2008-06-03 - SkyTools 2.1.7 - "Attack of the Werebugs"
-
- = Fixes =
-
- * Fix pgq trigger compilation with Postgres 8.2 (Marcin Stępnicki)
- * Replace `make` calls with $(MAKE) in Makefiles (Pierre-Emmanuel André)
- * londiste: Fix incompatibility with Python 2.3 (Dimitri Fontaine)
- * walmgr: Fix typo in config symlinking code. (pychecker)
- * bulk_loader: Fix typo in temp table check. (pychecker)
- * Install upgrade .sql files. Otherwise skytools_upgrade.py could
- be used only from source directory.
- * pgq.Consumer: Fix bug in retry/failed event handling. (pychecker)
- * pgq: Fix pgq.maint_retry_events() - it could create double events when
- amount of event to be moved back into main queue was more than 10.
-
- = Features =
-
- * Quoting of table and column names in Londiste and dispatcher scripts.
-
- = Upgrade procedure =
-
- * Database code under pgq and londiste schemas need to be upgraded.
- That can be done on running databases with following command:
-
- $ skytools_upgrade.py "connstr"
-
- Or by applying 'londiste.upgrade.sql' and 'pgq.upgrade.sql' by hand.
- The changes were only in functions, no table sctructure changed.
-
-2008-04-05 - SkyTools 2.1.6 - "Quick Bugfix Release"
-
- Now we have upgrade script, see 'man skytools_upgrade'
- for info how to upgrade database code.
-
- = Fixes =
-
- * Include upgrade sql scripts in .tgz
- * Fix 'londiste provider seqs'
- * Fix 'londiste provider fkeys' in no tables added.
- * Fix "londiste copy" pidfile timing race
- * Fix Solaris build - avoid grep -q / define HAVE_UNSETENV
- * Fix "make debXX" when several Postgres version are installed.
- * New-style AC_OUTPUT usage.
- * Disable manpage creation by default, --with-asciidoc to enable.
- They are still included in .tgz so users should have less problems now.
- * Restore iter-on-values behaviour for rows from curs.fetch*. The attempt
- to make them iter-on-keys seems now misguided, as iter-on-values is already
- used in existing code, and iter-on-keys returns keys in random order.
- * londiste subscriber add: Dont drop triggers on table if --expect-sync is used.
- * londiste copy: drop triggers and fkeys in case "replay" or "subscriber add" was skipped
- * walmgr restore: better detection if old postmaster is running (Charles Duffy)
- * walmgr xrestore: detect the death of parent process
- * walmgr restore: create pg_tblspc - its required for 8.3 (Zoltán Böszörményi)
- * walmgr restore: copy old config files over if exist (Zoltán Böszörményi)
-
- = Features =
-
- * Table name globbing for Londiste commands (Erik Jones)
- * New manpages for scripts (Asko Oja & me)
- * Upgrade script with manpage: scripts/skytools_upgrade.py
- * Add .version() function to pgq_ext & londiste schemas.
- * pgqadm: allow parameters without queue_ prefix in 'config' command.
- * skytools Python module:
- - intern() keys in db_urldecode() to decrease memory usage
- - udp-logger: more fields: hostaddr, service_name
- - udp-logger: dont cache udp socket, seem to hang in some cases
- - DBScript.get_database() allows explicit connect string
- - DBScript allows disabling config file loading
- - magic_insert on dicts allows missing columns, uses NULL
- - new parsing functions:
- - parse_pgarray(), parse_logtriga_sql(), parse_tabbed_table(),
- - parse_statements() - this one is used to split SQL install
- files to separate statements.
- - exists_function() checks both 'public' and 'pg_catalog'
- for unqualified functions names.
- - skytools.quoting: add C implementation for quote_literal, quote_copy,
- quote_bytea_raw, unescape, db_urlencode, db_urldecode. This gives
- 20x speedup for db_urlencode and 50x for db_urldecode on some
- real-life data.
- - unquote_ident(), unquote_literal()
-
-2007-11-19 - SkyTools 2.1.5 - "Enterprise-Grade Duct Tape"
-
- = Big changes =
-
- * Lot of new docs [Dimitri Fontaine, Asko Oja, Marko Kreen]
- * Support for fkey and trigger handling in Londiste. [Erik Jones]
- * Rewrite pgq.insert_event() and log triggers in C, thus SkyTools does
- not depend on PL/Python anymore.
-
- = Small changes =
-
- * pgq+txid: convert to new API appearing in 8.3 /contrib/txid/
- * Support psycopg2, preferring it to psycopg1.
- * Improved bulk_loader, using temp tables exclusively.
- * skytools.config: API change to allow usage without config file.
- * skytools module: quote_ident(), quote_fqident()
- * install .sql files under share/skytools in addition to contrib/
- * pgqadm: also vacuums londiste and pgq_ext tables, if they exist
- * londiste: provider add/remove --all [Hans-Juergen Schoenig]
- * backend modules support 8.3
- * pgq: switch pgq_lazy_fetch=NROWS for pgq.Consumer, which makes it use
- cursor for event fetching, thus allowing larger batches
- * txid: use bsearch() for larger snapshots
-
- = Fixes =
-
- * londiste fkeys: look also at dependers not only dependencies.
- * pgq.consumer: make queue seeking in case of failover more strict.
- * scriptmgr: dont die on user error.
- * pgq: there was still fallout from reorg - 2 missing indexes.
- * Due to historical reasons SerialConsumer (and thus Londiste)
- accessed completed tick table directly, not via functions.
- Make it use functions again.
- * londiste: set client_encoding on subscriber same as on provider
- * londiste: remove tbl should work also if table is already dropped [Dimitri Fontaine]
- * couple walmgr fixed [Martin Pihlak]
-
- = Upgrade procedure for database code =
-
- * PgQ (used on Londiste provider side), table structure, plpgsql functions:
-
- $ psql dbname -f upgrade/final/v2.1.5.pgq_core.sql
-
- * PgQ new insert_event(), written in C:
-
- $ psql dbname -f sql/pgq/lowlevel/pgq_lowlevel.sql
-
- * PgQ new triggers (sqltriga, logtriga, logutriga), written in C:
-
- $ psql dbname -f sql/pgq/triggers/pgq_triggers.sql
-
- * Londiste (both provider and subscriber side)
-
- $ psql dbname -f upgrade/final/v2.1.5.londiste.sql
-
- * pgq_ext:
-
- $ psql dbname -f upgrade/final/v2.1.5.pgq_ext.sql
-
-
-2007-04-16 - SkyTools 2.1.4 - "Sweets from last Christmas"
-
- = Fixes =
-
- * logtriga.c was supposed to survive mismatched column string,
- but the logic were buggy. Thanks go to Dmitriy V'jukov for
- good analysis.
- * Couple of scripts were not converted to new API. Fix it.
- * Quiet a warning in textbuf.c
- * Make configure and Makefiles survive on BSD's where 'make'
- is not GNU make. Thanks to Hans-Juergen Schoening.
-
- = Features =
-
- * Published WalMgr was an old version. Sync with internal code,
- where Martin has done lot of enhancements.
- * Small incompat change in PGQ: add fields to pgq.get_consumer_info()
- return type. Example upgrade procedure:
-
- DROP TYPE pgq.ret_consumer_info cascade;
- \i structure/types.sql
- \i functions/pgq.get_consumer_info.sql
-
- It will show some errors but thats ok. Its annoying but needed
- for the tick_id cleanup in SerialConsumer/Londiste.
-
-2007-04-10 - SkyTools 2.1.3 - "Brown Paper Bag"
-
- Still managed to sneak in a last-minute typo.
-
- * Fix copy-paste error in table_copy.py
- * Remember to bump version in pgq.version()
-
-2007-04-09 - SkyTools 2.1.2 - "Help screen works"
-
- Most fallout from reorg is hopefully cleaned now.
-
- * Dumb bug in ticker wich made it almost non-working,
- except it managed to complete the regression test...
- * Move --skip-truncate switch from 'copy' to 'londiste add'.
- * 'londiste install' also installs plpgsql+plpythonu.
- * SQL installer logs full path to file it found.
- * Change pgq.get_queue_info() FOR loop variable to record
- instead text that was reported to fail, although it did work here.
- * Remember where the SQL files were installed.
-
-2007-04-06 - SkyTools 2.1.1 - "Needs more thrust to fly"
-
- SkyTools got big reorg before release, but with the hoopla
- with the 3 projects at once, it did not get much testing...
- There are some untested areas still, but at least pgq/londiste
- are in better shape now.
-
- * pgqadm: finish conversion...
- * londiste.Syncer:
- - Convert to new API
- - Avoid ticking, parallel ticks are dangerous
- - Bad arg in repair
- * pgq:
- - too aggressive check in register_consumer
- - Algo desc for batch_event_sql
- * Add some knobs to make regtests for londiste pass
- more predictibly.
-
-2007-03-13 - SkyTools 2.1 - "Radioactive Candy"
-
- * Final public release.
+2012-xx-xx - SkyTools 3.0 - ""
+
+ * Cascaded queues
+ * Londiste: Parallel copy
+ * Londiste: EXECUTE
+ * Londiste: handlers
+ * qadmin
+ * pgqd
== Overview ==
-It contains following tools:
+It contains following modules:
=== PgQ ===
Documentation:
-- PgQ ticker daemon (pgqd) usage: link:pgqd.html[]
-- PgQ admin tool (qadm) usage: link:qadmin.html[]
-- PgQ SQL API overview: link:pgq-sql.html[]
-- PgQ SQL reference: http://skytools.projects.postgresql.org/skytools-3.0/pgq/
+- PgQ ticker daemon (pgqd) usage: link:doc/pgqd.html[]
+- PgQ admin tool (qadm) usage: link:doc/qadmin.html[]
+- PgQ SQL API overview: link:doc/pgq-sql.html[]
+- PgQ SQL reference: link:pgq/[]
=== Londiste ===
Documentation:
-- Londiste script usage: doc/londiste.cmdline.txt
+- Londiste script usage: doc/londiste3.txt
(also available as `man 1 londiste`)
-- Londiste configuration: doc/londiste.config.txt
- (also available as `man 5 londiste`)
-- Londiste reference: doc/londiste.ref.txt
+
+- Londiste HOWTOS: doc/howto/
=== walmgr ===
== Source tree contents ==
- doc/
+doc/::
Documentation in asciidoc format. Source for both html
and man pages.
- python/
+python/::
Python modules and primary executables - walmgr, londiste, qadmin, pgqadm.
- python/pgq/
+python/pgq/::
Python framework for PgQ.
- python/londiste/
+python/londiste/::
Londiste replication.
- python/skytools/
+python/skytools/::
Low-level utilities for writing database scripts in Python.
- sql/
+sql/::
Database modules.
- sql/pgq/
+sql/pgq/::
Table definitions and functions for PgQ queueing.
- sql/pgq_node/
+sql/pgq_node/::
Framework for cascaded consuming.
- sql/pgq_coop/
+sql/pgq_coop/::
Functions for cooperative consuming.
- sql/londiste/
+sql/londiste/::
Table definitions and functions for Londiste replication.
- sql/ticker/
+sql/ticker/::
PgQ ticker written in C.
- scripts/
+scripts/::
Python scripts with lesser priority.
- lib/
+lib/::
libusual C libary, for pgqd.
- debian/
+debian/::
Debian packaging. This is for creating private packages,
official Debian packages uses it's own packagin code.
- misc/
+misc/::
Random scripts used for building.
== Upgrade from 2.1 ==
EPYARGS = --no-private --url="http://pgfoundry.org/projects/skytools/" \
--name="Skytools" --html --no-private -v
-HTMLS = README.html INSTALL.html \
+TOPHTML = README.html INSTALL.html index.html
+DOCHTML = \
TODO.html pgq-sql.html pgq-nodupes.html \
- $(SCRIPT_HTMLS) faq.html set.notes.html skytools3.html devnotes.html
+ faq.html set.notes.html skytools3.html devnotes.html pgqd.html \
+ londiste3.html walmgr3.html qadmin.html scriptmgr.html \
+ skytools_upgrade.html queue_mover.html queue_splitter.html \
+ howto/londiste3_cascaded_rep_howto.html \
+ howto/londiste3_merge_howto.html \
+ howto/londiste3_partitioning_howto.html \
+ howto/londiste3_simple_rep_howto.html \
+ howto/setup_walmgr_replication.html
-SCRIPT_TXTS = londiste3.txt walmgr3.txt qadmin.txt scriptmgr.txt skytools_upgrade.txt \
- queue_mover.txt queue_splitter.txt
-SCRIPT_HTMLS = $(SCRIPT_TXTS:.txt=.html)
MAN5 =
MAN1_SFX = scriptmgr.1 skytools_upgrade.1 queue_mover.1 queue_splitter.1
MAN1 = qadmin.1 pgqd.1 walmgr3.1 londiste3.1
-FQHTML = $(addprefix html/doc/, $(HTMLS))
+FQHTML = $(addprefix html/doc/, $(DOCHTML)) $(addprefix html/, $(TOPHTML))
FQMAN1 = $(addprefix man/, $(MAN1))
FQMAN1_SFX = $(addprefix man/, $(MAN1_SFX))
FQMAN5 = $(addprefix man/, $(MAN5))
#AFLAGS = -a linkcss
#AFLAGS = -a stylesheet=extra.css
-all: $(FQMAN) $(FQHTML)
+all: $(FQMAN)
man: $(FQMAN)
html: $(FQHTML)
-install: $(FQMAN) $(FQHTML)
+install: $(FQMAN)
mkdir -p $(DESTDIR)/$(mandir)/man1
mkdir -p $(DESTDIR)/$(mandir)/man5
mkdir -p $(DESTDIR)/$(docdir)
for m in $(FQMAN5); do \
install -m 644 $$m $(DESTDIR)/$(mandir)/man5 || exit 1; \
done
+
+htmlinstall: $(FQHTML)
for h in $(FQHTML); do \
install -m 644 $$h $(DESTDIR)/$(docdir) || exit 1; \
done
-rsync -rtlz ../sql/londiste/docs/html/* $(web)/londiste/
clean:
- rm -rf html *.xml
+ rm -rf html *.xml api
distclean: clean
rm -rf ../sql/pgq/docs/pgq
realclean: distclean
- rm -f *.[15] *.xml *.html
+ rm -rf man *.xml *.html
ifneq ($(ASCIIDOC),no)
ifneq ($(XMLTO),no)
endif
html/doc/%.html: %.txt $(COMMON)
- @mkdir -p html/doc
+ @mkdir -p $(dir $@)
LANG=C cat $< \
| sed -e '/^include/b' -e 's,\([A-Za-z.0-9]*\)[.]txt,link:\1.html[],g' \
| $(ASCIIDOC) $(AFLAGS) -a toc `$(GETATTRS) $<` -o - - \
| sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \
> $@
-html/doc/README.html: ../README
- @mkdir -p html/doc
+html/README.html: ../README
+ @mkdir -p $(dir $@)
cat $< \
| sed -e 's,doc/\([!-~]*\)[.]txt,link:\1.html[],g' \
-e 's,http:[!-~]*,&[],g' \
- | $(ASCIIDOC) $(AFLAGS) -o - - \
+ | $(ASCIIDOC) $(AFLAGS) -a toc -o - - \
+ | sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \
+ > $@
+
+html/INSTALL.html: ../INSTALL
+ @mkdir -p $(dir $@)
+ $(ASCIIDOC) $(AFLAGS) -o - $< \
| sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \
> $@
-html/doc/INSTALL.html: ../INSTALL
- @mkdir -p html
+html/index.html: index.txt
+ @mkdir -p $(dir $@)
$(ASCIIDOC) $(AFLAGS) -o - $< \
| sed -e "/^[/][*] Workarounds/r ../misc/extra.css" \
> $@
endif
web: $(FQHTMLS)
- rsync -avz $(FQHTML) $(web)/doc/
+ rsync -avz html/* $(web)/
Gut feeling about priorities:
High::
- Needer for wider deployment / 3.0-final.
+ Needed soon.
Medium::
Good if done, but can be postponed.
Low::
Interesting idea, but OK if not done.
-== High Priority ==
+== Medium Priority ==
+
+* tests: takeover testing
+ - wal behind
+ - wal ahead
+ - branch behind
* londiste takeover: check if all tables exist and are in sync.
Inform user. Should the takeover stop if problems?
How can such state be checked on-the-fly?
Perhaps `londiste missing` should show in-copy tables.
-* docs:
- - londiste manpage
- - qadmin manpage
- - pgqd manpage
-
* cascade takeover: wal failover queue sync. WAL-failover can be
behind/ahead from regular replication with partial batch. Need
to look up-batched events in wal-slave and full-batches on branches
. Replay rest of batches fully
. Promote to root
-* tests: takeover testing
- - wal behind
- - wal ahead
- - branch behind
-
-== Medium Priority ==
+* Load public connect string for local node:
+ - from .ini (db=, public_db=?)
+ - with query sql
-* Load public connect string from .ini
* Load provider connect string:
- from ini (not good)
- describe connstr + query to look up node names?
- skytools.DBStruct
- londiste handlers
-* cascade watermark limit nodes. A way to avoid affecting root
- with lagging downstream nodes. Need to tag some nodes to coordinate
- watermark with each other and not send it upstream to root.
-
-* automatic sql upgrade mechanism - check version in db, apply .sql
- if contains newer version.
-
-* integrate skytools.checker. It is generic 'check/repair' script
- that can also automatically apply fixes. Should rebase londiste
- check/repair on that.
-
* londiste add-table: automatic serial handling, --noserial switch? Currently,
`--create-full` does not create sequence on target, even if source
table was created with `serial` column. It does associate column
* qadmin: merge cascade commands (medium) - may need api redesign
to avoid duplicating pgq.cascade code?
-* dbscript: configurable error timeout (currently it's hardwired to 20s)
-
* londiste replay: when buffering queries, check their size. Current
buffering is by count - flushed if 200 events have been collected.
That does not take account that some rows can be very large.
generally slow (GP, bad network, etc). Perhaps we can use
psycopg2 async mode for that. Or threading.
-* developer docs for:
- - DBScript, pgq.Consumer, pgq.CascadedConsumer?
- - Update pgq/sql docs for current pgq.
-
== Low Priority ==
* dbscript: switch (-q) for silence for cron/init scripts.
It is bunch of database management tools we use
and various frameworks / modules they depend on.
-Main components are:
-
-==== Python scripts ====
-
-Main tools:
-
- walmgr:: walshipping manager
- londiste:: replication on top of pgq
- pgqadm:: pgq administration and maintenance
- setadm:: cascaded pgq administration
-
-Special scripts:
-
- bulk_loader
- cube_dispatcher
- table_dispatcher
-
-Queue copy
-
- queue_mover:: copy queue contents from one db to another
- queue_splitter:: copy queue contents to another db splitting it into several queues
-
-Operate bunch of scripts together
-
- scriptmgr
-
-==== Python modules ====
-
- pgq
- skytools
- londiste
-
-==== SQL modules ====
-
- londiste
- pgq
- pgq_node
- pgq_ext
-
-=== Where is the code located? ===
-
-Code layout:
-
- debian/
- doc/
- python/bin/
- python/londiste/ - Londiste python modules
- python/modules/ - C extension modules for Python (string handling)
- python/pgq/ - pgq and cascaded pgq python modules
- python/skytools/ - python modules for generic database scripting
- scripts/ - Special purpose python scripts (python)
- sql/londiste/ - database code for londiste (plpgsql)
- sql/pgq/ - PgQ database code (C + plpgsql)
- sql/pgq_ext/ - PgQ event/batch tracking on remote database (plpgsql)
- sql/pgq_node/ - cascaded pgq support (plpgsql)
- sql/txid/ - Obsolete txid code for Postgres 8.2 and below
+Main components are `pgq` SQL module which implements generic
+queue in database, Python module for writing consumers for it,
+Londiste replication on top of them and walmgr for setting up
+WAL-based standby servers.
== PgQ - The generic queue ==
=== Dump-restore ===
+Database which contains `pgq` schema can be dumped and restored with
+`pg_dump`, but extra steps must be taken because PgQ tables
+contains transaction id-s and snapsnot which are extracted from
+Postgres code. PgQ depends on transaction id values going always
+higher. Thus restoring database in new Postgres clusten will break
+PgQ if that cluster has txids smaller than they were in old cluster.
+
+Postgres interally uses rolling 32-bit txids which on export are
+combined with 'txid epoch', which simply is count how many times
+the 32-bit txid has already cycled. Thus the way to make sure
+new cluster has highed txids than old one is to set the epoch higher
+in new cluster than it was in old cluster.
+
+To see epoch in old cluster in running database:
+
+ SELECT (txid_current() >> 32) as epoch;
+ epoch
+ -----
+ 1
+
+To see epoch on shut down database:
+
+ $ pg_resetxlog -n $DATADIR
+ ...
+ Latest checkpoint's NextXID: 0/3938 (epoch -> 0)
+ ...
+
+To set epoch in new cluster - it must be shut down first:
+
+ $ pg_resetxlog -e $NEWEPOCH $DATADIR
+
== Londiste - The replication tool ==
=== What type of replication it does? ===
--- /dev/null
+
+= How to Set up Cascaded Replication in londiste =
+
+== Basic cluster setup ==
+
+Basic configuration for cascaded replication setup.
+
+The configuration file for ticker (pgqd) restricting ticker to only the 5
+replicated databases:
+
+----
+$ cat conf/pgqd.ini
+[pgqd]
+database_list = db1,db2,db3,db4,db5
+logfile = log/pgqd.log
+pidfile = pid/pgqd.pid
+----
+
+The ini files for databases are are created similar to the one below, only
+the "db1" part changes
+
+----
+$ cat conf/londiste_db1.ini
+[londiste3]
+job_name = londiste_db1
+db = dbname=db1
+queue_name = replika
+logfile = log/%(job_name)s.log
+pidfile = pid/%(job_name)s.pid
+
+pgq_autocommit = 1
+pgq_lazy_fetch = 0
+----
+
+After creating the ini files we are ready to install londiste3 and initialize nodes
+
+----
+$ londiste3 -q conf/londiste_db1.ini create-root node1 dbname=db1
+$ londiste3 -q conf/londiste_db2.ini create-branch node2 dbname=db2 --provider=dbname=db1
+$ londiste3 -q conf/londiste_db3.ini create-branch node3 dbname=db3 --provider=dbname=db1
+$ londiste3 -q conf/londiste_db4.ini create-branch node4 dbname=db4 --provider=dbname=db2
+$ londiste3 -q conf/londiste_db5.ini create-branch node5 dbname=db5 --provider=dbname=db3
+----
+
+Now that schemas are installed, we can start the ticker
+
+----
+$ pgqd -q -d conf/pgqd.ini
+----
+
+To see the topology of replication, you can run the status command against any node:
+
+----
+$ londiste3 -q conf/londiste_db4.ini status
+Queue: replika Local node: node4
+
+node1 (root)
+ | Tables: 0/0/0
+ | Lag: 4s, Tick: 2, NOT UPTODATE
+ +--node3 (branch)
+ | Tables: 0/0/0
+ | Lag: 12s, Tick: 1, NOT UPTODATE
+ +--node2 (branch)
+ | Tables: 0/0/0
+ | Lag: 12s, Tick: 1, NOT UPTODATE
+ +--node4 (branch)
+ Tables: 0/0/0
+ Lag: 11s, Tick: 1, NOT UPTODATE
+----
+
+And you need a londiste worker process on each node to actually carry out
+the actions
+
+----
+$ londiste3 -q -d conf/londiste_db1.ini worker
+$ londiste3 -q -d conf/londiste_db2.ini worker
+$ londiste3 -q -d conf/londiste_db3.ini worker
+$ londiste3 -q -d conf/londiste_db4.ini worker
+$ londiste3 -q -d conf/londiste_db5.ini worker
+----
+
+== Adding tables and data ==
+
+Now let's play with data.
+
+Create table on root node and fill couple of rows
+
+----
+$ psql -d db1 -c create table mytable (id serial primary key, data text)
+$ psql -d db1 -c insert into mytable (data) values ('row1')
+$ psql -d db1 -c insert into mytable (data) values ('row2')
+$ psql -d db1 -c insert into mytable (data) values ('row3')
+$ psql -d db1 -c insert into mytable (data) values ('row4')
+----
+
+Creat some load on table
+
+----
+$ ./loadgen.py -d conf/gen1.ini
+----
+
+Register table on root node
+
+----
+$ londiste3 -q conf/londiste_db1.ini add-table mytable
+$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq
+----
+
+Register table on other node with creation
+
+----
+$ psql -d db2 -c create sequence mytable_id_seq
+CREATE SEQUENCE
+$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq
+$ londiste3 -q conf/londiste_db2.ini add-table mytable --create-full
+$ psql -d db3 -c create sequence mytable_id_seq
+CREATE SEQUENCE
+$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq
+$ londiste3 -q conf/londiste_db3.ini add-table mytable --create-full
+$ psql -d db4 -c create sequence mytable_id_seq
+CREATE SEQUENCE
+$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq
+$ londiste3 -q conf/londiste_db4.ini add-table mytable --create-full
+$ psql -d db5 -c create sequence mytable_id_seq
+CREATE SEQUENCE
+$ londiste3 -q conf/londiste_db1.ini add-seq mytable_id_seq
+$ londiste3 -q conf/londiste_db5.ini add-table mytable --create-full
+----
+
+
+== Change topology ==
+
+The main advantage of skytools3 cascaded replication is how easy it is to
+change the replication topology.
+
+
+----
+$ londiste3 -q conf/londiste_db4.ini change-provider --provider=node2
+$ londiste3 -q conf/londiste_db4.ini status
+Queue: replika Local node: node4
+
+node1 (root)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 57
+ +--node2 (branch)
+ | | Tables: 1/0/0
+ | | Lag: 1s, Tick: 57
+ | +--node4 (branch)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 57
+ +--node3 (branch)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 57
+ +--node5 (branch)
+ Tables: 1/0/0
+ Lag: 7s, Tick: 53
+ ERR: londiste_db5: duplicate key value violates unique constraint "mytable_pkey"
+----
+
+Now let's move it to node 3
+
+----
+$ londiste3 -q conf/londiste_db4.ini change-provider --provider=node3
+$ londiste3 -q conf/londiste_db4.ini status
+Queue: replika Local node: node4
+
+node1 (root)
+ | Tables: 1/0/0
+ | Lag: 0s, Tick: 59
+ +--node2 (branch)
+ | Tables: 1/0/0
+ | Lag: 3s, Tick: 59
+ +--node3 (branch)
+ | Tables: 1/0/0
+ | Lag: 3s, Tick: 59
+ +--node4 (branch)
+ | Tables: 1/0/0
+ | Lag: 3s, Tick: 58
+ +--node5 (branch)
+ Tables: 1/0/0
+ Lag: 12s, Tick: 53
+----
+
+
+
+----
+$ londiste3 -q conf/londiste_db5.ini change-provider --provider=node2
+$ londiste3 -q conf/londiste_db1.ini status
+Queue: replika Local node: node1
+
+node1 (root)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 64
+ +--node2 (branch)
+ | | Tables: 1/0/0
+ | | Lag: 1s, Tick: 64
+ | +--node5 (branch)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 64
+ +--node3 (branch)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 64
+ +--node4 (branch)
+ Tables: 1/0/0
+ Lag: 1s, Tick: 64
+----
+
+The topology change can also be accomplished from the "other" end,
+using the `takeover` command:
+
+----
+$ londiste3 -q conf/londiste_db3.ini takeover node2
+$ londiste3 -q conf/londiste_db2.ini status
+Queue: replika Local node: node2
+
+node1 (root)
+ | Tables: 1/0/0
+ | Lag: 0s, Tick: 66
+ +--node3 (branch)
+ | Tables: 1/0/0
+ | Lag: 0s, Tick: 66
+ +--node4 (branch)
+ | Tables: 1/0/0
+ | Lag: 3s, Tick: 66
+ +--node2 (branch)
+ | Tables: 1/0/0
+ | Lag: 0s, Tick: 66
+ +--node5 (branch)
+ Tables: 1/0/0
+ Lag: 3s, Tick: 65
+----
+
+The takeover command is in fact the only way to change the root node
+
+----
+$ londiste3 -q conf/londiste_db2.ini takeover node1
+$ londiste3 -q conf/londiste_db2.ini status
+Queue: replika Local node: node2
+
+node2 (root)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 72
+ +--node5 (branch)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 72
+ +--node1 (branch)
+ | Tables: 1/0/0
+ | Lag: 1s, Tick: 71
+ +--node3 (branch)
+ | Tables: 1/0/0
+ | Lag: 3s, Tick: 71
+ +--node4 (branch)
+ Tables: 1/0/0
+ Lag: 3s, Tick: 71
+----
+
+That's it!
+
+
--- /dev/null
+
+= How To Set Up "Merge" Replication of the Same Table from Multiple Partitions =
+
+== Introduction ==
+
+In this howto we will set up a replication scheme , where data is collected
+from multiple "partition" databases into a single table on "full" database.
+
+This situation is common when using PL/Proxy or other similar partitioning
+solution for OLTP, but still wanting the data in one table for Data Warehousing.
+
+Here we will demonstrate the simplest possible setup with 2 partition databases
+(`part1` and `part2`) replicating their data into one full database `full1`.
+
+=== Setting the nodes ===
+
+==== Partition databases ====
+
+On both partition databases you need to set up a londiste3 "root" node to
+replicate from. We will use the the following .ini files for that
+
+for database part1 `conf/l3_part1_q_part1.ini`:
+
+----
+[londiste3]
+job_name = l3_part1_q_part1
+db = dbname=part1
+queue_name = l3_part1_q
+logfile = log/%(job_name)s.log
+pidfile = pid/%(job_name)s.pid
+----
+
+and for part2 `conf/l3_part2_q_part2.ini`:
+
+----
+[londiste3]
+job_name = l3_part2_q_part2
+db = dbname=part2
+queue_name = l3_part2_q
+logfile = log/%(job_name)s.log
+pidfile = pid/%(job_name)s.pid
+----
+
+These ini files are then used for setting up the nodes and adding tables to root node
+
+Set up the root nodes on part1 and part2
+
+----
+$ londiste3 -v conf/l3_part1_q_part1.ini create-root part1_root dbname=part1
+$ londiste3 -v conf/l3_part2_q_part2.ini create-root part2_root dbname=part2
+----
+
+==== Full database ====
+
+On the full database, which will hold data from both partitions you need to
+set up two londiste nodes, one for each of the partition nodes.
+These will act as the receiving nodes to replicate to.
+
+These look very similar and differ only in queue name
+
+File `conf/l3_part1_q_full1.ini`:
+----
+[londiste3]
+job_name = l3_part1_q_full1
+db = dbname=full1
+queue_name = l3_part1_q
+logfile = log/%(job_name)s.log
+pidfile = pid/%(job_name)s.pid
+----
+
+File `conf/l3_part2_q_full1.ini`:
+----
+[londiste3]
+job_name = l3_part2_q_full1
+db = dbname=full1
+queue_name = l3_part2_q
+logfile = log/%(job_name)s.log
+pidfile = pid/%(job_name)s.pid
+----
+
+These are first used to set up the leaf nodes:
+
+----
+$ londiste3 -v conf/l3_part1_q_full1.ini create-leaf merge_part1_full1 dbname=full1 --provider=dbname=part1
+$ londiste3 -v conf/l3_part2_q_full1.ini create-leaf merge_part2_full1 dbname=full1 --provider=dbname=part2
+----
+
+and later also for launching the replication worker daemons
+
+But before launching the workers you need to start the pgqd or the "ticker daemon"
+
+----
+$ pgqd -v -d conf/pgqd.ini
+----
+
+the `conf/pgqd.ini` file for the command above looks like this:
+
+----
+[pgqd]
+database_list = part1,part2,full1
+logfile = log/pgqd.log
+pidfile = pid/pgqd.pid
+----
+
+Now that the ticker is running, it's time to launch londiste3 workers which will
+do tha actual replication:
+
+----
+$ londiste3 -v -d conf/l3_part1_q_full1.ini worker
+$ londiste3 -v -d conf/l3_part2_q_full1.ini worker
+----
+
+=== Setting up the tables ===
+
+in order to have something to replicate, we need some tables, so let's create
+them on partition nodes:
+
+----
+$ psql -d "part1" -c "create table mydata (id int4 primary key, data text)"
+$ psql -d "part2" -c "create table mydata (id int4 primary key, data text)"
+----
+
+and then add them to set of replicated tables on the root node
+
+----
+$ londiste3 -v conf/l3_part1_q_part1.ini add-table mydata
+$ londiste3 -v conf/l3_part2_q_part2.ini add-table mydata
+----
+
+Now we need some data in these tables, as replicating empty tables is no fun
+
+----
+$ psql -d "part1" -c "insert into mydata values (1, 'part1')"
+$ psql -d "part2" -c "insert into mydata values (2, 'part2')"
+----
+
+We can check that the tables are actually registered for replication in londiste:
+
+----
+$ psql -d "full1" -c "select * from londiste.table_info order by queue_name"
+ nr | queue_name | table_name | local | merge_state | custom_snapshot | dropped_ddl | table_attrs | dest_table
+----+------------+---------------+-------+-------------+-----------------+-------------+-------------+------------
+ 1 | l3_part1_q | public.mydata | f | | | | |
+ 2 | l3_part2_q | public.mydata | f | | | | |
+(2 rows)
+----
+
+Now lets subscribe them on full database. As the table is not yet created on
+full1, we specify `--create` so londiste creates the table on leaf node based on
+structure that is on root. The switch `--merge-all` tells londiste to add the table to all
+queues which have it on root side, not just the one from the .ini file
+
+----
+$ londiste3 -v conf/l3_part1_q_full1.ini add-table mydata --create --merge-all
+----
+
+And yes, there it is, subscribed from both queues:
+
+----
+$ psql -d "full1" -c "select * from londiste.table_info order by queue_name"
+ nr | queue_name | table_name | local | merge_state | custom_snapshot | dropped_ddl | table_attrs | dest_table
+----+------------+---------------+-------+-------------+-----------------+-------------+-------------+------------
+ 1 | l3_part1_q | public.mydata | t | | | | |
+ 2 | l3_part2_q | public.mydata | t | | | | |
+(2 rows)
+----
+
+Now we can put more data to partition tables:
+
+----
+$ psql -d "part1" -c "insert into mydata values (4 + 1, 'part1')"
+$ psql -d "part2" -c "insert into mydata values (4 + 2, 'part2')"
+----
+
+Wait a few seconds:
+
+----
+$ sleep 10
+----
+
+And check that the data has indeed appeared on full database:
+
+----
+$ psql -d "full1" -c "select * from mydata order by id"
+ id | data
+----+-------
+ 1 | part1
+ 2 | part2
+ 5 | part1
+ 6 | part2
+(4 rows)
+----
+
+the rows fith ids 1 and 2 where replicated during initial copy, the ones with
+5 and 6 were captured by triggers into event log on partition database and then
+replicated to full1 using the standard replication process.
+
+=== checking subscription ===
+
+Just to check if we really did achieve what we wanted, we see which tables
+are present and fully sobscribed ('ok')
+
+----
+$ psql -d "full1" -c "select * from londiste.table_info order by queue_name"
+ nr | queue_name | table_name | local | merge_state | custom_snapshot | dropped_ddl | table_attrs | dest_table
+----+------------+---------------+-------+-------------+-----------------+-------------+-------------+------------
+ 1 | l3_part1_q | public.mydata | t | ok | | | |
+ 2 | l3_part2_q | public.mydata | t | ok | | | |
+(2 rows)
+----
+
+ok, here we have the table public.mydata subscribed from 2 queues and it's
+merge_state is 'ok', meaning the initial copy process has been successfull
+
+
+That's it , we have successfully set up replication from two partition
+databases to one single full database.
--- /dev/null
+= Setting up Londiste3 replication to partitions =
+
+== Introduction ==
+
+This sample shows how to use Londiste `part` handler module to split
+one big table between two databases.
+
+The target databases will have `partconf` schema which is usually
+used to drive PL/Proxy. Here it is used simply to provide
+configuration to `part` handler.
+
+== Setting up the Root Database ==
+
+=== Create database ===
+
+Run the following SQL:
+----
+CREATE DATABASE l3part_root;
+----
+
+=== Set up pgbench schema ===
+
+In this HowTo we are using pgbench for setting up the schema,
+populating it with sampledata and later running SQL loads to be replicated.
+
+
+This command will create pgbanch tables and fill them with data:
+----
+/usr/lib/postgresql/9.1/bin/pgbench -i -s 2 -F 80 l3part_root
+----
+
+=== Change primary key columns to text ===
+
+Standard pgbench schema has integer primary key columns for its tables.
+The standard partitioning handler is able to partition only text columns,
+so we change the primary key column types to text
+
+
+----
+alter table pgbench_accounts alter column aid type text;
+alter table pgbench_branches alter column bid type text;
+alter table pgbench_tellers alter column tid type text;
+----
+
+Now create the partition databases to replicate to.
+Each of these will get roughly half of the individual data rows.
+
+
+Create database for partition #0:
+----
+createdb l3part_part0;
+----
+
+And create a partition configuration table in this database
+----
+
+CREATE SCHEMA partconf;
+CREATE TABLE partconf.conf (
+ part_nr integer,
+ max_part integer,
+ db_code bigint,
+ is_primary boolean,
+ max_slot integer,
+ cluster_name text
+);
+insert into partconf.conf(part_nr, max_part) values(0,1);
+
+----
+
+
+Create database for partition #1:
+----
+CREATE DATABASE l3part_part1;
+----
+
+
+----
+
+CREATE SCHEMA partconf;
+CREATE TABLE partconf.conf (
+ part_nr integer,
+ max_part integer,
+ db_code bigint,
+ is_primary boolean,
+ max_slot integer,
+ cluster_name text
+);
+insert into partconf.conf(part_nr, max_part) values(1,1);
+----
+
+Next create configuration files file for root node and both partitions
+
+st3partsplit/st3_l3part_root.ini
+----
+[londiste3]
+job_name = st3_l3part_root
+db = dbname=l3part_root
+queue_name = replika
+logfile = st3partsplit/log/st3_l3part_root.log
+pidfile = st3partsplit/pid/st3_l3part_root.pid
+----
+
+st3partsplit/st3_l3part_part0.ini
+----
+[londiste3]
+job_name = st3_l3part_part0
+db = dbname=l3part_part0
+queue_name = replika
+logfile = st3partsplit/log/st3_l3part_part0.log
+pidfile = st3partsplit/pid/st3_l3part_part0.pid
+----
+
+st3partsplit/st3_l3part_part1.ini
+----
+[londiste3]
+job_name = st3_l3part_part1
+db = dbname=l3part_part1
+queue_name = replika
+logfile = st3partsplit/log/st3_l3part_part1.log
+pidfile = st3partsplit/pid/st3_l3part_part1.pid
+----
+
+Then create root node:
+
+----
+londiste3 st3partsplit/st3_l3part_root.ini create-root node1 dbname=l3part_root
+----
+
+And start the worker on root:
+----
+londiste3 -d st3partsplit/st3_l3part_root.ini worker
+----
+
+
+
+And create leaf nodes and start the workers on partitions :
+
+----
+londiste3 st3partsplit/st3_l3part_part0.ini create-leaf node2_0 dbname=l3part_part0 --provider=dbname=l3part_root
+londiste3 -d st3partsplit/st3_l3part_part0.ini worker
+----
+
+Second node:
+----
+londiste3 st3partsplit/st3_l3part_part1.ini create-leaf node2_1 dbname=l3part_part1 --provider=dbname=l3part_root
+londiste3 -d st3partsplit/st3_l3part_part1.ini worker
+----
+
+
+
+Create config file st3partsplit/pgqd.ini for `pgqd` ("the ticker")
+----
+[pgqd]
+
+logfile = st3partsplit/log/pgqd.log
+pidfile = st3partsplit/pid/pgqd.pid
+
+----
+
+
+Start the ticker process :
+----
+pgqd -d st3partsplit/pgqd.ini
+----
+
+
+
+Now add the replicated tables to root and partitions.
+Here we use `--create` switch to add them to partition,
+which means Londiste takes schema from root node and
+creates tables on target nodes automatically.
+
+The `--handler=part` tells londiste to use the `part` handler for replication,
+the `--handler-arg=key=*id` specifyies which key field to partition on.
+
+
+
+Run command the following commands :
+----
+londiste3 st3partsplit/st3_l3part_root.ini add-table pgbench_accounts --handler=part --handler-arg=key=aid
+londiste3 st3partsplit/st3_l3part_part0.ini add-table pgbench_accounts --create --handler=part --handler-arg=key=aid
+londiste3 st3partsplit/st3_l3part_part1.ini add-table pgbench_accounts --create --handler=part --handler-arg=key=aid
+
+londiste3 st3partsplit/st3_l3part_root.ini add-table pgbench_branches --handler=part --handler-arg=key=bid
+londiste3 st3partsplit/st3_l3part_part0.ini add-table pgbench_branches --create --handler=part --handler-arg=key=bid
+londiste3 st3partsplit/st3_l3part_part1.ini add-table pgbench_branches --create --handler=part --handler-arg=key=bid
+
+londiste3 st3partsplit/st3_l3part_root.ini add-table pgbench_tellers --handler=part --handler-arg=key=tid
+londiste3 st3partsplit/st3_l3part_part0.ini add-table pgbench_tellers --create --handler=part --handler-arg=key=tid
+londiste3 st3partsplit/st3_l3part_part1.ini add-table pgbench_tellers --create --handler=part --handler-arg=key=tid
+----
+
+The following command will run pgbench full speed with 5 parallel
+database connections for 10 seconds.
+
+----
+/usr/lib/postgresql/9.1/bin/pgbench -T 10 -c 5 l3part_root
+----
+
+After this is done, you can check that the tables on both sides hanve the same data with
+
+----
+londiste3 st3partsplit/st3_l3part_part0.ini compare
+londiste3 st3partsplit/st3_l3part_part0.ini compare
+----
+
+Except of course that they dont - each partition will only have roughly half
+the data from the root. But the row counts and checksums of the partitions
+should both add up to the numbers on the master.
+
--- /dev/null
+= Setting up simple Londiste3 replication =
+
+== Introduction ==
+
+This sample does the following actions:
+
+ * sets up te databases
+ - creates a database 'l3simple_db1', which will be master
+ - populates this with pgbench schema and data
+ - adds primary and foreign keys to make the db more realistic
+ - makes a copy of the database ('l3simple_db2') to be used as slave
+ * sets up replication from the master to slave database
+ - creates the root node on 'l3simple_db1'
+ - creates a leaf node on 'l3simple_db2'
+ - starts the ticker daemon
+ - adds all tables to replication set on both databases
+ - waits the replication to complete
+
+It also runs pgbench a to test that the replication actually happens and works properly.
+
+
+== Set up schema for root database ==
+
+=== Create database ===
+
+Run the following SQL:
+----
+CREATE DATABASE l3simple_db1;
+----
+
+=== Set up pgbench schema ===
+
+In this HowTo we are using pgbench for setting up the schema,
+populating it with sampledata and later running SQL loads to be replicated.
+
+
+Run command :
+----
+/usr/lib/postgresql/9.1/bin/pgbench -i -s 2 -F 80 l3simple_db1
+----
+
+=== And add primary and foreign keys needed for replication ===
+
+Standard pgbench schema lacks Primary Key on the history table.
+As Londiste need primary keys we add one. We also add Foreign Keys between tables,
+as these may help detect possible replication failures.
+
+create file /tmp/prepare_pgbenchdb_for_londiste.sql with the following ...
+----
+
+-- add primary key to history table
+ALTER TABLE pgbench_history ADD COLUMN hid SERIAL PRIMARY KEY;
+
+-- add foreign keys
+ALTER TABLE pgbench_tellers ADD CONSTRAINT pgbench_tellers_branches_fk FOREIGN KEY(bid) REFERENCES pgbench_branches;
+ALTER TABLE pgbench_accounts ADD CONSTRAINT pgbench_accounts_branches_fk FOREIGN KEY(bid) REFERENCES pgbench_branches;
+ALTER TABLE pgbench_history ADD CONSTRAINT pgbench_history_branches_fk FOREIGN KEY(bid) REFERENCES pgbench_branches;
+ALTER TABLE pgbench_history ADD CONSTRAINT pgbench_history_tellers_fk FOREIGN KEY(tid) REFERENCES pgbench_tellers;
+ALTER TABLE pgbench_history ADD CONSTRAINT pgbench_history_accounts_fk FOREIGN KEY(aid) REFERENCES pgbench_accounts;
+
+----
+
+then load it into database:
+
+----
+psql l3simple_db1 -f /tmp/prepare_pgbenchdb_for_londiste.sql
+----
+
+
+Create and populate target database:
+
+----
+psql -d postgres -c "CREATE DATABASE l3simple_db2;"
+pg_dump -s l3simple_db1 | psql l3simple_db2
+----
+
+Create configuration file st3simple/st3_l3simple_db1.ini
+
+----
+[londiste3]
+job_name = st3_l3simple_db1
+db = dbname=l3simple_db1
+queue_name = replika
+logfile = st3simple/log/st3_l3simple_db1.log
+pidfile = st3simple/pid/st3_l3simple_db1.pid
+----
+
+
+Create Londiste root node:
+
+----
+londiste3 st3simple/st3_l3simple_db1.ini create-root node1 dbname=l3simple_db1
+----
+
+Run worker daemon for root node:
+
+----
+londiste3 -d st3simple/st3_l3simple_db1.ini worker
+----
+
+Create configuration file st3simple/st3_l3simple_db2.ini
+for worker daemon on target node:
+
+----
+[londiste3]
+job_name = st3_l3simple_db2
+db = dbname=l3simple_db2
+queue_name = replika
+logfile = st3simple/log/st3_l3simple_db2.log
+pidfile = st3simple/pid/st3_l3simple_db2.pid
+----
+
+Initialize node in target database:
+
+----
+londiste3 st3simple/st3_l3simple_db2.ini create-leaf node2 dbname=l3simple_db2 --provider=dbname=l3simple_db1
+----
+
+Launch worker daemon for target database.
+
+----
+londiste3 -d st3simple/st3_l3simple_db2.ini worker
+----
+
+Create config file `st3simple/pgqd.ini`
+for PgQ ticker daemon:
+
+----
+[pgqd]
+
+logfile = st3simple/log/pgqd.log
+pidfile = st3simple/pid/pgqd.pid
+----
+
+Launch ticker daemon:
+
+----
+pgqd -d st3simple/pgqd.ini
+----
+
+To generate some data traffic on the master database while replicating,
+run the following command in background:
+
+----
+/usr/lib/postgresql/9.1/bin/pgbench -T 120 -c 5 l3simple_db1 -f /tmp/throttled.pgbench
+----
+
+the /tmp/throttled.pgbench contains the standard pgbencg workload, except that
+ther are random length waits between commands.
+
+Now add all the tables to replication, first on root node and then on the leaf
+
+Run command :
+----
+londiste3 st3simple/st3_l3simple_db1.ini add-table --all
+londiste3 st3simple/st3_l3simple_db2.ini add-table --all
+----
+
+
+== Checking and testing ==
+
+To test our newly set up replication
+
+The following command will run pgbench full speed with 5 parallel
+database connections generating database traffic
+for 10 seconds
+
+----
+/usr/lib/postgresql/9.1/bin/pgbench -T 10 -c 5 l3simple_db2
+----
+
+After this is done, you can check that the tables on both sides have the same data with
+
+----
+londiste3 st3simple/st3_l3simple_db2.ini compare
+----
+
+Compare command will establish the same logical point in time on provider and
+subscriber nodes and then count and checksum the rows on both sides.
+
+The result will look like this
+
+----
+2011-12-25 08:24:42,138 29189 INFO Locking public.pgbench_accounts
+2011-12-25 08:24:42,147 29189 INFO Syncing public.pgbench_accounts
+2011-12-25 08:24:45,233 29189 INFO Counting public.pgbench_accounts
+2011-12-25 08:24:46,154 29189 INFO srcdb: 200000 rows, checksum=3864719477
+2011-12-25 08:24:46,953 29189 INFO dstdb: 200000 rows, checksum=3864719477
+2011-12-25 08:24:46,961 29189 INFO Locking public.pgbench_branches
+2011-12-25 08:24:46,965 29189 INFO Syncing public.pgbench_branches
+2011-12-25 08:24:50,528 29189 INFO Counting public.pgbench_branches
+2011-12-25 08:24:50,549 29189 INFO srcdb: 2 rows, checksum=-82377983
+2011-12-25 08:24:50,556 29189 INFO dstdb: 2 rows, checksum=-82377983
+2011-12-25 08:24:50,568 29189 INFO Locking public.pgbench_history
+2011-12-25 08:24:50,572 29189 INFO Syncing public.pgbench_history
+2011-12-25 08:24:53,641 29189 INFO Counting public.pgbench_history
+2011-12-25 08:24:53,660 29189 INFO srcdb: 1310 rows, checksum=-34927328558
+2011-12-25 08:24:53,670 29189 INFO dstdb: 1310 rows, checksum=-34927328558
+2011-12-25 08:24:53,675 29189 INFO Locking public.pgbench_tellers
+2011-12-25 08:24:53,677 29189 INFO Syncing public.pgbench_tellers
+2011-12-25 08:24:56,733 29189 INFO Counting public.pgbench_tellers
+2011-12-25 08:24:56,737 29189 INFO srcdb: 20 rows, checksum=518235101
+2011-12-25 08:24:56,740 29189 INFO dstdb: 20 rows, checksum=518235101
+----
+
+the "checksum" is computed by adding up hashtext() sums for all database rows.
+
+
+== Done ==
+
+The setup of simple 2 node cluster is done.
+
--- /dev/null
+= Setting up backup and streaming replication with walmgr3 =
+
+Hannu Krosing
+
+== Introduction ==
+
+This is a HowTo for setting up PostgreSQL for backup
+and then continue to setting up both wal-based and streaming'
+replication using waplmgr3
+
+The samle commandlines and configuration files are produced by
+running command
+
+-----
+python WALManager.py
+-----
+
+in directory python/testwrappers. This test/sample
+
+ * sets up a postgresql master server using directory /tmp/test_master
+ * creates a sample database tyhere using pgbench
+ * creates walmgr3 master configuration file /tmp/test_master/wal-master.ini
+ * creates a directory and configuration file "on slave" - /tmp/test_slave/wal-slave.ini
+ * runs walmgr3 setup command, which modifies postgresql.conf file on master server for replication
+ * restarts master server for changes of "walmgr3 ... setup" to take effect
+ * runs walmgr3 backup command, which creates a backup of master database and wal files on slace server
+ * modifies master's postgresql.conf and pg_hba.conf file some more for enabling Streaming Replication
+ * runs walmgr3 restore command ON SLAVE which moves backup directory into the right place for slave
+ server and then starts the slave server
+ * verifies that replication is running by running a SQL UPDATE query on master and checking the results on slave.
+
+== Preparation ==
+
+For things to work you need to have passwordless ssh access
+from master to slave set up using public/private key pair
+
+This can be done using walmgr3
+
+----
+1. Set up passwordless ssh-key on Master and write configuration file
+
+ master$ walmgr3 --ssh-keygen --init-master --slave <slave_hostname>
+
+2. Set up passwordless ssh authentication from Master to Slave and write configuration file on Slave
+
+ slave$ walmgr3 --init-slave --ssh-add-key=/tmp/id_rsa.pub
+----
+
+or directly from commandline
+
+----
+master$ test -f ~/.ssh/id_dsa.pub || ssh-keygen -t dsa
+master$ cat ~/.ssh/id_dsa.pub | ssh slave cat \>\> .ssh/authorized_keys
+----
+
+== setting up an population master database ===
+
+This part is for playing with walmgr3 without disturbing your existing
+databases.
+
+You can skip this part, if you already have a database you want to replicate.
+
+=== create database server ===
+
+Create database master server directory structure under /tmp/test_master
+by running the command :
+
+----
+/usr/lib/postgresql/9.1/bin/initdb -D /tmp/test_master
+----
+
+This database wil run as the user who was logged in at the time of
+creation, vs. the default user 'postgres'
+
+Change the port and socket directory (and set up some logging if you want
+to see what's going on in the database)
+
+Edit /tmp/test_master/postgresql.conf and set up the following
+
+----
+ unix_socket_directory = /tmp
+ port = 55401
+ # optional, for logging
+ log_connections = on
+ log_disconnections = on
+----
+
+Now you are ready to start up this server
+
+----
+/usr/lib/postgresql/9.1/bin/pg_ctl -D /tmp/test_master -l /tmp/test_master/postgresql.log start
+----
+
+you can use "tail /tmp/test_master/postgresql.log" to check that server started up correctly, and
+
+----
+psql -h /tmp -p 55401 -l
+----
+
+to check that it accepts connections
+
+=== create database and generate some data for it ===
+
+Create database to be used
+
+----
+createdb pgbdb;
+----
+
+and initialise a pgbench database structure and data in this database
+
+----
+/usr/lib/postgresql/9.1/bin/pgbench -i -s 1 -F 80 pgbdb -h /tmp -p 55401
+----
+
+== Setting up the replication ==
+
+OK. Now we have a database server to replicate, so lets configure walmgr3
+
+=== setting up master ===
+
+First we need a master configuration file /tmp/test_master/wal-master.ini :
+
+----
+[walmgr]
+job_name = wal-master
+logfile = /tmp/test_master/%(job_name)s.log
+pidfile = /tmp/test_master/%(job_name)s.pid
+use_skylog = 0
+
+master_db = port=55401 host=/tmp dbname=template1
+master_data = /tmp/test_master
+master_config = /tmp/test_master/postgresql.conf
+master_bin = /usr/lib/postgresql/9.1/bin/
+
+# set this only if you can afford database restarts during setup and stop.
+# master_restart_cmd = pg_ctlcluster 9.1 main restart
+
+slave = 127.0.0.1
+slave_config = /tmp/test_slave/wal-slave.ini
+
+walmgr_data = /tmp/test_slave_walmanager/backup/
+
+completed_wals = %(walmgr_data)s/logs.complete
+partial_wals = %(walmgr_data)s/logs.partial
+full_backup = %(walmgr_data)s/data.master
+config_backup = %(walmgr_data)s/config.backup
+
+# syncdaemon update frequency
+loop_delay = 10.0
+# use record based shipping available since 8.2
+use_xlog_functions = 0
+
+# pass -z to rsync, useful on low bandwidth links
+compression = 0
+
+# keep symlinks for pg_xlog and pg_log
+keep_symlinks = 1
+
+# tell walmgr to set wal_level to hot_standby during setup
+hot_standby = 1
+
+# periodic sync
+#command_interval = 600
+#periodic_command = /var/lib/postgresql/walshipping/periodic.sh
+----
+
+The things to takew notice hera ar that
+
+ * walmgr_data is a directory _on_the_slave_host_
+ * it is a bad idea to put slave_config in slave data directory (that
+ would be /tmp/test_slave/data/ as defined in wal-slave.ini below)
+ as then it gets overwritten when doing the restore.
+
+
+=== setting up slave ===
+
+You also need a walmgr3 conf file on slave, /tmp/test_slave/wal-slave.ini :
+
+----
+[walmgr]
+job_name = wal-standby
+logfile = /tmp/test_slave_walmanager/%(job_name)s.log
+use_skylog = 0
+
+slave_data = /tmp/test_slave/data
+slave_bin = /usr/lib/postgresql/9.1/bin/
+slave_stop_cmd = /usr/lib/postgresql/9.1/bin//pg_ctl -D /tmp/test_slave/data stop
+slave_start_cmd = /usr/lib/postgresql/9.1/bin//pg_ctl -D /tmp/test_slave/data -l /tmp/test_slave/data/postgresql.log start
+#slave_config_dir = /tmp/test_slave
+slave_config_dir = /tmp/test_slave/data
+
+walmgr_data = /tmp/test_slave_walmanager/backup/
+completed_wals = %(walmgr_data)s/logs.complete
+partial_wals = %(walmgr_data)s/logs.partial
+full_backup = %(walmgr_data)s/data.master
+config_backup = %(walmgr_data)s/config.backup
+
+backup_datadir = no
+keep_backups = 0
+# archive_command =
+
+# primary database connect string for hot standby -- enabling
+# this will cause the slave to be started in hot standby mode.
+primary_conninfo = host=127.0.0.1 port=55401 host=/tmp
+---
+
+=== Configuring postgreSQL for replication using walmgr3 ===
+
+Running the command :
+----
+walmgr3 /tmp/test_master/wal-master.ini setup
+----
+
+Modifies master postgresql ini file (/tmp/test_master/postgresql.conf)
+with these values
+----
+wal_level = 'hot_standby'
+archive_mode = 'on'
+archive_command = '/usr/local/bin/walmgr3 /tmp/test_master/wal-master.ini xarchive %p %f'
+----
+
+To enable streaming replication, you will need see that max_wal_senders
+is above 0 in /tmp/test_master/postgresql.conf
+
+----
+setting: max_wal_senders = 3
+----
+
+And yoy need to specifically enable the access to "replication" in /tmp/test_master/pg_hba.conf
+(Replication has to be enabled by name, database wildcard * does not cover it)
+
+----
+local replication ubuntu trust
+----
+
+it is "local" here, as this test/sample does the replication iover local socket,
+follow the commenst in pg_hba.conf for your case.
+
+And again, fo the changes to take effect, you need to restart the server
+
+----
+/usr/lib/postgresql/9.1/bin/pg_ctl -D /tmp/test_master restart
+----
+
+== Making the backup ==
+
+Once the configuration files are in place, making a backup is as simple as running
+
+----
+walmgr3 /tmp/test_master/wal-master.ini backup
+----
+
+If all goes well, this is what gets output (or written to logsfiles, if so configured)
+
+----
+0 2012-01-27 16:58:31,464 30870 INFO Backup lock obtained.
+2012-01-27 16:58:31,485 30750 INFO Execute SQL: select pg_start_backup('FullBackup'); [port=55401 host=/tmp dbname=template1]
+2012-01-27 16:58:36,779 30750 INFO Checking tablespaces
+2012-01-27 16:58:36,786 30750 INFO pg_log does not exist, skipping
+2012-01-27 16:58:36,873 30750 INFO Backup conf files from /tmp/test_master
+2012-01-27 16:58:38,599 31256 INFO First useful WAL file is: 000000010000000000000002
+2012-01-27 16:58:45,442 31633 INFO Backup lock released.
+2012-01-27 16:58:45,461 30750 INFO Full backup successful
+----
+
+and there will be a copy of your masters data directory on slave host
+under /tmp/test_slave_walmanager/backup/data.master, WAL files in
+files in /tmp/test_slave_walmanager/backup/logs.complete/ and copies
+of configuration files in /tmp/test_slave_walmanager/backup/config.backup/
+
+=== Backup is done ===
+
+If your aim is to just make copies, you can stop this howto here
+
+== Starting a Hot Standby replica ==
+
+=== Setting Up the replica ===
+
+If you want your replica to be usable for read-only queries, and not
+just report "server is starting up" when you try to connect, then your
+slave postgresql.conf neeeds to have "hot_standby = on" set:
+
+In our sample we also change the port, so we can run both servers on the
+same host.
+
+Editi /tmp/test_slave_walmanager/backup/config.backup/postgresql.conf
+
+----
+hot_standby = on
+port = 55402
+----
+
+=== starting the replica ===
+
+You restore backup to your the replica server and start it in one command
+
+----
+walmgr3 /tmp/test_slave/wal-slave.ini restore
+----
+
+Herew is what it outputs if everything runs ok
+
+----
+0 server starting
+2012-01-27 16:58:45,709 31636 WARNING backup_datadir is disabled, deleting old data dir
+2012-01-27 16:58:45,709 31636 INFO Move /tmp/test_slave_walmanager/backup//data.master to /tmp/test_slave/data
+2012-01-27 16:58:45,766 31636 INFO Write /tmp/test_slave/data/recovery.conf
+2012-01-27 16:58:45,768 31636 INFO Restoring configuration files
+2012-01-27 16:58:45,771 31636 INFO Starting postmaster: /usr/lib/postgresql/9.1/bin//pg_ctl -D /tmp/test_slave/data -l /tmp/test_slave/data/postgresql.log start
+----
+
+Now you have a streaming replica running at port 55402 and master at port 55401
+
+== Testing replication ==
+
+You can test that replication is really working by opening connections
+to both master and server and then watching changes done on master appear
+instantaneously oin the slave:
+
+Check the initial replicated state on slave
+
+----
+ubuntu@ubuntu-VirtualBox:~/skytools-markokr/python/testwrappers$ psql -h /tmp -p 55402 pgbdb
+psql (9.1.1)
+Type "help" for help.
+
+pgbdb=# select * from pgbench_tellers;
+ tid | bid | tbalance | filler
+-----+-----+----------+--------
+ 1 | 1 | 0 |
+ 2 | 1 | 0 |
+ 3 | 1 | 0 |
+ 4 | 1 | 0 |
+ 5 | 1 | 0 |
+ 6 | 1 | 0 |
+ 7 | 1 | 0 |
+ 8 | 1 | 0 |
+ 9 | 1 | 0 |
+ 10 | 1 | 0 |
+(10 rows)
+----
+
+Update the pgbench_tellers table on master
+
+----
+pgbdb=# \q
+ubuntu@ubuntu-VirtualBox:~/skytools-markokr/python/testwrappers$ psql -h /tmp -p 55401 pgbdb
+psql (9.1.1)
+Type "help" for help.
+
+pgbdb=# update pgbench_tellers set filler = random();
+UPDATE 10
+pgbdb=# select * from pgbench_tellers;
+ tid | bid | tbalance | filler
+-----+-----+----------+--------------------------------------------------------------------------------------
+ 1 | 1 | 0 | 0.755602441262454
+ 2 | 1 | 0 | 0.130802480038255
+ 3 | 1 | 0 | 0.725358869414777
+ 4 | 1 | 0 | 0.65205558296293
+ 5 | 1 | 0 | 0.0436737341806293
+ 6 | 1 | 0 | 0.797202748246491
+ 7 | 1 | 0 | 0.909699931740761
+ 8 | 1 | 0 | 0.981106289196759
+ 9 | 1 | 0 | 0.656265312805772
+ 10 | 1 | 0 | 0.759600875433534
+(10 rows)
+
+pgbdb=# \q
+----
+
+And check that it is the same on slave
+
+----
+ubuntu@ubuntu-VirtualBox:~/skytools-markokr/python/testwrappers$ psql -h /tmp -p 55402 pgbdb
+psql (9.1.1)
+Type "help" for help.
+
+pgbdb=# select * from pgbench_tellers;
+ tid | bid | tbalance | filler
+-----+-----+----------+--------------------------------------------------------------------------------------
+ 1 | 1 | 0 | 0.755602441262454
+ 2 | 1 | 0 | 0.130802480038255
+ 3 | 1 | 0 | 0.725358869414777
+ 4 | 1 | 0 | 0.65205558296293
+ 5 | 1 | 0 | 0.0436737341806293
+ 6 | 1 | 0 | 0.797202748246491
+ 7 | 1 | 0 | 0.909699931740761
+ 8 | 1 | 0 | 0.981106289196759
+ 9 | 1 | 0 | 0.656265312805772
+ 10 | 1 | 0 | 0.759600875433534
+(10 rows)
+
+pgbdb=#
+-----
+
+The replication is done now!
+
--- /dev/null
+= Skytools 3 Documentation =
+
+== Overview ==
+
+* link:README.html[Package overview]
+* link:INSTALL.html[Installation help]
+* link:doc/faq.html[FAQ]
+* link:doc/skytools3.html[Skytools 3.0 Release Notes]
+
+== HOWTOs ==
+
+* link:doc/howto/londiste3_simple_rep_howto.html[Setting up simple replication with 2 nodes]
+* link:doc/howto/londiste3_cascaded_rep_howto.html[Setting up cascaded replication with 5 nodes]
+* link:doc/howto/londiste3_merge_howto.html[Merging 2 partitions to one big database]
+* link:doc/howto/londiste3_partitioning_howto.html[Replicating from one master to 2 partitions]
+* link:doc/howto/setup_walmgr_replication.html[Setting up walmgr]
+
+== Manpages ==
+
+* link:doc/londiste3.html[londiste3] - Londiste command line interface.
+* link:doc/qadmin.html[qadmin] - psql-like console for managing queues
+* link:doc/queue_mover.html[queue_mover] - copy queue to another database
+* link:doc/queue_splitter.html[queue_splitter] - split queue into different queues
+* link:doc/scriptmgr.html[scriptmgr] - bulk start/stopping of skytools scripts
+* link:doc/skytools_upgrade.html[skytools_upgrade] - Update database modules
+* link:doc/walmgr3.html[walmgr3] - tool for managing WAL-base replication
+
+== API docs ==
+
+* Python link:api/[API] documentation
+* SQL API documentation:
+ - link:pgq[]: SQL module for generic queue
+ - link:pgq_coop[]: SQL module for sharing workload
+ - link:pgq_ext[]: batch tracking in target database
+ - link:pgq_node[]: cascaded queueing
+ - link:londiste[]: Londiste state
+
+== Internal ==
+
+* link:doc/devnotes.html[Notes for contributors]
+* link:doc/set.notes.html[Technical notes for cascading]
+* link:doc/TODO.html[TODO list]
+
+// == Old docs ==
+// * link:doc/pgq-nodupes.html[]
+// * link:doc/pgq-sql.html[]
+
* Cascading is implemented as generic layer on top of PgQ - *Cascaded PgQ*.
- Its goal is to keep identical copy of queue contents in several nodes.
- Not replication-specific - can be used for any queue.
- - Advanced admin operations: switchover, failover, change-provider, pause/resume.
+ - Advanced admin operations: takeover, change-provider, pause/resume.
- For terminology and technical details see here: set.notes.txt.
* New Londiste features:
were introduced in 2.1.x, but were not on by default. Now they are
used by default.
- - Londiste processes events via 'handlers'. (table partitioning)
+ - Londiste processes events via 'handlers'. Thus we can do table partitioning
+ in Londiste, instead of custom consumer, which means all Londiste features
+ are available in such situation - like proper initial COPY.
+ To see list of them: `londiste3 x.ini show-handlers`.
- Target table can use different name (--dest-table)
user-friendly, this is an experiment on interactive console with
heavy emphasis on tab-completion.
-* New multi-database ticker. It is possible to set up one process that
+* New multi-database ticker: `pgqd`. It is possible to set up one process that
maintains all PgQ databases in one PostgreSQL instance. It will
auto-detect both databases and whether they have PgQ installed.
This also makes core PgQ usable without need for Python.
-* New cascaded dispatcher script. Previous 3 dispatcher scripts
- (bulk_loader, cube_dispatcher, table_dispatcher) shared quite
- a lot of logic for partitionaing, differing on few details.
- So instead of porting all 3 to cascaded consuming, I merged them.
-
Minor improvements
------------------
* PgQ does not handle "failed events" anymore.
-Open questions
---------------
-
-* New ticker
- - Name for final executable: pgqd, pgq-ticker, or something else?
- - Should it serialize retry and maint operations on different dbs?
- - Should it serialize ticking?
-
-* Python modules
- - Skytools 3 modules should be parallel installable with Skytools 2.
- we decided to solve it via loader module
- (like http://faq.pygtk.org/index.py?req=all#2.4[pygtk]).
- The question is should we have Skytools-specific loader or more generic.
- And what should the name be?
-
- import skytools_loader
- skytools_loader.require('3.0')
-
- vs
+* Skytools 3 modules are parallel installable with Skytools 2.
+ Solved via loader module (like http://faq.pygtk.org/index.py?req=all#2.4[pygtk]).
+
+ import pkgloader
+ pkgloader.require('skytools', '3.0')
+ import skytools
- import pkgloader
- pkgloader.require('skytools', '3.0')
-
-* Londiste EXECUTE command
- - Should the scripts have ability to inform Londiste of the tables
- they operate on, so that nodes that do not have such tables
- can ignore the script.
-
-* Is there good reason not to drop following modules:
- - logtriga(), pgq.logtriga() - non-automatic triggers
- - cube_dispatcher, table_dispatcher, bulk_loader - they are merged into queue_loader
Further reading
---------------
-* Skytools 3 todo list: TODO.txt
-* QAdmin design and todo list: qadmin.txt
-* Technical notes about cascading: set.notes.txt
-* Notes for contributors: devnotes.txt
-
-* Python API docs:
- - http://skytools.projects.postgresql.org/skytools-3.0/api/[skytools, pgq, londiste modules]
-
-* Database API docs:
- - http://skytools.projects.postgresql.org/skytools-3.0/pgq/[PgQ]
- - http://skytools.projects.postgresql.org/skytools-3.0/pgq_node/[Cascaded PgQ]
- - http://skytools.projects.postgresql.org/skytools-3.0/pgq_coop/[Cooperative PgQ]
- - http://skytools.projects.postgresql.org/skytools-3.0/londiste/[Londiste]
-
-* Londiste Demo:
- - http://skytools.projects.postgresql.org/skytools-3.0/demo.html[]
+* http://skytools.projects.postgresql.org/skytools-3.0/[Documentation] for skytools3.
projects / skytools.git / commitdiff