pgpool2.git
3 years agoEnable streaming replication check by default.
Tatsuo Ishii [Wed, 13 Oct 2021 02:26:43 +0000 (11:26 +0900)]
Enable streaming replication check by default.

Change the compile time default value of sr_check_period from 0 to
10. This means if operating in streaming replication mode, now
streaming replication check is enabled by default. The parameter is
already set to 10 in pgpool.conf.sample-stream. So the only difference
is, when sr_check_period is commented out.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-October/004042.html

3 years agoFix there's no new line at the end of file.
Tatsuo Ishii [Tue, 12 Oct 2021 08:28:15 +0000 (17:28 +0900)]
Fix there's no new line at the end of file.

3 years agoFix nasty bug with pgpool_recovery() in 4.3 or later.
Tatsuo Ishii [Tue, 12 Oct 2021 06:16:49 +0000 (15:16 +0900)]
Fix nasty bug with pgpool_recovery() in 4.3 or later.

Reported by Kawamoto and Peng Bo.

3 years agoFix signal handler for SIGTERM, SIGINT and SIGQUIT.
Tatsuo Ishii [Tue, 12 Oct 2021 01:40:11 +0000 (10:40 +0900)]
Fix signal handler for SIGTERM, SIGINT and SIGQUIT.

It did not properly save errno and it was possible to overwrite errno.

3 years agoChange compile time default of log_standby_delay from 'none' to 'if_over_threshold'.
Tatsuo Ishii [Fri, 8 Oct 2021 01:55:53 +0000 (10:55 +0900)]
Change compile time default of log_standby_delay from 'none' to 'if_over_threshold'.

Before the compile time default was 'none' but the default value of
sample pgpool.conf was 'if_over_threshold'. Since 'if_over_threshold'
is more useful for most users, change the compile time default to
'if_over_threshold' as well.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-October/004035.html

4 years agoFix pcp_proc_info, show pool_processes and show pool_pools command.
Masaya Kawamoto [Wed, 6 Oct 2021 05:56:15 +0000 (05:56 +0000)]
Fix pcp_proc_info, show pool_processes and show pool_pools command.

I added new fields to the result of those commands in following commit:
https://git.postgresql.org/gitweb/?p=pgpool2.git;a=commitdiff;h=45152975fdf8d57deaff07dc6c34117fa8eeb3db

The usage of client_idle_limit was mistaken. Child processes are
restarted by child_life_time not client_idle_limit, and client_idle_limit
is related to Client idle duration and disconnects the connection.

So, I changed the display of Start time and Client idle duration.
I modify wait_for_connections function to calculate the time remaining
until the child process is restarted by child_life_time; Instead of
using child_life_time as the timeout for select(2), select(2) with 1 sec
timeout is looped through child_life_time times to get the elapsed time.

Moreover when there were multiple backends, Client idle duration was
only displayed in backend0, so I fix this too.

4 years agoDoc: fix documentation typos.
Bo Peng [Wed, 6 Oct 2021 01:30:24 +0000 (10:30 +0900)]
Doc: fix documentation typos.

4 years agoFix typos in documentation and sample scripts.
Bo Peng [Mon, 4 Oct 2021 10:56:49 +0000 (19:56 +0900)]
Fix typos in documentation and sample scripts.

Patch is created by Kazufumi Noto.

4 years agoAdd missing files to tar ball.
Bo Peng [Sun, 3 Oct 2021 12:12:00 +0000 (21:12 +0900)]
Add missing files to tar ball.

4 years agoDoc: fix yum install command typo in configuration example.
Bo Peng [Wed, 29 Sep 2021 14:47:17 +0000 (23:47 +0900)]
Doc: fix yum install command typo in configuration example.

4 years agoFix some compile time default values.
Tatsuo Ishii [Tue, 28 Sep 2021 10:48:55 +0000 (19:48 +0900)]
Fix some compile time default values.

Some of default values defined in src/config/pool_config_variable.c
were different from what the documentation expects. Namely:

insert_lock (changed from "true" to "false")
clear_memqcache_on_escalation (changed from "false" to "true")
wd_lifecheck_dbname (changed from "postgres" to "template1")
listen_backlog_multiplier (changed from "32" to "2")
authentication_timeout (changed from "0" to "60")

4 years agoFix for bug-732: Segmentation fault at failover ...
Muhammad Usama [Mon, 27 Sep 2021 17:07:48 +0000 (22:07 +0500)]
Fix for bug-732: Segmentation fault at failover ...

trigger_failover_command() had an assumption that old primary can never be NULL,
which of course is not the case.

4 years agoFeature: Import PostgreSQL 14 beta2 new parser.
Bo Peng [Mon, 27 Sep 2021 14:36:29 +0000 (23:36 +0900)]
Feature: Import PostgreSQL 14 beta2 new parser.

Major changes of PostgreSQL 14 parser include:

- Allow an alias to be used to a JOIN ... USING

  SELECT ... FROM t1 JOIN t2 USING (a, b, c) AS x

- Allow DISTINCT to be added to GROUP BY

  SELECT c1, max(c2) FROM t1 GROUP BY DISTINCT c1;

- New SEARCH and CYCLE clauses for common table expressions

4 years agoFix pg_config command path to avoid test failure.
Bo Peng [Thu, 23 Sep 2021 12:54:00 +0000 (21:54 +0900)]
Fix pg_config command path to avoid test failure.

4 years agoFix to support PostgreSQL version (e.g. alpha, beta).
Bo Peng [Thu, 23 Sep 2021 12:32:38 +0000 (21:32 +0900)]
Fix to support PostgreSQL version (e.g. alpha, beta).

4 years agoFix psql command path to avoid test failure.
Bo Peng [Thu, 23 Sep 2021 11:44:27 +0000 (20:44 +0900)]
Fix psql command path to avoid test failure.

4 years agoAdd new config parameter "failover_on_backend_shutdown".
Tatsuo Ishii [Wed, 22 Sep 2021 00:23:02 +0000 (09:23 +0900)]
Add new config parameter "failover_on_backend_shutdown".

If set to on, admin shutdown triggers failover (existing behavior).
If set to off, admin shutdown does not trigger failover (new
behavior), and is useful to completely avoid unwanted failover caused
by pg_terminate_backend() and SIGTERM signal sent to the backend
process.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-August/003999.html

4 years agoFix occasional hang in COPY FROM.
Tatsuo Ishii [Thu, 16 Sep 2021 06:44:51 +0000 (15:44 +0900)]
Fix occasional hang in COPY FROM.

If an error occurs while doing COPY FROM, it was possible the
Pgpool-II waited forever for a response from backend after COPY end
marker was sent from frontend. Pgpool expected a new message arrives
to socket, but it is possible that the message (in this case an error
message) is already in the backend read buffer. The fix is, check the
buffer is empty or not before reading from the socket.
New test case (07.copy_hang) is also added.

The bug was found by Bo Peng.

4 years agoDoc: fix incorrect file name in "Pgpool-II on Kubernetes".
Bo Peng [Mon, 13 Sep 2021 08:12:56 +0000 (17:12 +0900)]
Doc: fix incorrect file name in "Pgpool-II on Kubernetes".

4 years agoDoc: add release notes for Pgpool-II 4.2.5.
Bo Peng [Mon, 13 Sep 2021 05:57:59 +0000 (14:57 +0900)]
Doc: add release notes for Pgpool-II 4.2.5.

4 years agoFix for bug-731: Fails to execute follow_primary_command..
Muhammad Usama [Fri, 10 Sep 2021 09:38:25 +0000 (14:38 +0500)]
Fix for bug-731: Fails to execute follow_primary_command..

It was a segmentation fault happening because of a silly coding mistake in
parse_wd_exec_cluster_command_json() function.
The problem was the WDExecCommandArg** arg for returning the argument list
was not getting populated properly.
As part of the fix, to get rid of complexity and confusion, I have changed
the function signature to accept List* type argument for returning the
parsed command args instead of an array and its length.

Thanks to Bo Peng, Emond Papegaaij and Tatsuo Ishi for reviewing
and testing the fix

4 years agoAdd new fields to pcp_proc_info, show pool_processes, show pool_pools command.
Masaya Kawamoto [Wed, 8 Sep 2021 06:46:31 +0000 (06:46 +0000)]
Add new fields to pcp_proc_info, show pool_processes, show pool_pools command.

The purpose of this commit is to get more information about the process stats
and how pgpool is working with "client_idle_limit" and "child_max_connections".

Start time:
I add “remaining” that is the remaining time beforl the process is restarted by
client_idle_limit.

Client connection count:
The connection count from clients to this pgpool process. Useful to know how
many times clients can connect to this pgpool process before
child_max_connections reaches.

Backend connection time:
I just rename create_time for clarity.

Client connection time:
The time when the client last connected. If the client uses its own connection
pooling, this helps to know when the connection was actually made from pgpool's
point of view.

Client idle duration:
The elapsed time (in seconds) since the client became in idle state.

Client disconnection time:
This is the last time the client disconnects. The use case is the same as above.

Status:
This is the process status. This shows 4 status: “Wait for connections”, “Idle”,
“Idle in transaction”, “Execute command”.

4 years agoDoc: fixed "PGPOOL SHOW" documentaion that was missing in the previous commit 151e8f5...
Bo Peng [Mon, 6 Sep 2021 08:45:07 +0000 (17:45 +0900)]
Doc: fixed "PGPOOL SHOW" documentaion that was missing in the previous commit 151e8f54961b1a0394cfc86c2f4ddf715b41ceea.

4 years agoFix incorrect PGPOOL SHOW option.
Bo Peng [Fri, 3 Sep 2021 05:16:01 +0000 (14:16 +0900)]
Fix incorrect PGPOOL SHOW option.

Since 4.2 other_pgpool is removed, "PGPOOL SHOW other_pgpool" command should be changed to "PGPOOL SHOW watchdog".

4 years agoFix bug of "PGPOOL SHOW heartbeat" and "PGPOOL SHOW ALL" command.
Bo Peng [Fri, 3 Sep 2021 04:58:46 +0000 (13:58 +0900)]
Fix bug of "PGPOOL SHOW heartbeat" and "PGPOOL SHOW ALL" command.

The last member in heartbeat_* array was not displayed in the command result.

4 years agoFix to allow log_rotation_age to be set to 0.
Tatsuo Ishii [Thu, 2 Sep 2021 00:07:15 +0000 (09:07 +0900)]
Fix to allow log_rotation_age to be set to 0.

According to the docs, it is possible to set 0 to log_rotation_age so
that the log rotation feature is disabled. But the allowed minimum
value for it was mistakenly set to 10, which made it impossible to set
log_rotation_age to 0.

4 years agoSupport include directive in pgppol.conf file
Takuma Hoshiai [Tue, 31 Aug 2021 03:15:12 +0000 (12:15 +0900)]
Support include directive in pgppol.conf file

The pgpool.conf can be nested by 'include' directive.A depth of include have a
restriction for avoiding loop back of include files. 'include' is not regular
parameter, so SHOW command don't display it.

4 years agoFix pgpool_setup to do nothing when no new main node is available.
Tatsuo Ishii [Wed, 25 Aug 2021 00:52:39 +0000 (09:52 +0900)]
Fix pgpool_setup to do nothing when no new main node is available.

When no new main node is available due to there's only 1 node
remaining, pgpool_setup tried to promote a node with node id -1.  This
is impossible. Fix is just skipping whole failover process if new main
node id is -1.

4 years agoFix the incorrect display of load balancing node in raw mode.
Bo Peng [Wed, 18 Aug 2021 08:00:28 +0000 (17:00 +0900)]
Fix the incorrect display of load balancing node in raw mode.

In raw mode, Pgpool-II sends all queies to main node.
This is harmless, but it may confuse users.

4 years agoFix pgpool logger process eats 100% cpu.
Tatsuo Ishii [Thu, 12 Aug 2021 07:37:26 +0000 (16:37 +0900)]
Fix pgpool logger process eats 100% cpu.

The select(2) loop in the logger process did not initialize
timeout.tv_usec, which could cause error in select(2) because of too
big tv_usec. In this select(2) immediately returned and iterate the
for loop.

Problem reported and patch provided by Fang Jun.
Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-August/003993.html

4 years agoFix compiler warning.
Tatsuo Ishii [Wed, 11 Aug 2021 00:07:58 +0000 (09:07 +0900)]
Fix compiler warning.

4 years agoUpdate Makefile.in and configure.
Bo Peng [Tue, 10 Aug 2021 12:02:12 +0000 (21:02 +0900)]
Update Makefile.in and configure.

4 years agoFix "ar: `u' modifier ignored since `D' is the default (see `U')" warnings.
Tatsuo Ishii [Tue, 10 Aug 2021 10:07:59 +0000 (19:07 +0900)]
Fix "ar: `u' modifier ignored since `D' is the default (see `U')" warnings.

The warning can be eliminated by changing arguments of ar command from
"cru" to "cr".  There were two places of the warnings: src/libs/pcp
and src/watchdog.  For the former, we need to fix libtool, which is
generated by configure. So fix configure.ac.  For the latter, just fix
src/watchdog/Makefile.am.

4 years agoUpdate configure and src/config/pool_config.c due to the previous commit.
Bo Peng [Tue, 10 Aug 2021 07:52:03 +0000 (16:52 +0900)]
Update configure and src/config/pool_config.c due to the previous commit.

4 years agoFix compiler warnings generated by newer version of gcc.
Tatsuo Ishii [Tue, 10 Aug 2021 07:12:55 +0000 (16:12 +0900)]
Fix compiler warnings generated by newer version of gcc.

gcc on Ubuntu 20 (9.3.0) generates tons of compiler warnings something like:

In file included from /usr/include/string.h:495,
>                  from pg_md5.c:31:
> In function 。strncpy「,
>     inlined from 。update_pool_passwd_from_file「 at pg_md5.c:276:3,
>     inlined from 。main「 at pg_md5.c:136:3:
> /usr/include/x86_64-linux-gnu/bits/string_fortified.h:106:10: warning: 。__builtin___strncpy_chk「 output may be truncated copying between 0 and 128 bytes from a string of length 257 [-Wstringop-truncation]
>   106 |   return __builtin___strncpy_chk (__dest, __src, __len, __bos (__dest));
>       |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>

I have found a message in PostgreSQL hackers mailing list regarding similar issue:
https://www.postgresql.org/message-id/flat/21789.1529170195%40sss.pgh.pa.us#525c384d13505fa6f1f25c50b00d7a08

To fix these warnings:

1) fix misuse of strncpy() and friends

2) add -Wno-stringop-truncation to CLFAG

This commit implelents #1 and #2.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-August/003990.html

4 years agoFix SI mode to acquire a snapshot with an internal transaction.
Tatsuo Ishii [Tue, 10 Aug 2021 04:59:32 +0000 (13:59 +0900)]
Fix SI mode to acquire a snapshot with an internal transaction.

In SI mode a data modifying statement needs to start internal
transaction.  However it forgot to acquire a snapshot. This commit
fixes that in both simple query protocol and extended query protocol.

4 years agoUpdate Makefile and SPEC file to include "pgpool_recovery--1.3--1.4.sql" and "pgpool_...
Bo Peng [Tue, 10 Aug 2021 03:02:52 +0000 (12:02 +0900)]
Update Makefile and SPEC file to include "pgpool_recovery--1.3--1.4.sql" and "pgpool_recovery--1.4.sql".

4 years agoDoc: fix typo in Japanese release note.
Masaya Kawamoto [Tue, 10 Aug 2021 01:29:22 +0000 (01:29 +0000)]
Doc: fix typo in Japanese release note.

4 years agoDoc: Mention that double quotes are required in "PGPOOL SHOW" command if the paramete...
Bo Peng [Mon, 9 Aug 2021 01:43:47 +0000 (10:43 +0900)]
Doc: Mention that double quotes are required in "PGPOOL SHOW" command if the parameter contains uppercase letters.

4 years agoFix backend_flag* parameter shown twice while executing "pgpool show all".
Tatsuo Ishii [Sun, 8 Aug 2021 04:19:33 +0000 (13:19 +0900)]
Fix backend_flag* parameter shown twice while executing "pgpool show all".

There are two entries of "backend_flag" for "ALLOW_TO_FAILOVER" and
"ALWAYS_PRIMARY" in the config variable struct. This is mostly ok but
"pgpool show all" command displayed both backend_flag entries, which
looks redundant. The reason for this is, report_all_variables() shows
grouped variables first then other variables except already shown as
grouped variables.  Unfortunately build_variable groups() is not smart
enough to build grouped variable data: it only registers the first
backend_flag entry and leaves the second entry. Since the second entry
is not a grouped variable, backend_flag is shown firstly as a grouped
variable and then is show as a non grouped variable in
report_all_variables(). To fix this, mark that the second variable is
also a grouped variable (the flag is set by
build_config_variables()). See bug 728 for the report of the problem.

Also add/fix comments.

4 years agoDoc: add new parameter to recovery_1st_stage_command and recovery_2ndt_stage_command.
Tatsuo Ishii [Thu, 5 Aug 2021 08:39:00 +0000 (17:39 +0900)]
Doc: add new parameter to recovery_1st_stage_command and recovery_2ndt_stage_command.

4 years agoAdd missing changes in the previous commit.
Tatsuo Ishii [Thu, 5 Aug 2021 08:10:32 +0000 (17:10 +0900)]
Add missing changes in the previous commit.

4 years agoAdd new parameter to recovery_1st_stage_command and recovery_2nd_stage_command.
Tatsuo Ishii [Thu, 5 Aug 2021 07:45:46 +0000 (16:45 +0900)]
Add new parameter to recovery_1st_stage_command and recovery_2nd_stage_command.

The new parameter represents the primary (main) server host name which
is defined in backend_hostname parameter. The commands used to obtain
the host name by using hostname command. This is usually fine but in
some systems the host name returned by hostname command is not same as
backend_hostname. This brings problem to detach_false_primary feature.

Note that documentations are not completed in this commit because of
my development environment (Ubuntu 20) issue (openjade crashes). I
will push document changes later using another PC.

Discussion:
https://www.pgpool.net/pipermail/pgpool-hackers/2021-July/003943.html
https://www.pgpool.net/pipermail/pgpool-hackers/2021-July/003974.html

4 years agoAdd release notes.
Masaya Kawamoto [Tue, 3 Aug 2021 02:23:12 +0000 (02:23 +0000)]
Add release notes.

4 years agoDoc: Update configuration example "Pgpool-II on Kubernetes".
Bo Peng [Mon, 2 Aug 2021 16:58:31 +0000 (01:58 +0900)]
Doc: Update configuration example "Pgpool-II on Kubernetes".

4 years agoDoc: add more explanation about backend_application_name.
Tatsuo Ishii [Tue, 20 Jul 2021 07:01:08 +0000 (16:01 +0900)]
Doc: add more explanation about backend_application_name.

4 years agoAllow to log error messages in extended query mode in certain cases.
Tatsuo Ishii [Sat, 17 Jul 2021 07:37:27 +0000 (16:37 +0900)]
Allow to log error messages in extended query mode in certain cases.

In the extended query mode, it is possible that error messages are not
logged even after a sync message is received. With this commit
whenever ERROR response is arrived from backend in the extended query
mode, the error message is logged.

4 years agoImplementing the follow_primary command-locking over the watchdog channel.
Muhammad Usama [Fri, 16 Jul 2021 07:17:27 +0000 (12:17 +0500)]
Implementing the follow_primary command-locking over the watchdog channel.

Supplementary fix for [pgpool-hackers: 3892] Problem with detach_false_primary..

commit:455f00dd5f5b7b94bd91aa0b6b40aab21dceabb9 fixed a race condition between
detach_false_primary and follow_primary commands. Part of the fix was to make
sure that the detach_false_primary should only be executed on the
leader watchdog node.

The mentioned commit ensures the execution of detach_false_primary on the
watchdog leader by getting the watchdog status from within the main process.
The design is good enough for most cases, but has the potential to fail if
the cluster goes into the election process just after the main process
has read the status.

To fix that, this commit implements the synchronization of follow_primary_command
execution using the distributed locks over the watchdog channel.

The idea is, just before executing the follow_primary during the failover process
we instruct all standby watchdog nodes to acquire a lock on their respective
nodes to block the false primary detection during the period when the
follow_primary is being executed on the leader watchdog node.

Moreover to keep the watchdog process blocked on waiting for the lock the commit
introduced the pending remote lock mechanism, so that remote locks can get
acquired in the background after the completion of the inflight replication checks.

Finally, REQ_DETAIL_CONFIRMED flag is removed from degenerate_backend_set()
request that gets issued to detach the false primary, That means all quorum
and consensus rules must be satisfied for the detach to happen.

4 years agoDoc: Fix documentation typos.
Bo Peng [Wed, 14 Jul 2021 14:40:28 +0000 (23:40 +0900)]
Doc: Fix documentation typos.

4 years agoDoc: Fix documentation typos.
Bo Peng [Wed, 14 Jul 2021 14:05:16 +0000 (23:05 +0900)]
Doc: Fix documentation typos.

4 years agoUse milliseconds logging for Pgpool-II and PostgreSQL.
Tatsuo Ishii [Tue, 13 Jul 2021 13:21:44 +0000 (22:21 +0900)]
Use milliseconds logging for Pgpool-II and PostgreSQL.

This will make analyzing timing sensitive issues easier. Before we
could not analyze sub-second timing issues.

4 years agoAdd support for log time stamp with milliseconds.
Tatsuo Ishii [Tue, 13 Jul 2021 10:34:28 +0000 (19:34 +0900)]
Add support for log time stamp with milliseconds.

4 years agoFix 033.prefer_lower_standbu_delay
Masaya Kawamoto [Mon, 12 Jul 2021 04:52:40 +0000 (04:52 +0000)]
Fix 033.prefer_lower_standbu_delay

Sometimes this test failed due to the waiting time after pcp_reload_conf
was short. I modified the script to confirm the change by PGPOOL SHOW
before running the test.

4 years agoFix client side hang when describe message is followed by NoData response.
Tatsuo Ishii [Fri, 9 Jul 2021 02:59:33 +0000 (11:59 +0900)]
Fix client side hang when describe message is followed by NoData response.

For certain describe request for a query including "LISTEN", NoData
message can be returned. Since Pgpool-II buffered NoData, clients
could hang waiting for the NoData message in vain. To fix this, do not
buffer NoData message.

Problem reported and patch provided by Daniel van de Giessen.
Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-July/003951.html

4 years agoFix query cache to not cache SQLValueFunctions (CURRENT_TIME, CURRENT_USER etc.).
Tatsuo Ishii [Wed, 7 Jul 2021 04:03:59 +0000 (13:03 +0900)]
Fix query cache to not cache SQLValueFunctions (CURRENT_TIME, CURRENT_USER etc.).

Checking SQLValueFunctions was missed whether to be cached or not and
they were regarded as non function objects. As a result they were
cached.

Also add more test cases.

4 years agoDoc: fix typo in in memory query cache document.
Tatsuo Ishii [Tue, 6 Jul 2021 23:44:03 +0000 (08:44 +0900)]
Doc: fix typo in in memory query cache document.

4 years agoRevert "Fix 033.prefer_lower_standbu_delay"
Masaya Kawamoto [Mon, 5 Jul 2021 06:43:01 +0000 (06:43 +0000)]
Revert "Fix 033.prefer_lower_standbu_delay"

This reverts commit 61fb18a08d143c41d87ffa81dc1767c371f11917.
Previous commit was to deal with reload config sometimes not working. However that was not a good idea because it hides the real cause of the test failure.

4 years agoFix 033.prefer_lower_standbu_delay
Masaya Kawamoto [Mon, 5 Jul 2021 04:33:01 +0000 (04:33 +0000)]
Fix 033.prefer_lower_standbu_delay

4 years agoDoc: fix typo.
Tatsuo Ishii [Sun, 4 Jul 2021 02:24:25 +0000 (11:24 +0900)]
Doc: fix typo.

4 years agoFix typo in pgpool.conf samples.
Tatsuo Ishii [Sat, 3 Jul 2021 01:08:25 +0000 (10:08 +0900)]
Fix typo in pgpool.conf samples.

4 years agoAdd buffer length check.
Tatsuo Ishii [Fri, 2 Jul 2021 03:50:29 +0000 (12:50 +0900)]
Add buffer length check.

get_info_from_conninfo() did not check the size of the provided
buffers.  Add length parameters so that it can check the buffer size.

4 years agoFix compile warning
Takuma Hoshiai [Thu, 1 Jul 2021 06:32:08 +0000 (15:32 +0900)]
Fix compile warning

4 years agoFix sending invalid message in SI mode.
Tatsuo Ishii [Thu, 1 Jul 2021 04:41:34 +0000 (13:41 +0900)]
Fix sending invalid message in SI mode.

When a query is aborted by specific reason like serialization error,
Pgpool-II sends error query to abort transactions running on non main
nodes. The message length of the query was incorrect and it caused
"invalid string in message" error on backend.

4 years agoRemove /*NO LOAD BALANCE*/ comment
Takuma Hoshiai [Tue, 29 Jun 2021 02:29:39 +0000 (11:29 +0900)]
Remove /*NO LOAD BALANCE*/ comment

This comment was used for query load balancing controlling.
Currently, we can use allow_sql_comment and primary_routing_query_pattern_list
and things like that, so this speical comment is unnecessary.

4 years agoFix rsync parameter in pgpool_setup.
Tatsuo Ishii [Mon, 28 Jun 2021 05:59:06 +0000 (14:59 +0900)]
Fix rsync parameter in pgpool_setup.

It did not exclude "log" directory, which is the default logging
directory in recent PostgreSQL versions. This made hard to examine
PostgreSQL log, since it is copied from primary server.

4 years agoRemove unused static function.
Tatsuo Ishii [Sat, 26 Jun 2021 01:11:27 +0000 (10:11 +0900)]
Remove unused static function.

The function free_processInfo() is no longer needed due to the
refactoring c7690453.

4 years agoFix follow primary command creation in pgpool_setup.
Tatsuo Ishii [Fri, 25 Jun 2021 10:58:53 +0000 (19:58 +0900)]
Fix follow primary command creation in pgpool_setup.

If the target node of the follow primary script is primary and the
server version is before PostgreSQL 12, then the follow primary
command fails because there's no recovery.conf (12 or later is fine,
because "standby.signal" and "myrecovery.conf" are created).

To fix this, if server version is before 12, rename recovery.done to
recovery.conf.

Also this should fix 034.promote_node failure with PostgreSQL 11 or
before.

4 years agoFix the width of some struct member of LifeCheckNode.
Tatsuo Ishii [Thu, 24 Jun 2021 07:08:13 +0000 (16:08 +0900)]
Fix the width of some struct member of LifeCheckNode.

They were defined using constant numbers. Use more appropriate #define
symbols to make them consistent with other declarations.

The issue was detected by a Coverity run.

Also fix copyright year.

4 years agoFix follow primary script creation.
Tatsuo Ishii [Thu, 24 Jun 2021 06:09:35 +0000 (15:09 +0900)]
Fix follow primary script creation.

When PostgreSQL version is lower than 12, the script should have edit
recovery.conf, rather than myrecovery.conf.

4 years agoDoc: fix parameter name typo in ldap option
thoshiai [Mon, 21 Jun 2021 23:22:07 +0000 (08:22 +0900)]
Doc: fix parameter name typo in ldap option

4 years agoFix the commit b69e695758dcb2fea0dacfeaf1e40299352ccdaf and c75e1456b1259fc172c127d26...
Masaya Kawamoto [Mon, 21 Jun 2021 06:09:28 +0000 (06:09 +0000)]
Fix the commit b69e695758dcb2fea0dacfeaf1e40299352ccdaf and c75e1456b1259fc172c127d26b32285424399907

changing the permission of the regression test script and
adding prefer_lower_standby_delay to pool_process_reporting.c

4 years agoFix the commit b69e695758dcb2fea0dacfeaf1e40299352ccdaf
Masaya Kawamoto [Mon, 21 Jun 2021 05:19:19 +0000 (05:19 +0000)]
Fix the commit b69e695758dcb2fea0dacfeaf1e40299352ccdaf

4 years agoNew feature "prefer_lower_delay_standby"
Masaya Kawamoto [Mon, 21 Jun 2021 04:34:03 +0000 (04:34 +0000)]
New feature "prefer_lower_delay_standby"

Currently on streaming replication mode, when the node selected as the load
balancing node has streaming delay over delay_threshold, pgpool sends the
query to the primary regardless of other standbys.

When this parameter is set to on, if the delayed load balancing node is the
standby, pgpool selects the new load balancing node from the lowest delay standbys.

If all standbys are delayed or the primary is chosen as the load balancing node,
pgpool sends the query to the primary.

4 years agoFix 075.detach_primary_left_down_node.
Tatsuo Ishii [Mon, 21 Jun 2021 01:33:41 +0000 (10:33 +0900)]
Fix 075.detach_primary_left_down_node.

The test script executed pcp_detach_node but it always failed because
it tried to connect to UNIX domain socket.

4 years agoFix 031.connection_life_time.
Tatsuo Ishii [Mon, 21 Jun 2021 01:25:49 +0000 (10:25 +0900)]
Fix 031.connection_life_time.

The test script executed pcp_recovery_node but it always failed
because it tried to connect to UNIX domain socket.

4 years agoFix 018.detach_primary error in the log.
Tatsuo Ishii [Mon, 21 Jun 2021 01:19:23 +0000 (10:19 +0900)]
Fix 018.detach_primary error in the log.

The regression test itself passes but execution of pcp_watchdog_info
to get information failed because it tried to connect to UNIX domain
socket.

4 years agoFix pgpool_setup to generate portable follow_primary.sh.
Tatsuo Ishii [Mon, 21 Jun 2021 01:09:37 +0000 (10:09 +0900)]
Fix pgpool_setup to generate portable follow_primary.sh.

The script did not provide necessary information to execute pcp
commands: pcp port, PCPPASSFILE and path to pcp commands.

Also fix the failover.sh generation. Currently there's no user to use
the information but to avoid possible confusion in the future, the
information is also added.

These should fix the regression test failure of 034.promote_node.

4 years agoFix 034.promote_node test failure.
Tatsuo Ishii [Fri, 18 Jun 2021 23:23:12 +0000 (08:23 +0900)]
Fix 034.promote_node test failure.

It appeared that under buildfarm environment, pcp_socketdir is set to
/var/run/postgresql, but pcp client's socket directory are not changed
from /tmp (it is defined at the compile time). Probably we should
workaround this someday, but I just gives "-h localhost" parameter to
pcp_promote_node to avoid this problem.

4 years agoUpdate Copyright year.
Tatsuo Ishii [Thu, 17 Jun 2021 23:03:21 +0000 (08:03 +0900)]
Update Copyright year.

4 years agoFix regression test 034.promote_node failure.
Tatsuo Ishii [Wed, 16 Jun 2021 13:02:18 +0000 (22:02 +0900)]
Fix regression test 034.promote_node failure.

According to the build farm log, the cause of failure is, pcp server
was not yet ready.  So wait for up 10 seconds by calling pcp_node_info
before executing pcp_promote_node.

4 years agoRefactor "SHOW pool_pool_pools" and pcp_proc_info.
Tatsuo Ishii [Wed, 16 Jun 2021 08:26:01 +0000 (17:26 +0900)]
Refactor "SHOW pool_pool_pools" and pcp_proc_info.

These commands had many duplicated codes and did not use
infrastructures to make the code shorter. Now they are fixed and it is
much easier to add new data.

4 years agoFix pgpool_setup in creating base backup script.
Tatsuo Ishii [Tue, 15 Jun 2021 08:10:54 +0000 (17:10 +0900)]
Fix pgpool_setup in creating base backup script.

4 years agoAllow to specify node id to be promoted in pcp_promote_node.
Tatsuo Ishii [Tue, 15 Jun 2021 07:55:07 +0000 (16:55 +0900)]
Allow to specify node id to be promoted in pcp_promote_node.

For this purpose new parameter "-s" or "--switchover" is added.  If
the parameter is specified, current primary node is detached and the
target node id is passed to the failover script's argument "main
node". Since most of failover script promotes the "main node", this
results in promoting the node specified in pcp_promote_node.

The idea was initially proposed by Nathan Ward.  Patch was created by
me and reviewed by: Nathan Ward, Umar Hayat and Lachezar Dobrev.

Discussion:
https://www.pgpool.net/pipermail/pgpool-general/2021-April/007532.html
https://www.pgpool.net/pipermail/pgpool-hackers/2021-June/003921.html

4 years agoDo not use replication slot.
Tatsuo Ishii [Tue, 15 Jun 2021 07:52:10 +0000 (16:52 +0900)]
Do not use replication slot.

regress.sh enabled replication slot for pgpool_setup run.  The reason
is not clear but this makes some of tests occasionally failed.

4 years agoNew feature: Allow pcp_node_ino to list all backend nodes information.
Bo Peng [Tue, 15 Jun 2021 02:46:57 +0000 (11:46 +0900)]
New feature: Allow pcp_node_ino to list all backend nodes information.

4 years agoUpdate gram_minimal.c which is generated by gram_minimal.y
Bo Peng [Fri, 11 Jun 2021 16:26:39 +0000 (01:26 +0900)]
Update gram_minimal.c which is generated by gram_minimal.y

4 years agoFix orphan process is left when pgpool is going down.
Tatsuo Ishii [Wed, 9 Jun 2021 04:43:44 +0000 (13:43 +0900)]
Fix orphan process is left when pgpool is going down.

When pgpool is going down while follow primary command is ongoing,
some process started by follow primary child process could be left.
The follow primary child calls trigger_failover_command() to run the
follow primary script by using system(3). Unfortunately the process
started by system(3) was not tracked by anyone and it was possible
that the process(es) are left. To reproduce the problem you can edit
the test script (test,sh) for the regression test
075.detach_primary_left_down to change the timeout counter to small
number, say 1. i.e.,
change:
cnt=60
to
cnt=1

To fix the problem, the follow primary child process calls setsid(2)
to set new session id. When Pgpool-II main goes down, the exit handler
kills all the process including the process started by system(3), by
specifying kill(-(follow primary child process pid)).

Also in the initialization of the follow primary process, unmask
signals and assign default behavior to each signal (this should have
been done much earlier).

4 years agoFix pcp_detach_node leaves down node.
Tatsuo Ishii [Tue, 8 Jun 2021 10:06:13 +0000 (19:06 +0900)]
Fix pcp_detach_node leaves down node.

Detaching primary node using pcp_detach_node leaves a standby node
after follow primary command was executed.

This can be reproduced reliably by following steps:

$ pgpool_setup -n 4
$ ./startall
$ pcp_detatch_node -p 11001 0

This is caused by that pcp_recovery_node is denied by pcp child process:

2021-06-05 07:22:17: follow_child pid 6593: LOG:  execute command: /home/t-ishii/work/Pgpool-II/current/x/etc/follow_primary.sh 3 /tmp 11005 /home/t-ishii/work/Pgpool-II/current/x/data3 1 0 /tmp 0 11002 /home/t-ishii/work/Pgpool-II/current/x/data0
2021-06-05 07:22:17: pcp_main pid 6848: LOG:  forked new pcp worker, pid=7027 socket=6
2021-06-05 07:22:17: pcp_child pid 7027: ERROR:  failed to process PCP request at the moment
2021-06-05 07:22:17: pcp_child pid 7027: DETAIL:  failback is in progress

it complains that a failback request is still going. The reason why the
failback is not completed is, find_primary_node_repeatedly() is trying
to acquire the follow primary lock. However the follow primary command
has already acquired the lock and it is waiting for the completion of
the failback request. Thus this is a kind of dead lock situation.

How to solve this?

The purpose of the follow primary lock is to prevent concurrent run of
follow primary command and detach false primary by the streaming
replication check. We cannot throw it away. However it is not always
necessary to acquire the lock by find_primary_node_repeatedly(). If
it does not try to acquire the lock, failover/failback will not be
blocked and will finish soon, thus Req_info->switching flags will be
promptly turned to false.

When a primary node is detached, failover command is called and new
primary is selected. At this point find_primary_node_repeatedly() is
surely needed to run to find the new primary. However, once follow
primary command starts, the primary will not be changed. So my idea
is, find_primary_node_repeatedly() checks whether follow primary
command is running or not. If it is running, just returns the current
primary. Otherwise acquires the lock.

For this purpose, new shared memory variable
Req_info->follow_primary_ongoing was introduced. The flag is set/unset
by follow primary process.

New regression test 075.detach_primary_left_down_node is added.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-June/003916.html

4 years agoFix backend_application_name cannot be changed by reloading.
Tatsuo Ishii [Thu, 3 Jun 2021 05:39:06 +0000 (14:39 +0900)]
Fix backend_application_name cannot be changed by reloading.

The manual explicitly stats that it is possible to change
backend_application_name on the fly by reloading the configuration
file but the code set unnecessary restriction on that.

4 years agoDoc: fix wd_life_point description
Masaya Kawamoto [Wed, 2 Jun 2021 07:53:21 +0000 (07:53 +0000)]
Doc: fix wd_life_point description

The description of the default value was lacked in the Japanese doc.

4 years agoAdd comments to watchdog.c
Tatsuo Ishii [Mon, 31 May 2021 08:00:02 +0000 (17:00 +0900)]
Add comments to watchdog.c

4 years agoAdd comment to Req_info->conn_counter.
Tatsuo Ishii [Mon, 31 May 2021 07:56:10 +0000 (16:56 +0900)]
Add comment to Req_info->conn_counter.

4 years agoFix occasional failure of 028.watchdog_enable_consensus_with_half_votes test.
Tatsuo Ishii [Mon, 31 May 2021 06:19:40 +0000 (15:19 +0900)]
Fix occasional failure of 028.watchdog_enable_consensus_with_half_votes test.

It seems the failure was caused in this scenario:

Testing total nodes: 4. enable_consensus_with_half_of_the_votes: on
shutdown node pgpool2
2021-05-30 07:41:54: main pid 28819: LOG:  stop request sent to pgpool. waiting for termination....done.
shutdown node pgpool1
2021-05-30 07:41:56: main pid 28825: LOG:  stop request sent to pgpool. waiting for termination....done.
Quorum does not exist. Test failed

So the test failed at 07:41:56. In the mean time pgpool.log showed:
2021-05-30 07:41:54: watchdog pid 28569: LOG:  We have lost the cluster leader node "localhost:50008 Linux e1aa95e1fe13"
:
:
2021-05-30 07:41:59: watchdog pid 28569: LOG:  watchdog node state changed from [STANDING FOR LEADER] to [LEADER]
2021-05-30 07:41:59: watchdog pid 28569: LOG:  I am announcing my self as leader/coordinator watchdog node

The quorum was established at 07:41:59. That means the test for quorum
existence was too early.

To fix this, insert "sleep 5" after shutting down pgpool.

4 years agoEnhance watchdog_setup script.
Tatsuo Ishii [Sat, 29 May 2021 08:32:38 +0000 (17:32 +0900)]
Enhance watchdog_setup script.

shutdownall script generated by watchdog_setup shutdowns in the node
number order i.e.: 0, 1, 2...  This causes PostgreSQL backend shutdown
when pgpool0 node went down and node 1, 2... trigger failover event,
which is not necessary in the whole shutdown sequence.  Shutting down
in the reverse order (...2, 1, 0) should prevent this and shorten the
whole shutdown sequence.

Also this should prevent occasional 018.detach_primary and
028.watchdog_enable_consensus_with_half_votes test timeout (they use
watchdog_setup).

4 years agoFix occasional regression test 018.detach_primary error.
Tatsuo Ishii [Fri, 28 May 2021 07:47:45 +0000 (16:47 +0900)]
Fix occasional regression test 018.detach_primary error.

According to the buildfarm log, the test failed at after this:
with this:
$PGPOOL_INSTALL_DIR/bin/pcp_watchdog_info -v -w -p $PCP_PORT
error message was:
ERROR: connection to socket "/tmp/.s.PGSQL.50001" failed with error "No such file or directory"
This suggests that pcp_watchdog_info fails because pcp server has not started yet.

To fix this add wait_for_pgpool_startup before pcp_watchdog_info.

4 years agoFix maximum length of hostnames including domain name.
Tatsuo Ishii [Thu, 27 May 2021 10:15:46 +0000 (19:15 +0900)]
Fix maximum length of hostnames including domain name.

The maximum length of hostnames was 128, which is not incorrect.
Moreover there were multiple places where the maximum length of hostname is defined.
So create unified definition of it in libpcp_ext.h.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-May/003904.html

4 years agoFix watchdog communication race condition.
Tatsuo Ishii [Fri, 21 May 2021 12:11:27 +0000 (21:11 +0900)]
Fix watchdog communication race condition.

Watchdog sends information from the watchdog process to the Pgpool-II
main process using SIGUSR1. To pass detailed messages it uses shared
memory area. First it sets a message to the shared memory area then
sends SIGUSR1 to the main process. The main process received the
signal and the signal handler sets a global variable so that
sigusr1_interrupt_processor() processes it. However it is possible
that while sigusr1_interrupt_processor() is running new signal
arrives. In this case the new signal is caught but the global variable
is set to 0 after sigusr1_interrupt_processor() returns. This means
that the new message is not processed until new signal arrives, which
could cause significant delay before the message was processed.

To fix the problem, sigusr1_interrupt_processor() is repeatedly called
until there's no pending message.

Discussion: https://www.pgpool.net/pipermail/pgpool-hackers/2021-May/003901.html

4 years agoFix for bug:684: Watchdog node status not updating after rebooting.
Muhammad Usama [Thu, 20 May 2021 12:30:02 +0000 (17:30 +0500)]
Fix for bug:684: Watchdog node status not updating after rebooting.

A node should broadcast its status to the whole cluster after
joining the cluster as standby.

4 years agoFix "file name is too long (max 99)" error while creating a tarball.
Bo Peng [Wed, 19 May 2021 13:52:03 +0000 (22:52 +0900)]
Fix "file name is too long (max 99)" error while creating a tarball.

4 years agoAdd release notes.
Bo Peng [Wed, 19 May 2021 12:53:05 +0000 (21:53 +0900)]
Add release notes.