Postgres-XC 1.1 Documentation
Prev | Up | Chapter 16. Server Setup and Operation | Next

# 16.3. Starting the Postgres-XC Cluster

> Note: The following description applies only to Postgres-XC

Before anyone can access Postgres-XC (or XC in short) database, you must start XC database cluster. As described in the previous chapter, XC consists of various components. Minimum set of components are GTM, GTM-Proxy, Coordinator and Datanode. You must configure and start each of them. Following sections will give you how to configure and start them. `pgxc_clean` and `GTM-Standby` are described in high-availability sections.

## 16.3.1. Creating Databases

> Note: The following description applies only to Postgres-XC

You should initialize each database which composes Postgres-XC database cluster system. Both Coordinator and Datanode has its own database and you should initialize these database. Coordinator holds just database catalog and temporary data store. Datanode holds most of your data. First of all, you should determine how many Coordinators/Datanodes to run and where they should run. It is a good convention that you run a Coordinator where you run a Datanode. In this case, you should run `GTM-Proxy` on the same server too. It simplifies XC configuration and help to make workload of each servers even.

Both Coordinator and Datanode have their own databases, essentially PostgreSQL databases. They are separate and you should initialize them separately.

## 16.3.2. Starting GTM

> Note: The following description applies only to Postgres-XC

GTM provides global transaction management feature to all the other components in Postgres-XC database cluster. Because GTM handles transaction requirements from all the Coordinators and Datanodes, it is highly advised to run this in a separate server.

Before you start GTM, you should decide followings:

Where to run GTM

  Because GTM receives all the request to begin/end transactions and to refer to sequence values, you should run GTM in a separate server. If you run GTM in the same server as Datanode or Coordinator, it will become harder to make workload reasonably balanced.

  Then, you should determine GTM's working directory. Please create this directory before you run GTM.

Listen address and port of GTM

  Next, you should determine listen address and port of GTM. Listen address can be either the IP address or host name which receives request from other component, typically `GTM-Proxy`.

GTM id

  You have a chance to run more than one GTM in one Postgres-XC cluster. For example, if you need a backup of GTM in high-availability environment, you need to run two GTMs. You should give unique GTM id to each of such GTMs. GTM id value begins with one.

When this is determined, you can initialize GTM with the command initgtm, for example:

    $ initgtm -Z gtm -D /usr/local/pgsql/data_gtm

All the parameters related to GTM can be modified in `gtm.conf` located in data folder initialized by `initgtm`.

Then you can start GTM as follows:

    $ gtm -D /usr/local/pgsql/data_gtm

where `-D` option specifies working directory of GTM.

Alternatively, GTM can be started using `gtm_ctl`, for example:

    $ gtm_ctl -Z gtm start -D /usr/local/pgsql/data_gtm

## 16.3.3. Starting GTM-Proxy

> Note: The following description applies only to Postgres-XC

GTM-Proxy is not a mandatory component of Postgres-XC cluster but it can be used to group messages between GTM and cluster nodes, reducing workload and the number of packages exchanged through network.

As described in the previous section, `GTM-Proxy` needs its own listen address, port, working directory and GTM-Proxy ID, which should be unique and begins with one. In addition, you should determine how many working threads to run. You should also use GTM's address and port to start `GTM-Proxy`.

Then, you need first to initialize GTM-Proxy with `initgtm`, for example:

    $ initgtm -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxy

All the parameters related to GTM-Proxy can be modified in `gtm_proxy.conf` located in data folder initialized by `initgtm`.

Then, you can start `GTM-Proxy` like:

    $ gtm_proxy -D /usr/local/pgsql/data_gtm_proxy

where `-D` specifies `GTM-Proxy`'s working directory.

Alternatively, you can start GTM-Proxy using `gtm_ctl` as follows:

    $ gtm_ctl start -Z gtm_proxy -D /usr/local/pgsql/data_gtm_proxy

## 16.3.4. Configuring Datanode

> Note: The following description applies only to Postgres-XC

Before starting Coordinator or Datanode, you must configure them. You can configure Coordinator or Datanode by editing `postgresql.conf` file located at their working directory as you specified by `-D` option in `initdb` command.

Datanode is almost native PostgreSQL with some extension. Additional options in `postgresql.conf` for the Datanode are as follows:

max_connections

  This value is not just a number of connections you expect to each Coordinator. Each Coordinator backend has a chance to connect to all the Datanode. You should specify number of total connections whole Coordinator may accept. For example, if you have five Coordinators and each of them may accept forty connections, you should specify 200 as this parameter value.

max_prepared_transactions

  Even though your application does not intend to issue `PREPARE TRANSACTION`, Coordinator may issue this internally when more than one Datanode are involved. You should specify this parameter the same value as `max_connections`.

pgxc_node_name

  GTM needs to identify each Datanode, as specified by this parameter. The value should be unique and start with one.

port

  Because both Coordinator and Datanode may run on the same server, you may want to assign separate port number to the Datanode.

gtm_port

  Specify the port number of GTM-Proxy, as specified in `-p` option in `gtm_proxy` or `gtm_ctl`.

gtm_host

  Specify the host name or IP address of GTM-Proxy, as specified in `-h` option in `gtm_proxy` or `gtm_ctl`.

## 16.3.5. Configuring Coordinator

> Note: The following description applies only to Postgres-XC

Although Coordinator and Datanode shares the same binary, their configuration is a little different due to their functionalities.

max_connections

  You don't have to take other Coordinator or Datanode into account. Just specify the number of connections the Coordinator accepts from applications.

max_prepared_transactions

  Specify at least total number of Coordinators in the cluster.

pgxc_node_name

  GTM needs to identify each Datanode, as specified by this parameter.

port

  Because both Coordinator and Datanode may run on the same server, you may want to assign separate port number to the Coordinator. It may be convenient to use default value of PostgreSQL listen port.

gtm_port

  Specify the port number of GTM-Proxy, as specified in `-p` option in `gtm_proxy` or `gtm_ctl`.

gtm_host

  Specify the host name or IP address of GTM-Proxy, as specified in `-h` option in `gtm_proxy` or `gtm_ctl`.

max_pool_size

  Coordinator maintains connections to Datanode as a pool. This parameter specifies max number of connections the Coordinator maintains. Specify `max_connection` value of remote nodes as this parameter value.

min_pool_size

  This is the minimum number of Coordinator to remote node connections maintained by the pooler. Typically specify 1.

max_coordinators

  This is the maximum number of Coordinators that can be configured in the cluster. Specify exact number if it is not planned to add more Coordinators while cluster is running, or greater, if it is desired to dynamically resize cluster. It costs about 140 bytes of shared memory per slot.

max_datanodes

  This is the maximum number of Datanodes configured in the cluster. Specify exact number if it is not planned to add more Datanodes while cluster is running, or greater, if it is desired to dynamically resize cluster. It costs about 140 bytes of shared memory per slot.

enforce_two_phase_commit

  Enforce the usage of two-phase commit on transactions involving ON COMMIT actions or temporary objects. Usage of autocommit instead of two-phase commit may break data consistency so use at your own risk.

## 16.3.6. Starting Datanodes

Now you can start central component of Postgres-XC, Datanode and Coordinator. If you're familiar with starting PostgreSQL database server, this step is very similar to PostgreSQL.

You can start a Datanode as follows:

    $ postgres --datanode -D /usr/local/pgsql/data

`--datanode` specifies `postgres` should run as a Datanode. You may need to specify `-i` `postgres` to accept connection from TCP/IP connections or edit `pg_hba.conf` if cluster uses nodes among several servers.

## 16.3.7. Starting Coordinators

> Note: The following description applies only to Postgres-XC

You can start a Coordinator as follows:

    $ postgres --coordinator -D /usr/local/pgsql/Datanode

`--coordinator` specifies `postgres` should run as a Coordinator. You may need to specify `-i` `postgres` to accept connection from TCP/IP connections or edit `pg_hba.conf` if cluster uses nodes among several servers.

## 16.3.8. Server Start-up Failures

There are several common reasons the server might fail to start. Check the server's log file, or start it by hand (without redirecting standard output or standard error) and see what error messages appear. Below we explain some of the most common error messages in more detail.

    LOG:  could not bind IPv4 socket: Address already in use
    HINT:  Is another postmaster already running on port 5432? If not, wait a few seconds and retry.
    FATAL:  could not create TCP/IP listen socket

This usually means just what it suggests: you tried to start another server on the same port where one is already running. However, if the kernel error message is not `Address already in use` or some variant of that, there might be a different problem. For example, trying to start a server on a reserved port number might draw something like:

    $ postgres -p 666
    LOG:  could not bind IPv4 socket: Permission denied
    HINT:  Is another postmaster already running on port 666? If not, wait a few seconds and retry.
    FATAL:  could not create TCP/IP listen socket

A message like:

    FATAL:  could not create shared memory segment: Invalid argument
    DETAIL:  Failed system call was shmget(key=5440001, size=4011376640, 03600).

probably means your kernel's limit on the size of shared memory is smaller than the work area Postgres-XC is trying to create (4011376640 bytes in this example). Or it could mean that you do not have System-V-style shared memory support configured into your kernel at all. As a temporary workaround, you can try starting the server with a smaller-than-normal number of buffers (shared_buffers). You will eventually want to reconfigure your kernel to increase the allowed shared memory size. You might also see this message when trying to start multiple servers on the same machine, if their total space requested exceeds the kernel limit.

An error like:

    FATAL:  could not create semaphores: No space left on device
    DETAIL:  Failed system call was semget(5440126, 17, 03600).

does not mean you've run out of disk space. It means your kernel's limit on the number of System V semaphores is smaller than the number Postgres-XC wants to create. As above, you might be able to work around the problem by starting the server with a reduced number of allowed connections (max_connections), but you'll eventually want to increase the kernel limit.

If you get an "illegal system call" error, it is likely that shared memory or semaphores are not supported in your kernel at all. In that case your only option is to reconfigure the kernel to enable these features.

Details about configuring System V IPC facilities are given in Section 16.4.1.

## 16.3.9. Client Connection Problems

Although the error conditions possible on the client side are quite varied and application-dependent, a few of them might be directly related to how the server was started. Conditions other than those shown below should be documented with the respective client application.

    psql: could not connect to server: Connection refused
            Is the server running on host "server.joe.com" and accepting
            TCP/IP connections on port 5432?

This is the generic "I couldn't find a server to talk to" failure. It looks like the above when TCP/IP communication is attempted. A common mistake is to forget to configure the server to allow TCP/IP connections.

Alternatively, you'll get this when attempting Unix-domain socket communication to a local server:

    psql: could not connect to server: No such file or directory
            Is the server running locally and accepting
            connections on Unix domain socket "/tmp/.s.PGSQL.5432"?

The last line is useful in verifying that the client is trying to connect to the right place. If there is in fact no server running there, the kernel error message will typically be either `Connection refused` or `No such file or directory`, as illustrated. (It is important to realize that `Connection refused` in this context does not mean that the server got your connection request and rejected it. That case will produce a different message, as shown in Section 18.4.) Other error messages such as `Connection timed out` might indicate more fundamental problems, like lack of network connectivity.

Prev | Home | Next
Creating a Database Cluster | Up | Managing Kernel Resources