Monday, Nov 11, 2019
High Availability Web Application Setup
Today, we’re diving into the world of web applications with a focus on High Availability (HA). HA is key in ensuring that our applications remain operational even in the face of inevitable hiccups like server crashes or network issues.
We’ll be setting up our web application on Windows, focusing on the usage of Patroni for our PostgreSQL cluster, and Redis as our cache database - we will not cover load balancing and web server setup in this article. While most of the articles on this topic cover Linux installations, the Windows environment comes with a set of challenges that we need to solve. Ready to nerd out? Let’s dive in!
Patroni is an open source cluster solution, which takes care of automatic failover and high availability of your PostgreSQL database. We are using it in a majority of HA scenarios, as it is easy to use, especially in [Linux environments[(https://arctype.com/blog/postgres-patroni/)]. However, many people find it hard to deploy in Windows environments due to the many moving parts that it requires.
Patroni Environment Setup (PES)
PES makes the deployment of Patroni on Windows easy, fast, and user-friendly. It provides a convenient graphical interface (we are using Windows after all) to create cluster topology. You only need to set the number of nodes and their essential properties – like name, role (database/etcd/failover) – and you are done.
PES comes with one of the best setup guides I have worked with. You literally have to follow the procedure described there step by step, and, while there are many steps, we never had any major problems with it. Just run it on each of the servers that will be members of your cluster. You need to have at least three servers to avoid split brain scenario.
Among other things, PES installs etcd, a distributed key value store that is used by all database instances to track their status. By default, it syncs over the port 2379, so you can check who is the cluster leader by sending a GET request to http://your.server.ip.address:2379/v2/keys/service/pgcluster/leader.
By default, Patroni uses vip-manager service running on all cluster machines to manage a virtual IP based on the state kept in etcd, so that the IP could be used as a part of a database connection string. In most of our production scenarios, we used an external load balancer that directs traffic to the target database address instead of a vip-manager, especially if the servers were placed in different network segments.
Performing configuration changes and server restarts in a cluster requires a special approach. If you need to make database configuration changes, run the following from the folder where PES is installed:
C:\PES\patroni> python patronictl.py -c patroni.yaml edit-config
More on setting up PostgreSQL parameters in Patroni cluster can be found here. You might get a warning that the EDITOR environment variable has not been set. To solve this, run the following line from the elevated command prompt:
setx EDITOR notepad.exe /M
After this, you can change the parameters in the YAML file that is displayed using the edit-config command.
These types of configuration changes require restarting either individual servers or the whole cluster. Here is how to do it:
C:\PES\patroni> python patronictl.py -c patroni.yaml restart pgcluster
In this example, “pgcluster” is the name of the cluster to be restarted. More information on changing parameters can be found here.
To change a cluster leader, use the following:
python patronictl.py -c patroni.yaml switchover
More information on switchover and failover can be found here.
Redis is often our tool of choice for storing cache objects. However, as we are using Windows, we will have to download the installation package from GitHub. Here, we will describe a setup with 2 slave nodes, 3 sentinels and quorum 2, using 3 physical servers (you may use the same physical servers that are running PostgreSQL nodes, or choose separate nodes).
During the installation process, you will have to open the requested firewall ports. Configuration file is placed in C:\Program Files\Redis\redis.windows-service.conf, and no changes need to be done on it on the master node.
After the installation is finished, you need to create a new configuration file (using administrative privileges). I called it sentinel.windows-service.conf, and put the following content in it:
sentinel monitor db1 10.10.10.10 6379 2 sentinel down-after-milliseconds db1 5000 sentinel failover-timeout db1 45000 sentinel config-epoch db1 395 sentinel leader-epoch db1 395 port 26379 dir "c:\\Program Files\\Redis"
In this example, “db1” is the name of the master node, and 10.10.10.10 is its IP address. This file will be used as-is on the other two servers. After you install Redis on slave nodes, it will be automatically edited to add the following lines, containing IP addresses of slave nodes:
# Generated by CONFIG REWRITE loglevel verbose sentinel known-slave db1 10.10.10.11 6379 sentinel known-slave db1 10.10.10.12 6379 sentinel known-sentinel db1 10.10.10.12 26379 3aeaeb5de96ac6ecc07508cf05f3b7557c9ea138 sentinel known-sentinel db1 10.10.10.11 26379 6ed0acb5a516a3c4627c952567721c1259e6d80f sentinel current-epoch 395
After you finish editing the configuration file, run command prompt as admin, go to the Redis folder and run
redis-server.exe --service-install sentinel.windows-service.conf --service-name RedisSentinel --loglevel verbose --sentinel
Now you have an active master and sentinel on a single node.
You will do the same on slave nodes. Simply add the following line to the end of the windows-service.conf file:
slaveof 10.10.10.10 6379
This will tell each slave node where to find its master.
To check Redis replication status, start redis-cli and type
The output will be similar to this:
c:\Program Files\Redis>redis-cli 127.0.0.1:6379> info replication # Replication role:master connected_slaves:2 slave0:ip=10.10.10.11,port=6379,state=online,offset=139471,lag=0 slave1:ip=10.10.10.12,port=6379,state=online,offset=139197,lag=0 master_repl_offset:139471 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:2 repl_backlog_histlen:139470 127.0.0.1:6379>
To check sentinel status, type
redis-cli -p 26379
The commands you will want to use are sentinel masters and sentinel slaves db1.
That’s it - you have successfully set up a high availability web application on Windows using Patroni, PostgreSQL, and Redis.
Like all things tech, there’s always more to learn. High availability is a deep topic with a lot of nuances depending on your specific use case. But I hope this guide gives you a solid foundation to start from.
Remember to always test your setup thoroughly to ensure everything works as expected. Happy coding!