HAProxy configuration for Windows Exchange Server 2016/2019

July 24, 2021

HAProxy is a free, very fast and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. It is particularly suited for very high traffic web sites and powers quite a number of the world’s most visited ones.

graph TD A(mail.bidhankhatri.com.np) --> B[mail1.bidhankhatri.com.np] A --> C[mail2.bidhankhatri.com.np] A --> D[mail3.bidhankhatri.com.np] A --> E[mail4.bidhankhatri.com.np]

Nginx vs HAProxy: I prefer haproxy over open source nginx in this exchange application scenario. Haproxy provides better load balancing, better logging and provides nice stats interface than nginx. TCP load balancing is not available in open source nginx but nginx plus does provide it. Nginx plus is a commercial one so you have to pay to get that feature. We can also monitor and visualize haproxy metrics stats in real time as well as through Grafana as new haproxy is built with the Prometheus exporter.

The term SSL Termination means that you are performing all encryption and decryption at the edge of your network, such as at the load balancer (haproxy). The load balancer strips away the encryption and passes the messages in the clear to your backend servers. This is known as SSL offloading. We can SSL offload at haproxy end and ignored SSL verification to internal backend mail servers. You can also store the CA certificate on the load balancer and reference it. Here, I have used both.
SSL termination has few benefits. These include the following:

You can maintain certificates in fewer places, making your job easier.
You don’t need to expose your servers to the Internet for certificate renewal purposes.
Servers are unburdened from the task of processing encrypted messages, freeing up CPU time.

HAProxy can run in two different modes: TCP or HTTP. When operating in TCP mode, we say that it acts as a layer 4 proxy. In HTTP mode, we say that it acts as a layer 7 proxy.

Layer 7 – Application: application protocols like HTTP, SSH and SMTP
Layer 4 – Transport: data transfer protocols like TCP and UDP

We have used both HTTP & TCP mode here. If you need further details then you can click on this link. https://www.haproxy.com/blog/layer-4-and-layer-7-proxy-mode/

In Exchange 2016 and Exchange 2019, the Client Access services are stateless . In other words, because all processing for the mailbox happens in the backend services on the Mailbox server, it doesn’t matter which instance of the Client Access service in an array of Client Access services receives each individual client request. This means that session affinity is no longer required at the load balancer level. This allows inbound connections to Client Access services to be balanced by using simple load balancing techniques such as DNS round-robin. It also allows hardware load balancing devices to support significantly more concurrent connections.

Each web service has a virtual file to tell its state, called HealthCheck.htm. Let HAProxy use the contents of this file for the server health check. That way it’ll know to redirect clients if one of the services is down, even though the Exchange server in question may still be listening on port 443.

I am using haproxy version 2.0.13 on Ubuntu 20.04.1 LTS. We also tighten our TLS security by allowing only TLS 1.2 & 1.3. By default, Outlook clients 2010/2013 in windows 7 will try to connect exchange mail server through TLS 1.0 & TLS 1.1 so for that you have to do some changes in TLS part on your windows 7 pc. Windows 7 supports TLS 1.1 and TLS 1.2. But these protocol versions are not enabled on it by default. On Windows 8 and higher these protocols are enabled by default. Earlier I have written an article explaining enabling TLS 1.1 & TLS 1.2 in windows 7 pc. You can check this link if you are interested. https://bidhankhatri.com.np/windows/windows7-enable-tls12/

Configuration details:

Only allowed if HTTP host header contains mail.bidhankhatri.com.np www.mail.bidhankhatri.com.np autodiscover.bidhankhatri.com.np www.autodiscover.bidhankhatri.com.np
Exchange Admin Center is only allowed from specific networks.
TLS 1.2 & above are only allowed.
HAProxy metrics are monitored through Prometheus.

PEM file layout for HAProxy

cat certificate.crt intermediates.pem private.key > /etc/haproxy/proxy.pem

Mozilla SSL configuration generator tool.The goal of this tool is to help operational teams with the configuration of TLS. If you are interested then you can have a look. https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.13&config=intermediate

HAProxy config file

vim /etc/haproxy/haproxy.cfg

global
log /dev/log local0

maxconn 8000
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

#If you want to enable TLS 1.0 & TLS 1.1 also then use below line.
#ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:@SECLEVEL=1


# This ciphers should be on production: This should be used if u want to disable TLS1.0 & TLS1.1
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256

tune.ssl.default-dh-param 2048

defaults
log global
mode http
option httplog
option dontlognull

retries 3 # Try to connect up to 3 times in case of failure
timeout connect 5s # 5 seconds max to connect or to stay in queue
timeout http-keep-alive 1s # 1 second max for the client to post next request
timeout http-request 15s # 15 seconds max for the client to send a request
timeout queue 30s # 30 seconds max queued on load balancer
timeout client 300s
timeout server 300s



errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
no option http-use-htx

#---------------------------------------------------------------------
#HAProxy Monitoring Config
#---------------------------------------------------------------------
listen stats
bind :1111
mode http
option forwardfor
option httpclose
stats enable
stats uri /
stats refresh 5s
stats show-legends
stats realm Haproxy\ Statistics
stats auth admin:1password2$


frontend fe_prom_exporter
bind :8004
option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
stats enable
stats uri /stats
stats refresh 10s

#-----------------------
# FrontEnd Begins
# -------------------
#
#
frontend fe_mail
# receives traffic from clients
bind :80
http-response set-header X-Frame-Options SAMEORIGIN
http-response set-header X-Content-Type-Options nosniff
http-response set-header Strict-Transport-Security max-age=63072000


mode http
maxconn 10000

# Allow Exchange Admin Center to certain private network only
acl private_network src XXX.XXX.XXX.XXX/24 XXX.XXX.XXX.XXX/24
acl ecp_req url_beg /ecp
http-request deny if ecp_req !private_network


redirect scheme https code 301 if !{ ssl_fc } # redirect 80 -> 443 (for owa)
bind *:443 ssl crt /etc/haproxy/proxy.pem alpn h2,http/1.1

acl xmail hdr(host) -i mail.bidhankhatri.com.np www.mail.bidhankhatri.com.np autodiscover.bidhankhatri.com.np www.autodiscover.bidhankhatri.com.np

acl autodiscover url_beg /Autodiscover
acl autodiscover url_beg /autodiscover
acl mapi url_beg /mapi
acl rpc url_beg /rpc
acl owa url_beg /owa
acl owa url_beg /OWA
acl eas url_beg /Microsoft-Server-ActiveSync
acl eas url_beg /Microsoft-Server-activeSync
acl ecp url_beg /ecp
acl ews url_beg /EWS
acl ews url_beg /ews
acl oab url_beg /OAB
acl default_for_mail url_beg /

use_backend be_ex2019_owa if xmail owa
use_backend be_ex2019_autodiscover if xmail autodiscover
use_backend be_ex2019_mapi if xmail mapi
use_backend be_ex2019_activesync if xmail eas
use_backend be_ex2019_ews if xmail ews
use_backend be_ex2019_rpc if xmail rpc
use_backend be_ex2019_default if xmail default_for_mail


frontend fe_exchange_imaps
mode tcp
option tcplog
bind :993 name imaps # ssl crt /etc/ssl/certs/exchange_certificate_and_key_nopassword.pem <-- No need, certificate is read straight from the Exchange servers.
default_backend be_exchange_imaps

frontend fe_exchange_smtp
mode tcp
option tcplog
bind :25 name smtp 
default_backend be_exchange_smtp

#------------------------------
# Back-end section
#------------------------------

backend be_ex2019_autodiscover
mode http
# balance source
option httpchk GET /autodiscover/healthcheck.htm
option log-health-checks
http-check expect status 200
server mail1 mail1.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt
server mail2 mail2.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt

backend be_ex2019_mapi
mode http
balance source
option httpchk GET /mapi/healthcheck.htm
option log-health-checks
http-check expect status 200
server mail3 mail3.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt
server mail4 mail4.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt

backend be_ex2019_rpc
mode http
balance source
option httpchk GET /rpc/healthcheck.htm
option log-health-checks
http-check expect status 200
server mail3 mail3.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt

backend be_ex2019_owa
mode http
# balance source
option httpchk GET /owa/healthcheck.htm
option log-health-checks
http-check expect status 200
server mail1 mail1.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt
server mail2 mail2.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt
server mail3 mail3.bidhankhatri.com.np:443 check maxconn 1000 ssl ca-file /etc/ssl/certs/ca-certificates.crt



backend be_ex2019_activesync
mode http
#balance source
option httpchk GET /microsoft-server-activesync/healthcheck.htm
option log-health-checks
http-check expect status 200
server mail2 mail2.bidhankhatri.com.np:443 check ssl verify none
server mail1 mail1.bidhankhatri.com.np:443 check ssl verify none

backend be_exchange_imaps
mode tcp
#option tcplog
balance source
option log-health-checks
server mail1 mail1.bidhankhatri.com.np:993 weight 10 check
server mail2 mail2.bidhankhatri.com.np:993 weight 20 check
server mail3 mail3.bidhankhatri.com.np:993 weight 30 check

backend be_ex2019_ews
mode http
balance source
option httpchk GET /ews/healthcheck.htm
option log-health-checks
http-check expect status 200
server mail2 mail2.bidhankhatri.com.np:443 check ssl verify none
server mail1 mail1.bidhankhatri.com.np:443 check ssl verify none


backend be_ex2019_default
mode http
balance source
server mail2 mail2.bidhankhatri.com.np:443 check ssl verify none
server mail1 mail1.bidhankhatri.com.np:443 check ssl verify none


backend be_exchange_smtp
mode tcp
#balance source
balance source
option log-health-checks
server mail2 mail2.bidhankhatri.com.np:25 weight 20 check
server mail1 mail1.bidhankhatri.com.np:25 weight 10 check

haproxy -c -f /etc/haproxy/haproxy.cfg
    Configuration file is valid

systemctl start haproxy
systemctl enable haproxy

Now check TLS versions

openssl s_client -connect mail.bidhankhatri.com.np:443 -tls1_2    // to check tls1.2
openssl s_client -connect mail.bidhankhatri.com.np:443 -tls1    // to check tls 1.0

Monitoring

In prometheus server end.

vim /etc/prometheus/prometheus.yml

  - job_name: 'haproxy'
    static_configs:
    - targets: ['xxx.xxx.xxx.xxx:8004']

systemctl restart prometheus

HAProxy Real-Time Dashboard

image-center

HAProxy Prometheus Dashboard

image-center

Twitter Facebook LinkedIn

Bidhan Khatri

HAProxy configuration for Windows Exchange Server 2016/2019

HAProxy config file

Monitoring

HAProxy Real-Time Dashboard

HAProxy Prometheus Dashboard

Comments

You May Also Enjoy

Unassigned shards in Elasticsearch 7 and 8

Percona XtraDB Multi-Master Replication cluster setup between 3 nodes

Site-to-Site VPN between Mikrotik router and Ubuntu 22.04 through strongSwan using IPsec IKEv2

Monitor HA Cluster running Pacemaker and Corosync using Prometheus and Grafana using Docker