What is TCP Handshake?
A TCP handshake is the process used to establish a reliable TCP connection between a client and a server before any actual data is transferred. It’s also called a three-way handshake because it involves three steps exchanged between the two sides. SYN, SYN-ACK and ACK.

In this example, I will send “hello” data from client(10.32.10.21) to the server(10.32.3.69), which is listening on port 9600. We will analyze the packet flow between the client and the server. We will be using the ncat command on the client side to send the data and tcpdump on the server side to monitor the TCP packets.

For reference:
TCP Flags:

Step 1: Testing TCP Connectivity with ncat

echo "hello" | ncat 10.32.3.69 9600

Tcpdump output on server side.

1766716972.073906 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [S], seq 2335128560, win 64240, options [mss 1460,sackOK,TS val 2523678873 ecr 0,nop,wscale 9], length 0 
1766716972.074026 IP 10.32.3.69.9600 > 10.32.10.21.57270: Flags [S.], seq 1265705932, ack 2335128561, win 28960, options [mss 1460,sackOK,TS val 3672955275 ecr 2523678873,nop,wscale 7], length 0 
1766716972.074184 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [.], ack 1, win 126, options [nop,nop,TS val 2523678874 ecr 3672955275], length 0

First line, Client initiates TCP connection to port 9600. SYN [S] Flags can be seen as the 1st packet.
Second line, Server acknowledges SYN and sends its own sequence number. SYN+ACK [S.] in 2nd packet.
Third line, Client sends the ACK [.] packet to the server.
Now this completes the 3 way handshake. Connection is now open for data transfer and you will see TCP state as “ESTABLISHED” in the linux box.

If you check through ss command then you will see “ESTAB”. This confirms that the session between client and the server is active.

ss -pnt | grep 9600 
ESTAB 0 0 10.32.3.69:9600 10.32.10.21:39518 users:(("java",pid=33120,fd=84))

Now below line shows how data transfer packet looks like.

1766716972.075239 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [P.], seq 1:7, ack 1, win 126, options [nop,nop,TS val 2523678875 ecr 3672955275], length 6 
1766716972.075288 IP 10.32.3.69.9600 > 10.32.10.21.57270: Flags [.], ack 7, win 227, options [nop,nop,TS val 3672955276 ecr 2523678875], length 0

First line, Client sends data, [P.] = PSH + ACK push 6 bytes to server immediately. Payload size is 6 bytes.
Second line, Server Ack[.] it and confirms that I have received the 6 bytes (5 bytes for hello and 1 byte for newline) data. By default, echo adds a newline (\n) at the end. Notice the server’s ACK number is now 7. In TCP, the ACK number tells the sender the next sequence number the receiver expects. Since the client sent 6 bytes starting at sequence 1, the server ACKs with 7 (1 + 6 = 7), effectively saying: “I have received everything up to byte 6; please send byte 7 next.”

Usually server will also send data back to the clients and you will see similary kind of packets. This is actually a normal TCP data exchange packet flow between client and the server.

After the data exchange is completed, TCP teardown process will begin.

00:38:40.501716 IP 10.32.10.21.55952 > 10.32.3.69.9600: Flags [F.], seq 7, ack 1151, win 131, options [nop,nop,TS val 2522122312 ecr 3671385878], length 0 
00:38:40.501748 IP 10.32.3.69.9600 > 10.32.10.21.55952: Flags [.], ack 8, win 227, options [nop,nop,TS val 3671398702 ecr 2522122312], length 0
1766716972.075352 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [F.], seq 7, ack 1, win 126, options [nop,nop,TS val 2523678875 ecr 3672955275], length 0 
1766716972.081386 IP 10.32.3.69.9600 > 10.32.10.21.57270: Flags [F.], seq 1, ack 8, win 227, options [nop,nop,TS val 3672955282 ecr 2523678875], length 0 
1766716972.081531 IP 10.32.10.21.57270 > 10.32.3.69.9600: Flags [.], ack 2, win 126, options [nop,nop,TS val 2523678881 ecr 3672955282], length 0

First Packet, Client sends FIN + ACK [F.] which means, I have finished sending my data. I want to close my side of the connection. and it also acknowledges all the data received from the server side.
Second Packet, I received your request to close. I’m acknowledging it. Server ACK [.] the client FIN correctly.
Third Packet, Client sends a FIN to the server.
Fourth packet, Server also sends its FIN and ACK to the client.
Fifth packet, The client sends the final ACK.
Now the TCP connection is fully closed. This is a graceful connection close. You can observe a similar TCP teardown packet flow whenever a connection is closed.

Now to test the RST behaviour. Lets stop the service listening on Port 9600 on the server and we send the request on the same port from the client.

echo "hello" | ncat 10.32.3.69 9600
Ncat: Connection refused.

If we do that, the client will receive a “Connection refused” response from the server. If you observe the packet flow, it will look like this or a similar pattern. This is a connection rejection from the server side.

1766895916.006405 IP 10.32.10.21.56518 > 10.32.3.69.9600: Flags [S], seq 3541810289, win 64240, options [mss 1460,sackOK,TS val 2702622704 ecr 0,nop,wscale 9], length 0 
1766895916.006514 IP 10.32.3.69.9600 > 10.32.10.21.56518: Flags [R.], seq 0, ack 3541810290, win 0, length 0

1st Packet, client send a SYN packet.
2nd Packet, RST + ACK. This is the “Connection Refused” signal. The server acknowledges the request but immediately kills it with a Reset.

This tcpdump output shows a connection attempt that was immediately rejected with a TCP RST.

This kind of packet flow analysis can be very helpful during application troubleshooting from the network-level. Usually, if we see RST packets, it may be because the service that should be listening on that port is no longer listening or is not working properly. In such cases, the kernel sends an RST. RST packets can also be generated when the maximum number of connections is reached, causing new connection attempts to be forcefully reset. They may also occur when a service is killed, an incorrect protocol is used, or an RST is sent from the client side. Similarly, RST packets may be sent by firewall rules if such rules are configured.

In some cases, we want to allow connections only from specific networks and block connections from outside. For security reasons, we can configure firewall rules to send RST packets in those scenarios.

iptables -A INPUT -p tcp --dport 9600 -j REJECT --reject-with tcp-reset

This iptables rule will reject incoming request with RST on the port 9600.

The table below shows the different states of TCP communication from the client’s perspective while sending data. It describes which types of packets are sent and received, and what each of them means.

TCP Client State → Client Perspective (Sending Data to Server), Packet Flow (Sequential View)

When a client initiates a TCP connection and sends data, the connection can go through multiple states:

Handshake

Data Transfer

Connection Teardown

TCP connections often move so fast (milliseconds) that catching every state in real-time can be challenging.

ES_API_KEY="<your ES KEY>"
RABBIT_USER="admin"
RABBITT_PASS="pass"

main.yml file.

Output of the script:

2025-12-10 13:22:58 [INFO] Fetching queues from http://srv01.abc.com:15672 
2025-12-10 13:22:58 [INFO] Successfully fetched 365 queues from http://srv01.abc.com:15672 
2025-12-10 13:23:41 [INFO] Fetching queues from http://srv02.abc.com:15672 
2025-12-10 13:23:41 [ERROR] Failed to fetch queues from http://srv02.abc.com:15672: 401 Client Error: Unauthorized for url: http://srv02.abc.com:15672/api/queues 
2025-12-10 13:23:41 [INFO] Fetching queues from http://srv03.abc.com:15672 
2025-12-10 13:23:41 [INFO] Successfully fetched 28 queues from http://srv03.abc.com:15672

Screenshot from the Kibana.

Be cautious when using Docker on the same host as the Jenkins master—consider using a remote Docker daemon or sandboxing strategies to avoid privilege escalation. I will show you both approaches here.

First, spin up Jenkins. We will run Jenkins as a container. The Jenkins container runs as UID 1000 by default, so your host Jenkins directory must be writable by UID 1000.

mkdir /var/Docker/jenkins
chown -R 1000:1000 /var/Docker/jenkins
docker network create jenkins

If UID 1000 does not exist on the host machine, reserve UID 1000 for the Jenkins user.

sudo useradd -u 1000 -m -s /bin/bash jenkins

Run the Jenkins container with persistent storage, Docker access, jdk21 and port mappings.

docker run -d \
  --name jenkins \
  --network jenkins \
  --restart=always \
  -p 8080:8080 -p 50000:50000 \
  -v /var/Docker/jenkins:/var/jenkins_home \
  -v /var/run/docker.sock:/var/run/docker.sock \
  jenkins/jenkins:2.504.1-lts-jdk21

docker ps                                                                                                                                                                      
CONTAINER ID   IMAGE                               COMMAND                  CREATED        STATUS        PORTS                                                                                      NAMES
0970e84f4ffb   jenkins/jenkins:2.504.1-lts-jdk21   "/usr/bin/tini -- /u…"   1 hours ago   Up 2 minutes   0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp   jenkins

Get the initial admin password for jenkins

docker exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword
de9ae273099f4a09979e4c54fe0b1a20

Enter that password at Jenkins URL http://localhost:8080.
Install suggested plugins » Create your first admmin user. Dashboard » Manage Jenkins » System configuration » Nodes » clouds » Install a plugin ( Docker ) » Install » Tick on Restart Jenkins when installation is complete.
Now jenkins will restart enabling the docker plugin.

Docker cloud configuration

setting up Jenkins to dynamically provision Docker containers as build agents.
Manage Jenkins » Nodes » Clouds » New cloud
Give a name to your Docker cloud. I used ‘docker_cloud’ and select ‘Docker Type’ from the options.
Once you click on “create” now you will see 2 options:

1. Docker Cloud details
2. Docker Agent templates

1. Docker Cloud details:
This is where you define the Docker environment that Jenkins will use to spin up containers.
Name: A unique identifier for this Docker cloud.
Docker Host URI: URI to connect to Docker. E.g., unix:///var/run/docker.sock or tcp://…
Server Credentials: (Optional) TLS certs if Docker is secured with TLS (rare on local setups). In our case, we are not using any credentials.
Test Connection: Verifies Jenkins can reach the Docker daemon.
I’m trying to connect to the Docker daemon running on the same host where the Jenkins container is running, so I’m using docker.sock to connect to it.

If jenkins ia able to connect docker then you will see the docker version and api version detials. This confirms that Jenkins is correctly detecting the Docker engine version and API version when connecting to the Docker socket.
Version = 20.10.7, API Version = 1.41 Click on “Enabled” Leave other section as it is.
If you see errors like “java.net.BindException: Permission denied “ then you should add jenkins user to host docker group and restart it.
Get the docker group id from the host machine.

getent group docker
docker:x:986:bdn

docker exec -u0 -it jenkins bash
groupadd -g 986 docker
usermod -aG docker jenkins

docker restart jenkins

2. Docker Agent Templates:
Each template defines how a container should be spun up to act as a Jenkins agent.
Labels: Labels used by jobs/pipelines to request this agent type. Click on Enabled.
Name: Internal name (can be anything). use same as labels.
Docker Image: Image to use (e.g., jenkins/inbound-agent, maven:3.8-jdk-11 or your custom immage)
Instance Capacity: Max number of containers Jenkins can launch from this template.
Remote File System Root: Where the Jenkins workspace lives inside the container (e.g., /home/jenkins).
Usage: Use this node as much as possible
Connect method: Attach Docker container.
(All the images should have Java installed. Use the same Java version as the master. The Docker image CMD must either be empty or simply keep the container running indefinitely, e.g., /bin/bash. The Jenkins remote agent code will be copied into the container and then run using the Java installed in the container.)
Pull strategy: Never pull
(Pull all images every time if you’re pulling them from Docker Hub. If you have a local image, choose ‘Never pull.’)
In my case I’m choosing ‘Never pull’ as my image is a custom one.

Dashboard » Manage Jenkins » Clouds » docker_cloud » Configure

Here is the Dockerfile for a custom image which includes Java 21(OpenJDK) runtime, python 3 and ansible all based on Alpine linux.

FROM alpine:3.19

# Install base packages
RUN apk update && apk add --no-cache \
    bash \
    curl \
    openssh \
    git \
    python3 \
    py3-pip \
    ansible \
    libc6-compat \
    sshpass

RUN mkdir -p /opt/java && \
    curl -L -o /tmp/openjdk.tar.gz https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.2+13/OpenJDK21U-jdk_x64_alpine-linux_hotspot_21.0.2_13.tar.gz && \
    tar -xzf /tmp/openjdk.tar.gz -C /opt/java && \
    mv /opt/java/jdk-* /opt/java/openjdk && \
    rm -rf /tmp/openjdk.tar.gz


# Install Java 21 (Alpine-specific Temurin build)
ENV JAVA_HOME=/opt/java/openjdk
ENV PATH="$JAVA_HOME/bin:$PATH"


# Verify that Java has been installed correctly
RUN java -version

# Create non-root user
RUN adduser -D jenkins
USER jenkins

WORKDIR /home/jenkins

# Default command
CMD ["/bin/bash"]

docker build -t jenkins-agent:alp-j21-py3-ans .

Now create a new freestyle job

I have created one freestyle job which is “Hello-World”
Click on “Configuration” and in General, click “Restrict where this project can be run” field. put “jenkins-agent”

In build section, add build step, Execute shell echo “Hello World” Now run the job.

Here, what will happen is that Jenkins dynamically launches new containers as agents (jenkins-agent:alp-j21-py3-ans) on demand to run jobs(“Hello-World”). These containers are ephemeral and automatically destroyed after the job finishes. It’s a best-practice approach in Jenkins to use ephemeral Docker containers for running jobs.

That’s it. Now you can run as much as ephemeral containers to run your Jenkins Jobs.
If you want to run the job on the same host where Docker is installed, then the /var/run/docker.sock can be used. However, if you want to run the job on a remote server container, you first need to expose Docker over TCP. After that, your Jenkins controller instance can connect to the remote Docker server via TCP.

You can either modify the Docker configuration to enable TCP access, or, if you prefer not to touch the existing docker config, you can use an image like socat which will act as a relay between TCP and the Unix socket, which will expose Docker over TCP without altering the configuration. To connect from Jenkins controller you should define tcp://<RemoteIP:2375> in Docker Host URI field while configuring cloud.
Execute in remote server.

docker run -d \                                                                                                                                                                      
  --name socat-docker \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -p 2375:2375 \
  alpine/socat \
  tcp-listen:2375,fork,reuseaddr unix-connect:/var/run/docker.sock

docker ps                                                                                                                                                                     
CONTAINER ID   IMAGE                               COMMAND                  CREATED        STATUS        PORTS                                                                                      NAMES
9a3510394186   alpine/socat                        "socat tcp-listen:23…"   1 days ago     Up 1 second   0.0.0.0:2375->2375/tcp, :::2375->2375/tcp

Important: Make sure Docker is exposed securely to prevent unauthorized access. The above setup exposes your Docker daemon without authentication, which means anyone who can access port 2375 can control your Docker engine (run containers, delete images, etc.). Therefore, it is strongly recommended to configure TLS for secure access.

We will set up mutual TLS (mTLS) using two intermediate CAs—one for server certificates and another for client certificates—both signed by the same root CA.


Steps Overview:

1. Create Root CA
2. Create Two Intermediate CAs (one for server, one for client)
3. Generate Server Certificate (signed by Server Intermediate CA)
4. Generate Client Certificate (signed by Client Intermediate CA)
5. Start OpenSSL Server with mTLS
6. Test with curl

Step 1: Create the Root CA
A Root Certificate Authority (Root CA) is the top-level trust anchor in a Public Key Infrastructure (PKI) that issues and signs digital certificates. It is self-signed and is used to verify and establish trust in all certificates issued under it.
First commmand generate the private key as a file rootCA.key and second will creates a self-signed Root CA certificate (rootCA.crt) valid for 10 years.

openssl genrsa -out rootCA.key 4096 
openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 3650 -out rootCA.crt -subj "/C=US/ST=State/L=City/O=RootCA/OU=IT/CN=root-ca"  

🔹 genrsa → Generates a RSA private key.
🔹 -out rootCA.key → Saves the key as rootCA.key.
🔹 4096 → Specifies the key size (4096 bits) for strong security.
🔹 req → Runs OpenSSL’s Certificate Signing Request (CSR) command.
🔹 -x509 → Creates a self-signed certificate instead of a CSR.
🔹 -new → Generates a new certificate request.
🔹 -nodes → No password protection on the private key (useful for automation).
🔹 -key rootCA.key → Uses the previously generated private key.
🔹 -sha256 → Uses SHA-256 as the hashing algorithm for the signature.
🔹 -days 3650 → Sets the certificate’s validity period to 3650 days (10 years).
🔹 -out rootCA.crt → Saves the Root CA certificate as rootCA.crt.
🔹 -subj “/C=US/ST=State/L=City/O=RootCA/OU=IT/CN=root-ca”
/C=US → Country (US)
/ST=State → State name
/L=City → City name
/O=RootCA → Organization (Root CA)
/OU=IT → Organizational Unit (IT)
/CN=root-ca → Common Name (Root CA’s name)

Step 2: Create Two Intermediate CAs
Create a Server Intermediate Certificate that is signed by the Root CA.
CA:TRUE in Basic Constraints is required for a CA certificate, so let’s create a file to add Basic Constraints to the intermediate certificates.
vim intermediate_ca_ext.cnf

basicConstraints=CA:TRUE,pathlen:0
keyUsage = critical, keyCertSign, cRLSign
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer

Server Intermediate CA

openssl genrsa -out server-intermediateCA.key 4096
openssl req -new -key server-intermediateCA.key -out server-intermediateCA.csr -subj "/C=US/ST=State/L=City/O=ServerCA/OU=IT/CN=server-intermediate-ca"
openssl x509 -req -in server-intermediateCA.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out server-intermediateCA.crt -days 3650 -sha256 -extfile intermediate_ca_ext.cnf

First, This private key will be used to sign server certificates issued by this intermediate CA.
Second, This step generates a CSR that will be signed by the Root CA.
Third, This step creates a Server Intermediate CA certificate, signed by the Root CA.
🔹 server-intermediateCA.key → Private key for the Server Intermediate CA.
🔹 server-intermediateCA.csr > → CSR for the Server Intermediate CA.
🔹 server-intermediateCA.crt > → Intermediate CA certificate signed by the Root CA.
🔹 rootCA.srl > → Serial number file for tracking issued certificates.

Client Intermediate CA
Create a Client Intermediate Certificate Authority that is signed by the Root CA.

openssl genrsa -out client-intermediateCA.key 4096
openssl req -new -key client-intermediateCA.key -out client-intermediateCA.csr -subj "/C=US/ST=State/L=City/O=ClientCA/OU=IT/CN=client-intermediate-ca"
openssl x509 -req -in client-intermediateCA.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out client-intermediateCA.crt -days 3650 -sha256 -extfile intermediate_ca_ext.cnf

🔹 client-intermediateCA.key → Private key for the Client Intermediate CA.
🔹 client-intermediateCA.csr > → CSR for the Client Intermediate CA.
🔹 client-intermediateCA.crt > → Intermediate CA certificate signed by the Root CA.
🔹 rootCA.srl > → Serial number file for tracking issued certificates.

Step 3: Generate Server Certificate (Signed by Server Intermediate CA)
Generates a server certificate signed by the Server Intermediate Certificate Authority (CA).

openssl genrsa -out server.key 4096
openssl req -new -key server.key -out server.csr -subj "/C=US/ST=State/L=City/O=MyServer/OU=IT/CN=server.local"
openssl x509 -req -in server.csr -CA server-intermediateCA.crt -CAkey server-intermediateCA.key -CAcreateserial -out server.crt -days 3650 -sha256 -extfile intermediate_ca_ext.cnf

Server Certificate Chain:

Root CA
  ├── Server Intermediate CA
       ├── Server Certificate (server.crt)

Step 4: Generate Client Certificate (Signed by Client Intermediate CA)
Generates a client certificate signed by the Client Intermediate Certificate Authority (CA).

openssl genrsa -out client.key 4096
openssl req -new -key client.key -out client.csr -subj "/C=US/ST=State/L=City/O=MyClient/OU=IT/CN=client.local"
openssl x509 -req -in client.csr -CA client-intermediateCA.crt -CAkey client-intermediateCA.key -CAcreateserial -out client.crt -days 3650 -sha256

Client Certificate Chain:

Root CA
  ├── Client Intermediate CA
       ├── Client Certificate (client.crt)

Check the Basic Constraints: CA:TRUE in the Basic Constraints extension of an X.509 certificate indicates that the certificate is a Certificate Authority (CA), so it must be present in both the root and intermediate certificates.

openssl x509 -in server-intermediateCA.crt -noout -text | grep -A1 "Basic Constraints"              

            X509v3 Basic Constraints:
                CA:TRUE, pathlen:0

openssl x509 -in client-intermediateCA.crt -noout -text | grep -A1 "Basic Constraints" 

            X509v3 Basic Constraints:
                CA:TRUE, pathlen:0

Create CA bundle: Concating both Intermediate certs and root CA into a single file.

cat server-intermediateCA.crt client-intermediateCA.crt rootCA.crt > ca-bundle.crt

Verify certificates:

openssl verify -CAfile ca-bundle.crt server.crt
server.crt: OK

openssl verify -CAfile ca-bundle.crt client.crt
client.crt: OK

The output of these commands indicates that both server.crt and client.crt are valid and properly signed by a Certificate Authority (CA).

Step 5: Start OpenSSL Server with mTLS

Run the server, requiring client authentication:

openssl s_server -accept 8443 -cert server.crt -key server.key -CAfile ca-bundle.crt -Verify 2

🔹 s_server → Starts an SSL/TLS server.
🔹 -accept 8443 → The server listens on port 8443 for incoming TLS connections.
🔹 -cert server.crt → Uses server.crt as the server’s certificate.
🔹 -key server.key → Uses server.key as the private key for the server.
🔹 -CAfile ca-bundle.crt → Specifies a file containing trusted CA certificates (intermediate + root CAs) to verify client certificates.
🔹 -Verify 2 → Requires mutual TLS (mTLS), meaning: The server will request a client certificate. The depth 2 means it will accept client certificates issued up to 2 levels deep in the CA hierarchy (e.g., Root CA → Intermediate CA → Client Certificate).

Step 6: Test with curl

Use curl with client authentication:

curl -v --cert client.crt --key client.key --cacert ca-bundle.crt https://server.local:8443

🔹 -v → Enables verbose mode, showing detailed SSL handshake and request/response details.
🔹 –cert client.crt → Specifies the client certificate to authenticate with the server.
🔹 –key client.key → Specifies the private key for the client certificate.
🔹 –cacert ca-bundle.crt → Specifies the trusted CA certificate bundle to verify the server’s certificate.
🔹 https://server.local:8443 → The target URL of the server, using HTTPS on port 8443.

openssl s_server -accept 8443 -cert server.crt -key server.key -CAfile full-chain.crt -Verify 2

verify depth is 2, must return a certificate
Using auto DH parameters
ACCEPT

depth=2 C = US, ST = State, L = City, O = RootCA, OU = IT, CN = root-ca
verify return:1
depth=1 C = US, ST = State, L = City, O = ClientCA, OU = IT, CN = client-intermediate-ca
verify return:1
depth=0 C = US, ST = State, L = City, O = MyClient, OU = IT, CN = client.local
verify return:1
Write BLOCK
GET / HTTP/1.1
Host: server.local:8443
User-Agent: curl/8.7.1
Accept: */*

DONE
shutting down SSL
CONNECTION CLOSED
ACCEPT

First, Server verified the Root CA (which issued the intermediate CA). The server verified the Client Intermediate CA. The client’s certificate was issued by this intermediate CA.And, verify return:1 means verification was successful.

curl -v --cert client.crt --key client.key --cacert full-chain.crt https://server.local:8443

* Host server.local:8443 was resolved.
* IPv6: (none)
* IPv4: 127.0.0.1
*   Trying 127.0.0.1:8443...
* Connected to server.local (127.0.0.1) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: full-chain.crt
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, CERT verify (15):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=State; L=City; O=MyServer; OU=IT; CN=server.local
*  start date: Mar 10 17:18:54 2025 GMT
*  expire date: Mar  8 17:18:54 2035 GMT
*  common name: server.local (matched)
*  issuer: C=US; ST=State; L=City; O=ServerCA; OU=IT; CN=server-intermediate-ca
*  SSL certificate verify ok.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: server.local:8443
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off

Curl command successfully established a mutual TLS (mTLS) connection with the server. mTLS was successfully established. The server authenticated itself to the client.The client authenticated itself to the server.A secure session was created, and data exchange began.

To reassign all unassigned shards in Elasticsearch, you can use the following steps:

STEPS:

1. Check the status of unassigned shards
2. Check cluster allocation explanation
3. Automatic reallocation
4. Force reallocation of shards

Check current cluster status

curl -XGET -u elastic:password http://172.16.0.1:9200/_cluster/health?pretty=true

{
  "cluster_name" : "devcluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 24,
  "active_shards" : 24,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 26,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 48.0
}

Check the status of unassigned shards:
First, you need to identify which shards are unassigned. You can do this by using the _cat/shards API. This will give you a list of all shards in your cluster along with their status.

curl -XGET -u elastic:password http://172.16.0.1:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason

.monitoring-kibana-7-2024.06.27                               0 p STARTED
.monitoring-kibana-7-2024.06.27                               0 r UNASSIGNED CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2024.05.29-000001 0 p STARTED
.ds-.logs-deprecation.elasticsearch-default-2024.05.29-000001 0 r UNASSIGNED CLUSTER_RECOVERED
.apm-agent-configuration                                      0 p STARTED
.apm-agent-configuration                                      0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_7.17.13_001                                           0 p STARTED
.kibana_7.17.13_001                                           0 r UNASSIGNED CLUSTER_RECOVERED
.apm-custom-link                                              0 p STARTED
.apm-custom-link                                              0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.25                                   0 p STARTED
.monitoring-es-7-2024.06.25                                   0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.24                               0 p STARTED
.monitoring-kibana-7-2024.06.24                               0 r UNASSIGNED CLUSTER_RECOVERED
.ds-ilm-history-5-2024.05.29-000001                           0 p STARTED
.ds-ilm-history-5-2024.05.29-000001                           0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_task_manager_7.17.13_001                              0 p STARTED
.kibana_task_manager_7.17.13_001                              0 r UNASSIGNED CLUSTER_RECOVERED
.tasks                                                        0 p STARTED
.tasks                                                        0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.22                               0 p STARTED
.monitoring-kibana-7-2024.06.22                               0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.26                               0 p STARTED
.monitoring-kibana-7-2024.06.26                               0 r UNASSIGNED CLUSTER_RECOVERED
.kibana_security_session_1                                    0 p UNASSIGNED INDEX_CREATED
.kibana_security_session_1                                    0 r UNASSIGNED REPLICA_ADDED
.monitoring-kibana-7-2024.06.25                               0 p STARTED
.monitoring-kibana-7-2024.06.25                               0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.27                                   0 p STARTED
.monitoring-es-7-2024.06.27                                   0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.21                               0 p STARTED
.monitoring-kibana-7-2024.06.21                               0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.26                                   0 p STARTED
.monitoring-es-7-2024.06.26                                   0 r UNASSIGNED CLUSTER_RECOVERED
.geoip_databases                                              0 p STARTED
.geoip_databases                                              0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-kibana-7-2024.06.23                               0 p STARTED
.monitoring-kibana-7-2024.06.23                               0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.23                                   0 p STARTED
.monitoring-es-7-2024.06.23                                   0 r UNASSIGNED CLUSTER_RECOVERED
.kibana-event-log-7.17.13-000001                              0 p STARTED
.kibana-event-log-7.17.13-000001                              0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.22                                   0 p STARTED
.monitoring-es-7-2024.06.22                                   0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.21                                   0 p STARTED
.monitoring-es-7-2024.06.21                                   0 r UNASSIGNED CLUSTER_RECOVERED
.monitoring-es-7-2024.06.24                                   0 p STARTED
.monitoring-es-7-2024.06.24                                   0 r UNASSIGNED CLUSTER_RECOVERED
.security-7                                                   0 p STARTED
.security-7                                                   0 r UNASSIGNED CLUSTER_RECOVERED

Check cluster allocation explanation:
To understand why shards are unassigned, you can use the _cluster/allocation/explain API. This will provide a detailed explanation of the allocation decisions.

curl -XGET -u elastic:password http://172.16.0.1:9200/_cluster/allocation/explain?pretty"

    {
      "node_id" : "vO3fSByBTxOCFzNsql-95g",
      "node_name" : "dev01",
      "transport_address" : "172.16.0.2:9300",
      "node_attributes" : {
        "ml.machine_memory" : "6067675136",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "2147483648",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 2,
      "deciders" : [
        {
          "decider" : "enable",
          "decision" : "NO",
          "explanation" : "no allocations are allowed due to cluster setting [cluster.routing.allocation.enable=none]"
        }
      ]
    },
    {
      "node_id" : "kXPPZQDzTBu4ClLeWO95qQ",
      "node_name" : "dev02",
      "transport_address" : "172.16.0.3:9300",
      "node_attributes" : {
        "ml.machine_memory" : "6067666944",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "2147483648",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "weight_ranking" : 3,
      "deciders" : [
        {
          "decider" : "enable",
          "decision" : "NO",
          "explanation" : "no allocations are allowed due to cluster setting [cluster.routing.allocation.enable=none]"
        }

Automatic reallocation:
Sometimes, cluster settings might prevent the allocation of shards. You can check your current cluster settings with:

curl -XGET -u elastic:password http://172.16.0.1:9200/_cluster/settings?pretty"

{
    "peristent" : {
        "cluster" : {
            "routing" : {
                "allocation" : {
                    "enable" : "none"
                }
            }
        }
    }
}

If shard allocation is disabled just like above, you can enable it by updating the cluster settings:

curl -XPUT -u elastic:password http://172.16.0.1:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "all"
  }
}'

Force reallocation of shards:
If you want to manually allocate an unassigned shard to a specific node, you can use the _cluster/reroute API. Here’s how to force the allocation of a stale primary shard:

curl -XPOST -u elastic:password http://172.16.0.1:9200/_cluster/reroute" -H 'Content-Type: application/json' -d '{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "index_name",
        "shard": shard_number,
        "node": "node_name",
        "accept_data_loss": true
      }
    }
  ]
}'

Replace index_name, shard_number, and node_name with the appropriate values.
index: The name of the index to which the shard belongs.
shard: The shard number.
node: The name of the node to which you want to assign the shard.
accept_data_loss: Set to true if you want to allocate a stale primary shard, acknowledging that data loss might occur.

To allocate an unassigned replica shard, you can use:

curl -XPOST -u elastic:password http://172.16.0.1:9200/_cluster/reroute" -H 'Content-Type: application/json' -d '{
  "commands": [
    {
      "allocate_replica": {
        "index": "index_name",
        "shard": shard_number,
        "node": "node_name"
      }
    }
  ]
}'

To get the shard number and index name of unassigned shards in Elasticsearch, you can use the _cat/shards API. This API provides detailed information about all the shards in your cluster, including their index names, shard numbers, and current states.

Additional Steps and Considerations:
Check Node Availability: Ensure all nodes are up and running. Unassigned shards might be due to nodes being down or unreachable.

Disk Space: Make sure there is enough disk space on the nodes. Elasticsearch won’t allocate shards to nodes running low on disk space.

By following these steps, you should be able to diagnose and resolve issues related to unassigned shards in your Elasticsearch cluster.

Install Percona XtraDB Cluster on all hosts that you are planning to use as cluster nodes and ensure you have root access to the MySQL server on each node. In this setup, Multi-Master replication is implemented.

In Multi-Master replication, there are multiple nodes acting as master nodes. Data is replicated between nodes, allowing updates and insertions on a group of master nodes, resulting in multiple copies of the data.

Execute installation commands in all 3 nodes.

apt update
apt install -y wget gnupg2 lsb-release curl
wget https://repo.percona.com/apt/percona-release_latest.generic_all.deb
dpkg -i percona-release_latest.generic_all.deb
apt update
percona-release setup pxc80
apt install percona-xtradb-cluster

Encrypt PXC Traffic

There are two kinds of traffic in Percona XtraDB Cluster: client-server traffic (the one between client applications and cluster nodes), and replication traffic, which includes SST, IST, write-set replication, and various service messages.

Percona XtraDB Cluster supports encryption for all types of traffic. Replication traffic encryption can be configured either automatically or manually. In this guide, we are configuring automatic version.

SST (State Snapshot Transfer) is the full copy of data from one node to another. It’s used when a new node joins the cluster and needs to transfer data from an existing node.

IST (Incremental State Transfer) is a functionality which, instead of transferring the whole state snapshot, catches up with the group by receiving the missing writesets, but only if the writeset is still in the donor’s writeset cache.

Encrypt Relication Traffic

Replication traffic refers to the inter-node traffic, which includes SST, IST, and regular replication traffic.
Percona XtraDB Cluster supports a single configuration option to secure the entire replication traffic, often referred to as SSL automatic configuration. Alternatively, you can configure the security of each channel by specifying independent parameters.
The automatic SSL encryption configuration requires key and certificate files. MySQL generates default key and certificate files and places them in the data directory.
Percona XtraDB Cluster includes the pxc-encrypt-cluster-traffic variable, enabling automatic SSL encryption for SST, IST, and replication traffic.
By default, pxc-encrypt-cluster-traffic is enabled, ensuring a secured channel for replication. This variable is not dynamic and cannot be changed at runtime.
If you wish to disable encryption for replication traffic, you must stop the cluster and update the [mysqld] section of the configuration file on each node with pxc-encrypt-cluster-traffic=OFF. Then, restart the cluster.

But in our case we are not disabling the encryption traffic between the nodes so follow the below steps. Update/Add these variables in /etc/mysql/mysql.conf.d/mysqld.cnf file.

wsrep_node_name=node1
wsrep_node_address=172.16.0.12
pxc_strict_mode=ENFORCING
wsrep_provider=/usr/lib/galera4/libgalera_smm.so
wsrep_cluster_name=pxc-cluster
wsrep_cluster_address=gcomm://172.16.0.12,172.16.0.13,172.16.0.14

Similary add/update above variables in other 2 nodes too. wsrep_node_name and wsrep_node_address variables only need to be updated as per node. Other parameters remain same.

Now Bootstrap the first node. After you configure all PXC nodes, initialize the cluster by bootstrapping the first node. The initial node must contain all the data that you want to be replicated to other nodes.

systemctl start mysql@bootstrap.service

When you start the node using the previous command, it runs in bootstrap mode with wsrep_cluster_address=gcomm://. This tells the node to initialize the cluster with wsrep_cluster_conf_id variable set to 1. After you add other nodes to the cluster, you can then restart this node as normal, and it will use standard configuration again.

To make sure that the cluster has been initialized, run the following:

show status like 'wsrep%';

The output shows that the cluster size is 1 node, it is the primary component, the node is in the Synced state, it is fully connected and ready for write-set replication.

Now copy the certs from node1 to other 2 remaining nodes. Certs will be in path /var/lib/mysql.
It is important that your cluster use the same SSL certificates and keys on all nodes.

bdn@node02:~$ ls /var/lib/mysql/*pem
/var/lib/mysql/ca-key.pem       /var/lib/mysql/client-key.pem   /var/lib/mysql/server-cert.pem
/var/lib/mysql/ca.pem           /var/lib/mysql/private_key.pem  /var/lib/mysql/server-key.pem
/var/lib/mysql/client-cert.pem  /var/lib/mysql/public_key.pem

To verify that the server and client certificates are correctly signed by the same CA certificate, run the following command:

bdn@node02:/var/lib/mysql$ openssl verify -CAfile ca.pem server-cert.pem client-cert.pem
server-cert.pem: OK
client-cert.pem: OK

By default, it generates certificates valid for 10 years.

bdn@node02:/var/lib/mysql$ openssl x509 -enddate -noout -in server-cert.pem
notAfter=Apr 17 13:36:50 2034 GMT

Now start mysql in second node.

systemctl start mysql

Similary, start mysql service in third node too.
Now you can stop the mysql@bootstrap.service service in node1 and start mysql service
Now all 3 nodes are connected to the cluster.

You will see similar logs in /var/log/mysql/error.log file.

2024-04-21T17:21:17.442573Z 1 [Note] [MY-000000] [Galera] ===========================================
=====
View:
  id: 0c233cf9-fe50-11ee-a3e9-dfbfab2fa1fe:31
  status: primary
  protocol_version: 4
  capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ
, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
  final: no
  own_index: 0
  members(3):
        0: 60c1bc1f-0003-11ef-bae7-7347836516bcb, node01
        1: 7b9841ae-0003-11ef-8586-4b8504592b099, node02
        2: 8f92b85a-0003-11ef-b411-5a2ad6404a18f, node03

Now, any changes made to any MySQL server will automatically be reflected on the other nodes.

Install strongSwan in ubuntu

apt-get install strongswan libcharon-extra-plugins strongswan-pki

vim /etc/ipsec.conf

config setup
        charondebug="all"
        uniqueids=yes
        strictcrlpolicy=no
conn B1-TO-HO
        authby=secret
        left=%defaultroute
        leftid=2.2.2.2
        leftsubnet=172.16.2.0/24
        right=1.1.1.1
        rightsubnet=172.16.1.0/24  
        ike=aes256-sha2_256-modp1024!
        esp=aes256-sha256-modp1024!
        keyingtries=0
        ikelifetime=1h
        lifetime=8h
        dpddelay=30

config setup: This section contains global configuration options for StrongSwan.
charondebug=”all”: This option sets the debugging level for the IKE daemon (charon) to “all”, which means it will log all debugging messages. This can be useful for troubleshooting.
uniqueids=yes: This option ensures that each IKE_SA (Internet Key Exchange Security Association) has a unique ID. This helps avoid conflicts in case multiple connections are established.
strictcrlpolicy=no: This option specifies whether strict certificate revocation list (CRL) checking is enforced. Setting it to “no” means that CRL checking will not be strictly enforced.
conn B1-TO-HO: This section defines a specific connection between two peers.
authby=secret: This option specifies that authentication will be performed using a shared secret key.
left=%defaultroute: This option specifies that the local endpoint (left side) of the connection will be determined based on the default route.
leftid=2.2.2.2: This option specifies the identity (ID) of the local endpoint. In this case, it’s set to the IP address 2.2.2.2.
leftsubnet=172.16.2.0/24: This option specifies the local subnet that will be accessible through the VPN tunnel.
right=1.1.1.1: This option specifies the IP address of the remote endpoint (right side) of the connection.
rightsubnet=172.16.1.0/24: This option specifies the remote subnet that will be accessible through the VPN tunnel.
ike=aes256-sha2_256-modp1024!: This option specifies the IKE (Internet Key Exchange) encryption algorithm, integrity algorithm, and Diffie-Hellman group to be used for negotiating phase 1 of the IPsec tunnel. In this case, AES 256-bit encryption, SHA-256 hashing, and MODP 1024-bit Diffie-Hellman group are used.
esp=aes256-sha256-modp1024!: This option specifies the ESP (Encapsulating Security Payload) encryption algorithm, integrity algorithm, and Diffie-Hellman group to be used for negotiating phase 2 of the IPsec tunnel. In this case, AES 256-bit encryption, SHA-256 hashing, and MODP 1024-bit Diffie-Hellman group are used.
keyingtries=0: This option specifies the number of attempts for establishing the IKE_SA. Setting it to 0 means that no retries will be attempted.
ikelifetime=1h: This option specifies the lifetime of the IKE_SA (Phase 1) in hours.
lifetime=8h: This option specifies the lifetime of the CHILD_SA (Phase 2) in hours.
dpddelay=30: This option specifies the delay (in seconds) before Dead Peer Detection (DPD) is initiated. DPD is used to detect if the peer is still reachable. In this case, DPD will be initiated after 30 seconds of inactivity.

set preshared key:

vim /etc/ipsec.secrets

# This file holds shared secrets or RSA private keys for authentication.

# RSA private key for this host, authenticating it to any other host
# which knows the public part.

2.2.2.2 1.1.1.1 : PSK $I8WC#53D@$#%#

2.2.2.2: This is the local endpoint’s IP address.
1.1.1.1: This is the remote endpoint’s IP address.
PSK: This indicates that a pre-shared key is used for authentication.
$I8WC#53D@$#%#: This is the actual pre-shared key used for authentication.

So, in this case, the pre-shared key $I8WC#53D@$#%# is used for authentication between the local endpoint with IP address 2.2.2.2 and the remote endpoint with IP address 1.1.1.1

systemctl restart strongswan-starter

Now we need to enable IP forwarding. The configuration line "net.ipv4.ip_forward = 1" enables IP forwarding on a Linux system. IP forwarding allows the system to forward packets from one network interface to another, essentially acting as a router or gateway.

vim /etc/sysctl.conf
net.ipv4.ip_forward = 1

root@int-vpn-01:~# sysctl -p
net.ipv4.ip_forward = 1

Allow traffic through ufw firewall

ufw allow from 1.1.1.1/32 proto udp to any port 500,4500 comment 'Mikrotik' 
ufw route allow in on any out on any comment 'for VPN Traffic'

Mikrotik Configuration

 /ip ipsec profile
add name="DH_HO_Profile1" hash-algorithm=sha256 enc-algorithm=aes-256 dh-group=modp1024 lifetime=1d proposal-check=obey nat-traversal=yes dpd-interval=2m dpd-maximum-failures=5

/ip ipsec peer
add name="DH_HO_Peer1" address=2.2.2.2/32 profile=DH_HO_Profile1 exchange-mode=ike2 send-initial-contact=yes 

/ip ipsec identity
add peer=DH_HO_Peer1 auth-method=pre-shared-key secret="$I8WC#53D@$#%#" generate-policy=no

/ip ipsec proposal
add name="DH_HO_Proposal1" auth-algorithms=sha256 enc-algorithms=aes-256-cbc lifetime=30m pfs-group=modp1024

/ip ipsec policy
add dst-address=172.16.2.0/24 peer="DH_HO_Peer1" proposal="DH_HO_Proposal1" sa-dst-address=2.2.2.2 sa-src-address=1.1.1.1 src-address=172.16.1.0/24 tunnel=yes

Further NAT rule:

/ip firewall nat
add chain=srcnat action=accept src-address=172.16.1.0/24 dst-address=172.16.2.0/24 log=yes log-prefix=""

Allow port 500/4500/UDP in Mikrotik

/ip firewall filter
add chain=input action=accept protocol=udp src-port="" dst-port=500,4500 log=n> log-prefix="" 

check phase 1 status:

check phase 2 status:

open UDP port 500,4500 for remote server:

check connectivity:

check ipsec status in strongSwan VPN server:

root@int-vpn-01:~# ipsec status
Security Associations (1 up, 0 connecting):
 B1-TO-HO[13]: ESTABLISHED 5 minutes ago, 2.2.2.2[2.2.2.2]...1.1.1.1[1.1.1.1]
 B1-TO-HO{26}:  INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c006274a_i 01310acd_o
 B1-TO-HO{26}:   172.16.2.0/24 === 172.16.1.0/24
root@int-vpn-01:~#

Now our Mikrotik site can communicate with strongSwan site.

Additional, You have to configure the static route on every node behind the VPN gateway server. Otherwise, the 172.16.1.0/24 network will not be able to reach other 172.16.2.x IPs.

ip route add 172.16.1.0/24 via 172.16.2.10

Grafana dashboard which we will be using shows the details of a HA cluster running Pacemaker/Corosync. It is built on top of ha_cluster_exporter but it also requires Prometheus node_exporter to be configured on the target nodes, and it also assumes that the target nodes in each cluster are grouped via the job label.

INSTALL HA_CLUSTER_EXPORTER

mkdir /usr/local/ha_cluster_exporter
cd /usr/local/ha_cluster_exporter
wget https://github.com/ClusterLabs/ha_cluster_exporter/releases/download/1.3.3/ha_cluster_exporter-amd64.gz 
gunzip ha_cluster_exporter-amd64.gz
mv ha_cluster_exporter-amd64 ha_cluster_exporter
chmmod +x ha_cluster_exporter

Create systemd file.

vim /etc/systemd/system/ha_cluster_exporter.service

[Unit]
Description=HA Cluster Exporter

[Service]
User=root
WorkingDirectory=/usr/local/ha_cluster_exporter
ExecStart=/usr/local/ha_cluster_exporter/ha_cluster_exporter
Restart=always

[Install]
WantedBy=multi-user.target

Start and enable service.

systemctl enable --now ha_cluster_expoter

INSTALL NODE_EXPORTER

wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz 
tar xvf node_exporter-1.6.1.linux-amd64.tar.gz 
mv node_exporter-1.6.1.linux-amd64.tar.gz node_exporter
mv node_exporter /usr/local/
cd /usr/local
mv node_exporter-1.6.1.linux-amd64 node_exporter

vim /etc/systemd/system/node_exporter.service

[Unit]
Description=Prometheus Node Exporter

[Service]
User=root
WorkingDirectory=/usr/local/node_exporter
ExecStart=/usr/local/node_exporter/node_exporter --collector.systemd --collector.systemd.unit-include="pcsd|pacemaker|corosync)".service
Restart=always

[Install]
WantedBy=multi-user.target

systemctl enable --now node_exporter

Now Build Prometheus and Grafana Container
For the persistent container storage, execute below commands.

mkdir /var/Docker_home
cd /var/Docker_home
mkdir Prometheus Grafana
mkdir /Prometheus/PromDB
mkdir /Grafana/data
chmod 777 /prometheus/PromDB
chmod 777 /Grafana/data

Create Docker Compose file.

vim docker-compose.yml

version: '3'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    ports:
      - 9090:9090
    volumes:
      - ./Prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./Prometheus/PromDB:/prometheus

  grafana:
    image: grafana/grafana-oss
    container_name: grafana
    ports:
      - 3000:3000
    volumes:
      - ./Grafana/data:/var/lib/grafana

The code defines Docker services: Prometheus for monitoring (port 9090, config and data persistence), and Grafana for visualization (port 3000, data persistence). Docker volumes ensure host-container data sharing, allowing configuration and data to be stored persistently across restarts in respective directories.

Create prometheus config file.

vim /var/Docker_home/Prometheus/prometheus.yml

global:
  scrape_interval:     10s
  evaluation_interval: 10s

scrape_configs:
  - job_name: "ha-cluster"
    static_configs:
      - targets: ['ram-01.bidhankhatri.com.np:9664', 'ram-02.bidhankhatri.com.np:9664', 'ram-01.bidhankhatri.com.np:9100', 'ram-02.bidhankhatri.com.np:9100']

Port 9664 for ha_cluster_exporter and Port 9100 for node_exporter

docker-compose up -d

Pulling prometheus (prom/prometheus:)...
latest: Pulling from prom/prometheus
d5c4df21b127: Pull complete
2f5f7d8898a1: Pull complete
300c29bb5b04: Pull complete
be6ad5a51a35: Pull complete
ea6cf9f81dfe: Pull complete
b5ac85a4be54: Pull complete
d32980b63d51: Pull complete
502ed6d3bdc8: Pull complete
7bed70210741: Pull complete
3b19398e1689: Pull complete
d358eb0a0392: Pull complete
d6eaeaf54563: Pull complete
Digest: sha256:d6ead9daf2355b9923479e24d7e93f246253ee6a5eb18a61b0f607219f341a80
Status: Downloaded newer image for prom/prometheus:latest
Pulling grafana (grafana/grafana-oss:)...
latest: Pulling from grafana/grafana-oss
4db1b89c0bd1: Pull complete
312681f4cad0: Pull complete
8b7b65888846: Pull complete
dd9c3d04d541: Pull complete
959325519a8e: Pull complete
16cb2df7bffd: Pull complete
94d1f5f5bfea: Pull complete
e3281a1a7e8f: Pull complete
5c0c2b741753: Pull complete
Digest: sha256:423040d62678074111e4e72d7dcef23480a94eb4f21b9173204d1a5ee972ec59
Status: Downloaded newer image for grafana/grafana-oss:latest
Creating prometheus ... done
Creating grafana    ... done

docker ps   

CONTAINER ID   IMAGE                 COMMAND                  CREATED         STATUS              PORTS                                       NAMES
1fb48738a930   prom/prometheus       "/bin/prometheus --c…"   2 minutes ago   Up About a minute   0.0.0.0:9090->9090/tcp, :::9090->9090/tcp   prometheus
f5362d262246   grafana/grafana-oss   "/run.sh"                2 minutes ago   Up About a minute   0.0.0.0:3000->3000/tcp, :::3000->3000/tcp   grafana

Both Grafana and Prometheus are up now.

CONFIGURE GRAFANA

Go to http://YOURIPADDRESS:3000 to access grafana UI. Default username/password is admin/admin. change it after login.

1. 1st connect Grafana to Prometheus:
   Home » Connections » Add new connection » Search for Prometheus » Create a Prometheus data source » update prometheus server URL: http://YOURIPADDRESS:9090
   Click on Save & Test

2. Second, Import Grafana Dashboard for HA cluster. Ref: https://grafana.com/grafana/dashboards/12229-ha-cluster-details/
   Home » Dashboard » New [ drop down] » Import » Put ID 12229 and click on Load » Choose prometheus which we configured earlier
   Click on Import

Similar graph you will see.

GFS2 File system

GFS2 (Global File System 2) is a cluster file system designed for use in Linux-based environments. It is an enhanced version of the original GFS (Global File System), which was developed by Red Hat. GFS2 allows multiple servers to have concurrent read and write access to a shared file system, providing high availability and scalability for data storage.

GFS2 is commonly used in scenarios where multiple servers need simultaneous access to a shared file system, such as in high-performance computing (HPC) clusters, database clusters, or virtualization environments. It provides a reliable and scalable solution for managing data across a cluster of Linux servers.

Now lets start the setup.

1. Cluster setup with Pacemaker/Corosync
2. GFS2 Setup

Various components of the cluster stack (corosync, pacemaker, etc.) has to configured.

Pacemaker:

Pacemaker is a high-availability cluster resource manager — software that runs on a set of hosts (a cluster of nodes) in order to preserve integrity and minimize downtime of desired services (resources).

Corosync:

Corosync is an cluster messaging and membership service that is often used in conjunction with the Pacemaker software to build high availability (HA) clusters. It provides a reliable and scalable communication infrastructure for coordinating and synchronizing the activities of nodes in a cluster. Corosync operates as a cluster messaging layer, enabling the exchange of information and cluster-related events among the nodes.

Cluster setup with Pacemaker/Corosync

We will setup 2 node cluster and share the disk between them.
Nodes:

• ram-01.bidhankhatri.com.np
• ram-02.bidhankhatri.com.np

Prerequisites:

• Update both node with its IP in each other /etc/hosts file. Its always recommended to update host details in /etc/hosts file when you are trying to setup any cluster instead depending upon the external DNS.

vim /etc/hosts

10.12.6.10 ram-01.bidhankhatri.com.np
10.12.6.11 ram-02.bidhankhatri.com.np

• Ensure NTP is set up and synchronize the time between nodes to avoid significant time differences, which could adversely affect cluster performance. Verify this configuration on both nodes.

ntpstat

Now first enable the HighAvailability repo on both nodes and install packages. After that start and enable pcsd, pacemaker and corosync service.

yum install pcs pacemaker fence-agents-all
systemctl enable --now pcsd
systemctl enable --now pacemaker
systemctl enable --now corosync

Enable ports required for HighAvailability

firewall-cmd --permanent --add-service=high-availability
firewall-cmd --reload

The installed packages will create a hacluster user with a disabled password. While this is fine for running pcs commands locally, the account needs a login password in order to perform such tasks as syncing the corosync configuration, or starting and stopping the cluster on other nodes.

[root@ram-01 ~]# passwd hacluster
[root@ram-02 ~]# passwd hacluster

CONFIGURE COROSYNC

On either node, use pcs cluster auth to authenticate as the hacluster user:

[root@ram-01 ~]# pcs host auth ram-01.bidhankhatri.com.np ram-02.bidhankhatri.com.np
Username: hacluster
Password: ******
ram-01.bidhankhatri.com.np: Authorized
ram-02.bidhankhatri.com.np: Authorized
[root@ram-01 ~]#

NOTE: If you are using RHEL7/Centos7 or Fedora then command to authenticate hosts is:
pcs cluster auth ram-01.bidhankhatri.com.np ram-02.bidhankhatri.com.np

Next, use below command on the same node to generate and synchronize the corosync between the cluster nodes. We will be setting “gfs-cluster” as our cluster name.

[root@ram-01 ~]# pcs cluster setup gfs-cluster ram-01.bidhankhatri.com.np ram-02.bidhankhatri.com.np

NOTE: In Centos7, command is:
pcs cluster setup --name gfs-cluster ram-01.bidhankhatri.com.np ram-02.bidhankhatri.com.np

Check cluster status:

[root@ram-01 ~]# pcs cluster status
Cluster Status:
 Status of pacemakerd: 'Pacemaker is running' (last updated 2023-07-07 15:03:12 -04:00)
 Cluster Summary:
  * Stack: corosync
  * Current DC: ram-01.bidhankhatri.com.np (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum
  * Last updated: Fri Jul 7 16:11:18 2023
  * Last change: Fri Jul 7 16:11:00 2023 by hacluster via crmd on ram-01.bidhankhatri.com.np
  * 2 nodes configured
  * 0 resources instances configured
Node List:
  * Online: [ram-01.bidhankhatri.com.np ram-02.bidhankhatri.com.np ]

PCSD Status:
  ram-01.bidhankhatri.com.np: Online
  ram-02.bidhankhatri.com.np: Online

A Red Hat High Availability cluster requires that you configure fencing for the cluster. So before doing the GFS2 setup, we have to configure fencing first.

STONITH SETUP

pcs stonith create vmfence fence_vmware_rest pcmk_host_map="ram-01.bidhankhatri.com.np:ram-01-vm;ram-02.bidhankhatri.com.np:ram-02-vm" ip=http://192.168.2.1 ssl_insecure=1 username="adminuser@bidhankhatri.com.np" password="****" delay=10 pcmk_monitor_timeout=120s

The above provided command is used to create a fencing resource in the Pacemaker cluster manager using the VMware REST API. Fencing, also known as STONITH (Shoot The Other Node In The Head)

pcs stonith create vmfence: This is the command to create a new fencing resource named “vmfence” in Pacemaker.
fence_vmware_rest: This specifies the fence agent to be used, which in this case is the VMware REST API fence agent. This agent allows Pacemaker to communicate with VMware vSphere to fence a misbehaving node.
pcmk_host_map=”ram-01.bidhankhatri.com.np:ram-01-vm;ram-02.bidhankhatri.com.np:ram-02-vm”: This option specifies the mapping between the hostnames of the nodes in the Pacemaker cluster and the corresponding virtual machine (VM) names in the VMware environment. It indicates which VM corresponds to which cluster node.
ip=http://192.168.2.1 This option specifies the IP address or hostname of the VMware vCenter server.
ssl_insecure=1: This option tells the fence agent to ignore SSL certificate validation errors. Use this option only if you have a self-signed or untrusted SSL certificate on your vCenter server.
username=”adminuser@bidhankhatri.com.np”: This option specifies the username to authenticate with the VMware vCenter server.

GFS2 SETUP

Enable repo: rhel-8-for-x86_64-resilientstorage-rpms

On both nodes of the cluster, install the lvm2-lockd, gfs2-utils, and dlm packages.

yum install lvm2-lockd gfs2-utils dlm

On both nodes of the cluster, set the use_lvmlockd configuration option in the /etc/lvm/lvm.conf file to use_lvmlockd=1.

use_lvmlockd=1

Set the global Pacemaker parameter no-quorum-policy to freeze.

[root@ram-01 ~]# pcs property set no-quorum-policy=freeze

By default, when quorum is lost, all resources on the remaining partition are immediately stopped. This is the safest and most optimal option, but GFS2 requires quorum to function. If quorum is lost, both GFS2 applications and the GFS2 mount cannot be stopped correctly. To solve this, set no-quorum-policy to freeze when using GFS2. This means the remaining partition will remain inactive until quorum is regained.

To configure a GFS2 file system in a cluster, it is necessary to establish a dlm resource, which is a mandatory dependency. In this case, the provided example creates the dlm resource within a resource group called "locking."

[root@ram-01 ~]# pcs resource create dlm --group locking ocf:pacemaker:controld op monitor interval=30s on-fail=fence

Clone the locking resource group so that the resource group can be active on both nodes of the cluster.

pcs resource clone locking interleave=true

Set up an lvmlockd resource as part of the locking resource group.

pcs resource create lvmlockd --group locking ocf:heartbeat:lvmlockd op monitor interval=30s on-fail=fence

Check the status of the cluster to ensure that the locking resource group has started on both nodes of the cluster.

# pcs status --full

Cluster name: gfs-cluster
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-07-09 08:03:40 -04:00)
Cluster Summary:
  * Stack: corosync
  * Current DC: ram-01.bidhankhatri.com.np (version 2.1.4-5.e18_7.2-dc6eb4362e) - partition with quorum
  * Last updated: Sun Jul 09 08:03:41 2023
  * Last change: Sun Jul 9 08:03:37 2023 by root via cibadmin on ram-01.bidhankhatri.com.np
  * 2 ndoes configured
  * 5 resource instances configured

Node List:
  * Online: [ ram-01.bidhankhatri.com.np (1) ram-02.bidhankhatri.com.np (2) ]

Full List of Resources:
  * Clone Set: locking-clone [locking]:
    * Resource Group: locking:0:
      * dlm    (ocf::pacemaker:controld):       Started ram-01.bidhankhatri.com.np
      * lvmlockd    (ocf:pacemaker:lvmlockd):         Started ram-01.bidhankhatri.com.np
    * Resource Group: locking:1:
      * dlm    (ocf::pacemaker:controld):       Started ram-02.bidhankhatri.com.np
      * lvmlockd    (ocf:pacemaker:lvmlockd):         Started ram-02.bidhankhatri.com.np
  * vmfence     (stonith:fence_vmware_rest):    started ram-01.bidhankhatri.com.np

Migrration Summary:

Tickets:

PCSD Status:
  ram-01.bidhankhatri.com.np: Online
  ram-02.bidhankhatri.com.np: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

On one node, create a shared volume group. Create volume group "shared_vg"

vgcreate --shared shared_vg /dev/sdb
  Physical volume "/dev/vdb" successfully created.
  Volume group "shared_vg" successfully created
  VG shared_vg starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

In 2nd node.

vgchange --lockstart shared_vg
  VG shared_vg starting dlm lockspace
  Starting locking.  Waiting until locks are ready...

lvcreate --activate sy -l 100%FREE -n shared_lv shared_vg
  Logical volume "shared_lv" created.

[root@ram-01 ~]# mkfs.gfs2 -j2 -p lock_dlm -t gfs-cluster:gfs2 /dev/shared_vg/shared_lv
/dev/shared-vg is a symbolic link to /dev/dm-7
this will destory any data on /dev/dm-7
Are you sure you want to proceed? [y/n] y
Discarding device contents (may take a while on large devices): Done
Adding journals: Done
Building resource groups: Done
Creating quota file: Done
Writing superblock and syncing: Done
Device:              /dev/shared_vg/shared_lv
Block size:          4096
Device size:         8.00 GB (2096128 blocks)
Filesystem size:     8.00 GB (2096128 blocks)
Journals:            2
Journal size:        32MB
Resource groups:     34
Locking protocol:    "lock_dlm"
Lock table:          "gfs-cluster:gfs2"
UUID:                6c5a011c-188a-48d1-adc8-774b256b2850

• -p lock_dlm specifies that we want to use the kernel’s DLM.
• -j 2 indicates that the filesystem should reserve enough space for two journals (one for each node that will access the filesystem).
• -t gfs-cluster:gfs2 specifies the lock table name. The format for this field is clustername:fsname.
For clustername, we need to use the same value we specified originally with pcs cluster setup (which is also the value of cluster_name in /etc/corosync/corosync.conf). If you are unsure what your cluster name is, you can look in /etc/corosync/corosync.conf or execute the command.
pcs cluster corosync ram-01.bidhankhatri.com.np | grep -i "Cluster name".

Create an LVM-activate resource for logical volume to automatically activate that logical volume on all nodes.
Create an LVM-activate resource named sharedlv for the logical volume shared_lv in volume group shared_vg.
Below command creates a Pacemaker cluster resource called “sharedlv” that manages a logical volume named “shared_lv” belonging to the volume group “shared_vg.” The resource agent used is “ocf:heartbeat:LVM-activate,” and the LV is configured to be activated in shared mode, allowing multiple nodes to access it concurrently. The VG access mode is set to “lvmlockd” to enable distributed locking for concurrent VG access.

pcs resource create sharedlv --group shared_vg ocf:heartbeat:LVM-activate lvname=shared_lv vgname=shared_vg activation_mode=shared vg_access_mode=lvmlockd

pcs: It stands for “Pacemaker Configuration System” and refers to the command-line tool used for managing Pacemaker clusters.
resource create: This command is used to create a new resource in the Pacemaker cluster.
sharedlv: It is the name given to the resource being created.
–group shared_vg: It specifies that the resource should be added to the resource group named “shared_vg.” A resource group is a logical grouping of resources that are managed together.
ocf:heartbeat:LVM-activate: It specifies the resource agent to be used for managing the shared logical volume. In this case, the Heartbeat OCF resource agent is used. The LVM-activate resource agent is responsible for activating an LVM logical volume.
lvname=shared_lv: It specifies the name of the logical volume to be managed by the resource. In this case, the LV is named “shared_lv.”
vgname=shared_vg: It specifies the name of the volume group (VG) to which the logical volume belongs. In this case, the VG is named “shared_vg.”
activation_mode=shared: It specifies the activation mode for the LV. “Shared” mode indicates that the LV can be activated by multiple nodes simultaneously.
vg_access_mode=lvmlockd: It specifies the access mode for the volume group. "lvmlockd" is a locking daemon that provides a distributed lock manager for LVM, allowing multiple nodes to access the VG concurrently.

Clone the new resource group.

pcs resource clone shared_vg interleave=true

Configure ordering constraints to ensure that the locking resource group that includes the dlm and lvmlockd resources starts first.

[root@ram-01 ~]# pcs constraint order start locking-clone then shared_vg-clone

Configure colocation constraints to ensure that the vg resource group start on the same node as the locking resource group.

[root@ram-01 ~]# pcs constraint colocation add shared_vg-clone with locking-clone

On both nodes in the cluster, verify that the logical volumes are active. There may be a delay of a few seconds.

[root@ram-01 ~]# lvs
  LV         VG          Attr       LSize
  shared_lv shared_vg  -wi-ao-----  <8.00g

[root@ram-02 ~]# lvs
  LV         VG          Attr       LSize
  shared_lv shared_vg  -wi-ao-----  <8.00g

Create a file system resource to automatically mount each GFS2 file system on all nodes. You should not add the file system to the /etc/fstab file because it will be managed as a Pacemaker cluster resource.

Create a file system resource to automatically mount each GFS2 file system on all nodes. You should not add the file system to the /etc/fstab file because it will be managed as a Pacemaker cluster resource.

pcs resource create sharedfs --group shared_vg ocf:heartbeat:Filesystem device="/dev/shared_vg/shared_lv" directory="/app" fstype="gfs2" options=noatime op monitor interval=10s on-fail=fence

resource create: This command is used to create a new resource in the Pacemaker cluster.
sharedfs: It is the name given to the resource being created.
–group shared_vg: It specifies that the resource should be added to the resource group named “shared_vg.”
ocf:heartbeat:Filesystem: It specifies the resource agent to be used for managing the filesystem resource. In this case, the Heartbeat OCF resource agent for Filesystem is used.
device=”/dev/shared_vg/shared_lv”: It specifies the device associated with the filesystem resource. In this case, the filesystem is located on the logical volume /dev/shared_vg/shared_lv.
directory=”/app”: It specifies the directory where the filesystem should be mounted. In this case, the filesystem will be mounted on the directory /app.
fstype=”gfs2”: It specifies the filesystem type. In this case, the filesystem type is “gfs2,” which stands for Global File System 2.
options=acl,noatime: Specifies the mount options for the filesystem. The acl option enables Access Control Lists (ACLs), which provide more granular access control on files and directories. The noatime option disables updating the access time for files when they are accessed, potentially improving performance.
op monitor interval=10s on-fail=fence: It defines the resource’s monitoring behavior. The monitor operation will be performed on the resource every 10 seconds (interval=10s). If the monitor operation fails, the node where the resource is running will be fenced (on-fail=fence), indicating a failure and allowing the resource to be recovered on another node.

Verification steps:

mount | grep gfs2
/dev/mapper/shared_vg-shared_lv on /app type gfs2 (rw,noatime,acl)

mount | grep gfs2
/dev/mapper/shared_vg-shared_lv on /app type gfs2 (rw,noatime,acl)

df -h /mnt/gfs/
Filesystem                       Size    Used   Avail   Use%  Mounted on
/dev/mapper/shared_vg-shared_lv  8.0G    67M    8.0G    1%    /app

check cluster status:

# pcs status --full

Cluster name: gfs-cluster
Status of pacemakerd: 'Pacemaker is running' (last updated 2023-07-09 08:03:40 -04:00)
Cluster Summary:
  * Stack: corosync
  * Current DC: ram-01.bidhankhatri.com.np (version 2.1.4-5.e18_7.2-dc6eb4362e) - partition with quorum
  * Last updated: Sun Jul 09 08:03:41 2023
  * Last change: Sun Jul 9 08:03:37 2023 by root via cibadmin on ram-01.bidhankhatri.com.np
  * 2 ndoes configured
  * 9 resource instances configured

Node List:
  * Online: [ ram-01.bidhankhatri.com.np (1) ram-02.bidhankhatri.com.np (2) ]

Full List of Resources:
  * Clone Set: locking-clone [locking]:
    * Resource Group: locking:0:
      * dlm    (ocf::pacemaker:controld):       Started ram-01.bidhankhatri.com.np
      * lvmlockd    (ocf:pacemaker:lvmlockd):         Started ram-01.bidhankhatri.com.np
    * Resource Group: locking:1:
      * dlm    (ocf::pacemaker:controld):       Started ram-02.bidhankhatri.com.np
      * lvmlockd    (ocf:pacemaker:lvmlockd):         Started ram-02.bidhankhatri.com.np
  * vmfence     (stonith:fence_vmware_rest):    started ram-01.bidhankhatri.com.np
  * Clone Set: shared_vg-clone [shared_vg]:
    * Resource Group: shared_vg:0:
      * sharedlv    (ocf::heartbeat:LVM-activate):       Started ram-01.bidhankhatri.com.np
      * sharedfs    (ocf:heartbeat:Filesystem):         Started ram-01.bidhankhatri.com.np
    * Resource Group: shared_vg:1:
      * sharedlv    (ocf::heartbeat:LVM-activate):       Started ram-02.bidhankhatri.com.np
      * sharedfs    (ocf:heartbeat:Filesystem):         Started ram-02.bidhankhatri.com.np

Migrration Summary:

Tickets:

PCSD Status:
  ram-01.bidhankhatri.com.np: Online
  ram-02.bidhankhatri.com.np: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

Introduction

SAML authentication for OpenSearch Dashboards allows you to utilize your existing identity provider for offering single sign-on (SSO) on Amazon OpenSearch Service domains running OpenSearch or Elasticsearch 6.7 or later. To enable SAML authentication, fine-grained access control must be enabled.

Instead of authenticating through Amazon Cognito or the internal user database, SAML authentication for OpenSearch Dashboards allows you to utilize third-party identity providers to log in, manage fine-grained access control, search your data, and create visualizations. OpenSearch Service supports providers that adhere to the SAML 2.0 standard, including Okta, Keycloak, Active Directory Federation Services (ADFS), Auth0, and AWS IAM Identity Center (the successor to AWS Single Sign-On).

SAML authentication for Dashboards is specifically designed for accessing OpenSearch Dashboards through a web browser. It’s important to note that your SAML credentials cannot be used to directly make HTTP requests to the OpenSearch or Dashboards APIs.

SAML configuration overview

This documentation assumes that you have an existing identity provider and some familiarity with it.

The OpenSearch Dashboards login flow can take one of two forms:

1. Service provider (SP) initiated: You navigate to Dashboards (for example, https://my-domain.us-east-1.es.amazonaws.com/_dashboards), which redirects you to the login screen. After you log in, the identity provider redirects you to Dashboards.
2. Identity provider (IdP) initiated: You navigate to your identity provider ( E.g Okta), log in, and choose OpenSearch Dashboards from an application directory.

OpenSearch Service provides two single sign-on URLs, SP-initiated and IdP-initiated, but you only need the one that matches your desired OpenSearch Dashboards login flow. Here In our case, We are going to setup IdP initiated setup.

Regardless of the authentication type you choose, the objective is to log in through your identity provider and obtain a SAML assertion that includes your username (required) and any backend roles (optional, but recommended). This information enables fine-grained access control to assign permissions to SAML users. In external identity providers, backend roles are commonly referred to as ‘roles’ or ‘groups’.

STEPS:

1. Identity Provider (Okta) setup
2. Prepare OpenSearch Service for SAML configuration
3. SAML configuration in OpenSearch Service Domain and Okta
4. Validate Okta users access and role

Identity Provider (Okta) setup

( Create 2 groups “opensearch-admin” and “opensearch-user” and assign users to it. )

1. Signup for Okta Account
2. Create Users in Okta
3. Create Groups in Okta
4. Assign Users to Groups

Login to your Okta Admin page. If you don’t have Okta then you can singup for Okta free trial account which will be valid for 30 days. To get trail access, go to this page and fill the form. https://www.okta.com/free-trial/

Create Users in Okta: Directory » People » Add Person
Create multiple users according to your need. Here we created 2 users “bidhan.khatri” and “ram.thapa”

Create Groups in Okta: Directory » Groups » Add Group
We will create 2 groups: “opensearch-admin” and “opensearch-user.”
Assign user “bidhan.khatri” to “opensearch-admin” and user “ram.thapa” to “opensearch-user”.
“opensearch-admin” is a Okta group for admin access in OpenSearch Dashboard and “opensearch-user” for read access which we will configure later.

Assign Users to Groups:
Directory » Groups » opensearch-user » Assign people ( Click + sign of the user “ram.thapa” )
Similary, do for another group.
Directory » Groups » opensearch-admin » Assign people ( select user “bidhan.khatri”)

Create OpenSearch in AWS

If you haven’t created OpenSearch then follow the below steps, otherwise go to next section “SAML configuration in OpenSearch Service Domain and Okta”

Open AWS OpenSearch Service and Click on Create domain.
Domain Name: bdn
Domain Created Method: Standard create
Templates: Dev/test
Deployment Options(s): Domain without standby
Availability Zones(s): 1-AZ
Engine options: 2.5(latest)
Data nodes: t3.small.search
Number of Nodes: 1
Storage Type: EBS
EBS storage size per node: 10G
Network: Public Access
Fine-grained access control: Create master user // This credential will be used for logging opensearch dashboard.
Access Policy: Domain access Policy > Only use fine-grained access control
Other things as default
Now look at the summary and confirm for new domain and click “create”

SAML configuration in OpenSearch Service Domain and Okta

Go to Amazon OpenSearch Service and in your domain. Go to “bdn Domain”
Click on Actions » Edit security configuration

Tick on “Enable SAML Authentication”

We will be using the Service Provider entity ID and IdP-initiated SSO URL for Okta SAML configuration.

The OpenSearch Dashboards login flows below step:
You navigate to your identity provider (Okta), log in, and choose OpenSearch Dashboards from an application directory.

We will complete rest of the above OpenSearch Service SAML configuration after the Okta SAML configuration.
Above SAML information is required for Okta SAML setup.

Okta SAML Configuration

Go back to your OKTA ADMIN PAGE and choose Applications
Click on Create App Integration Choose SAML 2.0 and click NEXT

App name: OpenSearch
SAML Settings
Single sign-on URL: https://search-bdn-gu3ogldk6nakq776cama6dmjq4.us-east-2.es.amazonaws.com/_dashboards/_opendistro/_security/saml/acs/idpinitiated [ IdP - initiated SSO URL from bdn domain]
Put tick on Use this for Recipient URL and Destination URL
Audience URI (SP Entity ID): https://search-bdn-gu3ogldk6nakq776cama6dmjq4.us-east-2.es.amazonaws.com [ Service provider entity ID from bdn domain]
( Both URL’s are copied from SAML configuration in OpenSearch Service Domain )

Default RelayState: leave it blank
Name ID format: Select EmailAdress from drop down
Application Username: Okta username
Update application username on: Create and update

Attribute Statements (optional)
Name: http://schemas.xmlsoap.org/ws/2005/05/identity/claims/emailaddress
Name: Select URI Reference from dropdown
value: user.email

Group Attribute Statements (optional)
Name:http://schemas.xmlsoap.org/claims/Group
Normal Format: Unspecified
Filter: Starts with “opensearch”

Now click on Next

As our both Okta Groups “opensearch-admin” and “opensearch-user” naming starts with string opensearch. Here while configuring group section, we defined opensearch in Group attribute and used filter Starts with to catch both groups.

Are you a customer or partner? Tick on “I’m a software vendor. I’d like to integrate my app with Okta” » Finish

Choose Sign on menu. Under SAML Setup: Click on “View SAML setup instructions”
Go down and select and copy all the content of IDP metadata section. This is an Okta Identity provider metadata for SAML configutation. We have to use this XML content in OpenSearch service. Copy below XML information from the box to somewhere for now.

Assign Groups to Application “OpenSearch”:
Go to option Assignments menu and click on assign » Assign to Groups Choose both groups “opensearch-admin” and “opensearch-user”. Click on Assign

As you can see now both groups are assigned to application OpenSearch.

Back to AWS OpenSearch domain SAML authentication options

Under the Import IdP metadata section:
Metadata fromm IdP: Copy the Okta Identity Provider metadata which we copied earlier here in the IdP box.
IdP entity ID: It will auto populate once you provide IdP metadata.
SAML master username - optional: Leave it blank.
SAML master backend role - optional: opensearch-admin (Okta group which need full permission in OpenSearch Dashboard.)

Click on Additional settings:
Subject key - optional: Leave it blank.
Roles Key - optional: http://schemas.xmlsoap.org/claims/Group
Session time to live: 60 minutes
and save the changes.

Validate Okta users access and RBAC

Now login the Okta dashboard from your user “bidhan.khatri”.
Click on “My Apps” » under “work” » click on “OpenSearch” app to open the AWS OpenSearch Dashboard. User will get full access as user “bidhan.khatri” belongs to okta group “opensearch-admin”

All the users belongs to okta group “opensearch-admin” will receive full permission in OpenSearch Dashboard. We have defined to get full access for “opensearch-admin” while configuring SAML for AWS OpenSearch domain.
Similary, We can setup Role-based access control (RBAC) using Okta Group based on the OpenSearch Role. For okta group “opensearch-user” to give Read Only access in all index in OpenSearch dashboard, we have to login OpenSearch Dashboard with full permission user first and create a role in OpenSearch for “opensearch-user” group and map to it.

Give READ Access to Group “opensearch-user”

First, login OpenSearch from the user belonging to Okta group “opensearch-admin”. We need full permission user access to setup a new role. We will be giving only read permission to all the index to group user “opensearch-user”

Security » Roles » Create role » Role-Read » Index permissisions
Index: *
Index permmissions: read get
Tenant permissions: global_tenant

click Update

Go to Mapped users » Map users » In Backend roles » Add “opensearch-user” » Map

Backend Role is basically a Group in external authentication system like AD,LDAP with whom OpenSearch role is mapped so that instead of giving access to the 100 different users, we can simply provide access to a single AD group.

Now once user ram.thapa login OpenSearch Dashboard, he will see only read access given to all the indices.
To check your permission, Click on your user icon » click View roles and identities.”
He will get below roles defined.