Scalable CI/CD with Jenkins and Docker Cloud Agents
A Jenkins cloud agent using Docker refers to dynamically provisioning Jenkins build agents (also called slaves or nodes) in Docker containers—typically on demand—rather than using pre-provisioned static nodes. This allows Jenkins to scale efficiently and cleanly by creating isolated environments for each build. Today, I will show you how to configure Docker cloud on Jenkins.
Be cautious when using Docker on the same host as the Jenkins master—consider using a remote Docker daemon or sandboxing strategies to avoid privilege escalation. I will show you both approaches here.
First, spin up Jenkins. We will run Jenkins as a container. The Jenkins container runs as UID 1000 by default, so your host Jenkins directory must be writable by UID 1000.
mkdir /var/Docker/jenkins
chown -R 1000:1000 /var/Docker/jenkins
docker network create jenkins
If UID 1000 does not exist on the host machine, reserve UID 1000 for the Jenkins user.
sudo useradd -u 1000 -m -s /bin/bash jenkins
Run the Jenkins container with persistent storage, Docker access, jdk21 and port mappings.
docker run -d \
--name jenkins \
--network jenkins \
--restart=always \
-p 8080:8080 -p 50000:50000 \
-v /var/Docker/jenkins:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
jenkins/jenkins:2.504.1-lts-jdk21
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0970e84f4ffb jenkins/jenkins:2.504.1-lts-jdk21 "/usr/bin/tini -- /u…" 1 hours ago Up 2 minutes 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:50000->50000/tcp, :::50000->50000/tcp jenkins
Get the initial admin password for jenkins
docker exec -it jenkins cat /var/jenkins_home/secrets/initialAdminPassword
de9ae273099f4a09979e4c54fe0b1a20
Enter that password at Jenkins URL http://localhost:8080.
Install suggested plugins » Create your first admmin user.
Dashboard » Manage Jenkins » System configuration » Nodes » clouds » Install a plugin ( Docker ) » Install » Tick on Restart Jenkins when installation is complete.
Now jenkins will restart enabling the docker plugin.
Docker cloud configurationPermalink
setting up Jenkins to dynamically provision Docker containers as build agents.
Manage Jenkins » Nodes » Clouds » New cloud
Give a name to your Docker cloud. I used ‘docker_cloud’ and select ‘Docker Type’ from the options.
Once you click on “create” now you will see 2 options:
- Docker Cloud details
- Docker Agent templates
1. Docker Cloud details:
This is where you define the Docker environment that Jenkins will use to spin up containers.
Name: A unique identifier for this Docker cloud.
Docker Host URI: URI to connect to Docker. E.g., unix:///var/run/docker.sock or tcp://…
Server Credentials: (Optional) TLS certs if Docker is secured with TLS (rare on local setups). In our case, we are not using any credentials.
Test Connection: Verifies Jenkins can reach the Docker daemon.
I’m trying to connect to the Docker daemon running on the same host where the Jenkins container is running, so I’m using docker.sock to connect to it.
If jenkins ia able to connect docker then you will see the docker version and api version detials.
This confirms that Jenkins is correctly detecting the Docker engine version and API version when connecting to the Docker socket.
Version = 20.10.7, API Version = 1.41
Click on “Enabled”
Leave other section as it is.
If you see errors like “java.net.BindException: Permission denied “ then you should add jenkins user to host docker group and restart it.
Get the docker group id from the host machine.
getent group docker
docker:x:986:bdn
docker exec -u0 -it jenkins bash
groupadd -g 986 docker
usermod -aG docker jenkins
docker restart jenkins
2. Docker Agent Templates:
Each template defines how a container should be spun up to act as a Jenkins agent.
Labels: Labels used by jobs/pipelines to request this agent type. Click on Enabled.
Name: Internal name (can be anything). use same as labels.
Docker Image: Image to use (e.g., jenkins/inbound-agent, maven:3.8-jdk-11 or your custom immage)
Instance Capacity: Max number of containers Jenkins can launch from this template.
Remote File System Root: Where the Jenkins workspace lives inside the container (e.g., /home/jenkins).
Usage: Use this node as much as possible
Connect method: Attach Docker container.
(All the images should have Java installed. Use the same Java version as the master. The Docker image CMD must either be empty or simply keep the container running indefinitely, e.g., /bin/bash. The Jenkins remote agent code will be copied into the container and then run using the Java installed in the container.)
Pull strategy: Never pull
(Pull all images every time if you’re pulling them from Docker Hub. If you have a local image, choose ‘Never pull.’)
In my case I’m choosing ‘Never pull’ as my image is a custom one.
Dashboard » Manage Jenkins » Clouds » docker_cloud » Configure
Here is the Dockerfile for a custom image which includes Java 21(OpenJDK) runtime, python 3 and ansible all based on Alpine linux.
FROM alpine:3.19
# Install base packages
RUN apk update && apk add --no-cache \
bash \
curl \
openssh \
git \
python3 \
py3-pip \
ansible \
libc6-compat \
sshpass
RUN mkdir -p /opt/java && \
curl -L -o /tmp/openjdk.tar.gz https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.2+13/OpenJDK21U-jdk_x64_alpine-linux_hotspot_21.0.2_13.tar.gz && \
tar -xzf /tmp/openjdk.tar.gz -C /opt/java && \
mv /opt/java/jdk-* /opt/java/openjdk && \
rm -rf /tmp/openjdk.tar.gz
# Install Java 21 (Alpine-specific Temurin build)
ENV JAVA_HOME=/opt/java/openjdk
ENV PATH="$JAVA_HOME/bin:$PATH"
# Verify that Java has been installed correctly
RUN java -version
# Create non-root user
RUN adduser -D jenkins
USER jenkins
WORKDIR /home/jenkins
# Default command
CMD ["/bin/bash"]
docker build -t jenkins-agent:alp-j21-py3-ans .
Now create a new freestyle jobPermalink
I have created one freestyle job which is “Hello-World”
Click on “Configuration” and in General, click “Restrict where this project can be run” field. put “jenkins-agent”
In build section, add build step, Execute shell echo “Hello World” Now run the job.
Here, what will happen is that Jenkins dynamically launches new containers as agents (jenkins-agent:alp-j21-py3-ans) on demand to run jobs(“Hello-World”). These containers are ephemeral and automatically destroyed after the job finishes. It’s a best-practice approach in Jenkins to use ephemeral Docker containers for running jobs.
That’s it. Now you can run as much as ephemeral containers to run your Jenkins Jobs.
If you want to run the job on the same host where Docker is installed, then the /var/run/docker.sock can be used. However, if you want to run the job on a remote server container, you first need to expose Docker over TCP. After that, your Jenkins controller instance can connect to the remote Docker server via TCP.
You can either modify the Docker configuration to enable TCP access, or, if you prefer not to touch the existing docker config, you can use an image like socat which will act as a relay between TCP and the Unix socket, which will expose Docker over TCP without altering the configuration. To connect from Jenkins controller you should define tcp://<RemoteIP:2375> in Docker Host URI field while configuring cloud.
Execute in remote server.
docker run -d \
--name socat-docker \
-v /var/run/docker.sock:/var/run/docker.sock \
-p 2375:2375 \
alpine/socat \
tcp-listen:2375,fork,reuseaddr unix-connect:/var/run/docker.sock
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9a3510394186 alpine/socat "socat tcp-listen:23…" 1 days ago Up 1 second 0.0.0.0:2375->2375/tcp, :::2375->2375/tcp
Important: Make sure Docker is exposed securely to prevent unauthorized access. The above setup exposes your Docker daemon without authentication, which means anyone who can access port 2375 can control your Docker engine (run containers, delete images, etc.). Therefore, it is strongly recommended to configure TLS for secure access.
Comments