One of the cool little experiments in the home lab recently was seeing just how many Docker containers could run on a single physical Proxmox host, hosting a number of Docker container hosts. I could easily scale containers up to 1000 in each virtual machine using virtual machines and Docker Swarm. Let’s look at Proxmox Docker container host monster :). First though, let’s go through some basics of running Docker containers on Proxmox. You can skip directly to the section where I run 13000 containers in the table of contents.
Table of contents
Running Docker in VMs vs LXC Containers: A Quick Comparison
Let’s compare running Docker in a virtual machine vs. LXC containers. It’s worth comparing the two approaches.
Resource Efficiency: LXC containers are generally more resource-efficient than VMs, as they share the host system’s kernel and use fewer resources. This makes LXC containers a more lightweight option for running Docker containers, especially in environments where resources are limited.
Isolation: VMs provide a higher isolation level between the host and guest operating systems, as each VM runs its own kernel. This can be beneficial from a security perspective. However, LXC containers also provide a decent level of isolation, making them suitable for most use cases.
Compatibility: Docker containers should run consistently across different environments, whether in VMs or LXC containers. However, certain advanced features may be more challenging to implement in LXC containers compared to VMs. It is essential to test your specific use case to determine the best approach for your environment.
Ultimately, the choice between VMs and LXC containers for running your Proxmox Docker Container Monster setup depends on your needs and resource availability.
How to Set Up Docker Containers in Proxmox Using an LXC Container
Setting up Docker containers in Proxmox using an LXC container involves a few key steps. In this tutorial, we will walk you through creating an LXC container, installing Docker Engine, and setting up Docker Compose.
Use the Turnkey Core template
Another option for a base operating system is a Docker container host running on Turnkey Core Linux as a template you can download from your Proxmox host. Download the Turnkey Core container template from the Proxmox template repository. This will be used to create an LXC container for running Docker containers.
Downloading templates
Navigate to the storage location where you want to store the template, then click “Templates” and search for “Turnkey Core.” Download the template and wait for the process to complete. You can create a new LXC unprivileged container to run Docker as a Docker host on Proxmox VE.
Create a Ubuntu Server 22.04 VM
In preparing to run 13,000 containers. I found the LXC approach did not scale as well as virtual machines. I For me, going the route of virtual machine hosts for Docker provided the results I was after. How do you create a Docker container host using a virtual machine.
First, you must create a new VM running Ubuntu Server 22.04 as the base operating system. In the Proxmox web interface, click on “Create VM” and configure the necessary settings such as hostname, disk size, and storage location. Once the VM is created, proceed with installing Ubuntu Server 22.04.
Below, I have created an Ubuntu Server 22.04 virtual machine template for cloning additional VMs for container hosts.
Clone from your template virtual machine
Now that you have created the Ubuntu virtual machine template, we want to install Docker so we can use this installation to create a virtual machine template, which will save time. Once you have the VM template, you can clone from this virtual machine for your Docker container hosts. We can configure the container with the necessary resources, such as CPU, memory, and storage.
Installing Docker
Once the VM is created and running, access it via the Proxmox host console and run the following commands. Since I am using the Ubuntu Server platform, I am following the official guidance from Docker here on installing Docker in Ubuntu: Install Docker Engine on Ubuntu | Docker Documentation.
Proceed with the installation of a few prerequisites needed.
sudo apt-get update
sudo apt-get install
ca-certificates
curl
gnupgInstall the official GPG key:
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpgUse the following commands to setup the repository:
echo
"deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu
"$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" |
sudo tee /etc/apt/sources.list.d/docker.list > /dev/nullRun an apt update to update the package index:
sudo apt-get updateFinally, install Docker Engine on the Linux container.
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-pluginUse Portainer to manage your containers
To manage your Docker containers more easily, it’s recommended to install Portainer, a web-based interface for managing Docker environments. To install Portainer, run the following command:
docker run -d -p 9000:9000 --name portainer --restart always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ceOnce Portainer is installed, access its web interface by navigating to http://<your_proxmox_host_ip>:9000. Log in with your chosen username and password be able to manage your Docker containers.
Create Docker Containers in Portainer
Now that you have Portainer installed and configured, you can create Docker containers directly from the web interface. Click “Containers” in the left-hand menu, then click “Add container.” Provide the necessary details, such as the container name, image, and any required environment variables.
Docker compose and application stacks
To create containers using Docker Compose, you can either upload a Docker Compose YAML file or create one directly within the Portainer interface. Click “Stacks” in the left-hand menu, then click “Add stack.” Provide the necessary details, and Portainer will create the Docker containers based on the Docker Compose configuration.
You’ve successfully created a Proxmox Docker Container Monster environment using an LXC container with this setup. This allows you to run multiple Docker containers on your Proxmox host with minimal overhead and manage them easily using Portainer.
Install Docker Engine on the Linux Container host system
Before diving into the Docker installation process, ensure your LXC container is up to date by running the following commands:
sudo apt update sudo apt upgradeAfter updating the container, proceed with the installation of Docker by executing these commands:
curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo usermod -aG docker $USERThese commands download and install Docker Engine on your Linux container, allowing you to run Docker containers seamlessly.
Scaling up to 13000 Containers!
I wanted to see what my Supermicro E300-9D would do from a container perspective. It has 16 cores and 128 GB of memory. So I cloned the virtual machine template for VM Docker container hosts and was able to scale the containers in each one to 1000 containers each using 12 GB of memory configured.
I used Docker Swarm as an easy way to spin up (scaling) the containers to the desired number on each host.
docker swarm init
Below is an example of me testing a couple of containers on a single VM. As you can see below, I have spun up 800 nginx containers and 200 redis containers. However, I settled on just a single application container in the final test. So, I spun up 1000 containers on one host (nginx) and 1000 on another (redis), until I reached 13 VMs running 13000 containers.
To create a service, use this command:
docker service create --name nginx nginx:latestThen, to scale the service, you can use the command:
docker service scale nginx=1000
At 13,000 containers, the host’s resources were basically exhausted. I was running right at 100% CPU and memory was sitting at 95% memory. But it was up and running!
To recap:
I cloned 13 virtual machines in Proxmox (Ubuntu 22.04)
I initialized the virtual machine as a single Docker Swarm host
I then create a service (alternated between nginx and redis with each VM)
Then, I scaled the service in each virtual machine to 1000 containers
It was great to see that running so many containers was possible on a single server. It is a testament to just how efficient containers are, and when it comes to running applications at scale, they are superior to VMs. It is just a matter of pointing your DNS server to the applications configured in containers for end users to access.
Comments