Install Ollama Under Docker
Ahmad Mujahid Abdurrahman

Install Docker Engine
Update package indices and install the core Docker runtime components:
sudo apt update && sudo apt install -y curl docker.io util-linux-extra
Enable the Docker service to automatically start on boot:
sudo systemctl start docker
sudo systemctl enable docker
Create the Docker group
sudo groupadd docker
Configure non-root user permissions so you can run Docker commands without prefixing sudo:
sudo usermod -aG docker $USER
Note: You must log out and back into your Ubuntu user session for these group changes to apply.
Activate the new group membership in your current shell session:
newgrp docker
Option 1: CPU Only Installation
Install Ollama with CPU only is as simple as running the following command:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
Option 2: GPU Installation (NVIDIA)
To install Ollama with GPU support, you need to install the NVIDIA Container Toolkit. Follow these steps.
Install prerequisites for NVIDIA GPU support:
sudo apt update && sudo apt install -y --no-install-recommends \
ca-certificates \
curl \
gnupg2
Configure the NVIDIA package repositories and install the NVIDIA Container Toolkit:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
Update the package lists and install the NVIDIA Container Toolkit:
sudo apt update
Install the NVIDIA Container Toolkit and its dependencies:
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt install -y \
nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
Source: NVIDIA Container Toolkit
Install Ollama with GPU support in Docker:
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
Run Ollama
docker exec -it ollama ollama run llama2
List of avalilable models can be found at Ollama Models.