This guide explains how you can run virtual machines with VirtualBox 5.1 on a headless Ubuntu 16.04 server. Normally you use the VirtualBox GUI to manage your virtual machines, but a server does not have a desktop environment. Fortunately, VirtualBox comes with a tool called VBoxHeadless that allows you to connect to the virtual machines over a remote desktop connection, so there’s no need for the VirtualBox GUI.

1 Preliminary Note
I have tested this on an Ubuntu 16.04 server (host system) where I’m logged in as a normal user.

2 Installing VirtualBox
To install VirtualBox 5.1 on our Ubuntu 16.04 server, we add virtualbox sources to apt …

echo 'deb xenial contrib' | sudo tee /etc/apt/sources.list.d/virtualbox.list

Then we download the VirtualBox public key…
wget -q -O- | sudo apt-key add -

… and update our package database:
sudo apt-get update

Afterwards, we install VirtualBox 5.1 as follows:
sudo apt-get install linux-headers-$(uname -r) build-essential virtualbox-5.1 dkms

(The dkms package ensures that the VirtualBox host kernel modules are properly updated if the Linux kernel version changes.)

3 Installing VirtualBox Extensions
VirtualBox has “extension packs” that provide functionality like remote desktop connection support (VRDP).
VBOXVERSION=`VBoxManage --version | sed -r 's/([0-9])\.([0-9])\.([0-9]{1,2}).*/\1.\2.\3/'`
wget -q -N "$VBOXVERSION/Oracle_VM_VirtualBox_Extension_Pack-$VBOXVERSION.vbox-extpack"
sudo VBoxManage extpack install --replace Oracle*.vbox-extpack

8. Clean up and reboot
As good practice I always update and clean up before rebooting when installing new software
sudo apt update
sudo apt upgrade -y
sudo apt autoremove -y
sudo apt autoclean -y

sudo reboot now