Ubuntu 18.04 / 20.04 / 22.04 / 24.04:
Open a terminal and run this command:
gsettings set org.gnome.mutter center-new-windows 'true'
To change it back to the default behavior, run this command:
gsettings set org.gnome.mutter center-new-windows 'false'
Ubuntu 14.04 / 16.04:
Either search for CompizConfig in the Ubuntu Software Center, or type this command:
sudo apt install compizconfig-settings-manager
Once CompizConfig Settings Manager is installed, run it and go to Window Management > Place Windows and set "Placement Mode" to Centered.
Please note that this requires functioning 3D, Compiz, and Unity 3D (not 2D).