When you get to the stage where you get to choose what to install (it’s called tasksel and looks like : untick whatever desktop environment is selected. You’ll get a bare minimal install with no graphics server
Reboot into the installed system, and install whatever you want from there.
I would suggest: