Hi everyone,

I’ve been wanting to do this one for a long time, but I wanted a physical PC that I could show booting first, rather than a virtual machine. After the latest attempt at buying a used PC resulted in a blown PSU and a broken HDD, I have now given up with that :P. Instead, I’ll be showing you how to do it with a VM, which isn’t all that different, and how to customise some boot config and that kind of thing too, in case, like me, you have (had) annoying hardware that doesn’t work out of the box. Edit: Turns out my new laptop can do this – see the video below.

This isn’t meant to be a super-thorough guide – there’s too much complexity for one post here. Instead of putting all of it here, I will make a short series of howtos, and I will update some wiki pages with fresh information as well, as I found them to be wrong/out of date when I was figuring this out, which will also have more detailed information.

So, first off, this configuration is a bit weird; you might normally use a PXE server to install an OS on a lot of system over the network. What this post covers is setting it up so that you can boot a pre-installed OS, without modifying it, both over PXE, and from the hard drive where it is installed. I had a lot of trouble figuring this out :P. What’s all the more weird is that I used a virtual machine as the PXE server, which does actually work, interestingly. This guide will follow the steps I took, but as I mentioned before, there will be updates to the Ubuntu community wiki pages with more general help as well.

Links:

• See the demonstration in this post.
• See some extra server config for convenience in this post

Assumptions – quick summary

This guide assumes:

• You are using an Ubuntu-based PXE server.
• This server has access to the disks containing the operating systems you want to boot.
• You want to boot BIOS-based PXE clients – not UEFI.
• ^ UEFI will not be covered here – saved for a later post.
• You want to boot Ubuntu and/or Fedora-based OSs over PXE.
• You are prepared to accept that this guide may not be perfect, and may not work for your exact hardware configuration.
• … But I am happy to help in the above case 🙂
• You have patience and are prepared for things to be difficult and not work the first time 🙂

PXE Server Setup & Configuration

This guide uses a Ubuntu virtual machine as the PXE server, though of course it is possible to do it with other operating systems too, even Windows. I used Ubuntu server 16.04, but a desktop version will work fine too. This guide assumes you’ve done a fresh install of some Ubuntu-based OS.

So, sort of following the order of this wiki page, first you need to install the dependencies:

sudo apt-get install isc-dhcp-server tftpd-hpa syslinux pxelinux nfs-kernel-server initramfs-tools

This will pull in some other packages as well, but it should be fairly fast.

DHCP Server configuration

PXE Clients need some way of getting a network address so they can start to boot. The easiest way to do this is to have your PXE boot system set up as a DHCP server as well. We just installed the DHCP server software, so next we need to do the configuration. You can customise this as much as you like.

The configuration file can be found at /etc/dhcp/dhcpd.conf. Here is what I went for, keeping it simplistic:

allow booting;
allow bootp;

subnet 192.168.2.0 netmask 255.255.255.0 {
range 192.168.2.100 192.168.2.200;
option broadcast-address 192.168.2.255;

filename “/tftpboot/pxelinux.0”;
}

There are some other options as well, like specifying DNS servers, routers, and fixed IPs for particular machines, but they aren’t covered here to try to keep it simple. They’ll be on the wiki page. And perhaps in a follow up post.

TFTP Server Configuration

This is the next thing your TFTP client(s) will look for when booting up. These are used for (in the case of Linux) loading your kernel and initial ram filesystem, so your clients can start to boot up. To configure tftp-hpa, you need to create a /etc/default/tftp-hpa file with the following contents:


RUN_DAEMON="yes"
TFTP_USERNAME="nobody"
TFTP_OPTIONS="-l -s /tftpboot"
TFTP_ADDRESS=":69"


I will explain what all these options mean in a later post – coming soon – but for now just leave them like this.

TFTP boot directory

Next, you need somewhere to store your TFTP boot files. I just created a new folder at /tftpboot for this:

sudo mkdir /tftpboot

Here, you’ll need both the kernel and initial ram filesystem for each OS you want to boot, as well as the configuration for the boot menu and boot options.

Boot menu and options

This needs to go in a subfolder of your boot directory:

sudo mkdir /tftpboot/pxelinux.cfg

In here, you’ll need to copy some bootloader files across from your system’s directory. On Ubuntu 14.10 and later, you’ll need to run:


sudo cp /usr/lib/PXELINUX/pxelinux.0 /tftpboot
sudo mkdir -p /tftpboot/boot
sudo cp -r /usr/lib/syslinux/modules/bios /tftpboot/boot/isolinux


Next, you need to create your boot menu. Create /tftpboot/pxelinux.cfg/default, and follow this example for each boot option you need:


LABEL MyLinuxDistro
KERNEL name_of_kernel
APPEND root=/dev/nfs initrd=name_of_initrd nfsroot=(IP of your server):/nfsroot/ ip=dhcp rw


Obviously, replace name_of_kernel with the name of your kernel, and name_of_initrd with the name of your initial ram filesystem, but you can do that later. We’ll cover getting those files and putting them in the right place later on – don’t worry about it for now.

If you want you can set a default with:

DEFAULT=some_default_label

Or for a graphical menu (I use this):

DEFAULT=menu.c32

Make sure to put these at the beginning of the file.

Note: For Fedora clients, you’ll need to add the selinux=0 option, otherwise you won’t be able to log in!

Next, make sure this directory has the right permissions for clients to read the files:

sudo chmod -R 777 /tftpboot

And finally, you’ll need to start the TFTP server (or just reboot):

sudo /etc/init.d/tftpd-hpa start

Getting the kernel and initial ram filesystems into place

NOTE: Customising the initial ram filesystem will be covered in a later post and linked to here later.

IMPORTANT: These commands are run on the operating systems you want to boot over the network, NOT the boot server! Make sure you copy these files to the boot server somehow though, and put them in the right place.

For now, we’ll just go with using the kernel and initramfs with stock configuration.

Kernel

This is the easy part. For each operating system, find the kernel file, which should be in /boot somewhere, and called something like:

vmlinuz-(some-version-number)-generic

or:

vmlinuz-(some-version-number).fc(your-fedora-version).(your-architecture)

I suggest you make subfolders under /tftpboot, for example /tftpboot/ubuntu-16.04, and put the kernel for each OS in its own folder, so keep things organised. Remember, these folders are on the *boot server*, not the operating system you want to boot.

Initial ram filesystem

This is more difficult, depending on your operating system.

On Fedora

Your best bet is to download a netboot initrd for your version of fedora, which can be found at:

https://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/27/Everything/x86_64/os/images/pxeboot/

That example was for Fedora 27, but just change the version in the URL for whatever version of fedora you have. Once you have the file, put it in the same folder as the kernel for that version of Fedora, on the *boot server*.

On Ubuntu

Ugh, this was a pain. You have to create your own initrd, which means messing with boot settings, creating the initrd, then putting it with the kernel, and then reverting all the changes and rebuilding so the next boot from HDD doesn’t fail. Sigh.

So, first name a note of your kernel version, which you can get by running:

uname -r

On my machine at the time of writing, this returns “4.13.0-36-generic”.

Next, open /etc/initramfs-tools/initramfs.cfg as root, and change the BOOT flag from “local” to “nfs”:

BOOT=local

Becomes:

BOOT=nfs

Next change the MODULES flag from “most” to “netboot”:

MODULES=most

Becomes:

MODULES=netboot

If the options can’t be found, just add them near the top of the file, don’t worry about it.

Next you need to generate your new initrd:

sudo update-initramfs -u

Then, copy your new initrd file at:

/boot/initrd.img-your-kernel-version

To the right folder on the *boot server*, where you put your kernel file.

You are NOT done yet!

Next you need to revert *all* the changes you made to /etc/initramfs-tool/initramfs.cfg, and update your initrd again with “sudo update-initramfs -U” *before* you reboot, or you won’t be able to boot up again!

Note: I’ve found that you need to re-do this step (and the kernel steps) every now and then on Ubuntu – the drivers are eventually removed for the older kernels, so for example, your storage devices may stop being recognised after a while.

NFS Boot directory

What a minute, I’ve already got TFTP, what do I need to set NFS up for? Well, TFTP is used to the the kernel and initial ramdisk to your pxe booting clients, but they still need a way to access the operating system files so they can boot up properly. Using NFS is a good way to do this.

First, create your NFS “root directory”:

sudo mkdir /nfsroot

Also, make any subfolders you need for your operating systems. For example:


sudo mkdir /nfsroot/ubuntu-16.04
sudo mkdir /nfsroot/fedora-27


These subfolders will be where the files of each operating system will be available. You CAN copy all the files here, but I found it easier to just mount the local disks there instead. For example, if your fedora-27 installation disk is /dev/sdc, then run:

sudo mount /dev/sdc1 /nfsroot/fedora-27

Next, you need to explicitly allow access to each of these folders, so your clients have permission to boot – otherwise you’ll get permission denied errors. Edit /etc/exports to do this, adding a line like:


/nfsroot/ubuntu-16.04 192.168.2.*(rw,no_root_squash,async,insecure)


For each folder you need to share. Change 192.168.2.* to the network your clients will be on, or specify individual clients IPs with 192.168.2.86 for example.

Then run


sudo exportfs -rv


To activate this new configuration. As before, these options will be covered in more detail in a later post. To get your OSes to boot, just add the nofail option to all filesystems in /etc/fstab for each one. I will put more detail for this in a separate post as well.

A little bit of client configuration

/etc/fstab configuration

You need to edit you /etc/fstab file slightly, because otherwise when the system attempts to mount the local disks your boot will fail! There’s a handy option, “nofail”, you can add to work around this issue.

To apply the fix, each each line in /etc/fstab that mounts a filesystem and add nofail to the beginning of the options list.

For example:

/dev/sda1 / ext4 errors=remount-ro 0 1

Becomes:

/dev/sda1 / ext4 nofail,errors=remount-ro 0 1.

/etc/network/interfaces configuration (old Linux versions)

I only needed to do this on ubuntu 14.04, but you may find your system freezes at some stage during boot until you add the line:

iface eth0 inet manual

To /etc/network/interfaces. Of course, if you’re booting off eth1, then you’ll need to change the interface name in the file.

Speeding the boot process up

Note: The Fedora systems sometimes take a while to start – I’m not yet sure why.

You may have notices that boot up tends to hang for 90 seconds (on systemd boot systems at least) when the system looks for the local disks. You can change this timeout, but this is more complex, so I will add this in a separate post soon. Rest assured that the boot will finish, it’ll just have a longish delay.

Finally, you should now have a fully-configured PXE server!

The only thing left is to configure each of your clients to boot using PXE first (you can find these options in your firmware setup program), and make sure any routers you have between the server and the clients support forwarding DHCP BootP packets (most do, as far as I can tell). I recommend you test this with a client plugged straight into the server to start with – there’s a lot that can go wrong. I will add a troubleshooting page and link to it here in due course, but this post is more than long enough already I think. It certainly took me a long time to get this to work – all of these instructions are things I figured out using wiki pages and other instructions, most of which were wrong and out of date, and trial and error :).

I’ll be following this up shortly with a UEFI booting section, as well as short sub-guides for specialised bits of configuration, for if you have stupid hardware like me, or a specific purpose you need this guide to fill. There will also be a demonstration showing my exact configuration, and how to get your machines to boot.

Hamish