Booting Linux from USB drive

This exercise is somehow similar to booting up an embedded Linux system. The Linux kernel image and bootloader are read from a flash memory. However, one can first test some basic knowledge on Linux boot sequence on a PC.

Requirement

Booting Linux from USB is not supported on all PCs. In the following, I will discuss probably the minimum requirement of the PC/BIOS to support USB booting. However, there could be weird problems with the BIOS which prevents the drive from booting for various reasons.

  • Minimum requirement
    1. The PC BIOS can recognize USB devices and use it as a boot device
    2. The PC can boot up from the USB drive an FAT or NTFS partition
  • For GRUB support
    1. BIOS can emulate HDD (hard drive) instead of FD (floppy disk) for the USB memory. Otherwise, GRUB boot loaders can be confused by the BIOS disk driver and fails to load the system (stuck at GRUB title, no menu). If this condition is not satisfied, a workaround is to use syslinux which loads Linux from a FAT filesystem. Syslinux always works if the PC can boot up MS-DOS from USB.
    2. I found a solution about getting GRUB to work when BIOS only supports FD emulation of the USB memory. Click to open Solution.

Check USB boot setup

  1. Make sure the PC's BIOS supports USB boot-up.
  2. Usually, one would need to enable "(Legacy) USB support" in BIOS settings, and make the USB device to be the first boot device (above HDD, Floppy, Network, DVD, etc.)
  3. Insert a USB drive, restart the computer, and the LED on the USB should flash a couple of times during boot. This means the BIOS can detect the USB drive, and is trying to boot from it. This is a good sign that we can move on to make the USB boot-up stick. A caveat is that the BIOS may or may not support the USB memory as a HDD. If it doesn't emulate an HDD but a FD, this may cause some problems for GRUB, since GRUB under Linux will probably recognize it as a HD, and may have setup the bootloaders as such.

Make a bootable USB drive (MSDOS or Windows)

The point here may be just to start simple. There are existing automatic tools to make MSDOS/Windows bootable disks on USB. If you don't have a Windows system, or don't want to boot into DOS or Windows, you can safely skip this step.

  1. A simple tool for making a bootable USB drive is the HP USB Storage Formatter. It only requires an msdos or windows startup disk image, but it doesn't seem to require the boot sector fromr the image. The extract system files will suffice. For downloading the tools, and system files, please refer to this article: http://www.bay-wolf.com/usbmemstick.htm.
  2. There are other Windows tools to make a bootable USB, such as mkbt: http://www.nu2.nu/mkbt/. The point here is to test out the PC so that we are sure it supports USB boot-up.

Make a Linux USB boot drive

If you have tested out that the BIOS setup is correct, and your computer can indeed boot from the USB memory into DOS. Then, chances are you are only steps away being able to boot up Linux from the USB drive. Here are the steps.

Step 1. Partition the USB drive under Linux
Suppose the USB drive device node is /dev/sdX (e.g., sdX could be sda or sdb), run fdisk as

# fdisk /dev/sdb

Use the menu to delete the existing partition (e.g., FAT, NTFS, etc), and create a new primary partition. The USB mass storage device is appears to emulate a hard drive instead of a floppy disk. So before a filesystem can be put on the drive, you will need to create a partition, and specify the size of the partition. The USB drive is using LBA logic for block addressing, given that the LBA block address needs three numbers - cylinder, head, sector, the number of total cylinders on the USB drive could be fairly small when used to specify the partition size (the default number of heads is 255). After using the "n" command to create a partition, please don't forget to use "a" to mark it as active. Then, you can just "w" and exit fdisk.

Step 2. Format the USB drive partition using the ext2 or ext3 filesystem

# mke2fs /dev/sdb1

Step 3: Mount the drive, and install grub. The mount point can be arbitrarily selected (/mnt/memstick) is used.

# mount /dev/sdb1 /mnt/memstick
# cd /mnt/memstick
# grub-install --recheck --root-directory=/mnt/memstick /dev/sdb1

The "grub-install" command will setup the MBR and boot sector of the USB drive (GRUB stage1), and hopefully copy stage2 file to the correct position of the drive (sometimes this could fail, and after rebooting, GRUB may be stuck at stage1, with only the prompt "GRUB" on top of the screen, see trouble shooting below). Besides stage1 and stage2, it will copy grub files into /boot/grub, including the following files.
default
device.map
e2fs_stage1_5
fat_stage1_5
installed-version
jfs_stage1_5
minix_stage1_5
reiserfs_stage1_5
stage1
stage2
xfs_stage1_5

We can go ahead to delete *_stage1_5 files for unused file systems, because the only one we need is one file system for the partition. In this example, it is ext2_stage1_5, so we can delete fat_stage1_5, jfs_stage1_5, minix_stage1_5, reiserfs_stage1_5 and xfs_stage1_5.

Step 4: Copy linux files into the USB drive
Since linux kernel and ramdisk images, and other system files are not there yet. We need to copy them over to the USB drive. First, copy the kernel (vmlinuz-a.b.c-d) and the ramdisk image (initrd.img.a.b.c-d ) into the /boot directory of the USB partition (other directories are fine too). We will copy other system files after GRUB can boot up the kernel.

Step 5: Create the menu.lst file under /mnt/memstick/boot/grub. The menu.lst file defines the entries in the GRUB memu. For older version of GRUB, the file name is grub.conf. The file is critical for booting up Linux from USB. To set it up correctly, we need to know the BIOS device number and Linux device node for the USB partition on which we want to install Linux. Good news is that GRUB provides a good way to find them when the USB drive in mounted in the current Linux environment. The bad news, however , is that they may turn out to be different when we are booting the USB drive (see trouble shooting below). Suppose we know the correct BIOS device number and the Linux device node for the USB partition, the format of this file is

default=0
timeout=10
root=(hd0,0)

title Linux from USB
kernel /boot/vmlinuz-a.b.c-d root=/dev/sda1 ro
initrd /boot/initrd.img-a.b.c-d

The commands "kernel" and "initrd" will specify where the kernel and the initrd images files reside under the USB partition. The root entry specifies the partition in BIOS format.

The way to look for the BIOS device number for the USB partition under the current Linux environment is probably easy. You can just find the mapping in the file "device-map" generated by grub-install. For example, the first partition on the first IDE/SATA harddrive will be (hd0,0), the second partition (hd0,1). If Linux puts the USB drive after the hard drive, the USB partition will probably be (hd1,0). This can be double-checked by running "grub", and type the following command:

grub> find /boot/grub/menu.lst
 (hd0,4)
 (hd1,0)

This command will search all partitions for the specified menu.lst file we just put on the USB drive. On my system, the first drive (hd0,4) in on my hard drive, which has the file too (I had two Windows partitions, one Mac OS partition, an extended partition containing linux as the first logic drive, so it's hd0,4 or /dev/sda5). The second one (hd1,0) is the correct BIOS drive number for the USB drive. Therefore, we have got all information to fill out the menu.lst file.

After editing the menu.lst file, we don't need to reinstall GRUB to the disk. This menu.lst file is read by the binary code for stage2, and we don't need to update other files because of a change in menu.lst.

If we are lucky, after rebooting, we can see GRUB loads successful from the USB drive, displaying the choices in menu.lst, and it will simply load up the kernel correctly. However, many things may go wrong to prevent GRUB from working on USB. Here, I will first discuss some common problems, and discuss work-arounds, and alternative methods.

Setup other Linux system files on the USB drive

If the bootloader (GRUB or syslinux) can successfully load the kernel from the file system, and starts the kernel, then we are ready to obtain a fully working Linux system on the USB.

Troubleshooting USB booting

BIOS doesn't support HDD emulation of USB
When we mount the USB drive under Linux, the BIOS device may be (hd1,0). However, when it is driven by BIOS instead of Linux, it may change to (fd0)! To find the BIOS device number, one can boot to the usual GRUB from HD, type 'c' to enter the command line mode, and use "find /boot/grub/menu.lst" to search for the USB device during boot. If the USB partition shows up as a floppy device (fd0), then we know that booting GRUB from the USB disk on the system may be very difficult. The problem is that the USB might not boot up to "stage2". However, if it does load stage2 and show up the GRUB menu. We can still try to boot the kernel manually (note that the root filesystem is not yet set up here, but just to test the kernel booting).

grub> find /boot/grub/menu.lst
 (hd0,4)
 (fd0)
grub> root (fd0)
grub> kernel=/boot/vmlinuz-a.b.c-d
grub> boot

System is stuck at the "GRUB" prompt
The GRUB prompt is displayed by the "stage1" code of GRUB — the 512 bytes residing on the first sector of the drive/partition. GRUB stage1 can either be installed on MBR of the disk or the boot sector of a partition. If the boot-up process is stuck there, it means that "stage2" failed to load into RAM. This is likely to be failed sector/block addressing in "stage1" for the "stage2" code.

There is no good solution for this problem yet. Some things to try include using the FAT16 partition, instead of ext2/3. It may indeed be quite easy to load up the kernel from a FAT16 system, with the help of a DOS device drive. This option is provided by the tool syslinux, which can be easily installed on the USB FAT16 partition:

# syslinux /dev/sdb1
# cp /boot/vmlinuz-a.b.c.d /mnt/memstick/kernel.img

Then, reboot and type "kernel: /kernel.img" after ldlinux.sys has booted up, and asks for the kernel.img. The rest of the things is then setup the root filesystem.

It may still be worth to try to hack GRUB to do the (fd0) loading. I am still working on this problem of GRUB…. See the solution.

System is booting from USB but it loads the wrong menu (from HD) instead of the USB drive
This happens possibly because the grub files for the HD (the absolute /boot/grub/*) are copied to the USB drive. This problem itself be fixed by adding '—root-directory=/mnt/memstick' option to the "grub-install" command above.

The GRUB menu is loaded by "stage2". If this happens, the cause is that "stage1" (on the USB MBR) is attempting to load the "stage2" on a wrong drive. By referring to stage1 source code "stage1.S", we can see that it has a byte specifying the drive to load "stage2" from… the code snippet is here:

   87     /*
   88      * End of BIOS parameter block.
   89      */
   90 
   91 stage1_version:    
   92     .byte    COMPAT_VERSION_MAJOR, COMPAT_VERSION_MINOR
   93 boot_drive:    
   94     .byte    GRUB_INVALID_DRIVE    /* the disk to load stage2 from */
   95 force_lba:
   96     .byte    0
   97 stage2_address:
   98     .word    0x8000
   99 stage2_sector:
  100     .long    1
  101 stage2_segment:
  102     .word    0x800

The byte at Line 94 specifies the disk to load stage2 from. It's likely that this byte is wrong in this case. To resolve this, we can just run a correct "grub-install" command with "—root-directory" option or use "setup" command in "grub". If the problem persists, you may try to manually check if the 'boot_drive' byte (e.g., index 0x40 in Grub 0.97) is set correctly. The value 0x80 means it is hard-coded to be HD. If the value is 0xFF (invalid_drive or missing drive), then stage1 bootloader will use the drive letter passed by BIOS (which is hopefully the current boot device) in the x86 register DL. To do this one can run the following commands:
# dd if=/dev/sdb1 of=bootsect.bin bs=512 count=1
# hexdump -C bootsect.bin

The tool dd reads the first sector of the partition /dev/sdb1 and save it to the file bootsect.bin. "bs=512" specifies the block size, and count=1 specifies one sector to read. Hexdump will show the content in the stage1 boot loader in hex format, in which you can check the 'boot_device' byte (byte 0x40) and see if it has been hardcoded to be HD (0x80). If so, we would need to fix it unless the USB drive is indeed recognized as HD0 by BIOS.

References

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License