Remastersys Capink


 Transforming your Installation Manually into a Live DVD/CD

by capink from the Ubuntu Forums

This HOWTO is about making a live CD/DVD from the main system on your hard drive. This might be desired if you have customized your system and want to have it on CD. It can be useful also if you want to create a recovery CD from scratch, as you can make a minimal system using debootstrap and transform it into a live CD.

The live CD is usually created with a filesystem called squashfs. Squashfs is read only compressed filesystem that allow us to squeeze our system into a single CD. Note that your system has to be about 2GB (this might need some trial an error) to produce a compressed image that fits on the CD. Ofcourse compression incurs performance penalties. If you are using DVD and your installation is less than 4GB you don't have to use compression, instead you can use ordinary unix (e.g ext2) filesystem as pointed out at the end of this guide.

Quote:

Background on live CD/DVD

Note: This section is a clarification of how live CD works. You don't have to read it. You can skip it if you want.

A live CD/DVD is basically a normal linux installation just like an ordinary harddrive installation. However, simply copying the harddirve installation over to a CD/DVD is not enough to produce a working system. Why? because there are still minor differences between a live CD/DVD and on ordinary harddrive installation. So in addition to copying our harddirve installation to the CD/DVD we must address those differences as well.

So what are these differences?
  1. The CD or DVD is read only media. Linux needs to have write access to certain parts of the system to be able to operate properly (like "/dev" "/proc" "/var" "/tmp"). There are a lot of approaches to address this problem. All of which utilize the system RAM. Some of these approaches enable write access only to essential directories and files, and hence, they do not allow the user to modify the system or install new packages while in the live CD. Other approaches, like unionfs which is what is used in this guide, allows the user to write to any part of the system. This is done by merging part of the RAM with the read-only filesystem of the live CD/DVD and making the look like one filesystem that have read-write access. Unionfs has to be mounted at boot in a certain manner.
  2. With the harddrive installation the location of the root filesystem is fixed. So it is passed to the kernel at boot time using the root=/dev/... parameter. With a live CD/DVD, the location of the root device is not fixed as the user might have multiple cdrom drives, these drives can be ide, scsi ... etc. So for the root filesystem to be mounted, there must be a way to identify the root device, and then we have to load the suitable kernel modules (to be able to access the cdrom controller as well as the CD filesystem). All this has to be done even before we have a root filesystem mounted.
  3. To fit on a CD, the filesystem is usually compressed using squashfs. So we need to autodetect the filesystem type. We also need to have the proper modules for mounting it.




These considerations require special preparation at boot time, some of which must be performed even before mounting the actual filesystem. How can we do this?

Linux introduced a mechanism that allow for such preparations at boot time before the actual root filesystem is mounted. It is called the initial root filesystem or initramfs. This mechanism is used also in mounting normal harddirve installations, as it adds flexibilty to the boot process.


initramfs is virtual filesystem. It is a compressed cpio (cpio is an archive format similar to tar) archive that contains a minimal shell, kernel modules necessary for mounting the root filesystem and number of scripts that perform some tasks at boot time. The most important of these scripts is a script called init located at the root of the initramfs.

How does initramfs work?

The boot loader loads both the kernel and the initramfs into memory and starts the kernel. The kernel then unpacks the initramfs and mount it as initial root filesystem, and then looks for the init program within the initial filesystem, and once it finds it, it executes it and hand the boot process over to it. This init scirpt is responsible for finding the real root filesystem and mounting it. It is also responsible for any special preparations required at boot time.

So any special operations required for booting the system from live media can be coded into the initramfs boot scripts.

How is initramfs is created?

We do not have to create initramfs manually (although it can be done). There are tools for creating and updating initramfs like the command update-initramfs. Moreover, these tools can include custom scripts into the initramfs if they are placed in a certain preset locations (/usr/share/initramfs/scripts). So all we have to do is dump our custom scripts (which will do all the required preparation for booting the live CD/DVD) into these preset locations, and then create a custom initramfs by running update-initramfs.

We don't even have to write these scripts. Why? becuase there are packages that have scripts tailored for booting form live CDs. One of these packages is called casper (this is the package used in this howto). By installing casper into the system, it places the scripts in there proper locations (where they can be spotted by update-initrfamfs). The only thing we need to do after installing casper is running update-initramfs to create an initramfs suitable for live CD/DVD.







The live CD/DVD structure:

The directory tree of the live CD/DVD we are going to create is going to look like this:

Code:

(CD ROOT)
|-------+casper
|        |-------filesystem.${FORMAT}       
|        |-------filesystem.manifest
|        |-------filesystem.manifest-desktop
|       
|-------+boot
|        |--------+grub
|        |        |--------menu.lst
|        |        |--------stage2_eltorito
|        |
|        |-------vmlinuz
|        |-------initrd.gz
|        |-------memtest86+
|
|--------md5sum.txt


  • /casper/filesystem.${FORMAT}: This is the container of the linux filesystem we are going to copy from our harddisk. It is usually a compressed filesystem like squahsfs. But it can be an ordinary unix filesystem like ext2.
  • /casper/filesystem.manifest: This file is optional. You only need it if you decide to include the Ubuntu installer in the CD. The purpose of this file will be explained later.
  • /casper/filesystem.manifest-desktop: This file is optional. You only need it if you decide to include the Ubuntu installer in the CD. The purpose of this file will be explained later.
  • /boot/grub/menu.lst: File containing boot options for the live CD/DVD.
  • /boot/grub/stage2_eltorito: The boot loader of the CD. (stage2 of grub).
  • /boot/vmlinuz: The linux kernel. This is copied form the linux filesystem.
  • /boot/initrd.gz: the initramfs that contain the customizations necessary for the live CD/DVD.
  • /boot/memtest86+: Optional file used to test the RAM of the machine form the live CD/DVD.
  • /md5sum.txt: Optional file containing checksums for all the files in the CD.

What you need:

Outline of the steps:

A. Prepare Our work environment.

B. Copy the Source system to the target directory.

C. Chroot into the new system and make some modifications.

D. Prepare The CD directory tree.

E. Build the CD/DVD

Appendix 1. Using a non compressed filesystem.

Appendix 2. Adapting this guide to Debian.

Appendix 3. Building the live media form scratch using Debootstrap.

Conventions used in this HOWTO:

A. Preparing the environment

1. Set some variables

Code:

export WORK=~/work

export CD=~/cd

export FORMAT=squashfs

export FS_DIR=casper


The WORK Directory is where our temporary files and mount point will reside.

The CD is the location of the CD tree.

FORMAT is the filesystem type. We you are going to use a compressed squashfs

FS_DIR is the location of the actual filesystem image within the cd tree.

Replace only the values highlighted in Magenta.

< People opting for ext2 look at Appendix.1 for modification of this step >

UPDATE: Using ext2 is not necessary as you can use squashfs without compression. See appendix 1 for more details

2. Create the CD and the WORK directory structure:

Code:

sudo mkdir -p ${CD}/{${FS_DIR},boot/grub} ${WORK}/rootfs

3. Install some packages on the source system:

Code:

sudo apt-get update
Code:

sudo apt-get install mkisofs grub squashfs-tools linux-ubuntu-modules-$(uname -r) qemu

qemu is optional. It is only needed for testing the cd before burning it. It can be substituted with any other virtualization software like virtualbox.

linux-ubuntu-modules-$(uname -r) is only needed for Ubuntu Gutsy and later. If using an Ubuntu version prior to Gutsy omit this package as it is part of the main kerenl package.

< People opting for ext2 look at Appendix.1 for additional step to done right here >.

UPDATE: Using ext2 is not necessary as you can use squashfs without compression. See appendix 1 for more details

B. Copy your installation into the new filesystem.

Code:

sudo rsync -av --one-file-system --exclude=/proc/* --exclude=/dev/*\

 --exclude=/sys/* --exclude=/tmp/* --exclude=/home/*\

 --exclude=/lost+found / ${WORK}/rootfs


Note: rsync is used instead of cp to take advantage of the --one-file-system and the --exclude options.

If you have a separate boot partition you will have to copy it using the following command:

Code:

sudo cp -av /boot/* ${WORK}/rootfs/boot

(Optional) Copy settings in your home dir:

If you want to preseve your user account settings which are stored in your home directory, you can copy them to ${WORK}/rootfs/etc/skel/.

But first we have to define what files we want to copy. For example I am using xcfe4 as my DE, and it stores all it settings in a directory called .config in my home directory, so I am going to add .config to the variable $CONFIG:

C. Chroot into the new system and modify it:

1. Chroot into the copied system after mounting proc and dev:

Code:

CONFIG='.config .bashrc'

Now, Copy the CONFIG files using the following command:

Code:

cd ~ && for i in $CONFIG

do

sudo cp -rpv --parents $i ${WORK}/rootfs/etc/skel

done


 

Code:

sudo mount -o bind /dev/ ${WORK}/rootfs/dev
Code:

sudo mount -t proc proc ${WORK}/rootfs/proc
Code:

sudo chroot ${WORK}/rootfs /bin/bash

N.B: All commands in Blue are done within a chroot.

Now you are within chroot environment, type the following command:

Code:

LANG=

2. Install Packages Essential for live CD:

Code:

apt-get update
Code:

apt-get install casper discover1 xresprobe

casper contain the live scirpts.

discover1 & xresprobe are used for autodetectin hardware at startup.

3. (Optional) If you want your live cd to have an installer, install the Ubuntu installer:

Code:

apt-get install ubiquity

(Optional Step)Install any packages you want to be in the CD. Some of the following packages are useful in emergency situations:

sudo apt-get install gparted ms-sys testdisk wipe partimage xfsprogs reiserfsprogs jfsutils ntfs-3g ntfsprogs dosfstools mtools[/code]

gparted: patitioning tool. It is automatically installed as a dependecy of ubiquity.

ms-sys: writing a Microsoft compatible boot record (MBR).

testdisk: Partition scanner and disk recovery tool.

wipe: Secure file deletion.

partimage: backup partitions into a compressed image file (like norton ghost).

xfsprogs reiserfsprogs jfsutils: Tools for handling different filesystems.

mtools: Tools for manipulating MSDOS files

4. Update the initramfs:

First update modules.dep:

Code:

depmod -a $(uname -r)

 

Code:

update-initramfs -u -k $(uname -r)

As already metioned above, the initramfs is reponsible for much of the preparation required at the boot time of the CD/DVD. The updated initramfs now contain the live scirpts installed with casper.

5. Remove non system users

Code:

for i in `cat /etc/passwd | awk -F":" '{print $1}'`

do

        uid=`cat /etc/passwd | grep "^${i}:" | awk -F":" '{print $3}'`

        [ "$uid" -gt "999" -a  "$uid" -ne "65534"  ] && userdel --force ${i} 2>/dev/null

done


Non-system users are users created by you that have user id more than 999.

6. Delete these files.

Code:

for i in "/etc/hosts /etc/hostname /etc/resolv.conf /etc/timezone /etc/fstab /etc/mtab /etc/shadow /etc/shadow- /etc/gshadow  /etc/gshadow- /etc/gdm/gdm-cdd.conf /etc/gdm/gdm.conf-custom /etc/X11/xorg.conf /boot/grub/menu.lst /boot/grub/device.map"

do

        rm $i

done 2>/dev/null


These files are not needed in the CD/DVD. some of them are could interfer with the CD/DVD boot process. (e.g. shadow and gdm.conf-custom can interfere with autologin).

7. Clean the chroot environment form unnecessary files:

Code:

apt-get clean
Code:

find /var/run /var/log /var/mail /var/spool /var/lock /var/backups /var/tmp -type f -exec rm {} \;
Code:

rm -r /tmp/* /root/* 2>/dev/null

8. If you are using GDM recreate it's config file:

Code:

[ -f "/etc/gdm/factory-gdm.conf" ] && cp -f /etc/gdm/factory-gdm.conf /etc/gdm/gdm.conf 2>/dev/null


Sometimes a customized /etc/gdm/gdm.conf can interfere with the live CD/DVD autologin.

9. Create some files in /var/log:

Code:

for i in dpkg.log lastlog mail.log syslog auth.log daemon.log faillog lpr.log mail.warn user.log boot debug mail.err messages wtmp bootstrap.log dmesg kern.log mail.info

do

        touch /var/log/${i}

done


Most of these files are log files that have been cleaned in step 7. We created an empty files in their place to prevent the system from complaining at boot.

10. Exit chroot

Code:

exit

D. Prepare The CD directory tree:

1. Copy the kernel, the updated initrd and memtest prepared in the chroot:

Code:

sudo cp -vp ${WORK}/rootfs/boot/vmlinuz-$(uname -r) ${CD}/boot/vmlinuz
Code:

sudo cp -vp ${WORK}/rootfs/boot/initrd.img-$(uname -r) ${CD}/boot/initrd.gz
Code:

sudo cp -vp ${WORK}/rootfs/boot/memtest86+.bin ${CD}/boot

2. Generate manifest:

Note: This step is only needed if you installed the Ubuntu installer ubiquity. This step generates two files (filesystem.manifest & filesystem.manifest-desktop).

3. Unmount bind mounted dirs:

Code:

sudo chroot ${WORK}/rootfs dpkg-query -W --showformat='${Package} ${Version}\n' | sudo tee ${CD}/${FS_DIR}/filesystem.manifest
Code:

sudo cp -v ${CD}/${FS_DIR}/filesystem.manifest{,-desktop}
Code:

REMOVE='ubiquity casper live-initramfs user-setup discover1 xresprobe os-prober libdebian-installer4'
Code:

for i in $REMOVE

do

        sudo sed -i "/${i}/d" ${CD}/${FS_DIR}/filesystem.manifest-desktop

done


These two files are used by the ubiquity installer when installing to harddisk. These two files are just lists of packages. Ubiquity compares these two files and removes packages unique to filesystem.manifest. This way when installing to harddisk, packages like casper which is only useful in a live CD/DVD are removed. These packages that will be removed at install are defined in the variable $REMOVE

Code:

sudo umount ${WORK}/rootfs/dev ${WORK}/rootfs/proc

4. Convert the directory tree into a squashfs:

Code:

sudo mksquashfs ${WORK}/rootfs ${CD}/${FS_DIR}/filesystem.${FORMAT}

Note: Make sure the resulting file size can fit into your live media.

< People opting for ext2 look at Appendix.1 for modification of this step >

UPDATE: Using ext2 is not necessary as you can use squashfs without compression. See appendix 1 for more details

5. Make Grub the bootloader of the CD

Copy grub file:

Code:

sudo find /boot /usr/lib/grub/ -iname 'stage2_eltorito' -exec cp -v {} ${CD}/boot/grub \;

Make the menu.lst

Code:

sudo gedit ${CD}/boot/grub/menu.lst

Copy the following text into it and save it.

Code:

# By default, boot the first entry.

default 0

# Boot automatically after 30 secs.

timeout 30

color cyan/blue white/blue

title                Start Linux in Graphical Mode

kernel                /boot/vmlinuz BOOT=casper boot=casper nopersistent rw quiet splash

initrd                /boot/initrd.gz

title                Start Linux in Safe Graphical Mode

kernel                /boot/vmlinuz BOOT=casper boot=casper xforcevesa rw quiet splash

initrd                /boot/initrd.gz

title                Start Linux in Text Mode

kernel                /boot/vmlinuz BOOT=casper boot=casper nopersistent textonly rw quiet

initrd                /boot/initrd.gz

title                Start Presistent Live CD

kernel                /boot/vmlinuz BOOT=casper boot=casper persistent rw quiet splash

initrd                /boot/initrd.gz

title                Start Linux Graphical Mode from RAM

kernel                /boot/vmlinuz BOOT=casper boot=casper toram nopersistent rw quiet splash

initrd                /boot/initrd.gz

title                Memory Test

kernel                /boot/memtest86+.bin

title                Boot the First Hard Disk

root                (hd0)

chainloader +1


6. Calculate MD5

Code:

cd $CD && find . -type f -print0 | xargs -0 sudo md5sum | sudo tee ${CD}/md5sum.txt


E. Build the CD/DVD

1. Make the ISO file

Code:

sudo mkisofs -b boot/grub/stage2_eltorito \

-no-emul-boot -boot-load-size 4 -boot-info-table \

-V "Custom Live CD" -cache-inodes -r -J -l \

-o ~/live-cd.iso $CD


2. Test the CD

Test using qemu emulator

Code:

qemu -cdrom ~\live-cd.iso -boot d

Or use any other virtualization program you like.

3. (Optional) Clean our workspace

Final Notes:

Code:

[ -d "$WORK" ] && rm -r $WORK $CD

Note: If you are using ext2 instead of squashfs you may want to skip this step. Keeping these files in place will be handy whenever you update your CD.

UPDATE: Using ext2 is not necessary as you can use squashfs without compression. See appendix 1 for more details

 

 

Appendix 1. Building the CD/DVD using a non compressed filesystem (e.g. ext2)

Update: You can use squashfs without compression by adding the -noI -noD -noF switches to mksquashfs. This is easier than using a unix filesystem. It also has the advantage that you do not have to predict the size of the system in advance, which is needed for ext2 in order to create a virtual filesystem with enough space ( with squashfs you do not need to create a virtual filesystem, you just use mksquashfs to create the squashfs form a directory tree ). I have not tried this option yet. But I imagine it would make using a unix filesystem totally unnecessary.

Instead of using squashfs for the live DVD filesystem you can use an ordinary unix filesystem like ext2.

Advantages of using a unix filesystem:

- The CD/DVD will be faster than a CD/DVD with a compressed fileystem.

- Easier update of the live media provided you have not cleaned your workspace. An ordinary filesystem can be remounted and modified, then you can recreate the ISO image. Squashfs on the other hand, is a read only filesystem so it not as easy to update.

Disadvantages of using a unix filesystem:

- You lose the advantages of filesystem compression. So you will end up with large image.

- You need a DVD unless you are building a minimal system that can fit within a CD without compression (e.g. a non graphical rescue disk).

- Your system size cannot exceed 4GB as that is the maximum filesize supported by ISO9660. Of course this also applies when using squahsfs. But the differece is that a 4GB squashfs can contain roughly 12GB of data within it.

Note: If you are opting for the non compressed filesystem, it makes sesne to go for ext2 instead of ext3. It does not make sense to user a journaling filesystem with it's overhead in a read only media

To use ext2 instead of squashf follow the same steps above with the following modifications:

NOTE: The only filesystem I have tried is ext2. I have not tried any other filesystem and I do not know if they will work.

 

Appendix 2. Adapting this guide to Debian

This guide can be modified to apply to Debian systems as well. There are number of differeces between Ubuntu and Debian that we must take into account:

  1. As of Debian lenny, casper is deprecated and replaced with live-initramfs. live-initramfs is a fork of casper and it has the same options, with one difference in the CD directory tree sturcute. This can be solved by setting the variable FS_DIR=live instead of FS_DIR=casper.
  2. Sqaushfs modules and unionfs modules are in two separate packages.
  3. Ubiquity installer is not present in the Debain repositories. Ubiquity is only needed if you intend to install Linux from the live CD/DVD to the harddisk. I have not tried using ubiquity on Debian so I am not sure if it will work. To install it on Debian you have to add Ubuntu main repository to your sources.list.

So in light of the points mentioned above we have to make the following modifications to adapt the guide to Debian:

Appendix 3. Building the live media form Debootstrap installed system.

Instead of using your current installation to be the basis of you live CD, you can install a custom system into any directory in your system using debootstrap, and then use that as the basis of your CD. The modifications you have to make are

Replace ~/debootstrap/* with the path of the debootstrap installed system.[/list]

 

Edit (1): 20/02/08

  1. Added Suggestions for packages useful in rescue CD.
  2. Added ext2 tag as per RumorsOfWar suggestion.
  3. Modified step D.5 as per RumorsOfWar suggestion.

Edit (2): 22/02/08

If you want a non compressed filesystem (in case of DVD), using a unix filesystem like ext2 is not recommended anymore as you can use squashfs without compression by adding the -noI -noD -noF switches to mksquashfs. The option for creating ext2 filesystem is still present in the guide but with notes to use an uncom