How To Build a Minimal Linux System from Source Code <AUTHOR>Greg O'Keefe, <tt></tt> <DATE>v0.9, November 2000 <ABSTRACT> These are instructions for building a minimal linux system from source code. It used to be part of <URL URL="" NAME ="From PowerUp to Bash Prompt"> but I've separated it to keep both documents short and focussed. The system we build here is <EM>very</EM> minimal, and not ready for real work. If you want to build a practical system from scratch, see Gerard Beekmans' <URL URL="" NAME="Linux From Scratch HOWTO"> instead. </ABSTRACT> <TOC> <SECT>What You Will Need <P> We will install a Linux distribution like Red Hat in one partition, and use that to build a new Linux system in another partition. I will call the system we are building the ``target'' and the system we are using to build it with, the ``source'' (not to be confused with <EM>source code</EM> which we will also be using.) <P> So you are going to need a machine with two spare partitions on it. If you can, use a machine with nothing important on it. You could use an existing Linux installation as the source system, but I wouldn't recommend that. If you leave a parameter out of one of the commands we will issue, you could accidentally install stuff to this system. This could lead to incompatibilites and strife. <P> Older PC hardware, mostly 486's and earlier, have an annoying limitation in their bios. They can not read from a hard disk past the first 512M. This is not too much of a problem for Linux, because once it is up, it does its own disk io, bypassing the bios. But for Linux to get loaded by these old machines, the kernel has to reside somewhere below 512M. If you have one of these machines you will need to have a separate partition completely below the 512M mark, to mount as <TT>/boot</TT> for any partitions that are over that 512M mark. <P> Last time I did this, I used Red Hat 6.1 as a source system. I installed the base system plus <ITEMIZE> <ITEM>cpp <ITEM>egcs <ITEM>egcs-c++ <ITEM>patch <ITEM>make <ITEM>dev86 <ITEM>ncurses-devel <ITEM>glibc-devel <ITEM>kernel-headers </ITEMIZE> I also had X-window and Mozilla so I could read documentation easily, but that's not really necessary. By the time I had finished working, it had used about 350M of disk space. (Seems a bit high, I wonder why?) <P> The finished target system took 650M, but that includes all the source code and intermediate build files. If space is tight, you should do a <TT>make clean</TT> after each package is built. Still, this mind boggling bloat is a bit of a worry. <P> Finally, you are going to need the source code for the system we are going to build. These are the ``packages'' that I have discussed in this document. These can be obtained from a source cd, or from the internet. I'll give URL's for the USA sites and for Australian mirrors. <P> <LABEL ID="downloads"> <ITEMIZE> <ITEM>MAKEDEV <URL URL="" NAME="USA"> Another <URL URL="" NAME="USA"> site <ITEM>Lilo <URL URL="" NAME="USA">, <URL URL="" NAME="Australia">. <ITEM>Linux Kernel Use one of the mirrors listed at <URL URL="" NAME="home page"> rather than <URL URL="" NAME="USA"> because they are always overloaded. <URL URL="" NAME="Australia"> <ITEM>GNU libc itself, and the linuxthreads addon are at <URL URL="" NAME="USA"> <URL URL="" NAME="Australia"> <ITEM>GNU libc addons You will also need the linuxthreads and libcrypt addons. If libcrypt is not there it is because of some US export laws. You can get it at <URL URL="" NAME="libcrypt"> The linuxthreads addon is in the same places as libc itself <ITEM>GNU ncurses <URL URL="" NAME="USA"> <URL URL="" NAME="Australia"> <ITEM>SysVinit <URL URL="" NAME="USA"> <URL URL="" NAME="Australia"> <ITEM>GNU Bash <URL URL="" NAME="USA"> <URL URL="" NAME="Australia"> <ITEM>GNU sh-utils <URL URL="" NAME="USA"> <URL URL="" NAME="Australia"> <ITEM>util-linux <URL URL="" NAME="Somewhere else"> <URL URL="" NAME="Australia"> This package contains <TT>agetty</TT> and <TT>login</TT>. </ITEMIZE> <P> To sum up then, you will need: <ITEMIZE> <ITEM>A machine with two spare partitions of about 400M and 700M respectively though you could probably get away with less <ITEM>A Linux distribution (eg. a Red Hat cd) and a way of installing it (eg. a cdrom drive) <ITEM>The source code tarballs listed above </ITEMIZE> <P> I'm assuming that you can install the source system yourself, without any help from me. From here on, I'll assume that its done. <P> The first milestone in this little project is getting the kernel to boot up and panic because it can't find an <TT>init</TT>. This means we are going to have to install a kernel, and install lilo. To install lilo nicely though, we will need the device files in the target <TT>/dev</TT> directory. Lilo needs them to do the low level disk access necessary to write the boot sector. MAKEDEV is the script that creates these device files. (You can just copy them from the source system of course, but that's cheating!) But first of all, we need a filesystem to put all of this into. <SECT>The Filesystem <P> Our new system is going to live in a file system. So first, we have to make that file system using <TT>mke2fs</TT>. Then mount it somewhere. I'd suggest <TT>/mnt/target</TT>. In what follows, I'll assume that this is where it is. You could save yourself a bit of time by putting an entry in <TT>/etc/fstab</TT> so that it mounts there automatically when the source system comes up. <P> When we boot up the target system, the stuff that's now in <TT>/mnt/target</TT> will be in <TT>/</TT>. <P> We need a directory structure on target. Have a look at the File Heirarchy Standard (see section <REF ID="FHS" NAME="Filesystem">) to work out what this should be, or just <TT>cd</TT> to where the target is mounted and blindly do <VERB> mkdir bin boot dev etc home lib mnt root sbin tmp usr var cd var; mkdir lock log run spool cd ../usr; mkdir bin include lib local sbin share src cd share/; mkdir man; cd man mkdir man1 man2 man3 ... man9 </VERB> Since the FHS and most packages disagree about where man pages should go, we need a symlink <VERB> cd ../..; ln -s share/man man </VERB> <SECT>MAKEDEV <P> We will put the source code in the target <TT>/usr/src</TT> directory. So for example, if your target file system is mounted on <TT>/mnt/target</TT> and your tarballs are in <TT>/root</TT>, you would do <VERB> cd /mnt/target/usr/src tar -xzvf /root/MAKEDEV-2.5.tar.gz </VERB> <P> Don't be completely lame and copy the tarball to the place where you are going to extract it ;-> <P> Normally when you install software, you are installing it onto the system that is running. We don't want to do that though, we want to install it as though <TT>/mnt/target</TT> is the root filesystem. Different packages have different ways of letting you do this. For MAKEDEV you do <VERB> ROOT=/mnt/target make install </VERB> <P> You need to look out for these options in the README and INSTALL files or by doing a <TT>./configure --help</TT>. <P> Have a look in MAKEDEV's <TT>Makefile</TT> to see what it does with the <TT>ROOT</TT> varible that we set in that command. Then have a look in the man page by doing <TT>man ./</TT> to see how it works. You'll find that the way to make our device files is to <TT>cd /mnt/target/dev</TT> and do <TT>./MAKEDEV generic</TT>. Do an <TT>ls</TT> to see all the wonderful device files it has made for you. <SECT>Kernel <P> Next we make a kernel. I presume you've done this before, so I'll be brief. It is easier to install lilo if the kernel it is meant to boot is already there. Go back to the target <TT>usr/src</TT> directory, and unpack the linux kernel source there. Enter the linux source tree (<TT>cd linux</TT>) and configure the kernel using your favourite method, for example <TT>make menuconfig</TT>. You can make life slightly easier for yourself by configuring a kernel without modules. If you configure any modules, then you will have to edit the <TT>Makefile</TT>, find <TT>INSTALL_MOD_PATH</TT> and set it to <TT>/mnt/target</TT>. <P> Now you can <TT>make dep</TT>, <TT>make bzImage</TT>, and if you configured modules: <TT>make modules</TT>, <TT>make modules_install</TT>. Copy the kernel <TT>arch/i386/boot/bzImage</TT> and the system map <TT></TT> to the target boot directory <TT>/mnt/target/boot</TT>, and we are ready to install lilo. <SECT>Lilo <P> Lilo comes with a neat script called <TT>QuickInst</TT>. Unpack the lilo source into the target source directory, run this script with the command <TT>ROOT=/mnt/target ./QuickInst</TT>. It will ask you questions about how you want lilo installed. <P> Remember, since we have set <TT>ROOT</TT>, to the target partition, you tell it file names relative to that. So when it asks what kernel you want to boot by default, answer <TT>/boot/bzImage</TT> <EM>not</EM> <TT>/mnt/target/boot/bzImage</TT>. I found a little bug in the script, so it said <VERB> ./QuickInst: /boot/bzImage: no such file </VERB> But if you just ignore it, it's ok. <P> Where should we get <TT>QuickInst</TT> to put the boot sector? When we reboot we want to have the choice of booting into the source system or the target system, or any other systems that are on this box. And we want the instance of lilo that we are building now to load the kernel of our new system. How are we going achieve both of these things? Let's digress a little and look at how lilo boots DOS on a dual boot Linux system. The <TT>lilo.conf</TT> file on such a system probably looks something like this: <P> <VERB> prompt timeout = 50 default = linux image = /boot/bzImage label = linux root = /dev/hda1 read-only other = /dev/hda2 label = dos </VERB> <P> If the machine is set up this way, then the master boot record gets read and loaded by the bios, and it loads the lilo bootloader, which gives a prompt. If you type in <TT>dos</TT> at the prompt, lilo loads the boot sector from hda2, and it loads DOS. <P> What we are going to do is just the same, except that the boot sector in hda2 is going to be another lilo boot sector - the one that <TT>QuickInst</TT> is going to install. So the lilo from the Linux distribution will load the lilo that we have built, and that will load the kernel that we have built. You will see two lilo prompts when you reboot. <P> To cut a long story short, when <TT>QuickInst</TT> asks you where to put the boot sector, tell it the device where your target filesystem is, eg. <TT>/dev/hda2</TT>. <P> Now modify the <TT>lilo.conf</TT> on your source system, so it has a line like <VERB> other = /dev/hda2 label = target </VERB> run lilo, and we should be able to do our first boot into the target system. <SECT>Glibc <P> Next we want to install <TT>init</TT>, but like almost every program that runs under Linux, <TT>init</TT> uses library functions provided by the GNU C library, glibc. So we will install that first. <P> Glibc is a very large and complicated package. It took 90 hours to build on my old 386sx/16 with 8M RAM. But it only took 33 minutes on my Celeron 433 with 64M. I think memory is the main issue here. If you only have 8M of RAM (or, shudder, less!) be prepared for a long build. <P> The glibc install documentation recommends building in a separate directory. This enables you to start again easily, by just blowing that directory away. You might also want to do that to save yourself about 265M of disk space! <P> Unpack the <TT>glibc-2.1.3.tar.gz</TT> (or whatever version) tarball into <TT>/mnt/target/usr/src</TT> as usual. Now, we need to unpack the ``add-ons'' into glibc's directory. So <TT>cd glibc-2.1.3</TT>, and then unpack the <TT>glibc-crypt-2.1.3.tar.gz</TT> and <TT>glibc-linuxthreads-2.1.3.tar.gz</TT> tarballs there. <P> Now we can create the build directory, configure, make and install glibc. These are the commands I used, but read the documentation yourself and make sure you do what is best for your circumstances. Before you do though, you might want to do a <TT>df</TT> command to see how much free space you have. You can do another after you've built and installed glibc, to see what a space-hog it is. <P><VERB> cd .. mkdir glibc-build ../glibc-2.1.3/configure --enable-add-ons --prefix=/usr make make install_root=/mnt/target install </VERB> <P> Notice that we have yet another way of telling a package where to install. <SECT>SysVinit <P> Making and installing the SysVinit binaries is pretty straight forward. There is one minor hack to be done to the <TT>Makefile</TT> in the <TT>src/</TT> subdirectory. In the last four lines, you need to put <TT>$(ROOT)</TT> just before <TT>/dev/initctl</TT>, so that, for example <P><VERB> @ if [ ! -p /dev/initctl ]; then \ </VERB> becomes <P><VERB> @ if [ ! -p $(ROOT)/dev/initctl ]; then \ </VERB> <P> This <TT>initctl</TT> device file is a way of communicating with init. For example, the <TT>init</TT> man page says that this device file should be used instead of the SIGPWR to get init to shut down when the mains power is failing, and we are running on UPS power. The hack we just did ensures that it will go into the target system, not the source one. <P> Once that's done, from the <TT>src</TT> subdirectory, just do: <P><VERB> make ROOT=/mnt/target make install </VERB> <P> There are also a lot of scripts associated with <TT>init</TT>. There are example scripts with the SysVinit package, which work fine. But you have to install them manually. They are set up in a heirarchy under <TT>debian/etc</TT> in the SysVinit source code tree. You can just copy them straight across into the target <TT>etc</TT> directory, with something like <TT>cd ../debian/etc; cp -r * /mnt/target/etc</TT>. Obviously you will want to have a look before you copy them across! <P> Everything is in place now for the target kernel to load up <TT>init</TT> when we reboot. The problem this time should be that the scripts won't run, becasue <TT>bash</TT> isn't there to interpret them. Also, <TT>init</TT> will try to run <TT>getty</TT>'s, but there is no <TT>getty</TT> for it to run. Reboot now and make sure there is nothing else wrong. <SECT>Ncurses <P> The next thing we need is Bash, but bash needs ncurses, so we'll install it first. Ncurses replaces termcap as the way of handling text screens, but it can also provide backwards compatibility by supporting the termcap calls. In the interests of having a clean simple modern system, I think its best to disable the old termcap method. You might strike trouble later on if you are compiling an older application that uses termcap. But at least you will know what is using what. If you need to you can recompile ncurses with termcap support. <P> The commands I used are <P><VERB> ./configure --prefix=/usr --with-install-prefix=/mnt/target --with-shared --disable-termcap make make install </VERB> <SECT>Bash <P> It me took quite a lot of reading and thinking and trial and error to get Bash to install itself where I thought it should go. The configuration options I used are <P><VERB> ./configure --prefix=/mnt/target/usr/local --exec-prefix=/mnt/target --with-curses </VERB> <P> Once you have made and installed Bash, you need to make a symlink like this <TT>cd /mnt/target/bin; ln -s bash sh</TT>. This is because scripts usually have a first line like this <P><VERB> #!/bin/sh </VERB> <P> If you don't have the symlink, your scripts won't be able to run, because they will be looking for <TT>/bin/sh</TT> not <TT>/bin/bash</TT>. <P> You could reboot again at this point if you like. You should notice that the scripts actually run this time, though you still can't login, because there are no <TT>getty</TT> or <TT>login</TT> programs. <SECT>Util-linux (getty and login) <P> The util-linux package contains <TT>agetty</TT> and <TT>login</TT>. We need both of these to be able to log in and get a bash prompt. After it is instlalled, make a symlink from <TT>agetty</TT> to <TT>getty</TT> in the target <TT>/sbin</TT> directory. <TT>getty</TT> is one of the programs that is supposed to be there on all Unix-like systems, so the link is a better idea than hacking <TT>inittab</TT> to run <TT>agetty</TT>. <P> I have one remaining problem with the compilation of util-linux. The package also contains the program <TT>more</TT>, and I have not been able to persuade the <TT>make</TT> process to have <TT>more</TT> link against the ncurses 5 library on the target system rather than the ncurses 4 on the source system. I'll be having a closer look at that. <P> You will also need a <TT>/etc/passwd</TT> file on the target system. This is where the <TT>login</TT> program will check to find out if you are allowed in. Since this is only a toy system at this stage, we can do outrageous things like setting up only the root user, and not requiring any password!! Just put this in the target <TT>/etc/passwd</TT> <P><VERB> root::0:0:root:/root:/bin/bash </VERB> <P> The fields are separated by colons, and from left to right they are user id, password (encrypted), user number, group number, user's name, home directory and default shell. <SECT>Sh-utils <P> The last package we need is GNU sh-utils. The only program we need from here at this stage is <TT>stty</TT>, which is used in <TT>/etc/init.d/rc</TT> which is used to change runlevels, and to enter the initial runlevel. I actually have, and used a package that contains only <TT>stty</TT>, but I can't remember where it came from. Its a better idea to use the GNU package, because there is other stuff in there that you will need if you add to the system to make it useable. <P> Well that's it. You should now have a system that will boot up and prompt you for a login. Type in ``root'', and you should get a shell. You won't be able to do much with it. There isn't even an <TT>ls</TT> command here for you to see your handiwork. Press tab twice so you can see the available commands. This was about the most satisfying thing I found to do with it. <SECT>Towards Useability <P> It might look like we have made a pretty useless system here. But really, there isn't that far to go before it can do some work. One of the first things you would have to do is have the root filesystem mount read-write. There is a script from the SysVinit package, in <TT>/etc/init.d/</TT> which does this, and issues a <TT>mount -a</TT> so that everything gets mounted the way you specify in <TT>/etc/fstab</TT>. Put a symlink called something like <TT>S05mountall</TT> to it in the target's <TT>etc/rc2.d</TT>. <P> You may find that this script will use commands that you haven't installed yet. If so, find the package that contains the commands and install it. See section <REF ID="finding" NAME="Random Tips"> for clues on how to find packages. <P> Look at the other scripts in <TT>/etc/init.d</TT>. Most of them will need to be included in any serious system. Add them in one at a time, make sure everthing is running smoothly before adding more. <P> Check the File Heirarchy Standard (see section <REF ID="FHS" NAME="Filesystem">). It has lists of the commands that should be in <TT>/bin</TT> and <TT>/sbin</TT>. Make sure that you have all these commands installed. Even better, find the Posix documentation that specifies this stuff. <P> From there, it's really just a matter of throwing in more and more packages until everything you want it there. The sooner you can put the build tools such as <TT>gcc</TT> and <TT>make</TT> in the better. Once that is done, you can use the target system to build itself, which is much less complicated. <SECT>More Information <SECT1>Random Tips <LABEL ID="finding"> <P> If you have a command called <TT>thingy</TT> on a Linux system with RPM, and want a clue about where to get the source from, you can use the command: <VERB> rpm -qif `which thingy` </VERB> And if you have a Red Hat source CD, you can install the source code using <VERB> rpm -i /mnt/cdrom/SRPMS/ </VERB> <P> This will put the tarball, and any Red Hat patches into <TT>/usr/src/redhat/SOURCES</TT>. <SECT1>Links <P> <ITEMIZE> <ITEM> There is a mini-howto on building software from source, the <URL URL="" NAME="Software Building mini-HOWTO">. <ITEM> There is also a HOWTO on building a Linux system from scratch. It focuses much more on getting the system built so it can be used, rather than just doing it as a learning exercise. <URL URL="" NAME="The Linux From Scratch HOWTO"> <ITEM> <LABEL ID="FHS"> <URL URL="" NAME="Unix File System Standard"> Another <URL URL="" NAME="link"> to the Unix File System Standard. This describes what should go where in a Unix file system, and why. It also has minimum requirements for the contents of <TT>/bin</TT>, <TT>/sbin</TT> and so on. This is a good reference if your goal is to make a minimal yet complete system. </ITEMIZE> <SECT>Administrivia <SECT1>Copyright <P> This document is copyright (c) 1999, 2000 Greg O'Keefe. You are welcome to use, copy, distribute or modify it, without charge, under the terms of the <URL URL="" NAME="GNU General Public Licence">. Please acknowledge me if you use all or part of this in another document. <SECT1>Homepage <P> The lastest version of this document lives at <URL URL="" NAME="From Powerup To Bash Prompt"> <SECT1>Feedback <P> I would like to hear any comments, criticisms and suggestions for improvement that you have. Please send them to me <URL URL="" NAME="Greg O'Keefe"> <SECT1>Acknowledgements <LABEL ID="acknowledge"> <P> Product names are trademarks of the respective holders, and are hereby considered properly acknowledged. <P> There are some people I want to say thanks to, for helping to make this happen. <P> <DESCRIP> <TAG>Michael Emery</TAG> For reminding me about Unios. <TAG>Tim Little</TAG> For some good clues about <TT>/etc/passwd</TT> <TAG>sPaKr on #linux in efnet</TAG> Who sussed out that syslogd needs <TT>/etc/services</TT>, and introduced me to the phrase ``rolling your own'' to describe building a system from source code. <TAG>Alex Aitkin</TAG> For bringing Vico and his ``verum ipsum factum'' (understanding arises through making) to my attention. <TAG>Dennis Scott</TAG> For correcting my hexidecimal arithmetic. <TAG>jdd</TAG> For pointing out some typos. </DESCRIP> <SECT1>Change History <SECT2>0.8 -> 0.9 <P> <ITEMIZE> <ITEM> Added hack to sysvinit makefile. This info is due to Gerard Beekmans of ``Linux From Scratch'' fame. </ITEMIZE> <SECT2>0.8 <P> <ITEMIZE> <ITEM> Initial version. Separated from "From PowerUp to Bash Prompt". </ITEMIZE> <SECT1>TODO <P> <ITEMIZE> <ITEM> Convert to docbook. </ITEMIZE> </ARTICLE> How Linux Boots article.