Installing Linux on a G5 ClusterNode XServe

Now that I got this decent new dual G5 box, I wanted to install Linux. This turned out to be an extremely difficult job, as apparently nobody has ever tried to install Linux on any of the new XServe G5 Series machines, neither 32bit nor 64bit kernels.

There are a number of challenges:

  • No internal IDE or SCSI CD-ROM
  • Only serial console
  • A very new hardware with little Linux support

First I tried a number of ready-built installation ISO images, including the current sarge Debian-installer image for PPC, and the 32bit and 64bit live images of Gentoo.

The first thing I had to do is to disable autoboot and enable the serial console. Luckily, the box actually ships with a manual that instructs you how to put the OF boot console on the serial port. You have to press the admin (!) Button at the front of the box a magic number of times.

To permanently make the serial console work, use the following OF commands:

> setenv input-device scca
> setenv output-device scca

Next I had to figure out how to boot from the external firewire cdrom.. apparently this depends on your OF device tree and the GUID of your firewire device. On my particular box it works with

> devalias cd /ht/pci@5/firewire@e/node@00d04b3c50090210/sbp-2@c000/disk@0
Using Commands like
> dir cd:,\
I was then able to list files on the CD-ROM. To boot the yaboot loader on a Debian installer cd image, you can use
> boot cd:,\install\yaboot
sbp2:Open ->login?
speed=ffffffff 2 2 load-size=239a4 adler32=a5cf5aa0 

Loading ELF

sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 Config file read, 2907 bytes

sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 \
sbp2:Open ->login?
speed=ffffffff 2 2 Welcome to Debian GNU/Linux sarge!

This is a Debian installation CDROM,
built on 20040729.

The default option is 'install'. For maximum
control, you can use the 'expert' option.

If the system fails to boot at all (the typical
symptom is a white screen which doesn't go away),
use 'install video=ofonly' or 'expert video=ofonly'.

The plain options are for the powerpc family of
processors (from 601 to G4). The *-power3 options
are for IBM Power3 boxes, and the *-power4 options
are for IBM Power4 and Apple G5 boxes. Press the tab
key for a list of options, or type 'help' for help.

If in doubt, just choose 'install', and if that 
doesn't work, try 'install video=ofonly'.
Welcome to yaboot version 1.3.12
Enter "help" to get some basic usage information

sbp2:Open ->login?
speed=ffffffff 2 2 boot: 
I tried all of the provided images, with different options - no success. A common option to be used because of the serial port is "console=ttyS0,57600". All I got was:
boot: expert-power4
Please wait, loading kernel...

sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2    Elf32 kernel loaded...
copying OF device tree...done
starting cpu /cpus/PowerPC,G5...failed: 00000000 
Calling quiesce ...

erasing fff06000  of Micron B1 part
flashing fff06000  of Micron B1 part
swapping blocks
DO-QUIESCE finishedreturning 0x01400000 from prom_init

Playing with the Gentoo live cd images didn't bring me any further at all.

I then tried to compile a current 32bit ppc 2.6.8-rc2 kernel by hand (for G5 CPU's). Putting this kernel on the debian installer ISO didn't get me any further. So apparently either the serial port is not working, or the kernel crashes somewhere.

Using a cross-compiler running on my dual G4 PowerMac, I compiled the same 2.6.8-rc2 kernel for ppc64 target platform. Putting this on the debian boot cd helped a lot, I now got it as far as:

boot: expert-g5-64 console=ttyS0,57600
Please wait, loading kernel...

sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2 
sbp2:Open ->login?
speed=ffffffff 2 2    Elf64 kernel loaded...
Looking for displays
OF stdout is    : /ht@0,f2000000/pci@3/mac-io@7/escc@13000/ch-a@13020
Opening displays...
Calling quiesce ...

DO-QUIESCE finishedreturning from prom_init
Found U3 memory controller & host bridge, revision: 53
Mapped at 0xe000000080000000                          
Found a K2 mac-io controller, rev: 96, mapped at 0xe000000080041000
PowerMac motherboard: XServe G5                                    
Starting Linux PPC64 2.6.8-rc1 
naca                          = 0xc000000000004000   
naca->pftSize                 = 0x17              
naca->debug_switch            = 0x0 
naca->interrupt_controller    = 0x1
systemcfg                     = 0xc000000000005000
systemcfg->processorCount     = 0x2               
systemcfg->physicalMemorySize = 0x20000000
systemcfg->dCacheL1LineSize   = 0x80      
systemcfg->iCacheL1LineSize   = 0x80
htab_data.htab                = 0xc00000001f800000
htab_data.num_ptegs           = 0x10000           
[boot]0100 MM Init                                   
[boot]0100 MM Init Done
idle = native_idle     
Linux version 2.6.8-rc1 (laforge@dathomir) (gcc version 3.4.1) #4 SMP Sat Jul 31 16:12:42 CEST 2004
[boot]0012 Setup Arch
via-pmu: Server Mode is disabled
PMU driver 2 initialized for Core99, firmware: 0c
nvram: Checking bank 0...                        
nvram: gen0=204, gen1=205
nvram: Active bank is: 1 
Adding PCI host bridge /pci@0,f0000000
Found U3-AGP PCI host bridge. Firmware bus number: 240->255
Adding PCI host bridge /ht@0,f2000000                      
Can't get bus-range for /ht@0,f2000000, assume bus 0
U3/HT: hole, 0 end at 9fffffff, 1 start at b0000000 
Found U3-HT PCI host bridge. Firmware bus number: 0->239
Can't get bus-range for /ht@0,f2000000                  
PCI Host 0, io start: fffffffffd800000; io end: fffffffffdffffff
PCI Host 1, io start: 0; io end: 3fffff                         
Top of RAM: 0x20000000, Total RAM: 0x20000000
Memory hole size: 0MB                        
On node 0 totalpages: 131072
  DMA zone: 131072 pages, LIFO batch:16
  Normal zone: 0 pages, LIFO batch:1   
  HighMem zone: 0 pages, LIFO batch:1
[boot]0015 Setup Done                
Built 1 zonelists    
Kernel command line: ro debconf_priority=low devfs=mount,dall init=/linuxrc console=ttyS0,57600
PowerMac using OpenPIC irq controller at 0x80040000
[boot]0020 OpenPic Init                            
OpenPIC Version 1.2 (4 CPUs and 120 IRQ sources) at e000000082ccd000
OpenPIC timer frequency is 25.000000 MHz                            
[boot]0021 OpenPic Timer                
[boot]0022 OpenPic IPI  
[boot]0023 OpenPic Ext
[boot]0024 OpenPic Spurious
[boot]0025 OpenPic Done    
Slave OpenPIC at 0xf8040000 hooked on IRQ 56
[boot]0020 OpenPic U3 Init                  
OpenPIC (U3) Version 1.2  
[boot]0025 OpenPic2 Done
PID hash table entries: 16 (order 4: 256 bytes)
time_init: decrementer frequency = 33.333333 MHz
Console: colour dummy device 80x25              
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)   
Memory: 498688k available (3840k kernel code, 4120k data, 212k init) [c000000000000000,c000000020000000]
Calibrating delay loop... 66.56 BogoMIPS
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
PowerMac SMP probe found 2 cpus                           
Processor 1 found.             
Synchronizing timebase
Got ack               
score 299, offset 1000
score 299, offset 500 
score 299, offset 250
score 299, offset 125
score 299, offset 62 
score 299, offset 31
score 239, offset 15
score -107, offset 7
score 101, offset 11
score -5, offset 9  
score 63, offset 10
score -51, offset 9
Min 9 (score 5), Max 10 (score 87)
Final offset: 9 (61/300)          
Brought up 2 CPUs       
NET: Registered protocol family 16
PCI: Probing PCI hardware         
U3-DART: table not allocated, using direct DMA
PCI: Probing PCI hardware done                
PCI: no pci dn found for dev=0001:04:0f.0 Apple Computer Inc. K2 GMAC (Sun GEM)
PCI: no pci dn found for dev=0001:05:0c.1 PCI device 1166:0240 (ServerWorks)   
SCSI subsystem initialized                                                  
usbcore: registered new driver usbfs
usbcore: registered new driver hub  
nvram_init: Could not find nvram partition for nvram buffered error logging.
rtasd: no RTAS on system                                                    
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
devfs: 2004-01-31 Richard Gooch (   
devfs: boot_options: 0x1                              
Installing knfsd (copyright (C) 1996
Initializing Cryptographic API                          
pmac_zilog: 0.6 (Benjamin Herrenschmidt )
ttyS0 at MMIO 0x80013020 (irq = 22) is a Z85c30 ESCC - Serial port 
ttyS1 at MMIO 0x80013000 (irq = 23) is a Z85c30 ESCC - Serial port
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)                                         
sungem.c:v0.98 8/24/03 David S. Miller (
So apparently, there were some issues finding the OpenFirmware dn (distinguished name) for the Ethernet Chips and the ServerWorks chips. I tried to put some printk's into the arch/ppc64/pci_dn.c file to see what's going on. This then led me to the earlier error messages about the U3-DART. After reading some more code, it appeared like the DART is Apple's IOMMU, and it is supposed to be needed only when running with >2GB RAM. My box had 512MB, but I tried to force usage of the DART by putting "iommu=force" into the kernel commandline.

Great, this was apparently the problem, since now I got up to the point where it wanted to mount the root filesystem. I thought I didn't really need an initrd, since the kernel contained all drivers statically linked in. However, Debian installer seems to be running inside initrd only.

First try was just using one of the pre-supplied initrd.gz images. Yes, they have the wrong versions of the modules - but I don't want/need those modules anyway.

Of course this wouldn't work either:

RAMDISK: Compressed image found at block 0                 
Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)
 <0>Rebooting in 180 seconds..                       
No errror message, nothing. So I thought the problem is with devfs, and I tried passing several different root parameters ('root=/dev/ram', 'root=/dev/rd/0') without any success.

In the end I found out that the structure sizes of the cramfs superblock (include/linux/cram_fs_sb.h) are arch-dependent, so I cannot use an initrd that was built on a ppc32 machine. Unfortunately it is also endian-dependent, and at this time I only have 32bit big endian and 64bit little endian boxes at home.

Next step was to use an ext2 initrd, since reasonable filesystems don't have any strange host/byteorder/wordsize dependencies.

Now it is able to load the initrd, and mount it... although then some other stuff goes terribly wrong. No time yet to investigate this.