Now that I got this decent new dual G5 box, I wanted to install Linux.
This turned out to be an extremely difficult job, as apparently nobody has ever
tried to install Linux on any of the new XServe G5 Series machines, neither
32bit nor 64bit kernels.
There are a number of challenges:
- No internal IDE or SCSI CD-ROM
- Only serial console
- A very new hardware with little Linux support
First I tried a number of ready-built installation ISO images, including the
current sarge Debian-installer image for PPC, and the 32bit and 64bit live
images of Gentoo.
The first thing I had to do is to disable autoboot and enable the serial
console. Luckily, the box actually ships with a manual that instructs you how
to put the OF boot console on the serial port. You have to press the admin (!)
Button at the front of the box a magic number of times.
To permanently make the serial console work, use the following OF commands:
> setenv input-device scca
> setenv output-device scca
Next I had to figure out how to boot from the external firewire cdrom..
apparently this depends on your OF device tree and the GUID of your firewire
device. On my particular box it works with
> devalias cd /ht/pci@5/firewire@e/node@00d04b3c50090210/sbp-2@c000/disk@0
Using Commands like
> dir cd:,\
I was then able to list files on the CD-ROM. To boot the yaboot loader on a
Debian installer cd image, you can use
> boot cd:,\install\yaboot
sbp2:Open ->login?
speed=ffffffff 2 2 load-size=239a4 adler32=a5cf5aa0
Loading ELF
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 Config file read, 2907 bytes
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 \
sbp2:Open ->login?
speed=ffffffff 2 2 Welcome to Debian GNU/Linux sarge!
This is a Debian installation CDROM,
built on 20040729.
The default option is 'install'. For maximum
control, you can use the 'expert' option.
If the system fails to boot at all (the typical
symptom is a white screen which doesn't go away),
use 'install video=ofonly' or 'expert video=ofonly'.
The plain options are for the powerpc family of
processors (from 601 to G4). The *-power3 options
are for IBM Power3 boxes, and the *-power4 options
are for IBM Power4 and Apple G5 boxes. Press the tab
key for a list of options, or type 'help' for help.
************************************
If in doubt, just choose 'install', and if that
doesn't work, try 'install video=ofonly'.
************************************
Welcome to yaboot version 1.3.12
Enter "help" to get some basic usage information
sbp2:Open ->login?
speed=ffffffff 2 2 boot:
I tried all of the provided images, with different options - no success. A
common option to be used because of the serial port is "console=ttyS0,57600".
All I got was:
boot: expert-power4
Please wait, loading kernel...
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 Elf32 kernel loaded...
copying OF device tree...done
starting cpu /cpus/PowerPC,G5...failed: 00000000
Calling quiesce ...
erasing fff06000 of Micron B1 part
flashing fff06000 of Micron B1 part
swapping blocks
DO-QUIESCE finishedreturning 0x01400000 from prom_init
Playing with the Gentoo live cd images didn't bring me any further at all.
I then tried to compile a current 32bit ppc 2.6.8-rc2 kernel by hand (for G5
CPU's). Putting this kernel on the debian installer ISO didn't get me any
further. So apparently either the serial port is not working, or the kernel
crashes somewhere.
Using a cross-compiler running on my dual G4 PowerMac, I compiled the same
2.6.8-rc2 kernel for ppc64 target platform. Putting this on the debian boot cd helped a lot, I now got it as far as:
boot: expert-g5-64 console=ttyS0,57600
Please wait, loading kernel...
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 Elf64 kernel loaded...
Looking for displays
OF stdout is : /ht@0,f2000000/pci@3/mac-io@7/escc@13000/ch-a@13020
Opening displays...
Calling quiesce ...
DO-QUIESCE finishedreturning from prom_init
Found U3 memory controller & host bridge, revision: 53
Mapped at 0xe000000080000000
Found a K2 mac-io controller, rev: 96, mapped at 0xe000000080041000
PowerMac motherboard: XServe G5
Starting Linux PPC64 2.6.8-rc1
-----------------------------------------------------
naca = 0xc000000000004000
naca->pftSize = 0x17
naca->debug_switch = 0x0
naca->interrupt_controller = 0x1
systemcfg = 0xc000000000005000
systemcfg->processorCount = 0x2
systemcfg->physicalMemorySize = 0x20000000
systemcfg->dCacheL1LineSize = 0x80
systemcfg->iCacheL1LineSize = 0x80
htab_data.htab = 0xc00000001f800000
htab_data.num_ptegs = 0x10000
-----------------------------------------------------
[boot]0100 MM Init
[boot]0100 MM Init Done
idle = native_idle
Linux version 2.6.8-rc1 (laforge@dathomir) (gcc version 3.4.1) #4 SMP Sat Jul 31 16:12:42 CEST 2004
[boot]0012 Setup Arch
via-pmu: Server Mode is disabled
PMU driver 2 initialized for Core99, firmware: 0c
nvram: Checking bank 0...
nvram: gen0=204, gen1=205
nvram: Active bank is: 1
Adding PCI host bridge /pci@0,f0000000
Found U3-AGP PCI host bridge. Firmware bus number: 240->255
Adding PCI host bridge /ht@0,f2000000
Can't get bus-range for /ht@0,f2000000, assume bus 0
U3/HT: hole, 0 end at 9fffffff, 1 start at b0000000
Found U3-HT PCI host bridge. Firmware bus number: 0->239
Can't get bus-range for /ht@0,f2000000
PCI Host 0, io start: fffffffffd800000; io end: fffffffffdffffff
PCI Host 1, io start: 0; io end: 3fffff
Top of RAM: 0x20000000, Total RAM: 0x20000000
Memory hole size: 0MB
On node 0 totalpages: 131072
DMA zone: 131072 pages, LIFO batch:16
Normal zone: 0 pages, LIFO batch:1
HighMem zone: 0 pages, LIFO batch:1
[boot]0015 Setup Done
Built 1 zonelists
Kernel command line: ro debconf_priority=low devfs=mount,dall init=/linuxrc console=ttyS0,57600
PowerMac using OpenPIC irq controller at 0x80040000
[boot]0020 OpenPic Init
OpenPIC Version 1.2 (4 CPUs and 120 IRQ sources) at e000000082ccd000
OpenPIC timer frequency is 25.000000 MHz
[boot]0021 OpenPic Timer
[boot]0022 OpenPic IPI
[boot]0023 OpenPic Ext
[boot]0024 OpenPic Spurious
[boot]0025 OpenPic Done
Slave OpenPIC at 0xf8040000 hooked on IRQ 56
[boot]0020 OpenPic U3 Init
OpenPIC (U3) Version 1.2
[boot]0025 OpenPic2 Done
PID hash table entries: 16 (order 4: 256 bytes)
time_init: decrementer frequency = 33.333333 MHz
Console: colour dummy device 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Memory: 498688k available (3840k kernel code, 4120k data, 212k init) [c000000000000000,c000000020000000]
Calibrating delay loop... 66.56 BogoMIPS
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
PowerMac SMP probe found 2 cpus
Processor 1 found.
Synchronizing timebase
Got ack
score 299, offset 1000
score 299, offset 500
score 299, offset 250
score 299, offset 125
score 299, offset 62
score 299, offset 31
score 239, offset 15
score -107, offset 7
score 101, offset 11
score -5, offset 9
score 63, offset 10
score -51, offset 9
Min 9 (score 5), Max 10 (score 87)
Final offset: 9 (61/300)
Brought up 2 CPUs
NET: Registered protocol family 16
PCI: Probing PCI hardware
U3-DART: table not allocated, using direct DMA
PCI: Probing PCI hardware done
PCI: no pci dn found for dev=0001:04:0f.0 Apple Computer Inc. K2 GMAC (Sun GEM)
PCI: no pci dn found for dev=0001:05:0c.1 PCI device 1166:0240 (ServerWorks)
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
nvram_init: Could not find nvram partition for nvram buffered error logging.
rtasd: no RTAS on system
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
devfs: 2004-01-31 Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x1
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Initializing Cryptographic API
pmac_zilog: 0.6 (Benjamin Herrenschmidt )
ttyS0 at MMIO 0x80013020 (irq = 22) is a Z85c30 ESCC - Serial port
ttyS1 at MMIO 0x80013000 (irq = 23) is a Z85c30 ESCC - Serial port
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
sungem.c:v0.98 8/24/03 David S. Miller (davem@redhat.com)
So apparently, there were some issues finding the OpenFirmware dn
(distinguished name) for the Ethernet Chips and the ServerWorks chips. I tried
to put some printk's into the arch/ppc64/pci_dn.c file to see what's going on.
This then led me to the earlier error messages about the U3-DART. After
reading some more code, it appeared like the DART is Apple's IOMMU, and it is
supposed to be needed only when running with >2GB RAM. My box had 512MB, but I tried to force usage of the DART by putting "iommu=force" into the kernel commandline.
Great, this was apparently the problem, since now I got up to the point where
it wanted to mount the root filesystem. I thought I didn't really need an
initrd, since the kernel contained all drivers statically linked in. However, Debian installer seems to be running inside initrd only.
First try was just using one of the pre-supplied initrd.gz images. Yes, they
have the wrong versions of the modules - but I don't want/need those modules
anyway.
Of course this wouldn't work either:
RAMDISK: Compressed image found at block 0
Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)
<0>Rebooting in 180 seconds..
No errror message, nothing. So I thought the problem is with devfs, and I
tried passing several different root parameters ('root=/dev/ram',
'root=/dev/rd/0') without any success.
In the end I found out that the structure sizes of the cramfs superblock
(include/linux/cram_fs_sb.h) are arch-dependent, so I cannot use an initrd that
was built on a ppc32 machine. Unfortunately it is also endian-dependent, and
at this time I only have 32bit big endian and 64bit little endian boxes at
home.
Next step was to use an ext2 initrd, since reasonable filesystems don't have
any strange host/byteorder/wordsize dependencies.
Now it is able to load the initrd, and mount it... although then some other stuff goes terribly wrong. No time yet to investigate this.