Intel e1000 (82546) TX performance

After recent discussions with Robert Olsson at the netfilter workshop, I've decided to investigate a bit further, why the Intel e1000 gigabit MAC's are quite limited when it comes to TX performance and large numbers of pps.

My first assumption was that the in-kernel pktgen.c code might not keep the transmitter busy at all times, resulting in only 760kpps (out of the theoretical maximum of 1480kpps).

So I hacked the e1000 driver to hardcode a refill of the Tx queue with the same skb over and over again. Using a 2048 Tx descriptor ring, I was able to keep the transmitter busy at all times (E1000_ICR_TXQE interrupts).

Unfortunately, I still didn't get more than the 760kpps in this setup (PCI-X, 66MHz, Dual-Opteron 1.4GHz, DDR-333 (PC-2700) RAM. So either we're seeing a limitation of the 82546 chip, or the PCI-X bus / memory latency / whatever.

I'll try the same experiments on a different machine with PCI-X 100 / 133MHz in order to find out what exactly is causing this limit.

netfilter workshop / Linux Kongress 2004

I've not been able to write any articles for this log over the last few days, since I've been busy with the third netfilter developer workshop and Linux-Kongress 2004.

The netfilter workshop went really well, apparently the

Finishing preparations for upcoming netfilter developer workshop

I've spent a significant amount of time over the last couple of days with the final preparations of the upcoming 3rd netfilter developer workshop. This is the first one where I'm in charge of every tiny bit of the organization, and I hope I got everything right.

The first attendees are scheduled to arrive tomorrow. They might even arrive before me, since I'll be heading the 500km down south tomorrow.

Getting the external VGA of my Apple Powerbook (TiBook IV) working

If you've attended one of my presentations during the last 12 months, you will certainly have noticed the poor quality of the slides. Yes, the content and the presentation is poor, too - but I'm mostly referring to the optical quality.

I've already spent at least a whole day in the past in trying to get the external VGA working with Debian/ppc, with little success so far. I really don't care whether the external port mirrors the content of the display, or if it runs in dual head mode.

Today, I spent some three more hours in trail-and-error with the radeon driver of the dri-trunk XFree86. I tried CloneMode, Dual Head, with and without FBMode, and about any other parameter within XF86Config-4.

In the end it turned out that the man page was not up-to-date, and the preferred way to get it running was the so-called MergedFB mode. This wasn't as easy to configure as expected, and I still got lots of 'Signal 11' segfault-style crashes.

The crashes seem to be totally unrelated to my graphics setup. In fact, it crashes when eth0 is not configured yet, but works after the network device is up. Now please somebody step up and explain...

Started a new 2.6.x based mini router distribution

I'm in the process of deploying a couple of PC Engines WRAP.1C embedded x86 boards deployed in my apartment. They make neat little playgrounds for Router/NAT/VPN/WLAN/... style appliances.

Unfortunately I didn't find any embedded Linux distribution project that was up to my demands. Apparently they all use age-old kernels (2.4.17 or something ancient like that). And they very rarely come with a decent automatic build system that would allow you to rebuild it from scratch, adding your own patches, ...

So what did I do? I started my own :(. Not that I'm proud of it, but it was necessary. My home VLAN/firewall/PPPoE/NAT/VPN router is now running the very first image of this new distribution I called 'gRouter'.

It's main features are kernel 2.6.8.1, uClibc-0.9.26, busybox-1.00rc3, pppd with in-kernel PPPoE support, quagga, iptables-1.2.11, openvpn-1.6.0, and dropbear for SSH. It all fits in about 8MB of compact FLASH.

The build process is semi-automatic, apart from a few glitches the whole image compiles itself. I stole some of the build magic from the WISP-DIST project (part of LEAF), although this is all quite simple scripting.

After some more cleanups and testing, I plan to release this distribution. Please don't expect any support, or any configuration tools. It will be available for Linux experts who can configure and setup their system from scratch, and want to have the gadgets of the latest software releases.

On the todo list is cross-compilation support (well, since it is uClibc based, it already does cross-libc-compilation), madwifi support, and especially IPsec using the 2.6.x kernel implementation.

Fujitsu Siemens Corporation not fulfilling amicable agreement

As part of an amicable agreement, Fujitsu Siemens Corporation (FSC) agreed to make a donation to the German Unix Users Group. It came to me as a surprise, that GUUG has not yet received the funds even four months later!

Again, I am very disappointed by the behaviour of the former GPL violators. It should be in their own best interest not to produce any negative publicity.

More Allnet Devices contain Linux

I've now successfully proven that the ALL0185A, ALL0186, ALL1297, ALL2100, ALL2110 and ALL6100 devices contain the Linux kernel and are not distributed according to the GPL.

Considering the out-of-court agreement that I have concluded with them earlier this year in ALL0277, I have to say I'm a bit disappointed that this happened again. It should be in their own best interest to distribute within the GPL license terms, and not first try to infringe and wait until somebody complains.

I've contacted them, and they promised to publish the source code and adhere to the license within a short term. Let's see how this continues.

On VIA's failure to provide adequate Linux support

VIA is definitely one of the most innovative producers of PC-hardware. Their EPIA-series mini-ITX and nano-ITX mainboards are ideal for small appliances, such as firewalls, VPN-gateways, and especially home entertainment platforms such as PVR/DVR applications, DVB-Receivers, DVD/VCD/AVI-players, VideoLan receivers and such.

Just two days ago, VIA made a press release on their new VeXP 3.0 release, a VIA-enhanced fork of xine. To the unfamiliar reader, this press release raises the impression that VIA is really involved with Linux and the Free Software community.

This is just terribly wrong. They do anything but to support GNU/Linux. Comparing this press release with reality, I think VIA's Linux involvement as a whole is nothing more than a PR strategy.

I've recently investigated the "Linux support" they make available for their EPIA platforms. Even from the first glance it was obvious, that VIA just doesn't have any idea on on what it takes to "Support Linux".

All they do is to publish proprietary, pre-compiled kernel frame buffer and XFree86 display drivers for a limited number of particularly old GNU/Linux distributions.

Oh yes, I almost forgot it: They also publish the source to some 'lite' driver which lacks all the functionality needed for hardware-assisted MPEG2 decoding. This is obviously useless, since the whole point of buying a small fan-less board with hardware MPEG acceleration and TV-Out is to use the acceleration.

So their "Linux Support" is so good, that a number of people have to spend days and days in reverse engineering their binary proprietary drivers. You can find more information about the reverse engineering effort. My special thanks are going to Ivor Hewitt for doing all this work.

But wait, wasn't that what the Linux folks usually did with Windows drivers? Welcome to the world of "VIA Linux support", where instead of reverse engineering Windows drivers, we now have to do it with Linux drivers.

If VIA was really interested in providing good GNU/Linux support for their EPIA products, they would

  1. write full source code drivers licensed under appropriate Free Software licenses.
  2. make those drivers use standard interfaces, the respective project's coding style, contain useful comments.
  3. publish those drivers as patches against the latest development version of the respective project (kernel, XFree86, Xine)
  4. Work with the respective project maintainers to integrate those patches
  5. not have to care about maintaining RPMs for each and every distribution
  6. not have to care about porting their drivers to ever-changing API's, since they are included in the respective Free Software projects
  7. Provide documentation for their hardware down to the register level, so the Free Software community can continue development extending to features maybe not yet covered by the current driver.

Related Links:

  • http://lwn.net/Articles/99464/ VIA's original press release
  • http://www.viavpsd.com/ VIA's EPIA homepage
  • http://www.viaarena.com/ VIA's support forum and driver downloads
  • http://www.epiawiki.org/ The comprehensive source of EPIA/Linux related information
  • http://www.ivor.it/cle266/ The reverse engineered driver page
  • Video Documentation on 21C3

    I've attended a meeting on the subject of providing audio/video documentation at the 21st Chaos Communication Congress. During that meeting, I was appointed as being responsible for this part of the 21C3 conference.

    So we want to do on-the-fly encoding of four video signals from DC1394 cameras to DVD-compatible MPEG2, low-resolution MPEG4 for live-streaming, and OGG audio only for live streaming.

    I did some preliminary experiments with the available experimental x86_64 assembly patches for ffmpeg, and it turns out that at least theoretically a 1.6GHz AMD64 should have enough power of doing those three encodings at the same time.

    Unfortunately the dv1394 device at the moment only supports one encoder mmap() ing the ring buffer of incoming 1394 frames - but that should be fixed pretty easy.

    I'll do some more experiments in the next couple of weeks, stay tuned.

    Main netfilter.org server has been replaced

    Yesterday I finally got around moving almost all netfilter.org services from our old Sun Ultra5 to the new XServe ClusterNode.

    Unfortunately there were lots of complications, so I had to stay awake until 5am in order to get all services running again. At least for now, everything seems to run smoothly.

    Off-the-shelf multi-port serial cards and Linux

    This is now the third time I've bought some PCI serial multi-port card (6 to 8 ports) that claimed to have 'Linux support'. If you then read the document, the vendor bluntly tells you that Linux generally doesn't support more than four ports, so if you have two built-in ports, you can only use two more. I've never read such bullshit anywhere else ;)

    So after some minor twiddling, I now submitted a patch adding support for this particular 6port device. Apparently there is either a wide variety of such boards, or almost no Linux users... A couple of years ago I added support for an AFAVLAB 8port serial card, to the Linux serial driver.

    I think I now know way too much about the serial driver. Not stopping with those two PCI 8250 based boards, I did lots of serial driver hacking for the XServe G5 and also for my recent ARM embedded work. Let's hope I can again advance to some more exciting work in the future.

    Two hard drives dying in one week

    This week already the second hard drive in one of my workstations died.. both times it was the same model: IBM DTLA-307060, produced Nov 2000 in Hungary. If that isn't some coincidence. Maybe they have a built-in 'best before' date :(

    So both my main workstations (Dual PIII-733 and a Dual Apple G4-500) were inoperable, isn't that great? The good part is that they've been replaced with silent Samsung SP1213N models, significantly reducing the noise level in my office.

    Using a human-based data acquisition plugin

    Why buy expensive data acquisition boards, if you can have a cheap human being entering the data on some terminal? No, just kidding.

    Anyway, GSPC now has a gpsc_acquire_user.c plugin that retrieves measurement data via a ncurses-based dialog instead of any data acquisition board. This is useful for testing, but also in some real-world cases.

    Attaching an UW-SCSI hard disk to an embedded ARM922T

    No, I'm not doing this for fun, this is part of work. It turned out that nfsroot is a bit of a problem while you're hacking the core network stack (and everything breaks all the time). So I now attached an 18GB UW-SCSI disk to an old aic7xxx controller and plugged this into my ARM development board. Seems to work quite fine, as long as the aic7xxx_old driver is used. The new one apparently calls pci_alloc_consistent from interrupt context ?!?.

    IETF work on NAT behaviour

    Apparently some people within the IETF have started a new working group called 'BEHAVE'. It is about the behaviour of NAT devices on the internet, and their inconsistent and incompatible behaviour. The working group aims to give guidelines to ipmlementors, in order to assure interoperability with new applications such as VoIP and peer-to-peer protocols, as well as multicast and others.

    Certainly a topic that is in in the main focus of my interest, so I decided this is the right point in time to start participation in the IETF.

    For more information about behave, see the mailinglist.

    News on the GPL Violation Front

    It's been some time that I've reported news on the GPL violation side... Thus, no news is good news, one could think. Unfortunately to the contrary, I've been receiving a number of new GPL violation reports, unfortunately none of them containing my copyrighted work - and thus I am now looking for the respective copyright holders in order to get this issue sorted out.

    Stay tuned...

    Performance of system logging

    One of my customers recently had a serious performance issue with one of his installations. Surprisingly, it wasn't even the real applications software itself that had performance issues, but the mechanism used for logging from this application.

    So I started to think about the way logging usually works within a Linux-based system.

    The server applications can be divided within two groups. One of them logs via syslog(), the other logs directly to it's own files. The logging itself happens synchronously, i.e. blocking the normal code flow until the log line was written. In the case of syslog, it might block because the syslog pipe is full - in case of stand-alone files, the file/io might take some time to complete.

    Even in a multi-threaded or forked model of a network server program, this might pose considerable problems with regard to threads waiting for their log i/o to complete.

    Syslog itself might not be as bad, especially since the 2.6.x pipe implementation works with only the minimal necessary amount of copying, and supports larger pipe sizes to avoid writer blocking.

    Some people however tend to use something like syslogger in order to redirect the log output from programs with no syslog support also into syslog. This means that you have one pipe between your application and syslogger, and another pipe between syslogger and your real syslog daemon.

    Comparing this issue with networking is actually not too problematic. In networking, we have packets that are passed from one process to another... with logging it's not a packet but usually one or more lines of text (that is, about 60 to 240 characters per entry).

    You don't want to copy this data around and around... and in a lot of installations you'd rather want to use a couple of log lines than to slow down your application just for some statistics that you might collect.

    Of course, you don't want to modify any of the existing applications, too - they should just be able to use syslog() calls as usual. OF course you could load a LD_LIBRARY_PRELOAD lib and redirect the syslog() calls, if needed.

    So what I came up with, is something like a partially mmap()able pipe. The logging process would log to that pipe like it would with any other file descriptor. Internally, that 'pipe' has a ring buffer of configurable size. The pipe-reader could now mmap() this ring buffer into his address space in order to read the log.

    This scheme should have the advantage of not blocking the writer if the pipe is full (it would just wrap around the ring buffer), and it avoids copying the data from some in-kernel pipe buffer into the user-space of the pipe reader.

    Did you notice, this now looks perfectly like the DMA ring buffer of your Ethernet device and the Linux softirq handler ;)

    Anyway, as I didn't do any vm / vfs hacking in Linux so far, this is not a trivial thing to implement. And I have lots of other work at this point. However, I'd certainly like to investigate the possible performance gains [losses?] of this idea. Comments welcome.

    Upcoming Chaosradio episode on software patents

    The next Chaosradio radio show will be about the ongoing debade on software patents, especially the recent development within the European Union.

    Being part of the anti software patent movement for about 4-5 years now, I am more than happy to help with the radio show on this subject.

    The radio show will be on air on Sept 01, 10pm GMT+2. If you understand german, there's a MP3 live stream available on the homepage.

    Working on embedded Linux ARM SoC project

    While there hasn't been any update on this weblog for quite some time, I've been buried under a lot of work.

    One of the most interesting projects is an embedded ARM-based SoC project with special network acceleration hardware. Unfortunately I'm not allowed to talk too much about it at this point, but be assured it is very exciting, and of course runs Linux :)

    During development I found it quite comfortable to run the small embedded system with nfsroot mounted from some larger box. The nfsroot contains a debootstrap'ed installation of Debian sarge for ARM.

    The main problem for this kind of operation is the limited on-board memory. But I'm tempted to put a 64MB graphics card into one of the PCI slots and hack the Linux kernel to treat this framebuffer as (somewhat slow) RAM :)

    Booting from a md raid device on powerpc

    Apparently, nobody has ever tried to do this so far, since the mac partition handling code in the Linux kernel had no provisions for enabling auto-detection of md software raid.

    I've now written patch for Linux 2.6.8, available at http://gnumonks.org/ftp/pub/patches/linux-2.6.8-mac-autoraid.patch implementing this feature. All you need to do is apply that patch, and make sure your md partitions have the type 'Linux_raid_autodetect' in the mac partition table.

    Figured out the fan control on the XServe ClusterNode

    I spent the last couple of hours figuring out the missing bits of the fan/thermal control on Apples Dual XServe ClusterNode. Luckily it's very similar to the design Apple used in their Desktop G5 machines, so I can build on the work that Benjamin Herrenschmidt did with his thermal_pm72 driver.

    So in case anybody is interested in the technical details: Eight fans are controlled by the FCU (Fan Control Unit), which is attached to a i2c bus of the Apple U3 northbridge.

    There are three RPM controlled fans per CPU. The Left CPU (viewing from the front of the machine) has fans #1,2,3. The right CPU: #4,5,6.

    The other two fans are not RPM controlled, but just PWM controlled... so instead of setting an RPM, you have to set a pulse-width between 10 and 100%. PWM Fan #1 is located between RPM-fan 3 and 4 (between both CPU's) and it's job is to keep the U3 chip cool. PWM Fan #2 is located behind the PCI-X slots and thus cooling them (too bad in my machine there is no card to be cooled *g*).

    Regulating the CPU fans is quite easy, since there is a per-CPU temperature sensor, and also a voltage and current reading, so we can calculate the power consumption of each CPU and tune the fans accordingly.

    For the U3 it is a bit more difficult.. I have not yet found a way to get a temperature reading for it, but I'm quite sure there is some temperature sensor somewhere.

    As for PCI cards, there is apparently some way to read the power consumption - but of course again undocumented and not reverse engineered yet. As I don't have PCI boards in my box anyway, I personally don't care that much. But I should now stop arguing rationally, since a machine hosted in some rack-space is very unlikely to need fan control at all :)

    I'll try to make a somewhat cleaner unified driver for PowerMac7,2 and RackMac3,1 and post a patch in the next couple of days.

    I really wonder why Apple is not releasing their FCU driver source code for Darwin... it's really annoying. And I doubt they can claim that it contains any valuable intellectual property that their competitors are not allowed to see ;)

    Database Design + Content for GPL-Violations

    In order to keep track about the gpl violations that I am encountering myself or that are reported by fellow users, I really need some semi-automatic system to keep track of this.

    Being a RDBMS geek in my former life, I designed a SQL-based data model to cope with the individual objects such as vendors, products, product-firmware-versions, violations, settlements, compensations, comments, documents, contracts, ...

    It all turned out to be more complex than I thought initially. But I think it was really worth the effort.

    This database is for strictly internal use, since there is a lot of confidential information in there. However, according flags indicating the public/private nature of the data records are included in the data model. At some later point I might extract the public information to create some web pages at www.gpl-violations.org.

    It's main target is to allow me keep track with what's going on, and also keep track about what has been verified where, if for new upcoming firmware images the source code was made available, if the source was complete, ...

    I've already filled in lots of the existing data I have, but it's far from being complete. This needs some more time of filling in data records.

    And yes, I built some simple forms using GNU Enterprise Designer and Forms. It's still in 0.x stage, but usable for easy tasks.

    Finally the XServe ClusterNode runs Linux!

    Yes, it does. I now have two partitions: One running the experimental Gentoo ppc64 port, and another one running the overly-conservative Debian woody ppc32. The plan is to boot into Gentoo, and run publicly-accessible production services within the Debian woody chroot.

    So how did I make it? Well, I gave up on the idea that the usual installation process of any distribution would work. So instead of trying to fix up whatever goes wrong in the installation scripts, I just escaped to a shell ASAP, run mac-fdisk, mkfs.ext3, extracted the stage3.tar.gz and did the rest of the Gentoo install.

    Debian was then installed using the convenient debootstrap tool.

    One of the major remaining questions is however: Does the Apple XServe Hardware give you anything similar to Sun boxes, where you could just send break over the serial line and get into OpenFirmware? This is very convenient for remotely resetting machines without any local 'reset-staff' present.

    After some chatting with Benjamin Herrenschmidt, apparently nobody is working on getting fan rpm/speed/temperature control implemented on the XServe so far. Well, as it's a rack-mounted machine sitting in some hosting center I don't really care about the noise anyway.

    More interestingly, the Apple KeyLargo2 based machines have a Hardware Watchdog. Driver Source code is available within the public part of the Darwin kernel, so it should be easy to implement a Linux driver for this. Maybe I'll find some time to dive into this.

    IPv6 packet filter benchmarking

    It seems like a German university is currently doing feature analysis and benchmarking of IPv6 packet filters. Coincidentally, I'm going to near that university next week anyway, so I'll stop over for a short visit and help them with their ip6tables evaluation setup.

    I would be very interested to see some numbers on ip6tables... as we just discovered at the networking conference in Portland, nobody seems to be doing benchmarking / profiling on the Linux IPv6 code so far.

    Installing Linux on a G5 ClusterNode XServe

    Now that I got this decent new dual G5 box, I wanted to install Linux. This turned out to be an extremely difficult job, as apparently nobody has ever tried to install Linux on any of the new XServe G5 Series machines, neither 32bit nor 64bit kernels.

    There are a number of challenges:

    • No internal IDE or SCSI CD-ROM
    • Only serial console
    • A very new hardware with little Linux support

    First I tried a number of ready-built installation ISO images, including the current sarge Debian-installer image for PPC, and the 32bit and 64bit live images of Gentoo.

    The first thing I had to do is to disable autoboot and enable the serial console. Luckily, the box actually ships with a manual that instructs you how to put the OF boot console on the serial port. You have to press the admin (!) Button at the front of the box a magic number of times.

    To permanently make the serial console work, use the following OF commands:

    > setenv input-device scca
    > setenv output-device scca
    

    Next I had to figure out how to boot from the external firewire cdrom.. apparently this depends on your OF device tree and the GUID of your firewire device. On my particular box it works with

    > devalias cd /ht/pci@5/firewire@e/node@00d04b3c50090210/sbp-2@c000/disk@0
    
    Using Commands like
    > dir cd:,\
    
    I was then able to list files on the CD-ROM. To boot the yaboot loader on a Debian installer cd image, you can use
    > boot cd:,\install\yaboot
    sbp2:Open ->login?
    speed=ffffffff 2 2 load-size=239a4 adler32=a5cf5aa0 
    
    Loading ELF
    
    
    
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 Config file read, 2907 bytes
    
    
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 \
    sbp2:Open ->login?
    speed=ffffffff 2 2 Welcome to Debian GNU/Linux sarge!
    
    This is a Debian installation CDROM,
    built on 20040729.
    
    The default option is 'install'. For maximum
    control, you can use the 'expert' option.
    
    If the system fails to boot at all (the typical
    symptom is a white screen which doesn't go away),
    use 'install video=ofonly' or 'expert video=ofonly'.
    
    The plain options are for the powerpc family of
    processors (from 601 to G4). The *-power3 options
    are for IBM Power3 boxes, and the *-power4 options
    are for IBM Power4 and Apple G5 boxes. Press the tab
    key for a list of options, or type 'help' for help.
    
    ************************************
    If in doubt, just choose 'install', and if that 
    doesn't work, try 'install video=ofonly'.
    ************************************
    Welcome to yaboot version 1.3.12
    Enter "help" to get some basic usage information
    
    sbp2:Open ->login?
    speed=ffffffff 2 2 boot: 
    
    I tried all of the provided images, with different options - no success. A common option to be used because of the serial port is "console=ttyS0,57600". All I got was:
    boot: expert-power4
    Please wait, loading kernel...
    
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2    Elf32 kernel loaded...
    copying OF device tree...done
    starting cpu /cpus/PowerPC,G5...failed: 00000000 
    Calling quiesce ...
    
    erasing fff06000  of Micron B1 part
    flashing fff06000  of Micron B1 part
    swapping blocks
    DO-QUIESCE finishedreturning 0x01400000 from prom_init
    

    Playing with the Gentoo live cd images didn't bring me any further at all.

    I then tried to compile a current 32bit ppc 2.6.8-rc2 kernel by hand (for G5 CPU's). Putting this kernel on the debian installer ISO didn't get me any further. So apparently either the serial port is not working, or the kernel crashes somewhere.

    Using a cross-compiler running on my dual G4 PowerMac, I compiled the same 2.6.8-rc2 kernel for ppc64 target platform. Putting this on the debian boot cd helped a lot, I now got it as far as:

    boot: expert-g5-64 console=ttyS0,57600
    Please wait, loading kernel...
    
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2 
    sbp2:Open ->login?
    speed=ffffffff 2 2    Elf64 kernel loaded...
    Looking for displays
    OF stdout is    : /ht@0,f2000000/pci@3/mac-io@7/escc@13000/ch-a@13020
    Opening displays...
    Calling quiesce ...
    
    DO-QUIESCE finishedreturning from prom_init
    Found U3 memory controller & host bridge, revision: 53
    Mapped at 0xe000000080000000                          
    Found a K2 mac-io controller, rev: 96, mapped at 0xe000000080041000
    PowerMac motherboard: XServe G5                                    
    Starting Linux PPC64 2.6.8-rc1 
    -----------------------------------------------------
    naca                          = 0xc000000000004000   
    naca->pftSize                 = 0x17              
    naca->debug_switch            = 0x0 
    naca->interrupt_controller    = 0x1
    systemcfg                     = 0xc000000000005000
    systemcfg->processorCount     = 0x2               
    systemcfg->physicalMemorySize = 0x20000000
    systemcfg->dCacheL1LineSize   = 0x80      
    systemcfg->iCacheL1LineSize   = 0x80
    htab_data.htab                = 0xc00000001f800000
    htab_data.num_ptegs           = 0x10000           
    -----------------------------------------------------
    [boot]0100 MM Init                                   
    [boot]0100 MM Init Done
    idle = native_idle     
    Linux version 2.6.8-rc1 (laforge@dathomir) (gcc version 3.4.1) #4 SMP Sat Jul 31 16:12:42 CEST 2004
    [boot]0012 Setup Arch
    via-pmu: Server Mode is disabled
    PMU driver 2 initialized for Core99, firmware: 0c
    nvram: Checking bank 0...                        
    nvram: gen0=204, gen1=205
    nvram: Active bank is: 1 
    Adding PCI host bridge /pci@0,f0000000
    Found U3-AGP PCI host bridge. Firmware bus number: 240->255
    Adding PCI host bridge /ht@0,f2000000                      
    Can't get bus-range for /ht@0,f2000000, assume bus 0
    U3/HT: hole, 0 end at 9fffffff, 1 start at b0000000 
    Found U3-HT PCI host bridge. Firmware bus number: 0->239
    Can't get bus-range for /ht@0,f2000000                  
    PCI Host 0, io start: fffffffffd800000; io end: fffffffffdffffff
    PCI Host 1, io start: 0; io end: 3fffff                         
    Top of RAM: 0x20000000, Total RAM: 0x20000000
    Memory hole size: 0MB                        
    On node 0 totalpages: 131072
      DMA zone: 131072 pages, LIFO batch:16
      Normal zone: 0 pages, LIFO batch:1   
      HighMem zone: 0 pages, LIFO batch:1
    [boot]0015 Setup Done                
    Built 1 zonelists    
    Kernel command line: ro debconf_priority=low devfs=mount,dall init=/linuxrc console=ttyS0,57600
    PowerMac using OpenPIC irq controller at 0x80040000
    [boot]0020 OpenPic Init                            
    OpenPIC Version 1.2 (4 CPUs and 120 IRQ sources) at e000000082ccd000
    OpenPIC timer frequency is 25.000000 MHz                            
    [boot]0021 OpenPic Timer                
    [boot]0022 OpenPic IPI  
    [boot]0023 OpenPic Ext
    [boot]0024 OpenPic Spurious
    [boot]0025 OpenPic Done    
    Slave OpenPIC at 0xf8040000 hooked on IRQ 56
    [boot]0020 OpenPic U3 Init                  
    OpenPIC (U3) Version 1.2  
    [boot]0025 OpenPic2 Done
    PID hash table entries: 16 (order 4: 256 bytes)
    time_init: decrementer frequency = 33.333333 MHz
    Console: colour dummy device 80x25              
    Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
    Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)   
    Memory: 498688k available (3840k kernel code, 4120k data, 212k init) [c000000000000000,c000000020000000]
    Calibrating delay loop... 66.56 BogoMIPS
    Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
    PowerMac SMP probe found 2 cpus                           
    Processor 1 found.             
    Synchronizing timebase
    Got ack               
    score 299, offset 1000
    score 299, offset 500 
    score 299, offset 250
    score 299, offset 125
    score 299, offset 62 
    score 299, offset 31
    score 239, offset 15
    score -107, offset 7
    score 101, offset 11
    score -5, offset 9  
    score 63, offset 10
    score -51, offset 9
    Min 9 (score 5), Max 10 (score 87)
    Final offset: 9 (61/300)          
    Brought up 2 CPUs       
    NET: Registered protocol family 16
    PCI: Probing PCI hardware         
    U3-DART: table not allocated, using direct DMA
    PCI: Probing PCI hardware done                
    PCI: no pci dn found for dev=0001:04:0f.0 Apple Computer Inc. K2 GMAC (Sun GEM)
    PCI: no pci dn found for dev=0001:05:0c.1 PCI device 1166:0240 (ServerWorks)   
    SCSI subsystem initialized                                                  
    usbcore: registered new driver usbfs
    usbcore: registered new driver hub  
    nvram_init: Could not find nvram partition for nvram buffered error logging.
    rtasd: no RTAS on system                                                    
    VFS: Disk quotas dquot_6.5.1
    Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
    devfs: 2004-01-31 Richard Gooch (rgooch@atnf.csiro.au)   
    devfs: boot_options: 0x1                              
    Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
    Initializing Cryptographic API                          
    pmac_zilog: 0.6 (Benjamin Herrenschmidt )
    ttyS0 at MMIO 0x80013020 (irq = 22) is a Z85c30 ESCC - Serial port 
    ttyS1 at MMIO 0x80013000 (irq = 23) is a Z85c30 ESCC - Serial port
    RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
    loop: loaded (max 8 devices)                                         
    sungem.c:v0.98 8/24/03 David S. Miller (davem@redhat.com)
    
    So apparently, there were some issues finding the OpenFirmware dn (distinguished name) for the Ethernet Chips and the ServerWorks chips. I tried to put some printk's into the arch/ppc64/pci_dn.c file to see what's going on. This then led me to the earlier error messages about the U3-DART. After reading some more code, it appeared like the DART is Apple's IOMMU, and it is supposed to be needed only when running with >2GB RAM. My box had 512MB, but I tried to force usage of the DART by putting "iommu=force" into the kernel commandline.

    Great, this was apparently the problem, since now I got up to the point where it wanted to mount the root filesystem. I thought I didn't really need an initrd, since the kernel contained all drivers statically linked in. However, Debian installer seems to be running inside initrd only.

    First try was just using one of the pre-supplied initrd.gz images. Yes, they have the wrong versions of the modules - but I don't want/need those modules anyway.

    Of course this wouldn't work either:

    RAMDISK: Compressed image found at block 0                 
    Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)
     <0>Rebooting in 180 seconds..                       
    
    No errror message, nothing. So I thought the problem is with devfs, and I tried passing several different root parameters ('root=/dev/ram', 'root=/dev/rd/0') without any success.

    In the end I found out that the structure sizes of the cramfs superblock (include/linux/cram_fs_sb.h) are arch-dependent, so I cannot use an initrd that was built on a ppc32 machine. Unfortunately it is also endian-dependent, and at this time I only have 32bit big endian and 64bit little endian boxes at home.

    Next step was to use an ext2 initrd, since reasonable filesystems don't have any strange host/byteorder/wordsize dependencies.

    Now it is able to load the initrd, and mount it... although then some other stuff goes terribly wrong. No time yet to investigate this.