Number of GPL violations still rising
Over the last couple of days I've again verified a number of GPL violations.
It's a real pity that those companies still don't get the message.o
It hurts especially, that there are two cases (Netgear, Siemens) where
companies with whom we already had a amicable agreement published new devices
that again don't comply with the GPL (Netgear WGT634U and Siemens M740-AV). Apparently they don't really care despite the fact they should know better.
Also, we have another number of cases where companies signed an agreement with
us, but failed to fulfill that agreement only a couple of months later with
exactly the devices mentioned in the agreement.
I'm sick of those cases. What the hell is so difficult to put the source code
and the GPL license text on a CD-ROM that has 500MB unused and ships with the
device anyway?
[ /linux/gpl-violations |
permanent link ]
Preparing the 21st Chaos Communication Congress
As every year, the Chaos
Communication Congress takes place in Berlin, Germany.
For six years, I'm part of the team that takes care of audio and video
recording and streaming. Since this year I've become head of the a/v
documentation project, I decided to use a 100% Linux based solution instead of
the Apple Quicktime stuff that we've had for the last couple of years.
Thanks to the great ffmpeg software, we can
even encode four different streams on a off-the-shelf Pentium IV.
Today, I've been with the technicians at the congress center who set up the PA
and lighting. This was to make sure everything really reflects our demands,
and we have the correct audio signal delivered to the appropriate place, etc.
Setup of the congress will continue over the holidays. Especially the NOC
(Network Operations Centre) will have a hard time setting up the internal network for about 3000 attendees, certainly each bringing more than one networked device on average.
[ /linux/conferences |
permanent link ]
ffmpeg is undocumented, ffserver broken
I've been experimenting a lot with ffmpeg and ffserver over the last couple of
days. The fact that ffmpeg is very little documented is a pity, but not
exactly a problem for someone experienced with free software and C development
(use the source, Luke).
However, the ffserver program seems to be horribly broken in a number of ways.
Independent of the kind of configuration, it regularly segfaults, glibc
complains about double-free's, and valgrind or Electric Fence have numerous complaints.
All information you can find after browsing through mail archives, is that it's
apparently broken for a number of years. Maybe I'll spend some time at it and
fix it at least partially. So I spent about two days to familiarize myself
with the source of libavformat, libavcodec, ffmpeg and ffserver. It's not
exactly easy to understand, but I think I now got a good understanding of
what's going on where.
Another fundamental insufficiency of ffmpeg seems to be that it cannot put the
output of one codec into multiple output files. So let's say I want to encode
some MPEG2 video and AC3 audio. This is to be written to a .vob file and at
the same time sent as a transport stream over the network. The only way you
can achieve this now is to encode the input data twice - which I cannot afford due to CPU limitation.
So I was pondering something like streaming the output over multicast RTP plus
running something like rtpdump on the same machine to create the local file.
As a summary, I think it's a pity that there is good encoding software like
ffmpeg, and that nobody volunteered yet to fix the remaining issues required to
turn it into a good streaming and recording solution.
[ /linux |
permanent link ]
More and more cases
Today has been a sad day with regard to gpl-violations.org. I just ordered
five potentially infringing devices from three different vendors. Apparently
the message has not been conveyed to all respective parties yet...
So let's see how they will react if someone actually is in a position to ban
their products from all-important pre-christmas sale.
This really sucks. At some point I want to start coding on a day without having
to have information in my inbox about yet another gpl violation case.
[ /linux/gpl-violations |
permanent link ]
linux-bangalore 2003
I've just returned from lb/2003, the
major linux conference in India. I've had a great time there.
Besides giving two presentations (one about SMP
effects in kernel programming and another about the netfilter internals), I've
done some travelling to Mysore and Mumbai.
Thanks again to the lb/2003 organizers. They did a great job comforting the
speakers in any possible way.
[ /linux/conferences |
permanent link ]
Shopping in Bangalore
Today I went shopping in Bangalore. The first thing I had to learn, is that
you need a lot of travel through the heavy traffic in order to get to the
respective stores.
Secondly, buying/finding a Sari (Including the blouse and the Petticoat) is not
as easy as buying women's clothes in the western world. The choli (blouse) is
made-to-measure, and they require more information than the usual under bust /
over bust / waist measurements. So I only bought one this time, let's first see
how it fits Elisabeth before I buy more items that in the end don't fit.
Getting Hindi learner books (apart from the usual Devanagari alphabet training)
in Bangalore turned out to be more difficult than expected. Students tend to
get the books from the Schools, and the local language is Kannada. But finally
we managed to get them, too.
Finding Bollywood DVD's is obviously the most easy task ;) I got a stack of 8,
and I'll probably be buying more of them once I get to Mumbai on Tuesday.
[ /personal |
permanent link ]
Linux Bangalore is Over
The three-day lb2004 is now
over. About 80 presentations from all areas of free software, ranging from
hardcore technical subjects to user-experience.
One of the interesting parts was that one developer managed to port the "DotGNU
Portable.net" framework to the Simputer in only three days during the
conference. Apparently this spawned a lot of media interest.
In the end, the conference went really fine, if it wasn't for the strange rules
and regulations of the IISC that tried to undermine the event.
Oh yes, than there is the air condition to which I probably owe catching a cold last year - and this year again :(
[ /linux/conferences |
permanent link ]
Day one of Linux Bangalore 2004
So today lb2004 started, but unfortunately there are lots of problems, some of
them really outstandingly ridiculous.
The less problematic issue was that even though the zd1201 driver now works,
the access points would not actually get a link to a switch, independent of the kind of cable. So the whole wireless network idea was basically abandoned.
As for Internet access at the conference, there was none. There's not even
CDMA reception on top of the roof, and even though that the auditorium is part
of the Indian Institute of Sciences there is no connection to the IISC LAN
within the complex. Also, they IISC apparently has so little bandwidth, that
it's insufficient for their own purpose, let aside connecting some conference.
Then the really interesting thing came up: Because of about 2800 attendees,
there was an 500 seat additional auditorium built. Apparently the IISC gave
permission to build the auditorium tent on their ground, even charged money for
using the ground - but they informed the lb2004 organizers that they were not
allowed to use it. They've only given permission to build the auditorium, not
to actually use it to give any presentations in there, or even use it only as a
lounge.
Believe it or not, it became worse. Someone wanted to fetch food from the
catering to the speaker lounge. He was stopped by a security guard, stating
that in the room officially designated as speaker lounge by the IISC, there was
no food permitted, and a fine would apply if anyone actually tried to do so.
Oh yes, and they suddenly introduced a new rule, active on 1st of December,
that as soon as there are more than 25 cars parked on the grounds, another fine
would apply.
This is just incredibly ridiculous. This is the Indian Institute of Science,
and the conference is held in exactly the same premises for the third time.
None of those issues came up in the previous years.
Also, this is the same IISC which boasts himself to have denied an event with
Dr. Kalam (India's president and one of the biggest promoters of Free and Open
Source in India) at the J.N.Tata Auditorium.
It's very hard to understand that they just want to sabotage that kind of event
in any possible way. It makes me feel sick and sad. Somebody should organize
a demonstration. Call off half a day and make a 3000 attendee protest in front
of the office of the director of the IISC.
[ /linux/conferences |
permanent link ]
Making a broken ZyDAS zd1201 based USB Wireless work
It's amazing what kind of strange and broken USB devices there are. Here at
Linux Bangalore, they've got a bunch of 'combo USB WLAN and Flash Disk Sticks'
that turned out to be TwinMOS B241
devices. But let's forget about this for a moment and join me on my journey...
They ship with a Linux driver preinstalled onto the flash disk. Unfortunately
that driver consists of some hacked wlan-ng driver. For most people who've
worked with wlan-ng, they know that it's overly complex, and not really the standard Linux way of doing things.
That modified wlan-ng source code would only build for 2.4.x, the machines here are running Fedora Core 3.
Also, the machines would totally lock up their USB stack as soon as you would enable the WLAN part, even without any driver.
Since the wlan-ng was a modified prism2 USB driver, I though I could somehow merge the changes into the orinoco_usb driver that is in the standard kernel.
After some deeper look, it turned out that the device has no relation with
Intersil, and definitely doesn't have a Prism2 chip on the PCB, so my tries to get this working were useless.
Apparently, they didn't even do 'copy+paste', but they did 'edit and forget',
i.e. forget about prism2 devices and only support some totally different
chipset without actually changing file names or comments in the driver.
So I opened one of the devices and found a AU9254A21-CBS (4 port USB hub), a
K9F1G08UOM (the Flash memory for the USB drive), a IC1114-F48LQ (usb storage
controller for the flash), and some unknown chip labelled ZyDAS ARM. Also
there was a Cyprus semiconductor chip that I though of the EZ-USB controller
that connects the alleged prism2 to the USB bus. This fits the driver design,
since it has to download some 'bootup code' to the usb device before being able
to use it.
After some further analysis, the Cyprus CY62137CV30LL-70BVI turned out to be
some SRAM chip, and the ZyDAS ARM the real 802.11 MAC. And luckily, some
people are working on a very clean 2.6 style stand-alone driver.
And the driver even worked after just adding the USB device ID to it's list of known devices, at least on little endian platforms.
If the devices specs or documentation would have told us that it is a ZD1201,
the driver clearly indicated that it has no relation with prism2 or somebody
who wrote the driver actually had a clue how to do this, this would have saved me about four hours of time, at least.
Oh yes, and the usb stack lockup comes from violating the USB specification and
only supporting one particular flavour of USB bus enumeration. So nobody
actually ever tested it for USB spec compliance, even though there are
compliance tests available by the USB forum. *sigh*
[ /linux |
permanent link ]
Visiting Infosys
Today, the international speakers of LB/2004 were invited to visit the
sponsor Infosys, apparently India's
largest IT outsourcing company.
They've been growing from 7 to 35,000 engineers very rapidly, and their
Bangalore campus is certainly the most luxurious and westernized part of India
I've seen so far (not that I've seen much of India either).
Anyway, we were informed about their recent Linux and FOSS related activities,
met their internal InfyLUG (Infosys Linux User Group), met one of the seven
founders and Andi Kleen gave a lecture about the kernel development process,
that was attended by 300 employees and streamed to all the other Infosys
campuses.
[ /linux/conferences |
permanent link ]
Leaving for Linux-Bangalore/2004
I'm at the moment packing my suitcase, and I'll be sitting in the plane about
24 hours from now. Do not expect any fast email replies or IRC presence of me
before December 9th.
[ /linux/conferences |
permanent link ]
Make CyberJack drivers issue a key-press confirmation beep
This is a very useful feature, especially for blind people. Unfortunately
there is no unique way of issuing some beep sound on Linux-based systems, so
there needs to be some magic that determines whether running under X11 or not
and call the appropriate code for beeping.
[ /linux/cyberjack |
permanent link ]
Successful TomTom Visit
As indicated before, TomTom B.V. has invited Christian and me to visit them at
their offices. Apart from some consulting/training regarding Free Software
Licenses and the Free Software Community, they were particularly interested in
getting us involved with their Linux kernel related development.
I stressed the fact that it is very important to clean up all the drivers, make
them use standard interfaces and eventually get them merged to the mainline
kernel. As it seems, they agree and want to contract one or some of the
OpenTom developers to do so.
[ /linux/opentom |
permanent link ]
KNF Kongress: Meeting old friends
Today I've given my two gpl related presentations at the annual KNF Kongress.
Apparently it helped some people to understand legal requirements of dealing with various free software licenses, which is good.
Also, I was a the OpenTom presentation and could it actually see working with a
2.6.x kernel, sound, framebuffer, USB keyboard, USB CD-ROM and even playing
some low-res-movies with mplayer on the console. Great work.
Apart from that, I was just chatting with a lot of people. As a side-note, I've
also mentioned the CCCB's current search for a pc-based logic analyzer that
either comes with developer documentation or Linux software. People suggested
of building the logic analyzer on our own, by using available FPGA's, some SRAM
and a USB interface. If you think about it, this actually sounds quite
feasible. Now I'll do some research on FPGA's that ship with a free
development environment, unlike the proprietary stuff shipped by Altera & Co :(
[ /linux/conferences |
permanent link ]
Never ride trains on weekends
If I'm ever about to travel by train on a weekend, please somebody remind me
not to do so. All these crowds trying to find available seats, incredibly
busy, delayed trains, ...
Travelling during the week is just so much more convenient.
[ /personal |
permanent link ]
Two presentations at KNF Kongress coming up
I'll be giving two presentations at the upcoming KNF Kongress
2004, entitled "The GNU GPL Revisited" and "Copyright helps Copyleft".
Also, Christian Daniel from the OpenTom group is going to present on his re-engineering efforts.
If you happen to live in southern Germany, it's probably a good idea to check
out the yet small but great KNF Kongress. Looking forward to meeting you there.
[ /linux/conferences |
permanent link ]
I'm pleased to present at Linux Bangalore 2004
Following up my presence last year's Linux Bangalore 2003, I'm very pleased to again be invited to present at this year's incarnation.
Unfortunately I had to shift the main focus of my presentations a bit towards
political/legal issues, so there's one presentation about How to interact with
the Free Software Community, one about The GPL is not public domain, and for all the tech savvy guys, there's A tour through the Linux 2.6 network stack.
I'm happy to present on those political and legal issues, because I think this
is the opportunity to get this kind of knowledge into the Indian IT outsourcing
industry, before it is too late (like apparently happened with most of the
Taiwanese embedded Linux vendors).
I'm happy to see an increasing number of high profile speakers at Linux
Bangalore, and it's now becoming (to the best of my knowledge) a big
internationally recognized Linux event.
[ /linux/conferences |
permanent link ]
More work on the REINER SCT CyberJack drivers
I'm not sure if I did mention it on this blog, but I've been contracted by REINER SCT to work on a Linux driver for their CyberJack series of smart card readers for quite some time.
In the last days I've been spending quite an amount of time hunting down
user-reported bugs in the driver, which is good. Sometimes it's really
surprising to see in what kind of bugs stupid mistakes eventually result.
Also, I've now managed to make the driver work on x86_64, so it's working in
little-endian 32 and 64bit, big endian 32bit. I have to test it on my UltraSPARC box to see whether 64bit big endian also works.
[ /linux/cyberjack |
permanent link ]
Working on lots of Presentation Slides
I didn't even notice it before, but within two weeks I'm now scheduled to give
six presentations. Unfortunately, none of them is exactly the same subject on
which I've presented before, so the amount of recycling I can do is quite
limited.
I've always considered doing slides for a presentation as "necessary evil",
but it's OK if you do it once every so often. But preparing six presentations
in a row is no fun at all :(
You can follow the progress in the svn repository
I sometimes really feel the need for a secretary... or someone who does boring
small jobs like HTML/Postscript conversion of all my presentations, and makes
them more conveniently accessible on the net. *sigh*. Sorry guys..
[ /linux |
permanent link ]
No more time for OpenTom at the moment
Due to an increasing workload, I won't be able to work on the OpenTom project for at least some weeks. I've published the current state of the SD Card driver in my personal directory of the OpenTom Subversion repository. If you want to pick up, feel free. I'll answer questions by email.
[ /linux/opentom |
permanent link ]
wiki.opentom.org online
We've put together some information on our OpenTom efforts at wiki.opentom.org. Feel free to check it
out. Additions of content very welcome :)
[ /linux/opentom |
permanent link ]
More hacking on the SD Card driver
re-engineering the SD card stuff turns out to be more time consuming than
expected. Not that it's particularly fancy or complicated - just obfuscated.
Apparently there are some quite complex data structures involved, that are hard
to analyze by looking at the disassembly.
[ /linux/opentom |
permanent link ]
All GPL issues with TomTom B.V. settled
I'm very happy that the GPL issues with TomTom have now all been settled, and
despite some early disagreements we're now very happy with the way TomTom has
handled this case.
The TomTom GPL page contains the latest
source of their 4.42 firmware. Pretty much all of the drivers have been
released with their source code (touch-screen, framebuffer, USB device,
accelerometer, GPS). Only (obviously) the SD-Card driver is missing in the
source and provided as kernel module. This is due to the stupid SD Card
Alliance licensing agreement, which basically puts every recipient of the
Documentation under an NDA.
So at the moment you have to put all of the OS into the initrd, which is loaded by the bootloader.
We're working on a solution for the card reader, though. At least MMC Card
support should be available soon.
[ /linux/opentom |
permanent link ]
The OpenTom Project was founded
Our distributed efforts in opening up the TomTom GO have now found a common home, the
opentom.org domain. There's the OpenTom
website and the svn.opentom.org subversion server.
There's still a lot under construction, expect more news here in this blog and
in the subversion repository.
[ /linux/opentom |
permanent link ]
2.6.10-rc1 kernel for OpenTom
Christian Daniel has managed to get
2.6.10-rc1 running on the TomTom GO. This includes a 2.6.x-rewritten frame buffer driver, USB Host and Device support.
The kernel tree has been made available on svn.opentom.org.
[ /linux/opentom |
permanent link ]
Back blogging again
I had some severe hardware problems during last week, resulting in almost one
week of server outage. We had to change power supply, ram, mainboard and cpu
in order to get the machine back running again - basically a whole new machine.
Sorry for anybody trying to access www/ftp/.gnumonks.org over that time. Email was not affected, since email is dealt with on a totally different box.
Thanks to my Towersoft friends
who took care about the physical repairs of the machine (it's located some
500km from my place).
[ |
permanent link ]
Chaosradio about Biometric Information in Travel Documents
Yesterday I've participated in a Chaosradio show about the recent
international push towards biometrics in travel documents such as passports.
Our focus has been on the flaws of biometric systems, the current plans of the
ICAO about MRTD's (Machine Readable Travel
Documents), the risks involved and why they are not an applicable tool to prevent
terrorist attacks.
If you're interested in listening to a recording of the show, it is available
at the usual location, ftp.ccc.de.
[ /politics |
permanent link ]
GPL Agreement with TomTom B.V.
Two days ago I signed an amicable agreement with TomTom B.V., a Dutch vendor of GPS navigation
systems. The press release is as usual at the gpl-violations.org homepage.
According to the agreement, they have a grace period until Oct 30, but apparently they already published some source code.
Unfortunately it's still incomplete to some degree, but I'm looking forward to getting this sorted out.
Also, this source is not enough in order to run your own kernel on the TomTom
GO, you will need some information on the firmware image layout and a
particular blowfish key. For more details on the internals of the TomTom GO,
please see the OpenTom of
Christian Daniel.
I'm looking forward to convert to TomTom into a all-in-one car computer,
including wardriving (USB WLAN with kismet) support and MP3/Ogg-Player with USB
hard drive :) Not to forget bluetooth keyboard support, etc. :)
[ /linux/gpl-violations |
permanent link ]
TomTom and your own kernel
I've started to merge the TomTom specific patches into a plain 2.4.27 kernel.
Most of it is quite straight forward, since apparently they backported half of
the kernel to 2.4.18-rmk6 (which is what they use as base). I don't really get
it why companies still develop new products for 2.4.x, especially for really
old version like 2.4.18. In the windows world, nobody still writes windows
3.11 applications, why do they start this kind of crap with Linux? *sigh*
Anyway, I'm thinking about a 2.6.x kernel port at some point, but obviously
this is not an important issue on my agenda and I'd rather get some netfilter stuff running first.
[ /linux |
permanent link ]
Berlinux 2004
Some time ago I was approached if I would be able to give a presentation
at Berlinux 2004, Berlin's local
incarnation of a Linux conference, organized by the Berlin Linux User Group.
This should be the first contact to any user groups I've had for about five
years. I've tried to avoid Linux user groups exactly because of the 'User'
part. I have a hard time dealing even with Linux-savoy iptables users, let
aside users who need explanation how to install a given Linux distribution or
even how to use a file manager.
Unfortunately Berlinux seems to be very user-oriented, too. I arrived about 40
minutes early and am now waiting for a presentation explaining the principles
of mounting and the Linux file system layout to finish.
I'm surprised that Berlinux is so small, considering that Berlin is about seven
times the size of my old hometown of Nuernberg, and the ALIGN Linux Setup Parties had about the same
size.
Oh yes, does the idea trouble you that you know somebody at every international
Linux conference, from Bangalore to Ottawa - but at an event in your own
hometown you have a hard time finding any person whom you know? That's how I
feel. Misplaced, at the wrong event :(
[ /linux/conferences |
permanent link ]
Porting PPTP conntrack/nat helpers to 2.6.x
I've always refused to do the port of the PPTP conntrack/NAT helper I wrote for
2.4.x because there's higher priority items on my agenda.
Apparently it helped, as I was told Mandrake did a port to 2.6.x. I thought
that is great news, and I thought it'd take an hour or so to get it merged.
Unfortunately that 'port' was totally incomplete. NAT couldn't have worked at
all, and if you sent it a nonlinear TCP packet it would very likely crash your kernel.
In the end I spent the whole afternoon at it, with a resulting patch that is
about the same size as the original code :(
The code is now in our subversion repository, I didn't have the time test it so
far, so any testing you (yes, you, the reader) might give it would be
appreciated.
[ /linux/netfilter |
permanent link ]
Another patch submit day.
Today I've submitted hashlimit, CLUSTERIP and CONNMARK to the 2.6.x kernel.
After resolving some glitches with CLUSTERIP, DaveM took all three :)
This means we're again one step further submitting stuff from patch-o-matic into mainline, which is always a good thing.
[ /linux/netfilter |
permanent link ]
GPL Agreement with Gigabyte Technologies
I've managed to get an amicable agreement with Gigabyte Technologies B.V., yes
that's the big worldwide known vendor of Mainboards and other PC equipment :)
The press release is at the gpl-violations.org homepage
[ /linux/gpl-violations |
permanent link ]
I should do more press releases
I'm sorry for that. GPL-enforcement progresses meanwhile. I've been able to
obtain amicable agreements with three more vendors (D-Link, Gigabyte, TomTom),
and there are two more open / ongoing cases at this point.
Expect more news and even an official press release during next week
[ /linux/gpl-violations |
permanent link ]
Fun with incompetent BMW employees
So during the repairs of my BMW F650's carburetor, I lost the choke plunge.
Not a big deal, just a tiny part regulating the fuel/air ratio at engine
startup time.
So I picked up the phone and called the spare part department of BMW in Berlin,
and told them the exact part I wanted. "Chokekolben" is 100% not possible to
be misinterpreted, there is no other part with the same name. So I was told
that this part is not available on it's own, but just in a set bundled with the
linkage/string that actually attaches to the plunge.
One day later I was called that the part had arrived. It took me about an hour
to get to the BMW subsidiary, only to find out that they had ordered the choke
string, but it came without plunge.
They showed me the exploded view of the carburetor, and it was very clear that
the plunge is sold separately for about EUR 3. I have no idea how one can
misunderstand the exploded view and/or the spare part list associated.
After ordering the plunge, I asked them if they made the exploded views
available for customers, so they could directly order a particular spare part
number in order to avoid such misunderstandings. Apparently they only provide
those spare part catalogues to their BMW partners, and they see no way how they
could provide me a copy. *sigh*. So I will have to rely on some brain dead
spare part sales assistant who has most likely never disassembled that bike ..
Luckily, there's eBay and I found somebody who sold the original BMW spare part
catalogue on CD-ROM. What would the world be without eBay.
BMW, this happened about two weeks ago, and I still don't have that spare part.
[ /personal |
permanent link ]
Yet again more cases coming up
I've authorized my lawyer to act in five more new GPL violation cases. As
usual I will not disclose their names until some kind of agreement (or a court
order) is in place.
In one of the cases we unfortunately now had to go after a reseller, since the
warning notice to the Dutch vendor was unanswered. Apparently the strategy is
working, since the German reseller now put pressure on the Dutch vendor, who
suddenly now replies to us ;)
[ /linux/gpl-violations |
permanent link ]
Conntrack events for 2.6.x
I've separated out Patrick McHardy's conntrack events from the
nfnetlink-ctnetlink patch and ported it to 2.6.x. The patch was posted to
netfilter-devel, in case you're interested.
For those of you who don't know what this means: It means that the first part
of what is required for a 2.6.x ct_sync port is now done ;)
[ /linux/netfilter |
permanent link ]
ct_sync ethereal plugin
While doing some more ct_sync testing/debugging, I found out that for some
reason my ctnl_dump program didn't work anymore. Instead of fixing it, and updating it to CTSP (conntrack sync protocol) version 2, I decided to write a plugin for the well-known packet analyzer ethereal.
Due to the nature of the CTSP, it passes arch- endian- and
configuration-dependent data structures between master and slave. This means
that it is virtually impossible to write a analyzer that will work in any of
those combinations.
My plugin now assumes that you use a little-endian 32bit machine with the
pptp-conntrack-nat patch applied.
The plugin turned out to provide very useful information, and I was able to fix
some issues in ct_sync using it.
[ /linux/netfilter |
permanent link ]
No big news this week - I'm in Astaro labs
I'm about to do one week of benchmarking and profiling using an Ixia four-port
Gigabit Traffic generator and a Sun Fire v20z dual Opteron box in the Astaro labs. Let's hope I can find some code
pieces in the network stack that can be optimized in order to achieve higher
performance...
[ /linux |
permanent link ]
xfrm_user.c doesn't use netlink correctly
If you read the netlink documentation (and look on how existing users such as
rtnetlink or ipt_ULOG uses it), then all messages part of a dump have the
NLM_F_MULTI flag set, and the dump is terminated with a NLMSG_DONE message.
The code in net/xfrm/xfrm_user.c however dumps those messages without the
NLM_F_MULTI flag. I've hacked a first patch, but apparently it doesn't catch
all cases.
[ /linux |
permanent link ]
Motorbike problems
I wanted to take pictures of a recently detonated old building in Berlin. I
wanted to go there via motorbike. Unfortunately the bike got some problems:
After about 3km from my home, it suddenly stopped and refused to start again.
While trying to get it running, I suddenly noticed vast amounts of fuel leaking
from the air filter. That's a bad sign, it basically says that somehow the carburetor is getting fuel into the wrong direction.
I went home by public transport (no photos taken), and luckily found a truck
rental that was open on Sundays. So I managed to get the bike back home, take
everything apart and clean the carburetor. I couldn't find something serious
like a worn out fitting... all I found was a minimal amount of dirt.
I'll put the bike pieces back together tomorrow, let's see whether cleaning the dirt actually helped. Jeez, as if I hadn't enough to do already...
[ /personal |
permanent link ]
Linux Bangalore / 2004
The LB/2004 organizers have
officially appointed me as speaker recruiter ;). Apparently they have some
trouble in contacting various Linux developers due to over-reactive spam
filters (blocking everything from India, heh?).
This means I end up writing emails trying to convince folks such as Alan Cox,
Andrea Arcangeli, Russell King, Erik Andersen, Robert Love, ... to attend this wonderful Indian conference.
Did I mention that I'm going to be there this year, too ;)
[ /linux |
permanent link ]
2.4.x backport of neighbour cache rework
I've finished my 2.4.28 and 2.4.21 backports of our recent neighbour cache
re-work (see netdev of last two weeks in case you're interested). 2.4.28 was
quite straight-forward, just the missing per-CPU hurt a bit. 2.4.21 was pretty
hard, since the neighbour cache apparently changed quite a bit between 2.4.21 and 2.4.28.
But well, it's over now. Thank god :)
[ /linux |
permanent link ]
Generalized Linux network statistics
While working on the neighbour cache, I introduced some generic neighbour cache
statistics. They are done in the core, but exported to userspace for every
ncache separately (arp, ndisc, atm_clip, decnet). I used the same techniques and file format as rt_stat.
Martin Josefsson also recently introduced ctstat, the same kind of statistics
for ip_conntrack. He did a copy+paste 'port' of the rtstat userspace program.
I now also needed four more new copy+paste 'port's. And I couldn't do it.
Copy+Paste style ports are what I am fighting in the iptables world for two
years, so I certainly don't want to introduce them elsewhere..
The result is what I call lnstat. It's a generalized
version of rtstat, it works with neighbour cache, routing cache and conntrack
statistics - either separately or all at the same time. It has user-defined
formatting (field width) and key selection, as well as some other bells and
whistles. Let's hope this gets integrated with iproute2 soon, so people can
benefit from it.
I also thought about writing some daemon, but abandoned that idea in favour of
writing a ulogd2 plugin for it... this means ulogd2 will be able to log
per-packet, per-flow and generic things such as statistics...
[ /linux |
permanent link ]
First Solaris-based contract in four years
For more than four years, I did 100% linux based work. But apparently there
are still people interested in Solaris stuff, since I just got my first solaris
based contract in quite some time.
Spent an incredible amount of time getting Solaris 9 installed on my Ultra 5,
which was only running Linux before. I never understood how Sun could rectify Solaris being so much slower than Linux on their own hardware ;)
[ |
permanent link ]
Proceedings of Developer Workshop 2004 online
I finally managed to finish the write-up and markup of the proceedings. They
are available in a number of formats at the documentation section of the netfilter home page.
In theory, there could still be lots of semantic markup added, but well, who cares...
[ /linux/netfilter |
permanent link ]
pkttables finally making some progress
I've found some time to work on pkttables again. Isn't that great news? If my
brain is not completely broken, I've now worked out a RCU-powered way to have
full table traversal with a completely lock-less reader path, while providing
atomicity either on table- or chain level.
Also, I ripped the "struct nf_attr" and NFA_xx macros from the nfnetlink core,
since they get replaced by my vTLV (Versioned TLV) code.
With some luck I'll be able to continue my pkttables work next week
[ /linux/netfilter |
permanent link ]
CLUSTERIP is in patch-o-matic-ng
About one year ago I did some work for SuSE
in implementing load-balancer-less load-balancing clusters ;) This is achieved
by replying to ARP requests with a link-layer multicast address, so all nodes receive all packets. Hashing parts of the ip header now determines whether the packet is to be passed up the stack on a given node.
The result is called the iptables CLUSTERIP target, and I've now finally put it
in patch-o-matic-ng, since it was only available in my undocumented public CVS
tree so far.
[ /linux/netfilter |
permanent link ]
Siemens is violating the Settlement
Siemens is offering the SE-505 firmware on their homepage without any reference
to the source code, the GPL, or the GPL text. This is in violation of the signed settlement agreement that I have concluded with them.
The lawyer is already informed, and we'll see what kind of legal options we now have in pushing Siemens [again *sigh*] for GPL compliance.
[ /linux/gpl-violations |
permanent link ]
Reworking the Linux neighbour cache
Since I've lately had some customer issues with regard to neighbour cache
overflows, I studied the current code quite a bit. From my point of view, it has a couple of shortcomings.
The general problem goes like this: What do we do, if we're attached to let's
say a /16 (formerly 'Class B') network that has a theoretical limit of 65535
neighbours at layer 2, and somebody sends us a single packet for every one of
those neighbours. We now start to send ARP requests for all those neighbours,
and until those time out (1sec default), thus flooding our neigbour table.
The current Linux strategy is to configure a static limit (default: 1024), and as soon as we reach the limit, we start deleting old entries. 'old' entries are those for real hosts to which we've recently had connectivity... We do not expire any of the incomplete neighbour entries in order to avoid ARP-floods.
So if you want to avoid that, you always have to set the gc_thresh3 value to at
least the theoretical number of total machines that could be directly reachable
at layer 2. While this is not a problem with /16, it suddenly becomes one with
/8, or with the extremely large IPv6 prefixes.
The problem is further increased, since the number of hash buckets is very low
(static number of 32), and the used hash algorithm apparently has a bad
distribution. So either we increase the hash table, increase the number of
buckets and improve the hash algorithm, or we change the expiration scheme to
also drop incomplete entries. But the current situation is definitely not good.
So I picked up some old 2.4.x patches from Tim Gardner, ported them to 2.6.x
and brushed them up. The number of hash buckets is now a kernel boot
parameter (if not specified, the hash is dynamically sized, like the TCP
syn-queue, fragment queue or ip_conntrack hash). The hashing algorithm now
uses a Jenkins hash, just like all other parts of the kernel use, too. The
patch is in testing at my machines at the moment, but I think I'll push it
soon.
[ /linux |
permanent link ]
libiptc2 bugfix (upcoming iptables-1.3.0 prerelease)
Since the segfault-bug in my recent re-implementation of libiptc has now been
fixed, I think we're about one week before a iptables-1.3.0 prerelease for
public beta-testing.
[ /linux/netfilter |
permanent link ]
NAPIfied natsemi driver
I've now successfully NAPIfied the second NIC driver: natsemi.c... this was the
only remaining driver that I care about, since it is used in the PC Engines WRAP embedded systems that I use
as routers/bridges/wlan-gateways.
The result is that I can now get about 34kpps routed on an embedded 266MHz
Geode CPU at full 148kpps 64byte single-flow udp flood on the input NIC.
[ /linux |
permanent link ]
Adding NAPI support to the sungem.c Ethernet driver
Yesterday I implemented NAPI support for the sungem.c driver. This was done
because I was annoyed by the fact the my notebook (Apple Powerbook with on-board
Gigabit Ethernet) could still be killed by a machine running pktgen and
flooding it with some 700 kpps.
After submitting the patch, David Miller pointed out that he has added NAPI
support to sungem.c to the bitkeeper tree about four days ago :( So I spend a number of hours in duplicating work that was already there... not that I didn't have other stuff to do.
Well, at least I learned a bit more about Linux NIC drivers..
I'm now facing the task of implementing NAPI for the natsemi.c driver, which is
used in the PC Engines boards that I've
been using recently as embedded Routers / Firewalls.
[ /linux |
permanent link ]
Working on the summary / proceedings of the 3rd netfilter developer workshop
Spent a couple of hours putting the notes of the 3rd netfilter developer workshop together in a single file, adding lots of Docbook-XML markup, ...
It's still far from being complete, but I have to finish this ASAP..
[ /linux/netfilter |
permanent link ]
Intel e1000 (82546) TX performance
After recent discussions with Robert Olsson at the netfilter workshop, I've
decided to investigate a bit further, why the Intel e1000 gigabit MAC's are
quite limited when it comes to TX performance and large numbers of pps.
My first assumption was that the in-kernel pktgen.c code might not keep the
transmitter busy at all times, resulting in only 760kpps (out of the
theoretical maximum of 1480kpps).
So I hacked the e1000 driver to hardcode a refill of the Tx queue with the same
skb over and over again. Using a 2048 Tx descriptor ring, I was able to keep the transmitter busy at all times (E1000_ICR_TXQE interrupts).
Unfortunately, I still didn't get more than the 760kpps in this setup (PCI-X,
66MHz, Dual-Opteron 1.4GHz, DDR-333 (PC-2700) RAM. So either we're seeing a limitation of the 82546 chip, or the PCI-X bus / memory latency / whatever.
I'll try the same experiments on a different machine with PCI-X 100 / 133MHz in order to find out what exactly is causing this limit.
[ /linux |
permanent link ]
netfilter workshop / Linux Kongress 2004
I've not been able to write any articles for this log over the last few days,
since I've been busy with the third netfilter developer workshop and
Linux-Kongress 2004.
The netfilter workshop went really well, apparently the
[ /linux/netfilter |
permanent link ]
Started a new 2.6.x based mini router distribution
I'm in the process of deploying a couple of PC Engines WRAP.1C embedded x86 boards deployed in my apartment. They make neat little playgrounds for Router/NAT/VPN/WLAN/... style appliances.
Unfortunately I didn't find any embedded Linux distribution project that was up
to my demands. Apparently they all use age-old kernels (2.4.17 or something
ancient like that). And they very rarely come with a decent automatic build
system that would allow you to rebuild it from scratch, adding your own
patches, ...
So what did I do? I started my own :(. Not that I'm proud of it, but it was
necessary. My home VLAN/firewall/PPPoE/NAT/VPN router is now running the
very first image of this new distribution I called 'gRouter'.
It's main features are kernel 2.6.8.1, uClibc-0.9.26, busybox-1.00rc3, pppd
with in-kernel PPPoE support, quagga, iptables-1.2.11, openvpn-1.6.0, and
dropbear for SSH. It all fits in about 8MB of compact FLASH.
The build process is semi-automatic, apart from a few glitches the whole image
compiles itself. I stole some of the build magic from the WISP-DIST project
(part of LEAF), although this is all quite simple scripting.
After some more cleanups and testing, I plan to release this distribution.
Please don't expect any support, or any configuration tools. It will be
available for Linux experts who can configure and setup their system from
scratch, and want to have the gadgets of the latest software releases.
On the todo list is cross-compilation support (well, since it is uClibc based, it already does cross-libc-compilation), madwifi support, and especially IPsec using the 2.6.x kernel implementation.
[ /linux |
permanent link ]
Getting the external VGA of my Apple Powerbook (TiBook IV) working
If you've attended one of my presentations during the last 12 months, you will
certainly have noticed the poor quality of the slides. Yes, the content and
the presentation is poor, too - but I'm mostly referring to the optical quality.
I've already spent at least a whole day in the past in trying to get the
external VGA working with Debian/ppc, with little success so far. I really
don't care whether the external port mirrors the content of the display, or if
it runs in dual head mode.
Today, I spent some three more hours in trail-and-error with the radeon driver
of the dri-trunk XFree86. I tried CloneMode, Dual Head, with and without
FBMode, and about any other parameter within XF86Config-4.
In the end it turned out that the man page was not up-to-date, and the
preferred way to get it running was the so-called MergedFB mode. This wasn't
as easy to configure as expected, and I still got lots of 'Signal 11'
segfault-style crashes.
The crashes seem to be totally unrelated to my graphics setup. In fact, it
crashes when eth0 is not configured yet, but works after the network device is
up. Now please somebody step up and explain...
[ /linux |
permanent link ]
Finishing preparations for upcoming netfilter developer workshop
I've spent a significant amount of time over the last couple of days with the
final preparations of the upcoming 3rd netfilter developer workshop. This is
the first one where I'm in charge of every tiny bit of the organization, and I
hope I got everything right.
The first attendees are scheduled to arrive tomorrow. They might even arrive
before me, since I'll be heading the 500km down south tomorrow.
[ /linux/netfilter |
permanent link ]
More Allnet Devices contain Linux
I've now successfully proven that the ALL0185A, ALL0186, ALL1297, ALL2100, ALL2110 and ALL6100 devices contain the Linux kernel and are not distributed according to the GPL.
Considering the out-of-court agreement that I have concluded with them earlier
this year in ALL0277, I have to say I'm a bit disappointed that this happened
again. It should be in their own best interest to distribute within the GPL
license terms, and not first try to infringe and wait until somebody complains.
I've contacted them, and they promised to publish the source code and adhere to the license within a short term. Let's see how this continues.
[ /linux/gpl-violations |
permanent link ]
Fujitsu Siemens Corporation not fulfilling amicable agreement
As part of an amicable agreement, Fujitsu Siemens Corporation (FSC) agreed to
make a donation to the German Unix Users Group. It came to me as a surprise,
that GUUG has not yet received the funds even four months later!
Again, I am very disappointed by the behaviour of the former GPL violators. It
should be in their own best interest not to produce any negative publicity.
[ /linux/gpl-violations |
permanent link ]
On VIA's failure to provide adequate Linux support
VIA is definitely one of the most innovative producers of PC-hardware. Their
EPIA-series mini-ITX and nano-ITX mainboards are ideal for small appliances,
such as firewalls, VPN-gateways, and especially home entertainment platforms
such as PVR/DVR applications, DVB-Receivers, DVD/VCD/AVI-players, VideoLan
receivers and such.
Just two days ago, VIA made a press
release on their new VeXP
3.0 release, a VIA-enhanced fork of xine. To the unfamiliar reader, this press
release raises the impression that VIA is really involved with Linux and the
Free Software community.
This is just terribly wrong. They do anything but to support GNU/Linux.
Comparing this press release with reality, I think VIA's Linux involvement as a
whole is nothing more than a PR strategy.
I've recently investigated the "Linux support" they make available for their
EPIA platforms. Even from the first glance it was obvious, that VIA just
doesn't have any idea on on what it takes to "Support Linux".
All they do is to publish proprietary, pre-compiled kernel frame buffer and
XFree86 display drivers for a limited number of particularly old GNU/Linux
distributions.
Oh yes, I almost forgot it: They also publish the source to some 'lite' driver
which lacks all the functionality needed for hardware-assisted MPEG2 decoding.
This is obviously useless, since the whole point of buying a small fan-less
board with hardware MPEG acceleration and TV-Out is to use the acceleration.
So their "Linux Support" is so good, that a number of people have to spend days
and days in reverse engineering their binary proprietary drivers. You can
find more information about the
reverse engineering effort. My special thanks are going to Ivor Hewitt for
doing all this work.
But wait, wasn't that what the Linux folks usually did with Windows drivers?
Welcome to the world of "VIA Linux support", where instead of reverse
engineering Windows drivers, we now have to do it with Linux drivers.
If VIA was really interested in providing good GNU/Linux support for their EPIA
products, they would
-
write full source code drivers licensed under appropriate Free Software
licenses.
-
make those drivers use standard interfaces, the respective project's coding
style, contain useful comments.
-
publish those drivers as patches against the latest development version of
the respective project (kernel, XFree86, Xine)
-
Work with the respective project maintainers to integrate those patches
-
not have to care about maintaining RPMs for each and every distribution
-
not have to care about porting their drivers to ever-changing API's, since
they are included in the respective Free Software projects
-
Provide documentation for their hardware down to the register level, so
the Free Software community can continue development extending to features
maybe not yet covered by the current driver.
Related Links:
http://lwn.net/Articles/99464/
VIA's original press release
http://www.viavpsd.com/
VIA's EPIA homepage
http://www.viaarena.com/
VIA's support forum and driver downloads
http://www.epiawiki.org/
The comprehensive source of EPIA/Linux related information
http://www.ivor.it/cle266/
The reverse engineered driver page
[ /linux |
permanent link ]
Video Documentation on 21C3
I've attended a meeting on the subject of providing audio/video documentation
at the 21st Chaos Communication
Congress. During that meeting, I was appointed as being responsible for
this part of the 21C3 conference.
So we want to do on-the-fly encoding of four video signals from DC1394 cameras
to DVD-compatible MPEG2, low-resolution MPEG4 for live-streaming, and OGG audio
only for live streaming.
I did some preliminary experiments with the available experimental x86_64
assembly patches for ffmpeg, and it turns out that at least theoretically a
1.6GHz AMD64 should have enough power of doing those three encodings at the
same time.
Unfortunately the dv1394 device at the moment only supports one encoder mmap()
ing the ring buffer of incoming 1394 frames - but that should be fixed pretty
easy.
I'll do some more experiments in the next couple of weeks, stay tuned.
[ /linux |
permanent link ]
Main netfilter.org server has been replaced
Yesterday I finally got around moving almost all netfilter.org services from
our old Sun Ultra5 to the new XServe ClusterNode.
Unfortunately there were lots of complications, so I had to stay awake until
5am in order to get all services running again. At least for now, everything
seems to run smoothly.
[ /linux/netfilter |
permanent link ]
Using a human-based data acquisition plugin
Why buy expensive data acquisition boards, if you can have a cheap human being
entering the data on some terminal? No, just kidding.
Anyway, GSPC now has a gpsc_acquire_user.c plugin that retrieves measurement data via a ncurses-based dialog instead of any data acquisition board. This is useful for testing, but also in some real-world cases.
[ /linux/gspc |
permanent link ]
Two hard drives dying in one week
This week already the second hard drive in one of my workstations died.. both
times it was the same model: IBM DTLA-307060, produced Nov 2000 in Hungary. If that isn't some coincidence. Maybe they have a built-in 'best before' date :(
So both my main workstations (Dual PIII-733 and a Dual Apple G4-500) were
inoperable, isn't that great? The good part is that they've been replaced with
silent Samsung SP1213N models, significantly reducing the noise level in my
office.
[ |
permanent link ]
Off-the-shelf multi-port serial cards and Linux
This is now the third time I've bought some PCI serial multi-port card (6 to 8
ports) that claimed to have 'Linux support'. If you then read the document,
the vendor bluntly tells you that Linux generally doesn't support more than
four ports, so if you have two built-in ports, you can only use two more.
I've never read such bullshit anywhere else ;)
So after some minor twiddling, I now submitted a patch adding support for this particular 6port device. Apparently there is either a wide variety of such boards, or almost no Linux users... A couple of years ago I added support for an AFAVLAB 8port serial card, to the Linux serial driver.
I think I now know way too much about the serial driver. Not stopping with
those two PCI 8250 based boards, I did lots of serial driver hacking for the
XServe G5 and also for my recent ARM embedded work. Let's hope I can again advance to some more exciting work in the future.
[ /linux |
permanent link ]
Attaching an UW-SCSI hard disk to an embedded ARM922T
No, I'm not doing this for fun, this is part of work. It turned out that nfsroot is a
bit of a problem while you're hacking the core network stack (and everything
breaks all the time). So I now attached an 18GB UW-SCSI disk to an old aic7xxx
controller and plugged this into my ARM development board. Seems to work quite
fine, as long as the aic7xxx_old driver is used. The new one apparently calls
pci_alloc_consistent from interrupt context ?!?.
[ /linux |
permanent link ]
News on the GPL Violation Front
It's been some time that I've reported news on the GPL violation side... Thus,
no news is good news, one could think. Unfortunately to the contrary, I've
been receiving a number of new GPL violation reports, unfortunately none of
them containing my copyrighted work - and thus I am now looking for the
respective copyright holders in order to get this issue sorted out.
Stay tuned...
[ /linux/gpl-violations |
permanent link ]
Performance of system logging
One of my customers recently had a serious performance issue with one of his
installations. Surprisingly, it wasn't even the real applications software
itself that had performance issues, but the mechanism used for logging from
this application.
So I started to think about the way logging usually works within a Linux-based system.
The server applications can be divided within two groups. One of them logs via
syslog(), the other logs directly to it's own files. The logging itself
happens synchronously, i.e. blocking the normal code flow until the log line
was written. In the case of syslog, it might block because the syslog pipe is
full - in case of stand-alone files, the file/io might take some time to
complete.
Even in a multi-threaded or forked model of a network server program, this
might pose considerable problems with regard to threads waiting for their log
i/o to complete.
Syslog itself might not be as bad, especially since the 2.6.x pipe
implementation works with only the minimal necessary amount of copying, and
supports larger pipe sizes to avoid writer blocking.
Some people however tend to use something like syslogger in order to redirect
the log output from programs with no syslog support also into syslog. This
means that you have one pipe between your application and syslogger, and
another pipe between syslogger and your real syslog daemon.
Comparing this issue with networking is actually not too problematic. In
networking, we have packets that are passed from one process to another... with
logging it's not a packet but usually one or more lines of text (that is, about
60 to 240 characters per entry).
You don't want to copy this data around and around... and in a lot of
installations you'd rather want to use a couple of log lines than to slow down
your application just for some statistics that you might collect.
Of course, you don't want to modify any of the existing applications, too -
they should just be able to use syslog() calls as usual. OF course you could
load a LD_LIBRARY_PRELOAD lib and redirect the syslog() calls, if needed.
So what I came up with, is something like a partially mmap()able pipe. The
logging process would log to that pipe like it would with any other file
descriptor. Internally, that 'pipe' has a ring buffer of configurable size.
The pipe-reader could now mmap() this ring buffer into his address space in
order to read the log.
This scheme should have the advantage of not blocking the writer if the pipe is
full (it would just wrap around the ring buffer), and it avoids copying the
data from some in-kernel pipe buffer into the user-space of the pipe reader.
Did you notice, this now looks perfectly like the DMA ring buffer of your Ethernet device and the Linux softirq handler ;)
Anyway, as I didn't do any vm / vfs hacking in Linux so far, this is not a
trivial thing to implement. And I have lots of other work at this point.
However, I'd certainly like to investigate the possible performance gains [losses?] of this idea. Comments welcome.
[ /linux |
permanent link ]
IETF work on NAT behaviour
Apparently some people within the IETF have started a new working group called
'BEHAVE'. It is about the behaviour of NAT devices on the internet, and their
inconsistent and incompatible behaviour. The working group aims to give
guidelines to ipmlementors, in order to assure interoperability with new
applications such as VoIP and peer-to-peer protocols, as well as multicast and
others.
Certainly a topic that is in in the main focus of my interest, so I decided
this is the right point in time to start participation in the IETF.
For more information about behave, see the mailinglist.
[ |
permanent link ]
Upcoming Chaosradio episode on software patents
The next Chaosradio radio show will be
about the ongoing debade on software patents, especially the recent development within the European Union.
Being part of the anti software patent movement for about 4-5 years now, I am
more than happy to help with the radio show on this subject.
The radio show will be on air on Sept 01, 10pm GMT+2. If you understand
german, there's a MP3 live stream available on the homepage.
[ /politics/swpat |
permanent link ]
Working on embedded Linux ARM SoC project
While there hasn't been any update on this weblog for quite some time, I've
been buried under a lot of work.
One of the most interesting projects is an embedded ARM-based SoC project with
special network acceleration hardware. Unfortunately I'm not allowed to
talk too much about it at this point, but be assured it is very exciting, and
of course runs Linux :)
During development I found it quite comfortable to run the small embedded
system with nfsroot mounted from some larger box. The nfsroot contains a
debootstrap'ed installation of Debian sarge for ARM.
The main problem for this kind of operation is the limited on-board memory.
But I'm tempted to put a 64MB graphics card into one of the PCI slots and hack
the Linux kernel to treat this framebuffer as (somewhat slow) RAM :)
[ /linux |
permanent link ]
Booting from a md raid device on powerpc
Apparently, nobody has ever tried to do this so far, since the mac partition
handling code in the Linux kernel had no provisions for enabling auto-detection
of md software raid.
I've now written patch for Linux 2.6.8, available at http://gnumonks.org/ftp/pub/patches/linux-2.6.8-mac-autoraid.patch
implementing this feature. All you need to do is apply that patch, and make
sure your md partitions have the type 'Linux_raid_autodetect' in the mac
partition table.
[ /linux |
permanent link ]
Figured out the fan control on the XServe ClusterNode
I spent the last couple of hours figuring out the missing bits of the
fan/thermal control on Apples Dual XServe ClusterNode. Luckily it's very
similar to the design Apple used in their Desktop G5 machines, so I can build
on the work that Benjamin Herrenschmidt did with his thermal_pm72 driver.
So in case anybody is interested in the technical details: Eight fans are
controlled by the FCU (Fan Control Unit), which is attached to a i2c bus of the
Apple U3 northbridge.
There are three RPM controlled fans per CPU. The Left CPU (viewing from the front of the machine) has fans #1,2,3. The right CPU: #4,5,6.
The other two fans are not RPM controlled, but just PWM controlled... so
instead of setting an RPM, you have to set a pulse-width between 10 and 100%.
PWM Fan #1 is located between RPM-fan 3 and 4 (between both CPU's) and it's job
is to keep the U3 chip cool. PWM Fan #2 is located behind the PCI-X slots and
thus cooling them (too bad in my machine there is no card to be cooled *g*).
Regulating the CPU fans is quite easy, since there is a per-CPU temperature
sensor, and also a voltage and current reading, so we can calculate the power
consumption of each CPU and tune the fans accordingly.
For the U3 it is a bit more difficult.. I have not yet found a way to get a temperature reading for it, but I'm quite sure there is some temperature sensor somewhere.
As for PCI cards, there is apparently some way to read the power consumption -
but of course again undocumented and not reverse engineered yet. As I don't
have PCI boards in my box anyway, I personally don't care that much. But I
should now stop arguing rationally, since a machine hosted in some rack-space is
very unlikely to need fan control at all :)
I'll try to make a somewhat cleaner unified driver for PowerMac7,2 and
RackMac3,1 and post a patch in the next couple of days.
I really wonder why Apple is not releasing their FCU driver source code for
Darwin... it's really annoying. And I doubt they can claim that it contains
any valuable intellectual property that their competitors are not allowed to
see ;)
[ /linux |
permanent link ]
Finally the XServe ClusterNode runs Linux!
Yes, it does. I now have two partitions: One running the experimental Gentoo
ppc64 port, and another one running the overly-conservative Debian woody
ppc32. The plan is to boot into Gentoo, and run publicly-accessible
production services within the Debian woody chroot.
So how did I make it? Well, I gave up on the idea that the usual installation
process of any distribution would work. So instead of trying to fix up whatever
goes wrong in the installation scripts, I just escaped to a shell ASAP, run
mac-fdisk, mkfs.ext3, extracted the stage3.tar.gz and did the rest of the
Gentoo install.
Debian was then installed using the convenient debootstrap tool.
One of the major remaining questions is however: Does the Apple XServe
Hardware give you anything similar to Sun boxes, where you could just send
break over the serial line and get into OpenFirmware? This is very convenient
for remotely resetting machines without any local 'reset-staff' present.
After some chatting with Benjamin Herrenschmidt, apparently nobody is working
on getting fan rpm/speed/temperature control implemented on the XServe so far.
Well, as it's a rack-mounted machine sitting in some hosting center I don't
really care about the noise anyway.
More interestingly, the Apple KeyLargo2 based machines have a Hardware
Watchdog. Driver Source code is available within the public part of the Darwin
kernel, so it should be easy to implement a Linux driver for this. Maybe I'll
find some time to dive into this.
[ /linux |
permanent link ]
IPv6 packet filter benchmarking
It seems like a German university is currently doing feature analysis and
benchmarking of IPv6 packet filters. Coincidentally, I'm going to near that
university next week anyway, so I'll stop over for a short visit and help them
with their ip6tables evaluation setup.
I would be very interested to see some numbers on ip6tables... as we just
discovered at the networking conference in Portland, nobody seems to be doing
benchmarking / profiling on the Linux IPv6 code so far.
[ /linux/netfilter |
permanent link ]
Database Design + Content for GPL-Violations
In order to keep track about the gpl violations that I am encountering myself
or that are reported by fellow users, I really need some semi-automatic system to keep track of this.
Being a RDBMS geek in my former life, I designed a SQL-based data model to cope
with the individual objects such as vendors, products,
product-firmware-versions, violations, settlements, compensations, comments,
documents, contracts, ...
It all turned out to be more complex than I thought initially. But I think it
was really worth the effort.
This database is for strictly internal use, since there is a lot of
confidential information in there. However, according flags indicating the
public/private nature of the data records are included in the data model. At
some later point I might extract the public information to create some web
pages at www.gpl-violations.org.
It's main target is to allow me keep track with what's going on, and also keep
track about what has been verified where, if for new upcoming firmware images
the source code was made available, if the source was complete, ...
I've already filled in lots of the existing data I have, but it's far from
being complete. This needs some more time of filling in data records.
And yes, I built some simple forms using GNU Enterprise Designer and Forms. It's still in 0.x stage, but usable for easy tasks.
[ /linux/gpl-violations |
permanent link ]
Installing Linux on a G5 ClusterNode XServe
Now that I got this decent new dual G5 box, I wanted to install Linux.
This turned out to be an extremely difficult job, as apparently nobody has ever
tried to install Linux on any of the new XServe G5 Series machines, neither
32bit nor 64bit kernels.
There are a number of challenges:
- No internal IDE or SCSI CD-ROM
- Only serial console
- A very new hardware with little Linux support
First I tried a number of ready-built installation ISO images, including the
current sarge Debian-installer image for PPC, and the 32bit and 64bit live
images of Gentoo.
The first thing I had to do is to disable autoboot and enable the serial
console. Luckily, the box actually ships with a manual that instructs you how
to put the OF boot console on the serial port. You have to press the admin (!)
Button at the front of the box a magic number of times.
To permanently make the serial console work, use the following OF commands:
> setenv input-device scca
> setenv output-device scca
Next I had to figure out how to boot from the external firewire cdrom..
apparently this depends on your OF device tree and the GUID of your firewire
device. On my particular box it works with
> devalias cd /ht/pci@5/firewire@e/node@00d04b3c50090210/sbp-2@c000/disk@0
Using Commands like
> dir cd:,\
I was then able to list files on the CD-ROM. To boot the yaboot loader on a
Debian installer cd image, you can use
> boot cd:,\install\yaboot
sbp2:Open ->login?
speed=ffffffff 2 2 load-size=239a4 adler32=a5cf5aa0
Loading ELF
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 Config file read, 2907 bytes
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 \
sbp2:Open ->login?
speed=ffffffff 2 2 Welcome to Debian GNU/Linux sarge!
This is a Debian installation CDROM,
built on 20040729.
The default option is 'install'. For maximum
control, you can use the 'expert' option.
If the system fails to boot at all (the typical
symptom is a white screen which doesn't go away),
use 'install video=ofonly' or 'expert video=ofonly'.
The plain options are for the powerpc family of
processors (from 601 to G4). The *-power3 options
are for IBM Power3 boxes, and the *-power4 options
are for IBM Power4 and Apple G5 boxes. Press the tab
key for a list of options, or type 'help' for help.
************************************
If in doubt, just choose 'install', and if that
doesn't work, try 'install video=ofonly'.
************************************
Welcome to yaboot version 1.3.12
Enter "help" to get some basic usage information
sbp2:Open ->login?
speed=ffffffff 2 2 boot:
I tried all of the provided images, with different options - no success. A
common option to be used because of the serial port is "console=ttyS0,57600".
All I got was:
boot: expert-power4
Please wait, loading kernel...
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 Elf32 kernel loaded...
copying OF device tree...done
starting cpu /cpus/PowerPC,G5...failed: 00000000
Calling quiesce ...
erasing fff06000 of Micron B1 part
flashing fff06000 of Micron B1 part
swapping blocks
DO-QUIESCE finishedreturning 0x01400000 from prom_init
Playing with the Gentoo live cd images didn't bring me any further at all.
I then tried to compile a current 32bit ppc 2.6.8-rc2 kernel by hand (for G5
CPU's). Putting this kernel on the debian installer ISO didn't get me any
further. So apparently either the serial port is not working, or the kernel
crashes somewhere.
Using a cross-compiler running on my dual G4 PowerMac, I compiled the same
2.6.8-rc2 kernel for ppc64 target platform. Putting this on the debian boot cd helped a lot, I now got it as far as:
boot: expert-g5-64 console=ttyS0,57600
Please wait, loading kernel...
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2
sbp2:Open ->login?
speed=ffffffff 2 2 Elf64 kernel loaded...
Looking for displays
OF stdout is : /ht@0,f2000000/pci@3/mac-io@7/escc@13000/ch-a@13020
Opening displays...
Calling quiesce ...
DO-QUIESCE finishedreturning from prom_init
Found U3 memory controller & host bridge, revision: 53
Mapped at 0xe000000080000000
Found a K2 mac-io controller, rev: 96, mapped at 0xe000000080041000
PowerMac motherboard: XServe G5
Starting Linux PPC64 2.6.8-rc1
-----------------------------------------------------
naca = 0xc000000000004000
naca->pftSize = 0x17
naca->debug_switch = 0x0
naca->interrupt_controller = 0x1
systemcfg = 0xc000000000005000
systemcfg->processorCount = 0x2
systemcfg->physicalMemorySize = 0x20000000
systemcfg->dCacheL1LineSize = 0x80
systemcfg->iCacheL1LineSize = 0x80
htab_data.htab = 0xc00000001f800000
htab_data.num_ptegs = 0x10000
-----------------------------------------------------
[boot]0100 MM Init
[boot]0100 MM Init Done
idle = native_idle
Linux version 2.6.8-rc1 (laforge@dathomir) (gcc version 3.4.1) #4 SMP Sat Jul 31 16:12:42 CEST 2004
[boot]0012 Setup Arch
via-pmu: Server Mode is disabled
PMU driver 2 initialized for Core99, firmware: 0c
nvram: Checking bank 0...
nvram: gen0=204, gen1=205
nvram: Active bank is: 1
Adding PCI host bridge /pci@0,f0000000
Found U3-AGP PCI host bridge. Firmware bus number: 240->255
Adding PCI host bridge /ht@0,f2000000
Can't get bus-range for /ht@0,f2000000, assume bus 0
U3/HT: hole, 0 end at 9fffffff, 1 start at b0000000
Found U3-HT PCI host bridge. Firmware bus number: 0->239
Can't get bus-range for /ht@0,f2000000
PCI Host 0, io start: fffffffffd800000; io end: fffffffffdffffff
PCI Host 1, io start: 0; io end: 3fffff
Top of RAM: 0x20000000, Total RAM: 0x20000000
Memory hole size: 0MB
On node 0 totalpages: 131072
DMA zone: 131072 pages, LIFO batch:16
Normal zone: 0 pages, LIFO batch:1
HighMem zone: 0 pages, LIFO batch:1
[boot]0015 Setup Done
Built 1 zonelists
Kernel command line: ro debconf_priority=low devfs=mount,dall init=/linuxrc console=ttyS0,57600
PowerMac using OpenPIC irq controller at 0x80040000
[boot]0020 OpenPic Init
OpenPIC Version 1.2 (4 CPUs and 120 IRQ sources) at e000000082ccd000
OpenPIC timer frequency is 25.000000 MHz
[boot]0021 OpenPic Timer
[boot]0022 OpenPic IPI
[boot]0023 OpenPic Ext
[boot]0024 OpenPic Spurious
[boot]0025 OpenPic Done
Slave OpenPIC at 0xf8040000 hooked on IRQ 56
[boot]0020 OpenPic U3 Init
OpenPIC (U3) Version 1.2
[boot]0025 OpenPic2 Done
PID hash table entries: 16 (order 4: 256 bytes)
time_init: decrementer frequency = 33.333333 MHz
Console: colour dummy device 80x25
Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes)
Inode-cache hash table entries: 65536 (order: 7, 524288 bytes)
Memory: 498688k available (3840k kernel code, 4120k data, 212k init) [c000000000000000,c000000020000000]
Calibrating delay loop... 66.56 BogoMIPS
Mount-cache hash table entries: 256 (order: 0, 4096 bytes)
PowerMac SMP probe found 2 cpus
Processor 1 found.
Synchronizing timebase
Got ack
score 299, offset 1000
score 299, offset 500
score 299, offset 250
score 299, offset 125
score 299, offset 62
score 299, offset 31
score 239, offset 15
score -107, offset 7
score 101, offset 11
score -5, offset 9
score 63, offset 10
score -51, offset 9
Min 9 (score 5), Max 10 (score 87)
Final offset: 9 (61/300)
Brought up 2 CPUs
NET: Registered protocol family 16
PCI: Probing PCI hardware
U3-DART: table not allocated, using direct DMA
PCI: Probing PCI hardware done
PCI: no pci dn found for dev=0001:04:0f.0 Apple Computer Inc. K2 GMAC (Sun GEM)
PCI: no pci dn found for dev=0001:05:0c.1 PCI device 1166:0240 (ServerWorks)
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
nvram_init: Could not find nvram partition for nvram buffered error logging.
rtasd: no RTAS on system
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
devfs: 2004-01-31 Richard Gooch (rgooch@atnf.csiro.au)
devfs: boot_options: 0x1
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Initializing Cryptographic API
pmac_zilog: 0.6 (Benjamin Herrenschmidt )
ttyS0 at MMIO 0x80013020 (irq = 22) is a Z85c30 ESCC - Serial port
ttyS1 at MMIO 0x80013000 (irq = 23) is a Z85c30 ESCC - Serial port
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: loaded (max 8 devices)
sungem.c:v0.98 8/24/03 David S. Miller (davem@redhat.com)
So apparently, there were some issues finding the OpenFirmware dn
(distinguished name) for the Ethernet Chips and the ServerWorks chips. I tried
to put some printk's into the arch/ppc64/pci_dn.c file to see what's going on.
This then led me to the earlier error messages about the U3-DART. After
reading some more code, it appeared like the DART is Apple's IOMMU, and it is
supposed to be needed only when running with >2GB RAM. My box had 512MB, but I tried to force usage of the DART by putting "iommu=force" into the kernel commandline.
Great, this was apparently the problem, since now I got up to the point where
it wanted to mount the root filesystem. I thought I didn't really need an
initrd, since the kernel contained all drivers statically linked in. However, Debian installer seems to be running inside initrd only.
First try was just using one of the pre-supplied initrd.gz images. Yes, they
have the wrong versions of the modules - but I don't want/need those modules
anyway.
Of course this wouldn't work either:
RAMDISK: Compressed image found at block 0
Kernel panic: VFS: Unable to mount root fs on unknown-block(0,0)
<0>Rebooting in 180 seconds..
No errror message, nothing. So I thought the problem is with devfs, and I
tried passing several different root parameters ('root=/dev/ram',
'root=/dev/rd/0') without any success.
In the end I found out that the structure sizes of the cramfs superblock
(include/linux/cram_fs_sb.h) are arch-dependent, so I cannot use an initrd that
was built on a ppc32 machine. Unfortunately it is also endian-dependent, and
at this time I only have 32bit big endian and 64bit little endian boxes at
home.
Next step was to use an ext2 initrd, since reasonable filesystems don't have
any strange host/byteorder/wordsize dependencies.
Now it is able to load the initrd, and mount it... although then some other stuff goes terribly wrong. No time yet to investigate this.
[ /linux |
permanent link ]
Putting multiple SATA drives into a XServe ClusterNode G5
Apple is selling two different models of their Dual G5 XServe: One 'Normal'
model, and another 'ClusterNode' Model. They are pretty much the same, but the
ClusterNode doesn't have things you usually don't need in a rack-mounted 1U
server anyway: CD-ROM and VGA-Card. However, it is also limited to a single
hard drive.
I guess Apple's reason is that in a scientific cluster computing environment,
the node's local storage is insignificant - whereas on a real server you most
likely want multiple (mirrored) drives.
However, the significant price difference (Dual G5 ClusterNode has the same
price as the Single G5 XServe) made me ponder buying a ClusterNode and adding another drive.
Fortunately, the hardware is quite similar. It turns out that the Mainboard
has three SATA connectors, and the space for the 2nd and 3rd IDE drive was left
empty. Also, the Backplane for Apples hotplug drives is not fully assembled - it is missing the connectors for the 2nd and 3rd drive :(
So Putting the drive in place and attaching it via a fixed cable to the SATA
connector is no problem at all. However, Power is a slight problem. The whole
machine has not a single standard power connector, so my only remaining option
was to solder some wires onto the drive backplane PCB. This is ugly, but
well.. who cares ;)
I'll put some photos of the modification online soon.
[ /linux |
permanent link ]
David Miller survived my 13-patch patch-bomb
This is good news, DaveM accepted all the 13 netfilter related patches that I
had pending for 2.6.9. The patches included a number of optimizations, the
ctstat, connection-based accounting, TCP window tracking, and some conversions
to new in-kernel-API (seq_file, module_param).
Now let's hope that 2.6.8 will be released soon and we can start the 2.6.9 cycle...
[ /linux/netfilter |
permanent link ]
OLS2004 is over
After holding a BOF on GPL-Violations, and the traditional netfilter/iptables
BOF, OLS ended with Andrew Morton's Keynote.
Obviously, there also was the traditional OLS Social Event at the Black Thorn
Pub, which I left quite early in order to get some more work done on the ulogd2
flow accounting work.
[ /linux/conferences |
permanent link ]
Final court opinion on Sitecom Appeal released
The court handling the Sitecom appeals case has now released it's final
opinion. For those of you who happen to understand legal German, the 20 page document is available as PDF. An English translation will be available soon.
[ /linux/gpl-violations |
permanent link ]
Merging 2.6.8-rc2 changes into patch-o-matic ng
I just started the boring job of merging 2.6.8-rc2 with patch-o-matic-ng... I'm
happy that Jozsef, Martin and Patrick did this for the last couple of kernel
releases. However, I need to get more into this job again in order to
determine which patches still have to be submitted to the mainline kernel...
Expect some pom-ng breakage over the next couple of days...
[ /linux/netfilter |
permanent link ]
IPFIX / ulog integration
After some more in-depth study of the IPFIX IETF drafts, I finally started
coding. Having written the first dozens of lines, I discovered that on an
abstract layer IPFIX doesn't do something too different from my good old ulogd.
Ignoring the minor difference that ulogd deals with individual packets and
IPFIX with flows, the ulogd_iret_t structure is very similar to what IPFIX
templates are trying to describe.
So I now forked a ulogd2 branch off the current ulogd subversion tree and
started to reorganize the tree.
For more flexibility, I am going for a stackable plugin infrastructure, where
the sysadmin can configure stacks like: ULOG->ulogd_BASE->flow
aggregation->IPFIX-over-TCP-export or ctnetlink->IPFIX-over-SMTP-export.
[ /linux/netfilter |
permanent link ]
Group Photo of the Kernel Summit
At http://gnumonks.org/static/photos/ks2004/ are the group photos of this year's Kernel Summit. You obviously won't find me on those pictures, since I was behind the camera ;)
[ /linux/conferences |
permanent link ]
First day of OLS
OLS started today (well, it started with the official beer-drinking BOF
yesterday night). Like at the kernel summit, there are massive problems with
the wireless network, forcing me to operate in offline mode most of the time.
The presenters are apparently all running in slow motion, so I can allocate a
small time-slice to listen to them and spend most of the time working on some
code (conntrack-accounting/ipfix, qsearch, browsing through Rusty's patches). OLS thus starts more productive than I would have thought ;)
Had lunch with Daniel Phillips, who is now working on clustering infrastructure
at RedHat. We detected a general shift from the 'everything is a filesystem' to 'everything is a socket' mentality.
[ /linux/conferences |
permanent link ]
Working towards IPFIX based on conntrack
I've written a patch to add 64bit packet and byte counters for both directions
of every ip_conntrack. This should enable a clean and efficient implementation
of flow based accounting, when combined with ctnetlink events and a userspace
daemon picking up those events.
I need to study the IPFIX (IETF Working Group) specifications in more detail before writing the respective daemon...
The patch is apparently working, you can read the counters via
/proc/net/ip_conntrack and also use a modified/extended/updated version of the
'connbytes' match.
[ /linux/netfilter |
permanent link ]
Day one of the Kernel Summit
So this was day one of the famous kernel summit. Apart from meeting lots of
friends, this basically meant lots of in-depth technical discussions on various
subjects.
Most noticeable were long discussions about the deficiencies of the power
management API, problems with 3-level-page tables on AMD64, and last but not
least: The first-hand technical information from AMD, Intel and IBM on their
upcoming CPU generations.
My personal favourite (AMD) will be shipping dual core (not hyper-threading, but
two real cores) CPU's by mid 2005. They share the same Hyper-transport and
Memory interface, and therefore have to divide I/O Bandwidth between them.
Also had some interesting discussions with Jamal about netfilter performance
and the future l3 generalized connection tracking (called nf_conntrack). Maybe
I can talk him into attending the netfilter workshop for further discussion of
his ideas.
[ /linux/conferences |
permanent link ]
Pattern-matching API in the 2.6.x Kernel
There are various places in the kernel where we need to do some kind of pattern
matching on the packet contents. Applications range from connection tracking
helpers (looking for FTP PORT command, ...) over the 'string' match to
intrusion detection systems.
Two years ago, Phillip |