Update on the netfilter work

Ok, we've seen a terrible amount of bug-fixes going into the net-2.6.14 tree after my new nfnetlink/nfnetlink_log/nfnetlink_queue/... stuff was merged. It is my belief that we've now covered most of it.

As of now, I'm not planning to make any other big netfilter-related patch submissions. So nf_conntrack will probably have to wait for 2.6.15, especially since there are still a number of ip_conntrack/nf_conntrack compatibility issues to be resolved.

Lately I've been working on the userspace side. At least libnfnetlink_log and the libipulog compat API are finished now. libnfnetlink_queue is getting there, and the 'big' missing part is the libipq compat API.

So now I'm heading for some work on ulogd2, libnfnetlink_conntrack and the virtual Ethernet device (vdev) code. And if I still have some time left, there's exciting non-netfilter stuff like my RFID stack.

I'll be in Bangalore again :)

Well, according to the organizers it's just a formality, but "just for the record", I've now officially been invited to the-conference-formerly-known-as-Linux-Bangalore. It will happen Nov 29 to Dec 02, but due to timing overlap, I'll probably only be there from the 30th onwards.

I've already tried to raise awareness for this fabulous event with almost everybody I met during my vivid conference travel. Let's hope I have managed to convince a number of high-quality Linux hackers to consider submitting a paper (and let's hope the CfP will be published really soon now).

nf_conntrack now merged into local branch of netfilter-2.6.14.git

I've committed the last version of nf_conntrack, the layer-3-independent connection tracking code to my netfilter-2.6.14.git tree. It's a local branch called "nf_conntrack".

Yasuyuki and me have been working to port the latest mainline ip_conntrack changes to nf_conntrack. Now the tree should now be fully in sync with ip_conntrack of the same net-2.6.14 tree (this means that it supports CONNTRACK_ACCT and has it's own conntrack-event-api).

Major pieces that are missing from nf_conntrack are:

  • IPv4 NAT for nf_conntrack
  • nf_conntrack_netlink (aka ctnetlink for nf_conntrack)
  • support for ip(6)tables 'state', 'conntrack' and other matches
  • Finally, ct_sync

Visiting parents and friends in Nuernberg

This week I'll be visiting parents and friends in Nuernberg. I'm telling you that because this implicitly means that I'll most likely not be able to continue the pace of netfilter development like in the last couple of weeks.

It also means that I'll probably be doing some scheduled maintenance of the netfilter.org boxes (which are located in Nuernberg, too). So don't be surprised by some shortly-announced downtime. If you're curious what I'm planning: ganesha needs a RAM upgrade (512MB->1GB), and lakshmi needs an upgrade to Debian sarge. Maybe I'll also have time to work on the fail over solution, too.

I expect to read my mails daily, so there shouldn't be any delay in that.

Merging the PPTP helper to net-2.6.14

After having finished my work on the nfnetlink based subsystems, I've progressed to making the PPTP helper fit for mainline inclusion in 2.6.14.

First, it needed an update towards the 2.6.13 conntrack helper API changes (now that expect's have refcounts). Second, we don't have lockhelp.h anymore, and third I want to fall-back to ip_conntrack_proto_generic in case GRE version1 (RCF1701) packets are seen. Stay tuned.

nfnetlink_log submitted

I've submitted my nfnetlink_log patches to DaveM earlier today. So what is this about? It's a replacement for ipt_LOG, ip6t_LOG, ebt_ulog, ipt_ULOG. It introduces a layer-3 (AF_xxx) independent way of logging packets via a userspace logging process.

Again, one step towards code unification. One new piece of code that replaces four existing ones (of similar size), and obsoletes the need for any other such mechanisms that might have appeared for other protocols later on.

If you want to see how to use it from your favourite userspace app, please refer to libnfnetlink_log.

public netfilter-2.6.14 git tree

I've made public my netfilter-2.6.14 tree (based on DaveM's net-2.6.14 tree) at http://people.netfilter.org/laforge/scm/netfilter-2.6.14.git, also available via rsync://people.netfilter.org/users/laforge/scm/netfilter-2.6.14.git

Since this is the first time I'm making a public git tree available, please contact me in case you have any problems accessing it.

I still need to find out how to produce incremental git trees like the ipw2200 project does - this way I would not have to provide a full kernel tree, but only those changes that I do in the netfilter part of it.

iptables-1.3.3 is released

Today I've released iptables-1.3.3. Among some minor fixes (such as for the extremely important feature to SNAT and DNAT to/from ICMP ID _ranges_), it contains one major fix for an embarrassing use-after-free problem that was only introduced with 1.3.2. What do we learn from this? I need to review patches more carefully.

It also includes the NFQUEUE target, which is basically an extension to QUEUE. QUEUE only supports one queue number (0), so there can only be one userspace process be attached to it. This lead to the ugly hack of ipqmpd, the IP QUEUE multiplex daemon. Combining NFQUEUE with nfnetlink_queue (which is already in DaveM's net-2.6.14 tree), you can now have 65535 different queues, each heading to a separate userspace process. This is again one step ahead towards supporting "100% userspace conntrack helpers" which are sort of a strange hybrid variant of transparent proxies.

Data Retention is No Solution

EDRi and XS4ALL have started an online petition against the recent European Commission proposal on mandatory 12 month data retention of all telecommunications meta-data.

Much like the software patent issue, we again have a situation where the European Parliament (those who are directly elected by the public) is against the proposal, while the commission and some national governments are pushing it.

With your support (and at least your signature), there are chances that this data retention directive - like the proposed software patent directive - can be turned down. Please take your time and sign, thanks.

Please also consider supporting the EDRi. They recently announced that they're short of funding.

Intel releases Development manual for e1000 chips

Finally, within years, at least one hardware vendor does The Right Thing (TM): Intel releases hardware documentation about their Gigabit Ethernet Controller chips (known as 'e1000') in the Linux world. (For the curious ones: you can get it from the e1000 sourceforge page)

Even more surprising, they are doing it _despite_ providing a high-quality GPL licensed Linux driver. And by doing this, they show that they have understood that the many developers who are playing with their chip will in the end help them to perform even better, but only if they can actually read the hardware documentation.

There's a group of Linux networking developers who are constantly trying to optimize the driver and come up with new strategies on how to deal with high packet rates.. And at least until now, all the big current Gigabit Ethernet chips did not come with any kind of documentation.

Broadcom tg3 and Syskonnect/Marvell Yukon2 now have a severe competitive disadvantage. Let's see whether they get the clue, and release documentation, too.

I'm not a big fan of Intel, but what they're doing with regard to Linux and their e1000 and ipw2xxx chips is really good. Thanks, Intel!

Chaosradio on Electronic Health Card

Today I'll be moderating this months' episode of Chaosradio on the upcoming German Gesundheitskarte (Electronic Health Card, EHC).

This is the latest incarnation of the ever-increasing number of large-scale IT projects in public atministration. Following-up infamous examples such as TollCollect, the ALG2 software, INPOL-NEU, ELSTER, and last but not least the RFID enabled electronic Passport. And it will affect the data privacy and data protection of even more German citizens than any of the beforementioned systems!

I'm very pleased to announce Thomas Maus (ThoMaus), one (if not the) most prominent critical experts on the EHC as a live guest in the radio studio.

This subject is actually one that I think fits best into the idea of Chaosradio: Technical, but with vast implications on society. Even more than my last "favourite" data retention, but less than the upcoming Chaosradio show on "voting machines".

From my point of view there are too many issues currently at this border between technology, politics and society that need to be adressed. Too many to just talk about geeky technological stuff that is certainly also happening and woth covering it in Chaosradio.

Back home in Berlin

After one day for travel and sleeping-over-the-jetlag, I'm finally back on track at my home in Berlin.

I just decided to skip WTH, since it would require me to leave again in only two days (and I have another travel coming up on 1st August. So I'd rather spend the time to continue my current netfilter projects, taking care of accounting and tax declaration, etc.

Unfortunately I'm bound to using slower/older machines and my notebook, since the warranty replacement for my workstations' liquid cooling system has not yet arrived :(

RMS visits ASUS: Free Software beyond their notice ?!?

In his blog, Richard Stallman writes that he had a very unpleasant experience visiting ASUS in Taiwan.

This is outrageous, considering they are using Linux and other free software programs in their products and making business from it.

Their WL500g routers are using Linux, and did not comply with the GPL. So in 2004, I used my copyright to enforce the license. I have obtained a declaration to cease and desist from ASUS Headquarters in Taiwan, and they modified their product promptly to bring it into GPL compliance. See this news item on the netfilter.org project homepage.

Even today, ASUS seems to be using Free Software in a number of their latest devices, as I indicated in this blog entry.

Revamping netlink sockets

While writing on nfnetlink, ctnetlink, nfnetlink_queue and other bits of the 'new' netfilter infrastructure, I've run into a number of minor shortcomings in netlink that are surprisingly hard to overcome.

One of them is refcounting, i.e. making sure that the module implementing a particular functionality via netlink doesn't silently disappear by module unloading while sockets are still open from userspace.

I've now finished one implementation, but it might cause module refcount leaks if a kernel module implementing a netlink socket closes the socket in some other codepath but the module_exit() function.

The other problem (slightly harder) is module auto-loading. It's my position that the kernel should autoload the respective module once a userspace process opens a netlink socket. However, this can not be made obligatory, since multiple userspace processes might also just wish to communicate with themselves, with no listener/sender in the kernel at all.

OLS: Wireless Kernel Configuration BOF

James Ketrenos (the ipw2xxx maintainer) was running a BOF to get input on ideas for a new wireless kernel configuration API from the Linux community.

Due to excessive coding (see in some different entry of this journal), Patrick and me came in a bit late. We tried to convince the audience that netlink was the way to go, and that the current ioctl() interface could be served by some compatibility layer that converts the ioctl's to netlink messages.

Also, I raised the requirement for integrating this config interface with a unified userspace interface for association and authentication (i.e. management frames).

Unfortunately James had to leave quite early, so we couldn't finish the discussion in a more detailed way in a smaller group.

OLS: netfilter hacking with Patrick

Patrick McHardy and me sat together for a number of nights, reading and discussing various current issues with the networking code. It's surprising how much fallout we get from these discussions.

Apart from tons of new code (nfnetlink, ctnetlink, nfnetlink_queue, ...) there are apparently still quite a number of interesting bugs in esp. the NAT code that have been there for 5+ years without anybody noticing them.

What comes immediately to my mind is Rusty's famous quote "When we do something wrong, the users just hit reload. Nobody will notice, you never get bug reports". Especially when the NAT or conntrack code are doing something wrong that doesn't disrupt the protocol, it's relatively difficult to find those bugs.

So what did we find? For example, that ICMP ID NAT [yes, we do support that] had a number of endianness bugs. So when you wanted it to NAT ICMP ID's to a particular range [instead of any free ID], it would use totally different numbers that the administrator or the helper plugin actually specified - but only on little endian machines.

Some other bug was more severe, since it can theoretically cause memory corruption [a stale pointer could have been used since it was accidentally added to a list of 'static' variable declaration].

The IEEE and their policy on publication of standards.

The IEEE is a standardization body. Being a Linux network developer, access to their 802.x standards is sometimes quite valuable. A couple of years ago they introduced the "Get 802" program, where they would make available the 802 standards family some time after publication. This is great.

However, I recently needed a copy of the current draft of the 802.11e standard. They charge USD60 for this, which is a reasonable fee that I was willing to pay.

However, they only seem to be offering in some proprietary DRM format. This is totally unacceptable, since it would requires installation of the purchase and installation a proprietary operating system.

Networks (and especially the Internet) are built upon open and publicly available standards. Free and Open Source projects can only implement industry standards if they can actually access those standards. The availability of such standards is therefore an important aspect of their fast implementation and adoption.

I very much understand the requirement of standards organizations to charge reasonable fees (such as USD60 for the 802.11E draft) for purchasing copies of it.

However, after obtaining such a copy, I would like to print it or pages of it, I would like to view it on all of my computers, and I wan to do so while staying offline without any authentication that (I suppose) your DRM system requires.

By putting such incredible obstacles between the developers and the standardization body, they will achieve nothing but frustration and hamper the adoption of the standards which they care about.

OLS Day 1

I didn't actually visit any of the talks, but instead read some of the papers in the written proceedings, hacking lots of code and talking to various people.

I've also managed to convince GregKH that support for async URB submission from userspace needs CONFIG_BROKEN. libusb doesn't use it anyway, and the number of users of this interface is limited. Unfortunately one of my customers is one of the users, so I might be forced to implement a cleaner interface for the same purpose.

Lots of netfilter hacking over the last couple of days

Following-up meeting the other networking hackers at netconf, I got really extremely motivated and basically spent every single minute hacking code.

The projects include:

  • skb shrinkage (already merged in DaveM's net-2.6.14 tree)
  • nfnetlink (already merged in DaveM's net-2.6.14 tree)
  • conntrack event notifiers (already merged in DaveM's net-2.6.14 tree)
  • ctnetlink (reworked to use network byte order in all the payload)
  • nfnetlink_queue (a nfnetlink-based queue implementation)
  • vdev (a virtual device that allows you to use multiple mac addresses on one Ethernet device)
  • mmio_test (include support for machine-parseable reporting)

First day of netconf

The first day of netconf went quite fine, but we basically lost quite some amount of time waiting. First waiting for free tables at breakfast, then waiting for the bloated enrollment procedures of the Security Guards at the Ericsson venue...

Added with technical issues with the 800x600-only projector and the amount of time spent travelling from the hotel to the venue, we lost a lot of time and therefore actually didn't have the time to fit all talks into their respective slot, but only 60%.

The most cool work I've seen at this first day is Thomas Graf's work on a unified Linux kernel networking configuration and statistics tool...

Heading off to netconf in Montreal

Later today I'll be heading off towards Montreal for netconf 2005. I'm really looking forward to that event and the interesting discussions with my fellow Linux networking developers.

I'm actually meeting Patrick McHardy in Paris, as we'll be on the same transatlantic flight. I hope we can get some of the pending netfilter/iptables issues discussion meanwhile ;)

After netconf, most of us are heading to Ottawa for Kernel Summit and OLS. I've turned down the invitation to the kernel summit, since usually there is nothing on the agenda that even remotely touches the packet filter or even the core network stack, so I'd rather make space for somebody else.

I'm supposed to have network connectivity almost all the time, so I don't expect big delays in email responses.

Almost all vendors of console servers GPL incompliant

According to this German article (by Dr. Dirk Wetter), out of seven tested console servers (all Linux-based) of various vendors, only two even mentioned that GPL licensed software was used in the product. The majority of the devices did neither mention the GPL, nor make any source code offer.

The vendors have been contacted by the author of the article, and almost all promised to make their devices GPL compliant in the future. It has yet to be seen whether they actually fulfill that promise. I will ask each of them for a copy of the full corresponding source code, since the offer implicitly has to exist [the devices didn't ship with the source code, so 3a GPL is no longer possible].

It's really disappointing to see this happen again and again. Everybody seems to not care at all about the copyright of the code involved.

ASUS has a whole line of new gpl violating devices

Apparently, the AAM6020VI, AAM6020BI, AAM6030VI and AAM5030BI devices all contain Linux (including netfilter/iptables) -based firmware images, but no source code is made available.

None of the devices is sold here in Germany, so I can't go after ASUS Germany.

Estampie - Marco Polo (Live DVD)

Estampie is definitely one of my very favourite music bands ever. For the majority of my readers: They do serious medieval music. "serious" meaning they are doing this at the level of profession that you expect from classical musicians. Estampie is doing this for some 20 years, and they're not to be confused with the Spielmannsmusik that you recently find at any of the tourist-laden medieval festival.

At one of those dates when I was travelling to yet-another Free Software related conference, they played a programme called Marco Polo - Music of the Silk Route. Basically they tried to go beyond European medieval music and build bridges to other musical traditions of the same time, such as Khorasan Dotar music from Iran, traditional Mongolian music and some Indian Percussion.

They recently released a Live recording DVD from that project, and I am totally in love with the blend of music they have created. What they have created is "real" world music to me.

And there is more to come. As Michael Popp (the leader of the ensemble) points out in the interview section, "Marco Polo" was just the beginning of a trilogy. I'll definitely make sure that my travel schedule will adjust to the dates of the second and third part of the trilogy. There's no way I'll miss them.