LaForge's home page (Posts about netfilter)https://laforge.gnumonks.org/blog/tags/netfilter.atom2022-06-21T07:49:56ZHarald WelteNikolaInvited keynote + TTCN-3 talk at netdevconf 2.2 in Seoulhttps://laforge.gnumonks.org/blog/20171010-netdevconf/2017-10-10T00:00:00+02:002017-10-10T00:00:00+02:00Harald Welte<p>It was a big surprise that I've recently been invited to give a
<a class="reference external" href="https://www.netdevconf.org/2.2/session.html?welte-netfilterhistory-keynote">keynote on netfilter history at netdevconf 2.2</a>.</p>
<p>First of all, I wouldn't have expected netfilter to be <em>that</em> relevant
next to all the other [core] networking topics at netdevconf. Secondly,
I've not been doing any work on netfilter for about a decade now, so my
memory is a bit <em>rusty</em> by now ;)</p>
<p>Speaking of <em>Rusty</em>: Timing wise there is apparently a nice coincidence that
I'll be able to meet up with him in Berlin later this month, i.e. hopefully
we can spend some time reminiscing about old times and see what kind of useful
input he has for the keynote.</p>
<p>I'm also asking my former colleagues and successors in the netfilter
project to share with me any note-worthy events or anecdotes,
particularly also covering the time after my retirement from the core
team. So if you have something that you believe shouldn't miss in a
keynote on netfilter project history: Please reach out to me by e-mail
ASAP and let me know about it.</p>
<p>To try to fend off the <em>elder[ly] statesmen</em> image that goes along with
being invited to give keynotes about the history of projects you were
working on a long time ago, I also submitted an actual technical talk:
<a class="reference external" href="https://www.netdevconf.org/2.2/session.html?welte-ttcn3-talk">TTCN-3 and Eclipse Titan for testing protocol stacks</a>, in
which I'll cover my recent journey into TTCN-3 and TITAN land, and how I
think those tools can help us in the Linux [kernel] networking
community to productively produce tests for the various protocols.</p>
<p>As usual for netdevconf, there are plenty of other exciting talks in
<a class="reference external" href="https://www.netdevconf.org/2.2/schedule.html">the schedule</a></p>
<p>I'm very much looking forward to both visiting Seoul again, as well as
meeting lots of the excellent people involved in the Linux networking
subsystems. See ya!</p>Back from netdevconf 1.1 in Sevillehttps://laforge.gnumonks.org/blog/20160215-netdevconf/2016-02-15T00:00:00+01:002016-02-15T00:00:00+01:00Harald Welte<p>I've had the pleasure of being invited to <a class="reference external" href="http://www.netdevconf.org/1.1/">netdevconf 1.1</a> in Seville, spain.</p>
<p>After about a decade of absence in the Linux kernel networking
community, it was great to meet lots of former colleagues again, as well
as to see what kind of topics are currently being worked on and under
discussion.</p>
<p>The conference had a really nice spirit to it. I like the fact that it
is run by the community itself. Organized by respected members of the
community. It feels like Linux-Kongress or OLS or UKUUG or many others
felt in the past. There's just something that got lost when the Linux
Foundation took over (or pushed aside) virtually any other Linux kernel
related event on the planet in the past :/ So thanks to Jamal for
starting netdevconf, and thanks to Pablo and his team for running this
particular instance of it.</p>
<p>I never really wanted to leave netfilter and the Linux kernel network
stack behind - but then my problem appears to be that there are simply
way too many things of interest to me, and I had to venture first into
RFID (OpenPCD, OpenPICC), then into smartphone hardware and software
(Openmoko) and finally embark on a journey of applied telecoms
archeology by starting OpenBSC, OsmocomBB and various other Osmocom
projects.</p>
<p>Staying in Linux kernel networking land was simply not an option with a
scope that can only be defined as wide as wanting to implement <em>any
possible protocol on any possible interface of any possible generation
of cellular network</em>.</p>
<p>At times like attending netdevconf I wonder if I made the right choice
back then. Linux kernel networking is a lot of fun and hard challenges,
too - and it is definitely an area that's much more used by many more
organizations and individuals: The code I wrote on netfilter/iptables
is probably running on billions of devices by now. Compare that to the
Osmocom code, which is probably running on a few thousands of devices,
if at all. Working on Open Source telecom protocols is sometimes a
lonely fight. Not that I wouldn't value the entire team of developers
involved in it. to the contrary. But lonely in the context that 99.999%
of that world is a proprietary world, and FOSS cellular infrastructure
is just the 0.001% at the margin of all of that.</p>
<p>One the Linux kernel side, you have virtually every IT company putting in
their weight these days, and properly funded development is not that
hard to come by. In cellular, reasonable funding for anything (compared
to the scope and complexity of the tasks) is rather the exception than
the norm.</p>
<p>But no, I don't have any regrets. It has been an interesting journey and
I probably had the chance to learn many more things than if I had stayed
in TCP/IP-land.</p>
<p>If only each day had 48 hours and I could work both on Osmocom and on
the Linux kernel...</p>Another patch submit day.https://laforge.gnumonks.org/blog/20140221-patchsubmit/2014-02-21T03:00:00+01:002014-02-21T03:00:00+01:00Harald Welte<p>
Today I've submitted hashlimit, CLUSTERIP and CONNMARK to the 2.6.x kernel.
After resolving some glitches with CLUSTERIP, DaveM took all three :)
</p>
<p>
This means we're again one step further submitting stuff from patch-o-matic into mainline, which is always a good thing.
</p>Deutsche Telekom tried to register a trademark on netfilterhttps://laforge.gnumonks.org/blog/20110404-deutsche_telekom-netfilter_trademark/2011-04-04T03:00:00+02:002011-04-04T03:00:00+02:00Harald Welte<p>
I am currently doing some trademark related research, and just for fun I
queried the database of the DPMA (German trademark and patent office) for
"netfilter".
</p>
<p>
To my big surprise, you can find <a href="http://register.dpma.de/DPMAregister/marke/register/306442639/DE">this
record</a>, indicating that Deutsche Telekom AG has applied for a trademark
on the word "NetFilter" in July 2006.
</p>
<p>
I find that quite outrageous, as the netfilter project is using the name since
about 1999, i.e. 7 years earlier. To our luck, the trademark office refused the
application based on the generic nature of the name, i.e. "netfilter" being too
generic for anyone obtaining a trademark on it - at least in Germany, under German
laws.
</p>The 7th netfilter workshop is coming uphttps://laforge.gnumonks.org/blog/20101017-netfilter_workshop-coming_up/2010-10-17T03:00:00+02:002010-10-17T03:00:00+02:00Harald Welte<p>
The <a href="http://workshop.netfilter.org/2010/">7th Netfilter Workshop</a> is
just coming up next week in Seville, Spain. Once again it will be hosted at
the <a href="http://www.informatica.us.es/">ETS Ingeneria Informatica</a> of
the University of Seville.
</p>
<p>
I'd like to personally thank Pablo Neira for organizing and hosting the event
again in Seville.
</p>
<p>
As most readers of this blog will know, my current relationship to
netfilter/iptables is somewhat dormant. I haven't been writing any code for
probably something like five years ago, when I was seriously distracted with
stuff like OpenPCD, OpenPICC, OpenBeacon and later the Openmoko project.
</p>
<p>
Nonetheless, it is always great to learn what Patrick, Pablo, Martin, Jozsef,
Yasuyuki and the others have been up to. With a slight chance I may actually
still have some advice/ideas or other input I can contribute.
</p>The best linux kernel commit message everhttps://laforge.gnumonks.org/blog/20090428-hemminger_shakespeare/2009-04-28T03:00:00+02:002009-04-28T03:00:00+02:00Harald Welte<p>
As <a href="http://patchwork.kernel.org/patch/19739/">you can see at this
patch</a>, Stephen Hemminger has submitted what I would call the best
Linux Kernel commit message ever:
</p><pre>
In days of old in 2.6.29, netfilter did locketh using a
lock of the reader kind when doing its table business, and do
a writer when with pen in hand like a overworked accountant
did replace the tables. This sucketh and caused the single
lock to fly back and forth like a poor errant boy.
But then netfilter was blessed with RCU and the performance
was divine, but alas there were those that suffered for
trying to replace their many rules one at a time.
So now RCU must be vanquished from the scene, and better
chastity belts be placed upon this valuable asset most dear.
The locks that were but one are now replaced by one per suitor.
The repair was made after much discussion involving
Eric the wise, and Linus the foul. With flowers springing
up amid the thorns some peace has finally prevailed and
all is soothed. This patch and purple prose was penned by
in honor of "Talk like Shakespeare" day.
Signed-off-by: Stephen Hemminger <shemminger>
---
What hath changed over the last two setting suns:
* more words, mostly correct...
* no need to locketh for writeh on current cpu tis
always so
* the locking of all cpu's on replace is always done as
part of the get_counters cycle, so the sychronize swip
in replace tables is gone with only a comment remaing
include/linux/netfilter/x_tables.h | 55 ++++++++++++++--
net/ipv4/netfilter/arp_tables.c | 125 ++++++++++--------------------------
net/ipv4/netfilter/ip_tables.c | 126 ++++++++++---------------------------
net/ipv6/netfilter/ip6_tables.c | 123 ++++++++++--------------------------
net/netfilter/x_tables.c | 55 ++++++++--------
5 files changed, 188 insertions(+), 296 deletions(-)
</shemminger></pre>
<p>
Thanks Stephen, you made my day :)
</p>We don't do Advertisement on the netfilter.org homepagehttps://laforge.gnumonks.org/blog/20080411-netfilter_advertising/2008-04-11T03:00:00+02:002008-04-11T03:00:00+02:00Harald Welte<p>
For some reason, the amount of inquiries about companies who want to put ads
on netfilter.org has significantly increased. Since the content of that
site has not really changed much in the last (at least) four years, this
sudden interest is somewhat surprising to me.
</p>
<p>
However, we are absolutely not interested in advertisements. I personally
hate any form of advertisement, whether in print media, radio, TV, WWW or on
billboards. In fact, advertisements are the reason for me to not watch any
privately owned TV or radio stations for at least eight years.
</p>
<p>
So to all the advertising companies out there: Only over my dead body will
there be any kind of banner ads on any of the websites of the projects in which
I have anything to say.
</p>My last netfilter traininghttps://laforge.gnumonks.org/blog/20071108-netfilter_training/2007-11-08T03:00:00+01:002007-11-08T03:00:00+01:00Harald Welte<p>
Since I've been doing no netfilter/iptables related work recently, I've
announced that the <a href="http://www.heinlein-support.de/web/akademie/kurse/iptables-netfilter-firewalls/">three day training</a> is going to be the last one, at least for the time being.
</p>
<p>
Though stressful as usual (have you ever talked/presented straight 8 hours on
three consecutive days?) it was a quite joyful experience. Apart from the
netfilter/iptables workshop earlier this year, the only contact with my former
much-beloved project in 2007.
</p>
<p>
However, the training made me realize how outdated all the existing
documentation (and even my own training material) is. Basically everything was
written in the early 2.4.x days - and much has changed ever since.
</p>
<p>
There's all the nf_conntrack / nf_nat related changes, as well as the x_tables transition, which can cause many subtle errors due to old scripts expecting different kernel module names, etc.
</p>
<p>
None of the HOWTO's or similar documents talk about the conntrack userspace program yet, there's no documentation (and no release) for ulogd2, etc.
</p>
<p>
So I'll really try to sit down and find some time to improve some of those areas. It yet remains to be seen if I can actually make it. But I feel there's a real gap to be filled...
</p>netfilter developer workshop 2007 is overhttps://laforge.gnumonks.org/blog/20070914-coreteam-head/2007-09-14T03:00:00+02:002007-09-14T03:00:00+02:00Harald Welte<p>
The days of the netfilter workshop passed quite fast, and I'm finally back to
my home in Berlin now. In case you're interested, here is a link to the <a href="http://nfws.inl.fr/en/?p=55">group photo</a>.
</p>
<p>
Among other things, we've had the following major decisions at the workshop:
</p><ul>
<li><a href="http://nfws.inl.fr/en/?p=57">Patrick McHardy finally officially head of the coreteam</a></li>
<li>ulogd2 will see an official release candidate soon</li>
<li>we want to merge ipset soon</li>
<li>we will try to shift future developments in the direction of libnl and slowly deprecate libnetfilter_*</li>
<li>we will move the netfilter and netfilter-devel lists to vger.kernel.org since we don't have to care about spam filtering there. Other, lower-traffic lists remain on lists.netfilter.org</li>
<li>we will switch to git even for userspace code, at least for the iptables source code</li>
</ul>
<p>
Finally, I'd like to use this opportunity to thank all our <a href="http://workshop.netfilter.org/2007/sponsors.html">Workshop sponsors</a>,
particularly <a href="http://www.astaro.com/">Astaro</a> for their continuous
and generous support of netfilter/iptables throughout the last five years.
</p>Enjoying the netfilter workshop 2007https://laforge.gnumonks.org/blog/20070911-netfilter-workshop/2007-09-11T03:00:00+02:002007-09-11T03:00:00+02:00Harald Welte<p>
I've returned to Germany in order to attend the <a href="http://workshop.netfilter.org/2007/">5th netfilter development
workshop</a> in Karlsruhe. It's sponsored by Astaro, whose continuing support
of netfilter/iptables is really outstanding. Even after I took my "leave" to
work on <a href="http://www.openmoko.org/">OpenMoko</a>, they continue their
funding by paying for Patricks maintenance of the netfilter/iptables codebase,
and things like hosting the netfilter workshop.
</p>
<p>
It's really great to meet with the old colleagues with whom I've co-worked for
a number of years on netfilter/iptables. I really miss those days, basically
spending most of my day working together and communicating with cool people
hacking on similar problems. Quite a bit different from what I'm doing right now.
</p>
<p>
So while I'm here, I'm actually trying to spend most of my time related to
netfilter/iptables, which is really refreshing.
</p>Getting back into netfilter/iptables workhttps://laforge.gnumonks.org/blog/20070120-getting_back/2007-01-20T03:00:00+01:002007-01-20T03:00:00+01:00Harald Welte<p>
I've been gone for long enough. Even though neither my RFID projects nor
OpenMoko are anywhere close to be finished, I'm determined to get back into
netfilter work again.
</p>
<p>
Started to catch up with mailing lists. There has been amazing progress, most notably
the implementation of NAT for nf_conntrack, which finally should get us rid of the old
ip_conntrack code in one of the upcoming kernel releases. No more support of
two versions in parallel. And the ability to do IPv4 NAT and IPv6 connection tracking
on the same machine. Isn't that all that we wanted? Not quite...
</p>
<p>
So for now, I'm participating in the discussions again, and I'm now also working on
getting IPv6 interpreter plug-ins into ulogd2. The nfnetlink_log mechanism can happily
send IPv6 packets to user space, it's just that ulogd2 doesn't yet know what to
do with them. That needs to be changed.
</p>bugzilla.netfilter.org up+running againhttps://laforge.gnumonks.org/blog/20061222-bugzilla_patchwork/2006-12-22T03:00:00+01:002006-12-22T03:00:00+01:00Harald Welte<p>
Only two months after the involuntary absence of bugzilla.netfilter.org (due
to database corruption while doing a gentoo mysql update), I have finally
found some time (and a way) to fix the problem. Therefore, as of today,
<a href="http://bugzilla.netfilter.org/">bugzilla.netfilter.org</a> is now
up and running again.
</p>
<p>
This was possible due to the fact that the bugzilla tables were still present
in myISAM format. The mysql tables of <a href="http://patchwork.netfilter.org/">patchwork.netfilter.org</a> were not
that lucky. They were stored in exactly that InnoDB file that got corrupted.
However, the loss of archived (and lots of unmaintained) information on patches
that had been submitted on netfilter-devel is not really all that important
anyway.
</p>
<p>
However, let this be a lesson: Do daily dumps of all mysql tables in a cronjob before doing backups ;)
</p>netfilter.org releases (almost), update on my netfilter involvementhttps://laforge.gnumonks.org/blog/20060618-releases/2006-06-18T03:00:00+02:002006-06-18T03:00:00+02:00Harald Welte<p>
It's been terrible to be away from netfilter development for about two months
now. This really has to change, I have to cut down on other stuff if I don't
want to loose track completely.
</p>
<p>
Anyway, I finally did what I wanted to do at least for many weeks: To push new
releases of libnfnetlink, libnetfilter_log, libnetfilter_queue,
libnetfilter_conntrack and conntrack. The files are available from <a href="http://www.netfilter.org/projects/index.html">their usual location</a>.
Haven't been in the mood to write changelogs yet, so if you're really
interested in them, you'll have to wait for a bit more.
</p>
<p>
The main architectural change is that the internal api between libnfnetlink
and libnetfilter_* has changed, e.g. caller-allocated structures are now
callee-allocated. Apart from that, a very important bugfix was made in libnfnetlink,
one that actually affects future-compatibility of the kernel/userspace interface.
</p>
<p>
For anything else, it's mainly a maintenance release.
</p>
<p>
libnetfilter_queue doesn't yet contain the bits required for the 'upcoming'
libnetfilter_cthelper (userspace helpers), because I felt pushing that code
without having the rest of the infrastructure plus some test cases running
isn't really worth it.
</p>
<p>
So please include in your prayers that there are not too many gpl violations
during the next couple of weeks, that I finally get hold of that stupid PPTP problem
that is bugging me for many weeks. If that happens, I think I'll be back to
netfilter stuff early next week after returning from the Barcelona GPLv3 event.
</p>
<p>
Not sure whether I mentioned it already: I'm actually skipping OLS (and kernel
summit) this year in order to gain some time. Meeting folks and attending talks
is a lot of fun, but it also (including the travel overhead, jetlag, drinking, etc.)
eats a lot of time. So I'll actually take my long-announced <b>pkttables
holidays</b> when the rest of the Linux kernel developers are in Ottawa. For those
not familiar with the term: The idea is to 'go on holidays' (i.e. abandon anything
else like reading emails, etc) and stay focused working on netfilter stuff for
at least one week in order to finally see the ideas so far known as pkttables to finally
materialize in one way or the other.
</p>
<p>
Meanwhile, I have to extend my deepest thanks to Patrick McHardy, and all the work he's
been putting into netfilter maintenance over the last year or so.
</p>netfilter.org downtime - moving and updating servershttps://laforge.gnumonks.org/blog/20060404-servers/2006-04-04T03:00:00+02:002006-04-04T03:00:00+02:00Harald Welte<p>
I've spent the whole Monday in the hosting center where netfilter.org,
gnumonks.org and most of my other projects are hosted. The main reasons for
this visit were:
</p><ul>
<li>do kernel updates on two boxes that are known to be difficult with new kernels</li>
<li>move all five machines to a new rack, the old one is too crowded (no space for new machines, too hot)</li>
<li>add yet another new box (parvati.gnumonks.org), which makes the number of machines now six</li>
</ul>
<p>
As usual, Murphy's law applied, so about everything that could go wrong went wrong.
And, confirming Murphy's law, the most important machine (vishnu.netfilter.org)
had the longest downtime, something close to 9 hours.
</p>
<p>
This was mainly due to
the last Gentoo update overriding my custom-modified yaboot boot script (for
using the serial port, this is a headless XServe cluster node) with the default
one, which wants to use the non-existent framebuffer.
</p>
<p>
That combined with the fact that KDUMP-capable kernels can't be booted from
OpenFirmware (why isn't this indicated in the menuconfig help???) and thus the new
default boot kernel couldn't be booted from yaboot.
</p>
<p>
That day I've tried about anything, from attaching a powerbook with bootable cd
in firewire target mode to booting yaboot via tftp (which fails to load
yaboot.conf via tftp *sigh*).
</p>
<p>
Now I've learned my lesson: chattr +i on yaboot.conf and the modified boot
script for serial console.
</p>netfilter do_replace() bug is not remotely exploitablehttps://laforge.gnumonks.org/blog/20060322-bogus_vulnerability/2006-03-22T03:00:00+01:002006-03-22T03:00:00+01:00Harald Welte<p>
I don't know how people like <a href="http://www.securityfocus.com/bid/17178/discuss">securityfocus</a> and <a href="http://www.heise.de/newsticker/meldung/71128">heise.de</a> and others claim
that the recently-discovered and fixed 'do_replace()' bug is remotely exploitable.
</p>
<p>
In fact, the bug (which was found and fixed by Solar Designer while working for
the OpenVZ project) can only happen in a codepath that can be executed by the
local root user. Not even a non-root user, neither any remote parties can hit
that bug and/or exploit anything.
</p>Working on Bug 404https://laforge.gnumonks.org/blog/20060210-bug-404/2006-02-10T03:00:00+01:002006-02-10T03:00:00+01:00Harald Welte<p>
Isn't it a strange coincidence, that a reasonably non-trivial netfilter bug gets the bugzilla ID 404 ?
</p>
<p>
Well, before I try to build some conspiracy theories about somebody manipulating the bug id number sequence generation of our bugzilla installation, I'd rather concentrate on the real work.
</p>
<p>
Dave Remien is an excellent bug reporter, so as a maintainer you can actually
not expect anything more than his detailed <a href="https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=404">documentation</a>
(yes, I know, certificate has expired, too lazy and busy to update it right
now, stay tuned). From an outside perspective, it appears like packets get
'stuck' in nfnetlink_queue. In reality, it seems like the kernel is doing
everything fine, just the library eats some packets from time to time, meaning
that they remain inside the kernel queue and increase it's length (and thus
leak memory) one at a time.
</p>
<p>
The real cause has yet to be discovered, I'm confident that there will be some news tomorrow.
</p>iptables-1.3.5 is outhttps://laforge.gnumonks.org/blog/20060201-iptables-1_3_5/2006-02-01T03:00:00+01:002006-02-01T03:00:00+01:00Harald Welte<p>
I've released <a href="http://netfilter.org/projects/iptables">iptables-1.3.5</a> earlier today. This will probably mark the last 'new feature' release of the iptables-1.3.x branch.
</p>
<p>
I'm still working on the initial beta release of iptables-1.4.x, the userspace
counter part to what is now known in kernel space as 'x_tables'. Stay tuned.
</p>iptables-1.4 branch openedhttps://laforge.gnumonks.org/blog/20060121-iptables14/2006-01-21T03:00:00+01:002006-01-21T03:00:00+01:00Harald Welte<p>
Since we now have the x_tables kernel side code in the upcoming 2.6.16 series,
I'm working on getting iptables-1.4.x done to actually take advantage of
the new kernel's abilities.
</p>
<p>
The main reason why people are interested in this, is to get matches like
'state' and 'conntrack' working for IPv6. Even though 2.6.15 has nf_conntrack
and thus state tracking for IPv6, you cannot really use it from ip6tables yet.
</p>
<p>
The same goes for all native x_tables matches and targets. However, I think
we'll also release a new version of iptables-1.3.x just with 'state' and
'conntrack' support, since it gives a more stable foundation for production
users than a completely new 1.4.x branch with hundreds of kilobytes of patches.
</p>userspace conntrack helper code compiles (yet untested)https://laforge.gnumonks.org/blog/20060115-nfnl_helper/2006-01-15T03:00:00+01:002006-01-15T03:00:00+01:00Harald Welte<p>
Finally, both the kernel side (nfnetlink_helper) and the userspace side
(libnetfilter_cthelper) code for userspace conntrack helper support is
basically finished and compiles. I didn't yet dare to test it, and I'm rather
heading off to bed now. Testing will be done tomorrow.
</p>
<p>
So how is this supposed to work? Well, basically a new nfnetlink subsystem
exists, which can (on behalf of an userspace process) create dummy
"nf_conntrack_helper" structures inside the kernel. Such a dummy structure has
the usual properties (tuple, mask, timeout, etc.) but a dummy expectfn() which
only calls NF_QUEUE() to send the packet to userspace. Userspace can then look
at the packet, possibly modify it and re-inject it back into the kernel. Since
helpers are now processed at a different netfilter hookfn() than the rest of
the conntrack code, this actually works.
</p>
<p>
Now during the reception of such a packet in userspace, the process is likely
going to want to create a new expectations. Expectations can already be
created by means of libnetfilter_conntrack/nf_conntrack_netlink. However, in
order to create the expectation, a number of things are needed. Mainly the
tuple(s) of the master conntrack, but also other ancillary data such as ctinfo
are sometimes desired. As long as we don't do NAT, the process could derive
the tuple from the packet's IP[v6] header, and query nf_conntrack_netlink for
the remaining details. However, this is inefficient since we'd add another
kernel/userspace round-trip and the associated latency. So instead, I chose to
extend nfnetlink_queue a bit, and allow it to have a new queue_mode
(NFQ_MODE_PACKET_CT) in which there is a new nested attribute (NFQA_CT) which in
turn contains the tuple, id and ctinfo.
</p>
<p>
Userspace now has all informations to create a new expectation. But wait, what
do we do about expectfn()? We use the same magic as with helpfn(): Userspace
tells the kernel to which nfnetlink_queue queue_id packets hitting the
expectfn() should be sent. The 'minor' difficulty here is that expectfn() is
called from the middle of the conntrack code (init_conntrack() actually), and
when we get back from the queue (set_verdict or re-inject), then the netfilter
hook code would continue at the next hookfn, skipping most of the conntrack
code. But we can also return NF_REPEAT in order to call conntrack again.
Since our expectation is already confirmed, expectfn() will not be called and
it _SHOULD_ somehow just magically work, maybe with some tiny ugly hack here or
there.
</p>
<p>
The NFQA_CT way is still far from being optimal, since we copy the same
conntrack tuple for every packet of the control connection to userspace, no
matter that this information never changes, and no matter that we actually only
need it in those few cases where we want to raise an expectation. So the
mid-term plan is to make userspace keep a small copy of selected conntrack
state entries. This can be done by sending NEW and DELETE events for all
conntracks that have a helper assigned. We could create a new multicast group
specifically for this purpose, in order to keep the overhead and memory usage
low. Userspace keeps a hash table indexed by ct->id. Packets sent via
nfnetlink_queue will therefore only need a single 32bit ID attribute and not
the full tuple(s).
</p>
<p>
Apart from userspace helper code, I've been working on getting some x_tables /
nf_conntrack refcounting / dependency issues sorted out. Again another issue
where having a couple of dozens of inter-dependant netfilter modules seems to
become a major PITA. Sometimes I want to have back the simplicity of a
truly monolithic kernel.
</p>x_tables merged mainlinehttps://laforge.gnumonks.org/blog/20060114-xtables-included/2006-01-14T03:00:00+01:002006-01-14T03:00:00+01:00Harald Welte<p>
Linus has merged x_tables, even though I introduced some "doesn't build without
IPv6 support" breakage that only somebody not into networking would ever detect
(hey, would you build a kernel without ipv6?) ;)
</p>
<p>
Anyway, will try to be more cautious about these issues, as nobody wants to end
up with a "your patches break the kernel tree" reputation.
</p>ulogd-2.00beta1 releasehttps://laforge.gnumonks.org/blog/20060109-ulogd2beta1/2006-01-09T03:00:00+01:002006-01-09T03:00:00+01:00Harald Welte<p>
Finally, there is a first public beta version of ulogd 2.x
</p>
<p>
If you use (and like) ulogd-1.x, you should definitely have a look at the 2.x
release. Apart from packet-based logging, ulogd-2.x now also support
flow-based logging. This means that you can just run this daemon (and a recent
2.6.14/2.6.15 kernel) to log per-connection meta data into text files, syslog,
mysql, postgresql, or sqlite3 databases. If you enabled per-connection packet/byte counters in your kernel config, you even get flow-based accounting.
</p>
<p>
If you're interested, check it out at <a href="http://www.netfilter.org/projects/ulogd/">netfilter.org/projects/ulogd</a>.
</p>
<p>
Bug-reports welcome. Don't ask for too much documentation at this time, rather contribute some :)
</p>x_tables, take 5. nfsim tested.https://laforge.gnumonks.org/blog/20060108-xtables-fnsim/2006-01-08T03:00:00+01:002006-01-08T03:00:00+01:00Harald Welte<p>
Today I've posted the (hopefully) final version of x_tables, the in-kernel
generalization of {arp,ip,ip6}_tables to netfilter-devel.
</p>
<p>
After some nfsim hacking, I've been able to add x_tables support to nfsim and
have been successfully running the full nfsim testsuite. The testsuite found a single bug (which has been fixed) but otherwise all tests are passed.
</p>
<p>
Seems like we're going to push x_tables as well as the nf_conntrack port of
ctnetlink (nf_conntrack_netlink) for 2.6.16. Also, as I just noticed on <a href="http://people.netfilter.org/kaber/weblog/2006/01/07#20060107-blog">kaber's
blog</a>, his IPsec patches have made it in time, too. Userspace conntrack
helper support is definitely 2.6.17, though.
</p>ulogd2 now has an abstract SQL/db layerhttps://laforge.gnumonks.org/blog/20051209-ulogd-sql-layer/2005-12-09T03:00:00+01:002005-12-09T03:00:00+01:00Harald Welte<p>
This means that there is now very little code duplication between the mysql and
pgsql drivers, since all the high-level functionality is now 'abstracted away'.
</p>Moved ulogd repository from svn.gnumonks.org to svn.netfilter.orghttps://laforge.gnumonks.org/blog/20051124-ulogd-svn/2005-11-24T03:00:00+01:002005-11-24T03:00:00+01:00Harald Welte<p>
ulogd has practically always been a sub-project of the netfilter project, but was hosted at svn.gnumonks.org for historical reasons. I've now cleaned this up.
</p>
<p>
ulogd-1.x is now hosted at <a href="https://svn.netfilter.org/netfilter/trunk/ulog/ulogd/">https://svn.netfilter.org/netfilter/trunk/ulog/ulogd/</a>, ulogd-2.x at <a href="https://svn.netfilter.org/netfilter/branches/ulog/ulogd2/">https://svn.netfilter.org/netfilter/branches/ulog/ulogd2/</a>.
</p>2.6.14.y stable series lacks lots of netfilter fixeshttps://laforge.gnumonks.org/blog/20051115-stable-patches/2005-11-15T03:00:00+01:002005-11-15T03:00:00+01:00Harald Welte<p>
It seems like DaveM was away, there was some communication problem that lead to
the fact that none of the netfilter related fixes went into 2.6.14.y series (up
to 2.6.14.2) so far. I'm sorry for that, and all the fixes have been submitted
now.
</p>
<p>
So lets hope 2.6.14.3 will have no known netfilter related bugs.
</p>netfilter patch-bombhttps://laforge.gnumonks.org/blog/20051113-patchbombing/2005-11-13T03:00:00+01:002005-11-13T03:00:00+01:00Harald Welte<p>
To be more efficient in flooding DaveM with netfilter patches, I've now hacked
up a set of 'wrapper scripts' around my git tree. They enable me to
efficiently apply patches to my tree, generate sequential sets, and send them
off (actually not using a mail user agent).
</p>
<p>
This means, that for now my patch submissions are (like those of 99.9% of the other kernel hackers) not PGP/GPG signed. If I find some time, I'll add that feature to my script.
</p>
<p>
Anyway, I've sent off the first set of 10 netfilter patches and it worked like
a charm.
</p>nf_conntrack went mainline!https://laforge.gnumonks.org/blog/20051110-nf_conntrack-mainline/2005-11-10T03:00:00+01:002005-11-10T03:00:00+01:00Harald Welte<p>
Ok, finally. After David Miller has returned from his holidays, nf_conntrack
has 'magically' ended up in the mainline tree. Stateful IPv6 packet filtering
in vanilla 2.6.15 is therefore reality.
</p>
<p>
Thanks to Yasuyuki, DaveM, Acme and everybody else who has made this happen.
</p>lots of netfilter.org releaseshttps://laforge.gnumonks.org/blog/20051105-releases/2005-11-05T03:00:00+01:002005-11-05T03:00:00+01:00Harald Welte<p>
Today, I spent a lot of time doing <a href="http://www.netfilter.org/news.html#2005-11-05">releases</a> of libnfnetlink, libnetfilter_log, libnetfilter_queue, libnetfilter_conntrack and the conntrack program.
</p>
<p>
The amount of manual XML editing, copying of files, checking in stuff, ...
required to do a release is way too much. We definitely need some release
automatization.
</p>ulogd2 reaches beta statehttps://laforge.gnumonks.org/blog/20051105-ulogd2/2005-11-05T03:00:00+01:002005-11-05T03:00:00+01:00Harald Welte<p>
ulogd2 has now reached beta stage, and it now has almost all the plugins of
ulogd-1.x. Only the SQL database backends are missing. It also features a
ctnetlink input plugin for flow-based accounting with 2.6.14 kernels.
</p>
<p>
Next, I'll be working on documentation, testing and on some simple IPFIX output
plugin.
</p>iptables-1.3.4 has been releasedhttps://laforge.gnumonks.org/blog/20051103-iptables-1.3.4/2005-11-03T03:00:00+01:002005-11-03T03:00:00+01:00Harald Welte<p>
See <a href="http://www.netfilter.org/projects/iptables/downloads.html#iptables-1.3.4">the
1.3.4 release page</a> and the <a href="http://www.netfilter.org/projects/iptables/files/changes-iptables-1.3.4.txt">ChangeLog</a>.
</p>Bug reports after 2.6.14 is out.https://laforge.gnumonks.org/blog/20051101-bugs/2005-11-01T03:00:00+01:002005-11-01T03:00:00+01:00Harald Welte<p>
I've already received three different serious bug reports about problems with
netfilter/iptables in 2.6.14. This is frustrating, considering how long the
2.6.14 development cycle was. People should try new features of a new kernel
_before_ there is a release. Afterwards it's too late.
</p>2.6.14 is out, 2.6.15 has opened.https://laforge.gnumonks.org/blog/20051029-2615-nfct/2005-10-29T03:00:00+02:002005-10-29T03:00:00+02:00Harald Welte<p>
This means that I've immediately pushed three netfilter related changesets, the
biggest (307k unified diff, roughly 10k lines of code) was nf_conntrack.
</p>
<p>
Given the specific situation that David Miller is on holidays, and we have
Arnaldo Carvalho de Melo maintaining the network stack meanwhile, Linus hasn't
accepted that huge patch in the first round, since he lacked explanation why such a monster was required.
</p>
<p>
I hope my comments will convince him that nf_conntrack really is the way to
go.... let's hope we'll have nf_conntrack mainline in one or two days.
</p>
<p>
I hope Yasuyuki (the main author behind nf_conntrack) will make a big party with his USAGI friends once that happens ;)
</p>The modularity of iptables - or "ipt_SYSRQ"https://laforge.gnumonks.org/blog/20051027-modularity-sysrq/2005-10-27T03:00:00+02:002005-10-27T03:00:00+02:00Harald Welte<p>
One of the best early design choices of iptables was its support for plugin
matches and plugin targets. Over the last five years, we have seen some 100 of
such user-developed special-purpose plugins.
</p><p>
</p>
One that I find particularly funny is <a href="http://terminus.sk/~marek/prog/ipt_sysrq.shtml">ipt_SYSRQ</a>, a target
module that allows you to issue the "magic sysreq" command via a network
packet. This way you can sync, unmount and reboot a otherwise stuck machine that still responds to interrupts.
<p>
Obviously quite dangerous, but the author includes a time stamp and a
cryptographic signature, so replay attacks can only occur in a very small
time frame.
</p>
<p>
It's definitely a cool hack, although I'm not sure whether I'd want to put this
on a production system or not.
</p>Restructuring the netfilter.org project homepagehttps://laforge.gnumonks.org/blog/20051019-homepage/2005-10-19T03:00:00+02:002005-10-19T03:00:00+02:00Harald Welte<p>
Some years ago, the netfilter project only had the kernel side
netfilter/iptables code, and the userspace iptables program. Then we added
patch-o-matic(-ng), and more recently there were a number of more sub-projects
growing, like ipset, all the nfnetlink-related code, ctnetlink, etc.
</p>
<p>
Unfortunately the homepage design didn't really cope with the fact that there is
now a more hierarchical structure with many sub-projects.
</p>
<p>
It was always my hope that some "new webmaster" would take care of it. Unfortunately
we still don't have a webmaster, so I spent some time on it today. You can see
the results at <a href="http://www.netfilter.org">www.netfilter.org</a>.
</p>net-2.6.15 tree has openedhttps://laforge.gnumonks.org/blog/20051016-2615-submissions/2005-10-16T03:00:00+02:002005-10-16T03:00:00+02:00Harald Welte<p>
Since DaveM is on holidays, Acme is now in charge of running the net-2.6.15 tree. I've already
submitted nf_conntrack, the ip_conntrack hash table resizing code from Rusty, as
well as "revisions" support for {arp,ip6}_tables.
</p>
<p>
I'm also basically finished with x_tables now. Everything has been merged with
a post-nf_conntrack tree, and all the conntrack related matches/targets have been ported
to x_tables.
</p>
<p>
Now I need to do some serious testing (including nfsim), before it can be
submitted, too.
</p>More netfilter work at workshop coding day 1https://laforge.gnumonks.org/blog/20051007-netfilter-work/2005-10-07T03:00:00+02:002005-10-07T03:00:00+02:00Harald Welte<p>
After having terminated the traditional workshop part, we've today had day 1
of the <a href="http://workshop.netfilter.org/">workshop.netfilter.org</a>
hacking sessions.
</p>
<p>
Despite the different topic, I spent the better part of the day with Michael
Bellion and Henrik Nordstrom working out the details of nf-hipac / nfnetlink
integration.
</p>
<p>
Apart from that, there's now a nf_conntrack header cleanup in my git tree, I've
ported ebt_[u]log to nf[netlink]_log, fixed some minor Kconfig issues, merged
some patches from Yasuyuki and Pablo, and pushed forward a round of fixes and
updates to DaveM.
</p>Second day of netfilter workshophttps://laforge.gnumonks.org/blog/20051006-workshop/2005-10-06T03:00:00+02:002005-10-06T03:00:00+02:00Harald Welte<p>
If I would start to write about everything that we discussed or only about the
results from the discussions and presentations, I would probably need all night
to write this blog entry.
</p>
<p>
It's been a very productive two days, and I'm looking forward to the hacking
session that will happen on the next two days. Some of the TODO items for the
hacking session will be:
</p><ul>
<li>nfnetlink-enabling nf-hipac</li>
<li>resolving some header file issues for 2.6.14 / nfnetlink</li>
<li>using Gandalf's hashtrie as conntrack hash</li>
<li>nfnetlink-enabling ipset</li>
<li>using string search api for pattern matching in conntrack helpers</li>
<li>completing userspace conntrack helpers using nfnetlink_{queue,conntrack}</li>
<li>
</li></ul>
<p>
Ok, have to stop for now, too much exciting stuff keeping me busy here :(
</p>ulogd2 is workinghttps://laforge.gnumonks.org/blog/20051003-ulogd2-works/2005-10-03T03:00:00+02:002005-10-03T03:00:00+02:00Harald Welte<p>
I've managed to bring ulogd2 to a state where it finally does something. The
dynamic key resolval/linking of plugin stacks is working, and some basic
plugins (NFLOG input, IPV4 packet interpreter (BASE), LOGEMU output) are
working, too.
</p>
<p>
So the remaining work will mostly be in the plugin area. We're currently missing
</p><ul>
<li>ctnetlink input</li>
<li>packet->flow aggregation (basically 'nacctd')</li>
<li>IPFIX input and output</li>
<li>convert the old mysql/pgsql/sqlite output plugins</li>
</ul>
<p>
If you're interested, patches are always welcome. The code can be downloaded
via svn from <a href="http://svn.gnumonks.org/branches/ulog/ulogd2/">http://svn.gnumonks.org/branches/ulog/ulogd2/</a>.
</p>ulogd2 about to hit alpha statehttps://laforge.gnumonks.org/blog/20051002-ulogd2/2005-10-02T03:00:00+02:002005-10-02T03:00:00+02:00Harald Welte<p>
Yet another of my projects that never received the amount of attention that was
required is <a href="http://svnweb.gnumonks.org/branches/ulog/ulogd2">ulogd2</a>. If you
already know the ulogd-1.x series, then you know it as an efficient packet
filter policy violation logging daemon, with backends for files, syslog and
various SQL databases.
</p>
<p>
ulogd2 is much more than that. It's more abstract, and more universal. It's
no longer limited to receiving packets from the ULOG target, but is fully
modularized, with modules for ULOG, NFLOG (see linux-2.6.14), IPFIX, ctnetlink,
... Now you might wonder why there is something like IPFIX and ctnetlink?
That's because ulogd2 can also process (aggregate, export) per-flow
information.
</p>
<p>
The most difficult part of the implementation is the dynamic creation of
"plugin stacks", but I think I wrote about this earlier in my blog.
</p>
<p>
The good news is, that just before I went to bed, ulogd2 compiled for the first
time ;) This means I've waded through the tons of errors and warnings created
by all the changes introduced since it forked off ulogd-1.x about a year ago.
</p>
<p>
Now there are some bits of missing functionality here and there, and certainly
a large bunch of bugs. But if you are a software developer, you know it's much
easier (and rewarding) once the beast actually runs :)
</p>planet.netfilter.org goes livehttps://laforge.gnumonks.org/blog/20050928-planet-netfilter-org/2005-09-28T03:00:00+02:002005-09-28T03:00:00+02:00Harald Welte<p>
Following-up the recent site-wide installation of blosxom on <a href="http://people.netfilter.org/">people.netfilter.org</a>, I've now also
created our own <a href="http://www.planetplanet.org</a>planet</a>%20installation%0Aat%20<a%20href=" http:>planet.netfilter.org</a>. At the
moment, only three netfilter related blogs/journals/diaries are aggregated
there, but with some luck (and your help, since you will have to tell me what
other netfilter related weblogs) it will grow :)
</p>netfilter developer blogshttps://laforge.gnumonks.org/blog/20050927-developer_blogs/2005-09-27T03:00:00+02:002005-09-27T03:00:00+02:00Harald Welte<p>
I first wrote about this in early 2005: Having developer blogs on <a href="http://people.netfilter.org/">people.netfilter.org</a>. Unfortunately I
never finished that project so far. I'm not really a web guy at all, so doing
stuff related to (X)HTML and CSS always gives me the creeps. Why can't we just have a technically skilled web master volunteer for netfilter.org? *sigh*
</p>
<p>
For those who're curious, you check out <a href="http://people.netfilter.org/laforge/weblog/">a mirror of this blog</a>, or <a href="http://people.netfilter.org/gandalf/weblog/">the early beginning of Gandalf's blog</a>.
</p>
<p>
Every netfilter developer with an account on people.netfilter.org can easily
set up a blog, just by putting blog articles into ~/weblog/.
</p>Work on ulogd2https://laforge.gnumonks.org/blog/20050926-ulogd2/2005-09-26T03:00:00+02:002005-09-26T03:00:00+02:00Harald Welte<p>
I've continued work on ulogd2, the next generation netfilter userspace logging
daemon. In addition to packet-based logging, it supports flow-based logging.
</p>
<p>
It turns out my overly-flexible concept of plugin stacks ends up with quite
some implementation complexity. The problem can be viewed similar to a linker
problem (linking symbols of multiple objects), but in addition resolving
dynamically changing dependencies, with some 'symbols' being optional, and with
objects that you can ask "if I give you input symbol X, which output symbols
can you give me" ?
</p>
<p>
I really need to do resolve some tax issues before the netfilter workshop, so
I'm not sure whether I can finish it before.. especially since I've also started to merge years-old pkttables code into a recent kernel.
</p>released libnfnetlink, libnfnetlink_conntrack and conntrackhttps://laforge.gnumonks.org/blog/20050924-libnfnetlink-conntrack-release/2005-09-24T03:00:00+02:002005-09-24T03:00:00+02:00Harald Welte<p>
This triple-release is in anticipation of a 2.6.14 kernel release. The two
libs as well as the conntrack program are userspace counterparts to the "next
generation" subsystems inside the kernel netfilter part.
</p>
<p>
The release involved lots of painful learning-by-doing of autoconf/automake.
I'm not a fan of them at all, but I sill think it's less burden than trying to
invent everything on your own (like we did with the iptables package) and thus
forcing more burden onto the package maintainers of the distributions.
</p>
<p>
I'll probably release libnfnetlink_log and libnfnetlink_queue tomorrow... but I really don't have any time to work on netfilter at the moment, despite this <a href="http://people.netfilter.org/laforge/TODO">TODO</a> list :(.
</p>Submitted the PPTP conntrack/nat helper to the mainline kernelhttps://laforge.gnumonks.org/blog/20050915-pptp-mainline/2005-09-15T03:00:00+02:002005-09-15T03:00:00+02:00Harald Welte<p>
Following-up some serious testing today, I've finally submitted the latest
version of the PPTP helper from the netfilter-2.6.14#pptp tree to the mainline
kernel.
</p>
<p>
With some luck, it will be included before 2.6.14 gets final. It should go in,
since it doesn't modify existing code but is merely an addition.
</p>
<p>
Also, please note that the "ip_conntrack_proto_gre.ko" and "ip_nat_proto_gre.ko"
modules are gone with that 3.x version of the PPTP helper. The respective
code has been integrated into ip_{conntrack,nat}_pptp.ko. My initial dream
of doing some generic (non-PPTP) GRE connection tracking has evaporated, and
thus the PPTP helper now really only handles the special case of pptp-GRE.
</p>patchwork rulez!https://laforge.gnumonks.org/blog/20050831-patchwork/2005-08-31T03:00:00+02:002005-08-31T03:00:00+02:00Harald Welte<p>
Some time ago, <a href="http://ozlabs.org/~jk/">Jeremy Kerr</a> wrote the <a href="http://ozlabs.org/~jk/projects/patchwork/">patchwork</a> program as a
means to track patches sent to mailing-lists (specifically netfilter-devel in our case).
</p>
<p>
I'm now using it more-or-less frequently and it has already uncovered a number
of patches that got lost otherwise. Therefore I consider it a very helpful tool. Hopefully reports of netfilter-devel being "a write-only mailing-list" will
cease now..
</p>CLUSTERIP fixes/cleanuphttps://laforge.gnumonks.org/blog/20050830-clusterip/2005-08-30T03:00:00+02:002005-08-30T03:00:00+02:00Harald Welte<p>
Apparently we now have at least one corporate user of the ipt_CLUSTERIP target
(allowing load balancing without a load balancer). Krisztian Kovacs has
re-worked some of it's weak parts (like refcounting and procfs). I'll review the patches soon.
</p>Linus has merged the net-2.6.14 tree from DaveMhttps://laforge.gnumonks.org/blog/20050830-linus-merges-net2614/2005-08-30T03:00:00+02:002005-08-30T03:00:00+02:00Harald Welte<p>
This means that all the code from my netfilter-2.6.14 tree (master branch) are
now in the mainline kernel. The code in question mainly includes
</p>
<ul>
<li>conntrack event notifiers</li>
<li>nfnetlink layer </li>
<li>ctnetlink interface</li>
<li>nf_log API extension</li>
<li>nf_queue and nf_log /proc files</li>
<li>nfnetlink_log as successor of ipt_ULOG and ebt_ulog</li>
<li>nfnetlink_queue as successor of ip_queue and ip6_queue</li>
</ul>
<p>
We'll see whether nf_conntrack will also go into 2.6.14, at the moment I have
my doubts...
</p>Update on the netfilter workhttps://laforge.gnumonks.org/blog/20050809-netfilter-update/2005-08-09T03:00:00+02:002005-08-09T03:00:00+02:00Harald Welte<p>
Ok, we've seen a terrible amount of bug-fixes going into the net-2.6.14 tree
after my new nfnetlink/nfnetlink_log/nfnetlink_queue/... stuff was merged. It
is my belief that we've now covered most of it.
</p>
<p>
As of now, I'm not planning to make any other big netfilter-related patch
submissions. So nf_conntrack will probably have to wait for 2.6.15, especially
since there are still a number of ip_conntrack/nf_conntrack compatibility
issues to be resolved.
</p>
<p>
Lately I've been working on the userspace side. At least libnfnetlink_log and
the libipulog compat API are finished now. libnfnetlink_queue is getting
there, and the 'big' missing part is the libipq compat API.
</p>
<p>
So now I'm heading for some work on ulogd2, libnfnetlink_conntrack and the
virtual Ethernet device (vdev) code. And if I still have some time left,
there's exciting non-netfilter stuff like my RFID stack.
</p>Bug-fixing nfnetlink_log, nfnetlink_queue and nfnetlink_conntrackhttps://laforge.gnumonks.org/blog/20050805-nfnetlink-bugfixing/2005-08-05T03:00:00+02:002005-08-05T03:00:00+02:00Harald Welte<p>
Almost as expected, as soon as that code hits a somewhat more used tree (such as
Dave m's net-2.6.14 and the -mm tree), there are numerous bug-fixes piling up.
</p>
<p>
That's a bit embarrassing, though I'd rather fix it now than later when it is already in the mainline tree :)
</p>nf_conntrack now merged into local branch of netfilter-2.6.14.githttps://laforge.gnumonks.org/blog/20050804-nf_conntrack/2005-08-04T03:00:00+02:002005-08-04T03:00:00+02:00Harald Welte<p>
I've committed the last version of nf_conntrack, the layer-3-independent
connection tracking code to my netfilter-2.6.14.git tree. It's a local branch
called "nf_conntrack".
</p>
<p>
Yasuyuki and me have been working to port the latest mainline ip_conntrack
changes to nf_conntrack. Now the tree should now be fully in sync with
ip_conntrack of the same net-2.6.14 tree (this means that it supports
CONNTRACK_ACCT and has it's own conntrack-event-api).
</p>
<p>
Major pieces that are missing from nf_conntrack are:
</p>
<ul>
<li>IPv4 NAT for nf_conntrack</li>
<li>nf_conntrack_netlink (aka ctnetlink for nf_conntrack)</li>
<li>support for ip(6)tables 'state', 'conntrack' and other matches</li>
<li>Finally, ct_sync</li>
</ul>Merging the PPTP helper to net-2.6.14https://laforge.gnumonks.org/blog/20050730-pptp/2005-07-30T03:00:00+02:002005-07-30T03:00:00+02:00Harald Welte<p>
After having finished my work on the nfnetlink based subsystems, I've
progressed to making the PPTP helper fit for mainline inclusion in 2.6.14.
</p>
<p>
First, it needed an update towards the 2.6.13 conntrack helper API changes (now
that expect's have refcounts). Second, we don't have lockhelp.h anymore, and
third I want to fall-back to ip_conntrack_proto_generic in case GRE version1
(RCF1701) packets are seen. Stay tuned.
</p>nfnetlink_log submittedhttps://laforge.gnumonks.org/blog/20050730-nfnetlink_log/2005-07-30T03:00:00+02:002005-07-30T03:00:00+02:00Harald Welte<p>
I've submitted my nfnetlink_log patches to DaveM earlier today. So what is
this about? It's a replacement for ipt_LOG, ip6t_LOG, ebt_ulog, ipt_ULOG. It
introduces a layer-3 (AF_xxx) independent way of logging packets via a
userspace logging process.
</p>
<p>
Again, one step towards code unification. One new piece of code that replaces
four existing ones (of similar size), and obsoletes the need for any other such mechanisms that might have appeared for other protocols later on.
</p>
<p>
If you want to see how to use it from your favourite userspace app, please
refer to <a href="https://svn.netfilter.org/netfilter/trunk/libnfnetlink_log">libnfnetlink_log</a>.
</p>public netfilter-2.6.14 git treehttps://laforge.gnumonks.org/blog/20050730-git-netfilter/2005-07-30T03:00:00+02:002005-07-30T03:00:00+02:00Harald Welte<p>
I've made public my netfilter-2.6.14 tree (based on DaveM's net-2.6.14 tree)
at http://people.netfilter.org/laforge/scm/netfilter-2.6.14.git, also available
via rsync://people.netfilter.org/users/laforge/scm/netfilter-2.6.14.git
</p>
<p>
Since this is the first time I'm making a public git tree available, please
contact me in case you have any problems accessing it.
</p>
<p>
I still need to find out how to produce incremental git trees like the ipw2200
project does - this way I would not have to provide a full kernel tree, but
only those changes that I do in the netfilter part of it.
</p>iptables-1.3.3 is releasedhttps://laforge.gnumonks.org/blog/20050729-iptables-release/2005-07-29T03:00:00+02:002005-07-29T03:00:00+02:00Harald Welte<p>
Today I've released <a href="http://www.iptables.org/download.html">iptables-1.3.3</a>. Among some
minor fixes (such as for the extremely important feature to SNAT and DNAT
to/from ICMP ID _ranges_), it contains one major fix for an embarrassing
use-after-free problem that was only introduced with 1.3.2. What do we learn
from this? I need to review patches more carefully.
</p>
<p>
It also includes the NFQUEUE target, which is basically an extension to QUEUE.
QUEUE only supports one queue number (0), so there can only be one userspace
process be attached to it. This lead to the ugly hack of <a href="http://gnumonks.org/projects/project_details?p_id=2">ipqmpd</a>, the IP
QUEUE multiplex daemon. Combining NFQUEUE with nfnetlink_queue (which is
already in DaveM's net-2.6.14 tree), you can now have 65535 different queues,
each heading to a separate userspace process. This is again one step ahead
towards supporting "100% userspace conntrack helpers" which are sort of a
strange hybrid variant of transparent proxies.
</p>Lots of netfilter hacking over the last couple of dayshttps://laforge.gnumonks.org/blog/20050720-nfnetlink-ctnetlink/2005-07-20T03:00:00+02:002005-07-20T03:00:00+02:00Harald Welte<p>
Following-up meeting the other networking hackers at netconf, I got really
extremely motivated and basically spent every single minute hacking code.
</p>
<p>
The projects include:
</p>
<ul>
<li>skb shrinkage (already merged in DaveM's net-2.6.14 tree)</li>
<li>nfnetlink (already merged in DaveM's net-2.6.14 tree)</li>
<li>conntrack event notifiers (already merged in DaveM's net-2.6.14 tree)</li>
<li>ctnetlink (reworked to use network byte order in all the payload)</li>
<li>nfnetlink_queue (a nfnetlink-based queue implementation)</li>
<li>vdev (a virtual device that allows you to use multiple mac addresses on one Ethernet device)</li>
<li>mmio_test (include support for machine-parseable reporting)</li>
</ul>pptp-conntrack-nat for 2.6.11 and 2.6.12.x readyhttps://laforge.gnumonks.org/blog/20050704-pptp-2611/2005-07-04T03:00:00+02:002005-07-04T03:00:00+02:00Harald Welte<p>
I've finished the port of <a href="http://svn.netfilter.org/cgi-bin/viewcvs.cgi/trunk/patch-o-matic-ng/patchlets/pptp-conntrack-nat/">pptp-conntrack-nat</a> to the new 'rustynat' infrastructure of the 2.6.11 (and 2.6.12.x) kernels.
</p>
<p>
The frequent reader of this blog will have noticed my prior post. Despite
being just a minor kernel release, the conntrack/nat core got some recent
re-work which made porting of non-trivial helpers quite complex.
</p>
<p>
I've tested plain conntrack and SNAT/MASQUERADE so far. DNAT remains untested
for now, but <i>should</i> work. It's not as common so I deferred testing and
potential debugging - esp. since I'm going to be travelling again by tomorrow.
</p>
<p>
Thanks again to the cool guys from <a href="http://www.netboxblue.com/">NetBoxBlue</a> for funding this work. That made it a lot easier to put this in the top section of my TODO list.
</p>ct_sync, kernel 2.6.10, NAT and masqueradehttps://laforge.gnumonks.org/blog/20050629-ctsync-nat-masquerade/2005-06-29T03:00:00+02:002005-06-29T03:00:00+02:00Harald Welte<p>
Following up some thorough testing and debugging, I finally got both (SNAT,
DNAT) and MASQUERADe to work with ct_sync on a 2.6.10 kernel.
</p>
<p>
Apart from forgetting to disable TCP window tracking, there were some subtle
mistakes in #ifdef/endif of the code that actually prevented whole sections
from being built ;)
</p>
<p>
Debugging the problem however has forced me to update the ct_sync ethereal
plugin (<a href="http://gnumonks.org/~laforge/ct_sync_ethereal.png">screenshot</a>) to
parse almost every bit within the ct_sync protocol.
</p>Fighting with Docbook-Websitehttps://laforge.gnumonks.org/blog/20050628-fighting-with-docbook-website/2005-06-28T03:00:00+02:002005-06-28T03:00:00+02:00Harald Welte<p>
Almost all homepages I maintain are built using <a href="http://docbook.sourceforge.net/projects/website/index.html">docbook-website</a>.
</p>
<p>
Unfortunately I'm not a big XSLT guru, so I'm having a hard time finding and
fixing bugs in them. For that reason especially the netfilter.org homepage was suffering from problems with olinks.
</p>
<p>
Luckily, the 2.6.0 release of docbook-website seems to have fixed all the
olink-related bugs I was experiencing. I just re-built the page and now all
the cross-linking (including #localifo) is working fine now. Thanks to whoever
fixed it :)
</p>netfilter patch-o-matic-ng cleanup dayhttps://laforge.gnumonks.org/blog/20050627-patch-o-matic-clean-day/2005-06-27T03:00:00+02:002005-06-27T03:00:00+02:00Harald Welte<p>
Just a quick status update:
</p>
<p>
I've tried to make most of the patches in netfilter <a href="http://svn.netfilter.org/netfilter/trunk/patch-o-matic-ng">patch-o-matic-ng</a>
work with 2.6.12 today. It's amazing how fast the code bit-rots there.
</p>
<p>
I've also applied tons of cosmetic cleanup fixes, such as %zu and %ti format strings to avoid compiler warnings on 64bit archs.
</p>
<p>
Now it's time to head back to the PPTP-conntrack-nat port for 2.6.11+. Once
that is finished, I'm back to ct_sync work.
</p>
<p>
Oh, and yes, I almost forgot: <a href="http://ftp.netfilter.org/pub">ftp.netfilter.org</a> will have start having daily snapshots of <a href="http://svn.netfilter.org/netfilter/trunk/conntrack">conntrack</a> and <a href="http://ipset.netfilter.org/">ipset</a>.
</p>Adding missing features to libctnetlink and "conntrack" programhttps://laforge.gnumonks.org/blog/20050623-conntrack-ctnetlink/2005-06-23T03:00:00+02:002005-06-23T03:00:00+02:00Harald Welte<p>
I'm back to netfilter hacking, and it's more fun than ever :)
</p>
<p>
<a href="http://svn.netfilter.org/netfilter/trunk/libctnetlink">libctnetlink</a> was
extended to provide an API function to add an expectation. Also, the cool new <a href="http://svn.netfilter.org/netfilter/trunk/conntrack">conntrack control program</a> now has preliminary support to add expectations from the command line.
</p>
<p>
This means there is now the full chain in place (from kernel to userspace
library to command line tool) to allow expectations to be created from
userspace. I wonder how long it will take to see the first userspace ALG's to
show up. It would be a pleasure to finally see complex protocol handling done
in userspace rather than the kernel side.
</p>
<p>
While hacking at conntrack, I also added a man page and fixed some other bits
and pieces. Once the "do we want an ID, and if yes which kind of ID"
discussion has concluded on netfilter-devel, we can submit nfnetlink and
ctnetlink to the mainline kernel and make a first libnfnetlink, libctnetlink
and conntrack release.
</p>Just finished three days of teaching intensive netfilter/iptables coursehttps://laforge.gnumonks.org/blog/20050617-training/2005-06-17T03:00:00+02:002005-06-17T03:00:00+02:00Harald Welte<p>
I just finished my first three-day-in-a-row training for quite some time.
Seems like I almost forgot how exhausting it can be to talk for three full
days. However, it seems like the biggest part of the training went quite fine,
and the attendees were satisfied.
</p>
<p>
The most interesting part for me was to learn about the practical "real-world"
setups in which those users were actually using packet filters, NAT, bridges,
routers, etc. So basically it put me in touch with some of the more advanced
users, and taught me about their particular requirements. This will definitely
help during the further development process.
</p>Started to work on PPTP helper port for post-2.6.11https://laforge.gnumonks.org/blog/20050610-pptp-2611/2005-06-10T03:00:00+02:002005-06-10T03:00:00+02:00Harald Welte<p>
I've started to port the PPTP conntrack and NAT helper to the 2.6.11-and-later
API changes. Obviously that forced me to look at the code deeper than I did
for quite some time. That in turn led me to the discovery of a bug.
Obviously, the bug was not hit in most installations, because it's only a bug
in the error path.
</p>
<p>
Expectations used to be kmalloc()ed, so the helper could directly kfree() them
without a problem. Some time ago, we introduced a slab cache for expectations,
so that would no longer work. Now the code in svn was changed to use
ip_conntrack_expect_free().
</p>Back to ct_synchttps://laforge.gnumonks.org/blog/20050504-ctsync-continue/2005-05-04T03:00:00+02:002005-05-04T03:00:00+02:00Harald Welte<p>
I've managed to get back to work on ct_sync again. The final steps towards
full multi-master operation are underway. Apart from some changes to the
protocol on the wire, there is a major reorganization of almost all involved data structures.
</p>
<p>
I'm deeply sorry for not having been able to continue at the pace that I wanted
(and promised some customers), but there have been lots of issues that I
couldn't push back and had to deal with them immediately.
</p>ctnetlink now with flow-based accounting supporthttps://laforge.gnumonks.org/blog/20050416-ctnetlink/2005-04-16T03:00:00+02:002005-04-16T03:00:00+02:00Harald Welte<p>
Some months ago, I included per-connection packet and byte counters to
ip_conntrack (CONFIG_NF_CT_ACCT) into Linux-2.6 mainline. However, reading the
entries from /proc/net/ip_conntrack is not really a useful interface to access
those counters.
</p>
<p>
I've now merged Pablo Neira's latest ctnetlink/nfnetlink changes with mine, and
patch-o-matic-ng now includes support for dumping the counters to userspace.
</p>
<p>
With any userspace program (using libctnetlink) you can then retrieve the
counters. Either you wait until a connection dies (and receive the DELETE
message from the netlink socket, containing the counters), or you regularly
issue a request to list-conntracks-and-reset-counters-to-zero request.
</p>
<p>
The <a href="http://svn.netfilter.org/cgi-bin/viewcvs.cgi/trunk/conntrack/">conntrack</a> tool in subversion now already includes support for this, see the <b>conntrack -E conntrack</b> and <b>conntrack -L conntrack -z</b> commands.
</p>
<p>
I've also picked up working on ulogd2 again, to provide a all-in-one solution
that allows you to create IPFIX (aka NETFLOW) records or put the per-flow
accounting data directly into a SQL database. If everything works fine, I'll
be finished in a week or so.
</p>porting conntrack/nat helpers to post-2.6.11https://laforge.gnumonks.org/blog/20050413-conntrack_nat-2.6.11/2005-04-13T03:00:00+02:002005-04-13T03:00:00+02:00Harald Welte<p>
Unfortunately most of the conntrack/nat helpers in patch-o-matic were broken
ever since 2.6.11 was released. The reason is the new semantics of the
redesigned conntrack/nat helper API by Rusty Russell and Pablo Neira.
</p>
<p>
It's not an easy and straight-forward port, and as usual there were not many
people volunteering for that job. Max Kellermann is a positive example, he
ported the h323 helpers.
</p>
<p>
I've now ported the all remaining ones BUT the PPTP helper. At the moment I'm
not sure whether the PPTP/GRE helper can be ported/used at all with the new
infrastructure :( This will need some serious amount of thinking.
</p>
<p>
All the ported helpers are available from pom-ng. I don't have the possibility
to test them, since I don't actually use most of those protocols. Testing /
debugging / bug reporting is therefore very welcome. Anyone writing a test case
for nfsim would be my personal hero.
</p>More dual Opteron netfilter/iptables benchmarkshttps://laforge.gnumonks.org/blog/20050407-v20z-netfilter-benchmark/2005-04-07T03:00:00+02:002005-04-07T03:00:00+02:00Harald Welte<p>
The last two days I was at a <a href="http://www.stz-netze.de/">network performance lab</a> in Stralsund, Germany. We were testing dual Opteron 250 (2,4GHz) machines with e1000 cards and Linux.
</p>
<p>
One of the interesting results was that ip_conntrack [again] scales better as
the load generators. The generators couldn't establish more than 25,000 new
TCP connections per second and no more than 1 million total concurrent
connections ;)
</p>
<p>
Thus I'm now pretty much convinced that ip_conntrack scales quite reasonable,
and we should concentrate optimizations to other areas of netfilter/iptables.
</p>Microsoft due to invent packet filtering?https://laforge.gnumonks.org/blog/20050318-microsoft-packetfilter/2005-03-18T03:00:00+01:002005-03-18T03:00:00+01:00Harald Welte<p>
According to <a href="http://www.theregister.co.uk/2005/03/18/windows_server_firewall/">some
reports</a> the worlds most popular series of proprietary systems is suffering
from a severe lack of a packet filter. This is also documented at another <a href="http://www.securityfocus.com/columnists/307">article plus discussion</a>. </p>
<p>
Now apparently Microsoft will invent the idea of having a packet filter
integrated into the operating system with their WPF for Longhorn.
</p>
<p>
It's really amazing how innovative those guys are ;) Did I mention that Linux
has an embedded packet filter since more than a decade?
</p>ct_sync now fully modularhttps://laforge.gnumonks.org/blog/20050312-ctsync-fullymodular/2005-03-12T03:00:00+01:002005-03-12T03:00:00+01:00Harald Welte<p>
ct_sync is now able to run multiple instances on one node, allowing vrrp-like
setups! Thanks go to <a href="http://www.dreamlab.net/>dreamlab</a>%20for%0Asponsoring%20this%20work.%0A</p>%0A<p>%0AIf%20you're%20interested%20in%20looking%20at%20the%20current%20development%20version,%20check%20out%20<a%20href=" http:>http://svn.netfilter.org/netfilter/branches/netfilter-ha/linux-2.6-actact/</a>
</p>
<p>
The next couple of weeks will be focusing on testing and real active-active setups with multiple masters. My brain is already smoking from all the synchronization issues ;)
</p>Picked up working on ct_sync againhttps://laforge.gnumonks.org/blog/20050308-ctsync-status/2005-03-08T03:00:00+01:002005-03-08T03:00:00+01:00Harald Welte<p>
I've recently again picked up the work on ct_sync. The final goal ist to
support real active-active fail-over setups. Before the real work on that
particular issue can start, there are a number of prerequisites, like:
</p>
<ul>
<li>multiple cluster instances on one node</li>
<li>new sysfs-based configuration interface</li>
</ul>Getting conntrack+nat helpers to work with 2.6.11https://laforge.gnumonks.org/blog/20050305-conntrack-nat-helpers-2.6/2005-03-05T03:00:00+01:002005-03-05T03:00:00+01:00Harald Welte<p>
2.6.11 is out for a number of days, and we still don't have the conntrack/nat
helpers from patch-o-matic ported to Rusty's latest conntrack/nat helper
infrastructure changes.
</p>
<p>
It turns out that there are more changes necessary than I though initially.
It's strange that nat helpers now don't have a separate expectfn() anymore,
only the expectation has one. So I guess at least for talk, we'll have to call back into ip_conntrack_talk.c from ip_nat_talk.c.
</p>
<p>
With some luck I'll be finished by tomorrow and can again concentrate on the
fun stuff like active-active support for ct_sync.
</p>Dynamic port assignment of conntrack helper https://laforge.gnumonks.org/blog/20050214-helper-port_assignment/2005-02-14T03:00:00+01:002005-02-14T03:00:00+01:00Harald Welte<p>
I've coded a <a href="ftp://ftp.gnumonks.org/pub/patches/2.6.11-rc4-ip_conntrack_helper_proc.patch">patch</a> against 2.6.11-rc4 that allows dynamic (re-)configuration of
the port assignment of connection tracking helpers. This has been a TODO item
for at least three years on my TODO list ;)
</p>Porting patch-o-matic-ng to 2.6.11https://laforge.gnumonks.org/blog/20050213-pomng-2611/2005-02-13T03:00:00+01:002005-02-13T03:00:00+01:00Harald Welte<p>
Rusty's recent changes to the conntrack/nat helper API in 2.6.11-rcX have
rendered all conntrack/nat helpers in pom-ng unusable.
</p>
<p>
I've created a new <a href="https://svn.netfilter.org/netfilter/branches/patch-o-matic-ng/linux-2.6.11">svn 2.6.11 pom-ng branch</a> and started porting of all the helpers in there. The opportunity was also good to port all the 2.4.x only helpers to 2.6.x, so we won't have the big gap between 2.4.x and 2.6.x supported helpers.
</p>
<p>
I expect this to take a couple of days, and even after that, for most protocols
I have no opportunity to test (proprietary protocols, proprietary software,
...), so I'll have to rely on your <a href="https://bugzilla.netfilter.org/">feedback</a>.
</p>The iptables-1.3.0 release is outhttps://laforge.gnumonks.org/blog/20050213-iptables130-release/2005-02-13T03:00:00+01:002005-02-13T03:00:00+01:00Harald Welte<p>
I finally managed to get the <a href="http://iptables.org/news.html#2005-02-12">iptables-1.3.0</a> release out.
</p>Ulogd 1.20 releasehttps://laforge.gnumonks.org/blog/20050213-ulogd120-release/2005-02-13T03:00:00+01:002005-02-13T03:00:00+01:00Harald Welte<p>
After applying lots of updates that have accumulated in the last months, I've
released <a href="ftp://ftp.netfilter.org/pub/ulogd/">ulogd-1.20</a>. Changes
include dozens of fixes and a new PCAP and SQLITE3 output plugin.
</p>
<p>
This will probably the last new-feature release for 1.x, since I'm already working on 2.x with included support for flow-based (ct_acct) logging.
</p>Some more ct_sync fixeshttps://laforge.gnumonks.org/blog/20050210-ctsync24-loopfix/2005-02-10T03:00:00+01:002005-02-10T03:00:00+01:00Harald Welte<p>
The latest bug (endless loop) was caused by one of my last bugfixes.
Apparently I introduced an endless loop into a linked list (the nat bysource hash).
</p>Rusty producing more patches than I can review in fast timehttps://laforge.gnumonks.org/blog/20050122-patches/2005-01-22T03:00:00+01:002005-01-22T03:00:00+01:00Harald Welte<p>
There was s sudden surge in netfilter/iptables development in late December and
early January. I'm still reviewing some of the changes, and am not yet
convinced that all of them are the way to go.
</p>Work starting on ct_sync active-activehttps://laforge.gnumonks.org/blog/20050122-ctsync-active_active/2005-01-22T03:00:00+01:002005-01-22T03:00:00+01:00Harald Welte<p>
The swiss company <a href="http://www.dreamlab.net/">dremalab</a> wants to
sponsor me to work on an extension of ct_sync for active-active setups.
More detailed news will appear very soon on the netfilter page and/or on this blog. Stay tuned.
</p>Porting PPTP conntrack/nat helpers to 2.6.xhttps://laforge.gnumonks.org/blog/20041022-pptp_conntrack_nat_2.6/2004-10-22T03:00:00+02:002004-10-22T03:00:00+02:00Harald Welte<p>
I've always refused to do the port of the PPTP conntrack/NAT helper I wrote for
2.4.x because there's higher priority items on my agenda.
</p>
<p>
Apparently it helped, as I was told Mandrake did a port to 2.6.x. I thought
that is great news, and I thought it'd take an hour or so to get it merged.
</p>
<p>
Unfortunately that 'port' was totally incomplete. NAT couldn't have worked at
all, and if you sent it a nonlinear TCP packet it would very likely crash your kernel.
</p>
<p>
In the end I spent the whole afternoon at it, with a resulting patch that is
about the same size as the original code :(
</p>
<p>
The code is now in our subversion repository, I didn't have the time test it so
far, so any testing you (yes, you, the reader) might give it would be
appreciated.
</p>Conntrack events for 2.6.xhttps://laforge.gnumonks.org/blog/20041015-conntrack-events/2004-10-15T03:00:00+02:002004-10-15T03:00:00+02:00Harald Welte<p>
I've separated out Patrick McHardy's conntrack events from the
nfnetlink-ctnetlink patch and ported it to 2.6.x. The patch was posted to
netfilter-devel, in case you're interested.
</p>
<p>
For those of you who don't know what this means: It means that the first part
of what is required for a 2.6.x ct_sync port is now done ;)
</p>ct_sync ethereal pluginhttps://laforge.gnumonks.org/blog/20041014-ctsync-ethereal/2004-10-14T03:00:00+02:002004-10-14T03:00:00+02:00Harald Welte<p>
While doing some more ct_sync testing/debugging, I found out that for some
reason my <a href="https://svn.netfilter.org/netfilter/trunk/netfilter-ha/ctnl_dump/">ctnl_dump</a> program didn't work anymore. Instead of fixing it, and updating it to CTSP (conntrack sync protocol) version 2, I decided to write a plugin for the well-known packet analyzer <a href="http://www.ethereal.com/">ethereal</a>.
</p>
Due to the nature of the CTSP, it passes arch- endian- and
configuration-dependent data structures between master and slave. This means
that it is virtually impossible to write a analyzer that will work in any of
those combinations.
<p>
My <a href="https://svn.netfilter.org/netfilter/trunk/netfilter-ha/ethereal-plugin/">plugin</a> now assumes that you use a little-endian 32bit machine with the
pptp-conntrack-nat patch applied.
</p>
<p>
The plugin turned out to provide very useful information, and I was able to fix
some issues in ct_sync using it.
</p>Proceedings of Developer Workshop 2004 onlinehttps://laforge.gnumonks.org/blog/20040925-proceedings/2004-09-25T03:00:00+02:002004-09-25T03:00:00+02:00Harald Welte<p>
I finally managed to finish the write-up and markup of the proceedings. They
are available in a number of formats at the <a href="http://www.netfilter.org/documentation/index.html#documentation-events">documentation section</a><a> of the netfilter home page.
</a></p>
<p>
In theory, there could still be lots of semantic markup added, but well, who cares...
</p>pkttables finally making some progresshttps://laforge.gnumonks.org/blog/20040924-pkttables/2004-09-24T03:00:00+02:002004-09-24T03:00:00+02:00Harald Welte<p>
I've found some time to work on pkttables again. Isn't that great news? If my
brain is not completely broken, I've now worked out a RCU-powered way to have
full table traversal with a completely lock-less reader path, while providing
atomicity either on table- or chain level.
</p>
<p>
Also, I ripped the "struct nf_attr" and NFA_xx macros from the nfnetlink core,
since they get replaced by my vTLV (Versioned TLV) code.
</p>
<p>
With some luck I'll be able to continue my pkttables work next week
</p>CLUSTERIP is in patch-o-matic-nghttps://laforge.gnumonks.org/blog/20040921-clusterip-pom/2004-09-21T03:00:00+02:002004-09-21T03:00:00+02:00Harald Welte<p>
About one year ago I did some work for <a href="http://www.suse.com/">SuSE</a>
in implementing load-balancer-less load-balancing clusters ;) This is achieved
by replying to ARP requests with a link-layer multicast address, so all nodes receive all packets. Hashing parts of the ip header now determines whether the packet is to be passed up the stack on a given node.
</p>
<p>
The result is called the iptables CLUSTERIP target, and I've now finally put it
in patch-o-matic-ng, since it was only available in my undocumented public CVS
tree so far.
</p>libiptc2 bugfix (upcoming iptables-1.3.0 prerelease)https://laforge.gnumonks.org/blog/20040920-libiptc2-iptables13/2004-09-20T03:00:00+02:002004-09-20T03:00:00+02:00Harald Welte<p>
Since the segfault-bug in my recent re-implementation of libiptc has now been
fixed, I think we're about one week before a iptables-1.3.0 prerelease for
public beta-testing.
</p>Working on the summary / proceedings of the 3rd netfilter developer workshophttps://laforge.gnumonks.org/blog/20040914-nfworkshop-summary/2004-09-14T03:00:00+02:002004-09-14T03:00:00+02:00Harald Welte<p>
Spent a couple of hours putting the notes of the 3rd netfilter developer workshop together in a <a href="http://cvs.netfilter.org/documentation/conferences/nf-workshop-2004-summary.xml">single file</a>, adding lots of Docbook-XML markup, ...
</p>
<p>
It's still far from being complete, but I have to finish this ASAP..
</p>netfilter workshop / Linux Kongress 2004https://laforge.gnumonks.org/blog/20040908-nfworkshop-lk2004/2004-09-08T03:00:00+02:002004-09-08T03:00:00+02:00Harald Welte<p>
I've not been able to write any articles for this log over the last few days,
since I've been busy with the third netfilter developer workshop and
Linux-Kongress 2004.
</p>
<p>
The netfilter workshop went really well, apparently the
</p>Finishing preparations for upcoming netfilter developer workshophttps://laforge.gnumonks.org/blog/20040903-workshop-preparations/2004-09-03T03:00:00+02:002004-09-03T03:00:00+02:00Harald Welte<p>
I've spent a significant amount of time over the last couple of days with the
final preparations of the upcoming 3rd netfilter developer workshop. This is
the first one where I'm in charge of every tiny bit of the organization, and I
hope I got everything right.
</p>
<p>
The first attendees are scheduled to arrive tomorrow. They might even arrive
before me, since I'll be heading the 500km down south tomorrow.
</p>Main netfilter.org server has been replacedhttps://laforge.gnumonks.org/blog/20040825-netfilter.org-xserve/2004-08-25T03:00:00+02:002004-08-25T03:00:00+02:00Harald Welte<p>
Yesterday I finally got around moving almost all netfilter.org services from
our old Sun Ultra5 to the new XServe ClusterNode.
</p>
<p>
Unfortunately there were lots of complications, so I had to stay awake until
5am in order to get all services running again. At least for now, everything
seems to run smoothly.
</p>IPv6 packet filter benchmarkinghttps://laforge.gnumonks.org/blog/20040803-ipv6-filter-benchmarking/2004-08-03T03:00:00+02:002004-08-03T03:00:00+02:00Harald Welte<p>
It seems like a German university is currently doing feature analysis and
benchmarking of IPv6 packet filters. Coincidentally, I'm going to near that
university next week anyway, so I'll stop over for a short visit and help them
with their ip6tables evaluation setup.
</p>
<p>
I would be very interested to see some numbers on ip6tables... as we just
discovered at the networking conference in Portland, nobody seems to be doing
benchmarking / profiling on the Linux IPv6 code so far.
</p>David Miller survived my 13-patch patch-bombhttps://laforge.gnumonks.org/blog/20040725-davem-patches/2004-07-25T03:00:00+02:002004-07-25T03:00:00+02:00Harald Welte<p>
This is good news, DaveM accepted all the 13 netfilter related patches that I
had pending for 2.6.9. The patches included a number of optimizations, the
ctstat, connection-based accounting, TCP window tracking, and some conversions
to new in-kernel-API (seq_file, module_param).
</p>
<p>
Now let's hope that 2.6.8 will be released soon and we can start the 2.6.9 cycle...
</p>IPFIX / ulog integrationhttps://laforge.gnumonks.org/blog/20040722-ipfix-ulogd/2004-07-22T03:00:00+02:002004-07-22T03:00:00+02:00Harald Welte<p>
After some more in-depth study of the IPFIX IETF drafts, I finally started
coding. Having written the first dozens of lines, I discovered that on an
abstract layer IPFIX doesn't do something too different from my good old ulogd.
Ignoring the minor difference that ulogd deals with individual packets and
IPFIX with flows, the ulogd_iret_t structure is very similar to what IPFIX
templates are trying to describe.
</p>
<p>
So I now forked a ulogd2 branch off the current ulogd subversion tree and
started to reorganize the tree.
</p>
<p>
For more flexibility, I am going for a stackable plugin infrastructure, where
the sysadmin can configure stacks like: ULOG->ulogd_BASE->flow
aggregation->IPFIX-over-TCP-export or ctnetlink->IPFIX-over-SMTP-export.
</p>Merging 2.6.8-rc2 changes into patch-o-matic nghttps://laforge.gnumonks.org/blog/20040722-pomng-2.6.8-rc2/2004-07-22T03:00:00+02:002004-07-22T03:00:00+02:00Harald Welte<p>
I just started the boring job of merging 2.6.8-rc2 with patch-o-matic-ng... I'm
happy that Jozsef, Martin and Patrick did this for the last couple of kernel
releases. However, I need to get more into this job again in order to
determine which patches still have to be submitted to the mainline kernel...
</p>
<p>
Expect some pom-ng breakage over the next couple of days...
</p>Working towards IPFIX based on conntrackhttps://laforge.gnumonks.org/blog/20040721-connacct-connbytes/2004-07-21T03:00:00+02:002004-07-21T03:00:00+02:00Harald Welte<p>
I've written a patch to add 64bit packet and byte counters for both directions
of every ip_conntrack. This should enable a clean and efficient implementation
of flow based accounting, when combined with ctnetlink events and a userspace
daemon picking up those events.
</p>
<p>
I need to study the IPFIX (IETF Working Group) specifications in more detail before writing the respective daemon...
</p>
<p>
The patch is apparently working, you can read the counters via
/proc/net/ip_conntrack and also use a modified/extended/updated version of the
'connbytes' match.
</p>Pattern-matching API in the 2.6.x Kernelhttps://laforge.gnumonks.org/blog/20040719-qsearch/2004-07-19T03:00:00+02:002004-07-19T03:00:00+02:00Harald Welte<p>
There are various places in the kernel where we need to do some kind of pattern
matching on the packet contents. Applications range from connection tracking
helpers (looking for FTP PORT command, ...) over the 'string' match to
intrusion detection systems.
</p>
<p>
Two years ago, Phillipe Biondi once came up with something called <a href="http://www.cartel-securite.fr/pbiondi/libqsearch.html">libqsearch</a>. It implements a generic pattern matching API, supporting plugin based algorithm implementations.
</p>
<p>
I now took the liberty of porting this into a 2.6.x kernel, resulting in lots
of changes that make my qsearch port now incompatible with what Philipe wrote.
Anyway, I'm now in the process of combining this with Rusty's recent work on
skb_walk() and skb_iter(), so we can pattern-match against a
fragmented/nonlinear skb without any copy.
</p>New iptables-1.2.11 and patch-o-matic-ng-20040621 releasehttps://laforge.gnumonks.org/blog/20040621-iptables-reelease/2004-06-21T03:00:00+02:002004-06-21T03:00:00+02:00Harald Welte<p>
I have just released iptables-1.2.11 and patch-o-matic-ng-20040621 on the <a href="http://www.netfilter.org/">netfilter homepage</a>.
</p><p>
Seems like we'll never have an iptables release that doesn't introduce some
severe bug that requires releasing another version immediately later.
To some part, I blame the users. Seems like not enough of them try the CVS
snapshots and report bugs back to us.
</p>Doing lots of benchmarks / tuning / profiling latelyhttps://laforge.gnumonks.org/blog/20040421-opteron-benchmarks/2004-04-21T03:00:00+02:002004-04-21T03:00:00+02:00Harald Welte<p>
During the last weeks I've been working on tuning/benchmarking/profiling the
Sun V20z dual Opteron boxes for high-speed packet filtering purpose.
</p>
<p>
Some of my findings:
</p>
<ul>
<li>i386 kernels give you higher pps than x86_64 (because sk_buff is smaller)</li>
<li>e1000 are way faster than tg3 boards (could be hardware or driver issue)</li>
<li>Intel PRO/1000MT Quad e1000 boards suck (apparently problems with the onboard PCI-X bridge)</li>
<li>Connection Tracking performance is not that bad...</li>
<li>ip_tables performance sucks, even if the ruleset is empty ?!?</li>
<li>2.4.x has slightly worse results than 2.6.x if you use IRQ affinity, but really sucks if you don't, since the kernel doesn't balance IRQ's by itself (and irqbalance daemon only balances every 10 seconds)</li>
<li>You can route up to 1Mpps at 64bytes packet size</li>
<li>ip_conntrack and iptable_filter at suck at least 300kpps, giving 700kpps as a result</li>
</ul>
<p>
Expect a more detailed report within the next weeks.
</p>Some more ct_sync bug huntinghttps://laforge.gnumonks.org/blog/20040407-ctsync-bufixing/2004-04-07T03:00:00+02:002004-04-07T03:00:00+02:00Harald Welte<p>
It seems like there's still a number of bugs left in ct_sync. I've spent the
major part of the last three days hunting them down. Seems to be really hard
ones, that only appear when compiled with recent gcc-3.2 versions... Learned a lot about objdump and strange x86 "instruction encoding artefacts", though.
</p>Finally committing Pablo Neira's optimization patcheshttps://laforge.gnumonks.org/blog/20040327-pablo-optimizations/2004-03-27T03:00:00+01:002004-03-27T03:00:00+01:00Harald Welte<p>
Subject says it all... I've found some time to review his patches. With some
luck, DaveM will receive them later today.
</p>revived the dropped tablehttps://laforge.gnumonks.org/blog/20040326-dropped-table/2004-03-26T03:00:00+01:002004-03-26T03:00:00+01:00Harald Welte<p>
After about two years in deep freeze, I revived the idea of a dropped table.
For those of you who haven't heard about it in the past: The idea is to
gather all packets that are dropped at any place within the network stack.
This is very useful for auditing and debugging.
</p>
<p>
Userspace support is included in libiptc/iptables for ages, so all you need is patch-o-matic-ng from >= today.
</p>Added a new 'licensing' section on the netfilter homepagehttps://laforge.gnumonks.org/blog/20040229-licensing-homepage/2004-02-29T03:00:00+01:002004-02-29T03:00:00+01:00Harald Welte<p>
Since recently more and more vendors seem to disobey the terms of the GNU GPL,
I decided to put some more detailed information on how to comply with this
license online. It was written for the netfilter/iptables project, but should
apply to any other GPL licensed free software project. You can find the section <a href="http://www.netfilter.org/licensing.html">here</a>.
</p>