Harald Welte's blog
   

RSS

Harald's Web
gnumonks.org
hmw-consulting.de
sysmocom.de

Projects
OpenBSC
OsmocomBB
OsmocomTETRA
deDECTed.org
gpl-violations.org
gpl-devices.org
OpenMoko
gnufiish
OpenEZX
OpenBeacon
OpenPCD
librfid
openmrtd
opentom.org
netfilter/iptables

Categories

Archives

Other Bloggers
David Burgess
Zecke
Dieter Spaar
Michael Lauer
Stefan Schmidt
Rusty Russell
David Miller
Martin Pool
Jeremy Kerr
Tim Pritlove (German)
fukami (German)
fefe (German)
Bradley M. Kuhn
Lawrence Lessig
Kalyan Varma

Aggregators
kernelplanet.org
planet.netfilter.org
planet.openezx.org
planet.openmoko.org
planet.foss.in

Ohloh profile for laforge
identi.ca
twitter
flattr
Linked in
Xing

Creative Commons License
Articles on this blog/journal are licensed under a Creative Commons Attribution-NoDerivs 2.5 License.


blosxom


Contact/Impressum

       
Thu, 05 Sep 2013
Problems with OpenVPN on high-latency satellite links

So far I never had a need to look in detail how the OpenVPN protocol actually looks on the wire. It seems like not many people had that much of a close look, as the wireshark plugin is fairly recent (from 2012 I think) while OpenVPN is around for ten more years than that. If I was an OpenVPN developer, the wireshark plugin would be the first thing I'd write to help debugging and development. At least that's what I've been doing from OpenPCD to SIMtrace and through the various GSM and other protocols I encounter...

The reason for my current investigation is some quite strange and yet-unexplained problems when running OpenVPN on high-latency satellite links. I'm not talking about high-bandwidth VSAT or systems with dedicated / guaranteed bandwidth. The links I'm seeing often have RTT (as seen by ICMP echo) of 2 seconds, sometimes even 5. This is of course not only the satellite link, but includes queuing on the ground, possibly the space segment and of course the terminal, including (possibly) access arbitration.

What struck me _very_ odd is that OpenVPN is sending tons of UDP messages with ridiculously small size during the TLS handshake when bringing up the tunnel. Further investigation shows that they actually internally configure a MTU of '0' for the link, which seems to be capped at 100 bytes control payload, plus HMAC and OpenVPN header resulting in 124 to 138 bytes UDP payload.

Now you have to consider that the server certificate (possibly including even a CA certificate) can be quite large, plus all the gazillions of TLS handshaking options in ServerHello, the first message from server to client. This means that OpenVPN transmits that ServerHello in something like 40 to 60 fragments of 100 bytes each! And each of the fragments will have to be acknowledged by the remote end, leading 80 to 120 UDP/IP packets _only_ for the delivery of the TLS ServerHello.

Then you start reviewing the hundreds of OpenVPN configuration options, many of them related to MTU, MSS, fragmentation, etc. There is none for that insanely small default of 100 bytes for control packets during hand-shake. I even read through the related source code, only to find that indeed this behavior seems hard-coded. Some time later I had written a patch to add this option, thanks to Free Software. It seems to work on client and server and brings the ClientHello down to much smaller 4-6 messages.

The fun continues when you see that the timeout for re-transmitting fragments that have not been ACKed yet is 2 seconds. At my satellite RTT times this of course leads to lots of unneeded re-transmissions, simply because the ACK hasn't made its way back to the sender of the original message yet. Luckily there's a configuration option for that.

After the patch and changing that option, the protocol trace looks much more sane. However, I still have problems establishing a tunnel in a number of cases. For some odd reason, the last fragment of the ServerHello is not acknowledged by the client, no matter whether patched or unpatched OpenVPN is being used. I get acknowledgements always only up to fragment N-1 after having transmitted N. That last fragment is then re-transmitted by the server with exponential back-off, and finally some 60 seconds later the server gives up as the TLS handshake didn't finish within that time. Extending the TLS handshake timeout to 120 seconds also doesn't help.

I'm not quite sure why something like 39 out of 39 fragments all get delivered reliably and acknowledged, but always the last fragment (40) doesn't make it to the remote side. That's certainly not random packet loss, but a very deterministic one. Let's see if I can still manage to find out what that might be...

[ /linux | permanent link ]

Wed, 05 Jun 2013
Attending HITCON and COSCUP in Taipei

It is my pleasure to attend the HITCON 2013 and COSCUP 2013 conferences in July/August this year. They are both in Taipei. HITCON is a hacker/security event, while COSCUP is a pure Free/Open Source Software conference.

At both events I will be speaking at the growing list of GSM related tools that are available these days, like OpenBSC, OsmcoomBB, SIMtrace, OsmoSGSN, OsmoBTS, OsmoSDR, etc. As they are both FOSS projects and useful in a security context, this fits well within the scope of both events.

Given that I'm going to be back to Taiwan, I'm looking very much forward to meeting old friends and former colleagues from my Openmoko days in Taipei. God, do I miss those days. While terribly stressful, they still are the most exciting days of my career so far.

And yes, I'm also going to use the opportunity for a continuation of my motorbike riding in this beautiful country.

[ /gsm | permanent link ]

Mon, 03 Jun 2013
Rest In Peace, Atul Chitnis

Today, very sad news has reached me: Atul Chitnis has passed away. Most people outside of India will most likely not recognize the name: He has been instrumental in pioneering the BBS community in India, and the founder and leader of the Linux Bangalore and later FOSS.in conferences, held annually in Bangalore.

I myself first met Atul about ten years ago, and had the honor of being invited to speak at many of the conferences he was involved in. Besides that professional connection, we became friends. The warmth and affection with which I was accepted by him and his family during my many trips to Bangalore is without comparison. I was treated and accepted like a family member, despite just being this random free software hacker from Germany who is always way too busy to return the amount of kindness.

Despite the 17 year age difference, there was a connection between the two of us. Not just the mutual respect for each others' work, but something else. It might have been partially due to his German roots. It might have been the similarities in our journey through technology. We both started out in the BBS community with analog modems, we both started to write DOS software in the past, before turning to Linux. We both became heavily involved in mobile technology around the same time: He during his work at Geodesic, I working for Openmoko. Only in recent years his indulgence in Apple products was slightly irritating ;)

Only five weeks ago I had visited Atul. Given the state of his health, it was clear that this might very well be the last time that we meet each other. I'm sad that this now actually turned out to become the thruth. It would have been great to meet again at the end of the year (the typical FOSS.in schedule).

My heartfelt condolences to his family. Particularly to his wonderful wife Shubha, his daughther Anjali, his mother and brother. [who I'm only not calling by their name in this post as they deserve some privacy and their Identities is not listed on Atuls wikipedia page].

Atul was 51 years old. Way too young to die. Yet, he has managed to created a legacy that will extend long beyond his life. He profoundly influenced generations of technology enthusiasts in India and beyond.

[ /linux | permanent link ]

Fri, 29 Mar 2013
Hardware outage affectiong osmocom.org, deDECTed.org, gpl-violations.org

As usual, murphy's law dictates that problems will occur at the worst possible moment. One of my servers in the data center died on March 20, and it was the machine which hosts the majority of the free software projects that I've created or am involved in. From people.netfilter.org to OpenPCD and OpenEZX to gpl-violations.org and virtually all osmocom.org sites and services.

Recovery was slow as there is no hot spare and none of my other machines in the data center have backplanes for the old SCA-80 hard disks that are in use by that particular machine. So we had to send the disks to Berlin, wait until I'm back there, and then manually rsync everything over to a different box in the data center.

To my big surprise, not many complaints reached me (and yes, my personal and/or business e-mail was not affected in any way)

Recovery is complete now, and I'm looking forward to things getting back to normal soon.

[ /misc | permanent link ]

OsmoDevCon 2013 preparation update

OsmoDevCon 2013 is getting closer every day, and I'm very much looking forward to meet the fellow developers of the various Osmcoom sub-projects. Organization-wise, the catering has now been sorted out, and Holger has managed to get a test license for two ARFCN from the regulatory body without any trouble.

This means that we're more or less all set. The key needs to be picked up from IN-Berlin, and we need to bring some extra extension cords, ethernet switch, power cords and other gear, but that's really only very minor tasks.

There's not as much formal schedule as we used to have last year, which is good as I hope it means we can focus on getting actual work done, as opposed to spending most of the time updating one another about our respective work and progress.

[ /osmocom | permanent link ]

Fri, 08 Feb 2013
Update on what I've been doing

For the better part of a year, this blog has failed to provide you with a lot of updates what I've been doing. This is somewhat relate to a shift from doing freelance work on mainline / FOSS projects like the Linux kernel.

In April 2011, Holger and I started a new company here in Berlin (sysmocom - systems for mobile communications GmbH). This company, among other things, attempts to provide products and services surrounding the various mobile communications related FOSS projects, particularly OpenBSC, OsmoSGSN, OpenGGSN, but also OsmocomBB, and now also OsmoBTS + OsmoPCU, two integral components of our own BTS product called sysmoBTS.

Aside from the usual software development, this entails a variety of other tasks, technical and non-technical. First of all, I did more electrical engineering than I did in the years since Openmoko. And even there, I was only leading the hardware architecture, and didn't actually have to capture schematics or route PCBs myself. So now there are some general-purpose and some customer-specific circuits that had to be done. I really enjoy that work, sometimes even more than software development. Particularly the early/initial design phase can be quite exciting. Selecting components, figuring out how to interconnect them, whether you can fit all of them together in the given amount of GPIOs and other resource of your main CPU, etc. But then even the hand-soldering the first couple of boards is fun, too.

Of all the things I so far had least exposure to is casing and mechanical issues. Luckily we have a contractor working on that for us, but still there are all kinds of issues that can go wrong, where unpopulated PCB footprints can suddenly make contact with a case, or all kinds of issues related to manufacturing tolerances. Another topic is packaging. After all, you want the products to end up in the hands of the customer in a neat, proper and form-fitting package.

On the other hand, there is a lot of administrative work. Sourcing components can sometimes be a PITA, particularly if even distributors like Digikey conspire against you and don't even carry those low quantities of a component that we need for our 100-board low quantity runs. EMC and other measurements for CE approval are a fun topic, too. I've never been involved personally in those, and it has been an interesting venture. Luckily, at least for sysmoBTS, things are looking quite promising now. Customs paperwork, Import/Export related buerocracy (both in Germany as well as other countries) always have new surprises, despite me having experience in dealing with customs for more than 10 years now.

Also significant amount of time is spent on evaluating suppliers and their products, e.g. items like SIM/USIM cards, cavity duplexers, antennas, cables, adapters, power amplifiers and other RF related accessories for our products.

The thing that really caught me off-guard are the German laws on inventory accounting. Basically there is no threshold for low-quantity goods, so as a company on capital (GmbH/AG) you have to account for each and every fscking SMD resistor or capacitor. And then you don't only have to count all those parts, but also put a value at them. Depending on the type of item, you have to use either the purchasing price, or the current market price if you were to buy it again, or the price you expect to sell the item for. Furthermore, the trade law requirements on inventory accounting are different than the tax laws, not often with contradictory aims ;)

In the end it seems the best possible strategy is to put a lot of the low-value inventory into the garbage bin before the end of the financial year, as the value of the product (e.g. 130 SMD resistors in 0402 worth fractions of cents) is so much lower than the cost of counting it. Now that's of course an environmental sin, especially if you consider lots and lots of small and medium-sized companies ending up at that conclusion :(

So all in all, this should give you somewhat of an explanation why there might have been less activity on this blog about exciting technical things. On the one hand, they might relate to customer related projects which are of confidential nature. On the other hand, they might simply be boring things like dealing with transport damage of cavity duplexers from china, or with FedEx billing customs/import fees to the wrong address...

Overall I still have the feeling that I was writing a decent amount of code in 2012 - although there can never be enough :) Most of it was probably either related to OsmoBTS, OpenBSC/OsmoNITB or the various Erlang SS7/TCAP/MAP related projects. The list of more community-oriented projects with long TODO lists is growing, though. I'd like to work on SIMtrace MITM / card emulation support, the CC32RS512 based smartcard OS, libosmosim (there's a first branch in libosmocore.git). Let's hope I can find a bit more time for that kind of stuff this year. You should never give up hope, they say ;)

[ /misc | permanent link ]

Mon, 04 Feb 2013
Back from FOSDEM 2013

As (almost) every year, I attended the annual incarnation of FOSDEM. It is undoubtedly (one of?) the most remarkable events about Free Software in existence. No registration, no fees, 24 tracks in parallel, an estimated 5000 number of attendees. I also like that it brings together people from so many different communities, not _just_ the Linux or Gnome or KDE or Telephony or Legal people, but a good mixture of everything.

I have to congratulate the organizers, who manage to pull this off, year after year again. And as opposed to many other events, they do so quietly and without much recognition, I feel. I'd also like to thank the many volunteers working tirelessly before, at and after the event. Last, but not least, I'd like to thank the local university (ULB Solbosch) hosting the event.

What made me truly sad though, is the amount of littering that surprisingly many of the attendees did. This was particularly visible in the Cafeteria. Imagine an event run by volunteers, who put in a lot of time and effort. Imagine an event where food and drinks are sold by volunteers at such low prices that there can barely be any profit at all. And then imagine people eating there and leaving all their rubbish around, as if they were in some kind of restaurant where they are being served and where somebody is cleaning up after them. It really makes me feel very bitter to see this. Don't people realize that those very volunteers who are creating the event will then have to put in _their_ spare time just because those who just enjoyed their coffee or lunch didn't have the extra 30 seconds of bringing their trash to the trashcan? I feel ashamed for members of our community who behave this way. Please think next time before acting and show your respect to the people behind FOSDEM.

[ /linux/conferences | permanent link ]

Talk Idea: How to write code to make later enforcement easy

During FOSDEM 2013, I spoke with some fellow Free Software developers about how my knowledge on copyright and specifically legal aspects of software copyright has influenced the way how I write code, and particularly how I design architecture of programs.

This made me realize that this would probably make a quite interesting talk at Free Software conferences: How to architect and write code in order to make later [GPL] enforcement easy.

Of course there are all the general and mostly well-known rules like keeping track of who owns which part of the copyright, having proper copyright claims and license headers, etc.

But I'm more thinking in the sense of: How do I write code in a way to make sure people extending it in some way with their own code will be forced to create a derivative work. If that is the case, they will have absolutely no choice but to also license that under GPL.

This is particularly important in the case of GPL licensed libraries. The common understanding in the community is that writing an executable program against a GPL licensed library will constitute a derivative work and thus the main program must be licensed under the GPL, if it is ever distributed.

However, in reality there is of course no precedent, and in some particular cases, the legal framework, depending on the jurisdiction, might come to different conclusions if it ever ended up in court. The claim of a 'derivative work' would be particularly weak if the main program is only using a set of standard function calls whose function declarations are the same in many versions of the GPL licensed library you link against. So let's assume there was a GPL licensed standard C library for stuff like open(), close(), printf() and the like. I think it would be very difficult to argue in court that a program written against those functions and linked against such a library would constitute a derivative work of the library. As in fact, there are many other implementations providing the exact same interface, under different licenses, and the API was not even drafted by the author of the GPL licensed implementation.

So I think there are some things that an author of an (intentionally) GPL licensed library can do while writing the code, which will later help him to establish that an executable program is a derived work.

The same is true to some extent for executable programs, too. I very intentionally did not introduce a plug-in interface for BTS drivers in OpenBSC, even though while technically it would have been possible. I _want_ somebody who adds code for a different BTS to touch the main code of the program instead of just writing an external plugin. The mere fact that he has to edit the main program in order to add a new BTS driver indicates that he is creating a derivative work.

So I'll probably try to submit a talk on this topic to some upcoming conference[s]. If you think this is an interesting topic and want me to talk about it at a FOSS related event, please feel free to send me an e-mail.

[ /linux/gpl-violations | permanent link ]

Wed, 16 Jan 2013
Why I hate phone calls so much

The fact that I have more than 20 missed phone calls on my land line telephone after only half a day has passed triggers me to write this blog post.

It is simply impossible to get any productive work done if there are synchronous interruptions. If I'm doing any even remotely complex task such as analyzing code, designing electronics or whatever else, then the interruption of the flow of thoughts, and the context switch to whatever the phone call might be about is costing me an insurmountable amount of my productive efficiency. I doubt that I am the only one having that feeling / experience.

So why on earth does everybody think they are entitled to interrupt my work at any given point in time they desire? Why do they think whatever issue they have rectifies an immediate interruption in what I am doing? To me, an unscheduled phone call almost always feels like an insult. It is a severe intrusion into my work-flow, and has a very high cost to me in terms of loss of productivity.

Sure, there are exceptional absolute emergencies (like, a medical emergency of a family member). But just about anything else can be put in an e-mail, which I can respond to at a time of my choosing, i.e. at a time I am not deeply buried into some other task that requires expensive context switching and the associated loss of productivity. And yes, a response might be the same day, some days later, or even a week or more later. There are literally hundreds of mails of dozens of people that need to be responded to. I can never even remotely answer all of them in a timely manner, even if I'm working 12-14 hours a day up to 7 days a week.

Right now I'm doing the only reasonable thing that is left: Switch off all phones. And to anyone out there intending to contact me: Please think twice before calling me on the phone. Almost anything can be put in an e-mail. And if you really want to have a phone call, please request a scheduled phone call in an e-mail containing a very detailed agenda and explanation of the topic.

[ /misc | permanent link ]

Wed, 02 Jan 2013
Strain of bad luck

From roughly September to December 2012 I seem to have had a quite unusual strain of bad luck and set-backs. I don't want to go into the details here, as most of the issues are of quite private nature.

This has kept me quite distracted from a lot of my other activity. Projects like the various Osmocom sub-projects, gpl-violations.org are in desperate need of attention, and I have severely neglected my responsibilities in the Chaos Computer Club Berlin e.V. :(

I don't even want to talk about actual paid work, where customers also had to put up with repeated schedule slips and lack of availability.

I let down friends and colleagues at a number of occasions, as I was unable to keep up with anything that remotely resembles my typical work schedule.

Last but not least, I regrettably have also not felt much of an urge to write many blog posts here.

My sincere hope and expectation is that things are going to improve quickly in 2013. At least most of issues from the last half year have been resolved. Now I need to work through a considerable back-log of work and find more time for my volunteer projects in the FOSS and hacker worlds. However, this will need some time and I would like to ask for some patience. I do intend to be up to speed with things just like before.

In this spirit, I am looking forward to a productive and exciting 2013. Happy hacking und Viel Spass am Gerät

[ /personal | permanent link ]