Problems with OpenVPN on high-latency satellite links
So far I never had a need to look in detail how the OpenVPN protocol actually
looks on the wire. It seems like not many people had that much of a close
look, as the wireshark plugin is fairly recent (from 2012 I think) while
OpenVPN is around for ten more years than that. If I was an OpenVPN developer,
the wireshark plugin would be the first thing I'd write to help debugging and
development. At least that's what I've been doing from OpenPCD to SIMtrace and
through the various GSM and other protocols I encounter...
The reason for my current investigation is some quite strange and
yet-unexplained problems when running OpenVPN on high-latency satellite links.
I'm not talking about high-bandwidth VSAT or systems with dedicated /
guaranteed bandwidth. The links I'm seeing often have RTT (as seen by ICMP
echo) of 2 seconds, sometimes even 5. This is of course not only the satellite
link, but includes queuing on the ground, possibly the space segment and of
course the terminal, including (possibly) access arbitration.
What struck me _very_ odd is that OpenVPN is sending tons of UDP messages with
ridiculously small size during the TLS handshake when bringing up the tunnel.
Further investigation shows that they actually internally configure a MTU of
'0' for the link, which seems to be capped at 100 bytes control payload, plus
HMAC and OpenVPN header resulting in 124 to 138 bytes UDP payload.
Now you have to consider that the server certificate (possibly including even a
CA certificate) can be quite large, plus all the gazillions of TLS handshaking
options in ServerHello, the first message from server to client. This means
that OpenVPN transmits that ServerHello in something like 40 to 60 fragments of
100 bytes each! And each of the fragments will have to be acknowledged by the
remote end, leading 80 to 120 UDP/IP packets _only_ for the delivery of the TLS
Then you start reviewing the hundreds of OpenVPN configuration options, many of
them related to MTU, MSS, fragmentation, etc. There is none for that insanely
small default of 100 bytes for control packets during hand-shake. I even read
through the related source code, only to find that indeed this behavior seems
hard-coded. Some time later I had written a patch to add this option, thanks
to Free Software. It seems to work on client and server and brings the
ClientHello down to much smaller 4-6 messages.
The fun continues when you see that the timeout for re-transmitting fragments that
have not been ACKed yet is 2 seconds. At my satellite RTT times this of course
leads to lots of unneeded re-transmissions, simply because the ACK hasn't made
its way back to the sender of the original message yet. Luckily there's a
configuration option for that.
After the patch and changing that option, the protocol trace looks much more sane.
However, I still have problems establishing a tunnel in a number of cases. For
some odd reason, the last fragment of the ServerHello is not acknowledged by
the client, no matter whether patched or unpatched OpenVPN is being used. I
get acknowledgements always only up to fragment N-1 after having transmitted N.
That last fragment is then re-transmitted by the server with exponential
back-off, and finally some 60 seconds later the server gives up as the TLS
handshake didn't finish within that time. Extending the TLS handshake timeout
to 120 seconds also doesn't help.
I'm not quite sure why something like 39 out of 39 fragments all get delivered
reliably and acknowledged, but always the last fragment (40) doesn't make it to
the remote side. That's certainly not random packet loss, but a very
deterministic one. Let's see if I can still manage to find out what that might be...
[ /linux |
permanent link ]
Attending HITCON and COSCUP in Taipei
It is my pleasure to attend the HITCON
2013 and COSCUP 2013
conferences in July/August this year. They are both in Taipei. HITCON
is a hacker/security event, while COSCUP is a pure Free/Open Source
At both events I will be speaking at the growing list of GSM related
tools that are available these days, like OpenBSC, OsmcoomBB, SIMtrace,
OsmoSGSN, OsmoBTS, OsmoSDR, etc. As they are both FOSS projects and
useful in a security context, this fits well within the scope of both
Given that I'm going to be back to Taiwan, I'm looking very much forward
to meeting old friends and former colleagues from my Openmoko days in
Taipei. God, do I miss those days. While terribly stressful, they
still are the most exciting days of my career so far.
And yes, I'm also going to use the opportunity for a continuation of my
motorbike riding in this beautiful country.
[ /gsm |
permanent link ]
Rest In Peace, Atul Chitnis
Today, very sad news has reached me: Atul Chitnis has
passed away. Most people outside of India will most likely not
recognize the name: He has been instrumental in pioneering the BBS
community in India, and the founder and leader of the Linux Bangalore
and later FOSS.in
conferences, held annually in Bangalore.
I myself first met Atul about ten years ago, and had the honor of being
invited to speak at many of the conferences he was involved in. Besides
that professional connection, we became friends. The warmth and
affection with which I was accepted by him and his family during my many
trips to Bangalore is without comparison. I was treated and accepted
like a family member, despite just being this random free software
hacker from Germany who is always way too busy to return the amount of
Despite the 17 year age difference, there was a connection between the
two of us. Not just the mutual respect for each others' work, but
something else. It might have been partially due to his German roots.
It might have been the similarities in our journey through technology.
We both started out in the BBS community with analog modems, we both
started to write DOS software in the past, before turning to Linux. We
both became heavily involved in mobile technology around the same time:
He during his work at Geodesic, I working for Openmoko. Only in recent
years his indulgence in Apple products was slightly irritating ;)
Only five weeks ago I had visited Atul. Given the state of his health,
it was clear that this might very well be the last time that we meet
each other. I'm sad that this now actually turned out to become the
thruth. It would have been great to meet again at the end of the year
(the typical FOSS.in schedule).
My heartfelt condolences to his family. Particularly to his wonderful
wife Shubha, his daughther Anjali, his mother and brother. [who I'm
only not calling by their name in this post as they deserve some privacy
and their Identities is not listed on Atuls wikipedia page].
Atul was 51 years old. Way too young to die. Yet, he has managed to
created a legacy that will extend long beyond his life. He profoundly
influenced generations of technology enthusiasts in India and beyond.
[ /linux |
permanent link ]
Hardware outage affectiong osmocom.org, deDECTed.org, gpl-violations.org
As usual, murphy's law dictates that problems will occur at the worst
possible moment. One of my servers in the data center died on March 20,
and it was the machine which hosts the majority of the free software
projects that I've created or am involved in. From people.netfilter.org
to OpenPCD and OpenEZX to gpl-violations.org and virtually all
osmocom.org sites and services.
Recovery was slow as there is no hot spare and none of my other
machines in the data center have backplanes for the old SCA-80 hard
disks that are in use by that particular machine. So we had to send the
disks to Berlin, wait until I'm back there, and then manually rsync
everything over to a different box in the data center.
To my big surprise, not many complaints reached me (and yes, my
personal and/or business e-mail was not affected in any way)
Recovery is complete now, and I'm looking forward to things getting back
to normal soon.
[ /misc |
permanent link ]
OsmoDevCon 2013 preparation update
2013 is getting closer every day, and I'm very much looking forward
to meet the fellow developers of the various Osmcoom sub-projects.
Organization-wise, the catering has now been sorted out, and Holger has
managed to get a test license for two ARFCN from the regulatory body
without any trouble.
This means that we're more or less all set. The key needs to be picked
up from IN-Berlin, and we need to bring some extra extension cords,
ethernet switch, power cords and other gear, but that's really only very
There's not as much formal schedule as we used to have last year, which
is good as I hope it means we can focus on getting actual work done, as
opposed to spending most of the time updating one another about our
respective work and progress.
[ /osmocom |
permanent link ]
Update on what I've been doing
For the better part of a year, this blog has failed to provide you with
a lot of updates what I've been doing. This is somewhat relate to a
shift from doing freelance work on mainline / FOSS projects like the
In April 2011, Holger and I started a new company here in Berlin (sysmocom - systems for mobile communications
GmbH). This company, among other things, attempts to provide
products and services surrounding the various mobile communications
related FOSS projects, particularly OpenBSC, OsmoSGSN, OpenGGSN, but
also OsmocomBB, and now also OsmoBTS + OsmoPCU, two integral components
of our own BTS product called sysmoBTS.
Aside from the usual software development, this entails a variety of
other tasks, technical and non-technical. First of all, I did more
electrical engineering than I did in the years since Openmoko. And even
there, I was only leading the hardware architecture, and didn't actually
have to capture schematics or route PCBs myself. So now there are some
general-purpose and some customer-specific circuits that had to be done.
I really enjoy that work, sometimes even more than software development.
Particularly the early/initial design phase can be quite exciting.
Selecting components, figuring out how to interconnect them, whether you
can fit all of them together in the given amount of GPIOs and other
resource of your main CPU, etc. But then even the hand-soldering the
first couple of boards is fun, too.
Of all the things I so far had least exposure to is casing and
mechanical issues. Luckily we have a contractor working on that for us,
but still there are all kinds of issues that can go wrong, where
unpopulated PCB footprints can suddenly make contact with a case, or
all kinds of issues related to manufacturing tolerances. Another topic
is packaging. After all, you want the products to end up in the hands
of the customer in a neat, proper and form-fitting package.
On the other hand, there is a lot of administrative work. Sourcing
components can sometimes be a PITA, particularly if even distributors
like Digikey conspire against you and don't even carry those low
quantities of a component that we need for our 100-board low quantity
runs. EMC and other measurements for CE approval are a fun topic, too.
I've never been involved personally in those, and it has been an
interesting venture. Luckily, at least for sysmoBTS, things are looking
quite promising now. Customs paperwork, Import/Export related
buerocracy (both in Germany as well as other countries) always have new
surprises, despite me having experience in dealing with customs for
more than 10 years now.
Also significant amount of time is spent on evaluating suppliers and
their products, e.g. items like SIM/USIM cards, cavity duplexers,
antennas, cables, adapters, power amplifiers and other RF related
accessories for our products.
The thing that really caught me off-guard are the German laws on
inventory accounting. Basically there is no threshold for low-quantity
goods, so as a company on capital (GmbH/AG) you have to account for each
and every fscking SMD resistor or capacitor. And then you don't only
have to count all those parts, but also put a value at them. Depending
on the type of item, you have to use either the purchasing price, or the
current market price if you were to buy it again, or the price you
expect to sell the item for. Furthermore, the trade law requirements on
inventory accounting are different than the tax laws, not often with
contradictory aims ;)
In the end it seems the best possible strategy is to put a lot of the
low-value inventory into the garbage bin before the end of the financial
year, as the value of the product (e.g. 130 SMD resistors in 0402 worth
fractions of cents) is so much lower than the cost of counting it. Now
that's of course an environmental sin, especially if you consider lots
and lots of small and medium-sized companies ending up at that
So all in all, this should give you somewhat of an explanation why there
might have been less activity on this blog about exciting technical
things. On the one hand, they might relate to customer related projects
which are of confidential nature. On the other hand, they might simply
be boring things like dealing with transport damage of cavity duplexers
from china, or with FedEx billing customs/import fees to the wrong
Overall I still have the feeling that I was writing a decent amount of
code in 2012 - although there can never be enough :) Most of it was
probably either related to OsmoBTS, OpenBSC/OsmoNITB or the various
Erlang SS7/TCAP/MAP related projects. The list of more
community-oriented projects with long TODO lists is growing, though.
I'd like to work on SIMtrace MITM / card emulation support, the
CC32RS512 based smartcard OS, libosmosim (there's a first branch in
libosmocore.git). Let's hope I can find a bit more time for that kind
of stuff this year. You should never give up hope, they say ;)
[ /misc |
permanent link ]
Back from FOSDEM 2013
As (almost) every year, I attended the annual incarnation of FOSDEM. It is undoubtedly (one of?) the most
remarkable events about Free Software in existence. No registration, no fees,
24 tracks in parallel, an estimated 5000 number of attendees. I also like that
it brings together people from so many different communities, not _just_ the Linux or Gnome or KDE or Telephony or Legal people, but a good mixture of everything.
I have to congratulate the organizers, who manage to pull this off, year after
year again. And as opposed to many other events, they do so quietly and
without much recognition, I feel. I'd also like to thank the many volunteers
working tirelessly before, at and after the event. Last, but not least, I'd
like to thank the local university (ULB Solbosch) hosting the event.
What made me truly sad though, is the amount of littering that surprisingly
many of the attendees did. This was particularly visible in the Cafeteria.
Imagine an event run by volunteers, who put in a lot of time and effort.
Imagine an event where food and drinks are sold by volunteers at such low
prices that there can barely be any profit at all. And then imagine people
eating there and leaving all their rubbish around, as if they were in some kind
of restaurant where they are being served and where somebody is cleaning up
after them. It really makes me feel very bitter to see this. Don't people
realize that those very volunteers who are creating the event will then have to
put in _their_ spare time just because those who just enjoyed their coffee or
lunch didn't have the extra 30 seconds of bringing their trash to the trashcan?
I feel ashamed for members of our community who behave this way. Please think
next time before acting and show your respect to the people behind FOSDEM.
[ /linux/conferences |
permanent link ]
Talk Idea: How to write code to make later enforcement easy
During FOSDEM 2013, I spoke with some fellow Free Software developers
about how my knowledge on copyright and specifically legal aspects of
software copyright has influenced the way how I write code, and
particularly how I design architecture of programs.
This made me realize that this would probably make a quite interesting
talk at Free Software conferences: How to architect and write code in
order to make later [GPL] enforcement easy.
Of course there are all the general and mostly well-known rules like
keeping track of who owns which part of the copyright, having proper
copyright claims and license headers, etc.
But I'm more thinking in the sense of: How do I write code in a way to
make sure people extending it in some way with their own code will be
forced to create a derivative work. If that is the case, they will have
absolutely no choice but to also license that under GPL.
This is particularly important in the case of GPL licensed libraries.
The common understanding in the community is that writing an executable
program against a GPL licensed library will constitute a derivative work
and thus the main program must be licensed under the GPL, if it is ever
However, in reality there is of course no precedent, and in some
particular cases, the legal framework, depending on the jurisdiction,
might come to different conclusions if it ever ended up in court. The
claim of a 'derivative work' would be particularly weak if the main
program is only using a set of standard function calls whose function
declarations are the same in many versions of the GPL licensed library
you link against. So let's assume there was a GPL licensed standard C
library for stuff like open(), close(), printf() and the like. I think
it would be very difficult to argue in court that a program written
against those functions and linked against such a library would
constitute a derivative work of the library. As in fact, there are many
other implementations providing the exact same interface, under
different licenses, and the API was not even drafted by the author of
the GPL licensed implementation.
So I think there are some things that an author of an (intentionally)
GPL licensed library can do while writing the code, which will later
help him to establish that an executable program is a derived work.
The same is true to some extent for executable programs, too. I
very intentionally did not introduce a plug-in interface for BTS drivers
in OpenBSC, even though while technically it would have been possible.
I _want_ somebody who adds code for a different BTS to touch the main
code of the program instead of just writing an external plugin. The
mere fact that he has to edit the main program in order to add a new BTS
driver indicates that he is creating a derivative work.
So I'll probably try to submit a talk on this topic to some
upcoming conference[s]. If you think this is an interesting topic and
want me to talk about it at a FOSS related event, please feel free to
send me an e-mail.
[ /linux/gpl-violations |
permanent link ]
Why I hate phone calls so much
The fact that I have more than 20 missed phone calls on my land line
telephone after only half a day has passed triggers me to write this
It is simply impossible to get any productive work done if there
are synchronous interruptions. If I'm doing any even remotely complex
task such as analyzing code, designing electronics or whatever else,
then the interruption of the flow of thoughts, and the context switch to
whatever the phone call might be about is costing me an insurmountable
amount of my productive efficiency. I doubt that I am the only one
having that feeling / experience.
So why on earth does everybody think they are entitled to interrupt my
work at any given point in time they desire? Why do they think whatever
issue they have rectifies an immediate interruption in what I am doing?
To me, an unscheduled phone call almost always feels like an insult. It
is a severe intrusion into my work-flow, and has a very high cost to me
in terms of loss of productivity.
Sure, there are exceptional absolute emergencies (like, a medical
emergency of a family member). But just about anything else can be put
in an e-mail, which I can respond to at a time of my choosing, i.e. at a
time I am not deeply buried into some other task that requires expensive
context switching and the associated loss of productivity. And yes, a
response might be the same day, some days later, or even a week or more
later. There are literally hundreds of mails of dozens of people that
need to be responded to. I can never even remotely answer all of them
in a timely manner, even if I'm working 12-14 hours a day up to 7 days a
Right now I'm doing the only reasonable thing that is left: Switch off
all phones. And to anyone out there intending to contact me: Please
think twice before calling me on the phone. Almost anything can be put
in an e-mail. And if you really want to have a phone call, please
request a scheduled phone call in an e-mail containing a very detailed
agenda and explanation of the topic.
[ /misc |
permanent link ]
Strain of bad luck
From roughly September to December 2012 I seem to have had a quite
unusual strain of bad luck and set-backs. I don't want to go into the
details here, as most of the issues are of quite private nature.
This has kept me quite distracted from a lot of my other activity.
Projects like the various Osmocom sub-projects, gpl-violations.org are
in desperate need of attention, and I have severely neglected my
responsibilities in the Chaos Computer Club Berlin e.V. :(
I don't even want to talk about actual paid work, where customers also
had to put up with repeated schedule slips and lack of availability.
I let down friends and colleagues at a number of occasions, as I was
unable to keep up with anything that remotely resembles my typical work
Last but not least, I regrettably have also not felt much of an urge to
write many blog posts here.
My sincere hope and expectation is that things are going to improve
quickly in 2013. At least most of issues from the last half year have
been resolved. Now I need to work through a considerable back-log of
work and find more time for my volunteer projects in the FOSS and hacker
worlds. However, this will need some time and I would like to ask for
some patience. I do intend to be up to speed with things just like
In this spirit, I am looking forward to a productive and exciting
2013. Happy hacking und Viel Spass am Gerät
[ /personal |
permanent link ]