Osmocom.org GTP-U kernel implementation merged mainline

Have you ever used mobile data on your phone or using Tethering?

In packet-switched cellular networks (aka mobile data) from GPRS to EDGE, from UMTS to HSPA and all the way into modern LTE networks, there is a tunneling protocol called GTP (GPRS Tunneling Protocol).

This was the first cellular protocol that involved transport over TCP/IP, as opposed to all the ISDN/E1/T1/FrameRelay world with their weird protocol stacks. So it should have been something super easy to implement on and in Linux, and nobody should have had a reason to run a proprietary GGSN, ever.

However, the cellular telecom world lives in a different universe, and to this day you can be safe to assume that all production GGSNs are proprietary hardware and/or software :(

In 2002, Jens Jakobsen at Mondru AB released the initial version of OpenGGSN, a userspace implementation of this tunneling protocol and the GGSN network element. Development however ceased in 2005, and we at the Osmocom project thus adopted OpenGGSN maintenance in 2016.

Having a userspace implementation of any tunneling protocol of course only works for relatively low bandwidth, due to the scheduling and memory-copying overhead between kernel, userspace, and kernel again.

So OpenGGSN might have been useful for early GPRS networks where the maximum data rate per subscriber is in the hundreds of kilobits, but it certainly is not possible for any real operator, particularly not at today's data rates.

That's why for decades, all commonly used IP tunneling protocols have been implemented inside the Linux kernel, which has some tunneling infrastructure used with tunnels like IP-IP, SIT, GRE, PPTP, L2TP and others.

But then again, the cellular world lives in a universe where Free and Open Source Software didn't exit until OpenBTS and OpenBSC changed all o that from 2008 onwards. So nobody ever bothered to add GTP support to the in-kernel tunneling framework.

In 2012, I started an in-kernel implementation of GTP-U (the user plane with actual user IP data) as part of my work at sysmocom. My former netfilter colleague and current netfilter core team leader Pablo Neira was contracted to bring it further along, but unfortunately the customer project funding the effort was discontinued, and we didn't have time to complete it.

Luckily, in 2015 Andreas Schultz of Travelping came around and has forward-ported the old code to a more modern kernel, fixed the numerous bugs and started to test and use it. He also kept pushing Pablo and me for review and submission, thanks for that!

Finally, in May 2016, the code was merged into the mainline kernel, and now every upcoming version of the Linux kernel will have a fast and efficient in-kernel implementation of GTP-U. It is configured via netlink from userspace, where you are expected to run a corresponding daemon for the control plane, such as either OpenGGSN, or the new GGSN + PDN-GW implementation in Erlang called erGW.

You can find the kernel code at drivers/net/gtp.c, and the userspace netlink library code (libgtpnl) at git.osmocom.org.

I haven't done actual benchmarking of the performance that you can get on modern x86 hardware with this, but I would expect it to be the same of what you can also get from other similar in-kernel tunneling implementations.

Now that the cellular industry has failed for decades to realize how easy and little effort would have been needed to have a fast and inexpensive GGSN around, let's see if now that other people did it for them, there will be some adoption.

If you're interested in testing or running a GGSN or PDN-GW and become an early adopter, feel free to reach out to Andreas, Pablo and/or me. The osmocom-net-gprs mailing list might be a good way to discuss further development and/or testing.

Slovenian student sentenced for detecting TETRA flaws using OsmocomTETRA

According to some news report, including this report at softpedia, a 26 year old student at the Faculty of Criminal Justice and Security in Maribor, Slovenia has received a suspended prison sentence for finding flaws in Slovenian police and army TETRA network using OsmocomTETRA

As the Osmocom project leader and main author of OsmocomTETRA, this is highly disturbing news to me. OsmocomTETRA was precisely developed to enable people to perform research and analysis in TETRA networks, and to audit their safe and secure configuration.

If a TETRA network (like any other network) is configured with broken security, then the people responsible for configuring and operating that network are to be blamed, and not the researcher who invests his personal time and effort into demonstrating that police radio communications safety is broken. On the outside, the court sentence really sounds like "shoot the messenger". They should instead have jailed the people responsible for deploying such an insecure network in the first place, as well as those responsible for not doing the most basic air-interface interception tests before putting such a network into production.

According to all reports, the student had shared the results of his research with the authorities and there are public detailed reports from 2015, like the report (in Slovenian) at https://podcrto.si/vdor-v-komunikacijo-policije-razkril-hude-varnostne-ranljivosti-sistema-tetra/.

The statement that he should have asked the authorities for permission before starting his research is moot. I've seen many such cases and you would normally never get permission to do this, or you would most likely get no response from the (in)competent authorities in the first place.

From my point of view, they should give the student a medal of honor, instead of sentencing him. He has provided a significant service to the security of the public sector communications in his country.

To be fair, the news report also indicates that there were other charges involved, like impersonating a police officer. I can of course not comment on those.

Please note that I do not know the student or his research first-hand, nor did I know any of his actions or was involved in them. OsmocomTETRA is a Free / Open Source Software project available to anyone in source code form. It is a vital tool in demonstrating the lack of security in many TETRA networks, whether networks for public safety or private networks.

Developers wanted for Osmocom GSM related work

Right now I'm feeling sad. I really shouldn't, but I still do.

Many years ago I started OpenBSC and Osmocom in order to bring Free Software into an area where it barely existed before: Cellular Infrastructure. For the first few years, it was "just for fun", without any professional users. A FOSS project by enthusiasts. Then we got some commercial / professional users, and with them funding, paying for e.g. Holger and my freelance work. Still, implementing all protocol stacks, interfaces and functional elements of GSM and GPRS from the radio network to the core network is something that large corporations typically spend hundreds of man-years on. So funding for Osmocom GSM implementations was always short, and we always tried to make the best out of it.

After Holger and I started sysmocom in 2011, we had a chance to use funds from BTS sales to hire more developers, and we were growing our team of developers. We finally could pay some developers other than ourselves from working on Free Software cellular network infrastructure.

In 2014 and 2015, sysmocom got side-tracked with some projects where Osmocom and the cellular network was only one small part of a much larger scope. In Q4/2015 and in 2016, we are back on track with focussing 100% at Osmocom projects, which you can probably see by a lot more associated commits to the respective project repositories.

By now, we are in the lucky situation that the work we've done in the Osmocom project on providing Free Software implementations of cellular technologies like GSM, GPRS, EDGE and now also UMTS is receiving a lot of attention. This attention translates into companies approaching us (particularly at sysmocom) regarding funding for implementing new features, fixing existing bugs and short-comings, etc. As part of that, we can even work on much needed infrastructural changes in the software.

So now we are in the opposite situation: There's a lot of interest in funding Osmocom work, but there are few people in the Osmocom community interested and/or capable to follow-up to that. Some of the early contributors have moved into other areas, and are now working on proprietary cellular stacks at large multi-national corporations. Some others think of GSM as a fun hobby and want to keep it that way.

At sysmocom, we are trying hard to do what we can to keep up with the demand. We've been looking to add people to our staff, but right now we are struggling only to compensate for the regular fluctuation of employees (i.e. keep the team size as is), let alone actually adding new members to our team to help to move free software cellular networks ahead.

I am struggling to understand why that is. I think Free Software in cellular communications is one of the most interesting and challenging frontiers for Free Software to work on. And there are many FOSS developers who love nothing more than to conquer new areas of technology.

At sysmocom, we can now offer what would have been my personal dream job for many years:

  • paid work on Free Software that is available to the general public, rather than something only of value to the employer
  • interesting technical challenges in an area of technology where you will not find the answer to all your problems on stackoverflow or the like
  • work in a small company consisting almost entirely only of die-hard engineers, without corporate managers, marketing departments, etc.
  • work in an environment free of Microsoft and Apple software or cloud services; use exclusively Free Software to get your work done

I would hope that more developers would appreciate such an environment. If you're interested in helping FOSS cellular networks ahead, feel free to have a look at http://sysmocom.de/jobs or contact us at jobs@sysmocom.de. Together, we can try to move Free Software for mobile communications to the next level!

You can now install a GSM network using apt-get

This is great news: You can now install a GSM network using apt-get!

Thanks to the efforts of Debian developer Ruben Undheim, there's now an OpenBSC (with all its flavors like OsmoBSC, OsmoNITB, OsmoSGSN, ...) package in the official Debian repository.

Here is the link to the e-mail indicating acceptance into Debian: https://tracker.debian.org/news/755641

I think for the past many years into the OpenBSC (and wider Osmocom) projects I always assumed that distribution packaging is not really something all that important, as all the people using OpenBSC surely would be technical enough to build it from the source. And in fact, I believe that building from source brings you one step closer to actually modifying the code, and thus contribution.

Nevertheless, the project has matured to a point where it is not used only by developers anymore, and particularly also (god beware) by people with limited experience with Linux in general. That such people still exist is surprisingly hard to realize for somebody like myself who has spent more than 20 years in Linux land by now.

So all in all, today I think that having packages in a Distribution like Debian actually is important for the further adoption of the project - pretty much like I believe that more and better public documentation is.

Looking forward to seeing the first bug reports reported through bugs.debian.org rather than https://projects.osmocom.org/ . Once that happens, we know that people are actually using the official Debian packages.

As an unrelated side note, the Osmocom project now also has nightly builds available for Debian 7.0, Debian 8.0 and Ubunut 14.04 on both i586 and x86_64 architecture from https://build.opensuse.org/project/show/network:osmocom:nightly. The nightly builds are for people who want to stay on the bleeding edge of the code, but who don't want to go through building everything from scratch. See Holgers post on the openbsc mailing list for more information.

TelcoSecDay 2016: Open Source Network Elements for Security Analysis of Mobile Networks

Today I had the pleasure of presenting about Open Source Network Elements for Security Analysis of Mobile Networks at the Troopers 2016 TelcoSecDay.

The main topics addressed by this presentation are:

  • Importance of Free and Open Source Software implementations of cellular network protocol stacks / interfaces / network elements for applied telecom security research
  • The progress we've made at Osmocom over the last eight years.
  • An overview about our current efforts to implement at 3G Network similar to the existing 2G/2.5G/2.75G implementations.

There are no audio or video recordings of this session.

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2016/telcosecday/foss-gsm.html

Open Source mobile communications, security research and contributions

While preparing my presentation for the Troopers 2016 TelcoSecDay I was thinking once again about the importance of having FOSS implementations of cellular protocol stacks, interfaces and network elements in order to enable security researches (aka Hackers) to work on improving security in mobile communications.

From the very beginning, this was the motivation of creating OpenBSC and OsmocomBB: To enable more research in this area, to make it at least in some ways easier to work in this field. To close a little bit of the massive gap on how easy it is to do applied security research (aka hacking) in the TCP/IP/Internet world vs. the cellular world.

We have definitely succeeded in that. Many people have successfully the various Osmocom projects in order to do cellular security research, and I'm very happy about that.

However, there is a back-side to that, which I'm less happy about. In those past eight years, we have not managed to attract significant amount of contributions to the Osmocom projects from those people that benefit most from it: Neither from those very security researchers that use it in the first place, nor from the Telecom industry as a whole.

I can understand that the large telecom equipment suppliers may think that FOSS implementations are somewhat a competition and thus might not be particularly enthusiastic about contributing. However, the story for the cellular operators and the IT security crowd is definitely quite different. They should have no good reason not to contribute.

So as a result of that, we still have a relatively small amount of people contributing to Osmocom projects, which is a pity. They can currently be divided into two groups:

  • the enthusiasts: People contributing because they are enthusiastic about cellular protocols and technologies.
  • the commercial users, who operate 2G/2.5G networks based on the Osmocom protocol stack and who either contribute directly or fund development work at sysmocom. They typically operate small/private networks, so if they want data, they simply use Wifi. There's thus not a big interest or need in 3G or 4G technologies.

On the other hand, the security folks would love to have 3G and 4G implementations that they could use to talk to either mobile devices over a radio interface, or towards the wired infrastructure components in the radio access and core networks. But we don't see significant contributions from that sphere, and I wonder why that is.

At least that part of the IT security industry that I know typically works with very comfortable budgets and profit rates, and investing in better infrastructure/tools is not charity anyway, but an actual investment into working more efficiently and/or extending the possible scope of related pen-testing or audits.

So it seems we might want to think what we could do in order to motivate such interested potential users of FOSS 3G/4G to contribute to it by either writing code or funding associated developments...

If you have any thoughts on that, feel free to share them with me by e-mail to laforge@gnumonks.org.

Linaro Connect BKK16 Keynote on GPL Compliance

Today I had the pleasure of co-presenting with Shane Coughlan the Linaro Connect BKK16 Keynote on GPL compliance about GPL compliance.

The main topics addressed by this presentation are:

  • Brief history about GPL enforcement and how it has impacted the industry
  • Ultimate Goal of GPL enforcement is compliance
  • The license is not an end in itself, but rather to facilitate collaborative development
  • GPL compliance should be more engineering and business driven, not so much legal (compliance) driven.

The video recording is available at https://www.youtube.com/watch?v=b4Bli8h0V-Q

Slides are available at http://git.gnumonks.org/index.html/laforge-slides/plain/2016/linaroconnect/compliance.html

The video of a corresponding interview is available from https://www.youtube.com/watch?v=I6IgjCyO-iQ

Report from the VMware GPL court hearing

Today, I took some time off to attend the court hearing in the GPL violation/infringement case that Christoph Hellwig has brought against VMware.

I am not in any way legally involved in the lawsuit. However, as a fellow (former) Linux kernel developer myself, and a long-term Free Software community member who strongly believes in the copyleft model, I of course am very interested in this case - and of course in an outcome in favor of the plaintiff. Nevertheless, the below report tries to provide an un-biased account of what happened at the hearing today, and does not contain my own opinions on the matter. I can always write another blog post about that :)

I blogged about this case before briefly, and there is a lot of information publicly discussed about the case, including the information published by the Software Freedom Conservancy (see the link above, the announcement and the associated FAQ.

Still, let's quickly summarize the facts:

  • VMware is using parts of the Linux kernel in their proprietary ESXi product, including the entire SCSI mid-layer, USB support, radix tree and many, many device drivers.
  • as is generally known, Linux is licensed under GNU GPLv2, a copyleft-style license.
  • VMware has modified all the code they took from the Linux kernel and integrated them into something they call vmklinux.
  • VMware has modified their proprietary virtualization OS kernel vmkernel with specific API/symbol to interact with vmklinux
  • at least in earlier versions of ESXi, virtually any block device access has to go through vmklinux and thus the portions of Linux they took
  • vmklinux and vmkernel are dynamically linked object files that are linked together at run-time
  • the Linux code they took runs in the same execution context (address space, stack, control flow) like the vmkernel.

Ok, now enter the court hearing of today.

Christoph Hellwig was represented by his two German Lawyers, Dr. Till Jaeger and Dr. Miriam Ballhausen. VMware was represented by three German lawyers lead by Matthias Koch, as well as a US attorney, Michael Jacobs (by means of two simultaneous interpreters). There were also several members of the in-house US legal team of VMware present, but not formally representing the defendant in court.

As is unusual for copyright disputes, there was quite some audience following the court. Next to the VMware entourage, there were also a couple of fellow Linux kernel developers as well as some German IT press representatives following the hearing.

General Introduction of the presiding judge

After some formalities (like the question whether or not a ',' is missing after the "Inc." in the way it is phrased in the lawsuit), the presiding judge started with some general remarks

  • the court is well aware of the public (and even international public) interest in this case
  • the court understands there are novel fundamental legal questions raised that no court - at least no German court - had so far to decide upon.
  • the court also is well aware that the judges on the panel are not technical experts and thus not well-versed in software development or computer science. Rather, they are a court specialized on all sorts of copyright matters, not particularly related to software.
  • the court further understands that Linux is a collaborative, community-developed operating system, and that the development process is incremental and involves many authors.
  • the court understands there is a lot of discussion about interfaces between different programs or parts of a program, and that there are a variety of different definitions and many interpretations of what interfaces are

Presentation about the courts understanding of the subject matter

The presiding judge continued to explain what was their understanding of the subject matter. They understood VMware ESXi serves to virtualize a computer hardware in order to run multiple copies of the same or of different versions of operating systems on it. They also understand that vmkernel is at the core of that virtualization system, and that it contains something called vmkapi which is an interface towards Linux device drivers.

However, they misunderstood that this case was somehow an interface between a Linux guest OS being virtualized on top of vmkernel. It took both defendant and plaintiff some time to illustrate that in fact this is not the subject of the lawsuit, and that you can still have portions of Linux running linked into vmkernel while exclusively only virtualizing Windows guests on top of vmkernel.

The court went on to share their understanding of the GPLv2 and its underlying copyleft principle, that it is not about abandoning the authors' rights but to the contrary exercising copyright. They understood the license has implications on derivative works and demonstrated that they had been working with both the German translation a well as the English language original text of GPLv2. At least I was sort-of impressed by the way they grasped it - much better than some of the other courts that I had to deal with in the various cases I was bringing forward during my gpl-violations.org work before.

They also illustrated that they understood that Christoph Hellwig has been developing parts of the Linux kernel, and that modified parts of Linux were now being used in some form in VMware ESXi.

After this general introduction, there was the question of whether or not both parties would still want to settle before going further. The court already expected that this would be very unlikely, as it understood that the dispute serves to resolve fundamental legal question, and there is hardly any compromise in the middle between using or not using the Linux code, or between licensing vmkernel under a GPL compatible license or not. And as expected, there was no indication from either side that they could see an out-of-court settlement of the dispute at this point.

Right to sue / sufficient copyrighted works of the plaintiff

There was quite some debate about the question whether or not the plaintiff has shown that he actually holds a sufficient amount of copyrighted materials.

The question here is not, whether Christoph has sufficient copyrightable contributions on Linux as a whole, but for the matter of this legal case it is relevant which of his copyrighted works end up in the disputed product VMware ESXi.

Due to the nature of the development process where lots of developers make intermittent and incremental changes, it is not as straight-forward to demonstrate this, as one would hope. You cannot simply print an entire C file from the source code and mark large portions as being written by Christoph himself. Rather, lines have been edited again and again, were shifted, re-structured, re-factored. For a non-developer like the judges, it is therefore not obvious to decide on this question.

This situation is used by the VMware defense in claiming that overall, they could only find very few functions that could be attributed to Christoph, and that this may altogether be only 1% of the Linux code they use in VMware ESXi.

The court recognized this as difficult, as in German copyright law there is the concept of fading. If the original work by one author has been edited to an extent that it is barely recognizable, his original work has faded and so have his rights. The court did not state whether it believed that this has happened. To the contrary, the indicated that it may very well be that only very few lines of code can actually make a significant impact on the work as a whole. However, it is problematic for them to decide, as they don't understand source code and software development.

So if (after further briefs from both sides and deliberation of the court) this is still an open question, it might very well be the case that the court would request a techncial expert report to clarify this to the court.

Are vmklinux + vmkernel one program/work or multiple programs/works?

Finally, there was some deliberation about the very key question of whether or not vmkernel and vmklinux were separate programs / works or one program / work in the sense of copyright law. Unfortunately only the very surface of this topic could be touched in the hearing, and the actual technical and legal arguments of both sides could not be heard.

The court clarified that if vmkernel and vmklinux would be considered as one program, then indeed their use outside of the terms of the GPL would be an intrusion into the rights of the plaintiff.

The difficulty is how to actually venture into the legal implications of certain technical software architecture, when the people involved have no technical knowledge on operating system theory, system-level software development and compilers/linkers/toolchains.

A lot is thus left to how good and 'believable' the parties can present their case. It was very clear from the VMware side that they wanted to down-play the role and proportion of vmkernel and its Linux heritage. At times their lawyers made statements like linux is this small yellow box in the left bottom corner (of our diagram). So of course already the diagrams are drawn in a way to twist the facts according to their view on reality.


  • The court seems very much interested in the case and wants to understand the details
  • The court recognizes the general importance of the case and the public interest in it
  • There were some fundamental misunderstandings on the technical architecture of the software under dispute that could be clarified
  • There are actually not that many facts that are disputed between both sides, except the (key, and difficult) questions on
    • does Christoph hold sufficient rights on the code to bring forward the legal case?
    • are vmkernel and vmklinux one work or two separate works?

The remainder of this dispute will thus be centered on the latter two questions - whether in this court or in any higher courts that may have to re-visit this subject after either of the parties takes this further, if the outcome is not in their favor.

In terms of next steps,

  • both parties have until April 15, 2016 to file further briefs to follow-up the discussions in the hearing today
  • the court scheduled May 19, 2016 as date of promulgation. However, this would of course only hold true if the court would reach a clear decision based on the briefs by then. If there is a need for an expert, or any witnesses need to be called, then it is likely there will be further hearings and no verdict will be reached by then.

Software under OSA Public License is neither Open Source nor Free Software

It seems my recent concerns on the OpenAirInterface re-licensing were not unjustified.

I contacted various legal experts on Free Software legal community about this, and the response was unanimous: In all feedback I received, the general opinion was that software under the OSA Public License V1.0 is neither Free Software nor Open Source Software.

The rational is, that it does not fulfill the criteria of

  • the FSF Free Software definition, as the license does not fulfill freedom 0: The freedom to run the program as you wish, for any purpose (which obviously includes commercial use)
  • the Open Source Initiatives Open Source Definition, as the license must not discriminate against fields of endeavor, such as commercial use.
  • the Debian Free Software Guidelines, as the DFSG also require no discrimination against fields of endeavor, such as commercial use.

I think we as the community need to be very clear about this. We should not easily tolerate that people put software under restrictive licenses but still call that software open source. This creates a bad impression to those not familiar with the culture and spirit of both Free Software and Open Source. It creates the impression that people can call something Open Source but then still ask royalties for it, if used commercially.

It is a shame that entities like Eurecom and the OpenAirInterface Software Association are open-washing their software by calling it Open Source when in fact it isn't. This attitude frankly makes me sick.

That's just like green-washing when companies like BP are claiming they're now an environmental friendly company just because they put some solar panels on the roof of some building.