Harald Welte's blog

Leaving OpenMoko "Lead System Architect" position

The regular reader of this blog will have noticed a distinct lack of any OpenMoko related news. This is not a coincidence. As Mickey wrote in a blog entry, there has been quite a bit of internal friction lately.

Adding that to the enormous amount of stress over the last 18 months has made me feel quite a bit demotivated over time. I've tried to cut down on the amount of work I do, but it hasn't helped much. So now I'm at a point where I feel unable to work in any active/leading role inside the project and/or company. My deep apologies to the the project and its community.

But don't worry. OpenMoko is a team, and I'll do everything to help smoothen the transition. I'm more than willing to assist those who will take care of my various tasks.

From today on, I'm nothing more than a volunteer to the project. I'll likely continue a bit of hacking in my areas of personal interest. Just like in many of the other FOSS projects that I have been (or still am) involved.

All the best to the OpenMoko project, the OpenMoko company, FIC Mobility. The last 18 months was an intense experience. Thanks to everyone who has helped the project both inside FIC/OpenMoko and outside. Thanks to FIC for funding and supporting the project. I wish everyone the best, and I'll be the most happy person (next to Sean) at each and every milestone the project achieves in the future.

Heading back to Germany

So, after roughly two weeks of OpenMoko Taipei headquarters, I'm now heading back to my Home+Office in Berlin. And I'm really looking forward to it. During the last couple of months I've tried my best to help the transition from OpenMoko a a project inside a FIC business unit to OpenMoko, Inc. the independent company inside the FIC Group. I've helped with tons of things that are definitely by no means related to kernel/bootloader development, or even the hardware architectural planning that I've been heavily getting involved starting with GTA02.

So now it is a good time to finally focus again on what my actual and original task is: Software development. This can be done much better remotely, so I'll expect to be able to work way more from Berlin. Which makes me happy, since it always was and still is my favorite city. And I definitely missed it a lot during the last year of intensive OpenMoko work. Now I'm on my way back, and I'm looking forward to spending more time with my friends, the CCC Berlin. Being able to go to concerts and clubs that play music I actually like. Being able to work on finally improving (or rather: finishing) home improvement in my apartment, and many other things.

So don't get this wrong. I'm very much continuing my technical work for OpenMoko, and as the first developer on this entire project and a OpenMoko core team member, I'm always going to maintain an influential role in the project. But finally, I can go home, feel better, work more focused and efficiently, and improve the technical quality of our products even more :)

Since OpenMoko now actually has three full-time paid project members in Berlin, It's also going to be nice to closer cooperate with them. (or co-work, like the Chinese English speaking would say)

Back to Taipei

Today I got back to Taipei, almost three weeks later than originally anticipated. More news after at least one night of full sleep and the first day at the office...

Two days of intense u-boot hacking

After finishing most of the basic device support for GTA02v2 in both kernel and u-boot during the last week, I've finally turned back to implementing one of the longest standing issues in u-boot: GSM passthrough.

GSM passthrough allows you to basically ignore the smartphone part of the device and connect the GSM modem more or less directly to a host PC. The feature has been long known by various smartphone hacker projects such as e.g. OpenEZX (which as a side note has made quite some progress recently, much appreciated by me as retired project founder).

So GSM passthrough is mainly useful for rapid development (developing gsmd more efficiently by running it on the developers workstation, without cross-compiling and ipkg installs), but also if you want to use some legacy application that was written at a time where a phone really only was a phone (e.g. sim card managers, ...)

Now the GSM passthrough was always pushed back on the TODO list, since our usbtty code in u-boot was never very reliable and caused lots of data corruption such as bogus and/or missing characters. Quite useful for the human operator, but definitely not acceptable for getting a program with AT command parser to work. So that had to be fixed first (and it is now fixed).

As I pointed out in my announcement, the generic way of implementing this feature has actually quite interesting but much more obscure use cases such as dialling from a landline via GSM (CSD call) into your Neo, manually accepting the incoming call and then attaching the u-boot command line to it. That's sort of the feature you have on hosted/colocated servers, when you use a boot-loader with serial console support and attach a modem or terminal server to it.

So does this mean the Neo1973 is now ready for the enterprise? Not quite. Even though it has a built-in UPS (called battery), and GTA02 will even allow you to change the battery without shutting down the device, resulting in higher availability ;)

But then, the expectations / requirements for mobile communications devices are quite a bit different from that world. But the hackers community likes those kind of strange features. Have you ever heard of another smartphone with that capability?

Oh, and before I get any complaints about the security: This "feature" has to be explicitly enabled and every call manually accepted by typing a sequence of commands into the u-boot command line. So unless the attack involves tons of social engineering (getting the device owner to do all those things) there's not that much of a big deal. But maybe we should start to think of some kind of user authentication for u-boot now *rotfl*.

OpenMoko Taipei office network setup

For the last three days I've been busy setting up the network in the new OpenMoko office. Finally something that just works, without any major flaws. That's probably because the involvement of external entities is fairly small. But looking at it closer, actually exactly there the problems start. FIC purchasing e.g. only bought 50% of the switch equipment that I asked for, something that I still don't yet have received any details about, neither some kind of apology.

In any case, the network is up and running. We now have a pretty uncommon heterogeneous combination of Cisco, Dell and even some D-Link switches, with the core switch being a pretty impressive Dell 6248. The 10GBit transceivers are still missing, so the backbone is just running 1GBit for the time being.

There's a total of about 190 Ethernet access ports throughout this new office, and we have various different broadcast domains, safely separating guest network, office network, r&d network, ... from each other.

The server room also looks quite nice now, with five 19" full-height 19" racks, 2 layers of UPS (building-wide and extra small UPS in front of servers), with three fairly big servers in operation, and eight more pending to be used soon.

The Internet uplink is a 100MBit capable physical layer (fiber optics), of which we only subscribed to 8MBit initially. But one phone call (plus additional transfer of money) later, we can bump the speed to any rate below that 100MBit physical layer limit.

The operation of our own (netfilter/iptables based, what did you think?) firewall now also brings us back the long-missed basic fundamental networking needs of any free software hackers (called luxury around here) of all our developers being able to

access the web without restrictions and blocked sites
finally use IRC!
access external IMAP4 servers
have SSH and other interactive sessions not time out every couple of minutes

Progress with the new OpenMoko and FIC Mobility office

The final 24 hours of my current Taipei trip have started. This is a good time to reflect on what has happened in those last weeks since July 9.

As with the overall status of the project, I'm still extremely dissatisfied. The frequent reader of this blog will have noticed the last postings on this subject, full of discontent.

So the further we are into this project, the more time we put into it - the further I expect it to produce anything that I would consider reasonable results. Please don't confuse this with the commercial success, or the ability to produce working products. This is an entirely different matter.

To me, it is extremely important to do things systematically, with lots of planning, safeguards, checks, verifiable and reproducible processes, as much automatization as possible, little room for human error, etc. So as long as not everything from hardware to software development, mass production, production testing, distribution/logistics/sales, etc. follow a well-thought-through process, I will not be happy with the results. Because any such "results" are more or less the mere product of luck or randomness, and not a trustworthy basis upon which we can rely on.

So reflecting on those past weeks, I think the following things have made humble and moderate progress:

GTA02v2, the second prototype generation of GTA02 was finalized after many issues including unavailability of key components. I'm more than looking forward to see how it turns out
DebugBoard v3, the third version of the Neo1973 Debug Board was finalized and is actually also verified and can go in mass production
Our internal software team finally has proper leadership and guidance from somebody who is both Taiwanese and has a thorough understanding of Free Software: jserv
The new, second (intermediate) generation user interface was implemented and released. It's implemented mainly by O-Hand, since embarrassing as it is, we still don't yet have managed to build a proper internal software development team.
The first batch of Neo1973 GTA01 was sold, though with a entirely last-minute, error-prone and way-too manual process for order, payment and logistics.
We have found a capable sysadmin for our hosted, publicly-accessible servers. More news about that in September.
We have managed to find a extremely valuable senior technical person for our graphics driver and low-level UI work. This, too, will make big news in September.
The FiWin (FIC wireless networks) company, home to the team working on the it-exists-but-nobody-publicly-knows-what-it-is HXD8, was merged into OpenMoko and FIC Mobility
We have finalized the specification for the workstations of our software developers. It's incredibly complex to find something that's compliant with our requirements (mainboard with Intel 945/965 on-board graphics, Ethernet chip != attansic/realtek, dual core CPU with 2x2048kByte cache) in Taiwan. So now our developers will all get a Q6600 CPU (what nonsense!). I've tested it, and it compiles the GTA01 kernel in 1.59minutes. Guess they'll be happy about that.
Realize how many things really are fundamentally wrong internally. What we knew so far about our inheritance was just the beginning ;)

One major thing that finally started to move forward, with something like four to six weeks of completely unnecessary delays, is the new office. After it was decided that we will split FIC MCBU into the independent OpenMoko, Inc. and FIC Mobility (aka Mobile Communications), we also decided to move into bigger, scalable and independent offices.

To our big luck, two thirds of the 7th floor in the FIC headquarters were currently unused, and they're now undergoing quite a bit of renovation and reconstruction. Walls have been removed and brought in, floors have been properly removed and new ones laid - after days of fighting by Sean and myself.

The networking and phone cables get a major overhaul and will be tested. I've also seen the AC for the new OpenMoko server room being brought in. The contract for our own Internet plink has been finalized. The fiber will be put in place within the next week. The core switches have been configured, but we're still fighting very hard to get those damn 10Bit XFP transceivers from Dell.

So the current schedule is to move on August 17, one day after I'll be back from Germany. If that works out, I could spend the weekend 18/19 for doing the final network/server/router/firewall/... configuration.

Obviously, all of this causes quite significant resource drainage for everyone involved. But it's a more than necessary step forward to building an environment that we can actually work in. An environment where our developers have real Internet access, can join IRC channels, and can get in touch with the OpenMoko community without the obstacle of strange corporate policies. An environment where we can have a 'clean start', even in the most literal meaning of the word :)

So all in all, bear with us, have patience. The revolution might take significantly longer than anticipated. But we're still busy doing whatever it takes to get us to the product that the OpenMoko core team set out to build.

Sick but not insane (yet)

Some people were troubled by my last posting about 'insanity'. And I have to admit it is justified. It isn't nice to pull internal issues of some company out into the public that way. But I just couldn't do differently anymore. FIC should think about providing a free corporate psychiatrist for us. ;-) I think I have a fairly big tolerance when it comes to mishaps, incompetence and cluelessness, but some of the things we're experiencing here are just not bearable. Especially not if they lack any kind of rational explanation.

So my latest outburst was mainly related to the fact that despite being with flu and fever in bed, I had to personally hack that embarrassing little online shop we now have at https://direct.openmoko.com/ [my first perl CGI code in about ten years!!]. Can you believe it, we just did not have (and still don't have) anyone in this project who has ever done some real work on CGI's or 'technical web things' at all. And if this was not enough yet, the requirements kept constantly changing all the time, up to this day, more than one week after the webshop opened.

So for gods sake, how many months have we been knowing that at some point we need to ship? And then you have plenty of people who produce over many months three entirely different concepts on how to shop and distribute those devices. And in the end, you have something with a Chinese-only credit card processing page, no proper setup with regard to the carrier (UPS in our case), tons of people talking vapor, but from what I've heard drawing nice diagrams. And no working solution.

Some people might wonder why those shipping rates in the shop are so high. Shouldn't a large company like FIC get huge discounts? Yes, they do. But believe it or not, FIC does up to this day not know in advance how much a shipping costs. Only after it was made. UPS has something like a Rates and Service Selection API. UPS has also a nice and detailed manual on how this XML API reports those rebated 'negotiated rates' in addition to the standard rates.

But then till this very day, UPS keeps claiming that such an API does not exist. Despite the fact that off-the-shelf e-commerce applications _support_ this API, and despite the fact that UPS' very own API documentation goes into every detail describing how that information is XML-encoded. What comes to my mind is the O'Really: Distributing Clue to users in a slightly modified version: Distributing clue to UPS' own sales representatives. How on earth can you get or keep a job there if you haven't even read the company's own documentation on their products?

Then you have companies like WorldPay, who very broadly advertise the many different payment variants they accept, not only all the credit cards known to man, but even things like direct debit (Lastschrift) in Germany. But do you believe they are able to process American Express for us? No, obviously not. Customers located in Taiwan (like FIC/OpenMoko) cannot process AmEx. No explanation given. So much for the word WORLD in WorldPay.

Or would you believe that some large bank like Citibank (Taiwan) was able to charge credit cards (VISA/MasterCard) in USD? No, obviously, being in Taiwan they can only charge in NT$ (New Taiwan Dollars). WTF? It's the 21st century, and there are dozens of online credit card acceptance partners that allow you to bill in about any currency you want. But not a world-class bank like Citibank.

And now you might think that we're actually no longer working on development. That is wrong. Yet another funny story from the MokoUniverse is that one of the worlds largest semiconductor distributors just had a long meeting yesterday, with FAE's and tons of FIC hardware engineers to crack their heads about the question whether or not we can use a "new" silicon revision of a certain component, and where exactly the differences between the old and new revision might be. I refused to participate in such a meeting, indicating that I just want a document describing those differences. In writing. Soon after that, I find out that we have always been using that supposedly-new revision, for the better part of one year. Nobody actually bothered to look at a unit of hardware, or at all the high-res photographs of GTA01 that I posted on people.openmoko.org ages ago.

Then lets get to another of my favorites. Purchasing. I wrote an extensive list of networking gear that we desperately need in order to crate the internal IT environment for our fast growing new OpenMoko company. After many delays, today I finally got our two core switches. And what did they do? They "forgot" to order the transceivers, even though they were explicitly listed in the requirements/order list. God knows what happens in the heads of such people. So I'm now back to once again just mail-ordering things from Germany and then billing FIC for the expenses.

And to not bring up more embarrassing facts, and to at least preserve some level of confidentiality, I'm now going to stop. But believe me, there are _much_ more insane stories to be told. With some luck, at some distant point in the future, I'll get permission to publish them in my memoirs.

But that's basically the kind of thing that drives me close to insanity. Whether for that cause, or completely unrelated: My health is actually quite bad recently. Maybe the lack of regular sleeping hours, let aside regular intake of food and the tons of stress have contributed to the flu that lead to a severe fever attack today. You know there's something wrong if at 33 centigrade in the sun, you're shivering more than when riding your motorbike at -13 centigrade in German winter.

So I'm going to take it slowly for the time being, working much from what has now become my 2nd home (apartment in Taipei, provided gratuitously by FIC for people like Werner and myself, who are spending so much time over here).

My sincere apologies in advance. But given my multitude of roles/hats and functions in the OpenMoko project, I bet my health issues will now contribute to some further delays in about any area of the project. But there's little I can do. One day, we will get there. Let's hope you're still interested in our products at that point.

Insanity

If you want a status update on OpenMoko: I'm at a point where I won't go to the office again because I know the agression inside myself will turn into physically hurting someone there.

An update from the OpenMoko world

As Sean has now made his latest OpenMoko Announcement last night, this is a good point in time for me to write some more bits. Up to this announcement it was hard for me to publicly state anything, since the whole internal restructuring process has been underway, but not yet publicly announced.

I can only join Sean in his assessment about the superb support of FIC's senior management. They are providing us with the kind of resources we wanted.

But being the realist (well, Sean as optimist would call me pessimist, you know that old story), I also see severe challenges both right now, and ahead.

Whatever you might think: I bet that none of you has not the slightest idea about how many problems the OpenMoko team is fighting all day long. If you're thinking "what's the problem with a purely technical task, i.e. designing hardware and software for an Open phone"? Then my reply would be: It's not that big of a technical problem.

However, the really big issues start as soon as you leave the R&D world. On the one hand, there is the actual hardware production. Many components have incredible lead times (3 months or more), and our yet sort-of-unknown-but-initially-very-low quantities are not particularly helpful either. Any .tw OEM/ODM thinks in different terminology. The kind of production processes, shipping infrastructure, ... is just not meant for low-volume and direct shipment. We obviously knew this from the beginning, but everyone just happily works in their usual mode of operation, ignoring our concerns for many months.

Then think about the various customs / legal / trade issues. If you ship components from Taiwan to mainland china and use them to manufacture a product there, you need a special import license in order to get those products back into Taiwan. This license costs money (and, most of all: Time).

Or another example is the lack of double-tax agreements between Taiwan and the rest of the world. So payments to all our various external consultants all over Europe are taxed twice: Once in Taiwan, a second time in the respective country of residence.

For the last two weeks I have been working on finalizing the floor plan and infrastructure planning for the new FIC Mobile Communications and OpenMoko software groups offices in Taipei. And believe it or not, it is a very long and time consuming fight to ever get what you actually want. We know exactly what kind of servers, switches and routers we want. We know to which height we want to reduce the cubicles. We know what kind of Internet uplink we want. Still, it's close to impossible to get anything done. People will just outright refuse to do what they are asked (and paid!) to do.

Take our new servers as one minor example. You would assume that it is no problem at all to configure high-end servers around here. When doing this in Germany, I usually consult one of the many mailorder stores, go through their extensive list of mainboards and other components, select products based on their availability, price and features, and within 24hours I have everything delivered to my doorstep. 99% of those components are from Taiwanese companies.

Now enter Taiwan. First of all, you will discover that the concept of mailorder or extensive online product lists doesn't exist. "Taiwanese people don't trust e-commerce", is what they tell me. Secondly, you can't just call those places and ask them if they have a certain product, since apparently they would always say yes, only to get you into their store.

If you actually get into the various stores, you will see that almost all of the products you want are not available locally. "Not sold into the Taiwan market" is something that you hear very often. So e.g. the choice of Socket 478 mainboards from ASUS goes down from 52 (German online store) to something like 15-20.

So in the end we were really unable to find anything remotely decent (good performance, chipsets with excellent free software support) locally and I ended up importing Asus and Tyan mainboards from Germany into Taiwan, while buying the other components in Taiwan.

Now I could continue and name dozens of examples like this. If this project was just about _developing_ hardware and software, I would be a happy man, and we could look ahead to complete one device after the other. But it's all the other issues, administrative, political, cultural, sales, finance, accounting, shipping, ... which make people like Sean and me run at something like 20-25% of their usual efficiency, despite putting in at least 180% of regular working hours. And there is nobody who can help this, because nobody non-technical really understands what we're doing here, and why we need to do it different than whatever they might have done it before.

Visiting FIC's factory for GTA01 mass production in Suzhou

Yesterday I was on a ten-hour trip from Taipei to Suzhou in mainland China. It took about ten hours for something like a 300km line-of-sight distance, since the Taiwan/China political dispute over The Three Links doesn't allow any direct flights. So we had to go from Taipei to Hong-Kong, transfer onto a different flight to Shanghai, and then start for a couple of hours car ride to Suzhou.

The trip was quite impressive, especially since I have seen a lot more of Shanghai then during my summer 2006 trip to the FIC Shanghai branch. In fact, the sight of so many [strangely-looking] dense, high-rise apartment buildings with 25, 30 or more floors in the suburbs reminded me quite a lot of the classic 1927 Metropolis movie. This impression was probably further enhanced by the thick clouds of smog covering all of the sky, resulting in even non-grey objects look grey, giving the impression of a more-or-less gray-scale world...

In any case, let's not deliberate more about my general thoughts about China, Shanghai or Suzhou at this point, but get to the actual work: Today I spent at the FIC Suzhou factory, mainly doing final QA/Testing of the first 300 Neo1973 GTA01 phones. While I was doing this, another 192 phones went through assembly, resulting in a total of 492 units available at this time. We have another 500 units pending throughout the next two weeks.

The overall quality of my QA checks was quite good. The factory is doing a good job, and we could not detect any production-introduced bugs. The tools provided to the factory for programming/testing of the hardware leave quite a bit of room for optimization though. Will have to start this optimization process next week, after my return to Taipei.

This also means that we can finally make another announcement about the overall project status very shortly. And it means that as soon as the web-shop is up and running, developers will finally be able to purchase Neo1973 GTA01. 8 months too late. Sorry for that. Too much politics and too little actual technical work. All fixed now. Bright future ahead :)

Everyone is busy at OpenMoko

A number of people keep asking me what's going on with regard to openomko. I've even virtually stopped to update this blog quite some time ago. As much as I regret the lack of updates: Be assure they're just a sign of how busy everyone involved is.

I have, in fact, even cancelled my already-accepted paper and corresponding presentation at OLS this year :( I'm also not speaking at any other 'traditiona' event this year, not at Linuxtag, Linux-Kongress, CLUC, LTC, 0sec. Sorry, guys, maybe next year again.

I can't publicly state what's going on with regard to OpenMoko internally, but let me assure you: Good things are happening. We're working on a lot of internal changes that should enable us to approach the project with way more bandwidth.

The first couple of hundred GTA01Bv4 phnes have been produced by the FIC's mass production factory in mainland china. I'll personally do QA on 10% of those phones throughout the second week of June. We want to make sure we don't have any mishaps with our first customers, do we?

The first generation GTA02 prototypes have also showed up at FIC in Taiwan. More news on that at some later point :)

Introducing patchwork to the openmoko mailinglists

Some time ago, Jeremy Kerr wrote patchwork, a tool to semi-automatically keep track of patches submitted to mailinglists.

So far, it was mainly being used for the linuxppc and netfilter project with mixed results. I guess in both projects, in the end, nobody raelly maintained the patch status and stuff just ended up to get stale.

Now I've started an patchwork installation for OpenMoko, at least for the most common public mailinglists. So why do I think patchwork will be used more productively and receive better maintenance? Well, firstly, the number of patches on those lists is still quite small. Secondly, the number of people looking into reviewing those patches, within their primary paid-for working time, is relatively large.

I really believe that patchwork can provide an enormous benefit to a project. Let's hope we manage to use it correctly :)

Returning from FIC HQ / Taipei

Just returning from one of the probably busiest weeks in my life. The entire week was spent with meeting lots of FIC staff. Finally I'm able to connect faces to the members of the hardware and production testing team located in Taipei.

Significant time was spent talking to vendors of WiFi chipsets over the last week. The choice seems to have boiled down to designs around Atheros AR6K or Marvell 8686. The AR6K driver code is completely public for quite some time, even though it (and the mvista SDIO code it depends on) might need a bit of cleanup. From Marvell we yet have to find/receive the GPL licensed driver source code - at least from our [high level official] marvel contacts we have received positive confirmation. So actually, for now, only AR6K is a 'known possible' choice, whereas Marvell might become a choice, once we see the source :)

The other big task was sitting down with Werner Almesberger and doing the system level design for the next major hardware revision, and discussing this design with the hardware team. A lot of things are still in flux. But at least the potential of it is _really_ promising. Details are to be revealed at their appropriate time.

I've also had the chance to briefly meet with senior management such as the head of the mobile communications business unit, as well as the CEO and the chairman of the board of FIC. Everybody seems to be really excited and looking forward to the time ahead, now that we have identified many of the problems in the hardware design, production quality and internal processes and are heading to a much brighter future.

Another interesting opportunity was to present at Taiwan NTU University on the 'correct' way of doing Linux development, both technically and from the GPL / policy point of view. Let's hope we now have a couple of more software developers who will know these things before entering the industry :)

In retrospective, I should have done this trip way earlier on. But then, many things have changed over the last nine months, and it might have been an entirely different experience.

Taiwan/Taipei really seems to be an incredibly interesting place. A world of never-ending possibilities in the field of computer hardware. An industry that can produce about any device of your dream - but doesn't since it seems to be locked too much into either the master/slave role of traditional OEM/ODM business - or into producing the same things like all of their competitors, without really trying too much new/exceptional things.

So it seems, after all, my good intentions about travelling less in 2007 seem to be vaporizing sooner than I would have liked. I guess the productivity gain of me being in Taipei at least from time to time is worth it...

One-week trip to Taipeh, visiting FIC.

On rather short notice I am now on my way to Taipeh, Taiwan. As the interested reader of this journal will have noticed, OpenMoko and the Neo1973 are the seemingly endless story of delays and mishaps throughout the last nine months.

So the purpose of this trip is to discuss various options on how to make sure that we all learn the neccessary lessons out of this past experience, and ensure that OpenMoko eventually can at least keep up to some of its schedule promises.

So this will be a week full of meetings and discussions mostly with various managers and particularly the hardware team at FIC, our generous sponsor.

What is at least equally exciting are the planned meetings with various WiFi chipset vendors to discuss our requirements of a mobile-phone compatible low-size, low-power WiFi chipset with a fully GPL licensed Linux kernel driver.

The downside of all this is that I'm once again held back from writing some of the most important infrastructure code that OpenMoko still needs. Anyway, this compromise has to be made - after all, of what use is software without a high-quality, reliable, performant and available handset hardware :)

Since Werner doesn't have a blog, old-fashioned as he is, let me mention that he'll be there for the whole week, too. I think it's a 34-hour trip for him to get from Buenos Aires to Taipeh. I wonder how well he'll survive that...

Phase 0 software build

Whenever you think something is bad, it will come worse. We've had another hardware outage of one of our buildhosts during the creation of the images that were to be prepared for the "Phase 0" devices. Luckily, the build system is easy to set up on various machines, and we actually now have the build system on four hosts :)

It was a lot of stress to resolve the most important issues just before flashing those 50 phones. All kinds of last-minute problems showed up, such as a certain kernel module missing in the rootfs, etc.

Anyway, we managed to finalize it early Friday morning, and Friday + Saturday engineers in Taiwan used Werners fabulous devirginator to install u-boot, environment, splash screen, kernel and rootfs.

Unfortunately, due to all the delays we have been suffering, all those initial recipients will receive is a very basic Linux distribution:
We have hardware support for all the components in the boot loader and kernel. The device now has a boot menu that allows you to switch to serial or USB=tty console. It will happily boot into Linux, where it starts the X11 (kdrive) server and starts matchbox and looks like a pretty standard GPE desktop.

Suspend/Resume is now working from the software side, i.e. you can suspend the Linus system into RAM and get back from it, all drivers are coming back into operation nicely. You can access back light brightness, battery charger state, Bluetooth, etc. The easiest way to work on the device is currently by using USB Ethernet emulation and then ssh'ing into dropbear that gets started during bootup.

So you might think ok, I'll get a fully open source Linux PDA with super-bright display. But I wanted a phone!
We still have lots of stability issues with our native OpenMoko applications currently, so there will not be a working/stable "dialer" application yet. Instead, if you really want to try it, you can either manually use GSM, or use to

So yes, we are delayed. Still, we have decided to ship those units to allow more people to hack on it as early as possible. The foundation is there, and it so far appears to be quite stable, too.

Also, software can always be updated, especially now that we have USB DFU support.

And now, finally, after a long time, I am looking forward to touch the GSM code again. Let's hope there are not too many urgent issues interrupting me next week. After all, we want SMS and GPRS support rather sooner than later :)

A day full of new hardware problems

It wasn't sufficient enough that our main build server had memory corruption yesterday (in which case no RAID1 will help, because the buffer cache data is corrupt and gets written to both disks corrupt).

Today, I had the pleasant experience of finding something like three more or less independent severe hardware design bugs in the Neo1973. I know this is really sad news for those of you who are eagerly waiting for their "Phase 0" devices. But firstly, you have to understand how sad _we_ are about all this. Even more so, specifically, how sad I am, personally... [now working 14hours straight on this issue].

I was not involved in the early hardware design of the Neo1973, and was hired as a pure systems level software guy. Over the progress of this project, I've been involved more and more in hardware fixes / reviews / redesign. And it's been only now that I've had a more detailed look at the suspend/resume/wakeup related bits. Given the previous series of hardware bugs I should have probably been more cautious and thoroughly review the whole design from the beginning, but then: It is complex, time consuming, and I'm no hardware engineer either, just joe random hacker.

The good news is that we are able to fix all this in the next version (GTA01Bv4), and that there are likely-to-be-working hot-fixes for the already-produced Phase-0 devices.

Originally USB DFU support for u-boot was supposed to be finished yesterday. Now with those two days full of most serious hardware problems (server, prototype), I wonder what's going to prevent me from working on DFU tomorrow.

OpenMoko now runs 2.6.20

Despite the much-feared genirq and workqueue changes, it turned out to be way easier to merge our patches to 2.6.20 maintain than reviewing and back-porting all the relevant bug fixes from 2.6.20 down to our old 2.6.17.14 based system.

We probably wouldn't have been able to do this if Phase-0 wasn't held back due to Bluetooth hardware problems. So everything seems to have its positive side, too :)

Ben Dooks (S3C2410 Kernel Port Maintainer) has already picked some of our patches and is merging them, which is good. He also fixed a s3c2410fb bug in vanilla 2.6.20 which I discovered [and just worked around by porting s3c2410fb from 2.6.17.14. into 2.6.20, lazy as I am]

Today, I spent much time on restructuring our u-boot patches (getting them ready for submission) and actually submitting the first nine patches to the u-boot-users mailing list.

I also spent some time on proprietary software, after a _long_ time. I'm trying to get TI's GSM Modem Firmware updater ported from Windows to Linux. Eventually I want to be able to re-flash the GSM firmware from the S3C2410 side, not involving any PC and especially not any proprietary operating system ;)

I would love to see the firmware updater going public, too - but given the nature of the GSM business, that chance is close to zero. Which is ridiculous, since it doesn't reveal anything important at all. The GSM Modem will verify the cryptographic signature of the firmware image anyway, no matter what the downloader does. But well, we have different problems to solve than to engage in endless discussions anyway...

The first peak of load on openmoko.org servers

For the last seven hours I've been trying to organize the openmoko.org server setup into providing more efficiency / performance. It was really amazing. We were not on slashdot, not on any of the major news sites, but we were already having something like 40MBps aggregate outbound traffic peaks on our servers.

The two major performance bottlenecks were ViewCVS and Mediawiki. Quickly I installed memcached on one of our more idle boxes, and put two squid instances on two separate machines in front of the mediawiki, which then seemed to do mostly fine.

The ViewCVS apparently cannot be helped at all. What I found on the web is that it's apparently just very inefficient code, and there's little one can do without rewriting the code. I don't know whether that's still the current situation, but when the next peak comes, I'll probably just disable ViewCVS to save some CPU cycles.

In case you're interested: Our setup is currently running on four physical boxes, running a total of ten OpenVZinstances. One of the machines is dedicated to the GForge installation on projects.openmoko.org, the other one a (at least intended to be) dedicated buildhost where we do our OE builds.

While this obviously has been quite enough for the last half year, we now have different performance requirements. For Phase-0 this installation is probably still quite sufficient, since this first-couple-of-days peak is bound to cease at some point.

However, When Phase-1 starts (public availability of phones to developers by means of direct order), we will definitely need a more sophisticated infrastructure for our downloads.openmoko.org site, from where we will make available the full source code that our OE builds need, plus the full source and object code of every binary release we make. The idea is to have a round-robin DNS setup of geographically distributed machines.

So as of now, we're soliciting mirrors with large disk capacity. If you want to help us by providing a mirror (expected capacity requirement for 2007: something like 300GB) and bandwidth, please contact me at laforge@openmoko.org. We're particularly interested in the US and Asia regions.

Apart from that, a couple of secondary DNS servers would also help improving our availability. If you already have a bind installation somewhere and want to become a secondary DNS for openmoko.{org,com,net}, please contact me, too.

Finally, the good news is: We're down to 2..4MBps for now. Until the news appears on major news sites, I guess.

After having discovered that mailman doesn't use templates for its listinfo_overview page (the one you get when calling /mailman/listinfo without a list name), I quickly hard-coded our openmoko header into the python code. If somebody feels motivated to add proper templating support to all mailman pages (such as the 'options password prompt' and the before-mentioned listinfo_overview), you could help us out even with something like that.

Now I'll finally turn back to actual moko code. We have frame buffer support in u-boot since two days ago (splash screens are important for marketing), I'll now look into getting the CPU clock to 266MHz (we've had some issues with this) and finally, after all, TS07.10 multiplex and gsmd infrastructure, for which I consistently haven't found time until now.

openmoko.org goes public

Today, We have added public access to

OpenMoko / Neo1973 delayed, once again

As you can read in this announcement, there will be another [slight] delay in the release schedule for OpenMoko and the Neo1973 phone.

I'm really sad and sorry about this, since the core team has been working _very_ hard for the last couple of months to get this project somewhere. However, a combination of Murphy's law, our high demands on quality at every level, communication problems and a lack of FOSS-experienced developers have made progress quite a bit slower than expected. I won't even tell you how far we are behind the original internal schedule [It's an internal schedule, after all].

For somebody like me, who has primarily worked with and in the FOSS community, even in his day by day professional career for the last decade, there have been many cultural problems in this project.

I've originally been hired to take care of the low-level aspects of the system, i.e. boot loader+kernel porting, driver development, and last but not least the GSM communication infrastructure, as well as general consulting with regard to FOSS matters.

In the end (up to now) I have been doing tons of more things. I've been doing hardware related debugging, hot-fixing and consulting, providing lots of support for our internal development team, doing all the system administration, configuration and maintenance of our four physical and about 15 virtual machines (wiki, lists, gforge, svn, build server, etc.). Today I even spent a lot of time on web related issues [hey, I haven't done much web stuff since HTML4 and CSS1 came out], since we have committed to go public with our web sites public at some point.

We've had to teach people how to use request tracker, bugzilla, subversion, mailing lists, IRC. Those basic means of communication, natural for everyone ever involved in a FOSS project are all things that we had to bootstrap here.

Many of the things that are a complete given for me (and even us, the rest of the core team consisting of Sean, Werner, Mickey and myself) are not at all known, valued and/or respected [yet] by the various people and entities we had to relate in this project.

To give you a short outline about some of the issues we have been fighting to create our vision of a truly open device, targeted by developers for developers:

So e.g. for web designers, it's hard to cope with the demand that web pages should fully scale, not have any fixed-pixel width graphics/style, not contain flash, only make careful use of javascript, and should always pass XHTML / CSS validation. We don't want 'hacks', but clean themes/styles/templates in the native drop-in format that the respective applications (mediawiki, mailman, bugzilla, gforge, etc.) support.

For source code, people who have always worked in closed-source environment, and even with a concentration on embedded 'one time throw away' devices, it is a cultural change that the source code has to be maintainable, that it has to follow coding style rules, and that it has to be "pretty", since people will read it, and it will make a bad impression if the code quality sucks.

Our code is not OK if it builds once on some system, but the applications need to use autotools or similar mechanisms, be packaged for OE use, the package description needs to be verified and debugged, and the resulting builds need to be reproducible one everyone's system.

We do not use a vendor-supplied toolchain, but rather build our own. And yes, that toolchain also needs to be built 100% from scratch, and that process has to be solid and reproducible, just like the packaging.

For the hardware, we have to provide the interfaces that usually nobody wants to give people access to (serial console, JTAG, ..). We have to make it easy for people to update their firmware, rather than hard. We have to have safeguards in hardware, since we can't prevent people to reconfigure the battery charger algorithm in software. The battery should still survive this, no matter what happens.

Preferably using hardware components with open documentation (versus "the cheapest available that does the job") is a design criteria that almost nobody in the industry will be used to.

Also, whenever we use hardware with specs under NDA (which we tried to avoid in all place, the source code has), we have to submit that code to the vendor, and ask whether it is fine to release it.

All in all, this is a quite exciting thing. Making people think different, trying to get the values of Free Software into their mind set, even if only for this project. Most of the people (as in numbers of people) involved in the development so far have not had any relation to FOSS before. They might have done Linux based development, but only because somebody asked them to get something done, cheap and quickly. But that's now what we're after.

But this process takes time, and a lot of strength. And it's hard to scale if you only have three to four people who actually have a clue about all those things. So on the one hand, we have to learn how to scale better. We have to involve more people from the community, with the FOSS/hacker background, both as paid developers and the excited volunteers. So far Sean, Werner, Mickey, and myself have been powerful but 'lonesome' fighters in the corporate world with whom we had to interact.

I'd like to express my deepest thanks to FIC for funding this endeavor that must appear like strange little experiment to them. I'd like to thank for all the support we have received from the FIC hardware and software development teams. I know we've been always very critical, and probably still seem unsatisfied with many things. But it's nonetheless a unique chance to be able to do this at all.

I've suffered a lot during the last seven months, and I've never worked as hard in my entire life, spending at least 80 hours per week on this. Despite that, if you look at the actual results, both in software and hardware (in the upcoming days), you will probably see way more things that need to be done rather than have been done. You will probably think: What, you haven't written more code in that entire time frame? All I can do is ask you to read this mail to get a grasp about what has been going on.

From now on, I hope we can lead this project more into the community. Make it like any other FOSS project. Work will be way more fun. More creative people. More cool hacks. More freedom.

Please consider this as the beginning, not the end. We're just bootstrapping the world of as-open-as-possible mobile devices. There will be much cooler devices, and there will be much cooler software. I'm honoured to be given the chance to take a leading role in this.

USB serial support in Neo1973 boot loader (u-boot)

The last two days I was adding support for USB serial (cdc_acm compatible) to u-boot for the Neo1973 phone. This is definitely a nice way to give developers access to the boot loader prompt, without having to have any special cables, the debug board, hackers lunch box or else.

Obviously, it's not the same as a real serial port. But given that you have a working u-boot in flash, and given you don't want to do boot loader development and rather concentrate on kernel/user space issues, then this definitely is a nice option.

The basic cdc_acm patch came from the handhelds.org SX1 project, so I mainly had to add a s3c2410 USB device controller driver for u-boot. Usually this is not that much of a challenge. The lack of documentation by Samsung is compensated by the handhelds.org s3c2410_udc.c kernel driver, of which by now I know every single line.

However, this one was really hard. I couldn't even get the control pipe to do all the tasks to enumerate properly on the bus. And that after having implemented the control pipe handling for a different device (OpenPCD) only half a year ago, so I basically still knew quite detailed what had to be done.

I read, and re-read, and re-read the code. Looked and verified the assembler output. At some point, I was convinced the logic of my code is correct. It must be some auxiliary issue. PLL configured wrongly, GPIO settings not right, whatever. I still couldn't find it.

After two days (one of which had 16 hours straight) it turned out that the problem wasn't actually the logic of the code, but it was pure timing. The u-boot usbdcore.c EP0 implementation had done a memcpy of 18 bytes in a code-path that turned out to be extremely time critical. Just not copying the device descriptor (which is only done for on-the-fly patching the correct EP0 packet size) everything immediately worked.

What a relief. I can finally get back to work on GSM related stuff.

Informing recipients of free OpenMoko developer phones

Today [depending on the timezone maybe even yesterday], we started to inform those developers whom we have selected for the 'Phase 0', i.e. those who will receive a Neo1973 free of cost (including shipping). Those phones are scheduled to leave Taiwan on Feb. 11.

So this heads-up mail two weeks in advance is mainly to obtain shipping address, and ask whether there are any special customs related issues that need to be taken care of.

Yes, it is somewhat elitist to hand-pick people and then send them free hardware. But I don't really see a viable alternative approach for a start. Those recipients are people who are really known to contribute to the FOSS world, and of whom we think they would really like to contribute to this project.

Now some FOSS critics (or people critical of businesses engaging with FOSS) might say: Yeah, now you think the rest of your phone is developed for free. This is completely wrong.

This project is ran by people who believe in Free Software. It is from the community for the community. It is an important chance for Free Software and user freedom in this otherwise completely controlled mobile phone market.

Basically we have a hardware vendor (FIC) providing us with phone hardware, for which they

fund to hire selected developers within the community
provide complete hardware documentation and hardware support, even the ability to feed back hardware wish list items
give us complete freedom where we want to take this project

Now you might still think: "In the end, they will make the profit". This is only true to a certain degree. First of all, everything we develop is Free software. Everything but the hardware specific bits could easily be run on any other piece of hardware. So anyone who wants to either contribute code or hire capable developers could theoretically port the whole thing on different hardware.

Also, while we ourselves think that this product will rock, this is really a nice market. It's interesting for geeks, hackers and certain power users. Not unlike OpenWRT in the field of wireless routers - but with active support by the hardware manufacturer.

So I personally cannot believe any of those "they just want to get development for free" arguments and want to strongly encourage the interested community to join this effort and help us make Free Software a viable alternative in the mobile phone market.

An OpenMoko update

As the interested reader might have noticed, a couple of days ago, the OpenMoko project has officially announced a schedule for the upcoming months.

Behind the scenes, everybody is giving his best to be able to fulfill that schedule. Anybody who has been involved in a project of this size inevitably knows that there are many problems and bugs to be resolved.

Today, for example, I was happy to be able to play back MP3 (and politically correct: Ogg/Vorbis) files for the first time on the Neo1973 phone. Those speakers can really scream loud, I can tell you!

In my part of the project, the boot loader / kernel area, there are also many bugs to be fixed. The bugzilla indicates 11 blocker bugs and five major bugs in that key area of the project that need to be resolved. The quantity might seem low, but some of them are quite generic, such as some important mechanism not working yet.

By now, we've also come up with a quite complete list of names for the 'phase 0', i.e. those guys whom we will ship a free Neo1973 device on February 11 (actually 12, since 11th is a Sunday. Hey, we're working on Sundays, only the shipping company isn't *g*).

Oh, and yes. The planet.openmoko.org is finally public. Even though there's not a lot of activity yet, expect way more in the next weeks, especially after mid-february, when we've put 50 phones into the wild :)

I need a break: Debugging a driver for non-existent hardware

For the development of the OpenMoko system on the Neo1973 phone, we have two different development platforms, one is the phone prototype - the other being a generic S3C2410 development board. Obviously that generic board doesn't have all the phone specific bits on it - but it can interconnect to a GSM modem via DB9 serial port, providing a very close match to the actual product hardware.

For the better part of yesterday, I apparently forgot that this is not true for all the hardware devices. For hours and hours I tried to debug a problem with the power management unit (PMU) driver. The I2C core just didn't want to talk to it.

It is only today morning that I realize: The development board doesn't have this PMU. Doh! Obviously only the phone prototype has that PMU! For god's sake, it looks like I need a break (but can't afford one, time-line wise).

Now why am I 'broadcasting' this embarrassing notice in a blog? To demonstrate: We're all human beings, we all make - apparently stupid - mistakes from time to time.

Some details about the GSM infrastructure on the Neo1973 / OpenMoko

I've posted this publicly to some mailing lists in mid-November, but thought it was good to have this information in the blog, too:

First of all, there is a ts0710 multiplex layer, architecture-wise similar to what Motorola uses in their 2.4.x kernels. This ts0710 (de)multiplex takes care of handling GSM TS 07.10 "advanced mode" (the HDLC framing). It will be easy to add "basic mode" for chips that doesn't support advanced mode, and I'm also planning to add support for the Motorola proprietary 07.10 extensions (see OpenEZX wiki) once Neo1973 has been released.

This demultiplex is implemented as a line discipline. Therefore some userspace program (in our case the GSM daemon) attaches as a line discipline to the underlying physical UART.

devices that don't have a physical UART (such as the Motorola phones) will provide a small glue layer that provides a virtual UART on top of e.g. USB as underlying layer.

The GSM mux layer then provides itself one virtual serial port per DLC of the multiplex.

On top of those virtual serial ports, there is a GPRS line discipline, or a PPP line discipline for implementing full in-kernel data connection support, with no need for sending data packets for network traffic from/to userspace.

Both the GPRS line discipline and the ts0710 multiplex are written according to the style and requirements ("good taste") of kernel code, and will be submitted to the mainline kernel as soon as the Neo1973 goes public. I really hope to make this a standard component of the mainline kernel, supporting as many GSM modems as possible over time.

On top of the virtual serial ports, we have a GSM daemon. This daemon takes care of almost all communication with the GSM modem. The daemon initializes AT+CMUX and then attaches the kernel line discipline. It also attaches GPRS line discipline to a virtual serial port afterwards.

The daemon provides a Unix domain socket based protocol for other applications (at some later point this might become a network-enabled protocol by running it over TCP). The "other applications" (such as the contact manager, the dialer program, etc.) link against a library called "libgsmd" which wraps the protocol into a C language API.

This means that programs have a high-level API for initiating and receiving voice calls, for receiving and sending SMS, obtaining list of operators, reading/storing contacts from/to SIM card, etc.

The daemon will be GPL licensed, for the library we're not sure whether to GPL or LGPL it (probably LGPL). All applications shipped on the Neo1973 linking to the library are GPL licensed, so there will be enough example code for people to understand how that API works.

The gsmd/libgsmd code will be run (just like any other program on the Neo1973) as any other free software / open source program. Please understand that while FIC sponsors the OpenMoko project, they don't really exert control over it. So as soon as the device and code is released, I'm happy for any input and discussions the community has on improving such a system, including support for more devices, etc.

Oh, and yes, the daemon has a plug-in interface for vendor-specific extensions, since every GSM modem vendor has commands beyond the GSM07.07 specification. Also, the C API and the Unix domain protocol provide for transparent pass-through of AT commends from application to daemon. This is not meant to be a single-vendor-single-product code, but is at least designed to make it easy to add support for other devices.

Anyway, even without gsmd/libgsmd, I think the kernel-level serial multiplexer (which is not a very complicated thing) is a valuable feature to anyone doing GSM/GPRS on Linux - be it on a PC with GSM modem, or a smartphone.

The reason for doing this (de)multiplex in the kernel:

the individual virtual serial ports have all the features of real serial ports. hardware/software flow control, modem status lines, etc. - and the kernel has a standard API, well known in Unix over decades, to work with serial ports from a userspace program
especially when it comes to data sessions (packet data or circuit switched data), then you don't want to push all data to userspace and back in the kernel. you want to have a fast path for that, both from a CPU consumption (battery!) point of life, but also from a latency point of view. mobile data latencies are already high enough, we don't want to have additional unneccesary latencies in the handset

Please understand that at this time I have to focus on OpenMoko development, and cannot engage in lengthy discussions. This is about all the information I wanted to add about what's actually happening in our project, and this is the architecture the OpenMoko software on the Neo1973 phone has. Please bear with me until January. Once the code is out, I'm happy for any kind of discussion, modification, contribution, etc.