incommunicado for a while

It seems like my main mail server, ganesha.gnumonks.org, is facing some severe problems (ext3 corruption on a 3ware hardware RAID-1, i.e. something that clearly should not happen.

As per Murphy's law, this had to happen exactly while I was in-flight on my trip to Bangalore for FOSS.in 2007 :( Had it happened 2 days earlier, i would have actually within physical reach of the machine in question.

Luckily, all my hosted servers have remote consoles and actually even remote access to the BIOS setup. So I'm trying to recover what's to recover. The exim mail spool is on the affected /var partition. The much more important cyrus IMAPD spool is not affected. What a relief.

Still, everyone who tried to contact me: Please expect some delays in email based communication through the next few days. Sorry for the inconvenience.

Looking forward to FOSS.in

In a few days FOSS.in 2007 will start. I'm departing from Germany on Monday next week. I have a ton of things to do until then, including a trip to my family in Nuernberg on Friday,, visiting a industrial festival in Chemnitz on Saturday and packing my suitcases on Sunday ;)

So this will be the fifth year in a row that I visit Bangalore in late November/early December for the event formerly known as Linux Bangalore. What once started like a crazy reason for visiting India the first time (after enjoying Indian food and bollywood music from Germany for a couple of years), has turned into a regular mark in every years' calendar for me.

I've been told that FOSS.in this year will be very different from all the previous events. The focus has been shifted from doing just another round of 'this is free software and this is how to use it' event, the focus is now entirely on the community developer.

India still has, to my deep regret, shown relatively few significant contributors to Free Software - especially if you relate it to the size of the IT industry and the number of people working as software engineers in that country. Thus, I very much welcome any effort to nurture and foster the active, contributing part of the FOSS community there.

Meanwhile, the Schedule has been published by the organizers. Looking at the speakers and topics covered, it definitely looks more than promising!

Also: My openmoko-induced absence to the major Linux events in Europe and Canada have resulted in a way too long time since I've last met Rusty and James, my former fellow netfilter/iptables hackers :) Make sure you don't miss any of Rusty's talk. It's going to be fun :) And be prepared to switch your brain's English parser into high-speed mode :)

As a final side note: I'm happy to learn today that my application for a five year visa to India has been granted. During the last five years, I had to obtain a total of seven 6months visas - sufficient evidence to support my argument in favor of a 5 year visa.

Slowly getting back to my normal life

For the last couple of days I finally felt like in the "good old" (pre-openmoko) days. Suddenly things that have been piling up to higher and higher stacks are getting resolved. Tax related issues, various other administrative things like housing, insurance related, etc. I even find time for regular medical checkups like dentist's visit again.

I'm also slowly browsing through the various lists like netdev, which I haven't had time to read throughout the last year or so. Finally I get the feeling of being "in sync" again with what's happening in the rest of the Linux world.

I'm also making quick progress on gpl-violations.org (just ordered two new test purchases of allegedly infringing devices). And I even have time for the occasional homepage update of my various rotting (and rotten) websites.

I'm also almost finished with Ulrich Dreppers excellent paper "What Every Programmer Should Know About Memory". Being a kernel hacker, most of the issues have been known to me more or less, but it's good to read about all of them in a concisive paper, together with benchmarking on current hardware.

There ares still hundreds of pending issues here and there, and I yet have to find some time to e.g. finally do a ulogd2 release, work on integrating all the pending patches in librfid, work on ISO15693 support for it, and last but not least work on getting a lot of the work I did for openmoko on u-boot and kernel finally merged mainline. But if I'm able to continue getting things done at this pace, I'm very optimistic

It seems like all the time with OpenMoko has actually made me forget how easy and fast things have been moving all the time before ;)

Leaving OpenMoko "Lead System Architect" position

The regular reader of this blog will have noticed a distinct lack of any OpenMoko related news. This is not a coincidence. As Mickey wrote in a blog entry, there has been quite a bit of internal friction lately.

Adding that to the enormous amount of stress over the last 18 months has made me feel quite a bit demotivated over time. I've tried to cut down on the amount of work I do, but it hasn't helped much. So now I'm at a point where I feel unable to work in any active/leading role inside the project and/or company. My deep apologies to the the project and its community.

But don't worry. OpenMoko is a team, and I'll do everything to help smoothen the transition. I'm more than willing to assist those who will take care of my various tasks.

From today on, I'm nothing more than a volunteer to the project. I'll likely continue a bit of hacking in my areas of personal interest. Just like in many of the other FOSS projects that I have been (or still am) involved.

All the best to the OpenMoko project, the OpenMoko company, FIC Mobility. The last 18 months was an intense experience. Thanks to everyone who has helped the project both inside FIC/OpenMoko and outside. Thanks to FIC for funding and supporting the project. I wish everyone the best, and I'll be the most happy person (next to Sean) at each and every milestone the project achieves in the future.

My last netfilter training

Since I've been doing no netfilter/iptables related work recently, I've announced that the three day training is going to be the last one, at least for the time being.

Though stressful as usual (have you ever talked/presented straight 8 hours on three consecutive days?) it was a quite joyful experience. Apart from the netfilter/iptables workshop earlier this year, the only contact with my former much-beloved project in 2007.

However, the training made me realize how outdated all the existing documentation (and even my own training material) is. Basically everything was written in the early 2.4.x days - and much has changed ever since.

There's all the nf_conntrack / nf_nat related changes, as well as the x_tables transition, which can cause many subtle errors due to old scripts expecting different kernel module names, etc.

None of the HOWTO's or similar documents talk about the conntrack userspace program yet, there's no documentation (and no release) for ulogd2, etc.

So I'll really try to sit down and find some time to improve some of those areas. It yet remains to be seen if I can actually make it. But I feel there's a real gap to be filled...

Slowly getting back to work on gpl-violations.org

Today I've finally started to pro-actively work on gpl-violations.org again. I haven't been able to do any work on it for almost 1.5 years due to my intense involvement with OpenMoko.

Among my first tasks was to update the ssl certificate for our internal Request Tracker, which apparently expired quite some time ago. After that, I went through all RT tickets and deleted tons of spam from it. Now it finally looks like I can start working with it again :)

I'm also trying to catch up with all the gpl-violations.org related email, but please give me a couple of weeks, there's just way too much of it :(

Heading back to Germany

So, after roughly two weeks of OpenMoko Taipei headquarters, I'm now heading back to my Home+Office in Berlin. And I'm really looking forward to it. During the last couple of months I've tried my best to help the transition from OpenMoko a a project inside a FIC business unit to OpenMoko, Inc. the independent company inside the FIC Group. I've helped with tons of things that are definitely by no means related to kernel/bootloader development, or even the hardware architectural planning that I've been heavily getting involved starting with GTA02.

So now it is a good time to finally focus again on what my actual and original task is: Software development. This can be done much better remotely, so I'll expect to be able to work way more from Berlin. Which makes me happy, since it always was and still is my favorite city. And I definitely missed it a lot during the last year of intensive OpenMoko work. Now I'm on my way back, and I'm looking forward to spending more time with my friends, the CCC Berlin. Being able to go to concerts and clubs that play music I actually like. Being able to work on finally improving (or rather: finishing) home improvement in my apartment, and many other things.

So don't get this wrong. I'm very much continuing my technical work for OpenMoko, and as the first developer on this entire project and a OpenMoko core team member, I'm always going to maintain an influential role in the project. But finally, I can go home, feel better, work more focused and efficiently, and improve the technical quality of our products even more :)

Since OpenMoko now actually has three full-time paid project members in Berlin, It's also going to be nice to closer cooperate with them. (or co-work, like the Chinese English speaking would say)

Seeing netfilter/iptables boot-up messages in an airplane

I've read a couple of blog posts about the suspicion (or even confirmation) that some of the in-flight entertainment systems are using Linux. It's completely understandable that if you have to put 400+ such systems into a plane, you'd rather use free software for the economics of licensing costs.. imagine 400+ windows licenses :)

Now in any case, during my Taipei-Singapore flight enroute my trip back from OpenMoko headquarters to Berlin, I was amazed by the new big-screen entertainment systems that Singapore airlines is apparently using in some of their planes. This particular plane was a Boeing 777-300R. The screens are not the usual 4" or similar, but actually something like 10" - and that in economy class. Also, they're really high-res, and seem to be entirely controlled digital, i.e. no blurry PAL/NTSC or VGA resolution, but actually something on the order of (guessing) XGA.

The second unusual bit was the three connectors on the right hand side of the screen. Ethernet (!), USB host and composite video-in. I've never seen something similar in an airplane before.

Now unfortunately the system on my seat was stuck. I called the flight attendant, who then issued a remote system reset. To much of my surprise, I could soon see the BIOS boot screens of a VIA based embedded system. Afterwards, it executed RedBoot, followed by a Linux kernel, to be followed later by an X server (you could see the grey background pattern with the X cursor for quite some time), and then some custom X11 applications.

As the RedBoot message suggests, the system was implemented by Panasonic Avionics.

The first thing worth mentioning was the incredibly slow boot progress. They must be running those systems at low clock speed, and boot them over the network, even though the rootfs was ext2. I didn't see the details since I was too busy grabbing my camera to take this photograph.

The amazing thing about this system is that it has 512MB RAM per seat, dissipates a dangerous amount of heat (you can feel it getting very uncomfortably warm under that screen, where probably the entire system is located, similar to a "tablet PC" form-factor). Still, it is very slow. And then look at the details. Why on earth do you need a wifi stack and netfilter/iptables, including ip_conntrack on such a system inside the airplane? I severely doubt that they use packet filtering to prevent a hacker to get from one seat to another - and thus connection tracking adds anything aside a performance hit.

So what's it with those connectors? I couldn't get the Ethernet part. And for whatever reason, I didn't have a patch cable in my carry-on luggage either. Maybe the entertainment system might have been even more entertaining that way, who knows :(

The USB connector is meant for a user-provided USB memory stick, where you can then watch your own pictures, or use a word processor or spreadsheet to view / create / edit documents. The software used for this is - unsurprisingly - a customized version of StarWriter/StarCalc, based on OpenOffice.org. I've played a bit around with it. Using the controller, you can actually resize the window from full-screen to something smaller, but there's nothing interesting in the background. I've tried to see if one can somehow change the print command to "/bin/sh" and then print a document - but the printing functionality had been removed altogether, so no luck here.

I decided to sleep for the rest of the flight, rather than trying different attack vectors. If I run into such a system again, I'm probably quite tempted to do so again. Would be fund to get an entire plane full of Linux hackers (let's say: OLS attendees) and have them play around with it to see how well is is done from a security point of view. I guess the worst-case scenario is something like people connecting their USB hard drives (or laptops via Ethernet) and then ripping the entire entertainment library during the duration of a 12 hour intercontinental flight. I'd suppose they actually should have looked a fair bit at the security of such a system. But then, the same is true for many systems, and developers still neglect that aspect way too often.

Back to Taipei

Today I got back to Taipei, almost three weeks later than originally anticipated. More news after at least one night of full sleep and the first day at the office...

FOSS.in 2007 Call for Participation

The Call for Participation of this years incarnation [it's in India, after all] of FOSS.in has just been released.

This is great news. I've cut down on all other events this year: I haven't visited at OLS, LinuxConf Europe, linux.conf.au, ... Only FOSS.in I really cannot miss ;)

I haven't yet decided on the exact title of the lecture that I'm interested to submit, but it seems like I'll have to decide soon. It will be OpenMoko related, that's for sure ;)

If you have worked on something exciting in the FOSS world, please don't hesitate and submit it to the FOSS.in/2007 CfP. It's a great event with a very technical audience. And an ideal opportunity to catch a glimpse of India :)

Overwhelming participation at Demonstration against Germany's new surveillance laws

On Saturday, I attended the Freiheit statt Angst demonstration in Berlin, which aimed at protesting against the various new laws and regulations increasing the surveillance of the German government on its citizens. I assumed it would be again one of those niche events like the demonstrations against software patents, with some 200 people. To the contrary! The organizers counted 15,000 demonstrators, and even the police's initial estimate at the beginning of the demonstration was 8,000.

This really is a big step forward. Apparently it's not only the "generation Internet" that is sick of the ever increasing cut down on civil liberties, data protection and privacy. That being said, 15,000 is still a too small number for a topic that effects everyone in this country. But even a demonstration of that size doesn't happen every day in Berlin, so it's not that easy to completely ignore either...

Don't miss the photos of the demonstration

netfilter developer workshop 2007 is over

The days of the netfilter workshop passed quite fast, and I'm finally back to my home in Berlin now. In case you're interested, here is a link to the group photo.

Among other things, we've had the following major decisions at the workshop:

  • Patrick McHardy finally officially head of the coreteam
  • ulogd2 will see an official release candidate soon
  • we want to merge ipset soon
  • we will try to shift future developments in the direction of libnl and slowly deprecate libnetfilter_*
  • we will move the netfilter and netfilter-devel lists to vger.kernel.org since we don't have to care about spam filtering there. Other, lower-traffic lists remain on lists.netfilter.org
  • we will switch to git even for userspace code, at least for the iptables source code

Finally, I'd like to use this opportunity to thank all our Workshop sponsors, particularly Astaro for their continuous and generous support of netfilter/iptables throughout the last five years.

Enjoying the netfilter workshop 2007

I've returned to Germany in order to attend the 5th netfilter development workshop in Karlsruhe. It's sponsored by Astaro, whose continuing support of netfilter/iptables is really outstanding. Even after I took my "leave" to work on OpenMoko, they continue their funding by paying for Patricks maintenance of the netfilter/iptables codebase, and things like hosting the netfilter workshop.

It's really great to meet with the old colleagues with whom I've co-worked for a number of years on netfilter/iptables. I really miss those days, basically spending most of my day working together and communicating with cool people hacking on similar problems. Quite a bit different from what I'm doing right now.

So while I'm here, I'm actually trying to spend most of my time related to netfilter/iptables, which is really refreshing.

I now own two motorbikes

Besides my BMW F650ST in Berlin, I now (since 10 days ago) own a Yamaha TW200 in Taipei. To me, this is sort of a joke of a motorbike. A toy bike. 200cc feels like a bicycle with ancillary motor. No acceleration, no torque...

But then, Taiwan is an incredible strange country when it comes to motorbikes. And that TW is definitely better than one of those 2-stroke plastics scooters (I had one when I was 16: 3 jammed pistons in two years, plastics above the exhaust completely melted up to a point where I had to add custom-made aluminum pieces for it not to loose its structural integrity).

So it somehow fits the overall Taiwan experience: A never-ending compromise....

Two days of intense u-boot hacking

After finishing most of the basic device support for GTA02v2 in both kernel and u-boot during the last week, I've finally turned back to implementing one of the longest standing issues in u-boot: GSM passthrough.

GSM passthrough allows you to basically ignore the smartphone part of the device and connect the GSM modem more or less directly to a host PC. The feature has been long known by various smartphone hacker projects such as e.g. OpenEZX (which as a side note has made quite some progress recently, much appreciated by me as retired project founder).

So GSM passthrough is mainly useful for rapid development (developing gsmd more efficiently by running it on the developers workstation, without cross-compiling and ipkg installs), but also if you want to use some legacy application that was written at a time where a phone really only was a phone (e.g. sim card managers, ...)

Now the GSM passthrough was always pushed back on the TODO list, since our usbtty code in u-boot was never very reliable and caused lots of data corruption such as bogus and/or missing characters. Quite useful for the human operator, but definitely not acceptable for getting a program with AT command parser to work. So that had to be fixed first (and it is now fixed).

As I pointed out in my announcement, the generic way of implementing this feature has actually quite interesting but much more obscure use cases such as dialling from a landline via GSM (CSD call) into your Neo, manually accepting the incoming call and then attaching the u-boot command line to it. That's sort of the feature you have on hosted/colocated servers, when you use a boot-loader with serial console support and attach a modem or terminal server to it.

So does this mean the Neo1973 is now ready for the enterprise? Not quite. Even though it has a built-in UPS (called battery), and GTA02 will even allow you to change the battery without shutting down the device, resulting in higher availability ;)

But then, the expectations / requirements for mobile communications devices are quite a bit different from that world. But the hackers community likes those kind of strange features. Have you ever heard of another smartphone with that capability?

Oh, and before I get any complaints about the security: This "feature" has to be explicitly enabled and every call manually accepted by typing a sequence of commands into the u-boot command line. So unless the attack involves tons of social engineering (getting the device owner to do all those things) there's not that much of a big deal. But maybe we should start to think of some kind of user authentication for u-boot now *rotfl*.

OpenMoko Taipei office network setup

For the last three days I've been busy setting up the network in the new OpenMoko office. Finally something that just works, without any major flaws. That's probably because the involvement of external entities is fairly small. But looking at it closer, actually exactly there the problems start. FIC purchasing e.g. only bought 50% of the switch equipment that I asked for, something that I still don't yet have received any details about, neither some kind of apology.

In any case, the network is up and running. We now have a pretty uncommon heterogeneous combination of Cisco, Dell and even some D-Link switches, with the core switch being a pretty impressive Dell 6248. The 10GBit transceivers are still missing, so the backbone is just running 1GBit for the time being.

There's a total of about 190 Ethernet access ports throughout this new office, and we have various different broadcast domains, safely separating guest network, office network, r&d network, ... from each other.

The server room also looks quite nice now, with five 19" full-height 19" racks, 2 layers of UPS (building-wide and extra small UPS in front of servers), with three fairly big servers in operation, and eight more pending to be used soon.

The Internet uplink is a 100MBit capable physical layer (fiber optics), of which we only subscribed to 8MBit initially. But one phone call (plus additional transfer of money) later, we can bump the speed to any rate below that 100MBit physical layer limit.

The operation of our own (netfilter/iptables based, what did you think?) firewall now also brings us back the long-missed basic fundamental networking needs of any free software hackers (called luxury around here) of all our developers being able to

  • access the web without restrictions and blocked sites
  • finally use IRC!
  • access external IMAP4 servers
  • have SSH and other interactive sessions not time out every couple of minutes

Looking forward to the Chaos Camp 2007

In about 24 hours I'll be on my flight 'back' to Germany. In fact it's not really a flight back to Germany, but more like a temporary break of my extended stay in Taipei for the sake of OpenMoko.

The main reason for this trip is to attend the Chaos Camp 2007 of the CCC. I've so far dropped every conference or other technical event this year to concentrate on my work for OpenMoko, but I'm not able to compromise on the camp.

On the one hand, I'm looking forward to finally not having any official function at a CCC event. More than one year after vacating my task as leader of the video documentation effort, and after my somewhat minor involvement with the sputnik RFID tracking project at the congress last December, this is not really the first CCC event which I'll visit as a pure visitor. I haven't even submitted any paper.

So the camp will be holiday. Time to relax, talk with fellow hackers. Sure, lots of the German OpenMoko guys (roh, stefan, alphaone, and our newcomer gismo) will be there. So there will definitely be some kind of productive outcome for the OpenMoko project, too. But in a very different setting. Doing thighs that are fun, rather than all the things that have to be done :)

Progress with the new OpenMoko and FIC Mobility office

The final 24 hours of my current Taipei trip have started. This is a good time to reflect on what has happened in those last weeks since July 9.

As with the overall status of the project, I'm still extremely dissatisfied. The frequent reader of this blog will have noticed the last postings on this subject, full of discontent.

So the further we are into this project, the more time we put into it - the further I expect it to produce anything that I would consider reasonable results. Please don't confuse this with the commercial success, or the ability to produce working products. This is an entirely different matter.

To me, it is extremely important to do things systematically, with lots of planning, safeguards, checks, verifiable and reproducible processes, as much automatization as possible, little room for human error, etc. So as long as not everything from hardware to software development, mass production, production testing, distribution/logistics/sales, etc. follow a well-thought-through process, I will not be happy with the results. Because any such "results" are more or less the mere product of luck or randomness, and not a trustworthy basis upon which we can rely on.

So reflecting on those past weeks, I think the following things have made humble and moderate progress:

  • GTA02v2, the second prototype generation of GTA02 was finalized after many issues including unavailability of key components. I'm more than looking forward to see how it turns out
  • DebugBoard v3, the third version of the Neo1973 Debug Board was finalized and is actually also verified and can go in mass production
  • Our internal software team finally has proper leadership and guidance from somebody who is both Taiwanese and has a thorough understanding of Free Software: jserv
  • The new, second (intermediate) generation user interface was implemented and released. It's implemented mainly by O-Hand, since embarrassing as it is, we still don't yet have managed to build a proper internal software development team.
  • The first batch of Neo1973 GTA01 was sold, though with a entirely last-minute, error-prone and way-too manual process for order, payment and logistics.
  • We have found a capable sysadmin for our hosted, publicly-accessible servers. More news about that in September.
  • We have managed to find a extremely valuable senior technical person for our graphics driver and low-level UI work. This, too, will make big news in September.
  • The FiWin (FIC wireless networks) company, home to the team working on the it-exists-but-nobody-publicly-knows-what-it-is HXD8, was merged into OpenMoko and FIC Mobility
  • We have finalized the specification for the workstations of our software developers. It's incredibly complex to find something that's compliant with our requirements (mainboard with Intel 945/965 on-board graphics, Ethernet chip != attansic/realtek, dual core CPU with 2x2048kByte cache) in Taiwan. So now our developers will all get a Q6600 CPU (what nonsense!). I've tested it, and it compiles the GTA01 kernel in 1.59minutes. Guess they'll be happy about that.
  • Realize how many things really are fundamentally wrong internally. What we knew so far about our inheritance was just the beginning ;)

One major thing that finally started to move forward, with something like four to six weeks of completely unnecessary delays, is the new office. After it was decided that we will split FIC MCBU into the independent OpenMoko, Inc. and FIC Mobility (aka Mobile Communications), we also decided to move into bigger, scalable and independent offices.

To our big luck, two thirds of the 7th floor in the FIC headquarters were currently unused, and they're now undergoing quite a bit of renovation and reconstruction. Walls have been removed and brought in, floors have been properly removed and new ones laid - after days of fighting by Sean and myself.

The networking and phone cables get a major overhaul and will be tested. I've also seen the AC for the new OpenMoko server room being brought in. The contract for our own Internet plink has been finalized. The fiber will be put in place within the next week. The core switches have been configured, but we're still fighting very hard to get those damn 10Bit XFP transceivers from Dell.

So the current schedule is to move on August 17, one day after I'll be back from Germany. If that works out, I could spend the weekend 18/19 for doing the final network/server/router/firewall/... configuration.

Obviously, all of this causes quite significant resource drainage for everyone involved. But it's a more than necessary step forward to building an environment that we can actually work in. An environment where our developers have real Internet access, can join IRC channels, and can get in touch with the OpenMoko community without the obstacle of strange corporate policies. An environment where we can have a 'clean start', even in the most literal meaning of the word :)

So all in all, bear with us, have patience. The revolution might take significantly longer than anticipated. But we're still busy doing whatever it takes to get us to the product that the OpenMoko core team set out to build.

I'm single again

And this time actually looking forward to it. What kind of strange feeling.

Always in motion, the future is.

Sick but not insane (yet)

Some people were troubled by my last posting about 'insanity'. And I have to admit it is justified. It isn't nice to pull internal issues of some company out into the public that way. But I just couldn't do differently anymore. FIC should think about providing a free corporate psychiatrist for us. ;-) I think I have a fairly big tolerance when it comes to mishaps, incompetence and cluelessness, but some of the things we're experiencing here are just not bearable. Especially not if they lack any kind of rational explanation.

So my latest outburst was mainly related to the fact that despite being with flu and fever in bed, I had to personally hack that embarrassing little online shop we now have at https://direct.openmoko.com/ [my first perl CGI code in about ten years!!]. Can you believe it, we just did not have (and still don't have) anyone in this project who has ever done some real work on CGI's or 'technical web things' at all. And if this was not enough yet, the requirements kept constantly changing all the time, up to this day, more than one week after the webshop opened.

So for gods sake, how many months have we been knowing that at some point we need to ship? And then you have plenty of people who produce over many months three entirely different concepts on how to shop and distribute those devices. And in the end, you have something with a Chinese-only credit card processing page, no proper setup with regard to the carrier (UPS in our case), tons of people talking vapor, but from what I've heard drawing nice diagrams. And no working solution.

Some people might wonder why those shipping rates in the shop are so high. Shouldn't a large company like FIC get huge discounts? Yes, they do. But believe it or not, FIC does up to this day not know in advance how much a shipping costs. Only after it was made. UPS has something like a Rates and Service Selection API. UPS has also a nice and detailed manual on how this XML API reports those rebated 'negotiated rates' in addition to the standard rates.

But then till this very day, UPS keeps claiming that such an API does not exist. Despite the fact that off-the-shelf e-commerce applications _support_ this API, and despite the fact that UPS' very own API documentation goes into every detail describing how that information is XML-encoded. What comes to my mind is the O'Really: Distributing Clue to users in a slightly modified version: Distributing clue to UPS' own sales representatives. How on earth can you get or keep a job there if you haven't even read the company's own documentation on their products?

Then you have companies like WorldPay, who very broadly advertise the many different payment variants they accept, not only all the credit cards known to man, but even things like direct debit (Lastschrift) in Germany. But do you believe they are able to process American Express for us? No, obviously not. Customers located in Taiwan (like FIC/OpenMoko) cannot process AmEx. No explanation given. So much for the word WORLD in WorldPay.

Or would you believe that some large bank like Citibank (Taiwan) was able to charge credit cards (VISA/MasterCard) in USD? No, obviously, being in Taiwan they can only charge in NT$ (New Taiwan Dollars). WTF? It's the 21st century, and there are dozens of online credit card acceptance partners that allow you to bill in about any currency you want. But not a world-class bank like Citibank.

And now you might think that we're actually no longer working on development. That is wrong. Yet another funny story from the MokoUniverse is that one of the worlds largest semiconductor distributors just had a long meeting yesterday, with FAE's and tons of FIC hardware engineers to crack their heads about the question whether or not we can use a "new" silicon revision of a certain component, and where exactly the differences between the old and new revision might be. I refused to participate in such a meeting, indicating that I just want a document describing those differences. In writing. Soon after that, I find out that we have always been using that supposedly-new revision, for the better part of one year. Nobody actually bothered to look at a unit of hardware, or at all the high-res photographs of GTA01 that I posted on people.openmoko.org ages ago.

Then lets get to another of my favorites. Purchasing. I wrote an extensive list of networking gear that we desperately need in order to crate the internal IT environment for our fast growing new OpenMoko company. After many delays, today I finally got our two core switches. And what did they do? They "forgot" to order the transceivers, even though they were explicitly listed in the requirements/order list. God knows what happens in the heads of such people. So I'm now back to once again just mail-ordering things from Germany and then billing FIC for the expenses.

And to not bring up more embarrassing facts, and to at least preserve some level of confidentiality, I'm now going to stop. But believe me, there are _much_ more insane stories to be told. With some luck, at some distant point in the future, I'll get permission to publish them in my memoirs.

But that's basically the kind of thing that drives me close to insanity. Whether for that cause, or completely unrelated: My health is actually quite bad recently. Maybe the lack of regular sleeping hours, let aside regular intake of food and the tons of stress have contributed to the flu that lead to a severe fever attack today. You know there's something wrong if at 33 centigrade in the sun, you're shivering more than when riding your motorbike at -13 centigrade in German winter.

So I'm going to take it slowly for the time being, working much from what has now become my 2nd home (apartment in Taipei, provided gratuitously by FIC for people like Werner and myself, who are spending so much time over here).

My sincere apologies in advance. But given my multitude of roles/hats and functions in the OpenMoko project, I bet my health issues will now contribute to some further delays in about any area of the project. But there's little I can do. One day, we will get there. Let's hope you're still interested in our products at that point.

Insanity

If you want a status update on OpenMoko: I'm at a point where I won't go to the office again because I know the agression inside myself will turn into physically hurting someone there.

An update from the OpenMoko world

As Sean has now made his latest OpenMoko Announcement last night, this is a good point in time for me to write some more bits. Up to this announcement it was hard for me to publicly state anything, since the whole internal restructuring process has been underway, but not yet publicly announced.

I can only join Sean in his assessment about the superb support of FIC's senior management. They are providing us with the kind of resources we wanted.

But being the realist (well, Sean as optimist would call me pessimist, you know that old story), I also see severe challenges both right now, and ahead.

Whatever you might think: I bet that none of you has not the slightest idea about how many problems the OpenMoko team is fighting all day long. If you're thinking "what's the problem with a purely technical task, i.e. designing hardware and software for an Open phone"? Then my reply would be: It's not that big of a technical problem.

However, the really big issues start as soon as you leave the R&D world. On the one hand, there is the actual hardware production. Many components have incredible lead times (3 months or more), and our yet sort-of-unknown-but-initially-very-low quantities are not particularly helpful either. Any .tw OEM/ODM thinks in different terminology. The kind of production processes, shipping infrastructure, ... is just not meant for low-volume and direct shipment. We obviously knew this from the beginning, but everyone just happily works in their usual mode of operation, ignoring our concerns for many months.

Then think about the various customs / legal / trade issues. If you ship components from Taiwan to mainland china and use them to manufacture a product there, you need a special import license in order to get those products back into Taiwan. This license costs money (and, most of all: Time).

Or another example is the lack of double-tax agreements between Taiwan and the rest of the world. So payments to all our various external consultants all over Europe are taxed twice: Once in Taiwan, a second time in the respective country of residence.

For the last two weeks I have been working on finalizing the floor plan and infrastructure planning for the new FIC Mobile Communications and OpenMoko software groups offices in Taipei. And believe it or not, it is a very long and time consuming fight to ever get what you actually want. We know exactly what kind of servers, switches and routers we want. We know to which height we want to reduce the cubicles. We know what kind of Internet uplink we want. Still, it's close to impossible to get anything done. People will just outright refuse to do what they are asked (and paid!) to do.

Take our new servers as one minor example. You would assume that it is no problem at all to configure high-end servers around here. When doing this in Germany, I usually consult one of the many mailorder stores, go through their extensive list of mainboards and other components, select products based on their availability, price and features, and within 24hours I have everything delivered to my doorstep. 99% of those components are from Taiwanese companies.

Now enter Taiwan. First of all, you will discover that the concept of mailorder or extensive online product lists doesn't exist. "Taiwanese people don't trust e-commerce", is what they tell me. Secondly, you can't just call those places and ask them if they have a certain product, since apparently they would always say yes, only to get you into their store.

If you actually get into the various stores, you will see that almost all of the products you want are not available locally. "Not sold into the Taiwan market" is something that you hear very often. So e.g. the choice of Socket 478 mainboards from ASUS goes down from 52 (German online store) to something like 15-20.

So in the end we were really unable to find anything remotely decent (good performance, chipsets with excellent free software support) locally and I ended up importing Asus and Tyan mainboards from Germany into Taiwan, while buying the other components in Taiwan.

Now I could continue and name dozens of examples like this. If this project was just about _developing_ hardware and software, I would be a happy man, and we could look ahead to complete one device after the other. But it's all the other issues, administrative, political, cultural, sales, finance, accounting, shipping, ... which make people like Sean and me run at something like 20-25% of their usual efficiency, despite putting in at least 180% of regular working hours. And there is nobody who can help this, because nobody non-technical really understands what we're doing here, and why we need to do it different than whatever they might have done it before.

Visiting FIC's factory for GTA01 mass production in Suzhou

Yesterday I was on a ten-hour trip from Taipei to Suzhou in mainland China. It took about ten hours for something like a 300km line-of-sight distance, since the Taiwan/China political dispute over The Three Links doesn't allow any direct flights. So we had to go from Taipei to Hong-Kong, transfer onto a different flight to Shanghai, and then start for a couple of hours car ride to Suzhou.

The trip was quite impressive, especially since I have seen a lot more of Shanghai then during my summer 2006 trip to the FIC Shanghai branch. In fact, the sight of so many [strangely-looking] dense, high-rise apartment buildings with 25, 30 or more floors in the suburbs reminded me quite a lot of the classic 1927 Metropolis movie. This impression was probably further enhanced by the thick clouds of smog covering all of the sky, resulting in even non-grey objects look grey, giving the impression of a more-or-less gray-scale world...

In any case, let's not deliberate more about my general thoughts about China, Shanghai or Suzhou at this point, but get to the actual work: Today I spent at the FIC Suzhou factory, mainly doing final QA/Testing of the first 300 Neo1973 GTA01 phones. While I was doing this, another 192 phones went through assembly, resulting in a total of 492 units available at this time. We have another 500 units pending throughout the next two weeks.

The overall quality of my QA checks was quite good. The factory is doing a good job, and we could not detect any production-introduced bugs. The tools provided to the factory for programming/testing of the hardware leave quite a bit of room for optimization though. Will have to start this optimization process next week, after my return to Taipei.

This also means that we can finally make another announcement about the overall project status very shortly. And it means that as soon as the web-shop is up and running, developers will finally be able to purchase Neo1973 GTA01. 8 months too late. Sorry for that. Too much politics and too little actual technical work. All fixed now. Bright future ahead :)

Everyone is busy at OpenMoko

A number of people keep asking me what's going on with regard to openomko. I've even virtually stopped to update this blog quite some time ago. As much as I regret the lack of updates: Be assure they're just a sign of how busy everyone involved is.

I have, in fact, even cancelled my already-accepted paper and corresponding presentation at OLS this year :( I'm also not speaking at any other 'traditiona' event this year, not at Linuxtag, Linux-Kongress, CLUC, LTC, 0sec. Sorry, guys, maybe next year again.

I can't publicly state what's going on with regard to OpenMoko internally, but let me assure you: Good things are happening. We're working on a lot of internal changes that should enable us to approach the project with way more bandwidth.

The first couple of hundred GTA01Bv4 phnes have been produced by the FIC's mass production factory in mainland china. I'll personally do QA on 10% of those phones throughout the second week of June. We want to make sure we don't have any mishaps with our first customers, do we?

The first generation GTA02 prototypes have also showed up at FIC in Taiwan. More news on that at some later point :)

becoming a self-proclaimed election observer

Today I will leave to the German state of Sachsen-Anhalt, as part of a CCC group that will observe the use of electronic voting machines (rather: voting computers) at the elections there.

Our main focus is to witness and collect evidence of the many shortcomings in even the current (by no means sufficient) rules and laws on the security of those devices.

As a Dutch hacker group in cooperation with the Berlin CCC has demonstrated before, the voting computers in question are by no means safe against manipulations - neither are the corresponding safety procedures and measures.