Harald Welte's blog


Harald's Web




Other Bloggers
David Burgess
Dieter Spaar
Michael Lauer
Stefan Schmidt
Rusty Russell
David Miller
Martin Pool
Jeremy Kerr
Tim Pritlove (German)
fukami (German)
fefe (German)
Bradley M. Kuhn
Lawrence Lessig
Kalyan Varma


Ohloh profile for laforge
Linked in

Creative Commons License
Articles on this blog/journal are licensed under a Creative Commons Attribution-NoDerivs 2.5 License.



Wed, 06 Jul 2011
SIM-unlocking the Openmoko phones?

I think it's quite funny that SIM-unlicking vendors like RebelSIM actually advertise that their products are compatible with Openmoko, as you can see in this PDF file.

What's funny about this? Well, Openmoko phones have never been sold with any form of SIM or Operator locking. The entire idea was to have a phone that is under the control of the user, not the operator...

[ /linux/openmoko | permanent link ]

Tue, 07 Apr 2009
Closing the "Openmoko" section of this blog

I have decided to quit blogging in this section. I have left Openmoko Inc. quite some time ago. Openmoko Inc. has announced to discontinue Linux smartphones, and I'm not interested in any Project "B".

For those who want to follow what I have to say about Linux on mobile devices, I have created a new section linux/mobile in this blog.

[ /linux/openmoko | permanent link ]

Cancellation of GTA03 / thoughts on OM Inc.

As you can see at lwn and h-online, as well as slashdot: Openmoko Inc. has discontinued smartphone related development.

It's good that there is now some official statement from the company. They way this came about was less than pleasant, though. There would have been a number of ways to avoid the need for discontinuation.

For me, things now finally come to an end. I still very much believe in the idea of having more open mobile phones, both on the software stack as well as on the hardware side. I also believe that there used to be a number of very good people inside Openmoko who could drive the development to create such open products. But over time, I have started to have severe doubts whether Openmoko Inc. is really the most productive and/or best environment to do this kind of development. Priorities and directions changed a lot.

So as of now, even though Openmoko Inc. states it wants to re-start smartphone development at some later point, I am happy that I can finally let go. I do no longer have to hope that Openmoko Inc. gets their act together to actually get an (to my standards) acceptable product out into the market.

The pain and suffering is over. The engineers behind the 'open smartphone project' are all "free" now, and they are more than ever interested in continuing the mission that they once set out to get done. Very much like me some time ago. I've never stopped believing in the idea of building more open mobile phones. I just started to doubt if Openmoko Inc can do that, which is one of the key reasons why I left a very key position at the end of 2007. Now my doubts are "complete". People can move on and try to find a way to get what they were fighting for - probably with less interference and no side-tracking and constantly changing priorities.

Now some people might argue that I'm trying to do Openmoko-Inc-bashing here. That is not the point. I, as an individual, have just expressed my doubts and my assessment of the situation. I've held back a lot of grief for a long time, trying to not interfere with the business of Openmoko Inc. But since the Openmoko project is an open source project, and it is developed and supported by a much larger community, I feel more connected to that community than to the company. I feel obliged to let them know my opinion, since I have more insight into the project than most other people have. It's a personal opinion, I don't claim that it is "the one and only truth (TM)".

[ /linux/openmoko | permanent link ]

Tue, 31 Mar 2009
Openmoko [again] loosing some of their key engineers

I did not want to blog about it due to my intricate knowledge of Openmoko internals and my own past within the organization. But the news has made it to the various mailing lists now anyway: Andy e.g. is now longer working for Openmoko. He has been the main Openmoko kernel and bootloader developer (and maintainer) ever since I left in November 2007.

This is really sad news. There used to be really great engineers at Openmoko some time ago, but at least a number of good, senior folks are no longer working there at this point in time, or are working on a much smaller scope for Openmoko Inc.

Sure, it is not the right way to discuss the details of every HR matter in a public way. But I would have at least expected Openmoko to use the power they have to publish a statement on what this means _before_ the news get released in an out-of-control way by rumors and hearsay. If you allow this to happen, then you subject yourself to somebody else presenting their [distorted?] view of what he believes as reality first, and you (Openmoko Inc.) get into a defensive position.

[ /linux/openmoko | permanent link ]

Tue, 10 Feb 2009
Pavel Machek looking for Android exploits

In his recent bog post and mails, well-known Linux kernel developer Pavel Machek is looking for exploitable security holes in his Android phone in order to root it.

He is addressing the very fact that I also cannot re-iterate often enough: If I buy a product, then I own it. If I own it, only I decide what software runs on the device or doesn't run. The cell phone makers and mobile operators make us buy the devices, not rent them. So they have not the least right to restrict their customers from doing whatever they want on the product they own.

If those companies want to own their devices, they should resort to renting phones rather than selling them. But rather than following this logical consequence, they decide to use technical measures to distort the ownership/property situation of the product. They can still charge the customer for the full price of the product and literally sell it, while not actually transferring the inherent power of ownership. It's like selling a car but don't giving the keys along with it.

This is now the "Thanks" that the Linux Kernel developers get from companies like Google, Android or T-Mobile. They thank for being able to use the Linux kernel with locking out the very same people who wrote that kernel in the first place - as well as every other legitimate FOSS developer who wants to exercise his right to run modified versions of the program.

Welcome to the brave new world. I wish I had the time and resources to fight an example case against this kind of behavior. It is not against the wording of the GPLv2, but for me (and my lawyers) definitely against the spirit and the intent of it. Maybe I need to change priorities, as it now isn't only Motorola but also Google/Android/T-Mobile who are engaging in this outrageous act against what is probably the most important practical freedom of Free Software.

For the Motorola MAGX devices, OpenEZX hackers were luckily able to find relatively simple ways to circumvent its security, and thus the practical immediate need for any legal action. Let's hope the G1 has a similar fate soon.

[ /linux/openmoko | permanent link ]

Sat, 24 Jan 2009
Taking the DX900 apart, taking PCB photographs

I've taken the DX900 apart and taken the usual PCB photographs. You can find them attached to the DX900 page in the gnufiish wiki.

In the process, I learned some new bits about the DX900:

  • They have used a Sunplus 2.5G modem (next to their usual 3.5G modem design)
  • They have added a second small CPLD (2C64)
  • They are now using a Power Management IC (older glofiish don't)
  • The DX900 does no longer have the standard glofiish test pad arrangement like all earlier devices
  • There probably is no M900 (key-board version), since the PCB is now heavily using components and shielding covers on both sides, adding to the overall thickness of the device

[ /linux/openmoko | permanent link ]

Fri, 23 Jan 2009
Some reverse engineering on the E-TEN glofiish DX900

Today I've found a bit of time to start my reverse engineering efforts on the E-TEN glofiish DX900. In case you don't know what that is: It's E-TEN's latest PDA-phone model, 2.8" full-VGA touch-screen, GPS, WiFi, Bluetooth, 3.5G and dual-sim (it has a 2G modem next to the 3.5G modem). It runs the Samsung S3C6400 application processor. It is so new, that it doesn't even yet have FCC approval. Luckily it was available in Taiwan at PDA king for NT$ 16,900.

It seems like a great device. There's basically only one big flaw: It runs Windows Mobile. And it does that in a really bad way. From how sluggish and unresponsive the UI is, you can clearly see that they don't use any of the 2D acceleration functions of the S3C6400 hardware.

Now in order to get rid of Windows Mobile, we need to discover more about the device hardware. The first step for that is HaReT, the Hardware Reverse engineering Tool. Unfortunately the S3C6400 is so new that HaReT doesn't yet have support for it. So I had to dig into the code and add support for it. You can find the preliminary patch here.

The information that I was able to dig out using the first round HaReT can be found at this DX900 gnufiish.org wiki page. As expected even before touching the device:

  • the 2G modem connects to a UART
  • the 3.5G modem is SPI slave just like in the M800
  • the wifi is still Marvell 8686 connected to SPI
  • the GPS is still SIRF3 connected to a UART
  • the buttons still are the same, some connected to GPIO some not

I won't have much time to work on this right now, as too many other higher priority tasks are pending. But it seems like the DX900 is a nice s3c6400 based device to play with, and a Linux port to it is not really further away than for the M800 or X800

[ /linux/openmoko | permanent link ]

Mon, 12 Jan 2009
Photos from Korean barbecue with Samsung System LSI kernel developers

Just in case you're interested: Kwanghyun La has put up some pictures from the barbecue that I enjoyed with the kernel developer team there.

[ /linux/openmoko | permanent link ]

Sat, 10 Jan 2009
An exciting week at Samsung LSI

I've just finished one exciting week at Samsung LSI (the group inside Samsung responsible for the System-on-a-Chip line like the s3c24xx, s3c64xx as used in Openmoko, TomTom and E-TEN glofiish devices). Specifically, I had one week of close contact with the team there developing the Linux kernel port and drivers for those products.

I do not want to go into too many details here in public, but it was a very productive week, and I think everyone involved is drawing a very positive conclusion. Let's hope the actual results from the now-improved understanding of the Linux and FOSS development model will be of mutual benefit to the community as well as Samsung. I'm also hoping for better and faster integration of Samsung's codebase into mainline Linux.

Parts of this Openmoko-triggered drive can be already seen on git.kernel.org, where Samsung LSI has started to push their current development trees to.

Since this is not the first time that I'm trying to lobby for more understanding of the Linux development process, the benefits of going mainline, etc. - I wonder if this kind of work should not be done in a much more scaled and proactive way by somebody like the Linux Foundation. Especially in the embedded world, there are many companies who are probably very interested, but just don't know where to start or whom to talk to. There just is no (or no easily identifiable) entity catering to their needs - and since they are always busy with developing new products and working on ports for other operating system, the initiative should probably come from the outside.

Right now this field is left to embedded distributors who have their own agenda and are always only representing a small fragment of the Linux stakeholders

[ /linux/openmoko | permanent link ]

Fri, 19 Dec 2008
FreeSmartphone.Org (FSO) developer meeting in Braunschweig

Yesterday I had the pleasure of attending a developer meeting of the FSO core developers Mickey, Daniel, Stefan and Jan in Braunschweig, Germany.

So far my actual involvement with FSO code has been minimal, apart from some profiling here and there, as well as a couple of comments/opinions, mostly offline and not on the mailing lists. I think this is sad, since FSO is the best thing that ever happened inside Openmoko. Focussing on the actual platform/architecture/middleware, standardizing and implementing the interfaces by which application programs interface with the actual device.

So while I haven't been able to contribute much to the python reference implementations, I hope I can contribute a bit more in the future. Also, my involvement with Swisscom Innovations as well as the gnufiish project will probably sooner or later drive me into touching some more FSO code.

It was good to hear the various reports and to see how much thought is given to the various details. Most notably was the quite lengthy debate about how a suitable battery / power supply API should look like. The devil is in the details.

As far as my actual work at the meeting is concerned: I've been asked by the FSO guys to show them how to play back PCM audio into a GSM voice call, and how to record a GSM voice call. Allegedly nobody has ever done this inside Openmoko again, after I demoed it ages ago, most likely still on GTA01.

The resulting information can now be found in the wiki. Unfortunately the actual capture is not working, apparently due to a ASoC driver kernel bug which I tried to debug but gave up after some intermediate results, since my understanding of the audio subsystem is limited and I have tons of other tasks.

The other bit I've been working on is a serial port LED trigger, i.e. the ability to make a LED class device blink if RX or TX activity is detected on a serial port. The code is not finished yet, but will be hopefully soon.

We've also been talking a bit on how to integrate keyboard-based devices with FSO, i.e. how the framework should indicate this to the window manager. The key part is that we're not as much interested in the actual existence of the keyboard, but the fact of whether it is slided out or not (for devices with a slide, such as the glofiish M800 or the TyTN/Kaiser).

Some further bits were spent with Stefan trying to hook up the libertas GSPI driver with the S3C24xx SPI host driver in order to get WiFi to work in project gnufiish.

[ /linux/openmoko | permanent link ]

Mon, 15 Dec 2008
glofiish M800 keyboard driver

I've been hacking on a glofiish M800 keyboard driver, and most of it is now implemented. Even the Caps and Fn LED are in operation, and sliding out the keyboard causes a led_trigger to enable the keyboard backlight.

However, I still get the timing wrong when to read the CPLD. In most cases it works, but sometimes I get bogus characters and the like. Also pending: A more comprehensive keymap, plus support for the Fn-shift-key.

[ /linux/openmoko | permanent link ]

Wed, 10 Dec 2008
ALSA SoC driver working on E-TEN glofiish M800

After some hacking on this on the airplane back from India, and some finishing touches today, the audio output on the M800 is now working under Linux. The AK4641 codec driver and M800 ASoC device driver are to be found in the gnufiish git tree.

What I yet have to verify are the interconnections with the GSM Modem audio, as well as the FM radio. At least general audio playback is working, both through the earphone and ringtone speakers, as well as on the headset.

[ /linux/openmoko | permanent link ]

Tue, 09 Dec 2008
Received a Google/Android/T-Mobile G1

Due to a friendly person in Taiwan, I have now received a Google/Android/T-Mobile G1 device. Not that I'm interested in actually using it as my regular phone, but just to play a bit around with it. I've already had a chance to use the device for a bit, and I think it's surprisingly unimpressive. Mechanical quality of the swivel and keyboard is OK but definitely not as good as e.g. the HTC Kaiser/TyTN2.

What I'm much more interested in is to confirm my my earlier suspicions on the lack of openness of Android, or at least the actual devices based on Android.

As it seems, in fact it only accepts cryptographically signed kernels, and the signatures that are accepted are not signatures of the user who has bought and owns the device, but rather the signatures of Google/Android/T-Mobile/HTC.

I still think it's extremely weird that you actually buy a device, and then don't own it. I would have no problem if the device is rented from the manufacturer or the mobile network operator. Sure, then in this rented device, only they control what kind of software you use. But this is not the case. People buy it, pay money, legally own the device but technically don't.

I've seen that there are some hacks, including one one that puts the engineering bootloader on the G1. Sure, there will always people who exploit known security issues in hardware, ROM or software, just like the guys who work on Linux on the iPhone, or the now-proclaimed-dead Motorola MAGX platform. However, this doesn't address my point. Those kind of hacks will be closed at least with the next hardware release, and they can only be performed by a really small portion of the actual users (owners).

A truly open device (including mobile phone) has to give the user the freedom of choice what code he runs... Just like on the PC. You can run any OS you want, any App you want. Hell, you can even go ahead and write your own OS if you want. And the G1 is definitely not giving you that kind of freedom.

[ /linux/openmoko | permanent link ]

Thu, 20 Nov 2008
E-TEN glofiish M800 Linux tree now has public git tree and a name

You can find the git tree here and clone it from git://git.gnufiish.org/gnufiish.git. Thanks to Stefan for setting up the tree and doing the initial push (ssh+git from Taipei is so slow that it cannot finish the initial commit of a kernel Tree before the DSL modem reconnects 12 hours later).

I've called the project "gnufiish" just for the fun of it. It's quite close to the original name, and therefore a 'language hack'. Though it's obvious that there is no real official connection to the GNU project, since Linux is not part of that.

[ /linux/openmoko | permanent link ]

Mon, 17 Nov 2008
Digging into the internals of WinCE / WinMobile

My E-TEN glofiish Linux porting efforts have made me investigate a lot of internals of the E-TEN ROM file format, WinMobile ROM files in general, XIP, Microsoft flash partition tables, imagefs and other bits and pieces.

I'm basically able to fully 'decompose' a ROM image into all its individual bits, including the pre-installed CAB's, the pre-linked DLLs in the XIP and the contents of the imagefs. And all of that on Linux, if it wasn't for the weird XPRS / SRPX compression in CE5.x imagefs. Allegedly it's the same Xpress algorithm as used e.g. in hibernation files and certain M$ network protocols, but I was trying that and didn't get anywhere. Luckily, the tools at least ran inside wine.

It's surprising how little information there is about those internals of the operating system. You can find bits and pieces in the 'ROM cooking' scene, but those are mostly HTC specific and don't always apply to E-TEN. And most of the tools that people tend to create in this community are not FOSS either :(

Anyway, once I find some time I'll probably pack/publish the stuff that I've done now. Obviously the coolest thing would be to do a GPL'd implementation of e.g. imagefs and get that into Linux. Would be fun (I've never ventured into filesystem land!), but then, it's not like I have any spare time at hands.

Last night I was trying to make sense of some of the M800 hardware drivers (sergsm.dll, keypad.dll, keybddr.dll, etc.) but it's actually quite a bit harder than I thought.

I also wrote some perl script that uses haret TRACE capability to reconstruct the I2C command/response stream. so you can basically perform any action on the device, like pushing one of the capacitive touch buttons, and see what kind of I2C communication the CPU initiates as a result. The problem with this, though: The I2C bus runs too fast, so it loses some bytes. I tried to work around it by increasing the I2C clock divider, but it seems the driver actually re-sets the divider with every transfer (rather than just once when bringing the I2C host controller up).

I'm trying to find other options (I could clock the entire system down, but then this affects things like the LCM refresh and other important clocks), since I believe a clean I2C tracer is the right thing to do.

I've also spent a bit of time on the Marvell 8686 driver, as there is already some (not entirely polished and thus not mainline yet) GSPI transport code for the libertas driver. However, I didn't finish this since it is not the biggest priority right now. Also, interestingly, the GPIO and other related bits regarding the wifi chip are all present in the winCE registry. Marvell apparently made the driver in a way that E-TEN and others don't need any access to its source code but can fully parameterize it through the registry.

So as a summary: I was spending basically every awake minute during the last days on this project, but there's no real visible progress yet. I've just learned a lot, and hope to use that information soon to further improve the Linux port.

Oh, and by the way: It seems like I'll be talking about some of this work (and actually showing how it was done) at FOSS.in 2008 next week. Stay tuned for some good old fun ;)

As with actual progress on the device itself: I've spent quite a bit today again with reverse engineering some drivers, thereby discovering two GPIO's that seem to be related to GSM modem power management. Maybe that's the reason why my own humble attempt at a Linux GSM driver has so far been unsuccessful. I also seem to find an awful lot of indication that UART0 is actually connected to the GSM modem, too. This might be some strange copy+paste artefact from older glofiish devices' linux driver, or actually they might have two independent communications channel to the modem - wouldn't be the first time to see this.

Some other bits have hinted at an externally-provided UART clock, but that is apparently just a workaround of a S3C2442 serial controller bug.

I', still having fun wading through tons of ARM disassembly. It's been a long time since I last had any good reason to use IDA (Interactive Dis-Assembler) that much. It's the only proprietary software that I've been willing to license (and thus pay for) in something like a decade.

[ /linux/openmoko | permanent link ]

Fri, 14 Nov 2008
Glofiish M800 GSM/UMTS Modem interface reverse engineered

During my seven hour stopover in Bangalore, I decided not to sleep and rather have a look about what I could do to find out more about the yet-unknown interface between the S3C2442B application processor and the 3.5G Modem in the E-TEN Glofiish M800.

Some initial poking in the WM6.1 registry led me to the (wrong) conclusion that UART0 might be used. It would have been a lot of data for a UART anyway...

So as it seems, they're using a SPI based interface. Not a bad choice, considering the various suboptimal alternatives. USB is way too heavyweight and power-consuming, and leads to inevitable problems when you want to resume the application processor from suspend (e.g. on incoming call). You just simply cannot afford the time to enumerate the USB, etc. Some shared memory / dual ported RAM interface like it is found in more integrated chipsets requires quite a bit of software work (synchronization of a shared memory region between two processors that have no common resources!) and requires a quite deep interface into the modem side. Something that E-TEN would unlikely get from Ericsson, I would say.

So SPI it is. Interestingly, the SPI master is in the modem, the S3C2442 acts as SPI slave. This adds the need for some kind of mechanism how the application processor can tell the modem that it actually wants to transmit an AT command. A simple GPIO line does that trick. The Modem responds by asserting the slave select line.

Interestingly, they even use DMA accelerated data transfers. So receiving data from the modem is less CPU intensive than reading data from NAND. What a crazy world.

Some more bits are found in the wiki.

I've already started to hack up a Linux driver. The SPI side is really simple. What is much harder is the fact that the Linux SPI core has no support for slave mode, and thus neither the in-kernel s3c24xx SPI driver. Furthermore, many of the traditional serial line analogies (baud rate, modem control lines, handshake, break, ...) no longer apply.

On top of the SPI, they seem to be running pretty standard AT commands. Nothing fancy at all. Thus, I'm optimistic that once the kernel driver is there, FSO or other userland can make use of it quite easily.

[ /linux/openmoko | permanent link ]

Tue, 11 Nov 2008
Running Linux on E-TEN glofiish M800

Ever since my blog post about certain E-TEN glofiish devices in late August, it might have been obvious that I've been up to something.

In fact, I didn't have much time, as usual. Finally, after something like about two full days of work, I can present some preliminary results:

root@glofiish-m800:/proc# cat /proc/cpuinfo 
Processor       : ARM920T rev 0 (v4l)
BogoMIPS        : 176.53


Hardware        : Glofiish M800
Revision        : 0000
Serial          : 0000000000000000

You can also find a preliminary wiki page about the current status of hardware reverse engineering in the OpenEZX wiki. It doesn't really related to EZX or OpenEZX at all, but it somehow is related to the same thing: Bringing kernel+rootfs based 100% on open source to phones without vendor support. It also doesn't really fit into the Openmoko wiki, since as you can assume, this is by no means a project of Openmoko, Inc.

So far, it was pretty easy. I was taking the 'stable' branch of the Openmoko kernel git tree, adding minimal platform support to it (to get framebuffer, microSD and USB device working), and using haret to boot into a fso-terminal-image located on a microSD card.

Of course the really hard work now starts, getting all the hardware properly supported, especially the communication with the GSM Modem, as well as the power management related bits. Nonetheless, a foundation is laid, and I expect it to be not too hard to continue from here.

So maybe, if I can find sufficient time, we will see FSO on a 3G phone at some not-too-distant point in the future.

Now some of you might be asking: Why am I not working on improving the code for the Openmoko, Inc. handsets GTA01 and GTA02? Isn't it bad to support a non-open hardware manufacturer, plus pay the Windows Mobile license tax on a device, ...? After all, Openmoko, Inc. current business model is centered around the sales of their own hardware to support for the software development!

I don't think that this is much of a competition to Openmoko. Obviously, everyone wants truly open hardware, such as what Openmoko, Inc. is trying to do. Nonetheless, people (especially geeks/nerds/hackers) want devices with 3.5G or at least 3G, they want devices with real keyboards, higher capacity batteries, better mechanical design, camera, etc. This is just something that Openmoko Inc. has not been able to provide so far. There's probably not many people on this planet who feel as sad about this fact as I do.

[ /linux/openmoko | permanent link ]

Mon, 27 Oct 2008
Some more S3C24xx NAND speed observations

I've now moved on to other topics, but I still want to mention some of my thoughts on the still quite poor NAND performance on the GTA02 (and generally, the S3C24xx).

It seems like we are down to a point where the CPU is 100% busy reading from NAND, which is odd. Why would reading from a mass storage device make the CPU so busy? Well, because Samsung "forgot" to add DMA support to all of their integrated NAND controllers, from the old 2410 through the 2440, 2442, 2443 and up to the shiny new 6410, all the NAND controllers don't support DMA. In fact, they don't even have a FIFO or some kind of internal buffer for the received data. This is really weird, considering the facts that

  • every other peripheral (SD/MMC, SPI, UART, ...) can use DMA
  • Samsung as provider of both NAND flash and SoC should be experts in providing good flash performance
  • I cannot see any strong architectural limitation. The data is read into a register. The register should be replaced with a FIFO, and a DMA can regularly read from that register or FIFO and put it somewhere else into memory. It's not any different from e.g. SPI or UART DMA.

In the current mainline Linux driver for S3C2440 NAND, we busy-wait (poll) for completion of the read request. This is of course sub-optimal. I implemented a version that uses the Read/Busy line IRQ and a 'struct complation' based mechanism and to my big surprise the CPU usage and throughput was identical. It seems like that NAND flash in the GTA02 is fast enough to max out the CPU.

So probably all that can be done is to optimize the actual code that is executed during the NAND read. It might be worth implementing some small hand-optimized assembly implementation as standalone code (not using the existing driver) to see how far the hardware can actually go.

Isn't it sad that you use Samsungs SoC and Samsungs NAND (packaged together in one component as multi-chip-package by Samsung) and still get less than 50% of the performance of the NAND chip, according to the data sheets :(

[ /linux/openmoko | permanent link ]

Thu, 23 Oct 2008
Android and its perceived 'openness' :(

As many other people have been blogging and news sites have been reporting: The Android source code has been released. This is definitely good news. However, freedom-loving people already discover in blog posts that there's a remote kill switch by which Google can disable an already installed application and that some features are reserved to vendor-signed applications.

To me, those things are not a big surprise. As soon as you try to get in bed with the big operators, they will require this level of control. Android is not set out to be a truly open source mobile phone platform, but it's set out to be a sandbox environment for applications.

And even with all the android code out there, I bet almost (if not all) actual devices shipping with Android and manufactured by the big handset makers will have some kind of DRM scheme for the actual code: A bootloader that verifies that you did not modify the kernel, a kernel that ensures you do not run your own native applications.

Thus, Android so far was little more to me than yet-another-J2ME. Some sandbox virtual machine environment where people can write UI apps for. In other words: Nothing that gets me excited at all. I want a openness where I can touch and twist the bootloader, kernel, drivers, system-level software - and among other things, UI applications.

I actually think it's a bit of an insult if people think of Motorola's EZX or MAGX (and now Android) phones as "Linux phones". Because all the freedoms of Linux (writing native applications against native Linux APIs that Linux developers know and love, being able to do Linux [kernel] development) are stripped.

In the end, to what good is Linux in those devices? Definitely not to any benefit of the user. It's to the benefit of the handset maker, who can skip a pretty expensive Windows Mobile licensing fee. Oh and, yes, they get better memory management than on Symbian ;)

That's the brave new world. It makes me sick.

Luckily, it's 50 B.C. and the Romans occupy all of Gaul, except for one small village that has been able to avoid the invaders.

[ /linux/openmoko | permanent link ]

Mon, 20 Oct 2008
Openmoko GTA02 NAND performance improvements

On Sunday night, after returning from a weekend trip to Hamburg, I sat down and looked at the NAND and S3C2442B data sheet to figure out the actual timing performance. Interestingly, the NAND timings were much more verbose and detailed (and had different names) than the timings described in the NAND controller section of the S3C24xx manual - and both are from Samsung ;)

Anyway, it seems like the current timing settings for the various stages (reading u-boot by the stepping stone mechanism, reading the kernel by u-boot as well as actual mtd-based access inside the Linux kernels) were extremely suboptimal. They're basically standard timings designed to work with most NAND flashes out there, ignoring the fact that GTA02 uses one specific flash with very good (fast) timings, at least according to the data sheet. There should also be no PCB / routing related issues such as capacitive overload preventing higher speeds, since the NAND flash die is stacked onto the CPU die in one package, and the NAND controller signals are not routed on the PCB anywhere.

Some initial experiments based on the calculations show that the performance can be easily improved by 41% over the stock GTA02 NAND performance. However, the actual speed (6.59MBytes/sec) is still much lower than the theoretical maximum read performance of 15.64MBytes/sec. It seems there is more room for improvement inside the MTD layer of the Linux kernel.

It's again quite amazing how much room there is for improvement in GTA02 performance, both power consumption wise (see my recent post about framebuffer blanking), as well as actual data throughput. Those are really low-hanging fruits, and it's very surprising that nobody working for Openmoko or in the Openmoko community has been able to spend some time to look into those...

[ /linux/openmoko | permanent link ]

Wed, 08 Oct 2008
Significant power savings during framebuffer blanking

As I've posted today to openmoko-kernel, there have been some quite significant power savings during the "backlight off but still not suspended yet" operational mode of the GTA02. The power savings are in the order of 49%, which is really massive, particularly for applications that run in the background while the screen is blanked, like typical mp3/ogg player applications.

It is sad to me that something like this is found long after the GTA02 has actually shipped. It seems like there are still fairly low-hanging fruit around to do some significant power saving.

Since all the measurements can be done on the device itself, using the built-in high precision coulomb counter of the battery, everyone is able to do the measurements without any special equipment. It also means that power management related issues can be tested automatically.

I would love to see somebody working on software that switches certain hardware on (and off) again or cycles through various differnent power states of every hardware unit an then reads the power consumption. The resulting readings can then be manually checked if they're consistent with expectations based on the hardware design. Furthermore, this program could be used for regression testing against new uboot/kernel/OS releases in order to ensure we don't get into power consumption regressions.

[ /linux/openmoko | permanent link ]

Tue, 07 Oct 2008
Actually working on Openmoko again

It's an interesting feeling to spend some days working full-time on Openmoko again. Swisscom was stating a number of high-priority bugs (for them) which I tried to resolve.

The first two are u-boot related, namely: get GSM passthrough working again, and fix USB DFU Upload on GTA02. Those two should be doing quite fine now.

I've also been investigating possible ways to optimize CPU usage of frameworkd, although it is not yet clear which of the possible solutions should be implemented in the end

Right now I'm working on some power management related issues, mostly glamo/backlight/LCM related, as well as re-investigating the hardware-ECC work by Zecke.

However, after a significant break from _using_ the Openmoko devices and the software available for them for a number of months, I have to say that the overall experience was really disappointing.

  • Whatever Openmoko builds as their daily builds available on downloads.openmoko.org is the most unintuitive UI that I've ever seen (is that ASU?). After some attempts, I gave up. unusable for me.
  • FSO images can be installed, but are incredibly slow
  • Documentation in either openmoko wiki or FSO wiki is horribly outdated
  • It's _really_ hard to get devirginator running since lowlevel_foo and others are not available on downloads.openmoko.org, but devirginator insists on downloading them from a website rather than copying them from a local directory
  • there's a neverending fragmentation
  • core aspects of the system level have not been addressed, like replacing sysvinit with something like upstart, some serious boot speed optimizations and various power management related bits
  • Nobody has yet had the time and resources to investigate a thorough, flexible and performant storage and API subsystem for contact + related data

All this makes me really sad and gets me back to the point where I feel like when I left OpenMoko, Inc. last year: Too many insurmountable problems, and very few that can be addressed in a way that they are solved once and forever. Everyone runs their own personal little pet system, magic scripts, revision control system, overlay files, images, etc. Still too many people think OE is a tool to develop+crosscompile application programs.

[ /linux/openmoko | permanent link ]

Thu, 25 Sep 2008
Swisscom Research is evaluating Openmoko

At OpenExpo, Swisscom Research had it's own small stand (wouldn't call it a booth) to demonstrate thei research and evaluation work based on Openmoko. This is definitely exciting news, first of all since it is the research department of an established carrier, i.e. Openmoko is considered seriously even by them.

Secondly, they have many interesting ideas, some of which they have implemented. They have created a much more simplified UI, as well as an interesting input method based on gesture recognition. They've also been working on some crypto and security related bits.

I can now also disclose the fact that both Rasterman and myself have been (and stilll are) providing a bit of consulting and R&D services for Swisscom.

[ /linux/openmoko | permanent link ]

Wed, 26 Mar 2008
I don't work for Google - no matter what the rumors say

A number of people have recently independently approached me about rumours that I'm now working for Google/Android, after having left OpenMoko, Inc. in November 2007.

According to one source, some friend who visited Android was told by Android that I would be now working for them. There is no truth to this.

Please put an end to those rumours. I'm not working with or for either Google or Android. There also are no plans to do so, and there have never been any negotiations, aside from the usual Google headhunters that approach anyone visible in the FOSS world every so often - which I always decline, indicating that I am not interested in a dependent employment position, no matter for which company.

I will continue to be doing freelance contract work on projects that are interesting to me and within my fields of expertise. Should anyone chose to approach me with an interesting technical Android system-level and/or hardware related project, that would certainly potentially be interesting. But I'd look at it like any other inquiry.

[ /linux/openmoko | permanent link ]

Thu, 13 Dec 2007
Final cleanup of OpenMoko Neo1973 kernel patches

I'm doing one final review+cleanup iteration for the OpenMoko Neo1973 GTA01 related kernel patches before pushing them for review later tonight or at some point tomorrow.

The cleanups are mostly dead code removal, avoiding compile-time warnings as well as cosmetic cleanups such as adding MODULE_DESCRIPTION to all modules, and using consistent naming for files and driver names.

GTA02 will have to wait a bit more. On the one hand, changes that the kernel developers want me to do on PCF50606 will likely appear in the PCF50633 driver, too. On the other hand, the entire Smedia Glamo driver core has not been polished yet.

[ /linux/openmoko | permanent link ]

Wed, 05 Dec 2007
Merging OpenMoko patches with u-boot and kernel mainline trees

Those following the OpenMoko commitlog will have noticed quite a bit of activity in the areas of u-boot and kernel patchset. For u-boot we always tried to track mainline git. For the kernel, there is now a patchset against the current Linus git tree (2.6.24-rc4 should also work) at http://svn.openmoko.org/branches/src/target/kernel/2.6.24.x/patches.

I'm intending to do some level of testing after the merge, and then submit the majority of the stuff. For kernel, I don't expect many issues. For u-boot, it will be a quite painful process.

So if you now think this means that I'm back working for OpenMoko, this is wrong. I'm doing this for a personal reason: I merely want to make sure that the code I wrote throughout the last 18 months will not bit rot somewhere but is actively merged into mainline.

[ /linux/openmoko | permanent link ]

Fri, 16 Nov 2007
Leaving OpenMoko "Lead System Architect" position

The regular reader of this blog will have noticed a distinct lack of any OpenMoko related news. This is not a coincidence. As Mickey wrote in a blog entry, there has been quite a bit of internal friction lately.

Adding that to the enormous amount of stress over the last 18 months has made me feel quite a bit demotivated over time. I've tried to cut down on the amount of work I do, but it hasn't helped much. So now I'm at a point where I feel unable to work in any active/leading role inside the project and/or company. My deep apologies to the the project and its community.

But don't worry. OpenMoko is a team, and I'll do everything to help smoothen the transition. I'm more than willing to assist those who will take care of my various tasks.

From today on, I'm nothing more than a volunteer to the project. I'll likely continue a bit of hacking in my areas of personal interest. Just like in many of the other FOSS projects that I have been (or still am) involved.

All the best to the OpenMoko project, the OpenMoko company, FIC Mobility. The last 18 months was an intense experience. Thanks to everyone who has helped the project both inside FIC/OpenMoko and outside. Thanks to FIC for funding and supporting the project. I wish everyone the best, and I'll be the most happy person (next to Sean) at each and every milestone the project achieves in the future.

[ /linux/openmoko | permanent link ]

Fri, 19 Oct 2007
Heading back to Germany

So, after roughly two weeks of OpenMoko Taipei headquarters, I'm now heading back to my Home+Office in Berlin. And I'm really looking forward to it. During the last couple of months I've tried my best to help the transition from OpenMoko a a project inside a FIC business unit to OpenMoko, Inc. the independent company inside the FIC Group. I've helped with tons of things that are definitely by no means related to kernel/bootloader development, or even the hardware architectural planning that I've been heavily getting involved starting with GTA02.

So now it is a good time to finally focus again on what my actual and original task is: Software development. This can be done much better remotely, so I'll expect to be able to work way more from Berlin. Which makes me happy, since it always was and still is my favorite city. And I definitely missed it a lot during the last year of intensive OpenMoko work. Now I'm on my way back, and I'm looking forward to spending more time with my friends, the CCC Berlin. Being able to go to concerts and clubs that play music I actually like. Being able to work on finally improving (or rather: finishing) home improvement in my apartment, and many other things.

So don't get this wrong. I'm very much continuing my technical work for OpenMoko, and as the first developer on this entire project and a OpenMoko core team member, I'm always going to maintain an influential role in the project. But finally, I can go home, feel better, work more focused and efficiently, and improve the technical quality of our products even more :)

Since OpenMoko now actually has three full-time paid project members in Berlin, It's also going to be nice to closer cooperate with them. (or co-work, like the Chinese English speaking would say)

[ /linux/openmoko | permanent link ]

Sun, 07 Oct 2007
Back to Taipei

Today I got back to Taipei, almost three weeks later than originally anticipated. More news after at least one night of full sleep and the first day at the office...

[ /linux/openmoko | permanent link ]

Sat, 01 Sep 2007
Two days of intense u-boot hacking

After finishing most of the basic device support for GTA02v2 in both kernel and u-boot during the last week, I've finally turned back to implementing one of the longest standing issues in u-boot: GSM passthrough.

GSM passthrough allows you to basically ignore the smartphone part of the device and connect the GSM modem more or less directly to a host PC. The feature has been long known by various smartphone hacker projects such as e.g. OpenEZX (which as a side note has made quite some progress recently, much appreciated by me as retired project founder).

So GSM passthrough is mainly useful for rapid development (developing gsmd more efficiently by running it on the developers workstation, without cross-compiling and ipkg installs), but also if you want to use some legacy application that was written at a time where a phone really only was a phone (e.g. sim card managers, ...)

Now the GSM passthrough was always pushed back on the TODO list, since our usbtty code in u-boot was never very reliable and caused lots of data corruption such as bogus and/or missing characters. Quite useful for the human operator, but definitely not acceptable for getting a program with AT command parser to work. So that had to be fixed first (and it is now fixed).

As I pointed out in my announcement, the generic way of implementing this feature has actually quite interesting but much more obscure use cases such as dialling from a landline via GSM (CSD call) into your Neo, manually accepting the incoming call and then attaching the u-boot command line to it. That's sort of the feature you have on hosted/colocated servers, when you use a boot-loader with serial console support and attach a modem or terminal server to it.

So does this mean the Neo1973 is now ready for the enterprise? Not quite. Even though it has a built-in UPS (called battery), and GTA02 will even allow you to change the battery without shutting down the device, resulting in higher availability ;)

But then, the expectations / requirements for mobile communications devices are quite a bit different from that world. But the hackers community likes those kind of strange features. Have you ever heard of another smartphone with that capability?

Oh, and before I get any complaints about the security: This "feature" has to be explicitly enabled and every call manually accepted by typing a sequence of commands into the u-boot command line. So unless the attack involves tons of social engineering (getting the device owner to do all those things) there's not that much of a big deal. But maybe we should start to think of some kind of user authentication for u-boot now *rotfl*.

[ /linux/openmoko | permanent link ]

Sun, 19 Aug 2007
OpenMoko Taipei office network setup

For the last three days I've been busy setting up the network in the new OpenMoko office. Finally something that just works, without any major flaws. That's probably because the involvement of external entities is fairly small. But looking at it closer, actually exactly there the problems start. FIC purchasing e.g. only bought 50% of the switch equipment that I asked for, something that I still don't yet have received any details about, neither some kind of apology.

In any case, the network is up and running. We now have a pretty uncommon heterogeneous combination of Cisco, Dell and even some D-Link switches, with the core switch being a pretty impressive Dell 6248. The 10GBit transceivers are still missing, so the backbone is just running 1GBit for the time being.

There's a total of about 190 Ethernet access ports throughout this new office, and we have various different broadcast domains, safely separating guest network, office network, r&d network, ... from each other.

The server room also looks quite nice now, with five 19" full-height 19" racks, 2 layers of UPS (building-wide and extra small UPS in front of servers), with three fairly big servers in operation, and eight more pending to be used soon.

The Internet uplink is a 100MBit capable physical layer (fiber optics), of which we only subscribed to 8MBit initially. But one phone call (plus additional transfer of money) later, we can bump the speed to any rate below that 100MBit physical layer limit.

The operation of our own (netfilter/iptables based, what did you think?) firewall now also brings us back the long-missed basic fundamental networking needs of any free software hackers (called luxury around here) of all our developers being able to

  • access the web without restrictions and blocked sites
  • finally use IRC!
  • access external IMAP4 servers
  • have SSH and other interactive sessions not time out every couple of minutes

[ /linux/openmoko | permanent link ]

Sat, 04 Aug 2007
Progress with the new OpenMoko and FIC Mobility office

The final 24 hours of my current Taipei trip have started. This is a good time to reflect on what has happened in those last weeks since July 9.

As with the overall status of the project, I'm still extremely dissatisfied. The frequent reader of this blog will have noticed the last postings on this subject, full of discontent.

So the further we are into this project, the more time we put into it - the further I expect it to produce anything that I would consider reasonable results. Please don't confuse this with the commercial success, or the ability to produce working products. This is an entirely different matter.

To me, it is extremely important to do things systematically, with lots of planning, safeguards, checks, verifiable and reproducible processes, as much automatization as possible, little room for human error, etc. So as long as not everything from hardware to software development, mass production, production testing, distribution/logistics/sales, etc. follow a well-thought-through process, I will not be happy with the results. Because any such "results" are more or less the mere product of luck or randomness, and not a trustworthy basis upon which we can rely on.

So reflecting on those past weeks, I think the following things have made humble and moderate progress:

  • GTA02v2, the second prototype generation of GTA02 was finalized after many issues including unavailability of key components. I'm more than looking forward to see how it turns out
  • DebugBoard v3, the third version of the Neo1973 Debug Board was finalized and is actually also verified and can go in mass production
  • Our internal software team finally has proper leadership and guidance from somebody who is both Taiwanese and has a thorough understanding of Free Software: jserv
  • The new, second (intermediate) generation user interface was implemented and released. It's implemented mainly by O-Hand, since embarrassing as it is, we still don't yet have managed to build a proper internal software development team.
  • The first batch of Neo1973 GTA01 was sold, though with a entirely last-minute, error-prone and way-too manual process for order, payment and logistics.
  • We have found a capable sysadmin for our hosted, publicly-accessible servers. More news about that in September.
  • We have managed to find a extremely valuable senior technical person for our graphics driver and low-level UI work. This, too, will make big news in September.
  • The FiWin (FIC wireless networks) company, home to the team working on the it-exists-but-nobody-publicly-knows-what-it-is HXD8, was merged into OpenMoko and FIC Mobility
  • We have finalized the specification for the workstations of our software developers. It's incredibly complex to find something that's compliant with our requirements (mainboard with Intel 945/965 on-board graphics, Ethernet chip != attansic/realtek, dual core CPU with 2x2048kByte cache) in Taiwan. So now our developers will all get a Q6600 CPU (what nonsense!). I've tested it, and it compiles the GTA01 kernel in 1.59minutes. Guess they'll be happy about that.
  • Realize how many things really are fundamentally wrong internally. What we knew so far about our inheritance was just the beginning ;)

One major thing that finally started to move forward, with something like four to six weeks of completely unnecessary delays, is the new office. After it was decided that we will split FIC MCBU into the independent OpenMoko, Inc. and FIC Mobility (aka Mobile Communications), we also decided to move into bigger, scalable and independent offices.

To our big luck, two thirds of the 7th floor in the FIC headquarters were currently unused, and they're now undergoing quite a bit of renovation and reconstruction. Walls have been removed and brought in, floors have been properly removed and new ones laid - after days of fighting by Sean and myself.

The networking and phone cables get a major overhaul and will be tested. I've also seen the AC for the new OpenMoko server room being brought in. The contract for our own Internet plink has been finalized. The fiber will be put in place within the next week. The core switches have been configured, but we're still fighting very hard to get those damn 10Bit XFP transceivers from Dell.

So the current schedule is to move on August 17, one day after I'll be back from Germany. If that works out, I could spend the weekend 18/19 for doing the final network/server/router/firewall/... configuration.

Obviously, all of this causes quite significant resource drainage for everyone involved. But it's a more than necessary step forward to building an environment that we can actually work in. An environment where our developers have real Internet access, can join IRC channels, and can get in touch with the OpenMoko community without the obstacle of strange corporate policies. An environment where we can have a 'clean start', even in the most literal meaning of the word :)

So all in all, bear with us, have patience. The revolution might take significantly longer than anticipated. But we're still busy doing whatever it takes to get us to the product that the OpenMoko core team set out to build.

[ /linux/openmoko | permanent link ]

Tue, 17 Jul 2007
Sick but not insane (yet)

Some people were troubled by my last posting about 'insanity'. And I have to admit it is justified. It isn't nice to pull internal issues of some company out into the public that way. But I just couldn't do differently anymore. FIC should think about providing a free corporate psychiatrist for us. ;-) I think I have a fairly big tolerance when it comes to mishaps, incompetence and cluelessness, but some of the things we're experiencing here are just not bearable. Especially not if they lack any kind of rational explanation.

So my latest outburst was mainly related to the fact that despite being with flu and fever in bed, I had to personally hack that embarrassing little online shop we now have at https://direct.openmoko.com/ [my first perl CGI code in about ten years!!]. Can you believe it, we just did not have (and still don't have) anyone in this project who has ever done some real work on CGI's or 'technical web things' at all. And if this was not enough yet, the requirements kept constantly changing all the time, up to this day, more than one week after the webshop opened.

So for gods sake, how many months have we been knowing that at some point we need to ship? And then you have plenty of people who produce over many months three entirely different concepts on how to shop and distribute those devices. And in the end, you have something with a Chinese-only credit card processing page, no proper setup with regard to the carrier (UPS in our case), tons of people talking vapor, but from what I've heard drawing nice diagrams. And no working solution.

Some people might wonder why those shipping rates in the shop are so high. Shouldn't a large company like FIC get huge discounts? Yes, they do. But believe it or not, FIC does up to this day not know in advance how much a shipping costs. Only after it was made. UPS has something like a Rates and Service Selection API. UPS has also a nice and detailed manual on how this XML API reports those rebated 'negotiated rates' in addition to the standard rates.

But then till this very day, UPS keeps claiming that such an API does not exist. Despite the fact that off-the-shelf e-commerce applications _support_ this API, and despite the fact that UPS' very own API documentation goes into every detail describing how that information is XML-encoded. What comes to my mind is the O'Really: Distributing Clue to users in a slightly modified version: Distributing clue to UPS' own sales representatives. How on earth can you get or keep a job there if you haven't even read the company's own documentation on their products?

Then you have companies like WorldPay, who very broadly advertise the many different payment variants they accept, not only all the credit cards known to man, but even things like direct debit (Lastschrift) in Germany. But do you believe they are able to process American Express for us? No, obviously not. Customers located in Taiwan (like FIC/OpenMoko) cannot process AmEx. No explanation given. So much for the word WORLD in WorldPay.

Or would you believe that some large bank like Citibank (Taiwan) was able to charge credit cards (VISA/MasterCard) in USD? No, obviously, being in Taiwan they can only charge in NT$ (New Taiwan Dollars). WTF? It's the 21st century, and there are dozens of online credit card acceptance partners that allow you to bill in about any currency you want. But not a world-class bank like Citibank.

And now you might think that we're actually no longer working on development. That is wrong. Yet another funny story from the MokoUniverse is that one of the worlds largest semiconductor distributors just had a long meeting yesterday, with FAE's and tons of FIC hardware engineers to crack their heads about the question whether or not we can use a "new" silicon revision of a certain component, and where exactly the differences between the old and new revision might be. I refused to participate in such a meeting, indicating that I just want a document describing those differences. In writing. Soon after that, I find out that we have always been using that supposedly-new revision, for the better part of one year. Nobody actually bothered to look at a unit of hardware, or at all the high-res photographs of GTA01 that I posted on people.openmoko.org ages ago.

Then lets get to another of my favorites. Purchasing. I wrote an extensive list of networking gear that we desperately need in order to crate the internal IT environment for our fast growing new OpenMoko company. After many delays, today I finally got our two core switches. And what did they do? They "forgot" to order the transceivers, even though they were explicitly listed in the requirements/order list. God knows what happens in the heads of such people. So I'm now back to once again just mail-ordering things from Germany and then billing FIC for the expenses.

And to not bring up more embarrassing facts, and to at least preserve some level of confidentiality, I'm now going to stop. But believe me, there are _much_ more insane stories to be told. With some luck, at some distant point in the future, I'll get permission to publish them in my memoirs.

But that's basically the kind of thing that drives me close to insanity. Whether for that cause, or completely unrelated: My health is actually quite bad recently. Maybe the lack of regular sleeping hours, let aside regular intake of food and the tons of stress have contributed to the flu that lead to a severe fever attack today. You know there's something wrong if at 33 centigrade in the sun, you're shivering more than when riding your motorbike at -13 centigrade in German winter.

So I'm going to take it slowly for the time being, working much from what has now become my 2nd home (apartment in Taipei, provided gratuitously by FIC for people like Werner and myself, who are spending so much time over here).

My sincere apologies in advance. But given my multitude of roles/hats and functions in the OpenMoko project, I bet my health issues will now contribute to some further delays in about any area of the project. But there's little I can do. One day, we will get there. Let's hope you're still interested in our products at that point.

[ /linux/openmoko | permanent link ]

Wed, 11 Jul 2007

If you want a status update on OpenMoko: I'm at a point where I won't go to the office again because I know the agression inside myself will turn into physically hurting someone there.

[ /linux/openmoko | permanent link ]

Thu, 28 Jun 2007
An update from the OpenMoko world

As Sean has now made his latest OpenMoko Announcement last night, this is a good point in time for me to write some more bits. Up to this announcement it was hard for me to publicly state anything, since the whole internal restructuring process has been underway, but not yet publicly announced.

I can only join Sean in his assessment about the superb support of FIC's senior management. They are providing us with the kind of resources we wanted.

But being the realist (well, Sean as optimist would call me pessimist, you know that old story), I also see severe challenges both right now, and ahead.

Whatever you might think: I bet that none of you has not the slightest idea about how many problems the OpenMoko team is fighting all day long. If you're thinking "what's the problem with a purely technical task, i.e. designing hardware and software for an Open phone"? Then my reply would be: It's not that big of a technical problem.

However, the really big issues start as soon as you leave the R&D world. On the one hand, there is the actual hardware production. Many components have incredible lead times (3 months or more), and our yet sort-of-unknown-but-initially-very-low quantities are not particularly helpful either. Any .tw OEM/ODM thinks in different terminology. The kind of production processes, shipping infrastructure, ... is just not meant for low-volume and direct shipment. We obviously knew this from the beginning, but everyone just happily works in their usual mode of operation, ignoring our concerns for many months.

Then think about the various customs / legal / trade issues. If you ship components from Taiwan to mainland china and use them to manufacture a product there, you need a special import license in order to get those products back into Taiwan. This license costs money (and, most of all: Time).

Or another example is the lack of double-tax agreements between Taiwan and the rest of the world. So payments to all our various external consultants all over Europe are taxed twice: Once in Taiwan, a second time in the respective country of residence.

For the last two weeks I have been working on finalizing the floor plan and infrastructure planning for the new FIC Mobile Communications and OpenMoko software groups offices in Taipei. And believe it or not, it is a very long and time consuming fight to ever get what you actually want. We know exactly what kind of servers, switches and routers we want. We know to which height we want to reduce the cubicles. We know what kind of Internet uplink we want. Still, it's close to impossible to get anything done. People will just outright refuse to do what they are asked (and paid!) to do.

Take our new servers as one minor example. You would assume that it is no problem at all to configure high-end servers around here. When doing this in Germany, I usually consult one of the many mailorder stores, go through their extensive list of mainboards and other components, select products based on their availability, price and features, and within 24hours I have everything delivered to my doorstep. 99% of those components are from Taiwanese companies.

Now enter Taiwan. First of all, you will discover that the concept of mailorder or extensive online product lists doesn't exist. "Taiwanese people don't trust e-commerce", is what they tell me. Secondly, you can't just call those places and ask them if they have a certain product, since apparently they would always say yes, only to get you into their store.

If you actually get into the various stores, you will see that almost all of the products you want are not available locally. "Not sold into the Taiwan market" is something that you hear very often. So e.g. the choice of Socket 478 mainboards from ASUS goes down from 52 (German online store) to something like 15-20.

So in the end we were really unable to find anything remotely decent (good performance, chipsets with excellent free software support) locally and I ended up importing Asus and Tyan mainboards from Germany into Taiwan, while buying the other components in Taiwan.

Now I could continue and name dozens of examples like this. If this project was just about _developing_ hardware and software, I would be a happy man, and we could look ahead to complete one device after the other. But it's all the other issues, administrative, political, cultural, sales, finance, accounting, shipping, ... which make people like Sean and me run at something like 20-25% of their usual efficiency, despite putting in at least 180% of regular working hours. And there is nobody who can help this, because nobody non-technical really understands what we're doing here, and why we need to do it different than whatever they might have done it before.

[ /linux/openmoko | permanent link ]

Fri, 15 Jun 2007
Visiting FIC's factory for GTA01 mass production in Suzhou

Yesterday I was on a ten-hour trip from Taipei to Suzhou in mainland China. It took about ten hours for something like a 300km line-of-sight distance, since the Taiwan/China political dispute over The Three Links doesn't allow any direct flights. So we had to go from Taipei to Hong-Kong, transfer onto a different flight to Shanghai, and then start for a couple of hours car ride to Suzhou.

The trip was quite impressive, especially since I have seen a lot more of Shanghai then during my summer 2006 trip to the FIC Shanghai branch. In fact, the sight of so many [strangely-looking] dense, high-rise apartment buildings with 25, 30 or more floors in the suburbs reminded me quite a lot of the classic 1927 Metropolis movie. This impression was probably further enhanced by the thick clouds of smog covering all of the sky, resulting in even non-grey objects look grey, giving the impression of a more-or-less gray-scale world...

In any case, let's not deliberate more about my general thoughts about China, Shanghai or Suzhou at this point, but get to the actual work: Today I spent at the FIC Suzhou factory, mainly doing final QA/Testing of the first 300 Neo1973 GTA01 phones. While I was doing this, another 192 phones went through assembly, resulting in a total of 492 units available at this time. We have another 500 units pending throughout the next two weeks.

The overall quality of my QA checks was quite good. The factory is doing a good job, and we could not detect any production-introduced bugs. The tools provided to the factory for programming/testing of the hardware leave quite a bit of room for optimization though. Will have to start this optimization process next week, after my return to Taipei.

This also means that we can finally make another announcement about the overall project status very shortly. And it means that as soon as the web-shop is up and running, developers will finally be able to purchase Neo1973 GTA01. 8 months too late. Sorry for that. Too much politics and too little actual technical work. All fixed now. Bright future ahead :)

[ /linux/openmoko | permanent link ]

Sat, 02 Jun 2007
Everyone is busy at OpenMoko

A number of people keep asking me what's going on with regard to openomko. I've even virtually stopped to update this blog quite some time ago. As much as I regret the lack of updates: Be assure they're just a sign of how busy everyone involved is.

I have, in fact, even cancelled my already-accepted paper and corresponding presentation at OLS this year :( I'm also not speaking at any other 'traditiona' event this year, not at Linuxtag, Linux-Kongress, CLUC, LTC, 0sec. Sorry, guys, maybe next year again.

I can't publicly state what's going on with regard to OpenMoko internally, but let me assure you: Good things are happening. We're working on a lot of internal changes that should enable us to approach the project with way more bandwidth.

The first couple of hundred GTA01Bv4 phnes have been produced by the FIC's mass production factory in mainland china. I'll personally do QA on 10% of those phones throughout the second week of June. We want to make sure we don't have any mishaps with our first customers, do we?

The first generation GTA02 prototypes have also showed up at FIC in Taiwan. More news on that at some later point :)

[ /linux/openmoko | permanent link ]

Sun, 01 Apr 2007
Introducing patchwork to the openmoko mailinglists

Some time ago, Jeremy Kerr wrote patchwork, a tool to semi-automatically keep track of patches submitted to mailinglists.

So far, it was mainly being used for the linuxppc and netfilter project with mixed results. I guess in both projects, in the end, nobody raelly maintained the patch status and stuff just ended up to get stale.

Now I've started an patchwork installation for OpenMoko, at least for the most common public mailinglists. So why do I think patchwork will be used more productively and receive better maintenance? Well, firstly, the number of patches on those lists is still quite small. Secondly, the number of people looking into reviewing those patches, within their primary paid-for working time, is relatively large.

I really believe that patchwork can provide an enormous benefit to a project. Let's hope we manage to use it correctly :)

[ /linux/openmoko | permanent link ]

Sun, 25 Mar 2007
Returning from FIC HQ / Taipei

Just returning from one of the probably busiest weeks in my life. The entire week was spent with meeting lots of FIC staff. Finally I'm able to connect faces to the members of the hardware and production testing team located in Taipei.

Significant time was spent talking to vendors of WiFi chipsets over the last week. The choice seems to have boiled down to designs around Atheros AR6K or Marvell 8686. The AR6K driver code is completely public for quite some time, even though it (and the mvista SDIO code it depends on) might need a bit of cleanup. From Marvell we yet have to find/receive the GPL licensed driver source code - at least from our [high level official] marvel contacts we have received positive confirmation. So actually, for now, only AR6K is a 'known possible' choice, whereas Marvell might become a choice, once we see the source :)

The other big task was sitting down with Werner Almesberger and doing the system level design for the next major hardware revision, and discussing this design with the hardware team. A lot of things are still in flux. But at least the potential of it is _really_ promising. Details are to be revealed at their appropriate time.

I've also had the chance to briefly meet with senior management such as the head of the mobile communications business unit, as well as the CEO and the chairman of the board of FIC. Everybody seems to be really excited and looking forward to the time ahead, now that we have identified many of the problems in the hardware design, production quality and internal processes and are heading to a much brighter future.

Another interesting opportunity was to present at Taiwan NTU University on the 'correct' way of doing Linux development, both technically and from the GPL / policy point of view. Let's hope we now have a couple of more software developers who will know these things before entering the industry :)

In retrospective, I should have done this trip way earlier on. But then, many things have changed over the last nine months, and it might have been an entirely different experience.

Taiwan/Taipei really seems to be an incredibly interesting place. A world of never-ending possibilities in the field of computer hardware. An industry that can produce about any device of your dream - but doesn't since it seems to be locked too much into either the master/slave role of traditional OEM/ODM business - or into producing the same things like all of their competitors, without really trying too much new/exceptional things.

So it seems, after all, my good intentions about travelling less in 2007 seem to be vaporizing sooner than I would have liked. I guess the productivity gain of me being in Taipei at least from time to time is worth it...

[ /linux/openmoko | permanent link ]

Sun, 18 Mar 2007
One-week trip to Taipeh, visiting FIC.

On rather short notice I am now on my way to Taipeh, Taiwan. As the interested reader of this journal will have noticed, OpenMoko and the Neo1973 are the seemingly endless story of delays and mishaps throughout the last nine months.

So the purpose of this trip is to discuss various options on how to make sure that we all learn the neccessary lessons out of this past experience, and ensure that OpenMoko eventually can at least keep up to some of its schedule promises.

So this will be a week full of meetings and discussions mostly with various managers and particularly the hardware team at FIC, our generous sponsor.

What is at least equally exciting are the planned meetings with various WiFi chipset vendors to discuss our requirements of a mobile-phone compatible low-size, low-power WiFi chipset with a fully GPL licensed Linux kernel driver.

The downside of all this is that I'm once again held back from writing some of the most important infrastructure code that OpenMoko still needs. Anyway, this compromise has to be made - after all, of what use is software without a high-quality, reliable, performant and available handset hardware :)

Since Werner doesn't have a blog, old-fashioned as he is, let me mention that he'll be there for the whole week, too. I think it's a 34-hour trip for him to get from Buenos Aires to Taipeh. I wonder how well he'll survive that...

[ /linux/openmoko | permanent link ]