The first peak of load on openmoko.org servers

For the last seven hours I've been trying to organize the openmoko.org server setup into providing more efficiency / performance. It was really amazing. We were not on slashdot, not on any of the major news sites, but we were already having something like 40MBps aggregate outbound traffic peaks on our servers.

The two major performance bottlenecks were ViewCVS and Mediawiki. Quickly I installed memcached on one of our more idle boxes, and put two squid instances on two separate machines in front of the mediawiki, which then seemed to do mostly fine.

The ViewCVS apparently cannot be helped at all. What I found on the web is that it's apparently just very inefficient code, and there's little one can do without rewriting the code. I don't know whether that's still the current situation, but when the next peak comes, I'll probably just disable ViewCVS to save some CPU cycles.

In case you're interested: Our setup is currently running on four physical boxes, running a total of ten OpenVZinstances. One of the machines is dedicated to the GForge installation on projects.openmoko.org, the other one a (at least intended to be) dedicated buildhost where we do our OE builds.

While this obviously has been quite enough for the last half year, we now have different performance requirements. For Phase-0 this installation is probably still quite sufficient, since this first-couple-of-days peak is bound to cease at some point.

However, When Phase-1 starts (public availability of phones to developers by means of direct order), we will definitely need a more sophisticated infrastructure for our downloads.openmoko.org site, from where we will make available the full source code that our OE builds need, plus the full source and object code of every binary release we make. The idea is to have a round-robin DNS setup of geographically distributed machines.

So as of now, we're soliciting mirrors with large disk capacity. If you want to help us by providing a mirror (expected capacity requirement for 2007: something like 300GB) and bandwidth, please contact me at laforge@openmoko.org. We're particularly interested in the US and Asia regions.

Apart from that, a couple of secondary DNS servers would also help improving our availability. If you already have a bind installation somewhere and want to become a secondary DNS for openmoko.{org,com,net}, please contact me, too.

Finally, the good news is: We're down to 2..4MBps for now. Until the news appears on major news sites, I guess.

After having discovered that mailman doesn't use templates for its listinfo_overview page (the one you get when calling /mailman/listinfo without a list name), I quickly hard-coded our openmoko header into the python code. If somebody feels motivated to add proper templating support to all mailman pages (such as the 'options password prompt' and the before-mentioned listinfo_overview), you could help us out even with something like that.

Now I'll finally turn back to actual moko code. We have frame buffer support in u-boot since two days ago (splash screens are important for marketing), I'll now look into getting the CPU clock to 266MHz (we've had some issues with this) and finally, after all, TS07.10 multiplex and gsmd infrastructure, for which I consistently haven't found time until now.