Lenz Grimmer's blog

As I mentioned some time ago, Software Freedom Day 08 will take place on Saturday, 20th of September 2008.

Coincidentally, the a large number of Sun/MySQL Engineers and other Sun folks will be in Riga, Latvia for an internal developer meeting around this day. To make use of this opportunity, we plan to give a number of sessions and presentations (in english) about various topics and to contribute to this global celebration of Open Source Software.

We've set up a Team Page on the Software Freedom Day web site for this event - the venue will be the Cafeteria Conference room in the basement of the University of Latvia, Riga, which can accomodate 60-80 people:

Raiņa bulvāris 19
Rīga, LV-1586

There is no entrance fee and you don't have to register - just come by and meet with us! There will be free coffee, refreshments and cake during the breaks.

In the evening, Sun will host a social event (incl. free drinks and food) in the SAS Radisson Daugava hotel, starting at 19:30:

Radisson SAS Daugava Hotel
Kugu 24, Rīga, LV-1048, Latvia
Tel.:+371 6706 1147; Fax: +371 6706 1101

We've set up a tentative schedule (45 minutes per session plus 15 minutes of Q&A), please check the Wiki for eventual last-minute changes!

11:00-12:00: MySQL/Open Source in Latvia (Evijs Taube, Sun Microsystems)
12:00-13:00: Open Source Business Models: how to build a business around free software (Speaker TBD)
13:00-13:30: Lunch Break / Ask the Guru your tech questions
13:30-14:30: MySQL in the Enterprise: Customer references, commercial offerings (Rob Young/Robin Schumacher, Sun Microsystems)
15:00-16:00: MySQL Community Overview: How to engage and contribute (Giuseppe Maxia/Jay Pipes/Lenz Grimmer, Sun Microsystems)
16:15-17:15: MySQL Performance tuning best practices (Jay Pipes, Sun Microsystems)
17:15-18:15: Maintaining your Open source project with Bazaar and Launchpad (Lenz Grimmer/Giuseppe Maxia, Sun Microsystems)
19.30: Social event: Software demonstration, buffet and free beer in the SAS Radisson Daugava hotel

We'd like to thank Leo Trukšāns, Michael Dexter and Georg Richter for their help and support in getting this event arranged and organized! I look forward to being there and help to spread the word about the stuff that keeps me occupied for more than 13 years now

The MySQL 5.1 Use Case Competition is in full swing - we've already received a number of cool and interesting submissions, which we will turn into articles that will be published on the MySQL Developer Zone over the course of the coming weeks. Today we received a note from Jakub Vrána from the Prague, Czech republic. He's the author of phpMinAdmin, a MySQL management tool written in PHP. Here's what he wrote:

In the beginning of September 2008, I have implemented MySQL 5.1 Events to the database management tool phpMinAdmin. I've used the Windows version of MySQL 5.1.26 for the development.

As phpMinAdmin tries to be full-featured, yet compact management tool, I have implemented Events to allow users of MySQL 5.1 manage it.

The implementation was quite straightforward, it took only 4.5 kB of PHP code. Events management is well described in MySQL documentation and easy to understand.

During the development, I've reported three bugs: Bug#39163, Bug#39165, Bug#39173. I have been positively surprised by the speed of reaction to these reports

Thank you very much for your submission and the support, Jakub! We appreciate it.

If you're reading this and are using MySQL 5.1 and any of it's new features: have you considered telling your story yet? You may even win something when doing so!

Just to remind you that Packt Publishing is running their Open Source CMS Award again:

The Packt Open Source Content Management System Award is designed to encourage, support, recognize and reward Open Source Content Management Systems (CMS) that have been selected by a panel of judges and visitors to www.PacktPub.com. Now entering its third year, the Award has established itself as an important measure for quality and the popularity of Open Source Content Management Systems.

You have two more weeks to submit your favourite CMS in the following categories:

• Overall Winner
• Most Promising Open Source CMS
• Best Open Source PHP CMS
• Best Other Open Source CMS

As for the last two years, I'll be a member of the team of judges that have to choose from the finalists that received the most nominations during the nomination stage.

I look forward to the list of finalists - it's always interesting to find out about new developments in this area and how the established projects in this market have developed over the course of the year!

I recently added two new packages to my repository on the openSUSE Build Service:

• Maatkit is a collection of essential command-line utilities for MySQL. Each is completely stand-alone, without dependencies other than core Perl and the DBI drivers needed to connect to MySQL, and doesn't need to be "installed" - you can just execute the scripts. This makes the tools easy to use on systems where you can't install anything extra, such as customer sites or ISPs.
• protobuf - Protocol Buffers - Google's data interchange format. Protocol Buffers are a way of encoding structured data in an efficient yet extensible format. Google uses Protocol Buffers for almost all of its internal RPC protocols and file formats.

The protobuf package is required, if you want to compile drizzle. Packages are available for openSUSE, Fedora and Mandriva Linux. Feedback is welcome!

In a recent article, Matt Asay was musing about the aspects of hosting an Open Source project by yourself vs. using a public project hosting service like SourceForge, GitHub or Launchpad. He concluded that it's important for commercial/sponsored open source projects in particular to do the hosting by themselves, so they can maintain full control and can gain more insight, which hopefully will turn into more revenue at some point.

However, Matt seems to reduce "hosting" to "providing downloads" only:

Control and visibility. Given the importance of customer conversions, it becomes hugely valuable information to know that it takes, say, eight months on average for someone to buy the "Enterprise" version of your code after downloading the software. With Sourceforge et al., you have no way of connecting the dots between download and purchase. But if you host your downloads, you can suddenly link a download to a purchase using marketing automation software like Loopfuse.
[...]
It can tell you many things, but the key is to be able to glean insight from the earliest stage of your interaction with a potential customer, and that means you have to host your own downloads. Otherwise, you have no idea how or when a would-be customer downloads your code, which makes the "why" they download it less interesting, because it becomes less actionable.

I understand and agree to Matt's point in principle - you want to know more about the users that download and use your stuff. Here are some related thoughts about this topic.

Project hosting is not just about downloads

First: project hosting is much more than just providing a download/mirror infrastructure for your product releases. On the one hand, you have the regular users of your product who are primarily interested in having easy and fast access to the latest builds for their platform of choice and a platform to exchange their problems and experiences with other users.

But project hosting facilities also address a completely different audience, with different needs. These are the developers, who want to have easy access to the latest source code, be able to submit bug reports and patches and want a direct communication path to the project's developers.

I think it is important to ensure that you serve both the developer community as well as the user community as best as you can, which could of course mean you should provide the full range of project hosting all by yourself. But by doing so, you also create an island that makes it difficult to benefit from the "cross-pollination effects" between your project and others. This can partially be remedied if you don't only set up a project hosting infrastructure for your own purposes, but also open it for projects related to your project (and which not maintained by your own team), e.g. how SugarForge is doing it. But the cost and effort involved in setting up and maintaining such an infrastructure should not be underestimated.

There is more to distribute than releases

At MySQL, we just recently moved away the MySQL Server source trees from the proprietary BitKeeper revision control system to Bazaar. Along with this migration, we also relocated the public repositories from mysql.bkbits.net to Launchpad.net, to make it easier for external developers to access and work with the code. Currently, MySQL only makes use of the source repository hosting capabilities - downloads, bug reports and most other things like mailing lists or forums are all maintained by ourselves and hosted on mysql.com.

Due to the distributed nature of Bazaar, we could of course also provide the source repos from our own servers (similar to how we do it for several of our projects that are still maintained in Subversion). But I think it makes a lot of sense to use Launchpad for that, as it allows a tighter integration and collaboration with contributors and other related projects, and it gives us more visibility within the developer community.

Drizzle has taken this even further: the project utilizes all of Launchpad's facilities, including Blueprints, Bug reporting, mailing lists. It's going to be an interesting learning experience to see how this affects and improves community interaction/participation. I'd love to see MySQL move more into this direction as well (especially the bug database and worklog would be good candidates), but this probably will take some more time.

I too recently moved the source tree of my own personal project from a Subversion repository on my private server to Launchpad. Several reasons motivated me to do this, one of them being the opportunity to gain more practical experience with Bazaar and getting away from a central source code repository that makes me the bottleneck in making changes and applying patches. A distributed revision control system makes much more sense from a community contribution point of view, which Ian Clatworthy summarizes quite well in his paper "Distributed Version Control Systems - Why and How". In a way I deliberately give away some of the control over my project. And I must say I like how Launchpad integrates the various available subsystems like blueprints, code branches and bug reports - things are much better connected and they provide useful workflows that make the entire system much more productive to use than e.g. SourceForge.

I still provide downloads of released versions from my own site (as does MySQL), but mostly because I actually did not know until recently that Launchpad offered this kind of service - I will look into that for the next release. I am more interested in making sure that my users have easy access to properly packaged versions of my project for their operating system of choice. Therefore I work closely with the packagers from various distributions and make sure they integrate new releases quickly. In addition to that, I make use of hosted services like the OpenSUSE Build Service, which automatically provides package repositories for a number of platforms. I aim for wide distribution on as many channels as possible, instead of trying to be the sole provider of my product. This brings me to another point:

Downloads stats are overrated

Direct downloads from your project's web site usually are only one part of the distribution system. I believe that being included in the various Linux or other Open Source Operating System Distributions (e.g. Free/OpenBSD, OpenSolaris, etc.) plays a much bigger role in gaining popularity and reaching more users. Most users usually go with what they get as part of the package, as the distributor usually has taken care of a tight integration and proper packaging of your project within his own product and also takes care of providing updates and fixes.

Unfortunately it's almost impossible to gather any detailed intelligence about the number of users of a project this way, as distributions usually don't keep track of (or don't disclose) their download figures and which packages on their releases are the most popular. Debian's Popularity Contest is probably the only exception to this, but it's unclear how reliable that information is. Here I must agree with Matt again, if we just look at project hosting services acting as download providers only and include distributions in this equation:

As open source becomes more commercial, someone is going to need to step up to offer such visibility into these hosted services, or we're going to find the hosted services proving useful for ever decreasing amounts of time.

I guess we all would love to know more about the users that don't download a package from our site, but go with the one provided by their distribution of choice instead or download it from somewhere else. But so far, this is a blank spot on our radar screen.

 Another caveat that results from these multiple distribution channels: just looking at your own download stats may actually give you a skewed picture of your user base, particularly if you look at the platforms (which will probably be dominated by Windows or Mac OS X, as these OSes usually don't ship your code as part of their own product).

So instead of trying to force downloads through a single instance only, I think it's much more important to ensure widespread distribution and a top-notch first hand experience. If users like your product, they are much more inclined to consider coming back and purchasing something from you than if you annoy them by making your product hard to download and install or require them to register before they can obtain a copy of your product. It's all about lowering the barriers as much as you can, even if you have to give up some control in exchange.

JFYI: today I migrated the mylvmbackup source tree from my local Subversion repository on http://www.lenzg.org/ to a Bazaar repository on Launchpad.net.

This will hopefully make it easier for contributors to work on the code and share their modifications with others, removing me as the bottleneck for applying and testing patches for new releases. I chose Bazaar primarily because I wanted to get some more hands-on practice with it, now that the MySQL Server source trees have been transferred to it as well (see Kaj's announcement for details).

As mylvmbackup is closely related to the MySQL Server project, it made sense to choose the same platform and enjoy the cross-pollination effects and the infrastructure that Launchpad provides. Additionally, the distributed nature of Bazaar makes it more convenient to work with the code history and commiting changes locally without having to be online and connected to the SVN server.

I am sure that other DSCMSs like Git, Mercurial or darcs would have done the job equally well - nowadays it's very hard to choose

The "trunk" branch is now hosted on Launchpad. I assume that I will soon open up a development branch, that will receive heavier modifications first. I also plan to use the site for bug tracking and keeping track of feature requests (via Blueprints).

To create a local branch of the "trunk" repository, you can use the following command:

bzr branch lp:mylvmbackup

I also maintain a copy of that branch on my home server, just in case: http://www.lenzg.org/bzr/mylvmbackup/

To avoid confusion, I removed the Subversion repository on http://www.lenzg.org. Please use the Bazaar tree on Launchpad from now on. Thanks!

Even though we had been preparing the migration to Bazaar for a while now, today's announcement kind of caught me by surprise. But I am very happy about this move!

While BitKeeper is an excellent tool and served us well the past eight (!) years, I was quite annoyed when BitMover decided to remove the fully functional free BitKeeper client, which effectively put our development back into a Cathedral: even though our source trees remained accessible via bkbits.net, the crippled bk client was only capable of cloning and pulling new revisions from there - it was not possible for an external developer to commit changes locally or to create patches. This turned out to become a severe roadblock for users that wanted to participate in the development of the server. While some of our other internal projects (like the documentation, connectors and GUI tools) moved to Subversion, this was not that easily doable for the MySQL Server source trees. Lots of internal processes and tools had been tightly coupled with the source code management and there never seemed to be the right time to make the switch, which potentially could have disrupted the ongoing development work.

By moving to a truly open, distributed revision control system, it will become much easier for external contributors to work on patches and maintain modifications or enhancements outside of the main MySQL Server source trees. In fact, work has already begun! When I look at the various repositories hosted on Launchpad.net, I already notice several source trees that are not maintained by MySQL Engineers, e.g. Marc Callaghan's Google patches, Percona's work on microsecond resolution or a branch of MySQL 5.0 that is maintained and used by the folks from Wikimedia. I too helped publishing a source tree today - Holyfoot's work on improving the GIS functionality is now also available from there! In addition to the MySQL Server, many other MySQL-related projects are hosted on LaunchPad. And by utilizing services like the fabolous OpenSUSE Build Service, it's actually quite easy to provide readily installable packages of these as well!

I hope this change will significantly lower the barriers for external contributors and make it easier for our own developers to merge and accept changes that have been created by others and received sufficient community testing. This is much more convenient than having to apply a patch that was posted a long time ago and may have already been abandoned by the contributor or suffered from bit-rot. Being able to maintain a patch within a live tree that can be kept in sync with the main repository is one of the great features of distributed revision control (and which I think is one of the technical reasons why the Linux Kernel development model works so well).

Kudos to everybody involved in making this happen!

If you are working with MySQL as a DBA or developer, I'd like to encourage you to consider taking the MySQL Magazine Survey, which was compiled by Keith Murphy and Mark Schoonover.

The survey takes around 10-15 minutes to complete and runs until June 16th. The results will be published in the summer issue of MySQL Magazine, due on July 15th. The questions cover a broad range of topics, from details about your MySQL experience and job description over connectors and languages to operating systems and MySQL versions.

Thanks in advantage for your support and input! The results of this survey will be interesting for us as well.

IBM plans a residency to write a Redbook about the IBM DB2 for i5/OS as a Database Engine for MySQL:

Seize this opportunity to be among the first, elite IBM System i professionals worldwide to gain hands-on and practical experience with the MySQL on i5/OS. Working alongside and exchanging knowledge with some of the top database experts in the System i, you will develop a competitive edge that will serve you well in successful service delivery to System i clients. MySQL is used extensively with opensource PHP applications. As our customers deploy PHP web applications, many of them are looking to run MySQL as well.

The residency will take place in Rochester, MN from April 21st to May 16th. This book is going to be a nice followup to the first Redbook titled "Discovering MySQL on IBM i5/OS", which was finished in December last year. If you are curious to learn how to install MySQL on this platform (or would like to prepare yourself for participating in this upcoming residency), you can get a PDF from there, too!

I initially wanted to attend this residency myself, as I have been involved in writing a Redbook before and really enjoyed the experience. However, recent changes forced me to retract my offer to volunteer. IBM covers the travel expenses and will take care of accomodation for the time of the residency. If you are interested and think you have the time and skills and you would like to expand your knowledge about MySQL on the i5/OS Platform, please contact me at firstname at MySQL dot com!

As I felt the itch to do some quick hacking yesterday, I decided to provide an RPM spec file for the MySQL proxy. The changes have been commited to the SVN trunk now and I added some hints to the INSTALL file on how to perform an RPM build.

Here is a quick summary of how to convert the current SVN code into an installable RPM. You build environment needs to fulfill a few additional prerequisites (a gcc compiler and the C library header files are taken for granted here), I added the versions I used on my openSUSE 10.2 system for reference:

• autoconf 2.56 or newer (autoconf-2.60)
• automake 1.9 or newer (automake-1.9.6)
• MySQL 5.0.x include files (MySQL-devel-5.0.41)
• pkg-config (pkgconfig-0.20)
• libevent 1.x or newer (libevent-1.1)
• lua 5.1 or newer (lua-5.1.1)
• glib2 2.4.0 or newer (glib2-2.12.4)

# Check out the trunk of the SVN repository
$ svn co http://svn.mysql.com/svnpublic/mysql-proxy/trunk mysql-proxy

# Enter the directory and bootsrap autoconf/automake
$ cd mysql-proxy
$ sh ./autogen.sh

# Build the source distribution tarball
$ ./configure && make distcheck

# Convert the source tarball into a binary RPM
$ rpmbuild -tb --clean mysql-proxy-<version>.tar.gz

RPM will now perform the compiling and packaging. The end result will be an installable binary RPM package which will end up in the RPMS directory. Depending on your distribution this location will vary, on my openSUSE system the package ended up in in the following location from where I installed it directly:

$ sudo rpm -Uhv /usr/src/packages/RPMS/i586/mysql-proxy-0.5.1-0.i586.rpm

The mysql-proxy will be installed into /usr/sbin, the documentation and example scripts are being put into the default documentation directory (/usr/share/doc/packages/mysql-proxy in my case).

For now, you still have to start the proxy manually. Next thing is to create an init script that starts up the process at boot time.

For both worlds, the concept of collaboration, participation and giving more power to their users is a key component. OSS projects need contributors for patches and bug reports, but also for feedback, translations, artwork, advocacy in order to be popular and healthy. The project's developers need to be open for suggestions, listening to their user base on where the project should be heading. They also usually strive for open standards and APIs, allowing easy combination and interaction with other Open Source Software so that other projects can build their work on top of theirs or compliment it. A successful OSS application gains popularity by the snowball effect: users of the product find it useful and recommend it to other users seeking for similar functionality. These too start to contribute to the project in various ways, making it in turn more useful and attractive for even more users and so on.

The same is true for Web 2.0 sites: they depend on a user base that creates and provides content that makes their site useful and popular, e.g. blog postings, pictures, articles and other content, but also by providing feedback and bug reports about the site itself and recommending it to others. A successful Web2.0 site needs to be as open to user feedback as a regular OSS project in order to remain attractive and maintain a loyal user base. So it's a lot about participation and empowering the users to contribute and control the content - the part that actually makes a site useful and attractive. Many of these Web2.0 sites also provide public APIs (e.g. SOAP, REST, XML-RPC or RSS, just to name a few acronyms) that allow others to access their services and functionality and reuse or combine them with their own work. And by using technologies like AJAX, most of these sites actually look and feel like regular desktop applications already.

But there is a strikingly difference: even while most Web2.0 sites are based on Open Source technology (e.g. the LAMP stack or Ruby on Rails), only very few ones actually carry on the OSS philosophy and make their own code available as Open Source as well. Not that they would have to do so, as there is this so-called ASP loophole in the GPL and most other OSS licenses: web sites are not actually distributing their application code, so they are not required to disclose the source of their applications.

But why do they break the chain here? One could argue that the obvious reason for that is that most web sites are created and maintained by companies, that are doing this for business reasons and want to maintain their competitive edge over others. However, many sites started as a hobby project by an individual or a small group and were converted into commercial entitities once they became successful. Even in these early stages it's rare that sites provide their source code. But isn't Web2.0 mostly about the users and content and not so much about the technology behind it? Would it really make a difference if the source code of a site is freely available?

In the OSS world, users could just fork off a project, if they don't feel the project is going the right direction or it has been abandoned by the original author. This sometimes leads to much better applications than what the original project came up with. This also leads to a healthy competition and to much better software with less duplication of work in the long run. And this is being recognized by more and more software companies as well - there is a large and growing number of OSS applications that are not maintained by a group of volunteers, but by a commercial entity.

In the Web2.0 world, people start creating new sites from scratch again and again, resulting in a vast amount of sites that serve a similar purpose, all with their own warts and deficiencies, splitting up the user base and available content. This causes fragmentation and makes it much harder to create one useful resource. And they are all on their own in maintaining and improving their sites, to fulfill the requests of their user base and keeping them happy and loyal.

I wonder if and when Web2.0 developers will embrace and extend the concept of sharing their code as well, to let others help them to become even more powerful and popular. Once a site has gained a certain popularity, there is plenty of new features that their users will ask for, or bugs and deficiencies that are in need of fixing. Making the source code available would allow others to jump in here, opening yet another channel of possible contributions and community-building. The Open Source community provides numerous lessons to learn from. True, others may then be able to take the code and create a site that provides the same services and experience. But they still would have to gain traction by creating a community and content before they would become a serious competition.

One great example of how this could be done is Wikipedia - they maintain Mediawiki (the application that powers their site) as a regular OSS project, which has made it one of the most popular and powerful Wiki applications around and has allowed others to create Wiki sites about all kinds of topics on their own.

And while it's not really true Web2.0 company, I also welcome Linden Lab's decision to open up the Second Life client source code. I am confident that the experiences they make with this will encourage them to consider opening the sources of the server code as well at some point...

But they both have grasped that it's not so much about the technology and software, but much more about the users and the content created by the community that makes them popular and successful. This is something one can't simply take and create a competing offering, even though the source code is available.