libjpeg-turbo | About / General Sponsorship, libjpeg-turbo 1.5, and the Meaning of Life

Some of you have already heard me beat the patronage drum many times with regards to this project, but I thought it necessary to coalesce all of my thoughts on this topic into a single article, in hopes of explaining why a General Fund for libjpeg-turbo is going to be critical to our project's continued health. I hope that posting this may solicit help from organizations who are able to contribute toward such a fund.

libjpeg-turbo is a unique project. Whereas most open source projects (OSP's) are funded by a company or a foundation, libjpeg-turbo is run by a single developer working as an independent contractor. I am not actually selling a product but rather providing a product for free and making money through funded development opportunities. The reason why I do things this way is because it allows me to produce a high-quality product that is as platform-agnostic as possible. For instance, if O/S Company A were paying me to work on this, then it is unlikely that many of the fixes/features necessary to integrate libjpeg-turbo into O/S B would be possible. If Chip Company A were paying me to work on this, then it is unlikely that SIMD extensions for Chips B or C would be possible. If another company was paying me to work on this, then it's possible that they would want to sell libjpeg-turbo as an SDK and prevent me from distributing high-quality binary packages through the project. Back when I worked for Sun Microsystems (primarily on my other pet projects, VirtualGL and TurboVNC, which Sun was productizing), I spent an inordinate amount of time solving Solaris/SPARC-specific problems. While Sun did move the VirtualGL and TurboVNC projects forward, I've been able to move them forward at a much much faster rate as a solo operative. The patronage model under which I work may be unique to software development, but it's not unusual in other fields-- automobile racing, for instance, or music. Car companies fund racing teams because the work of the racing teams ultimately leads to better car technology.

I am a believer in the mantra that "quality happens only when someone takes responsibility for it" (refer to https://queue.acm.org/detail.cfm?id=2349257), but quality takes time, and time is money. I don't believe in either the Cathedral or the Bazaar. A cathedral is too rigid, and a bazaar is not rigid enough. The best open source projects tend to be those that have their equivalent of Linus, and while I would never presume to compare myself to him, at least within the limited scope of this project, my role is similar. If I were financially secure, then I'd be more than happy to donate a lot of my free time to libjpeg-turbo. I believe in what this project can do, but I've unfortunately become a victim of its success. The more ubiquitous it becomes, the more free work I have to do to move it forward. For instance, every time a large OSP (such as Chrome or Mozilla or Debian, etc.) discovers a bug, it usually requires many hours of my unpaid labor to help diagnose the bug, to discuss possible solutions, to review patches contributed by those developers (or sometimes to craft a patch on my own), to perform regression tests on multiple CPUs and O/S's to ensure that the solution doesn't affect performance, etc. And that's just for a bug fix. It often takes as much or more work for me to integrate a contributed feature patch as it took for the contributor to write it. For instance, my development work to clean up, integrate, and test the features that will be necessary to make libjpeg-turbo the default JPEG library in Android (RGB 565, partial image decoding, etc.) cost me the equivalent of two months of my already meager income. There is no immediate way for me to monetize that work, since Android support is of no immediate benefit to VirtualGL or TurboVNC.

In addition to Android support, here are some other examples of unpaid projects that have been necessary to move libjpeg-turbo forward in recent years and that have required significant pro bono labor on my part:

Researching the usefulness of the DCT scaling and SmartScale features in libjpeg v8, in the context of evaluating them for possible inclusion in libjpeg-turbo, and writing a lengthy article discussing the findings
Researching the issues surrounding libjpeg v9 API/ABI support and writing a lengthy article explaining why I chose to stop treating libjpeg as an upstream project
Responding to legal challenges from the current Independent JPEG Group maintainer
Attempting to diagnose and fix bugs in the MIPS DSPr2 SIMD implementation that were discovered long after the feature was contributed (the original developers could not be contacted)
Upgrading the official builds to a newer version of autotools, to fix a variety of build issues (this work will land in libjpeg-turbo 1.5)
Extensive performance testing with newer versions of GCC, and based on those findings, upgrading the official builds to use GCC 5.1.x (this work will land in libjpeg-turbo 1.5)
Extensive diagnosis and performance testing, in an attempt to work around issues in the clang optimizer that cause libjpeg-turbo to perform 20% more slowly when built with that compiler instead of GCC
Maintaining current benchmarking information to guard against performance regressions
Refactoring the libjpeg-turbo code base to remove obsolete features and formatting conventions inherited from libjpeg
Developing new TurboJPEG functions that allow for encoding from/decoding to separate YUV planes
12-bit JPEG support
Migrating the code to GitHub

I hate asking for money, but the reality is that libjpeg-turbo cannot move forward without a General Fund. A general fund will pay for integrating contributions from the community, build system upgrades, providing free support to the community through the mailing lists, working on bug fixes, and forward-looking research that may not immediately (or ever) pan out. None of that stuff is sexy enough for anyone to want to sponsor it on a project-specific basis, but all of it is vital to the overall health of libjpeg-turbo.

I certainly appreciate it when people contribute feature patches to libjpeg-turbo, but those patches are never really free. Even the most high-quality of patches still takes a lot of time to integrate. There are just too many platforms and compilers to test, too many things to do to make the code conform to the existing (20-year-old) coding standards, too many regression tests to run to make sure everything is stable and performant (performance regression testing, in and of itself, can often take the better part of a day's work.) Plus, when I adopt a feature, that means that I am taking the responsibility to fix any bugs that might be discovered with that feature down the road. Even if the development of the feature has been paid for, it is often the case that I have to donate labor to fix it if (when) bugs are discovered months or years later.

The VirtualGL Project (which produces VirtualGL and TurboVNC) has a pool of 200 "general" hours per year that are paid for by a large company with hundreds of seats of those products. The company is saving $1 million/year by using those products, so throwing a tiny fraction of that my way to keep the projects healthy makes a lot of financial sense for them. I am hoping that there are companies reading this right now who have similarly benefited from libjpeg-turbo and would be willing to similarly invest in this project to keep it moving forward. As another example-- mozjpeg (which built upon and continues to build upon my work in libjpeg-turbo) received a large grant from Facebook that is fueling development of their project.

Being an independent developer means that I don't draw a salary for my work on libjpeg-turbo. No one is paying for my health insurance or office space or sick days or retirement contributions or other things that are provided to most full-time open source developers by their corporate employers. Most feature contributions I receive to the project are from these full-time developers. From their point of view, they have already contributed labor to develop the patch, so why should they contribute money for its integration? But from my point of view, if it takes me 20 hours to clean up and integrate a patch, that's 20 hours during which I'm not making money.

I of course want people to help with the project, but I don't think it's boastful to say that no one knows this code as well as I do. Therefore, basically any contribution is going to have to be thoroughly reviewed and tested (and often rewritten) before it can be integrated into libjpeg-turbo without compromising our existing quality standards. If someone contributes a feature for one platform, for instance, it is rare that they will have tested the feature on other platforms. If contributed code isn't very project-friendly, then my choices usually boil down to: (a) don't accept it, (b) spend many many hours of my unpaid labor to integrate it in a manner that maintains the quality of the overall product, or (c) ask for money to pay for that integration labor. I have already done (b) quite a bit in recent years, and it's literally about to make me go broke. Thus, I am now forced to do either (a) or (c).

I work kind of like a research institution, which means that I live on grants, and my research is steered in the direction of whoever is paying those grants. libjpeg-turbo was originally the product of a grant from Cendio AB to develop an open source accelerated JPEG codec for TigerVNC. To put this another way, if I didn't work the way I work, then this project would not have ever existed. Red Hat, who was a member of the TigerVNC Project at the time, decided to integrate libjpeg-turbo into Fedora, and that started the ball rolling for getting it into other O/S distributions. Almost all of the major features that moved the project forward in the first few years of its existence were paid for by a company or other organization, but in the last couple of years, the trend has been more in the direction of organizations contributing patches instead of money. Unfortunately, this often puts me in the awkward position of having to donate my own development labor for free in order to adopt the contributed features. Grants are what move this project forward, and when I do donate pro bono labor, I need it to be an investment that has a good chance of producing more grants.

I wrote this essay in part because it's germane to the topic of a future libjpeg-turbo 1.5 release. In order to put out a new release, there have to be enough new features to entice people to upgrade. If people see the 1.5.x feature list and yawn, then they're probably just going to keep using 1.4.x, and the project stagnates. There are basically two big features now in the 1.5 branch (git main):

AltiVec SIMD support: this was paid for by IBM, but because I was locked into a flat rate for the project and I ran into unforeseen difficulties, I ended up having to eat about $4000 of cost. I don't blame them for that. I made a bad business decision, but the reality is that that feature has very little prospect of generating any paid follow-on work for me. From a developer's point of view, I'm happy that libjpeg-turbo is now accelerated on a whole new class of platforms, but from a business point of view, the project created a financial hardship.
Partial JPEG decoding: this was initiated by a patch submission from Google, because this feature is needed by the higher-level Android APIs. However, despite the relatively high quality of the patch, it still took about $2000 of my unpaid labor to integrate it. In addition to the initial integration/refactoring of the patch, testing, and the diagnosis and fixing of bugs, I also had to design a regression test suite that fully covers all of the code paths in the jpeg_skip_scanlines() function with the least possible number of tests (very tedious work.) This test suite additionally had to be ported to work with 12-bit samples and the Windows build system, and building/testing on a variety of platforms revealed some other issues that needed to be fixed. The jpeg_skip_scanlines() function is a great addition to the libjpeg API, but I think it really needs to be extended into the TurboJPEG API as well, both for completeness and also to entice people to upgrade to 1.5. Also, adding support for this feature to the TurboJPEG API would allow me to benchmark it. I know for a fact that there are TurboJPEG API users who have requested a partial decode feature. However, it will probably take an additional 25-30+ hours to extend the feature up into the TurboJPEG C and Java APIs, document the new methods, test performance, etc.

I had intended for the main "selling feature" of libjpeg-turbo 1.5 to be AVX2 SIMD support (for faster performance on newer Intel and AMD processors.) A company stepped forward to sponsor the purchase of development hardware for this (for which I am very grateful), and I spent some time with this new hardware laying the groundwork for the feature. Through that research, however, I discovered that it's unfortunately not going to be as straightforward of a project as I would have hoped. Right now, it looks like it would take many many hours to implement AVX2 support the right way, and so far, no one seems able or willing to invest in that project without any guarantee of a major speedup.

Hence the need for a General Fund. I really think the above features are necessary in order to evolve libjpeg-turbo 1.5 sufficiently relative to 1.4.x to justify releasing it. If I have to do all of the above work pro bono, then it is unclear when the new release can land. The recent unpaid feature work I've done has broken the bank, and I am simply not going to be able to work on any major new features in libjpeg-turbo unless they are funded, either through a general fund or through project-specific contracts.