Most of the challenges relate to the fact that I am an independent open source developer without a lot of resources. Almost every major feature that has been implemented in VirtualGL and my other open source projects since 2009 has been implemented because a company or other organization needed that feature badly enough to pay for my labor to implement it. The value proposition for potential project sponsors is that, by using VirtualGL, they are already reaping the benefit of decades of labor. It would probably cost US$millions to create such a project from scratch, but organizations can instead pay US$hundreds or US$thousands to fund the development of a specific feature or fix they need. Still, though, I constantly have to remind people that, although the software is provided free of charge, it isn’t free to produce. I have to hustle just to make enough money to pay my bills. Depending on the year, I earn about 1/4 to 1/3 of what a software engineer with my experience and skill set would normally earn in the U.S. market. I am willing to do that because I believe strongly in what VirtualGL and my other open source projects are accomplishing, but those projects are always on the ragged edge. Without “general sponsors”— companies and other organizations that provide a fixed amount of funding every year that can be used for project maintenance, spinning releases, build system enhancements, investigating and fixing issues reported by the community, answering questions from the community, and other necessary labor that is unlikely to attract outside funding— the projects wouldn’t survive. However, our level of general funding is only about half of what the projects need in order to be healthy and sustainable. I almost always have to eat significant labor cost— sometimes hundreds of hours’ worth— in order to finish a major release. If I cease being able to pay my bills through open source development, then I would have no choice but to pursue a different career, which would force me to abandon full-time work on VirtualGL and my other open source projects. In 2018, that almost happened. When major releases from all three projects landed in the same year, I went into debt. As an independent developer, I don’t get paid for vacation or sick days or bathroom breaks or lunch breaks. In the U.S., self-employed people have to pay both the employer’s and employee’s share of Medicare and Social Security, and individual health insurance is much more expensive than the group health insurance that corporate employees can typically get. Thus, I take home much less money than a corporate employee with the same gross income would. (And, again, corporate employees in my line of work typically have a much higher gross income to begin with.) I haven’t been able to meaningfully contribute to my retirement account since 2009, and in fact, I had to start borrowing against the interest on it a few years ago in order to pay my bills. I mention all of this not to complain. There are many other people in the world who have it much worse than I do. I mention all of this primarily to underscore the fact that these open source projects, which are making or saving millions of dollars for multiple companies worldwide, are not on stable financial footing. Stabilizing the project revenue streams would make it easier to speculate on new feature development, and it would increase my efficiency, since I wouldn’t have to spend as much time hustling.
Independent open source development is always a delicate balancing act. You have to give away a certain amount of free milk but not so much that people won’t buy the cow. Bug fixes are always free-- covered by General Fund money or, when that money inevitably runs out halfway into the fiscal year, by pro bono labor. That is just good PR— making sure that the software is always as stable and intuitive and performant as it can be. Doing so attracts new users, and the more new users there are, the more likely it is that some of them work for companies that can fund the development of VirtualGL. This patronage-based business model has deep trade-offs, though. The upside is that no one organization controls the agenda of my open source projects, meaning that they are not beholden to any particular variety of CPU or GPU or operating system. Any feature that makes sense can probably happen as long as someone is willing to pay for it. The downside, however, is that lack of funding frequently makes it difficult to work on the features that the projects need the most. Sometimes I have to speculate, to use pro bono labor to implement a feature or finish a release in order to attract new funded development. Long story short: the biggest challenge is money.
A related challenge is lack of access to the latest hardware. Basically all of the equipment I use to develop and test VirtualGL was donated or reimbursed by corporate sponsors. I was able to purchase a new Linux workstation using part of a grant that libjpeg-turbo was awarded in 2019, but prior to that, my newest Linux machine was nearly 10 years old. I test most operating systems using VMWare, which works reasonably well for most purposes, but sometimes, I’ll encounter issues that can’t be reproduced on my equipment (usually because the issues involve interaction between a specific O/S and specific hardware.) I have one recent AMD GPU and one old (2012 vintage) nVidia GPU at my disposal. I have no access to commercial 3D applications for testing, so if an interaction issue between VirtualGL and such an application can’t be reproduced in any other way, I’m usually stuck unless someone can provide access to the application (which isn’t always possible.) I also have no access to the official Khronos GLX conformance tests, because they charge tens of thousands of dollars for that. Thus, I basically had to write my own.
The technical challenges around developing VirtualGL mostly have to do with “stupid application tricks” and “stupid driver tricks.” It’s less the case now than it used to be, but historically, commercial 3D applications did some really “unique” things. Thus, most of the complexity in VirtualGL stems from workarounds for such applications. In terms of drivers, there are unfortunately some vendor-specific GLX implementations that are just plain wrong. Even more unfortunately, one of them is from a major GPU vendor, who has yet to fix the conformance issues despite me providing test cases and supporting documentation for all of them. Thus, I had to devote significant labor (mostly pro bono) toward working around some of those issues in VirtualGL. I wasn’t able to work around all of them, though, so I continue to have to field tech support requests from users of those GPUs, explaining why VirtualGL can’t currently work properly with them.
Again, I mention all of this not to complain but rather to paint the picture that, despite being a very lean operation with low overhead and relatively modest needs in terms of money and equipment, it’s still a struggle to get what I need-- even when what I need is just a little bit of tech support from a particular vendor.
Even though VirtualGL is open source, it is too specialized to allow for much collaborative development. (Translation: it’s way too easy to break things if you don’t know what you’re doing.) Depending on the scope, it may take me dozens of hours to properly regression test, benchmark, review, clean up, and document a submitted patch that implements a major new feature. If there is no funding to cover that labor, then I have to really ponder whether the feature is strategically important enough to integrate using pro bono labor. Since I have nearly 20 years of experience with this stuff, paying me to implement a new feature is almost always less costly than paying someone else to implement it and then paying me to integrate it. I do, however, welcome patches that fix minor issues. I also welcome bug reports. Bugs are fixed for free (within reason), and every squashed bug improves the reputation and reach of VirtualGL, which potentially attracts more funded development.
I have 200 hours of funding per year from Santos, which is used for maintaining both VirtualGL and TurboVNC, but VirtualGL would be much healthier if it had 200 hours of its own. If your organization has benefited significantly from VirtualGL and is in a position to provide that general funding, please contact me.
Another way that organizations can contribute is by funding the development of specific features. You can find a list of outstanding features (most of which I consider strategically important) that are in need of funding here. And I am, of course, open to any other ideas that you might have. Funded development helps the project in two ways. In the near term, it helps pay my bills, and in the long term, the new feature helps attract new users, some of whom might be in a position to fund further development.
Organizations that fund $1000 or more of labor can opt to have their logo displayed on the VirtualGL Sponsors page.
Apart from that, you can find a link for individual donations on the landing page of VirtualGL.org. Every penny I receive goes directly toward funding the labor needed to maintain VirtualGL.
Also, the easiest way that most people can contribute is simply to test VirtualGL with their favorite applications and let me know if any issues are discovered.
I generally prefer users to file an issue in the GitHub issue tracker or to post on one of the mailing lists. This gives us a public, searchable record of the issue or question and its resolution, so others in the community can potentially benefit from it. Feel free to use either forum for bug reports or just general questions about VirtualGL. I am happy to answer them.
) Very interesting reading.