I don't generally have a very high opinion of humans (myself included). I think we're all heavily inclined towards being short sighted, irrational, selfish and easily bored (especially once our basic needs are met). It may be tempting to dismiss that as a projection of my own issues onto others (since I certainly consider it an apt description of myself), but there's plenty of objective evidence to back up my perspective (I highly recommend Danny Kahneman's "Thinking, Fast & Slow" on this topic).
However, despite these flaws, we've figured out a way to build institutions that are more farsighted, more rational, more altruistic and less easily distracted than we are as individuals.
These institutions come in many flavours, including local, state and federal governments, corporations, NGOs, professional bodies, but also (and increasingly) in loosely governed self-forming communities.
The US Constitution was actually a pretty well designed document, and it owes a lot to one fundamental premise: humans cannot be trusted with unlimited power, and this goes double (or more than double) for humans with an active interest in seeking power.
Thus the principle of the separation of powers of born, with the intent being for the judiciary, legislature and executive to act as checks on each other, and the general population to act as checks on them all.
It turns out that this system can go horribly wrong once unregulated mass market propaganda enters the mix (and thus government becomes about funding your re-election propaganda rather than serving your constituents), but that just serves to confirm the original premise - if humans are given the opportunity to achieve power without responsibility and accountability, many of us will take it.
However, while it's relevant as background, the balance of power in the US government isn't what prompted the creation of this thoughtstream. Rather, it was this sad-but-all-too-familiar tale of a corporation descending into the mire of internecine internal bureaucratic warfare.
Microsoft is both a competitor and a partner for my employer (Red Hat). We sell competing systems, but for the sake of our respective customers, we still try to ensure our tools work well together. I've been in the kind of bureaucratic hell described in that article in the past and it's not a particularly pleasant experience (although you can put up with a surprising amount of it if you're well paid and are getting to work on interesting problems with talented people most of the time - my personal limit turned out to be more than a decade).
On the flipside, we have this piece from Valve's resident economist, musing on the nature of corporations as one of the least market-driven elements of nominally market-driven societies.
The discipline of the market is a truly wonderful thing (in cases where we can figure out how to design the market appropriately), yet much corporate activity seems expressly designed to bypass that discipline (especially acquisitions and mergers to grow into monolithic autocratic mega-corporations).
One of the reasons I love working for Red Hat these days is that this is a company that is genuinely trying to be different. It doesn't always succeed, and trying to balance the competing values of Freedom, Accountability, Courage and Commitment can make for some rather emotive exchanges (there's a reason our CEO mentions memo-list as an example of one of the things that makes Red Hat tick).
But even though our adherence to those four principles is imperfect (we're still only human), it remains a fact that Red Hat follows a policy of "upstream first" for new product development (even for previously proprietary products acquired through acquisitions), and aims to only diverge from upstream versions for sustaining engineering purposes (like backporting backwards compatible fixes to early versions).
Even some of our major tools (like Beaker, the product I am development lead for) are open source rather than proprietary, simply because we think that collaborating in the open is a better way to build software, even if all the developers on that project currently work for the same company.
Defaulting to open is about creating opportunities for serendipity - assuming nobody else would be interested in our software, and hence keeping it a secret would be a self-fulfilling prophecy. Assuming competitors would be interested and being overly restrictive on what we release could also be counterproductive (and I think we're actually undersharing in several respects at the moment - I've opened up some aspects more since taking over as dev lead, but we still have a lot more to do before Beaker is a truly open project that clearly explains its potential value to users and provides good avenues for them to progress to becoming contributors).
And that finally brings me to the point that prompted this whole ramble. A recent Wired profile of Monty Taylor, one of the OpenStack developers, made the point that the developers on key open source projects often have multiple companies interested in employing them, and that this is a well-recognised fact.
What this does is help to finally create a true engineering centre of power, distinct from the traditional corporate power centres of finance and people management. When upstream is heavily volunteer driven, with all activity conducted in public, you simply can't get away with the traditional proprietary BS where the features that are implemented get dictated in an autocratic fashion from on high.
Instead, changes almost always have to be justified on the basis of their value to the project itself or to actual users of that project. "We have to implement this feature because Bob from Sales told this customer we could already handle it, and we really want their business" just doesn't fly - the feature has to be valuable in its own right, or it won't happen.
The right to fork is another critical element in making this work - even the power of core developers on a project is limited, as if they do anything too ridiculous, then they may find the community is unwilling to follow them.
Of course, this varies wildly across different projects, but it's applicable to many of the larger ones, including the ones that are upstream of Red Hat's products. It may seem bizarre to refugees from traditional proprietary companies, but "we can't dictate that, it's a decision that needs to be made by the community" is a legitimate reason for not doing something, even for projects like Fedora.
We may champion and promote a change, but actually making a change must go through the regular community channels, even if it was originally proposed internally at Red Hat.
Beaker is more autocratic than many Red Hat open source projects (since we're upstream of an internal testing system rather than of a product we ship to customers), something I refer to as being open-source-but-not-open-governance.
I'd love for Beaker's capabilities to grow to a point where it makes sense for people without a direct tie to Red Hat to be using it and to start complaining about the relatively closed governance model. With any luck, I'll be able to start further down that road at Flock later this year, but that's highly dependent on getting travel funded.
Yay, my Flock talk (and travel funding request) got approved - I'll get the opportunity to pitch Beaker as a project upstream Fedora QA should be interested in, rather than it being solely of interest to the product QA teams.
Dave Neary wrote a good post for community.redhat.com on The Value of Open Source (the formatting of the comments is a bit hard to read, but I added a few thoughts of my own anyway).
I think this perspective is also the basis for much of my frustration with the business models of GitHub and Atlassian: while these two companies use open source heavily, and contribute greatly to the community (in the form of providing free-as-in-beer software and services, as well as through conference sponsorships, organising events and releasing ancillary code as truly free software), they're not taking the steps I see as necessary to preserve a strong engineering culture as the companies grow.
An engineer that disagrees with the strategic direction of GitHub or Atlassian can't walk away from them and start a competitor. That's great for the bottom line of those companies, but it lays the foundation for them becoming the next Microsoft or Oracle, rather than the next Red Hat.
On the other hand, it's easier to find a business model that doesn't rely on source code secrecy when you're a platform provider like Red Hat. Red Hat Enterprise Linux, JBoss, OpenShift Online/Enterprise - these are complex technology platforms, where people are willing to pay handsomely for Red Hat's quality assurance, sustaining engineering, security threat monitoring, software delivery mechanisms, certification programs (for software, hardware and staff), supports services, consulting services, access to our customer community forums, etc.
By contrast, when your main product is a relatively straightforward web application that could be deployed on top of a PaaS, then the idea of potential customers "rolling their own" using your open source version is a much more significant strategic threat.
This article considers the problem of "free loaders" for an open source project: those that just download and use the software, and never contribute even a bug report or feature request, let alone helping other users, offering patches or (gasp) becoming a paying customer of the associated product vendor.
The success of Red Hat Enterprise Linux has given Red Hat the luxury of taking more risks than smaller companies with our approach to open source. OpenShift Online ran in beta for two years before we flipped the switch on offering a paid tier (without making any changes to the free tier).
Even today, the "roll your own!" community offering (OpenShift Origin) is given equal prominence on the home page with the two commercial offerings (Red Hat/AWS hosted and self-hosted).
Once the service actually goes live, it will be interesting to see if Pivotal One (aka VMWare/EMC) are as up front with their users about the availability of CloudFoundry as Red Hat is about OpenShift Origin.
But what do you do when you're a relatively small company, and there's a big strategic risk that open sourcing your core product would gut your ability to survive long enough to establish your credibility as an organisation worth investing in to ensure the availability of future updates?
I've realised there's a model that works at that scale which I love, and it's the one being pursued by Continuum Analytics for Numba vs NumbaPro: the "delayed open source" model.
What this model does is help establish the credibility of the company as a centre of engineering excellence in their own right (and thus worth investing in, either directly or as a customer), while still providing the risk management benefits for employees and customers: if senior management at the company lose the plot and start taking actions that are customer or employee hostile, then any business risk only applies to the last 6 or 12 months of features and employees that decide to strike out on their own are starting from a point that isn't too far behind.
Scaling up a company without losing the culture that originally made it great is incredibly hard. If you're below 150 employees, it's still possible to function using instinctive tribal mechanisms: Dunbar's Number is a big deal when it comes to effective organisational design.
Once you get beyond that, though, you need a plan to avoid descending into the toxic nature of the typical modern enterprise, which is the last bastion of feudal authority. We rejected feudalism as a model for running our societies, why should we accept it as a suitable model for running a business?
Red Hat's "default to open" ethos is one such strategy. It turns out, that beyond a certain point, sharing information within a company is actually harder than just sharing it with the world at large. So Red Hat pairs up its products with a community project, and we have community projects for many of our internal tools as well.
On the community side, it doesn't matter that much what your official job title is - it's all about ability and the investment of time, and that applies even if you have "@redhat.com" in your email address.
On the product side, we try to provide a far more streamlined experience, as we aim to provide customers with a clearer picture of what's going on in the more free-wheeling upstream communities, and which parts are actually relevant to them, and which parts they don't need to worry about (at least for now).
Information sharing is a hard problem in general. Limited information sharing is actually even harder than sharing with the general public.
At a recent presentation in Brisbane, Pia Waugh noted that for Australian government departments, getting permission to share information online is only a little bit more work than getting permission to share information with one other department. By doing that little bit of extra work to make the information available to the Australian public, they also make it available to every other government department.
That's a powerful benefit, and something the enterprise world would do well to pay attention to.
One of the triggers for the latest batch of updates here was this article about HubSpot. This was the one that got me think about Dunbar's Number again, and the associated break point where you have to shift from tribal governance models to more scalable structures.
A couple more interesting and somewhat related articles:
The Antifragile Organisation (via Robyn Bergeron)
Glyph on programmer ethics (I'd seen this before, but was recently reminded of it by Gary Bernhardt - the significance of this becomes ever greater as we outsource more and more activities to online service providers)
I've also been reflecting further on the reasons why I care so much about the business models of GitHub and Atlassian in particular, but am comparatively unconcerned about the business models of other proprietary services I use (like Gmail, Feedly, Xero, ShareSight, Dropbox, or ThoughtStreams itself).
I think it mostly has to do with the role both companies now play in introducing many people to the idea of open collaboration (through GitHub and BitBucket respectively). No matter how much either company says they support open source, their actions say that they fundamentally believe the notion "You can't make money writing open source software".
There's no other way to rationalise their actions - GitHub have even stated explicitly they will never open source the core Rails application. If you'd love to see a cool new feature on GitHub, you can't send them a pull request for it - you have to hope that someone in the blessed inner circle with access to the source code will offer.
Atlassian have exactly the same problem with BitBucket - BB 2184 (adding CNAME support like that offered by GitHub Pages) likely would have been added long ago if they accepted external contributions. As it is, the lack of that feature is the only reason I have any repos on GitHub - the site publication repo for Curious Efficiency lives there.
The other aspect is that I think both companies do amazing work, and have done a lot of good in supporting the open source community, and I want to be an unabashed fan of their efforts.
However, I can't be an unreserved advocate for any company that reserves the right to screw over its customers under the law, and that's exactly what proprietary software licensing is designed to do: it relies on the full coercive power of the government as an incentive for customers to keep paying their license fees if they want to continue using the software they already have installed. See Atlassian's EULA for an example of this - that's one of the most customer friendly proprietary EULAs you're going to find on the planet, but it still deliberately enforces a "single provider" model in order to inflate the switching costs for migration to any alternate solution.
By contrast, a customer that decides a Red Hat subscription isn't providing them with value for money will usually be able to keep using that software under the open source licensing terms, even after they decide to stop paying us. The fact we can't rely on prohibitive switching costs keeping our customers locked in to our software is one hell of an incentive for us to ensure our customers are happy with the ongoing services we're offering (and it's a credit to our sales, support and other service divisions, as well as the Red Hat Customer Portal developers, that so many of our customers do renew their subscriptions, and even increase them). We've made it incredibly easy for them to walk away from us, and they stay anyway.
The real value of a company like Red Hat, or GitHub, or Atlassian isn't in the software, it's in the people. Talented groups of people focusing on understanding customers problems and collaborating effectively to solve them.
However, for a company to survive, it's not enough for the company to understand that, their customers have to understand it, too. We're all better off if we can invest in a few specialist organisations that guide the development of common tools that can then be used to the benefit of everyone.
Without careful attention to the business model, there's a high risk that the "Tragedy of the Commons" will kick in, and not enough organisations will contribute to ensure the centre remains strong.
Red Hat has successfully navigated these treacherous waters at the operating system layer by splitting out certification and support from the underlying software (other Linux vendors that haven't made that split don't appear to be faring quite so well). If you look up any of our public financial statements, you can see that model is also working well for us at the Java application server level, and I don't see any reason why it won't work for the IaaS and PaaS markets. They're all platforms where it's useful to have a vendor acting as a filter between the early adopting technology enthusiasts and the wider majority who just want something stable that works, so they can get on with their own business (whatever that may be).
If phrases like "fiduciary responsibility", "due diligence", "risk management" and "successor in interest" are part of your every day concerns, then a good relationship with a trusted vendor is important, because it's actually cheaper than employing the expertise you need to interact with upstream directly. (Keep in mind that even tech companies struggle to find enough good staff - at this point in history, there's nowhere near enough technologically savvy people available for every business to interact with upstream projects directly).
There are several groups trying to navigate the issues of open source development in the developer tools hosting market, too. Sourceforge lost a lot of its early dominance due to both reliability issues (as bigger projects decided it was worth the hassle of switching to self-hosting to gain more control) and being slow in offering new version control systems (initially Subversion, later Git and Mercurial).
They're still one of the bigger open source players in the hosting game (Allura is AGPL), but the revenue models of GitHub and Atlassian actually give them an advantage here, as their free-as-in-beer tier can be focused entirely on providing an awesome user experience, without needing to worry about making room for ads.
Zero cost, an excellent user experience and the network effects of GitHub and BitBucket are too tempting for even me to resist - while it annoys me that GitHub and Atlassian reserve the right to screw their users, that's very different from believing that they would actually do so. That's another reason I find the situation so frustrating - they're reserving the right to do something I don't think the current leadership would ever do, but I can't say the same for their successors in interest, and strategic missteps have led to changes in control at far larger companies than those two (compare the way Sun handled open source projects to the way those projects have been handled by Oracle post acquisition).
In addition to Sourceforge with Allura, other open source projects like Gitorious, Gitlab, etc, serve to keep the leading proprietary providers honest. As noted in the discussion here, if you avoid the integrated issue tracker and wiki, migrating away from GitHub or BitBucket would be an inconvenience rather than a disaster (and it wouldn't be that bad for most projects, even if they did lose their tracker data).
The user network effects of GitHub have been enough to lure even some of Red Hat's upstream projects (most notably, OpenShift Origin).
I ended up spending most of the evening working on this, instead of on my PyCon AU talk as I originally planned. When I notice a conflict between what I think I believe (previously free software needs free-as-in-freedom tools), and what I actually do (including using both GitHub's and Atlassian's free-as-in-beer proprietary services for my own hosting needs, backing the OpenShift team's decision to do the same for OpenShift Origin in order to lower the barrier to contribution for users, and acknowledging the use of various proprietary tools and services as a reasonable away to address various business requirements at Red Hat when the existing open source options are manifestly inadequate, and investing directly in improving them ourselves isn't a reasonable course of action, since we need the problems solved now, rather than at some point in the future when the open source version has caught up), it's hard for me to let the question go until I've figured out the source of the discrepancy.
A colleague made a comment to me recently along the lines of "The licensing fees for vendor product <x> are almost as much as it would cost us to hire a whole extra developer!". Hearing that kind of comment is why I think companies are right to be worried about people not understanding what "value for money" means in this context. Not only did a colleague make that comment to me, but it took me a couple of days to realise how precisely backwards it is.
Here's a different, and I think far more reasonable, way of looking at those licensing fees: "For less than the cost of even one developer, we get to benefit from the accumulated knowledge of a development team that spends their entire time thinking about these issues, whereas for us, it's merely a distraction from our main goal of helping the rest of Red Hat deliver high quality software to our subscribers".
Someone that pays for a Red Hat subscription is effectively outsourcing a lot of tasks that need to be done if you're going to use open source responsibly in a large organisation. By paying us, our customers get to benefit from it being done once, and well, rather than each of them having to duplicate that effort.
Paying an organisation like Atlassian or GitHub to think seriously about encouraging effective collaboration amongst software developers and others should be something large enterprises do as a matter of course. Yet I'm not confident either company could AGPL their core applications (as they exist today), without seeing a lot of potential revenue disappear down the drain.
Tim O'Reilly coined the wonderful phrase "Create more value than you capture", but there's an unspoken corollary to that: "Capture enough value to thrive".
I'd like to live in a world where companies like Jive, GitHub, Atlassian and Continuum Analytics could open source their core applications and collaborate openly, without it posing an existential threat to their business.
I'd like to think that software consumers were enlightened enough to realise that there's only so much that can be done with purely volunteer efforts. Seriously: there's lots of important stuff involved in doing software development really well that doesn't get done with only volunteers. That's why the MSDN docs for Windows APIs are so much better than what we can provide for Python on python.org, why the level of review, auditing and testing Red Hat can put into our products outstrips what upstream can muster, and why the overall road map for Python is so hazy it barely even exists (PEP 0 gives some guidance, particularly since I cleaned out a few entries from the "Open PEPs" section, but it's not a real road map).
But we don't live in that world (yet, anyway). Red Hat's a billion dollar company, and we have plenty of people that aren't willing to pay reasonable subscription fees to ensure tools we depend on continue to exist into the future.
And just as I decry laments about AGPL software as "friendly fire", I'm coming around to the point of view that I consider giving back to the community through means other than open collaboration on core applications to be a reasonable approach.
I'll continue to hope to one day be able to welcome the likes of GitHub, Atlassian, Jive or Continuum Analytics to the club of true open source vendors. I'd certainly like it if the first three considered open sourcing versions of their core applications from 6-12 months ago to help constrain the actions of their successors in interest (whoever they may be).
Ultimately though, my preference for open source licensing comes down to a combination of ideology and risk management (remember, unlike a truly open company like Gittip, Red Hat's motto is "default to open", not "everything is open").
It turns out that when my risk assessment for a proprietary dependency is "low" (either because I don't care all that much about the data or workflow involved, or because I understand the vendor's business model, and it seems both sustainable and unlikely to cause their interests to diverge substantially from mine), or when the functionality gap is sufficiently large, then I think it makes sense to just run with the proprietary vendor (at least in the short term).
So, for me personally, "Does it use an open development model?" is an important factor to be considered, but it's still open to being traded away in favour of other features, particularly when the market is already in a position to severely punish vendor misbehaviour. Open models are most important when the risks of data lock-in or workflow lock-in are high, and especially critical when there's an established pattern of a vendor taking advantage of a dominant market position to the detriment of their customers.
That's certainly not a universally shared opinion within Red Hat (there's still plenty of "go open or go home" folks, particularly amongst those that work almost entirely on upstream projects), but that kind of contention is a large part of what makes Red Hat work as an effective bridge between the collaborative open source community focused world view, and the software-is-just-a-necessary-evil perspective of many customers.
Heh, added Robyn's link above based on the headline, without reading the article first. That said, some aspects of it are still relevant to what I'm discussing here.
What I'm talking about here is the way organisations fail as they grow - becoming warring petty kingdoms that lose sight of the original goals of the overall organisation. The notes towards the end of the article about DevOps keeping developers focused on the end user experience are an interesting aspect of that.
Open development is another such tool, since it allows dedicated users more direct access to the development team, rather than filtering everything through multiple layers of analysts.
I should also mention DISQUS in the list of proprietary-but-gives-back-in-other-ways (like a great free tier) services I use
David Cramer's Sentry is an instance of a true open source company (using the SaaS model, where people pay for the hosted version). (Thought prompted by scanning the EuroPython schedule and seeing David's talk outline)
Loggly are another company that (like Sentry) focus on their hosted service, meaning they can open source logstash without needing to worry about monetizing deployments inside corporate firewalls.
New Relic are an open source based company where they use a proprietary model, but I'm not sure they need to. Bringing the data and the analytical tools together is a key part of what they do, and there's a clear and obvious benefit to paying someone to improve the analytical tools available for already collected data. Effective data analysis is one of those bottomless wells where there is no such thing as "done" - there are always going to be new questions to ask, and new code to be written to answer them.
New Relic's enterprise offering is also New Relic hosted. As with Loggly, the storage management side of what they do is painful enough that it makes sense to pay to outsource it.
If they offered an encrypted S3 data export on an open source software base (ala Loggly), that would be a spectacularly low risk service at a very reasonable price. As it is, you have to allow for the fact that if New Relic shuts down, or raises their prices, you have no exit strategy (short of the "build your own metrics infrastructure").
However, the big difference between New Relic and a company like Atlassian is that New Relic don't make any claims to being anything other than a classic proprietary vendor. By contrast, it's hard to reconcile proprietary licensing with Atlassian's stated values.
An AGPL version of BitBucket could be a very interesting prospect. Atlassian have stated explicitly that BitBucket will always be SaaS only, so subscribing there will always be about the hosting, rather than the software itself.
The Enterprise version is called Atlassian Stash, and that has a bunch of features BitBucket doesn't need, but could potentially be very useful in a corporate environment (like selective publication of repos outside the firewall).
David Cramer (@zeeg) has the slides for his EuroPython talk online.
Some good thoughts on running a SaaS hosted service as an open-source-first upstream.
See my A Question of Trust for more on this, including a discussion of RhodeCode.
RhodeCode is particularly interesting because it uses an approach proposed by Monty Widenius of MySQL AB fame: Business Source Licensing
In the RhodeCode case, the underlying functional software is all open source right now. However, several non-functional aspects of the front end (the "chrome" if you will) are not open source, but are instead under a business source license that means they will go open source automatically in a few years time. They also have a generous program to make the full version available to other open source projects and traditionally cash-strapped organisations.
This creates some really nice incentives:
traditional enterprise customers are strongly encouraged to treat it as a typical software purchase/subscription. Sure, in theory they could strip the not-yet-open-source chrome bits and replace them, but it isn't worth the hassle at the prices RhodeCode charge
other open source projects can use the full RhodeCode version free of charge
if RhodeCode ever do anything ridiculous that alienates their community, the right to fork remains available, as replacing the business source licensed chrome parts wouldn't be that difficult, and the worst case scenario is to just wait a few years until the currently business sourced parts become fully open source.
Coming back to this after another 18 months or so: I'm now of the view that Red Hat's own core operating system business model is actually of the "delayed open source" variety, since the definition of the core package set for RHEL & CentOS is entirely up to Red Hat, with the source code being delivered to CentOS at the same time as a new version of RHEL becomes generally available to subscribers. Thus, RHEL is to CentOS as Android is to AOSP. The difference is that Android doesn't have any equivalent to Fedora as an upstream integration project that creates a pool of public components, for Google's to filter and create the base Android platform.
Third party contributors getting a change accepted into Fedora isn't a guarantee of seeing that same change accepted into mainline RHEL (and hence CentOS), it just dramatically lowers the barriers to having that happen.