Planet JM

Category: supply chain

  • AI Native and the Open Source Supply Chain

    AI Native and the Open Source Supply Chain

    I recently wrote 2 essays on the subject of AI Native Automation over on the AINT blog. The gist of them is simple:

    It’s that latter point that I want to dive a bit deeper into here, but first a disclaimer:

    We have no idea what the ultimate impact of "AI" is to the world, but there are some profoundly negative ramifications that we can see today: misinformation, bigotry and bias at scale, deep fakes, rampant surveillance, obliteration of privacy, increasing carbon pollution, destruction of water reservoirs, etc. etc. It would be irresponsible not to mention this in any article about what we call today "AI". Please familiarize yourself with DAIR and it's founder, Dr. Timnit Gebru.

    When I wrote that open source ecosystems and InnerSource rules were about to become more important than ever, I meant that as a warning, not a celebration. If we want a positive outcome, we’ll have to make sure that our various code-writing agents and models subscribe to various agreed-upon rules of engagement. The good news is we now have over 25 years of practice for open source projects at scale that gives us the basis to police whatever is about come next. The bad news is that open source maintainers are already overwhelmed as it is, and they will need some serious help to address what is going to be an onslaught of “slop”. This means that 3rd party mediators will need to step up their game to help maintainers, which is a blessing and a curse. I’m glad that we have large organizations in the world to help with the non-coding aspects of legal protections, licensing, and project management. But I’m also wary of large multi-national tech companies wielding even more power over something as critical to the functioning of society as global software infrastructure.

    We already see stressors from the proliferation of code bots today: too many incoming contributions that are – to be frank – of dubious quality; new malware vectors such as “slopsquatting“; malicious data injections that turn bots into zombie bad actors; malicious bots that probe code repos for opportunities to slip in backdoors; etc – it’s an endless list, and we don’t yet even know the extent to which state-sponsored actors are going to use these new technologies to engage in malicious activity. It is a scary emerging world. On one hand, I look forward to seeing what AI Native automation can accomplish. But on the other, we don’t quite understand the game we’re now playing.

    Here are all the ways that we are ill prepared for the brave new world of AI Native:

    • Code repositories can be created, hosted, and forked by bots with no means to determine provenance
    • Artifact repositories can have new projects created by bots with software available for download before anyone knows no humans are in the loop
    • Even legitimate projects that use models are vulnerable to malicious data injections, with no reliable way to prove data origins
    • CVEs can now be created by bots, inundating projects with a multitude of false positives that can only be determined by time-consuming manual checks
    • Or, perhaps the CVE reports are legitimate, and now bots scanning for new ones can immediately find a way to exploit one (or many) of them and inject malware into an unsuspecting project

    The list goes on… I fear we’ve only scratched the surface of what lies ahead. The only way we can combat this is through the community engagement powers that we’ve built over the past 25-30 years. Some rules and behaviors will need to change, but communities have a remarkable ability to adapt, and that’s what is required. I can think of a few things that will limit the damage:

    • Public key architecture and key signing: public key signing has been around for a long time, but we still don’t have enough developers who are serious about it. We need to get very serious very quickly about the provenance of every actor in every engagement. Contributed patches can only come from someone with a verified key. Projects on package repositories can only be trusted if posted by a verified user via their public keys. Major repositories have started to do some of this, but they need to get much more aggressive about enforcing it. /me sideeyes GitHub and PyPi
    • Signed artifacts: similar to the above – every software artifact and package must have a verified signature to prove its provenance, else you should never ever use it. If implemented correctly, a verified package on pypi.org will have 2 ways to verify its authenticity: the key of the person posting it, and the signature of the artifact itself.
    • Recognize national borders: I know many folks in various open source communities don’t want to hear this, but the fact is that code that emanates from rogue states cannot be trusted. I don’t care if your best friend in Russia has been the most prolific member of your software project. You have no way of knowing if they have been compromised or blackmailed. Sorry, they cannot have write access. We can no longer ignore international politics when we “join us now and share the software”. You will not be free, hackers. I have to applaud the actions of The Linux Foundation and their legal chief, Michael Dolan. I believe this was true even before the age of AI slop, but the emergence of AI Native technologies makes it that much more critical.
    • Trust no one, Mulder: And finally, if you have a habit of pulling artifacts directly from the internet in real time for your super automated devops foo, stop that. Now. Like.. you should have already eliminated that practice, but now you really need to stop. If you don’t have a global policy for pushing all downloads through a centralized proxy repository – with the assumption that you’re checking every layer of your downloads – you are asking for trouble from the bot madness.
    • Community powered: It’s not all paranoid, bad stuff. Now is a great opportunity for tech companies, individual developers, enterprises, and software foundations to work out a community protocol that will limit the damage. All of these actors can sign on to a declaration of rules they will follow to limit the damage, quarantine known bad actors, and exchange vital information for the purpose of improving security for everyone. This is an opportunity for The Linux Foundation, Eclipse, and the Open Source Initiative to unite our communities and show some leadership.
    • Bots detecting bots: I was very hesitant to list this one, because I can feel the reactions from some people, but I do believe that we will need bots, agents, and models to help us with threat detection and mitigation.

    I have always believed in the power of communities to take positive actions for the greater good, and now is the perfect time to put that belief to the test. If we’re successful, we can actually enjoy revamped ecosystems that will be improved upon by our AI Native automation platforms. If successful, we will have safer ecosystems that can more easily detect malicious actors. We will also have successful communities that can add new tech capabilities faster than ever. In short, if we adapt appropriately, we can accelerate the innovations that open source communities have already excelled at. In a previous essay, I mentioned how the emergence of cloud computing was both a result of and an accelerant of open source software. The same is true of AI Native automation. It will inject more energy into open source ecosystems and take them places we didn’t know were possible. But what we must never forget is that not all these possibilities are good.

  • The Revenge of the Linux Distribution

    The Revenge of the Linux Distribution

    Some things appear in hindsight as blindingly obvious. And to some of us, perhaps they seemed obvious to the world even at the time. The observations of Copernicus and Galileo come to mind. To use a lesser example, let’s think back to the late 2000s and early 2010s when this new-fangled methodology called “devops” started to take shape. This was at a moment in time when “just-in-time” (JIT) was all the rage, and just-in-time continuous integration (CI) was following the same path as just-in-time inventory and manufacturing. And just like JIT inventory management had some weaknesses that were exposed later (supply chain shocks), so too were the weak points of JIT CI similarly exposed in recent security incidents. But it wasn’t always thus – let’s roll back the clock even further, shall we?

    Party like it’s 1999

    Back when Linux was first making headway towards “crossing the chasm” in the late 90s, Linux distributions were state of the art. After all, how else could anyone keep track of the all the system tools, core libraries, and language runtime dependencies without a curated set of software packaged up as part of a Linux distribution? Making all this software work together from scratch was quite difficult, so thank goodness for the fine folks at Red Hat, SuSE, Caldera, Debian, and Slackware for creating ready-made platforms that developers could rely on for consistency and reliability. They featured core packages by default that would enable anyone, so long as they had hardware and bandwidth, to run their own application development shop and then deliver those custom apps on the very same operating systems, in one consistent dev-run-test-deploy workflow. They were almost too good – so good, in fact, that developers and sysadmins (ahem, sorry… “devops engineers”) started to take them for granted. The heyday of the linux distribution was probably 2006, when Ubuntu Linux, which was based on Debian, became a global phenomenon, reaching millions of users. But then a funny thing happened… with advances in software automation, the venerable Linux distribution started to feel aged, an artifact from a bygone time when packaging, development, and deployment were all manual processes, handled by hand-crafted scripts created with love by systems curmudgeons who rarely saw the light of day.

    The Age of DevOps

    With advances made in systems automation, the question was asked, reaching a crescendo in the early to mid-2010’s, “why do we need Linux distributions, if I can pull any language runtime dependency I need at a moment’s notice from a set of freely available repositories of artifacts pre-built for my operating system and chip architecture? Honestly, it was a compelling question, although it did lead to iconic graphics like this one from XKCD:

    For a while it was so easy. Sure, give me a stripped down platform to start with, but then get the operating system out of the way, and let me design the application development and deployment layers. After all, any competent developer can assemble the list of dependencies they will need in their application. Why do I need Red Hat to curate it for me? Especially when their versions are so out of date? The rise of Docker and the race to strip down containers was a perfect example of this ethos.

    A few incidents demonstrated the early limitations of this methodology, but for the most part the trend continued apace, and has remained to this day. But now it feels like something has changed. It feels like curation is suddenly back in vogue. Because of the risks from typo-squatting, social engineering hacks, and other means of exploiting gaps in supply chain security, I think we’ve reached somewhat of a sea change. In a world where the “zero trust” buzzword has taken firm hold, it’s no longer en vogue to simply trust that the dependencies you download from a public repository are safe to use. To compensate, we’ve resorted to a number of code scanners, meta data aggregators, and risk scoring algorithms to determine whether a particular piece of software is relatively “safe”. I wonder if we’re missing the obvious here.

    Are We Reinventing the Wheel?

    Linux distributions never went away, of course. They’ve been around the whole time, although assigned to the uncool corner of the club, but they’re still here. I’m wondering if now is a moment for their return as the primary platform application development. One of the perennial struggles of keeping a distribution up to date was the sheer number of libraries one had to curate and oversee, which is in the tens of thousands. Here’s where the story of automation can come back and play a role in the rebirth of the distribution. It turns out that the very same automation tools that led some IT shops to get too far ahead over their skis and place their organizations at risk also allow Linux distributions to operate with more agility. Whereas in the past distributions struggled to keep up the pace, now automated workflows allow curation to operate quickly enough for most enterprise developers. Theoretically, this level of automated curation could be performed by enterprise IT, and indeed it is at some places. But for teams who don’t have expertise in the area of open source maintainership or open source packaging, the risk is uncertain.

    Is It Time for a Comeback?

    I don’t know for a fact that Linux distributions are poised to return to the center of application development, but I do know that much of what we’re doing to isolate and mitigate risk – security scanning, dependency curation, policy enforcement, and scorecards – feels an awful lot like what you get “out of the box” with a distribution. Enterprise IT has moved to a different delivery model than what existed previously, and moving away from that is not trivial. But if I were looking to start an organization or team from scratch, and I wanted to reduce the risk of supply chain attacks, I would probably elect to outsource risk mitigation to a curated distribution as much as possible.

  • The Open Source Supply Chain Was Always Broken

    I’ve written a number of articles over the years about open source software supply chains, and some of the issues confronting open source sustainability. The ultimate thrust of my supply chain advocacy culminated in this article imploring users to take control of their supply chains. I naively thought that by bringing attention to supply chain issues, more companies would step up to maintain the parts that were important to them. When I first started brining attention to this matter, it was November 2014, when I keynoted for the first time at a Linux Foundation event. Over the next 3 years, I continued to evolve my view of supply chains, settling on this view of supply chain “funnels”:

    Diagram of a typical open source supply chain funnel, where upstream comments are pulled into a distribution, packaged for widespread consumption and finally made into a product
    Diagram of open source supply chian funnel

    So, what has happened since I last published this work? On the plus side, lots of people are talking about open source supply chains! On the downside, no one is drawing the obvious conclusion: we need companies to step up on the maintenance of said software. In truth, this has always been the missing link. Unfortunately, what has happened instead is that we now have a number of security vendors generating lots of reports that show thousands of red lights flashing “danger! danger!” to instill fear in any CISO that open source software is going to be their undoing at any given moment. Instead of creating solutions to the supply chain problem, vendors have instead stepped in to scare the living daylights out of those assigned the thankless task of protecting their IT enterprises.

    Securing Open Source Supply Chains: Hopeless?

    Originally, Linux distributions signed on for the role of open source maintainers, but the world has evolved towards systems that embrace language ecosystems with their ever-changing world of libraries, runtimes, and frameworks. Providing secure, reliable distributions that also tracked and incorporated the changes of overlaid language-specific package management proved to be a challenge that distribution vendors have yet to adequately meet. The uneasy solution has been for distribution vendors to provide the platform, and then everyone re-invents (poorly) different parts of the wheel for package management overlays specific to different languages. In short, it’s a mess without an obvious solution. It’s especially frustrating because the only way to solve the issue in the current environment would be for a single vendor to take over the commercial open source world and enforce by fiat a single package management system. But that’s frankly way too much power to entrust to a single organization. The organizations designed to provide neutral venues for open source communities, foundations, have also not stepped in to solve the core issues of sustainability or the lack of package management standardization. There have been some efforts that are noteworthy and have made a positive impact, but not the extent that is needed. Everyone is still wondering why certain critical components are not adequately maintained and funded, and everyone is still trying to undertand how to make language-specific package ecosystems more resilient and able to withstand attacks from bad-faith users and developers. (note: sometimes the call *is* coming from inside the house)

    But is the supply chain situation hopeless? Not at all. Despite the inability to solve the larger problems, the fact is that every milestone of progress brings us a step closer to more secure ecosystems and supply chains. Steps taken by multiple languages to institute MFA requirements for package maintainers, to use but one example, result in substantial positive impacts. These simple, relatively low-cost actions cover the basics that have long been missing in the mission to secure supply chains. But that brings us to a fundamental issue yet to be addressed: whose job is it to make supply chains more secure and resilient?

    I Am Not Your Open Source Supply Chain

    One of the better essays on this subject was written by Thomas Depierre titled “I Am Not a Supplier“. While the title is a bit cheeky and “clickbait-y” (I mean, you are a supplier, whether you like it or not) he does make a very pertinent – and often overlooked – point: developers who decide to release code have absolutely no relationship with commercial users or technology vendors, especially if they offer no commercial support of that software. As Depierre notes, the software is provided “as is” with no warranty.

    Which brings us back to the fundamental question: if not the maintainers, whose responsibility is open source supply chains?

    The 10% Rule

    I would propose the following solution: If you depend on open source software, you have an obligation to contribute to its sustainability. That means if you sell any product that uses open source software, and if your enterprise depends on the use of open source software, then you have signed on to maintain that software. This is the missing link. If you use, you’re responsible. In all, I would suggest replacing 10% of your engineering spend with upstream open source maintenance, and I’ll show how it won’t break the budget. There are a number of ways to do this in a sustainable way that leads to higher productivity and better software:

    • Hire a maintainer for software you depend on – this is a brute force method, but it would be valuable for a particularly critical piece of software
    • Fund projects dedicated to open source sustainability. There are a number of them, many run out of larger software foundations, eg. The Linux Foundation, the ASF, Eclipse, the Python Software Foundation, and others.
    • Pay technology vendors who responsibly contribute to upstream projects. If your vendors don’t seem to support the upstream sources for their software, you may want to rethink your procurement strategies
    • Add a sustainability clause to your Software Bills of Materials (SBOM) requirements. Similar to the bullet above, if you start requiring your vendors to disclose their SBOMs, add a requirement that they contribute to the sustainability of the projects they build into their products.

    There is, of course, still a need to coordinate and maximize the impact. Every critical piece of software infrastructure should be accounted for on a sustainability metric. Ideally, software foundations will step up as the coordinators, and I see some progress through the Alpha and Omega project. It doesn’t quite reach the scale needed, but it is a step in the right direction.

    If you work for a company that uses a lot of open source software (and chances are that you do) you may want to start asking questions about whether your employers are doing their part. If you do the job well of sustaining open source software and hardening your supply chains, you can spend a lot less on “security” software and services that generate reports that show thousands of problems. By coordinating with communities and ecosystems at large, you can help solve the problem at the source and stop paying ambulance chasers that capitalize on the fear. That’s why spending 10% of your IT budget on open source sustainability will be budget neutral for the first 2 years and deliver cost savings beyond that. Additionally, your developers will learn how to maintain open source software and collaborate upstream, yielding qualitative benefits in the form of greater technology innovation.

  • Open Source and SaaS

    Now that I work in an engineering environment tailored for SaaS development, I’ve developed a better understanding of the challenges they face when open sourcing their code. I wrote it up for OpenSource.com in a 2-part article, “How to decide whether to open source your SaaS solution.”

    Some tidbits:

    The decision to open source code requires a fair bit of planning if you want to do it right, especially when it comes to user support and documentation. In the case of SaaS, the required planning is different, although it shares some factors with any open source effort. In my series, How to Make Money from Open Source Platforms, I focused on software that exists solely to be deployed on a computer, whether on a local machine, in a data center, or in a cloud platform (yes, I know the last two are redundant).

    There was a simple reason for this focus: It was what I knew. In my career, I have always worked with software, of commercial and community origins, to be installed somewhere. Now I work directly with engineers who take software designed to work solely on their website and with their particular infrastructure, automation, and orchestration. The fact they have been able to take this software and offer it to others in a way that is not only usable but can actually power other businesses is a testament to their commitment to an open source world.

    This article attempts to summarize their experience and point out lessons to draw from it. I’ll also try to identify how open source efforts relate to business and product strategy for SaaS models.

    I try to go into some level of detail, using my favorite tool: supply chain funnel analysis. If you’re looking into taking your SaaS code open source, I hope this helps you.

    Read the full article. 

  • Podcast: Shane Coughlan of Openchain

    Podcast: Shane Coughlan of Openchain

    Shane Coughlan is the founder and manager of the Openchain Project, which “builds trust in open source by making open source license compliance simpler and more consistent.” As any software asset management person can tell you, they get cross-eyed when it comes to open source license compliance. My opinion has always been that this was due to lack of information outside of the immediate sphere of open source developers. The Openchain Project aims to remedy that, and in this podcast we talked about the challenges of doing that. It’s a great listen!

  • Sustainable Open Source – Where Are the Vendors?

    Sustainable Open Source – Where Are the Vendors?

    Harvard Business Review has an article comparing old, crusty open source code to the Y2K ordeal. Go ahead and read it – it’s worth your time.

    Joshua Gans, the author, lists open source projects that are maintained by lonely developers who don’t make much money (if any) for producing their craft. He specifically calls out ntpd:

    What if I told you that the entire NTP relies on the sole effort of a 61-year-old who has pretty much volunteered his own time for the last 30 years? His name is Harlan Stenn…

    For a number of years Stenn has worked on a shoestring budget. He is putting in 100 hours a week to put patches on code, including requests from big corporations like Apple… And this has led to delays in fixing security issues and complaints.

    Then Gans includes this bit, which is also a personal favorite of mine whenever I talk about open source product management:

    …Last year we saw the consequences from this when a 28-year-old developer briefly “broke“ the internet because he deleted open-source code that he had made available. The drama occurred because the developer’s program shared a name with Kik, the popular Canadian messaging app, and there was a trademark dispute. The rest of the story is complicated but has an important takeaway: Our digital infrastructure is very fragile.

    Gans then adds links and descriptions of two efforts in the code sustainability area, which are worth mentioning here:  Open Collective and libraries.io. Strangely, he didn’t mention the Core Infrastructure Initiative, sponsored by the Linux Foundation, which works to address similar issues in the space.

    I agree with much of what Gans writes. There is indeed a problem with unmaintained crusty code, which manifests itself in the form of security vulnerabilities and things that break more easily than they should. In fact, it’s become such a well-known issue that GitHub and others recently sponsored a conference in SF to talk about it. But in all this discussion, and in going through the non-profit organizations dedicated to working on sustainable open source code, I have to ask: where are the vendors?

    As became all too apparent after reading Swapnil Bhartiya’s excellent primer on the state of IoT security (haha), this is a failure of product lifecycle management. The fact that much of the code in question is open source doesn’t really matter from a production point of view – code is code, after all. But because it is open source, this becomes a much wider scale problem, due to the ease with which it spreads from one poorly maintained repository to another. Philip DesAutels, the Linux Foundation’s IoT lead, said it best in Bhartiya’s article when he called out vendors for failing at product development basics.

    On one hand, I’m glad some vendors are participating in initiatives like the CII from the Linux Foundation, but I wish there was more pressure on said vendors to collaborate on these things before it becomes a problem. This is, of course, the essence of why I created OSEN in the first place. Not enough people talk about this stuff, and I want to do my part to fix that. But I don’t think it does us much good to talk around the problem; let’s put the pressure where it should be – on the vendors. They should be working with upstream maintainers, collaborating and devoting resources where necessary, and perhaps even taking on more responsibilities with the project leadership. I know this can lead to governance issues, but the alternative is more dire.

    Whether putting more effort into projects like the CII or voting with our wallets for better product development and testing, it’s time to start raising this as more of an issue. Sustainability of open source development is certainly important for future economic development, and vendors, who greatly benefit from the existence of open source code, have a responsibility to do their part.

  • It’s the Ecosystem, Stupid

    It’s the Ecosystem, Stupid

    I published a bit over at OpenSource.com.

    Read the full article here.

    It’s a plea to look externally and figure out how your technology relates to all that’s happening in the greater ecosystem. There are still way too many companies who suffer from NIH and end up saddled with way too much technical debt. Don’t do that. Take the time and effort to make inroads into all those communities that are making all the new innovations.

     

     

  • Product Development in the Age of Cloud Native

    Product Development in the Age of Cloud Native

    In defense of the community distribution

    Ever since the mass adoption of Agile development techniques and devops philosophies that attempt to eradication organizational silos, there’s been a welcome discussion on how to optimize development for continuous delivery on a massive scale. Some of the better known adages that have taken root as a result of this shift include “deploy in production after checking in code” (feasible due to the rigorous upfront testing required in this model), “infrastructure as code”, and a host of others that, taken out of context, would lead one down the path of chaos and mayhem. Indeed, the shift towards devops and agile methodologies and away from “waterfall” has led to a much needed evaluation of all processes around product and service delivery that were taken as a given in the very recent past.

    In a cloud native world, where workloads and infrastructure are all geared towards applications that spend their entire life cycle in a cloud environemnt, One of the first shifts was towards lightning fast release cycles. No longer would dev and ops negotiate 6 month chunks of time to ensure safe deployment in production of major application upgrades. No, in a cloud native world, you deploy incremental changes in production whenever needed. And because the dev and test environments have been automated to the extreme, the pipeline for application delivery in production is much shorter and can be triggered by the development team, without needing to wait for a team of ops specialists to clear away obstacles and build out infrastructure – that’s already done.

    Let me be clear: this is all good stuff. The tension between dev and ops that has been the source of frustration over the centuries has left significant organizational scar tissue in the form of burdensome regulations enforced by ops teams and rigid hierarchies which serve to stifle innovation and prevent rapid changes. This is anathema, of course, to the whole point of agile and directly conflicts with the demands of modern organizations to move quickly. As a result, a typical legacy development pathway may have looked like this:

    Screen Shot 2017-05-19 at 9.27.45 AM
    3-stage development process, from open source components to 1st software integration to release-ready product

    In the eyes of agile adherents, this is heretical. Why would you waste effort creating release branches solely for the purpose of branching again and going through another round of testing? This smacks of inefficiency. In a cloud native world, developers would rather cut out the middle step entirely, create a better, comprehensive testing procedure, and optimize the development pipeline for fast delivery of updated code. Or as Donnie Berkholz put it: this model implies waterfall development. What a cloud native practitioner strives for is a shortened release cycle more akin to this:

    Screen Shot 2017-05-19 at 10.20.03 AM
    2-stage process, from open source components to deployable product

    Of course, if you’ve read my series about building business on open source products and services, you’ll note that I’m a big advocate for the 3-step process identified in figure 1. So what gives? Is it hopelessly inefficient, a casualty of the past, resigned to the ash heap of history? I’ll introduce a term here to describe why I firmly believe in the middle step: orthogonal innovation.

    Orthogonal Innovation

    In a perfect world, innovation could be perfectly described before it happens, and the process for creating it would take place within well-defined constructs. The problem is that innovation happens all the time, due to the psychological concept of mental incubation, where ideas fester inside the brain for some indeterminate period of time, until finding its way into a conscious state, producing an “Aha!” moment. Innovation is very much conjoined with happenstance and timing. People spawn innovative ideas all the time, but the vast majority of them never take hold.

    As I wrote in It Was Never About Innovation, the purpose of communities created around software freedom and open source was never to create the most innovative ecosystems in human history – that was just an accidental byproduct. By creating rules that mandated all parties in an ecosystem were relative equals, the stage was set for massively scalable innovation. If one were to look at product lifecycles solely from the point of view of engineering efficiency, then yes, the middle piece of the 3-stage pathway looks extraneous. However, the assumption made is that a core development team is responsible for all necessary innovation, and none more is required. That model also assumes that a given code base has a single purpose and a single customer set. I would argue that the purpose of the middle stage is to expose software to new use cases and people that would have a different perspective from the primary users or developers of a single product. Furthermore, once you expose this middle step to more people, they need a way to iterate on further developments for that code base – developments that may run contrary to the goals of the core development team and its customers. Let’s revisit the 3-stage process:

    Screen Shot 2017-05-19 at 10.48.43 AM

    In this diagram, each stage is important for different reasons. The components on the left represent raw open source supply chain components that form the basis for the middle stage. The middle stage serves multiple entities in the ecosystem that springs up around the code base and is a “safe space” where lots of new things are tried, without introducing risk into the various downstream products. You can see echoes of t his in many popular open source-based products and services. Consider the Pivotal Cloud Foundry process, as explained by James Watters in this podcast with Andreessen Horowitz: raw open source components -> CloudFoundry.org -> Pivotal Cloud Foundry, with multiple derivatives from CloudFoundry.org, including IBM’s.

    As I’ve mentioned elsewhere, this also describes the RHEL process: raw components -> Fedora -> RHEL. And it’s the basis on which Docker is spinning up the Moby community. Once you’ve defined that middle space, there are many other fun things you can do, including building an identity for that middle distribution, which is what many open source communities have formed around. This process works just as well from an InnerSource perspective. Except in that case, the downstream products’ customers are internal, and there are multiple groups within your organization deriving products and services from the core code base in the middle stage. Opening up the middle stage to inspection and modification increases the surface area for innovation and gives breathing room for the more crazy ideas to take shape, possibly leading to their becoming slightly less crazy and useful for the other downstream participants.

    For more on this, come to our meetup at the Microsoft NERD center in Cambridge, MA on May 23, where I’ll present on this subject.

    Addendum: none of the above necessitates a specific development methodology. It could be agile, waterfall, pair programming or any other methodology du jour – it’s immaterial. What matters is constructing a process that allows for multiple points of innovation and iterative construction, even – or especially – where it doesn’t serve the aims of a specific downstream product. You want a fresh perspective, and to get that, you have to allow those with different goals to participate in the process.

  • Supply Chain Case Study: Canonical and Ubuntu

    canonical-logo1

    I love talking about supply chain management in an open source software context, especially as it applies to managing collaborative processes between upstream projects and their downstream products. In the article linked above, I called out a couple of examples of supply chain management: an enterprise OpenStack distribution and a container management product utilizing Kubernetes and Docker for upstream platforms.

    What about anti-patterns or things to avoid? There are several we could call out. At the risk of picking on someone I like, I’ll choose Canonical simply because they’ve been in the headlines recently for changes they’ve made to their organization, cutting back on some efforts and laying off some people. As I look at Canonical from a product offering perspective, there’s a lot they got right, which others could benefit from. But they also made many mistakes, some of which could have been avoided. First, the good.

    What Canonical Got Right About Supply Chains

    When the Ubuntu distribution first started in 2004, it made an immediate impact; the kind of impact that would frankly be impossible today for a Linux distribution. Remember what was happening at the time: many, many Red Hat Linux distribution users were feeling left out in the cold by Red Hat’s then groundbreaking decision to fork their efforts into a community stream and a product stream. One of the prevailing opinions at the time was that Fedora would be treated like the red-headed stepchild and starved for resources. Unfortunately, Red Hat played into that fear by… initially treating Fedora like the red-headed stepchild and almost willfully sabotaging their own community efforts. (for a good run-down of the 2004 zeitgeist and some LOL-level hilarity, see this page on LWN).

    Ubuntu never had that problem. From the very outset, there was never any doubt that Mark Shuttleworth and crew meant what they said when they set out to deliver an easy-to-use, free distribution. Lots of people tried to do that, but Ubuntu went about it more intelligently and made a lot more progress than its predecessors. Where did they succeed where others failed?

    1. They chose a great upstream platform. Instead of building something from scratch (which would have taken forever) or using the abandoned Red Hat Linux or even Mandrake, which were both going through awkward transitional phases (one to Fedora and the other out of the Red Hat orbit), they built Ubuntu on a rock-solid, dependable, community-maintained Linux distribution: Debian. openSUSE was not yet a thing, and building on SuSE Linux would have tied Ubuntu to the fortunes of SuSE and Novell, which would have been a bad idea. Slackware was long in the tooth, even then. Debian had its own challenges, not the least of which was a clash of cultures between free culture diehards and a group of people starting a for-profit entity around Debian, but it worked for Ubuntu’s purposes. It was also a challenge to install, which provided a great opportunity for an upstart like Ubuntu.
    2. Their supply chain was highly efficient, which is directly related to the above. Contrast this to what I’ll say below, but in the case of the base platform they started from, the software supply chain that made up Ubuntu was reliable and something its developers and users could depend on.
    3. They invested in the user experience and community. Ubuntu classified itself, at least back then, as “Linux for humans”, which spoke to the fact that, up until then, using Linux was an  esoteric and mistake-prone set of tasks. It was the sort of thing you did in your spare time if you were a CS or EE major looking to construct a new toy. Ubuntu changed all that. They made Linux much easier than any previous desktop Linux initiative. From a supply chain perspective, they did this great UX work as participants in the greater Gnome community. I realize some Gnome folks may blanch at that statement, but by and large, Ubuntu was very much depicted as Gnome people, and they were making contributions to the greater effort.
    4. They scaled globally, from the beginning. It was awe-inspiring to see all the local communities (LoCos, in Ubuntu parlance) spring up around the world dedicated to evangelizing Ubuntu and supporting its users. This happened organically, with Canonical providing support in the form of tooling, broad messaging, and in some cases, on the ground resources. Ubuntu also employed a formidable community team helmed by Jono Bacon, who accelerated Ubuntu’s growth (I once chided Jono on a community managers panel at OSCON about how easy he had it as the Ubuntu community lead. I still chuckle over that). One cannot emphasize enough that when this massive global community kicks into gear, the effect on upstream supply chains is tremendous. A hefty number of these global users and developers also became participants in many of the upstream communities that fed into Ubuntu. It’s one of the great examples of increased participation yielding positive results for everyone in the ecosystem, including Canonical.
    5. They were early adopters of “cloud native” workloads. As Simon Wardley will never let us forget, Canonical bought into cloud-based workloads before any other Linux vendor. It was their work in 2008-2009 that really cemented Ubuntu’s status as *the* primary and default platform for all new cloud and server-based technologies, which continues to this day. Even now, if a new project wants to get early adoption, they release .DEBs on Ubuntu and make sure it builds properly for those Ubuntu users and developers who download the source code. It gives Ubuntu and Canonical an incredible advantage. Again, from a supply chain perspective, this was gold. It meant that the upstream supply chain for cloud native tools was heavily Ubuntu-centric, wrapping it all together with a nice bow.

    Where it Went Pear-shaped

    In writing about everything they got right, I am of course using the rhetorical device of setting up the reader for a barrage of things they got wrong. This is that section. For all of their incredible acumen at scaling out a global community around a Linux distribution, they failed to learn from their supply chain success, and instead started down the path of NIH syndrome. You’ll see lots of critiques elsewhere pertaining to other challenges at Canonical, but I’ll focus on their supply chain strategy, and how it failed them.

    1. Launchpad. The first sign that Canonical were moving away from their established supply chain methodology was when they first released Launchpad, a web-based service for developers to create, share, and collaborate on software projects. It also featured an auto-build service and an easy way to release and manage unofficial builds of bleeding edge software: the “Personal Package Archive” or PPA. The service was great for its time and highly ambitious. And when Canonical announced they were open-sourcing it, even better. But there were problems: maintaining a code base for a service as complex as Launchpad is really difficult. Even with an entire company devoted to such a concept, there are still major challenges. There were a couple of ways to deal with that complexity: upstream as much as possible to defray the cost of maintaining the code and/or create a long-term revenue model around launchpad to sustain its development. Canonical did neither. In fact, it was the worst of both worlds: they neither upstreamed the project nor created a revenue model to sustain development. In other words, Launchpad became a proverbial albatross around the neck, both in terms of technical debt to be maintained solely by the Launchpad team and in the lack of funding for future development. It was the first sign that Canonical was on the wrong track from a business strategy viewpoint. The polished user experience that Ubuntu users came to expect from their software was missing from Launchpad, giving GitHub the opportunity to build something larger.
    2. Juju. It might be premature to write off Juju entirely, but it hasn’t become the force Canonical and Ubuntu intended it to be. Written at a time when Puppet and Chef were the young upstarts, and Ansible was but a gleam in Cobbler’s eye, Juju was Canonical’s answer to the problem of configuration management in the age of cloud. It might have had a better chance if Canonical had decided to be a bit looser with its user base. Puppet and Chef, for example, were always available on a variety of platforms, whereas Juju was specifically tied to Ubuntu. And while Ubuntu became the de facto standard for building cloud tools, the enterprise was still dominated by Windows, Unix, and RHEL. Developers may have built many tools using Ubuntu, but they deployed in production on other platforms, where Juju was not to be found. If you were an enterprising young devops professional, going with a Juju-only approach meant cutting off your air supply. Because it was Ubuntu-only, and because it was never a core part of the Debian upstream community, the impact made by Juju was limited. Canonical was unable to build a collaborative model with other developer communities, which would have improved the supply chain efficiency, and they weren’t able to use it to add value to a revenue-generating product, because their capacity for generating server revenue was limited. It’s another case of great software hobbled by a poor business strategy.
    3. Unity. If Launchpad and Juju gave observers an inkling that Canonical was going off the rails, the launch of Unity confirmed it. From the beginning, Canonical was always a participant in the Gnome desktop community. This made sense, because Ubuntu had always featured a desktop environment based on Gnome. At some point, Canonical decided they could go faster and farther if they ditched this whole Gnome community thing and went their own way. As with Launchpad and Juju, this makes sense if you’re able to generate enough revenue to sustain development over time with a valid business model. I personally liked Unity, but the decision to go with it over stock Gnome 3 drove an even larger wedge between Ubuntu and the rest of the Linux desktop ecosystem. Once again, Ubuntu packagers and developers were caught in a bubble without the support of an upstream community to stabilize the supply chain. This meant that, once again, Canonical developers were the sole developers and maintainers of the software, further straining an already stretched resource.
    4. Mir. I don’t actually know the origins of Mir. Frankly, it doesn’t matter. What you need to know is this: the open source technology world participated in the Wayland project, whose goal was to build a modern successor to the venerable X.org windows server, and Canonical decided to build Mir, instead. The end. Now, Mir and Unity are, for all intents and purposes, mothballed, and Wayland is the clear winner on the desktop front. Supply chains: learn them, live them, love them – or else.
    5. Ubuntu mobile / Ubuntu phone. The mobile space is extremely difficult because the base hardware platforms are always proprietary, as mandated by the large carriers who set the rules for the entire ecosystem. It’s even more difficult to navigate when you’re launching a product that’s not in your area of expertise, and you try to go to market without a strong ecosystem of partners. The iPhone had AT&T in its corner. The Ubuntu phone had… I’m not even sure.  Ubuntu phone and the mobile OS that ran on it were DOA, and they should have understood that much sooner than they did.
    6. Ubuntu platform itself. I know, I spent the first half of this article talking up the great success of Ubuntu, but there is one place where it never excelled: it never became a large enough revenue generator to sustain the many other projects under development. There was also never a coherent strategy, product-wise, around what Ubuntu should grow up to become. Was it a cloud platform? Mobile platform? Enterprise server? Developer workstation? And there was never an over-arching strategy with respect to the complementary projects built on top of Ubuntu. There was never a collective set of tools designed to create the world’s greatest cloud platform. Or enterprise server. Or any of the other choices. Canonical tried to make Ubuntu the pathway to any number of destinations, but without the product discipline to “just say no” at the appropriate time.

    I get no joy from listing the failings of Canonical. I remain a great fan of what they accomplished on the community front, which as far as I can tell, is without parallel. Not many companies can claim with any credibility that they fostered a massive, global community of users and developers that numbered in the millions and covered every country and continent on the planet, driven by organic growth and pursued with a religious zeal. That is no small feat and should be celebrated. My hope is that this is what Ubuntu, Canonical, and yes, Mark Shuttleworth, are known for, and not for any business shortcomings.

    I’m not suggesting that a company cannot be successful without building out an upstream supply chain – there are far too many counter-examples to claim that. What I am suggesting is that if you have limited resources, and you choose to build out so many products, you’re going to need the leverage that comes from massive global participation. If Canonical had chosen to focus on one of the above initiatives, you could argue that supply chain would not have been as important. I will note, for the record, that none of the challenges listed above are related to the fact that they were open source. Rather, to sustain their development, they needed much broader adoption. In order to sustain that model, they would have had to create a successful product with a high growth in revenue, which never came. The lesson: if you want more control over your software products, you need an accompanying product strategy that supports it. If I were at Canonical, I would have pushed for a much more aggressive upstream strategy to get more benefits from broader open source participation.

  • An Open Letter to Docker About Moby

    Congratulations, Docker. You’ve taken the advice of many and gone down the path of Fedora / RHEL. Welcome to the world of upstream/downstream product management, with community participation a core component of supply chain management. You’ve also unleashed a clever governance hack that cements your container technology as the property of Docker, rather than let other vendors define it as an upstream technology for everyone. Much like Red Hat used Fedora to stake its claim as owner of an upstream community. I’ll bet the response to this was super positive, and everyone understood your intent perfectly! Oh…

    So yes, the comparison to Fedora/RHEL is spot on, but you should also remember something from that experiment: at first, everyone *hated* it. The general take from the extended Linux community at the time was that Red Hat was abandoning community Linux in an effort to become “the Microsoft of Linux”. Remember, this level of dissatisfaction is why CentOS exists today. And the Fedora community rollout didn’t exactly win any awards for precise execution. At first, there was “Fedora Core”, and it was quite limited and not a smashing success. This was one of the reasons that Ubuntu became as successful as it did, because they were able to capitalize on the suboptimal Fedora introduction. Over time, however, Red Hat continued to invest in Fedora as a strategic community brand, and it became a valuable staging ground for leading edge technologies from the upstream open source world, much like Moby could be a staging area for Docker.

    But here’s the thing, Docker: you need to learn from previous mistakes and get this right. By waiting so long to make this move, you’ve increased the level of risk you’re taking on, which could have been avoided. If you get it wrong, you’re going to see a lot of community pressure to fork Moby or Docker and create another community distribution outside of your sphere of influence. The Fedora effort frittered away a lot of good will from the Linux community by not creating an easy to use, out of the box distribution at first. And the energy from that disenchantment went to Ubuntu, leaving Fedora in a position to play catchup. That Red Hat was able to recover and build a substantial base of RHEL customers is a testament to their ability to execute on product management. However, Ubuntu was able to become the de facto developer platform for the cloud by capitalizing on Fedora’s missteps, putting them on the inside track for new cloud, IoT, and container technologies over the years. My point is this: missteps in either strategy and execution have a large and lasting impact.

    So listen up, Docker. You need to dedicate tremendous resources right now to the Moby effort to make sure that it’s easy to use, navigate, and most importantly, ensure that your community understands its purpose. Secondly, and almost as importantly, you need to clearly communicate your intent around Docker CE and EE. There is no room for confusion around the difference between Moby and Docker *E. Don’t be surprised if you see a CentOS equivalent to Docker CE and/or EE soon, even though you’re probably trying to prevent that with a freely available commercial offering. Don’t worry, it will only prove your model, not undermine it, because no one can do Docker better than Docker. Take that last bit to heart, because far too many companies have failed because they feared the success of their free stuff. In this case, that would be a colossal unforced error.