Blog

  • The Revenge of the Linux Distribution

    The Revenge of the Linux Distribution

    Some things appear in hindsight as blindingly obvious. And to some of us, perhaps they seemed obvious to the world even at the time. The observations of Copernicus and Galileo come to mind. To use a lesser example, let’s think back to the late 2000s and early 2010s when this new-fangled methodology called “devops” started to take shape. This was at a moment in time when “just-in-time” (JIT) was all the rage, and just-in-time continuous integration (CI) was following the same path as just-in-time inventory and manufacturing. And just like JIT inventory management had some weaknesses that were exposed later (supply chain shocks), so too were the weak points of JIT CI similarly exposed in recent security incidents. But it wasn’t always thus – let’s roll back the clock even further, shall we?

    Party like it’s 1999

    Back when Linux was first making headway towards “crossing the chasm” in the late 90s, Linux distributions were state of the art. After all, how else could anyone keep track of the all the system tools, core libraries, and language runtime dependencies without a curated set of software packaged up as part of a Linux distribution? Making all this software work together from scratch was quite difficult, so thank goodness for the fine folks at Red Hat, SuSE, Caldera, Debian, and Slackware for creating ready-made platforms that developers could rely on for consistency and reliability. They featured core packages by default that would enable anyone, so long as they had hardware and bandwidth, to run their own application development shop and then deliver those custom apps on the very same operating systems, in one consistent dev-run-test-deploy workflow. They were almost too good – so good, in fact, that developers and sysadmins (ahem, sorry… “devops engineers”) started to take them for granted. The heyday of the linux distribution was probably 2006, when Ubuntu Linux, which was based on Debian, became a global phenomenon, reaching millions of users. But then a funny thing happened… with advances in software automation, the venerable Linux distribution started to feel aged, an artifact from a bygone time when packaging, development, and deployment were all manual processes, handled by hand-crafted scripts created with love by systems curmudgeons who rarely saw the light of day.

    The Age of DevOps

    With advances made in systems automation, the question was asked, reaching a crescendo in the early to mid-2010’s, “why do we need Linux distributions, if I can pull any language runtime dependency I need at a moment’s notice from a set of freely available repositories of artifacts pre-built for my operating system and chip architecture? Honestly, it was a compelling question, although it did lead to iconic graphics like this one from XKCD:

    For a while it was so easy. Sure, give me a stripped down platform to start with, but then get the operating system out of the way, and let me design the application development and deployment layers. After all, any competent developer can assemble the list of dependencies they will need in their application. Why do I need Red Hat to curate it for me? Especially when their versions are so out of date? The rise of Docker and the race to strip down containers was a perfect example of this ethos.

    A few incidents demonstrated the early limitations of this methodology, but for the most part the trend continued apace, and has remained to this day. But now it feels like something has changed. It feels like curation is suddenly back in vogue. Because of the risks from typo-squatting, social engineering hacks, and other means of exploiting gaps in supply chain security, I think we’ve reached somewhat of a sea change. In a world where the “zero trust” buzzword has taken firm hold, it’s no longer en vogue to simply trust that the dependencies you download from a public repository are safe to use. To compensate, we’ve resorted to a number of code scanners, meta data aggregators, and risk scoring algorithms to determine whether a particular piece of software is relatively “safe”. I wonder if we’re missing the obvious here.

    Are We Reinventing the Wheel?

    Linux distributions never went away, of course. They’ve been around the whole time, although assigned to the uncool corner of the club, but they’re still here. I’m wondering if now is a moment for their return as the primary platform application development. One of the perennial struggles of keeping a distribution up to date was the sheer number of libraries one had to curate and oversee, which is in the tens of thousands. Here’s where the story of automation can come back and play a role in the rebirth of the distribution. It turns out that the very same automation tools that led some IT shops to get too far ahead over their skis and place their organizations at risk also allow Linux distributions to operate with more agility. Whereas in the past distributions struggled to keep up the pace, now automated workflows allow curation to operate quickly enough for most enterprise developers. Theoretically, this level of automated curation could be performed by enterprise IT, and indeed it is at some places. But for teams who don’t have expertise in the area of open source maintainership or open source packaging, the risk is uncertain.

    Is It Time for a Comeback?

    I don’t know for a fact that Linux distributions are poised to return to the center of application development, but I do know that much of what we’re doing to isolate and mitigate risk – security scanning, dependency curation, policy enforcement, and scorecards – feels an awful lot like what you get “out of the box” with a distribution. Enterprise IT has moved to a different delivery model than what existed previously, and moving away from that is not trivial. But if I were looking to start an organization or team from scratch, and I wanted to reduce the risk of supply chain attacks, I would probably elect to outsource risk mitigation to a curated distribution as much as possible.
  • Whither the OSPO?

    Whither the OSPO?

    I read Dirk Riehle’s recent post on the OSPO Lifecycle, and it conjured up some thoughts that I’ve had recently and have been meaning to write down. Something has been bothering me about the concept of Open Source Program Offices (OSPOs) within corporations and where they fit in value stream discussions, especially since a few OSPOs suffered waves of layoffs and saw a reduction in scope. As a professional OSPO guy, it certainly turned my head and made me think. In Dirk’s post, he points out that the OSPO provides an important leadership function, mostly at the start. Over time, as the company’s open source involvement matures, the OSPO reaches an inflection point and transitions from a thought leadership role to one of coordination and support. The mature OSPO performs a support function for open source governance and compliance, as well as procedural guidance for the few lucky engineers who get to actively participate in external communities. This makes sense if you think of the OSPO as an atomic entity, riding a 5-year lifecycle from inception to “business as usual”.

    But what if OSPOs are not atomic entities? When I think about how OSPOs function, what is often missed is its role in developer productivity. Back when OSPOs were first stood up inside tech vendors, before they were even called OSPOs, a big incentive was vendors wanting to capture value from software produced by collaborative communities. Vendors wanted to be able to reliably use community-produced software embedded within products that they sold. This required a different view of supply chain and product management than had ever existed before, and OSPOs were the chosen vehicle for doing so. Along the way, these vendors discovered that an additional source of value was learning how to collaborate in an open source way. Suddenly, they weren’t just pulling software down from communities, they were actively collaborating with these communities. What OSPOs helped vendors achieve was producing products using the principles of open source collaboration. To me, the enablement of community collaboration and the embrace of open source principles was always the primary value of an OSPO. In that light, to constrain the scope of an OSPO to one of coordination and support is to miss the primary opportunities for value.

    What’s in a Name?

    I think a maturing OSPO needs a name that reflects its aspirational scope. If the ultimate value of an OSPO is measured in developer productivity, then perhaps what’s holding it back is the name. A “program office” may seem like an interesting place to invest if you’re a tech vendor, but the words “program office” have a very different meaning inside large enterprises, one largely associated with bureaucratic functions.

    One of the messages I have incorporated into a lot of my talks since 2013 is that open source communities have been the greatest source of innovation for over two decades, going back to the linux boom of the late 90’s. Any large enterprise would do well to at least attempt to replicate the success of open source communities and instill open source principles into its engineering teams. And if you can expand your “shift left” methodologies to include open source supply chains in your SDLC, then you benefit direclty from the innovation produced by these communities. This is where an OSPO can add the most value, if that value is recognized and invested in. I don’t know that the name necessarily should be, but since accelerated innovation and higher developer productivity are the end goals, then that should be reflected.

    I think when OSPOs grow up, they should become Centers of Innovation and Developer Productivity. Let’s face it, the term “open source” doesn’t grab people like it used to. It became what we always thought it would be – a means to an end. A tool. Instead, let’s focus on the outcome we’re looking to drive: Innovation and Developer Productivity.

  • Guess Who’s Coming to Dinner

    Content warning: the following post contains racial slurs and frank depictions of racist hate speech

    I’ve spent weeks trying to figure out my approach to writing down my memory of this event. I considered writing a dramatic reenactment as a screenplay. I might still, but after much thought, I think I shouldn’t hide behind anything less than a direct retelling of the story as I remember it. Or, as the other person who was there put it, “wasn’t it dramatic enough?” lolol I couldn’t agree more. I guess I was still afraid of putting this incident to “paper”, even though it took place almot 30 years ago. So…. direct and to the point I shall be. All names have been changed to protect the innocent… and guilty.

    Setting the stage: I had just graduated from Yale in the spring of 1995. Sort of. I was 1 credit shy (don’t ask) so I had to stay over the summer and take 1 remaining course to graduate. I stayed in an apartment with a few classmates and also worked as custodial staff for the Special Olympics, which was held that year on the Yale campus. It was at the Special Olympics job that I met Liz, and we started dating. Because of a series of unfortunate events, she needed to crash at our apartment for the last few weeks in August. At the end of the month, my parents were driving over from Arkansas to pick me up, and we had the brilliant idea to have dinner with them before we all went back to our respective homes. I feel the urge to tell this story because it demonstrates how little I understood about racism at the time, how deeply brainwashed I was by white nationalist evangelical culture, and how we subject those we care about to needless harm and trauma when we don’t stand up to racism and misogyny when we encounter it. The one thing I wish I understood about racism at the time is that there’s no such thing as an innocent bystander. If you’re a passive bystander witnessing someone else’s racism, you are allowing them to inflict harm on others, effectively aiding and abetting them in the process.

    In hindsight, I should have known how this was going to turn out, but at the time I was naive (22) and still not far enough removed from my evangelical Christian upbringing to understand how toxic and hurtful my family was to others. This was long before I was aware of the famous Maya Angelou line, “When they tell you who they are, believe them.” In this case, when they (my father) used the word “coon” in his first phone conversation with Liz just to see how she would react, that was probably a good clue. See, she was of mixed heritage with a Puerto Rican mother and a mostly nordic father. And since my father knew nothing about Puerto Ricans other than what he saw on TV, he was, uh… “curious” about her ethnicity. And since he was the only father I had ever known, it didn’t seem the least bit weird to me when he asked to speak to her during one of our phone calls once he learned of her existence. He was my father, and I did as I was told. Once on the phone with her, my dad proceeded to interrogate her, including the question, “what do you call black people?” I don’t know exactly how Liz answered that question or how the conversation unfolded afterwards, but somehow my dad thought it pertinent to cheerily volunteer that “out here, we call ’em coons.” He hadn’t yet even seen a picture of Liz, but what he *really* wanted to know, without stating it, was, “does she look black” and “how black is she.” Because I was a product of his tutelage and hadn’t yet addressed my own racism and misogyny, I thought I was being helpful when I said, “No, no – she doesn’t look Puerto Rican (black) at all.” I don’t remember if this all happened during the same phone conversation or a different one, but it doesn’t really matter. This was all a prelude to the main event – my parents were picking me up from college, and I was leaving New Haven to go back to Arkansas, until I could put together a plan for San Francisco, where I wanted to relocate.

    I am painfully aware of how terrible this all sounds. All of it. Casually dropping a racial slur in conversation. The inappropriate line of questioning. The bizarre interest in skin color and ethnicity. The idea that it was acceptable to interrogate someone you’ve never met about their ethnicity and personal family history. To this day, Liz maintains that she wasn’t particularly offended, because for her it was an “anthropological experiment” and she knew she would never have to see these people (my parents) again. That said, it definitely alarmed her at the time that someone could be so brazenly racist. She had not encountered that before. For me, it seemed all too normal. I wish I could say I learned my lesson from this incident, but… I did not, at least not completely. That may have to wait for another post.

    On the fateful day that we were expecting my parents to arrive, we were not exactly calm. Liz had spoken to my father a few days earlier for the first time, and now she would be meeting him in person. Liz decided to make arroz con pollo, because she wanted to make something authentically Puerto Rican. I don’t remember much from that day before they arrived; I just recall a slow-burning and ceaseless state of elevated anxiety while trying to relax. And then came the phone call – they were here! Time to swing into action. The food was mostly prepped, but it would take an hour or so to cook. In the meantime, we would chat and, ya know, get to know each other. Liz felt like she was viewing another species of human – it was a real-life anthropology lab. At some point, my mom smelled the pot of chicken and remarked, “that smells very…” searching for just the right word and then looking at Liz before finding it: “ethnic”. One of Liz’s great qualities is that she’s able to put people at ease because she’s very talkative and can easily draw people into conversation. At some point, she started talking to my mom about social workers and the difficult job they have. She mentioned a family with a young boy who took care of his mother, who was disabled, and that the overworked social worker assigned to them was in a bit of an ethical quandary. My mother helpfully jumped to the conclusion that the mother must have been on drugs, and Liz paused to explain that no, the ethical dilemma was about not reporting the family because the likely outcome would be foster care and the mother would be without her caregiver. I don’t believe that race or ethnicity were ever mentioned in this story, but I’m pretty sure that my mother assumed the family was black. I remember being shocked at how little my mother understood of the world.

    It just kept getting better from there. At some point, we finally finished cooking and sat down to eat. The rest of the evening is pretty much a blur, but 2 things stand out. For one, my father decided that one racial slur wasn’t enough. No no, he had to say it again – for the same reason as before: to make Liz as uncomfortable as possible. And the 2nd thing that still stands out is Liz and I decided to go to the rooftop of the apartment building to be alone, because frankly, it was a lot. I mean, imagine being in a summer fling, your last chance at carefree fun before being forced to deal with the realities of post-college life, and you are subjected to… <waves hands around> all of *this*. It was… a lot. For our sanity’s sake, we needed 15 minutes alone on the rooftop in order to keep it together. I didn’t fully appreciate just how bad it was at the time, but I certainly do now. Reliving this evening to write it down is equal parts catharsis and relived trauma.

    As the evening wound down, Liz’s father picked her up to take her home, and I was alone. My parents were there, but I never felt more alone than in that moment. They stayed for the night, and in the morning we packed up my things, and I left New Haven forever. To me, this evening will forever live on as the moment where, for the first time, I saw the stark relief of a clash of civilizations. Up until that moment, I could live in the self-delusion that we all lived in the same universe, obeying the same laws and social mores. From that moment on, it became increasingly clear to me that this simply wasn’t the case. We did not, in fact, obey the same laws and abide by the same moral code. I felt trapped between both – desperately wanting to escape my family, but never quite accepted by the prep school kids who dominated college life. It was a long drive from Connecticut to Northeast Arkansas, and for most of the trip, Simon and Garfunkel’s “Homeward Bound” rang through my head as tears threatened to well up at any moment.

    When we finally got back home, there were a few conversations about Liz. One was my father telling me that he “approved” of her. Uh… thanks? But the other was when I overheard him talking to someone else about Liz, about how she was some “Puerto Rican girl” as if she just fell out of a West Side Story production into my life. I protested with the “defense” of how she “looked white” to which my dad responded, “oh yeah, with dark eyes and dark hair.” At the time, I didn’t understand how deeply racist my response was, but I still remember being very confused by his response. Dark hair and dark eyes? Would this description not equally apply to my own mother? I’m pretty sure I had not heard of the “1 drop rule” at the time, but this was the first time I came face to face with it. Unlike stories of Sally Hemmings or other “white-passing” slaves from the 19th century, this was an actual event I lived through toward the end of the 20th century, involving someone I cared about. It brought home how deeply ingrained white supremacy is in American culture in a way that no history textbook ever could. As white people, we too easily dismiss the harms of racism as something in the distant past, something we have evolved beyond. That is simply not the case.

    Some years later, I married an immigrant from Hong Kong. I wish I could say I was smarter and had learned from my previous experience. I had not. There were the same suspicions; the same interrogations; the same dismissiveness of her experience; and the feeling that she never quite belonged and was not “one of us”. It took some years, almost 2 decades in fact, until she decided enough was enough, and it was either me or my parents, but not both. It was only at that point that I finally understood and came to terms with my own tacit approval of and participation in white supremacy. It was then and only then that I understood how I had to turn the page on my own family and choose to move forward with my spouse. But not before years of trauma and harm were visited on people that I love. It was a hard lesson, but the takeaway is thus: there are no innocent bystanders to bigotry. When you “stand back and stand by” while bigotry is perpetrated on others, you are silently sanctioning the harm done to them. You are aiding and abetting the willful commission of white supremacist hate on your neighbors, friends, lovers, and yes, family. We are all children of Jim Crow, although we only apply that term to the black communities who suffered under its persecution – and prosecution. We seem reluctant to apply that term to white communities and families even though they were very much influenced by the Jim Crow era, segregation, desegregation, and bussing.

    We are all children of Jim Crow. We just lived on different sides of it, and its legacy is very much with us today, no matter how much we would like to dismiss it and pretend otherwise.

  • The Open Source Supply Chain Was Always Broken

    I’ve written a number of articles over the years about open source software supply chains, and some of the issues confronting open source sustainability. The ultimate thrust of my supply chain advocacy culminated in this article imploring users to take control of their supply chains. I naively thought that by bringing attention to supply chain issues, more companies would step up to maintain the parts that were important to them. When I first started brining attention to this matter, it was November 2014, when I keynoted for the first time at a Linux Foundation event. Over the next 3 years, I continued to evolve my view of supply chains, settling on this view of supply chain “funnels”:

    Diagram of a typical open source supply chain funnel, where upstream comments are pulled into a distribution, packaged for widespread consumption and finally made into a product
    Diagram of open source supply chian funnel

    So, what has happened since I last published this work? On the plus side, lots of people are talking about open source supply chains! On the downside, no one is drawing the obvious conclusion: we need companies to step up on the maintenance of said software. In truth, this has always been the missing link. Unfortunately, what has happened instead is that we now have a number of security vendors generating lots of reports that show thousands of red lights flashing “danger! danger!” to instill fear in any CISO that open source software is going to be their undoing at any given moment. Instead of creating solutions to the supply chain problem, vendors have instead stepped in to scare the living daylights out of those assigned the thankless task of protecting their IT enterprises.

    Securing Open Source Supply Chains: Hopeless?

    Originally, Linux distributions signed on for the role of open source maintainers, but the world has evolved towards systems that embrace language ecosystems with their ever-changing world of libraries, runtimes, and frameworks. Providing secure, reliable distributions that also tracked and incorporated the changes of overlaid language-specific package management proved to be a challenge that distribution vendors have yet to adequately meet. The uneasy solution has been for distribution vendors to provide the platform, and then everyone re-invents (poorly) different parts of the wheel for package management overlays specific to different languages. In short, it’s a mess without an obvious solution. It’s especially frustrating because the only way to solve the issue in the current environment would be for a single vendor to take over the commercial open source world and enforce by fiat a single package management system. But that’s frankly way too much power to entrust to a single organization. The organizations designed to provide neutral venues for open source communities, foundations, have also not stepped in to solve the core issues of sustainability or the lack of package management standardization. There have been some efforts that are noteworthy and have made a positive impact, but not the extent that is needed. Everyone is still wondering why certain critical components are not adequately maintained and funded, and everyone is still trying to undertand how to make language-specific package ecosystems more resilient and able to withstand attacks from bad-faith users and developers. (note: sometimes the call *is* coming from inside the house)

    But is the supply chain situation hopeless? Not at all. Despite the inability to solve the larger problems, the fact is that every milestone of progress brings us a step closer to more secure ecosystems and supply chains. Steps taken by multiple languages to institute MFA requirements for package maintainers, to use but one example, result in substantial positive impacts. These simple, relatively low-cost actions cover the basics that have long been missing in the mission to secure supply chains. But that brings us to a fundamental issue yet to be addressed: whose job is it to make supply chains more secure and resilient?

    I Am Not Your Open Source Supply Chain

    One of the better essays on this subject was written by Thomas Depierre titled “I Am Not a Supplier“. While the title is a bit cheeky and “clickbait-y” (I mean, you are a supplier, whether you like it or not) he does make a very pertinent – and often overlooked – point: developers who decide to release code have absolutely no relationship with commercial users or technology vendors, especially if they offer no commercial support of that software. As Depierre notes, the software is provided “as is” with no warranty.

    Which brings us back to the fundamental question: if not the maintainers, whose responsibility is open source supply chains?

    The 10% Rule

    I would propose the following solution: If you depend on open source software, you have an obligation to contribute to its sustainability. That means if you sell any product that uses open source software, and if your enterprise depends on the use of open source software, then you have signed on to maintain that software. This is the missing link. If you use, you’re responsible. In all, I would suggest replacing 10% of your engineering spend with upstream open source maintenance, and I’ll show how it won’t break the budget. There are a number of ways to do this in a sustainable way that leads to higher productivity and better software:

    • Hire a maintainer for software you depend on – this is a brute force method, but it would be valuable for a particularly critical piece of software
    • Fund projects dedicated to open source sustainability. There are a number of them, many run out of larger software foundations, eg. The Linux Foundation, the ASF, Eclipse, the Python Software Foundation, and others.
    • Pay technology vendors who responsibly contribute to upstream projects. If your vendors don’t seem to support the upstream sources for their software, you may want to rethink your procurement strategies
    • Add a sustainability clause to your Software Bills of Materials (SBOM) requirements. Similar to the bullet above, if you start requiring your vendors to disclose their SBOMs, add a requirement that they contribute to the sustainability of the projects they build into their products.

    There is, of course, still a need to coordinate and maximize the impact. Every critical piece of software infrastructure should be accounted for on a sustainability metric. Ideally, software foundations will step up as the coordinators, and I see some progress through the Alpha and Omega project. It doesn’t quite reach the scale needed, but it is a step in the right direction.

    If you work for a company that uses a lot of open source software (and chances are that you do) you may want to start asking questions about whether your employers are doing their part. If you do the job well of sustaining open source software and hardening your supply chains, you can spend a lot less on “security” software and services that generate reports that show thousands of problems. By coordinating with communities and ecosystems at large, you can help solve the problem at the source and stop paying ambulance chasers that capitalize on the fear. That’s why spending 10% of your IT budget on open source sustainability will be budget neutral for the first 2 years and deliver cost savings beyond that. Additionally, your developers will learn how to maintain open source software and collaborate upstream, yielding qualitative benefits in the form of greater technology innovation.

  • Will It Let Me Fire Some Guys?

    Cory Doctorow published an excellent essay in Locus about the AI bubble and what will happen when (not if) it goes “bloop” as bubbles are wont to do. Namely, the money in the AI ecosystem is only sustainable if it allows programs to replace people, and due to the prevalence of high risk applications, that seems highly unlikely. I think he’s absolutely right – read that first.

    Ok, done? Cool…

    Reading Cory’s essay jogged my memory about some experiences I’ve had over my tech career. The first thought that came to mind was, haven’t we been through this before? Yes, we have. Several times. And each time we learn the same lesson the hard way: paradigm shifting tech transformations do not, in fact, result in large reductions of workers. Sometimes there may be displacement and reallocation, but never reductions. No, large reductions happen when businesses decide it’s time to trim across the board or exit certain businesses altogether.

    One particular moment from my career came to mind. I was a product manager at large storage vendor. We had a assembled a small group of large company CTOs and were telling them about our latest roadmap for storage management automation. We had launched an automation product 3 years prior, and we wanted to assure them that we were committed to continuing our investment (spoiler alert: we were not, in fact, committed to that). So we went through the song and dance about all the great new things we were bringing to the product suite, about how it would solve problems and help our customers be more productive.

    I’ll never forget one exchange with a particular CTO that is forever seared into my memory. He began by carefully choosing his words, mindful of their impact, but he finally said what was really on his mind, and likely for the rest of the group as well: “Will this let me fire some guys?” I was unprepared for this question. We had just spent the last 2 hours talking about increased productivity and efficiency from automation, so he drew what seemed to him to be a very logical conclusion from that. That is, if the product is as efficient and productive as we claimed, then surely he would be able to reduce staff. We hemmed and hawed and finally admitted that, no, we could not guarantee that it would, in his words, let him “fire some guys.” It was as if the air completely left the room. Whatever we said after that didn’t really matter, because it wouldn’t be the magic bullet that let everyone fire a bunch of staff.

    This is a lesson that we keep learning and unlearning, over and over again. Remember cloud? Remember how that spelled the end of sysadmins and half of IT staff? Yeah, they’re still here, but their job titles have changed. Just because you moved things to the cloud doesn’t mean you can be hands off – you still need people to manage things. Remember Uber? None of these gazillion dollar swallowing enterprises or sub-industries of tech have generated anywhere near the original perceived value. And don’t even get me started on crypto, which never had any actual value. Cory’s point is the same: do you really think hospitals are going to fire their radiologists and put all patient screening and lab results in the hands of a machine learning (ahem: advanced pattern recognition) bot? Of course not. And so, a hospital administrator will ask, what’s the point? Do you really believe that hospitals are going to add tens or even hundreds of thousands of dollars to their annual budget to have both bots AND people? Don’t be absurd. They’ll be happy to make use of some free database provided by bots, but the humans in the loop will remain. Cory’s other example was self-driving cars. Do you think taxi or other transportation companies are going to pay both drivers (remote or otherwise) and bots for transit services? Be serious. And yet, that’s the only logical outcome, because there is no universe where humans will be taken out of this very high risk loop.

    The problem is that this is no justification for the billions of dollars being invested in this space. End user companies will happily make use of free tools, keep their humans, and spend as little as possible on tech. That part will not change. So who, then, is going to justify the scope of current investments? No one. That’s why it’s a bubble. Cory’s right. The only thing that remains to be seen is who gets harmed in the aftermath and how badly.

    The intended buyers of this technology are going to ask the same question as that CTO from years ago: will it let me fire some guys? The answer is no. It is always no.

  • This Time It’s Different

    Those of us who have been around the block in the high tech space can point to a number of moments where the hype went way beyond the actual value. The worst example of this was probably crypto and NFTs, which are slot machines built on a casino where the house definitely has the upper hand. The world of AI is the successor to crypto, with one very important difference: the tools that have been lumped under “AI” are actually useful, or potentially useful. But that is also part of the problem: because there are some well-known use cases, there’s a tendency to exaggerate the usefulness of the technology. There’s also a tendency to exaggerate the possibilities of the technology to the point of delusion.

    Let’s start with the first problem: the term itself, “Artificial Intelligence”. It is neither “artificial” nor “intelligent”. What it actually is is advanced pattern recognition and language automation. For that insight, I credit Dr. Emily M. Bender, professor of linguistics and computational linguistics at the University of Washington. Labeling language automation tools as “AI” brings about the worst comparisons to dystopian sci-fi, but it also is, frankly, just wrong. No large language model is remotely sentient. None of the language automation tools are paving the way to Artificial General Intelligence (AGI) – the type of technology that “wakes up” and… makes us breakfast? provides tips on the betterment of humanity? decides humans have had their day and builds skynet? All of these scenarios are a bit silly, and the hype beasts concern trolling over implausible outcomes has become most wearisome.

    While we were distracted by the dystopia vs utopia non-debate, real harms have been perpetrated against real humans with these tools. And with the increasing compute power behind these language models, the degree of potential harm grows with each passing day. Real harms in the form of disinformation, bias, devaluing of creative works, and a growing inability to retract or prevent any of these harms. Add to that the growing body of research that shows LLMs are vulnerable to data poisoning and reverse engineering of its training data and it’s clear that we haven’t quite thought out the ramifications of relying on these tools.

    I’ll wrap up this blog post by (hopefully) stating the obvious: LLMs are obviously here to stay and can already do a number of useful things. I know I look forward to having an LLM fulfill my more mundane, rote tasks. But it’s crucial that we don’t anthropomorphize LLMs and ascribe to them characteristics that are definitely not there, however much we might wish them to be. It’s equally important not to buy into the dystopian doomerism about rogue AI, which is its own form of egregious hype. The more we worry about implausible hypotheticals, the more we risk missing the danger that’s here today. Humans were already good at institutionalizing bias and spreading misinformation. Now, with LLMs, we can do it faster and at a much larger scale. Buckle up!

    My guiding lights on this topic are the amazing people of the DAIR Institute, led by founder Dr. Timnit Gebru. Other influences are Kim Crayton and the aforementioned Dr. Bender. Read them today – don’t believe the hype.

  • Son of a Preacher Man

    1985: Rural Northeastern Arkansas

    When I was 12, I had an… well, I don’t know quite what to call it, but I think of it as an existential crisis. It started as an overwhelming sense of dread whenever we would drive in to our place of work. We ran a crafts business, and I was one of the employees – me, my older brother (17), and our parents. It was just us. We had moved to my mother’s small, rural hometown in northeast Arkansas to launch a business and capitalize on her family’s help in the form of free or cheap housing and office space, not to mention sweat equity partners like my aunt and uncle.

    Anyway, every morning we would make the short drive to the shop, and every morning I would feel a sense of overwhelming dread. A sense of neverending doom and dispair that this is it. This is my life. It’s never going to evolve from this into something better. Such was my mental state that when I somehow heard about Descartes’ “I think, therefore I am” in response to the philosophical question of whether or not we are real or merely living in someone else’s dream, my brain went absolutely wild. I went from an overwhelming sense of gloom to a full-on panic. Every day, I would question whether my world was real or imagined by someone else, and every day I would come to the unsettling conclusion that I didn’t know. An uneasy feeling settled in the pit of my stomach, and it wouldn’t budge. It was at this point that I started to wonder, “Is this what it feels like to go insane?” Cue another panic attack, but now, instead of thinking about the uncertainty of existence, I was dogged by the uncertainty of my sanity. Naturally, I dealth with these issues by… never telling anyone.

    At some point, after some months of mental anguish, I decided that if this is a dream, then I may as well make it a good dream and have fun with it. And that’s how I came to the conclusion that I wasn’t crazy. No crazy person could make such a logical deduction! Looking back, I like to think that it couldn’t have been that bad if I came up with a way to cope with it. But there’s a reason why that particular time period was scary for us and why it led to the summer of panic in 1985 – and also explains why this blog/newsletter is called “son of a preacher man”. 1984-1985 was a period of great uncertainty for us, much of it self-inflicted by my parents, and specifically my father.

    Flashback: Southwestern Missouri and the Ministry

    Before launching our business, we were a church-leading family in the Southern Baptist denomination. We were “in the ministry”, and my father had been a music leader, youth pastor, and associate pastor in Northwestern Arkansas, and then became a head pastor in 1980 in Southwestern Missouri. After being voted out of his first church after his first year (I’ll come back to this episode in a future installment) he and his followers decided they were going to start a new church, Victory Baptist Church. My father developed some rather strident views: The Southern Baptist Convention was “too liberal” and was insufficiently unkind to those in queer communities. He also was not fond of recent trends to ordain women. He was extremely bigoted against blacks and immigrants, as was our mother. As with most southern moms, she held the same views but was highly skilled in hiding it with a veneer of niceness and civility. They would never admit it, but they were functionally segregationists.

    Victory Baptist was where we were free to be us, shedding the official ties to the “liberal” SBC and putting all of our fundamentalist beliefs out in the open: scientific evidence of creation theory, avoiding the path to damnation paved with the gay agenda, abortion is murder, and the end times and the rapture were just around the corner. The rapture scared the shit out of me. I lived much of my childhood convinced that at any given moment, my mom would disappear and I would be left behind. Cue a number of moments where I would desparately try to find my mother out of fear that she had been taken away. Underneath that anxiety was the dual fear that I had been left behind because I didn’t pass muster as a Christian. So we created a small school to avoid the herecy rampant in our government schools and teach our kids the values of homophobic, racist, fundamentalist Christianity.

    Over time, my father grew increasingly frustrated with this ambitious project. I’ve honestly never quite understood why. From a career perspective, he probably felt that he could never achieve greatness as a politico-spiritual leader. One of the themes I’ll return to in this blog is my father’s narcissism – and my grandmother’s. But there was also undiagnosed mental illness and severe bouts with depression. He certainly didn’t have any changes in belief – his core beliefs are the same now, several decades on. Whatever the reason(s), by the summer of 1984, he was done, and he quit, throwing the family into chaos. We were unable to make mortgage payments, and we lost our house by that fall. This precipitated one of the most unsettling incidents I’ve ever witnessed. While we were moving out of our foreclosed house, my father suffered what I believe was a nervous breakdown and blacked out over a period of several hours. He started acting weirdly, eg. while loading a moving truck, placing furniture in a position on the edge where it would certainly fall off onto the road. When someone pointed this out, he shrugged it off and walked away. I’ll never forget walking into our house looking for my parents, and seeing my dad with his head buried in his hands. He kept repeating, “I can’t do it. I can’t do it” with my mom assuring him, “Yes, you can.” By this point, we had frantically begun to look for others to help us with the move move, and thankfully, they showed up to do the thankless job of making sure my dad didn’t place himself or others in harm’s way. When it came time to make the first delivery to our new rental home, a 30-mile drive on rural roads, they convinced him to get on the truck but would not let him drive – he was clearly too incapacitated to trust behind the wheel. He was characterized as “off” and not really present. At some point during this drive, he “work up” and wondered where he was. He had completely blacked out and had no recall of the preceding events.

    What It All Means

    It would be all too easy to look at these events and say, aha, that was some real trauma, and believe that this was the extent of it. But the fact is that our lives in the evangelical community prepped us for a lifetime of trauma and abuse. The irony is that the difficulties I outlined above are part and parcel of a lifetime spent moving from one traumatic moment to the next. The trauma of never knowing if you were good enough to get into heaven. The trauma of believing in a literal hell that awaits you if you don’t measure up. The trauma that stems from a continuous fear of being “left behind” by the better Christians. The trauma of believing we were heading into the “end times” and preparing for the 2nd coming of Christ. The trauma of living in a household with a Father who saw himself as the anointed head of household and head of the church, our “Christian flock”, coupled with the stress and paranoia that stemmed from all of the above. And then, ultimately, how it all fell apart when we could no longer maintain that veneer that we had strived for so long to present to the outside world.

    I tell this story, because, while my personal crisis the following year pales in comparison to my father’s, it shows a direct link between a traumatic period of our family’s life and my inability as a child to process all of the prior trauma. This period of time, during my most formative years, had a profound effect on who I am today. As an adult, looking backwards, I often return to that traumatic time, haunted by its many ramifications: a brother who later came out as gay, whom I would describe as “psychologically broken” by my fundamentalist parents; a father and mother who never evolved emotionally, choosing to remain steadfast in their awfulness; and a strong desire to seek a replacement for the certainty of fundamentalist Christianity, as abusive as it was, which meant I have often been vulnerable to charismatic grifters with good storytelling skills.

    In many ways, our family’s story of the past 40 years is America’s story of the past 40 years, especially evangelical Christian America. Abusive relationships with authoritarian Christian leaders, hateful bigotry, an ambition to purge America of its sinful waywardness, a desire for the freedom to dominate others that we deem to be lesser, and most of all, political striving – it’s all there in our family. I don’t think most Americans truly understand evangelicals and the dangers of their beliefs. In this blog series, I plan to peel back the layers that we wanted everyone to see and show the seedy underbelly of how this culture functions – or, rather, doesn’t. I will lay bare our unapologetic racism. I will expose our suspicion of democratic principles and our cavalier dismantling of them. And I will hopefully show that there is no compromise with those who sincerely believe that they are liberating America from Satan. But I’m not going to create boring academic lectures; I’m going to pull examples from our family’s history to *show* these principles in action, laying bare the subtext and speaking the unspoken. I continue to be disappointed with how most of our media cover the evangelical movement. I hope that by putting a human face on this movement, I can help others to understand this world more fully.

    Also on:

    brid.gy Medium

  • Why is This Site Called Pro-Life?

    You may have noticed the name of this blog and wondered what this is all about. Am I going to scream at you that abortion is murder and stopping the baby killers? No. Well… unless the subject is infant and maternal mortality in the United States, in which case I will tell you that our terrible racist healthcare “system” and lack of reproductive rights does in fact put babies, and their mamas, at risk. The United States leads the industrialized world in infant and maternal mortality, and not in a good way.

    There are a number of reasons why this is the case:

    • Lack of comprehensive health care – the US leads the world in bankruptcies from illness
    • Rampant poverty, especially among younger women of color of childbearing age
    • High rates of unwanted pregnancies (for a number of reasons – will go into detail in a future blog post)
    • Relatively poor health: high rates of diabetes and other chronic debilitating health issues as well as lowest life expectancy of industrialized countries
    • Lack of prenatal care (will address this in the future – know that this is connected to the US’ overall rejection of reproductive rights for women)

    In every point made above, there is a readily available solution. In fact, every other industrialized nation has solved this problem, and it would be relatively easy for the US to address these issues. The irony is that those most opposed to abortion – those with the gall to call themselves “pro-life” – have resisted every opportunity to address any of the above issues. Every. Single. Time. In fact, they are the ones most vehemently opposed to addressing these problems. Sickening, no? Isn’t it odd that those who call themselves “pro-life” are actually ensuring that more women and children die?

    One of the reasons I started this blog and named it “We Are Pro-Life” is because we, those of us who actually care about people in our communities, we are the real pro-life advocates. We are the ones who advocate for trans lives. We are the ones who defend black lives. We are the ones with the core belief that everyone is equal in the eyes of our creator.

    We. Are. Pro. Life.

    Not those other clowns.

  • There is No Open Source Community

    There is No Open Source Community

     

    In January, 2006, I published this article on O’Reilly’s OnLAMP.com site, which was recently shut down. I’ve always been proud of this essay, because I think I got a lot right.  I’m republishing it now in the hopes that it will continue to educate others – and perhaps  allow others to critically evaluate where I fell short in my arguments.  The central thesis is here:

    The commoditization of software and a gradual, long-term reduction in price have played far more important roles than previously recognized. Business strategy designed to leverage open source should focus more on economies of scale (in terms of user and developer bases) and less on pleasing a mythical, monolithic community.

    Basically, stop treating open source as a social movement, because it’s not. This false assumption has caused much harm to software developers and users alike (more on that in a follow-up article). However, while I’m busy patting myself on the back for writing about software commoditization, I missed something fairly big: the value of source code itself is essentially worthless. This may have actually been more important than the price of software.

  • Tales of Privilege

    Here’s my latest on Medium:

    Much has been said recently about privilege and, specifically, white male privilege. How it feeds into the success of many people, especially those who benefit from institutions that privilege whiteness, maleness, and more specifically, maleness that falls within the strict bounds of gender and sexuality norms. It has been said that it’s impossible to separate the role of privilege from one’s success. That they are tightly coupled, and to suggest that one can have success without acknowledging the role of social privilege is highly disingenuous and tantamount to thievery. In other words, check your privilege. This is an attempt to put my story in this context and to show how conventional wisdom doesn’t always apply to individual stories.

    Read the complete post.