Andy Matuschak argues that closed apps and specialist programming are holding back invention, while code agents could make software more malleable and prototype-friendly.
For years software promised freedom, but delivered silos. Matuschak starts from there, arguing that two choices born almost by inertia—the app model and programming culture—have narrowed what users can imagine and modify. His claim is that code agents are changing the relationship between ideas and implementation again, especially in complex interfaces. The point, he says, is not just to let non-programmers build software, but to put invention itself back into circulation.
Matuschak opens with a simple, uncomfortable claim: personal software promised custom tools, but became a landscape of monoliths and fences. The app model, he argues, pushed makers to build bundles suited to the broadest possible market, and users were left able to touch only the knobs the vendor provided. The result, he says, is that workflows bend poorly to what people actually need.
We sleepwalked into two accidental theories. The first is the application model, and that led to one-size-fits-all packages.
We live in a landscape of walled gardens, and the result is that our workflows often don’t work the way we want.
The next target is programming itself, which Matuschak does not describe as a universal language but as a specialization that has occupied too much space. Instead of becoming mass literacy, he says, it stayed in the hands of a small technical class, and that ended up slowing invention even inside industry and HCI* research. This is not just a complaint about passive users; it is an accusation against the culture that decides who gets to imagine new tools.
We didn’t create a new universal literacy, we created a specialized priesthood.
The familiar version of that complaint is that end users are relegated to the role of passive consumers of the dynamic medium.
Matuschak says code agents are finally making an old HCI ambition practical: letting users extend the software they actually use, without rewriting it from scratch. The target is not marginal features, but the central surfaces of work—the ones that today remain locked inside huge, rigid apps. In his telling, the change is not yet a mass revolution, but the first openings agents create inside the everyday lives of non-programmers.
I want to see the dream of personal dynamic media realized for the most important parts of people’s practice.
Those parts often happen in huge, extremely complex apps.
His measure of the shift is cautious. He says he sees personal dashboards, automations, scripts that connect incompatible systems, but admits that for now agents mainly produce glue code at the margins of real work, not deep extensions of primary interfaces. It is an important transition: the technology that promises to democratize programming seems, for now, better at stitching than reinventing.
For Matuschak, the problem is structural: traditional plugins live in separate panels, while the part of the software that really matters stays untouched. In apps like Photoshop or Premiere, he observes, an extension that touches the main surface would have to recreate almost all of the app’s complexity, and that discourages both developers and automated tools. , he says in effect, because the alternative would be chaos that is hard to govern.
If we want to get out of siloed monoliths, we’ll need a substrate more malleable than today’s zero-to-one code agents.
If plugins could get in and change things, chaos would arrive quickly.
In the demo, the point is not another flashy feature, but the fact that two different extensions can coexist in the same editor without rebuilding everything else from scratch. Matuschak first shows a plugin that anchors citations to the EPUB, then another that syncs audio and transcript, to arrive at a simple thesis: , while composition makes it genuinely usable.
Right now it’s still just a Markdown document. But from here on it’s plain text, so I can extend my comments, move everything into the document, copy it, paste it, treat it as a medium.
Here I have a new idea, a different kind of Markdown directive that I called a transcript directive. I can press play and get audio playback synchronized with the text transcript.
The sequence serves a precise demonstration. If a new behavior can be inserted, split apart, copied, and even combined with another behavior without losing the rest of the editor, then the plugin is not decoration but infrastructure. That is where the argument shifts from individual tricks to the architecture that makes them possible.
What’s exciting about this demo is not the interaction design of those plugins. What’s exciting is that this isn’t a research system: I brought it into Obsidian, a production word processor that I actually use.
Here Matuschak insists on the boundary between lab and everyday use. An interesting interface, he implies, is poor if it lives only in a paper, because the user still has to leave their own environment and pay the cost of separate software. Obsidian thus becomes proof that useful invention is not just what works, but what enters the normal flow of work.
At Matuschak shifts from apps to reading. If writing systems can be extended by plugins that act on the same text and the same states, he says, then the real question is why reading is still confined to PDFs and static pages, as if cognitive support had to end at the margin. His move is to generalize the logic of CodeMirror to a reader that does not merely display content, but makes it combinable, annotatable, alive.
The issue, really, is worse than needing extensible tools for professionals. Serious readers don’t have professional tools in the first place.
Our reading environments have evolved very little. And yet there is so much room for them to evolve.
Here his argument changes scale. Matuschak insists that the reading medium should do for understanding what Figma does for layout or a mixer does for sound: absorb part of the mental load and make the effects of choices visible in near real time. Today, he argues, people still work on paper images on a screen, while the text remains inert and the reader carries all the weight.
When I iterate on a design system in Figma, I can see in real time how my choices affect layout and weight on every screen of the app I’m designing. When I mix a track, the spectrum visualizers help me quickly identify and correct muddy sections.
To show he is not speaking abstractly, Matuschak lists a small genealogy of augmented readers already in existence: Skim, SiteSee, Papio, Liquid Text, Space Ink, Quantum Country*. But the point is not to celebrate each project individually. It is that they all remain segregated within their own systems, forcing anyone who wants to use them to switch readers every time, just like the first plugin demos remained trapped in a lab prototype.
If you want SiteSee’s colored schema, you have to use its prototype as your PDF reader. And then, if you want features from each of the other systems I just mentioned, you’d have to switch readers completely for each one.
For Matuschak, the problem is no longer just writing code faster. The real bottleneck is that programming has occupied too much cultural space within interface invention, leaving design and domain knowledge in the background. So, he argues, even when the field celebrates creativity, it often ends up producing cautious variations on already familiar forms.
I think the central problem is that inventing in the dynamic medium requires too much programming.
If new interfaces are blocked mostly by imaginative design work and deep domain knowledge, then this asymmetry puts selective pressure on the wrong skills.
Matuschak invokes Alan Cooper and *The Inmates Are Running the Asylum*, the book that in the 1990s accused programmers of imposing ugly and hostile interfaces. He says that criticism already produced a useful correction, because specialization made room for interaction design and interface patterns spread. But for him, the old problem was only the first layer.
Cooper was worried about an accidental tyranny of programmers subjecting people to comically bad user interfaces. I’m worried about an accidental tyranny of programmers holding imagination back in user interfaces.
The decisive shift, in his reading, is cultural: almost every environment that talks about interfaces ends up populated by programmers, trained more in analysis than in design or domain work. Matuschak insists this creates a distorted selection, because non-programmers remain dependent on others precisely when they should be iterating, failing, and actually using the prototype. At that point, he says, work drifts toward what is easiest to do alone—often visual design—while the truly dynamic part remains unfinished.
Programming without design or domain knowledge can produce working interfaces, even if they’re often boring or flawed, while design or domain knowledge without programming gets stuck on the drawing board.
Matuschak closes by shifting the weight from code to the mental cost of spending hours on it, without backing away from the bet. Agents, he says, can fragment attention and make it easy to fall into a kind of productive agitation that looks too much like an escape from slow thinking; and yet they remain, for him, the best way to get more ideas into the product pipeline early.
The costs are real, but we’re bearing them for now because code agents are good news for invention.
I want to increase the imaginative flow at the top of the funnel. That early phase puts less pressure on software quality.
His distinction is sharp: a prototype needs to be only real enough for designers and colleagues to feel how an idea behaves in context. That is where he cites the value of high-fidelity prototypes for dynamic media, contrasted with static artboards and clickable demos that say little about how an interface really behaves.
A prototype must be real enough to let the designer understand how their idea develops in an authentic context and communicate it clearly to others.
A high-fidelity prototype can clearly specify a new dynamic behavior.
What is Matuschak’s central thesis?
His thesis is that modern software has created two blocks: apps that are too closed and programming that is too specialized. Code agents, he argues, can lower the barrier to inventing and customizing interfaces.
Why does he mention Obsidian and CodeMirror?
He cites them as proof that a UI can be built compositionally, so plugins do not remain confined to the margins. For him, that is the kind of architecture that lets agents make a real difference.
Why does he talk about reading and not just editing?
Because he wants to show that the problem is not only writing, but also reading and annotating in more dynamic environments. His idea is that reading, too, can become an extensible medium like an editor.
According to him, what is the real bottleneck?
It is not just the technical quality of prototypes, but the fact that invention requires design, domain knowledge, and programming all at once. He says the system today too often selects technical talent and too little imagination.
What is the critique of traditional product teams?
Matuschak argues that non-programmers often depend too much on programmers to iterate on an idea. That slows discovery, because it prevents people from working early and often enough with a living prototype.
The Inmates Are Running the Asylum (1999)
Matuschak cites the book to contrast his thesis with Cooper’s: he is not only worried about bad interfaces, but about programming slowing down invention.
AI-assisted summary of Andy Matuschak's podcast, verified against the original transcript.