Something is rotten in the state of Denmark Google. That much is obvious.

Looking at the Hacker News comments, the consensus is that someone should make a better search engine. More specifically, a search engine that “doesn’t include those spammy blogs“ or that “searches for the exact tokens in my query, without all this machine learning nonsense.” While those suggestions may yield a better search engine, it is not obvious that is what the world needs anymore.

This post makes the case that the tool that we need to navigate the information landscape of today will not look like a search bar and ten blue links.

Google took its current form because of the nature of the World Wide Web.

The Web is a network of content (i.e. webpages) authored by humans, whether they were amateurs, journalists, or experts. These creators interacted in loose and implicit communities by linking to each other’s pages. Google’s claim to fame was using this hyperlink structure to derive the relative importance of web pages. Today, PageRank is outweighed by other signals, but at its core, Google still relies on this network structure to build its index.

Google’s other claim to fame – the one that has made it the most profitable machine in human history – is as an economic engine for the creation of content on the Web. It aggregates user intention and attention and directs it towards highly-ranked websites. This model makes it profitable for website owners to create content in all sorts of niches by converting attention into ad revenue. Conveniently enough, Google has some tools to help with that.

If you strip away the implementation details of its UI and ranking algorithms, Google’s purpose is to create a map of a content ecosystem and provide the engine that fuels the expansion of that ecosystem. Or, to use Google’s own words, to organize the world’s information and make it universally accessible and useful.

There’s just one problem for Google and any new entrants to the search engine game: the Web is dead.

Just as demand for cheap fat sources led to the razing of rainforests to grow palm oil plantations, the demand for free content has turned the once lush Web into a spammy, SEO-ridden monoculture. No amount of virtue signaling about RSS or reminiscing about dialup-core aesthetics can bring back the Web that once was. After years of poor stewardship and exploitation, this ecosystem may be past the point of no return.

While we should mourn the loss of this common good, there is a reason for hope. We are in a renaissance of the creation of quality, informational content. It just looks different than what we’re used to.

When you think of new content formats, many think of entertainment content in walled gardens such as Instagram and TikTok. But there is also a boom in the publishing of open and semi-open formats such as podcasts, Twitter threads, newsletters, and forum discussions. While not all are purely informational, the knowledge found in these new media is often unavailable on the Web.

The people who create content are increasingly defecting from the ad-supported model. Through platforms like Patreon, Substack, OnlyFans, and even NFTs, content creators are monetizing more of the demand curve. This new revenue model introduces new challenges in gaining and retaining an audience of true fans.

The past two decades were undoubtedly the Age of the Algorithm. Google and Facebook became exceedingly effective at focusing the public’s attention in ways that were profitable for them, at the expense of nearly everyone else.

But in lieu of algorithms, information is increasingly percolated through identity-based reputation networks (e.g. Twitter) or collectively-moderated community resources (e.g. Reddit). And cryptocurrencies introduce new tools to solve problems of reputation, trust, and incentivized curation of content.

The new landscape is replete with activity and high-quality information, but it is also fragmented and illegible. This is a new ecosystem just waiting to be mapped, cultivated, and nurtured. Sound familiar?

It’s important to keep in mind that this new information engine will not feel like this:

but more like this:

Which is to say, in classic disruptor fashion, it will be worse at everything except the few things that matter.

There is no point trying to make a new information tool that tries to compete for questions like [mcdonald’s hours] or [how to stop nosebleed]. Google is perfectly fine at those and it's unlikely that the answers would be found in a Substack newsletter.

A new information engine might instead be built using this playbook:

If you look at a map of the Twitter graph, it is organically organized into pockets of activity. Instead of domains and URLs and pages, a subculture contains clusters of people, their identities, and their work across various platforms. By indexing this territory, an information engine can allow an informal network to become more densely connected and discoverable.

It is astonishing how bad the current tools are at mapping these Internet subcultures. For example, if I wanted to know who is involved in DAOs today and what they have to say about them, neither Twitter nor Google will satisfy me.

This does not necessarily mean throwing a search bar on top of whatever corpus. Some form of content would lend themselves to that type of organization: it would be cool to search over podcast transcripts, newsletter archives, and Reddit threads. But other kinds of content might benefit from being organized into directories or ordered lists.

This also means that algorithms are not the source of truth for making sense of this data. By giving visibility to their subculture, the community might in turn help curate and maintain the index of their own content.

An information aggregator grows its dominance by growing the pie and rewarding its stakeholders. Google, at least in theory, did that for the Web. At this point, the “creator economy” is already carved up by the likes of Patreon and Substack. But creators are not the only stakeholders in the system anymore. Can curators and moderators be incentivized to maintain the quality of the information on the platform? Can consumers and fans fund requests for the content they’d like to exist?

The one thing I’m not sure about is whether this template will result in many hyper-focused information engines or one big one. Focusing on a particular subculture at the beginning does not preclude broader adoption later on. Reddit started as a news aggregator for techies but turned into the front page of the Internet. Discord started out with gamers but has since transcended that demographic. It may happen here too.

Does this vision of the new information ecosystem make sense to you? Let us know what you think. We’d love to hear your thoughts, skepticism, and alternative approaches to this problem. Better yet, come map this new world with us.

We’re [email protected].