From medium by Thomas Jay Rush

As it was, as it could have been, as it may be in the future

I first heard of Ethereum in late 2015. A few weeks later, I heard about the Mist Browser [1, 2].

What a nightmare.

Luckily, by then, I was so far down the rabbit hole that I was willing to endure any pain to use a “programmable blockchain.” I had read about BitCoin a year earlier and the discussions about “abstracting out programmability,” so Ethereum made perfect sense.

What didn’t make sense was why I had to go through so much pain to use it.

The proverbial “we are so early” came to mind. But — I mean — just how early could we have been?

It was like entering the Garden of Eden only to be accosted with signs saying, “Pardon our appearance during construction.” I’m like, “God, what is going on here? Your garden sucks.”

Mist on Ethereum (the gate to Ethereum’s Garden of Eden) was “too early”

What could Mist have done better? Is it too late to resurrect Mist at this point?

What Was, What Could Have Been, What May Be

This article shows three sequence diagrams and short descriptions of each. The diagrams are called “Mist As It Was,” “Mist As It Should Have Been,” and “Mist II: Revenge of the Nerds.”

We discuss two main problems that (among others) ultimately killed Mist. In our opinion, the death of Mist foreshadowed the death of building truly decentralized and truly private applications. But it’s not too late.

What Were The Two Problems?

We think Mist had at least two problems that doomed it from the start:

Mist would start Ethereum node software (Geth) during its initialization. This was an understandable but ultimately lousy choice. The end user had to wait to use the app until the node software finished syncing. This delay worsened as the blocks became more and more complicated.
Mist relied on the node’s RPC endpoint to access the blockchain data. The RPC was inadequate to serve applications because of a simple fact: it’s not a database. As I’ve been saying for six years, “The RPC sucks.” It cannot adequately serve “real” applications. The RPC indexes log topics, but that’s not what it should index. It should index what we call address appearances. (This problem is fully explained in the Unchained Index Specification.)

There were other problems, such as security issues and a lack of suitable alternatives to Electron for building desktop applications, but the above two are the main user-facing problems. (We’ve used “Wails,” an effective and easy-to-use alternative to Electron, to build TrueBlocks Browse. Check it out.)

In the remainder of this article, we describe three sequence diagrams showing how Mist worked, how it could have worked, and how it might work if reimagined.

Mist As It Was

Our first diagram shows a stylized (perhaps slightly inaccurate — it’s been a while) view of how Mist worked. See the numbered notes below, which correspond to the black circles in the image.

The user starts Mist locally on their machine.
Internally, Mist starts up a local node (Geth). Because the node has not been continually running, it must catch up to the chain’s head. Depending on how long it's been, it may have taken a few minutes to a few hours to catch up.
Because the node takes time to sync, Mist has to wait to “see” the latest block and/or interact with any smart contracts.
Once the node is synced, the next problem manifests. However, this problem was subtle and “inside the system,” so it was hard to see. The problem can be summarized thus: “The RPC is not a database.” The RPC did not (and still does not) act as a proper database. It doesn’t have an index (other than log topics, which is inadequate, in our opinion). Evidence of this inadequacy is everywhere: (a) one must use a block explorer to view one’s personal transactional history (not permissionless, not private). (b) one must use Dune or “for-pay” analytics to understand what’s happening (not uncensorable, rate limited, slow, costly). Privacy is non-existent (if you use a remote service, they know who you are). Nearly every dApp you use today is either anemic or a Web 2.0 privacy-invading nightmare. Compare any dApp you use to a “real” application like Microsoft Word or Excel. dApps are inconsequential or simply web 2.0 websites. This is because the RPC does not have a proper index.
When the user closes Mist, the Node shuts down; therefore, the head of the chain is not maintained.

Back in 2016 or 2017, there were reasons why Mist behaved this way (the fear was that the node would overwhelm the user’s machine), but we think this is no longer true. We know this because we’ve seen an Erigon node running on our own laptop. You don’t believe me? Here’s proof.

The most pressing reasons Mist failed were because of its extremely slow startup time and its anemic data access. The node must be continually synced, and the RPC must be an actual database.

Mist As It Could Have Been

One simple thing that could have been done with Mist would have been to install the Geth Node as a service. Other desktop apps (such as Docker or 1-Password) do this.

I call this type of application a “long-running process” because it starts when the machine boots and runs until it shuts down. Others call it a “daemon” or “service.” This is how software that keeps up with continually incoming data must work.

The following sequence diagram shows how Mist might have worked and why it would have solved the first of these two problems. Notice the black circles in the image below. They correspond to the numbered notes following.

In this imagined version of Mist, the Node is started when the machine boots. In this way, the Node stays in sync. This fixes the issue with the original Mist having to wait to begin until the node syncs up.
The node stays continually up to date with the head of the chain. In 2016 and 2017, this may have been a problem, but with the recent advent of chips like the Apple M4 (and a large hard drive — ours has 8TB), it is no longer a concern. While we no longer do so, we’ve had an Erigon node running with no ill side effects on our M1 laptop in the past. You don’t believe me? Here’s proof.
If the node had been running, Mist would have started immediately—no waiting — like every other application you’ve ever used.
Mist would have still suffered from the “RPC is not a database” issue. Each application (assuming people had built more applications than Mist) would have had to extract the chain data it was interested in, cache the results of past extractions, and somehow deliver the data quickly to the app. In other words, every application must have implemented a scraper/indexer/monitor. Anticipating the image below, the need for a better indexing/caching layer inside the node is apparent.
In this imagined version, when Mist is closed, the node stays running; therefore, the node stays continually up to date.

Mist II — Revenge of the Nerds

Why was Mist so important? Why did running local first software directly against a peer-to-peer network of data matter? What’s the big deal?

Direct, unfettered, permissionless access to the blockchain data is vital for many reasons: speed, depth of detail, 18-decimal place accuracy, no “provider” and therefore no censorship, perfect privacy, and near-zero cost. These things are at the core of Ethereum’s ethos. None of these things are available in the current system. Accessing data through Web 2.0 data providers (including remote access to RPCs) destroys every one of these things we say we value so much.

Another critical aspect (at least for me) is that decentralized blockchain networks were supposed to “yank” power from the powerful forces of the Web 2.0 ecosystem. (Such forces will never give the power back to us because we ask them kindly — we have to “yank” it back permissionlessly without asking). This is what the nerds who built BitCoin did. They yanked the power away from the powerful.

This is not how Ethereum works today.

You‘re welcome to lie to yourself if you wish, but the system we’re building is not permissionless. The system is not decentralized if people are not running local nodes and using permissionless local applications. The system is not fast. It’s not cheap (it may be free, but you pay through privacy invasion). Nor — to my mind — is it interesting.

This is why I call for resurrecting Mist. It’s “The Revenge of the Nerds.” We must “yank” the power back from the powerful before they capture it again.

How Mist (i.e., dApps) Might Work in the Future

The final diagram presents just one vision of how Mist (or any decentralized desktop application) might work.

We introduce a new component called Khedra (pronounced “kay-drah”), which is a standalone “daemon,” but it could be incorporated into the node software as an optional daemonic component.

Khedra (here’s the history behind its name) solves the second of the two problems mentioned above. Khedra turns the node into a proper database. It extends the node by providing an indexing, caching, and monitoring layer that runs right next to the node software.

The Unchained Index and Khedra are very well explained in many places [1, 2, 3]. We won’t go too deeply into it here.

Our Final Image

For this final image, the numbered notes below correspond to the black circles in this image.

A Mythical “Garden of Eden” of How dApps Could Work if We Wanted Them To

Both the Ethereum node and Khedra are started when the machine boots. They act as long-running daemons, continually syncing the chain (in the case of the node) and indexing and monitoring for data (in the case of Khedra). Both processes keep their respective databases up to date.
A quick note on the distinction between indexing and monitoring: “indexing” means building and sharing the Unchained Index to IPFS and the smart contract. This is described in great detail here. “Monitoring” means that the end user — you — provides a list of addresses to “watch.” Your addresses. The smart contracts that you’re interested in. This keeps the caches minimal. This is why our system still works (and can be super detailed and, therefore, super accurate) on laptops. Khedra does not extract the entire blockchain, unlike block explorers — it extracts only those addresses you tell it to extract. And…those address histories are continually cached, making them super fast when you query them and available to your dApps.
When Mist (or any local desktop dApp) starts, there’s no need to wait — identical to every application you’ve ever used (other than Mist I). Obviously.
With Khedra and the TrueBlocks SDKs, your local-first desktop apps can treat the node as an actual database indexed by Ethereum addresses. Khedra (and the SDK) provide blindly fast transactional histories for any address (if the address is monitored). Other available data includes balance histories by block or period, reconciled bank statements by block or period, more than 35,000 named addresses, ABI files (downloaded from trusted sources), the Unchained Index manifest, index chunks, and Bloom filters. Khedra turns the previously anemic node RPC into a fully functional local-first application server database.
Your application can still access the RPC to complete any task your application completed previously, including interacting with the chain to send transactions, sign messages, or do anything else. No change here.
When your application closes, the node and Khedra continue to operate and maintain the chain's index and head.

Conclusion

Mist’s fundamental assumption that a decentralized local application must run a local node was exactly correct. Without a local node, your application cannot possibly be decentralized.

Mist’s implementation failed because the Garden of Eden was under construction. The node must be run as a long-running process. The data on the node must be indexed — minimally so it continues to run on small machines. Only addresses of direct interest to the end user should be monitored and cached.

It’s time to revisit our desire to live in an earthly paradise. Do you want to live in Eden? I do.

If we want it, we can have it —all we have to do is build it that way — yank it that way — on purpose.

Your Support is Welcome

TrueBlocks is funded from personal funds and grants from The Ethereum Foundation (2018, 2022, 2024), Optimism Retro PGF (2022, 2023), Moloch DAO (2021), Filecoin/IPFS (2021), Consensys (2019), and our lovely GitCoin donors.

If you like this article and wish to support our work, please donate to our GitCoin grant using ETH or any token. If you’re an Optimism badge holder, vote for our project in future Retro rounds. Or send us a tip directly at trueblocks.eth or 0xf503017d7baf7fbc0fff7492b751025c6a78179b.

Appendix

I’ve had conversations with many people about the idea of resurrecting Mist. These conversations are sketchy, infrequent, and poorly documented. We invite you to join us. Below (and above) are some…very rough…poorly edited…incoherent thoughts. It’s a start, and for better or worse, you go to start somewhere.

In the following, we assume there’s a two-pronged approach — one develops “toward the node” which means it tries to improve or extend the RPC. The other proceeds towards the user, which means one must encourage users, developers, and the community to think more deeply about ideas like Mist.

Toward the node

Indexing is missing from the Ethereum node software — it should have been included in the RPC from the start.
TrueBlocks, through the Unchained Index, fixes the “missing index” problem by adding chifra scrape, chifra export, chifra list and other tools to the mix,
The Unchained Index is fully specified, robust, and fast and includes a detailed description of how to build an indexer/scraper.
Khedra is a stand-alone GoLang package that implements, among other things, a daemon to build the Unchained Index, monitor addresses, and other things.
The Unchained Index is well-specified enough to be easily incorporated into the node software (depending on how you define “easily”). There is both an existing implementation and deep documentation.
Khedra already works as a stand-alone command line application on your desktop. Using your system’s existing systems for running services or daemons turns it into a long-running process.
Khedra can easily be made into a docker image that could run alongside a node. The docker version can be used to make a dAppNode package.
Data sharding and sharing are purposefully designed into Khedra and the Unchained Index. Sharing happens simply through usage.
The Unchained Index is naturally sharded…but…the user doesn’thave to know this.

Toward the user

The ultimate ulterior motive of our work is to get more people to run local-first desktop applications.
Users are attracted by:■ convenience — local-first dApps must be easy to install, run and maintain,■ speed —local-first apps can be 100x faster than web applications,■ accuracy — this speed allows for deeper data extraction, which allows for 18-decimal reconciliations every 12 seconds,■ privacy — no third-party APIs to snoop on your wallets.
Users don’t appear if developers don’t develop:■ upshot — the path to users is through developers,■ solution — improving the RPC’s capabilities and UX (through the Unchained Index and TrueBlocks-Core — a well-documented SDK) increases the chance of new apps being built.
Developers won’t build if there are no easy-to-follow examples:■ trueblocks-browse — is an example of a local-first dApp using our SDK, Khedra, and the Unchained Index.■ examples and tutorials — including video tutorials are available,■ Auto code generation aids in producing robust, well-documented code
Auto code generation allows:■ consistent, well-documented code throughout the project,■ creating SDKs in other languages (we already have experimental Python and Typescript SDKs).

Collaboration

Collaboration with others is a prerequisite. Promote participation through:■ producing a “wish list” of potential dApps (Dao manager, Staking Manager, etc.)■ directly funding of “local-first” developers■ provide grants to “local-first” developers■ sponsor hackathon prizes
Potential partners:■ EF (Buidl-Guidl, Portal Network, Sourcify)■ Rotki■ Rick Dudley■ Pocket Network■ Otterscan — Willian■ Node software teams: Erigon (GoLang), Geth (GoLang), Reth (extensible)
Potential L2 partners:■ layer twos need indexing as much as (if not more) than layer ones,■ any such indexing should be standardized if there’s any hope of a truly decentralized future.
Other potential partners:■ Uniswap — Ken Ng■ Optimism (Test-In-Production)■ Other friends: Giveth, Gitcoin, Hopr

/fin