- The bulk of Ethereum’s DeFi ecosystem went dark earlier today after a latent bug in the Ethereum code split the network’s transaction history in two.
- The split resulted from a code change that was surreptitiously inserted into a previous Geth update and activated today; some Ethereum node operators ignored the update, which ironically was meant to prevent the very split that occurred.
- The nodes that did not upgrade were under the impression the update was minor and did not know it included a change to Ethereum’s consensus design.
- A post-mortem released today by Geth indicates that a hard fork to push the update was intentionally triggered today. The case is perhaps Ethereum’s greatest challenge since the 2016 DAO fork, and it raises questions about Ethereum’s oft-touted decentralization and the effectiveness of its developer coordination going into Ethereum 2.0.
At first, it was an apparent issue with Infura, the ConsenSys-run servers that keep a majority of decentralized finance (DeFi) applications synced to the Ethereum network.
Infura went down around 8:00 UTC Wednesday, and with it, some of Ethereum’s most popular applications like Metamask, MakerDAO, Uniswap, Compound and MyCrypto, among others. Shortly after, Binance halted Ethereum trading after noticing conflicting transactions on its Ethereum node. As other exchanges suspended trading as well, the real issue became clear: A bug in the Go Ethereum (Geth) client, whose code underpins 80% of Ethereum’s applications, had split the Ethereum blockchain in two.
The two conflicting transaction histories meant Etheruem users were temporarily interacting with different versions of the Ethereum blockchain. More than causing delays, this put user funds at risk by knocking out the majority of Ethereum’s DeFi applications for a few hours.
Infura has fixed the issue, as have other service providers who were affected by the snafu, by updating their nodes. These stakeholders were running an older version of Geth, and the split was caused by an “unannounced hard fork” that was put into a recent update (but which was only just activated) that Infura and Blockchair, among others, ignored.
Besides these two service providers, other Ethereum users and wallet providers were also affected because they didn’t update their code, developers have told CryptoX.
The fiasco has critics challenging Ethereum’s perceived decentralization, while stakeholders are wondering why the hard fork was pushed in secret without coordination between Geth and other development teams.
To some, the split is the most pressing challenge for Ethereum since the infamous DAO hack of 2016.
Ethereum’s chain split: How it happened
In a just-published post-mortem, Péter Szilágyi, a team lead at Ethereum, wrote that a hard fork “was (deliberately) triggered on the Ethereum network.”
A representative from Optimism, an Ethereum scaling project, recently posted that the project was responsible for triggering the hard fork.
A hard fork is an update which is incompatible with earlier versions of a blockchain’s software, so when the hard fork activated today, it created two versions of the Ethereum transaction ledger: one with transactions from updated Geth clients, and one with transactions from older Geth clients (like Infura).
“The fix was deployed several months ago and only today a transaction that caused that split came in,” Nikitia Zhavoronkov, the lead developer at Blockchair, an Ethereum block explorer who was affected by the hard fork, told CryptoX in a direct message.
Thinking the update was “a minor change to the code,” Blockchair didn’t bother with the update because it wouldn’t be worth the downtime for their services. But more than minor, developers apparently made a quiet change to Geth’s consensus mechanism in the update, as well.
“The Geth team indeed changed the consensus implementation in the v1.9.17 release, however the team did not create any new rules that the Ethereum community didn’t know about or agree to,” Szilágyi writes in the post, saying these rules were laid out in an Ethereum Improvement Proposal three years ago.
“If you don’t consider accidentally introducing a bug a ‘consensus upgrade,’ then you should also not consider fixing the said bug a few months later a ‘consensus upgrade’,” he argued.
A call for transparency
Ironically, the hard fork itself was intended to fix the very consensus issue that caused the split.
The Ethereum bounty program recently recognized John Yang, a newcomer to Ethereum’s open-source community, for discovering this and another vulnerability. Geth developer and Ethereum security expert Martin Swende tweeted the changes in the update fix the disclosed issues, intimating that the debacle is a “reminder to keep your node(s) up to date!”
Swende continues to say in the tweet thread that developers did not announce the big change to avoid drawing attention to the flaw. In his own explanation, Szilágyi said that “silently” fixing the bug invited less “disruption.”
Still, other Ethereum stakeholders are wondering why the bug could not have been disclosed in private with the teams that are building on Geth.
“Each major project that the dev team is in close contact with should have a security contact that can help manage and coordinate a smooth upgrade, and we should work together,” Matt Luongo, the founder of Thesis, told CryptoX.
“When hard forks are surprises, anyone who has built atop Ethereum like we have could lose money,” he continued.
Thesis builds the Keep Network, which issues tBTC, a form of tokenized bitcoin for the Ethereum blockchain. Luongo said the fork put tBTC users’ funds at risk, but not because of the chain split, which has been resolved after Infura and others updated their Geth clients.
It’s because the downtime meant that users staking Ethereum in Keep Network couldn’t coordinate with the Ethereum mainchain; as a result, they risked having part of their stakes “slashed” for not meeting their fiduciary requirements.
Despite the problems the split caused, prices for ether, the Ethereum blockchain’s native cryptocurrency, rose 4.6% Wednesday after the news emerged, suggesting that traders see little systemic or long-term threat from the snafu.
Picking up the pieces
Zhavoronkov said the split was no doubt “unexpected” by the Geth developers, who he said would have announced the hard fork (per precedent) if they thought it would cause problems. Luongo shared a similar sentiment, saying that the Geth team are “good developers” but that they lack “experience running infrastructure” and are “underfunded.”
Both Zhavoronkov and Luongo said that they’re going to wait for Geth’s post-mortem before weighing in with a definitive takeaway.
Because the core question – why was the hardfork triggered – has yet to be answered. It may have been accidentally activated, or maybe Geth, assuming users had activated, flipped the switch on the upgrade manually.
The post-mortem will “shed more light on things,” Szilágyi told CryptoX.
Zhavoronkov said that the mess-up was not malicious, but that “if [Geth] knew such thing could happen, they should’ve prepared a guide for node operators.” Luongo shared similar frustrations, saying that the Geth team are “good developers” but that they lack “experience running infrastructure” and are “underfunded.”
The comments key in on a frustration some Ethereum stakeholders share pertaining to why Geth kept the hard fork a secret. Going further, why did Infura, the backbone of Ethereum’s decentralized finance ecosystem, among others, not know about a consensus-breaking bug in Ethereum’s code before a hard fork had been activated to fix it?
“This is a bit of a grey area and requires a case-by-case discussion,” Szilágyi explains in the post. “We all agree that transparency is king and that we should strive as much as possible towards it, but it’s also important to look at all the details before heads start rolling.
“In the case of Ethereum, it takes a lot of time (weeks, months) to get node operators to update even to a scheduled hard fork. Highlighting that a release contains important consensus or DoS fixes always runs the risk of someone trying to beat updaters to the punch line and taking the network down. Security via obscurity is definitely not something to aim for, but delaying a potential attack by enough to get most node operators immune may be worth the temporary ‘hit’ to transparency,” he continued.
Ultimately, Geth’s team believed there was too much risk in disclosing the vulnerability, so they decided that pushing the update surreptitiously invited the least risk.
“We’d argue that it actually did work,” Szilágyi says. Even though the update “took an unexpected turn with yesterday’s network split,” Geth’s team still believes keeping the hard fork hush “was the right call”
As Ethereum approaches its largest upgrade ever in Eth 2.0, the case could be a critical study in client coordination for the Ethereum ecosystem.
“The most important thing here IMO is that the folks who made this call are transparent about the reasoning, own any mistakes, and grow,” Luongo said. “Monero has dealt with [consensus bugs] well in the past, as has Bitcoin and Zcash. There are many examples, and while it’s always tricky to coordinate across an industry, eschewing any sort of coordination is extremely dangerous.
“I hope this fork leads to tighter relationships and rethinking how projects on Ethereum interact with client development.”
