Announcing DeFi Audits & The Holistic bZx Post-Mortem)
DeFi Audits (due diligence beyond static Solidity code) & bZx Post-Mortem (explaining what happened in detail and what we can learn from it)
This week has been very eventful in DeFi land with the bZx flash loans. I wanted to take the time in this edition to go deep about the events that unfolded for everyone and to announce the birth of DeFi Audits!
If you've been following how the arb happened, feel free to scroll through about 40% of this. Otherwise, enjoy the simple pictures to understand what went down so you can know it well enough to explain it to your friends!
About 2 months ago, the DeFi community became aware that flash loans were a thing. The idea around them was simple, borrow as much money as you want, provided you can return it back in the same transaction. If you don't, then the transaction will fail to actually finish executing
Why would anyone implement this? Well because it doesn't cost anything and the risk is mitigated by the fact borrowers HAVE to pay it back by virtue of code constraints.
Now what someone realised is that you can now have the power of a whale without actually being a whale! One general problem with markets with low liquidity (amount of money) is that any large movements of money can artificially tank or rise the price . Flash loans + illiquid markets are very potent combo as we learn with bZx - a lending protocol launched a few weeks ago.
Here's a simplified diagram of what happened with bZx, I'll explain each step in greater detail below.
Mallet goes to dYdX to acquire 10,000 ETH with no collateral. The only guarantee is the fact the loan has to be repaid by the end of the transaction.
Next, Mallet goes to Compound (another lending/borrowing protocol) and borrows 112 wBTC (a form of BTC on Ethereum).
His next action is to open a wBTC short position on bZx. This means he's betting on the price of wBTC going down. An important step to note: bZx uses Kyber Network (an on-chain decentralised exchange) to get the price of wBTC.
Mallet then sells his 112 wBTC acquired from Compound on Uniswap (another on-chain decentralised exchange).
Kyber uses Uniswap as a liquidity source if it doesn't have any wBTC itself. This causes the price of wBTC on Kyber via Uniswap which actually tanks the price on bZx where the attacker opened a short position.
Since the price of wBTC has "gone down" or been manipulated to drop, the attacker can successfully close his/her short position and profit a clean $350,000!
To make sure that all of these steps succeeds, he also pays back his 10,000 ETH loan he acquired at the start.
All of the above happened in one transaction! As soon as it did, bZx did the following:
Paused their smart contract using their admin key (to prevent future losses)
Told users funds are safe
Investigated the issue and released an official post-mortem here: https://bzx.network/blog/postmortem-ethdenver
During this time I decided to a small audit of bZx's admin controls and code review to see whether there were any other dangers lurking around. Unfortunately I came to a few shocking conclusions:
Their contracts can be unilaterally changed to anything for a certain time period (take funds, change logic etc)
The admin key has been used thousands of time to make changes
They don't rely on any automated testing to write code which is an extremely important practice in writing financial software with large amounts of money. In addition, they've only done one audit since launch but changed the protocol many times after that audit.
For more information, you can read more here: https://defiweekly.substack.com/p/how-decentralised-is-bzx
bZx had already lost a lot of the community's trust, but also their user's trust. Regardless, given the sophistication of the attack - many were able to excuse them since slip-ups can happen and everyone should get a second chance to rectify their mistakes. Many, including myself, had assumed things would be okay and as a result many lenders still kept their money in bZx. Unfortunately it took less than 6 hours after they unpaused the protocol to get exploited again!
This time someone made off with 2,400 ETH or $650,000 USD. What happened? Basically the same thing as the last time, with just 2-3 steps changed. Explained below is how the second attack happened:
Our fictional character, Chad, decided to get a 7,500 ETH flash loan. Once again with no collateral, just a few cents worth of Ether. The best bit is that the flash loan was from bZx since they decided to launch a new feature while releasing a critical patch in a short period of time (with no audits to vet their code).
Chad decides for this first move he's going to purchase $900,000 synthetic US Dollars (generated from the synthetic protocol Synthetix) from his ETH flash loan
Using the sUSD purchased, he opened a long position using the sUSD as collateral. In simple english, he's borrow plenty of ETH because he thinks the price will go up. The way he's able to borrow this is by using the US Dollars as deposit.
Since bZx didn't really think to remove Kyber as an oracle source, Chad decided that he'd make sure they did after his next moves.
Kyber (the on-chain decentralised exchange) has two reserves it sells assets from, it's native reserve and then Uniswap as a backup. What Chad did was slowly drain Kyber's sUSD reserve by selling 20 ETH 18 times. This meant that the next sUSD purchase would have to come from Uniswap.
Here's Chad's masterpiece move: he buys 900 ETH worth of sUSD to drive up the price significantly (1 sUSD is now worth $2). This is because Uniswap didn't have much sUSD to begin with and exponentially increases the price if someone buys majority the liquidity in the pool.
Going back to step 3, Chad's collateral is now worth $1.8m by the nature of sUSD being priced $2 on Uniswap! As a result he's able to p 6796 ETH from bZx since the protocol thinks that his deposit is worth more now.
Using the newly acquired 6796 ETH + his unused flash loan funds, he repays his 7,500 ETH flash loan and profits 2,400 ETH. His long position is left in debt with collateral the protocol actually doesn't have. This is the loss that the lenders remaining in bZx have to take on.
While the team still claims "funds are safe", it's hard to see how that statement holds true as the ETH in the system does not have collateral equal to the value of the ETH left inside. bZx claims they will gradually pay back lenders but for those who got out last, their funds are probably gone unless bZx pays the losses fully out of pocket.
Looking back, it's hard to wonder if any of this was avoidable to begin with. The answer is yes, it was. Taylor Monahan from MyCrypto highlights all of the 6 prior incidents bZx had to correct and remove attack vectors in their protocol.
Rather than spending the necessary time to address the core issue, repeated bandaid solutions were used. SamCZSun, a prominent crypto white-hacker revealed to bZx that flash loans could be used to manipulate price oracles and cause serious damage. Rather than looking into how to not to use on-chain price oracles, the team decided to do a simple spread check which didn't address the core problem of prices being easily manipulated. The following post was written on 30th September 2019 and shows this exact attack:
By relying on an on-chain decentralized price oracle without validating the rates returned, DDEX and bZx were susceptible to atomic price manipulation. This would have resulted in the loss of liquid ETH in the ETH/DAI market for DDEX, and loss of all liquid funds in bZx. Fortunately, no funds were actually lost.
Furthermore, the second attack could have been clearly avoidable should there have been a proper audit done and time given to the community to expose any more potential exploits.
So what's next?
Firstly, DeFi is the world's largest bug bounty, that's open 24x7 and can never be fixed until it's too late due to the immutable nature of on-chain code. Code is law in this new realm. Many DeFi teams are now aggressively working to make sure they aren't vulnerable to hackers with access to millions of dollars to cause more trouble. The most notable example is MakerDAO where all the collateral can be stolen should 80k MKR vote to do so. There was 15k MRK available publicly until a large whale withdrew that liquidity on Uniswap.
While it would be easier to simply be an idle commenter and tell people what they should do, I thought that instituting change would be a better idea.
That's why I want to announce a new audit: DeFi Audits.
So what is a DeFi Audit? Well, it goes through the following key points:
Code quality: Trusting teams have implemented best practices for developing high quality, tested software should not be taken for taken for granted. Detailed due diligence needs be done at regular intervals.
Ownership structure & upgrade logic: Knowing the ownership structure of the protocol is much more important than just saying "oh look it has an admin key"! Most of the time the admin is a contract which has its own logic. Furthermore analysis around how often the admin key is used should be done including how easy is to audit what the changes actually are.
Liquidity Analysis: On-going monitoring around the attack vector of flash loan liquidity compared to assets used by platform liquidity. This is something that you can kind of gauge at the moment but isn't really easy to continually check. Teams should think about what safe-guards they're taking to ensure bootstrapping liquidity slowly isn't vulnerable to flash loan attacks. Governance tokens beware.
Oracle Analysis: If you're going to use on-chain oracles, don't just assume a large whale won't manipulate them, actually do the math! I think before it's been too easy to dismiss the fact that a large whale wouldn't risk their capital. Now anyone can become a whale without the capital!
Bootstrap Insurance Liquidity: With the advent of new insurance protocols such as Nexus and Opyn, taking the other side of the insurance is a great way to signal confidence to your users that you're willing to pay out losses in the case that they want insurance. It probably won't even cost that much considering a static solidity audit runs $50k - $200k USD and most people won't purchase insurance. Adding to this, just lend out funds in a lending protocol earning interest while you have your ETH/DAI parked inside.
Liquidity Caps: As much as everyone wants to ship fast, rushing the deployment process and not capping your down-side can be a hazard. If you're going to cut corners, ensure that the total amount lost is on a much smaller scale. MakerDAO is a great example of this by having the cap on total amount of DAI being able to be minted. Lending protocols might want to think about adding a cap similar to this as well.
So how do we make DeFi Audits a reality? Well, firstly - I'd love your help if you're able to help with gathering any of the information above to compile. Simply drop me a DM at https://twitter.com/kermankohli.
I have no doubt that teams like bZx do have the best intentions, but it's also up to us as a community to help them where they might not be as strong. This is un-chartered territory and regulation won't help due to the nature of the technology.
Like most things in life, everything is a spectrum. No one should assign the label of "DeFi or not" but rather drive people to ask "what characteristics of DeFi does this exemplify"?
I hope that this effort educates a wider audience about what's really going and the risks involved with DeFi protocols.
Have a protocol in particular you’d like to see get a DeFi Audit first? Reply to this email!