Hegic vs Trail of Bits & The Issue with "Audits"
A recap into the Hegic vs Trail of Bits fiasco but also a critical examination around the practice of smart contract audits in the space
Either this title is self-explanatory or it's not. In case it's not feel free to read below, otherwise scroll down!
Last week I started doing the usual crypto twitter scroll until I came across what appeared to be drama with Hegic having a "typo" in their code and something about a vulnerability. Being the curious person I am, I decided to dig in.
Seems like a bit more than a typo huh? Well I decided to scroll to see what the commenters were saying. Until I came across this comment which made me go "oh yeah, what happened to the tests?"
I decided to jump into the Hegic repo and check out what was going on here. The crazy bit here is that the only thing inside this repo was a bunch of solidity files. No automated tests (standard software development practice, especially for contracts), no linting, deployments scripts or other basics. Just basic contracts, sitting in the wild.
What's inside these files anyways? Well essentially, just some Solidity code - but poorly formatted. For those of you that aren't technical, reading this file is kind of like reading English but you can tell that it's someone who has just learned to write the basics. Same thing here but for software engineering. I wouldn't say all poorly formatted code == bad code but more so it's an indicator of the person writing the code.
Conclusion at this point at time: what a terrible developer who didn't know how to write code and put user's funds in danger. If only he/she got an audit... Turns out Hegic did by none other than Trail of Bits!
Enter Trail of Bits
Trail of Bits decided to go back on the offence with the CEO writing a full tweet storm about the whole incident. The quick and short summary is:
Trail of Bits found the vulnerability and advised holding off the deployment
Trail of Bits said they did a "3-day" review rather than a full audit and says Hegic mis-represented their review
They concluded in the report that code "might have more bugs"
Okay so I guess this leaves in a situation where:
Hegic didn't follow proper development guidelines and cut security in order to get to market quick (not the first time this has happened in DeFi)
Trail of Bits had done a "3-day code review" and had found issues but stands firm that Hegic mis-represented their report.
By now you've probably made up your mind and deciding who is to "blame". However, the story gets deeper and this is probably the more important bit of this entire fiasco that I'd like to touch on.
Shortly after this goes down, Molly Wintermute (anon founder of Hegic) writes a fairly lengthy post mortem of the entire situation. Read it here: https://firstname.lastname@example.org/post-mortem-hegic-unlock-function-bug-or-three-defi-development-mistakesthat-i-feel-sorry-about-5a23a7197bce
Through his/her post we get a slightly different view of the situation:
Molly requested an audit from Trail of Bits for the Hegic codebase
Trail of Bits responds saying that a 3 day audit is more than enough to provide good coverage of the smart contracts
Molly responds saying that 3 days seems a bit short and that he/she is happy to do a 1 week audit for further coverage
Trail of Bits pushes back and insists that 3 days is enough, oh and that they'll charge $9600 for the audit.
Molly asks what kind of report he/she will get after paying $9600 for 3 days of work. Trail of Bits responds with this:
After inquiring whether this was a full proper audit or a short one, no clear answer was given - just a "this should be enough"
Last, and most importantly, the Trail of Bits report contains the following line:
The Issue with Security Audits
Having been a developer/maker in the space, I have to say: security auits are one of the most frustrating processes in the entire development space. Why? Because they charge a fortune while saying very little.
I personally know of situations where auditors will charge $50k for a full audit of the protocol and then when teams ask "is this secure" they get a vague response to the tune of "this depends on your risk parameters". I'm not even kidding. Furthermore, every auditor claims that their reputation is on the line when they do an audit - however even if a vulnerability is found they'll find a way to see how they didn't see that coming/people don't really hold them accountable.
Let's look at this screenshot from above:
If I'm a developer who is paying $10,000 for a reputed company to audit my smart contracts, surely I should be knowing that my code is secure. However we can clearly see in Trail of Bits response they respond with:
This is what most people need to make a risk decision about the project
Not really. The reason why developers pay a fortune is so that they can go to experts to get a critical analysis of how their protocol works. I decided to look into the report and I found the last line in this paragraph particularly worrisome.
More issues might be present in the code base
Well isn't that what an audit is meant to cover if you're paying $10,000? Trail of Bits claims that this was a "3-day" code review and not a proper audit. However as we can see in the email thread that Trail of Bits was the one that insisted 3 days was enough to audit Hegic.
Now I wouldn't mind if auditors stuck to their guns and said that "this is only a check and shouldn't be a final of whether the protocol is safe", but the rest of the community actually does determine if a protocol is safe by the auditors and not the developers. Furthermore, if auditors in the real world are held liable for damages, why aren't smart contract auditors, where there is real money on the line?
This isn't even the most annoying part of this all.
Creating high quality content on a consistent basis is really demanding. By upgrading to premium you’re supporting me. Read more about my decision and why DeFi Weekly is switching to a Freemium model: