Maintaining the security of DigiByte
It’s now a year since Barry had all but finished DigiByte Core 8.21 code (Dec ‘20). This includes ProgPoW and RandomX, along with other actual feature & performance improvements. The genuine progress, and how to get there, is what this post is about.
(This was originally written 5+ months ago but never published, I’ve now updated it and finally gotten around to getting it out there for the world)
Contrast that to the ongoing delays we’ve seen with the alleged “DigiByte 8.22” from Jared, which in reality is not even a “7.22” given it’s got no actual improvements and none of the code for the fork / upgrade. In fact, it’s gone backwards, and they’ve removed Dandelion at the moment.
It’s also been pretty unfortunate to see that the improved development process of DigiByte has been co-opted. Where the intentions were to ensure all code was thoroughly reviewed (not just “does it run”), it’s now become as bad as it ever was. So I want to give you a little history into some of the previous fails, and provide you with the answers you need for a solution to ensure they don’t happen again.
This can be summarized with: Big merges are bad, rebases should be the defacto process.
Side note: Some of this didn’t make it into the History of DigiByte book, but you should go and support Chris and pick up a copy of this coz it’s a pretty good read.
Hopefully I can explain all of this with minimal images, but in a simple way that somebody with even just basic knowledge of blockchain / DigiByte can understand.
How does a merge differ from a rebase?
I’m glad you asked because that underpins a lot of this!
A merge is where you take all of the code changes that have happened from upstream Bitcoin, and apply them all to DigiByte. Where Bitcoin changes the name of a file, or moves some code from one file to another, or refactors certain parts of it, those changes then get laid over the top of DigiByte.
Sounds straight-forward enough, except it’s not.
Merges are great if you’re doing smaller or more isolated changes. For example if you’re working on RandomX and I’m working on ProgPoW, but we both make changes to the way the consensus will be activated, it’s easy to look at it and go “Ah yeah we should actually do this slightly differently to accomodate both of what we’re trying to accomplish”.
However when you’re merging the code from hundreds of thousands of commits, each commit could possibly break things if not carefully reviewed in the context for which it was intended at the time. It becomes a nightmare to have someone else review those changes afterwards too.
Rebasing though, is different, more streamlined.
You take a “known good” starting point (Bitcoin Core) and apply JUST the changes needed to make DigiByte “DigiByte”, leaving behind all the stale / unused code.
You’ve probably seen the drum being beaten about DigiShield, how it was the greatest thing since sliced bread, and MultiShield too now for MultiAlgo. You’d be forgiven for thinking it’s tens of thousands of lines of code. You know how many it is?
A thousand?
A hundred?
A dozen?
Less!
In fact, the Ethereum difficulty adjustment code was just a single line:
block_diff = parent_diff + parent_diff // 2048 * max(1 — (block_timestamp — parent_timestamp) // 10, -99) + int(2**((block.number // 100000) — 2))
(It wrapped though here on Medium, coz it’s a long line)
So when you talk about a rebase bringing across DigiShield (which it has to do because it needs to validate the older blocks), we’re bringing across just a couple of lines. MultiShield isn’t even a dozen.
Rebasing is what Litecoin was doing too. They would take Bitcoin, add in Scrypt, change the max supply / supply emissions / block timing, and that’s basically it. Thrasher- churned out the last Litecoin Core in 2020 in what looks like less than two days worth of effort. What a champion!
That’s the way DigiByte should be proceeding too.
So why is merging bad then?
Well, because so much fluff gets left in for starters, but also because it’s going to be impossible to review the code. Things can get slipped in there, intentionally or inadvertently, which you naturally don’t want.
Let me explain, with some historical examples:
DigiSpeed code removed
When DigiByte merged in the code from BTC Core in 2017, the SegWit code broke DigiSpeed block size doubling. As such, that block size increase code was removed.
The reason?
SegWit changes block “max size” to “ max weight”, and by Jareds own admission at the time they didn’t fully understand how that worked, nor how it interacted with DigiSpeed (because DigiSpeed increased the block max size, not the max weight).
It would have been simple enough to adjust that block max-size to max-weight but because of the myriad of changes happening as part of the merge, when the SegWit code came up for merging it was a case of “Hmm dunno what this does, let’s just bring it in and we’ll fix what breaks later”. Which happened. The max block size code, despite being unused, was still there until I removed it cleaning things up a little back in 2019.
That’s part of what I mean about historical “fluff” being left there, which can in time cause bugs, security issues and more.
Double spends on DigiByte
I’ve talked about this before in depth (at the end of the article), but it needs repeating:
Jared merging BTC code caused the DigiByte double spends of 2018
It’s just another reason why merges are less than ideal, given it broke the way that MultiAlgo looked at the weighting of SHA256. The merge was done in 2017, but it wasn’t exploited until 2018.
This wasn’t just one double spend either, to my knowledge it occurred 16x over the course of several weeks, hitting several exchanges:
Yes, that’s right: Jared merging code caused over a dozen major double-spends on DigiByte. But he still wants to proceed with merging as a plan?
These double spends are yet another reason why merges are bad. Bringing in that code on-top of DigiBytes effectively overrode the MultiShield code. Rebasing wouldn’t have had that issue given you apply the DigiByte changes on top of the Bitcoin code, ensuring you have brought across what you need in the codebase.
The crazy thing? This was an earlier merge from 2017. More code merges had been done since, and it had been in there the whole time.
Code can easily be slipped through when merging
Unless somebody is looking over every single line of code, and vetting it thoroughly, you can slip through a LOT of things. It might not be malicious, it may be inadvertent issues that get through, but it can still happen.
In fact that was how the fix for the aforementioned double spends went through: As part of a large merge, to hide it, so people didn’t see it!
Knowing nobody would review the code (Because frankly nobody could) for a big merge, the code for the double spend fix was included in an attempt to get it incorporated and keep it all on the down-low. This is why anybody trying to tell you (*cough* Jared *cough*) that all the code from a merge will be reviewed, is lying to you.
This is the merge that was done pulling BTC Core 0.16.2, and included in it is the fix for the double spend issues. Did you know it was in there?
I bet you didn’t, because even by the time you get to see it on GitHub it doesn’t show you it by default because of the size of the code merge:
But it was in there!
Now imagine if somebody wanted to slip in something malicious, given there’s hundreds of thousands of lines changed, how would you ever know?
Or what if it was an accident that caused some other major security issue, how would you ever know?
Inability to import older wallets
Merging also broke the ability to restore wallets with private keys from the earlier 2014 days of DigiByte!
Yep, if you had your private keys from being an early DigiByte supporter, restoring those keys wouldn’t work in 6.16.2, the fix for that came in 6.16.3.
Yay chalk up another win for merging.
Broke fresh blockchain syncing
Yep, did you know that when 6.16.5 was released, further code from upstream was merged which broke the ability for DigiByte to sync from scratch?
Yep, the testing was great on that one… rather than just adding the one-line fix to prevent the crash from duplicate inputs, code was merged, and it broke the ability to sync DigiByte from scratch for two whole days until the new release was pushed out. It was fine if you were upgrading, but you couldn’t delete your blocks directory and start a sync from the beginning of the blockchain, it just wouldn’t work!
Again that’s far less likely to have occurred with a rebase (not to mention some half-decent testing).
Thanks to Murat who found and reported that issue back in the day 🥰
So why do Jared and GTO90 insist on a merge?
I genuinely don’t know. It’s completely asinine at this point.
I’ve clearly shown a myriad of issues that have all stemmed from merging BTC Core code, one of which literally resulted in the blockchain being exploited and double spent on for several weeks. Who knows what else may be hiding there from a previous merge? You can go and view and verify these things I’ve asserted for yourself too, they’re even talked about in the Release Notes of DigiByte too.
There is no good reason at this stage for the merge to continue, especially given Barry had shown a rebase can be done. It’s now 12 months since the 8.21 rebase was all but completed (Dec ‘20).
TWELVE MONTHS!
This merge drama from Jared has been a complete sideshow, just so that his ego doesn’t get bruised, because he’d become and still is irrelevant.
What work has been done on the merge in July this year, when Jared said he was giving 110%, was actually mostly Yoshis work (thanks Yoshi!). Yoshi too has previously stated he’d be fully onboard with a rebase.
Plus, that merge work done on 8.22 is realistically still a 7.22 because there’s no other fork upgrades / improvements, no algo swaps, nothing.
Will a rebase really solve these issues though?
Great question, and I’ll ask you another in return before answering:
What was the point of changing the development process?
It was to get ALL the code thoroughly reviewed before it was accepted.
There is not a chance in hell of that happening with that merge. Jared would sooner denounce his conspiracy theories, than that code has of getting reviewed line-by-line.
A rebase on the other hand, means there’s significantly less to review and less to break. At worst, you’d be bringing across earlier “unknown” issues, but there’s way less chance of bringing in new ones.
In fact, the DigiByte repository shouldn’t have the full BTC Core code there, it doesn’t need it. Ideally it would be a set of build-scripts that simply applies the DigiByte changes on-top of Bitcoin Core at build-time.
Sounds scary, how the heck would that work?
It’s pretty straight-forward actually:
99% of the code in that repository is still Bitcoin (excluding the hashing algo libraries).
Take Bitcoin Core (let’s say 0.22) and use that as a base. However, rather than having a monolithic set of code in DigiByte repository, ideally it would only have the changes to Bitcoin needed to make DigiByte “DigiByte”, such as the changes to the maximum supply, algo changes etc
Those could be applied, as a series of patches during the build process. The libraries pulled (such as for Keccak, used in Odocrypt) would be pulled as a whole file, untouched, and can be hashed against earlier versions to see that no changes have been made.
You could even save those file hashes to the DigiByte blockchain so that people can see the hashes in perpituity for those files to ensure they haven’t been changed.
What DigiByte would be left with is a far, far smaller set of changes. It would be an infinitely more easily reviewable codebase, and it would be significantly simpler to upgrade things in the future to base off a new version of BTC Core too. The DigiByte “changes” would be packaged up nice and neatly so they could be reviewed, without having to go through all of the hundreds of thousands of changes that BTC Core have already done and reviewed themselves.
Barry even started on this back in July, pulling most of the hard yards already. It would take far less to get that ready to roll than it would to finish the 8.22 (7.22) merge. It’s already far closer to what Litecoins rebase commits look like too!
There would be further effort needed to fully segregate the DigiByte code from Bitcoin, and optimize it for in the future, but it would make DigiByte improvements SIGNIFICANTLY simpler in the future because there’s no wondering “Is this something Bitcoin related I’m changing? Or am I changing DigiByte?”. Not to mention it will still be less work than the merge has been, or still needs, and it would be able to be reviewed.
The files for things like Keccak can be compared against existing ones (Looks the same to me) automatically too with a build-script to ensure they’ve not changed, removing 99% of the effort as well.
So what’s the way forward?
Demand better.
Don’t settle for the lies Jared keeps shilling on Gitter and Telegram.
Demand that the Foundation and DGBAT members who have stood by and watched this all since January actually step up and step in. Rudy, Hans, Rutger, they were all there in the chat watching when Jared torpedo the development in January, and they stood by idly. Time those Core Values of the Foundation were put to use. I point blank told them to get a 3rd party opinion, they didn’t, and opted to blindly follow Jared who has let them down again.
The DigiByteCoin Twitter account that dgb_ycagel primarily runs, needs to stop sharing Jareds nonsense and giving the community the false impression that Jared is still involved. Throw down the gauntlet, and if Jared throws a wobbly (haha that’s never happened before has it?) then move on and start a new DigiByte twitter account that he isn’t pulling the strings on.
Jared’s been absent from the project for the years, simply giving the illusion of involvement.
Nobody can tell me anything over the last 3 (nearly 4) years he’s actually done. Why?
Because even when it came to 7.17.2 and Odocrypt, he wasn’t really even involved back then in 2019. I was helping him setup his Cyclone-V FPGA prior to the countdown livestream that took place, just 30 minutes before the algo upgrade was due to be activated (It was already locked-in) and he still had no idea how it worked!
Build a better future, without having him pulling the project down. Without him getting it delisted from the likes of Poloniex, or hamstringing the Binance listing progress.
Ever stopped to wonder about why Coinbase hasn’t listed DigiByte?
I wonder if it’s because of the history of merge issues, and the code can’t be reviewed? Or maybe it’s because of Jared, as it was for Binance? I don’t know for certain, but it’s not that far-fetched is it?
But you can do this community! I believe in you.
Just to clarify though: I don’t want to have access to any of this. I won’t be pushing it (I’m actually working full-time on a non-crypto business now sorry), but for all those who’ve reached out to me asking about the way forward that will ensure the security of the blockchain, ensure that the code is reviewed and is more maintainable in the future: That’s how you do it. You don’t do it with the merge. It’s impossible to review it.
The question is: will the DigiByte community settle?
Or will the “faster, more secure, forward thinking…” mantra finally come back to life?
