youtube

spotify

apple_podcasts

[00:00:00] Dee: Having used a hammer before. This person behind you is trolling. 

[00:10:20] Dmitriy: I am at  their door. And they're like, what are you doing right now?

[00:22:06] Jessie: All right. Welcome, everybody, to another episode of hashing it out. Today we have three guests for you. We have Vitalik, Danker and Dimitri here to talk a little bit about specifically, a little bit about the history of sharding and Ethereum, as well as its evolution into current IPA 44 four as the first part of the journey to full sharding.

Welcome. can we, hear maybe a little bit from you, Vitalik, about kind of the history of, sharding in the context of Ethereum, and maybe you can start start there.

[00:52:00] Vitalik: Sure. So sharding has been part of the Ethereum roadmap, almost since the beginning. so people who have, been around you know, like since 2050 and might remember the scalability paper and, the scalability blog posts, back in 2014, where, you know, we talked about things like the hub and spoke model and hyper cubes. and, I think, 2015 was, the time when, like we first really several understood that, execution and, data availability are the things that both have to be scaled and that there is some, like, separate concerns for the two, and, solution that, handle scalable, verifiable computing and doesn't automatically handle data.

And so we have to do some, pretty different things for it. I think, the idea for Ethereum sharding at the beginning was, theory L1 focused, vision of sharding or basically just said that, you know, you would just have the ECM and, you basically just have a whole bunch of parallel copies of the ECM that would talk to each other or potentially you would like somehow split the existing account space up between a bunch of shards.

And then, if you had calls that, go between addresses that are in different shards, then they just have to be asynchronous or something like that. So the thinking at the very beginning was basically that, you know, we'd have this very monolithic sharding systems that would handle data, it would handle execution. It would basically try to convert a single threaded world computer into a thousand threaded world computer, where L1 would be handling all of the logic.

And then 2017 was the time when we really started taking the data availability problem seriously. Right? This is basically this, problems that if someone publishes a very large, blocks that contains, more data than what one node can practically download, which is necessarily the case of any, scalable system, then, like it? There is fundamental limits, to the safety of any mechanism that, tries to determine whether or not the data actually was published on time.

And like basically, there is a limits to what you can do with checking data availability that you just cannot do, or, or, or limits that just do not exist with fraud proofs, for example. Right. And, I came up with side data availability sampling, back then, back in 2017, there was a, a post on, I think, somewhere on GitHub.

And then there was, a code implementation. Back then, we were just using two dimensional binary trees and, then, that just kind of kept, the state in a corner, and we weren't really thinking about it too much because we were just so far away from actually able that take sharding. And then in 2020, we started thinking about the roll up centric roadmap.

And, one of the interesting things about the role of centric roadmap, right, is that it basically splits the sharding problem in half, and it basically says we still do a sharding, but, the L1 only handles the data, half of sharding, you know, the execution, half of sharding actually gets handled at the rollup level. And, we basically say you can create your own shard, and if you create your own shard, then, like, you can actually go and have your own rules on your own shard.

And so it ends up also expanding the space of innovation quite a lot at the same time. And then, shortly after that, you know, we started thinking about, like, much more concrete ways where we could actually try to get, data availability space on chain and, like, like that's one kg and doing sharding and all of these other research ideas started coming in.

And, yeah, I'm sure it's Dankrad has quite a lot to say about those.

[04:36:18] Jessie: Now we can move the conversation to Dankrad. And you can speak a little bit about how you've been thinking about doing sharding as it exists now. And the namesake.

Yeah. I mean, it was, a few years in the making. It's actually like it's, it's quite funny how it worked out, because, we first came across, k the g commitments, I think, like, early 2019 was the first time I heard about them and got interested in them. And that was happen to be also the year when, pretty normal commitments and his G commitments started becoming really famous later on because then FLoC was built based on it.

but initially I looked at them and was like, okay, this is really cool because in theory, this, this idea, the point of my commitments, like, solves two really big problems that, we have with, sharding and data availability. the proof of custody and, data availability something. Or at least it makes them much easier then, then what we had available before, like the previous construction, like we'd have Vitalik talked about, talked about, construction using, Merkle trees, two dimensional ones, from 2017.

And, like, it has some major downsides, such as, like, needing fraud proofs, which, like, limits really like, how much you can scale with them because it means that, there has to be a large number of nodes, all the, all the consensus nodes that have to watch everything. and with covenants, we didn't have those downsides anymore.

But back then it looked like, oh my god. And some the performance that just terrible. I could flow like just take like hours to compute them and then computing the proofs and so on. And then what really happened over the next 1 or 2 years is that, like from everywhere I came, massive performance improvements. with respect to these and we basically, got the performance of computing the commitments and the proofs down from the.

Yeah, like basically several hours to millisecond. And, and we, we noticed, I think like, yeah, like relatively soon after. Okay. Like now, actually like, based on this, we can, we can actually get a very practical data availability sampling system built now. And, it's, it it's really it's actually going to work like before that it was always like, okay, there's this piece that we will need to do at the end that's actually like really, really hard.

And so that was one of the big steps. and then the other one was realizing at some point I think like for a long time, Ethereum had, clearly had the goal of like making everything extremely decentralized and basically all nodes being more or less the same and no nodes like taking it on more work than others significantly.

and I think like from the movie space, we started getting the, the message, hey, like this doesn't work anyway. Like there will be some concentration. and, we need instead of, like, trying to ignore it, we need to focus on managing the concentration, on making sure that those parties that do emerge, think they can exist, but they shouldn't have power over the system.

They should be easily replaceable. They shouldn't be able to, like, do invalid things and other bad things and, and from this started the other strand, that came into having in the end, which was the idea that instead of like having all these separate charts that, like, separately provide data availability, to like, make it just one big system that provides a big block of data available to every block.

And, and that's what massively simplify things. And, the downside is basically just one, which is you need, a somewhat larger node that produces these large blocks that makes sure that, the all the data samples are well distributed after that. but like in return you get, an immensely like, simplified system, like when we like, push the diff, it's like really like 70% or so of the, of the code goes.

And that's like a big difference in terms of client complexity. and it also solved several of the problems we've been having like, how do rollups actually get the data on chain? How do they make sure they get it once and not five times? How do they pay for it? And all these things just like neatly fell out when you when you redesign the system.

And so this, happened like a little bit more than two years ago that we came up with that. And, yeah, since then we have started, actually implementing down on the roadmap of that. So like, we now actually have the first, piece of this, live on Ethereum, the, so called prototype sharding, which is a small piece that, implements all the cryptography that we're going to need for, for sharding.

and just, doesn't actually do the networking part yet. So it's a little bit of a cheat, but, it's both fun. Make it much simpler going from here to full sharding because, from now on, for example, rollups will not have to care anymore. Like any rollup that has upgraded to using, prototype sharding yet before it for four is ready for full sharding and will not need any further upgrades for that and, yeah, like, we're now working on the next pieces.

[10:05:06] Jessie: One of the things that I know that you guys are facing so regarding like networking, allergens, challenges, in the context of block distribution, and then also ensuring that, all the nodes have, certain amount of, I guess, bandwidth associated so that they can, make sure that the block is propagated throughout the network. What are the technical challenges that you've been facing there?

Maybe this is something Dmitriy can speak a little bit about, in terms of the kind of efforts that the simulator has been trying to help solve.

[10:36:02] Dmitriy: Yeah, definitely. So, yeah, we've been working on essentially trying to understand the tradeoffs at, network can layer and the data structure, levels, essentially, we've been looking at, so we've been building tools for this. We've been doing some simulations, similar, you know, figuring out what the different, parameters of the systems, what performance characteristics they, they, they yield, and in general, we've been trying to assess the, I guess, the direction and design of, of, of full gas.

I found this interesting, and I think it, it should be reiterated that your theorem is in somewhat of a unique perspective here, unique scenario here. And that there's additionally applied constraints to how this has to work. namely like it's in a, in a very adversarial setting. So you need to do this distribution with a good portion of the nodes being able to manipulate it or try and manipulate it, and also, some resource constraints and trying to keep the costs of an individual node as low as possible, because you'd like home stackers to be a part of the distribution.

Like, how does that change the roadmap and the difficulty of of pushing this work forward and getting something that actually work, or you feel actually works?

[12:06:24] Dankrad: I mean, I would say, yeah. So what's unique about Ethereum as, as you said, like we have the requirements that we want the network to be able to run with almost only, small nodes, like we're willing to accept a smaller number of like larger nodes, like, say, builders or something. but, but they should, they should be in such a way, like, like the worst case of these failing should at most be a liveness failure, and ideally not even that.

And, so yeah, what that means is that we have to design a system so that, for example, at least for, all the essential features, for example, reconstructing all the data, it fails, we don't want to rely on any, on any node that needs to get on the data. And so this means that, for example, the Celestia design, does not apply to Ethereum.

Celestia simply says, like all the consensus nodes have to have all the data. And I think like it's an interesting idea to have some nodes who have just who have all the data. I think like, that that does in all cases just improve the resilience of the network. So it's a good thing. But I think like our problem with the way they are doing it is that it requires all the contentious nodes to do that.

So there would be no more home stake. So that's not really like something that Ethereum could accept. and so that means that, yes, like we have much higher requirements for like when we actually do real scanning. So like right now, as I said, for the sharding doesn't actually scale, it just implements all the cryptography and, introduce the new fee market.

but like once we have, scaling, then we already need to, like, get to a distributed network structure. And what's hard about this is that nobody has really implemented these, these, distributed structures such as really under highly adversarial conditions like, so far, all the analysis is on under pretty friendly conditions. and so like with now especially like with the Codex team working on like finding a way of making them work and, the extremely pessimistic assumptions that we want to make for, if they are not.

And, yes, I mean, this like, will definitely mean that, it takes longer to get this into production. But on the other hand, I think, like we will have something that is far more robust than any other system that exists right now.

One thing that I think is, valuable to think about here is the very, gradual, nature of the transition that we're going to have from, the status quo to full, data availability sampling. Right. So in, data availability sampling, basically each node only, downloads a small portion of, each of the blobs. And so for every 128 kilobyte blob, you might imagine every node only downloading a couple of kilobytes or actually like know really full size.

you actually go even further and you say for the entire grid of all of the blobs. So you might have 128 blobs for each of, actually 28kB. the user might only download like 50kB total. That's just, like randomly sampled from across the grid. And, the challenge of that kind of design, right, is that the requirements on the peer to peer network, as you said, they go much higher, right?

Because, instead of a node being able to or needing to just have access to one other honest node somewhere on the network, because if you have one other honest, they can give you everything. There's like a significantly higher number of, honest two nodes that, a node needs to have access to so that the node can find all of the different pieces of the block that it's, querying for.

Right? And so there is this, like, very linear, kind of like very graduated way in which the robustness demands on the network increase as the actual level of scale of the network, increases. And what's happening is that, as of today, right, we have prototyping, sharding, which, as know we've mentioned, implements all of the cryptographic scaffolding of jank, sharding and all.

That's all of the data for about the level. That's all of the logic. but there isn't actually any data sharding going on in the sense that, everyone still downloads all the blobs. The next step from this is what we call peer does. And in peer does, the parameters are set pretty conservatively. Like basically the average node is going to download something like one quarter or one eighth of, the blocks.

And actually it's probably going to be even more conservative than that. Right? In the sense that realistically you'll probably have like small nodes, doing the peer strategy. But then every node that can't afford to handle it will probably still ends up by downloading the, the entire, contents of all the blobs. And so you have this, set up where, like, realistically, like even if the gas part breaks completely, the network is going to get, like, a pretty severe disruption, but it'll still keep on going, like it might stop finalizing, but, you know, the the full nodes will still carry it through and, like you basically.

Yeah, you naturally get this kind of training wheels phase, right? Like basically the way that I think about it is we're going to get this like step ladder of, expansions going from, you know, three blobs today to, eight blobs tomorrow to 24 blobs, into yourself, then 64 blobs, then 128 blobs and, and keep on going.

And, every phase of the step ladder is like the training wheels for the next phase or the step ladder. Right. And so we just, learn of the challenges associated with, each level of data dispersion and over time and, we get this nice, slow and steady path toward increasing scale. And, at each step, making sure that we can do it safely.

[18:03:20] Corey: It makes sense that we got to the point of, like, okay, you'd mentioned that L2 don't have to care at this point, which is really nice. And so we've set the training wheels of starting to solve some of the networking problems. And we'll, they'll, they implement the roll those out to do each one. But like Dmitri, as you think about this as a networking problem because that's where most of the problems lie now.

Like how do you what what bottlenecks do you see? How do you break this problem up? Like if we look at the idea of erasure coding, it's basically taking a chunk of blob and breaking it up into a bunch of different pieces, adding redundancy to it and then dispersing those across a bunch of peers. Like when you think about this problem and optimizing it the way Ethereum wants it to be optimized, like how do you what what do you think about immediately?

Like, how do you how do you break that problem up? 

[18:52:00] Dmitriy: Yeah, I mean, that's a great question. So there's different stages of how data travels through for this, through this network and then how this data is, sampled I guess, or downloaded. You can, you could say it's like data as well. but essentially we've, we've been looking at it as the initial setting of the, of the network with the samples, and then the actual sampling process itself.

And it turns out that they obviously they both have their own specific challenges. But what's more interesting, I guess, is that the current way that that the current primitives that are being used in Ethereum, they, for example, goes up, which is used to, to propagate blocks. And at the and the current beacon chain, they have their own set of limitations and they, they impose a specific structure that might not necessarily match the sampling.

what? But you know what sampling would impose, right? The, the flow, that sampling would impose. So I think that's where a lot of the, I think that's where the, the intermediate solution of or of pure dask came from. It's essentially realizing that we have a, you know, we have existing primitives in place that we can use there, battle tested.

They're well understood, and they will provide some, some level of scaling. They're not the full thing because in order to get to the full thing, very likely we'll have to probably, depart from this, from this primitives, right. From concepts up, for example, in some cases. And then also there is, specific assumptions made at the network in, stack level in the loop itself, which is, you know, stream oriented, connection oriented.

And this also might not necessarily fit very well. The, the, the test model is, as we as we envision it. And so that's where we are, focusing on that's what we're focusing on right now. It's essentially understanding how data propagates through the network, what the actual data distributed data structure looks like, what sort of addressing schemes are being, are applicable there and most optimal for, for, for this model.

And yeah. So we've divided this into, you know, dispersal of data, through the network. How does that look? And then the actual sampling process itself. 

Yeah, that's a that's an important distinction to make. I think that, most people don't quite into it. And that is like if you have the first step which is get a block broken up, you disperse it.

And then the next question is the sampling process of how do we know it's still there and recoverable, because you've broken it up and dispersed it across a bunch of different nodes you've charted such that, not like no individual has it all or needs to have it all. And then the sampling process is is a is a basically a way to checkpoint the system to say like, can I still get it back?

Right? Is that is that kind of the the differentiation there? And like what is the we understand the reasoning. You would like to make sure that the system isn't broken in the most minimally impactful way. What is the bottleneck there of like breeding a data data availability sampling process that doesn't undo the difficulties or doesn't undo the benefits of sharding itself, or right?

The sampling is what gives the system its its properties. Right. So that's that's, the, the actual thing that that happens to, you know, verifying that that something is, true in this case, whether this data has been actually published to the network. But the publishing part of it is, is equally important, obviously, because if you can't publish it fast enough, or inefficiently now, often you won't have enough time, to sample, for example.

Right. Like I said, makes sense. It's like, it's constantly asking the question, is this still available in the most minimal way?

I mean, it's similar maybe to like if you think of, you know, as proof system then, like, so the part you care about as a client is can I get the proof? But of course, someone needs to produce the proof. It's like the sampling is verifying the proof. Like I locally see. Oh, yes, that's truly available. Whereas like, yeah, someone needs to in the first place, distribute those samples to make sure that everyone else can check it.

By clarifying that. So I think it's it's lost on a lot of people on like how the system actually works and what we're talking about when we're thinking about improvements to the sampling process. 

Yeah. And it's interesting because the sampling, in order to be efficient and effective, and scale to the, to the numbers that we, we hope it scales.

yeah, it, it requires some very specialized, approaches. So, like, off the shelf, or potentially well understood and well tried out, structures might not necessarily scale as well as, as, as one would, but, you know, as much as we require them to scale, like then only, for example, it's, very likely has some bottlenecks that we're trying to now, understand better.

And, at the same time, we, we, we're trying to look at different addressing schemes. And, you know, there's a relatively well defined structure, in this network already. And we can we have to make sure that the this overlay matches the structure, right. This natural structure of the network already. So, that's where we're focusing on right now.

And we're trying to understand this again, the trade offs and then also the mechanisms that are available to us to, to make sampling more efficient. yeah.

[25:00:23] Dee: So, you know, how do we ensure that, you know, there's good communication, seamless integration, between all this data and the execution layer? Because, you know, if all, you know, if every node has a different piece and some nodes are different size pieces than others, doesn't that when they create like a lag between the execution or how is it, how does it work seamlessly?

So what you have to remember is that the data layer, doesn't doesn't actually communicate directly with execution layer. The only thing you get on the execution layer is a commitment to the data, and a guarantee that any of these commitments that you can get on the execution data layer, if it was included in a shard, like in a blob transaction, it was made available.

But you don't get the actual data. So like there's no process of like in the execution layer, retrieving that data from the, from the nodes that like a separate process. So for example, so in the, in ZK rollups, you never actually need to do that. Right. Because you just prove that like the execution was correct. So the execution doesn't never needs the data itself.

In an optimistic rollup you might need the data sometimes, like when you, when you do like one of those five proof games, like people need access to the data, but it's like it's the the role of the fraud provers or like whoever takes that role or, well, it could be the sequence, right. But they obviously always have the data, to put that data on chain when it's needed.

so what we need is that they are guaranteed to get the data, but we know that because data is available, but there's no way for directly for the execution layer to ask, hey, give me that piece of data. They can only basically incentivize someone to give it to them, say, like, I'll pay you like half an eath if you give me that data, and then someone will do it for them.

Yeah. So what are the solutions for that? Like what are the current people trying to build things to, to be that, data retrieval process?

I'm not currently following that. I mean, I think there are people building those things. I think, like, it's, interesting. It is not essential for, the type of, like. So, I mean, it really depends on what you're building, but for all apps, nobody really needs to build that system, because what, what you want to have is, roll up nodes to just follow the roll up.

And we just need to know that they are able to get all the data and they are like they will just follow the required subnets for that and they will get the data and they will have it. And if the fraud occurs they will have the ability to prove that fraud occurred. So this is all we really need.

All I can in the case of rollups, they also need the data because they need to be able to have the current state in case the sequence that drops out or the prover and refuses to do their work, then someone else can take over. So they need to be able to get the current state. That's also why they they need the data.

but yeah, like we get the guarantee that they can have it for this purpose. So we don't really need to build a dedicated data retrieval system or anything. There might be other applications where you want that, but like this is not I'm sure people are building that well.

[28:16:05] Dee: They're good money in it? I know a guy. He might he might want to make some money knocking don't care.

I guess I have a question when it comes to, like, applications that, need to go back and kind of retrieve old states that may be pruned at this point. How do you guys think about that in terms of providing some sort of persistence layer to enable that?

Right. so I think like this is like, I mean, that the different questions here, but I think like, so what Ethereum like, as the most basic service provides is this, this machine that advanced is the state that says like, okay, if this was the state in the book before, then it's this side. Now. And it's not a history machine.

It's not a storage engine. It's it's just these are the things that it is not. And it's never been I think it's like it's like a little bit of a, like a misconception where people think, oh, Ethereum history is available from all the nodes. So that's all it does a store things. No it does not that currently I can currently like.

Yes. No it's currently don't provide that as a flag. But you could change the node software right now and stop it from storing all the history. And it would work perfectly fine. So it's not really a guarantee that Ethereum provides in any like crypto economic sense that this is like promise like this is just happens to be an implementation detail of current nodes.

So there's really not a big difference. Like people make a big deal out of this like two week storage, window. It's really not it's like a practical implementation detail where we like. Yes. Have really said, okay, like now we really need to like be a bit more strict about what we're keeping because otherwise it becomes impractical. But this has always been the case.

Episode hosts - Demetrick Fergurson, Corey Petty, Jessie Santiago

Vitalik Buterin and Dankrad Feist of Ethereum, and Dmitriy Ryajov of Codex, join HIO for a discussion about the challenges and solutions in the realm of data availability and scalability in blockchain systems, particularly Ethereum.  They address the complexities of the issues, from Buterin's perspective on the long-term vision, to Ryajov and Feist's insights into the specific projects like Codex and the data sharding.

<p>Hashing It Out is a highly-technical bi-weekly show that brings you the pioneering developers driving the Cypherpunk movement. In each show we take a deep dive through privacy, sovereignty, and network state technology with some of web3's most influential innovators.</p><section class="footnotes-container">&nbsp;</section>

Hashing It Out is a highly-technical bi-weekly show that brings you the pioneering developers driving the Cypherpunk movement. In each show we take a deep dive through privacy, sovereignty, and network state technology with some of web3's most influential innovators.

Dr. Corey Petty

Jessie Santiago

Demetrick Ferguson

Hashing It Out

<p><a href="https://x.com/VitalikButerin">Vitalik Buterin</a> and <a href="https://x.com/dankrad">Dankrad Feist</a> of Ethereum, and <a href="https://x.com/dryajov">Dmitriy Ryajov</a> of Codex, join HIO for a discussion about the challenges and solutions in the realm of data availability and scalability in blockchain systems, particularly Ethereum.</p><p>&nbsp;</p><p>&nbsp;They address the complexities of the issues, from Buterin's perspective on the long-term vision, to Ryajov and Feist's insights into the specific projects like Codex and the data sharding.</p><section class="footnotes-container">&nbsp;</section>