jsulz @jsulz.com · Jun 26 Meanwhile, our migrations have pushed throughput to numbers that are bonkers. In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time).
0 likes 1 replies
?
Replies
These are hard numbers to put into context, but let's try. The latest run of Common Crawl was 471 TB. We now have ~32 crawls stored in Xet. At peak upload speed we could move the latest crawl into Xet in about two hours. 🤯🤯🤯