fig (aka:[phil]) @bad-example.com · May 1

Couldn't resist looking back at firehose optimization after getting STAR-lite mostly working. Looks like 3x firehose bandwidth reduction is achievable with implicit MST + previous-commit cache of CIDs per repo.

34 likes 3 replies

?

Replies

fig (aka:[phil]) · May 2

with a few more hours of data, zstd 9 compression, bigger output bundles (marginal improvement) (neat that you currently only need a 200K-entry cache to get ~90% hit rate on DIDs)

fig (aka:[phil]) · May 1

(the cid cache hit rate is really around 65% (it's computing against all cids, not potentially-cacheable cids), and you can actually get it to almost 90% with a different cache approach (LRU instead of last-commit), but keeping the client's cache in sync would be way harder/messier.)

Mike P · May 2

Any CID that exists in the MST could be expressed as a path through the tree, i.e. a sequence of small integers describing which child node to choose at each step, to take you from the root to that CID. I bet this issmaller than encoding the full CID, and means you don't need a separate cache.