fig (aka:[phil])

@bad-example.com

art and transistors, they/them plant-mom, community infra for the atmosphere @microcosm.blue making/running constellation.microcosm.blue, slingshot, UFOs, spacedust, lightrail, relays, plc mirrors, hubble, service SLA & atproto consulting available

hubble memory rss

flat at 6G until just after 10:55, then a small gap, then it picks up at around 3G with an initial small rise, drop, and then probably stabilization at just under 3G

actors working (stacked)

a very spiky busy chart, y-axis 0–over 1000 over 3 hours. a "ScheduledResync" (purple) task dominates, but a lot of "Firehose#commit" can be seen underneath. "Bootstrap#wake" makes up most of the rest, with some smaller things also happening but which are harder to make out.

hubble memory rss (line chart)

a slightly noisy line that initially has some peaks just over 4GiB, but tapers slightly to staying between 3 and less than 4GiB

over about 3 hours

table-rendered json ('desynchronized_by_reason' is a subsection):

synchronized 75
desynchronized 24320
deactivated 0
gone 0
desynchronized_by_reason
app_requested 0
firehose_account_desync 0
firehose_account_throttled 0
firehose_fail 0
firehose_sync 0
first_seen 24320
future_rev 0
sync11_lax 0
throttled 0
unresolvable_identity 0

rapidly clicking "refresh" on a plain-text page that says only:

"hi this is hubble. there are 5615160 records here."

(the number counts up quickly but in small and big jumps)

the url is localhost:3000

$use jacquard_axum::{ExtractXrpc, XrpcErrorResponse}; use jacquard_common::xrpc::XrpcError; use hubble_sync::{AccountStatus, CommitObject, Did}; use rocksdb::{IteratorMode, ReadOptions}; use star_lite::{CarBuilder, ConvertOptions};$

// 6. check the commit's prevData against ours
if commit.prev != self.prev {
return Sync11Outcome::Desync;
}

account deletion rate and jetstream event rate: there's a big spike last night in the event rate from the ny nicks win, which was about halfway decayed by midnight. account deletion rate then spikes before 1am for a bit

screen crop of mu.social on a eurosky.social post with verification badge shown: the network tab is open, with details of a request:

Host: constellation.microcosm.blue
Filename: /xrpc/blue.microcosm.links.getBacklinks
--
source: app.bsky.graph.verification:subject
subject: did:plc:ooensn4mr5mhznzypvxelfa3

not accepted
not accepted
not accepted
deadline missed
submitted

sample 1: 20MB -> 21MB CAR snapshots

70-day survival rate

records (54,436): 99.97% (-14)
layer 0 (10,217): 99.82% (-18)
layer 1 ( 2,501): 99.32% (-17)
layer 2 ( 622): 97.75% (-14)
layer 3 ( 177): 94.92% (- 9)
layer 4 ( 39): 82.05% (- 7)
layer 5 ( 9): 66.67% (- 3)
layer 6 ( 2): 0.00%
layer 7 ( 1): 0.00%

ignored ( 888): 99.14% (- 4)

Producers SHOULD emit blocks in pre-order traversal of the included repository portion: header, commit object, root MST node, then a recursive depth-first interleaving of subtree nodes and the records they reference.

Preorder traversal enables streaming verification of repositories, allowing parsers to walk the MST structure and output key-to-record mappings while maintaining minimal MST state in memory. This approach supports efficient stream processing of large repositories without requiring complete buffering of the serialized data.

Parsers MUST tolerate other block orderings, duplicate occurrences of the same block, and additional unrelated blocks. Specifically:

box diagram: RepoActor (a tokio task)

external entry: registry.send - finds or wakes the repo actor

then all inside the actor:

intake rx: lil mpsc mailbox, goes to

intake loop: overflow policy, push or coalesce tasks, goes to

task queue, an internal VecDeque, to

process loop: actually runs tasks, does repo state mutations, with:

app callback, storage atomic write batch

milkcap.us-west.host.bsky.network (none) 448580
morel.us-east.host.bsky.network (none) 47256
mottlegill.us-west.host.bsky.network (none) 418860
mycena.us-west.host.bsky.network (none) 418703
oyster.us-east.host.bsky.network (none) 482727
oysterling.us-west.host.bsky.network (none) 417003
panthercap.us-east.host.bsky.network (none) 446256
panus.us-west.host.bsky.network (none) 424458
parasol.us-east.host.bsky.network (none) 445929
phellinus.us-west.host.bsky.network (none) 151380
pholiota.us-west.host.bsky.network (none) 515311
pioppino.us-west.host.bsky.network (none) 420639
poisonpie.us-west.host.bsky.network (none) 421514
polypore.us-west.host.bsky.network (none) 419107
porcini.us-east.host.bsky.network (none) 156186
psathyrella.us-west.host.bsky.network (none) 425332
puffball.us-east.host.bsky.network (none) 482488
reishi.us-east.host.bsky.network (none) 447059
rhizopogon.us-west.host.bsky.network (none) 312673
rooter.us-west.host.bsky.network (none) 147911
russula.us-west.host.bsky.network (none) 220797
scalycap.us-west.host.bsky.network (none) 416729
scarletina.us-east.host.bsky.network (none) 447930
shaggymane.us-west.host.bsky.network (none) 448767
shiitake.us-east.host.bsky.network (none) 81535
shimeji.us-east.host.bsky.network (none) 483372
splitgill.us-east.host.bsky.network (none) 332322
stinkhorn.us-west.host.bsky.network (none) 419532
stropharia.us-west.host.bsky.network (none) 611182
stropharia.us-west.host.bsky.network br 1
suillus.us-west.host.bsky.network (none) 418673
truffle.us-east.host.bsky.network (none) 446280
velvetfoot.us-east.host.bsky.network (none) 446359
verpa.us-west.host.bsky.network (none) 194347
waxcap.us-west.host.bsky.network …

a solitary tree on a small front lawn, covered in blooming pink flowers, obscures a small blue house. two bright green trees are behind, and a close dark green tree borders the right edge of the frame. in the far distance is the side of a mountain with more trees, but the top quarter of the mountain background is obscured by solid cloud

non_bsky_repos: 77,663
raw_car: 50.1 GiB
starlite_zstd3: 9.7 GiB
blobs: 4.5 M
blobs_size: 2.7 TiB

Too Many Requests

temporarily disabled 12 hrs after repo creation

consumer_events_processed rate

log scale line chart, "commit" events are a yellow series that never actually goes to zero during "dead" times -- just drops to ~6 events per second. then eventually jumps up high to catch up, and settles at live-firehose steady-state, and then ... for some reason... drops to 6evt/s for a bit again.

jetstream-bsky-network going to zero emit rate between 7am and 9:45am, then emitting at a high rate to catch up by 10am.

screenshot: Links to did:plc:hdhoaan3xa3jiuq4fg4mefid

7 links from sh.tangled.graph.vouch:..

Links, most recent first:

[one sample shown]

"operation": "create", "collection": "sh.tangled.graph.vouch"...
"reason": "really lovely person with nice vat pics "...

"operation": "update", "collection": "sh.tangled.graph.vouch"...
"reason": "really lovely person with nice cat pics "...

same as the last chart, just with the longer flatter trends extended over.

the compression performance gets a tiny bit better (under 30% estimated sztd bundled output w implicit cid), did hit rate breaks 90%

estimated client apps (stacked)

constellation growing from about 80 to around 130/day
slingshot (starting around march) stacked on top, taking it from ~150 to ~200 daily active apps.

two scatter charts:
first has two paired series: "raw: cid cache" + "raw: implicit cid" (second is smaller by a constant ~6%); and "zstd cid cache (+bundled) + zstd: estimated implicit", again the second improves by a roughly constant 6%.

both follow a sort of exponential decay toward a limit: first pair bottoms out around 60%; second around 33%.

the second chart has "did-cid cache hit ratio" and "overall cid hit ratio". both are kind of opposite of the last chart: rapid initial rise, slowing to some apparent limit.

did-cid cache: corners around 65% but keeps growing to around 80%, hasn't quite levelled off. overal cid hit ratio: levels at about 30%. might grow a bit more but not much.

three bar charts, each with multiple series:

Compressed size vs uncompressed CAR (series: CAR, blue; STAR-lite, red)
Y-axis: 0–100%.
X groups: uncompressed, zstd --fast 1, zstd 3, zstd 9, zstd 9 +long, zstd 22
CAR starts at 100% (by definition) and lands just under 33%. there isn't a big change from zstd3 through zstd 22.
STAR-lite starts at a bit over 75% and lands around 16%. there's a bit more improvement at the higher zstd levels.
The final bars of each series are labelled: 3.09x (CAR), 6.29x (STAR-lite)

Bucketed STAR-lite / CAR by compressor
Same percentage Y-axis, series are each compression level including uncompressed.
X groups are: "< 100KiB", "100 KiB–1MiB", "1–10MiB", "10–100MiB", ">= 100MiB", "weighted average"
Uncompressed series: about 70% for <100KiB, levels off around 75% over the other buckets
zstd --fast 1: about half-way between uncompressed and zstd-3 for all buckets
zstd 3: just over 50% uncompressed, rising to perhaps 55% by 10–100MiB. bit over 50% weighted average.
higher compressions start very flat (little benefit over zstd 3) at <100KiB, and show more spread as the CAR size increases. At >=100MiB, zstd22 takes a dramatic drop to about 34%.
The weighted average shows a pretty clean spread: zstd3 is much better than zstd --fast 1, zstd9 is a little better, zstd 9 +long is similarly a little better, and zstd 22 is better again by a bit more.

Bucketed count of repos & weight (total size on disk)
X-groups: same buckets without weighted average
Left Y-axis and "count of CARs" series: power-curve rapid dropoff: about 350K repos < 100KiB, fewer than 100K 100K–1M, much less than 50K 1–10M, barely visible bar 10–100M; invisible at >= 100M
Right Y-axis and "total bucket size" series (the weight of the weighted average): <100KiB has the smallest total, less than 10GiB; then the last bucket, >=100MiB has slightly more. Middle buckets in order: 100K–1M a bit over 25GiB; 1–10M around 90GiB; 10–100M almost 100GiB

CAR vs STAR-lite for various compression configs: [bar chart]

raw: car 1x (by definition… it’s the comparison point). star is about 0.75x the size

gzip is around 1/3 for car and under 0.25 for star

zstd 3 and 9 are similar, a bit better for both (star well under the 0.25x line, hard to judge further kind of)

overall (N=4866, raw CAR=2.18 GiB, raw STAR=1.62 GiB):

setting star vs raw CAR star vs same coder
mean med wt mean med wt

────────────
raw 0.668 0.678 0.746 — — —
gzip 0.292 0.232 0.215 0.568 0.556 0.552
zstd --fast 1 0.333 0.295 0.286 0.632 0.635 0.671
zstd 3 0.281 0.224 0.195 0.566 0.551 0.553
zstd 9 0.276 0.218 0.183 0.562 0.544 0.542

< 10 KiB (N=2168, raw CAR=6.31 MiB, raw STAR=4.04 MiB):

setting star vs raw CAR star vs same coder
mean med wt mean med wt
────────────
raw 0.614 0.611 0.640 — — —
gzip 0.395 0.370 0.301 0.624 0.640 0.585
zstd --fast 1 0.413 0.402 0.342 0.653 0.663 0.628
zstd 3 0.383 0.363 0.295 0.621 0.630 0.578
zstd 9 0.381 0.361 0.292 0.622 0.634 0.578

10 KiB – 1 MiB (N=2346, raw CAR=379.92 MiB, raw STAR=276.84 MiB):

setting star vs raw CAR star vs same coder
mean med wt mean med wt
───────────
raw 0.706 0.710 0.729 — — —
gzip 0.209 0.210 0.208 0.517 0.520 0.526
zstd --fast 1 0.266 0.270 0.270 0.607 0.615 0.629
zstd 3 0.199 0.198 0.193 0.519 0.521 0.529
zstd 9 0.193 0.191 0.184 0.511 0.513 0.519

1 MiB – 100 MiB (N=352, raw CAR=1.80 GiB, raw STAR=1.35 GiB):

raw 0.749 0.746 0.750 — — —
gzip 0.216 0.216 0.216 0.556 0.557 0.557
zstd --fast 1 0.288 0.290 0.289 0.676 0.675 0.680
zstd 3 0.192 0.194 0.195 0.547 0.543 0.558
zstd 9 0.180 0.182 0.183 0.534 0.532 0.547

a log COVERED with mushrooms, cropped in to show detail. they are greenish but otherwise similar to Nick’s mushrooms

rows and rows of little fans aligned horizontally around the log, which is resting on a leafy forrest floor

bandwidth on the network interface for relay.fire.hose.cam: starts at about 3x TX over RX (so ~3 clients connected) and then is pretty erratic over the day (but elevated) until the last few hours when TX starts really climbing high (more consumers)

RX is pretty erratic on its own, since the entire network throughput has been shaken constantly by bsky's app outages all day

key restart interval for 4k blocks and 32k blocks

4k blocks: very small size reduction at interval=32; nothing more at 64

8k: similar small reduction at 32; very small further reduction at 64

the jump up at interval=8 is higher, and the jump at interval=4 even higher still (but still ultimately pretty small)

z3 no dict vs 16k dict, and z6 no dict vs 16k dict, across block sizes

there's a large split between the dictionary and non-dictionary series at 4k block size (zst3 with dict is significantly better than zstd9 without)

at 8k blocks z3+dict still beats z9 without but they're closing in

z3 dict/no-dict are close at 16k block size, z9 still has a significant benefit

at 32k the z3 dict benefit is gone, z9 benefit slight

at 64k and 128k the dicts have no benefit

(it's possible that larger dictionaries could have helped...)

zstd-1 dict(16k) vs no dict, across block sizes

there's a significant win at 4k block size, a small win at 8k, and essentially zero gain at 16k and above.

i suspect that larger dictionaries might have helped with larger block sizes.

total size vs block size by zstd level, for zstd-1, 3, 6, 9

two pairs (1 and 3; 6 and 9) start out overlapping: 6+9 are about 1.6% smaller than 1+3 at 4k block size.

they remain similarly split at 8k, but all with about 10% better space efficiency

at 16k (another 7% gain) the 1+3 pair diverge very slightly (with zstd-1 actually slightly outperforming zstd-3)

at 32k they all fan out into a very even separation (1 worst, 3 better, 6 even better, 9 best). zstd-1 here is slightly outperforming z-9 at 16k.

at 64 and 128k the even fan-out remains, with similar (if slightly growing) gaps and diminishing returns.

lz4 total size is consistently much higher than zst1 without dictionary training, across all block sizes (x-axis). both series have the same shape (exponential decay) but lz4 is just higher (total disk size)

pretty much same as the keys-per-account chart looked before, but staying slightly rounder through the end (less obvious inflection)

screen recording of the quoted link showing the top four charts, with all the interactive widgets being clicked

first shows space vs lz4 or zstd compression ratios (lz4 is bad, all zstd levels are closer than i would have expected)

next is total size vs compaction time (higher zst levels take longer, larger block sizes compress better)

next is dictionary training for 4k and 8k blocks, with a weird step-like behaviour (performance is bad until the training size crosses a threshold, then is good. all are kind of similarly-bad or similarly-good on either side of the threshold, and the threshold varies by dictionary size.

last is space efficiency vs key rstart interval. boring. small effect at both 4k and 32k block sizes, pretty levelled-off by 32.

"value size distribution" and "keys per account distribution" histograms

value size rises rapidly to peak at the 128–256 byte bucket, then tapers off smoothly down to 256M (both axes log scale)

keys per account is so weird: it's almost flat until 4096 keys (most accounts covered) after which it drops off on a smooth accelerating downward curve, with a significant drop after 100k keys. log scale on both axes again.

the second is just really like, a big lump. it gradually rolls down like a border-radius until the last few buckets when it very slightly inflects back out.

also its very first bucket (1 key) is a bit lower than the smooth shape would have, and its next bucket (2–3 keys) is slightly higher. after that it's butter.

$screenshot of monospace json containing some stats about atproto records. most of it is two histogram sets, which both have triples of numbers on each line, creating visual channels that flow down and slightly rightward. the first one is "value_size_hist", ranging from 16–131072 on a log scale, with numbers steadily getting wider as it goes down. the right-most column of numbers bulges significantly in the first two thirds (peaking at 128) and then shrinks down to single digit for the last bucket. the second one is more uniform: "keys per account hist" ranges from 1 to 1M-sized buckets (2x each time) with the third-most column holding out at four digits wide until 16384, at which point its digits decrease almost matching the increasing bucket digits, with diagonal channels between numbers. the full dataset: ``` { "total_keys": 110471379, "total_accounts": 78568, "value_size_hist": [ [16, 32, 32], [32, 64, 30165], [64, 128, 13339156], [128, 256, 79513656], [256, 512, 13262139], [512, 1024, 3916643], [1024, 2048, 384505], [2048, 4096, 20803], [4096, 8192, 3652], [8192, 16384, 472], [16384, 32768, 68], [32768, 65536, 44], [65536, 131072, 37], [131072, 262144, 7] ], "keys_per_account_hist": [ [1, 2, 2630], [2, 4, 8451], [4, 8, 6955], [8, 16, 7082], [16, 32, 7424], [32, 64, 7898], [64, 128, 7817], [128, 256, 7196], [256, 512, 6323], [512, 1024, 5230], [1024, 2048, 4087], [2048, 4096, 2869], [4096, 8192, 2038], [8192, 16384, 1223], [16384, 32768, 747], [32768, 65536, 379], [65536, 131072, 165], [131072, 262144, 45], [262144, 524288, 6], [524288, 1048576, 2], [1048576, 2097152, 1] ] } ```$