Programming for the Planet @propl.dev · Jun 15 1 / 2
Icechunk is a cloud-native Rust-based chunked, compressed storage format for ndarray. It's got a hierarchical data model which cherrypicks from the best ideas in netCDF and so on and crucially promotes rigor via version control, isolated transactions and snapshots, says @deepakcherian.bsky.social
1 likes 1 replies
?
Replies
Since it's a big storage system, stability is super important for a cloud-native weather format. Public providers are moving to it, for example for weather forecasts and cloud data. It supports 'virtual' datasets built from piles of other data so you can compose vast datasets via 'symlinking'