Programming for the Planet @propl.dev · Jun 15 Deepak's a great example of someone who wouldn't ordinarily attend @sigplan-pldi.bsky.social; he's a 'professional scientist' who has worked on terabyte scale oceanic models, and now maintains important scientific infrastructure like xarray
4 likes 2 replies
Replies
Oceanography has an inspiring history of open source from the 90s zenodo.org/records/1061... but more sophisticated ideas and services never built a network effect and so we've ended up with 'standard compliant self-describing metadata-rich piles of files'
Each of these massive storage arrays is bespoke, and each consumer repeats huge amounts of computation without any sharing or peer verification. Updates and ingest are also invisible, and so users are exposed to raw filenames and how its partitioned. This is happening with data from NASA, JPL etc!