Tanel Poder @tanelpoder.com · Oct 22 Filtering data in real time (at CERN) After early filtering, this snippet from the article: "At this stage, the remaining events are recorded to a 40 Petabyte disk buffer." A 40 PB *buffer* for data 🫠 lhcb-outreach.web.cern.ch/filtering-da...
13 likes 2 replies
?
Replies
I remember an outline of what has to be their old system by now, and an important first step was a highly optimized chunk of code that read the stream of data out and marked 90% of it as not relevant, because it was known things, and needed to be pared so the remainder could be processed.
At 120 GB/s rate, that 40 Petabyte buffer can hold ~3.8 days worth of data.