Daniel MacArthur

@dgmacarthur.bsky.social

5634 Followers 1293 Following

Genomics, big data, open science, diversity. Director of the Centre for Population Genomics, focused on building a more equitable future for genomic medicine. Opinions my own.

ChatGPT chat interface. The prompt is “draw a pedigree illustrating second cousins” and the response is a nonsensical pedigree including sibling consanguinity between a pair of brothers
User study results: EvAgg use reduced the time to complete a case by 34%, and resulted in a >2x increase in the number of papers assessed by the curator.
Benchmarking overview and results. (A) Individual steps of the EvAgg pipeline were separately evaluated against the curated dataset to assess tool performance in different aspects of the task. (B) Task performance when benchmarked against the evaluation subset from the curated dataset: precision and recall for the paper selection and observation finding tasks, accuracy (i.e., the frequency of matching curated dataset content) for the content extraction task.
.
Benchmark curated dataset generation process. 1) Forty-five genes were randomly selected from the subset of ClinGen submissions to GenCC with only moderate, limited, or no known gene-disease relationship (GDR). 2) Approximately 4400 papers were returned by a PubMed search for these genes and manually assessed for diagnostic relevance using a standardized rubric, with ~400 papers deemed relevant (Supplementary methods). 3) From these ~400 relevant papers, we selected up to 2 papers for each of the 45 genes and curated detailed content for observations of individuals with genetic variants in the gene of interest. The curated dataset, consisting of the complete list of ~400 relevant papers and curated content for ~200 observations, was used for subsequent benchmarking.
Figure showing the type of table generated by EvAgg, which starts with a gene symbol and produces structured information on individual papers analysed, the individual patient IDs, variant HGVS nomenclature, associated phenotypes, population frequency, variant type, zygosity, and inheritance modes.
Figure 1 from the EvAgg paper, which compares the traditional manual workflow of variant curators to the ability of the EvAgg tool to assess multiple papers, extract information, and provide a structured summary.
Flyer for the LPSHG - details in the post
GPT-4o gets it wrong