Amanda Bertsch

@abertsch.bsky.social

2215 Followers 546 Following

PhD student @ CMU LTI. working on text generation + long context https://www.cs.cmu.edu/~abertsch/

Score by answer type and task type for Oolong-synth. The month+year and date types are the hardest for many models, corresponding with the difficulty of the timeline tasks.
Graph showing that the performance with labels in-context for Oolong synth is only slightly better.
A figure demonstrating the two splits of Oolong: the left side has the question “Were there more news articles about the economy in September or August?”, from Oolong-synth; the right side has the question “How many times in this set of episodes does the character Jester cast Healing Word?”, from Oolong-real. Both questions require the model to label sections of input, identify the important segments, and aggregate across these to answer the question.
Performance of a sweep of models on Oolong-synth and Oolong-real. Performance decreases with increasing context length, sometimes steeply.