Atharva Sehgal

@aseg.bsky.social

78 Followers 439 Following

Postdoc scholar @Caltech | PhD from @UTAustin

Even with visual programming, the LLM proposing the program has no idea about the execution semantics of the underlying VLM. Things still don't work.
gpt-o3, which has probably seen this image before, reasons incorrectly about the type of lizard and gets it wrong. Visual feedback is extremely important here!
An example from inaturalist of two scientist deliberating how to classify a rare lizard. The first scientists gets it wrong because they aren't trained as a herpetologist. The second scientist is a trained herpetologist, and reasons in natural language how to correctly identify the image.