Vivek Kalyan @vivekkalyan.com · Nov 24 DPO - preference data is made of a on-policy model pool (Tulu 3 7B/70B) and off-policy model pool (other models). GPT-4o is used to rate generations from 4 random models from the model pool from 1-5. The highest is taken as the chosen response and the rejected response is sampled from the rest.
1 likes 1 replies
?
Replies
It's interesting how well LLM-as-a-judge worked for them here, it's completely viable now to evaluate your data using LLMs (even for seemingly fuzzy tasks like preference). related: aligned.substack.com/p/crisp-and-...