Nish Tahir @nishtahir.com · Apr 22
1 / 2

I'm training the model using a self distill, so output loss is KL calculated against the base model token outputs. Model is Qwen3-0.6B, frozen all that gets trained is the encoder. Training on wildchat, running for about 30mins.

1 likes 1 replies

?

Replies

Nish Tahir · Apr 22

I haven't done any optimization or anything so outputs are about as expected but does show that the model is learning something. Probably needs a bigger/better encoder, batch size of > 1, maybe more sophisticated LLM. But fun start.