To familiarize myself with some RL techniques and how to fine-tune the latest open-source models, I took a pass at fine-tuning Gemma with some of the traces from larger models like Fable. The results showed there was some headroom in this approach for allowing open-source models to learn.
Check it out here.
I don't have much GPU to spare, so I had to take some shortcuts like some older evals, quantization to 4-bit, and LoRA instead of full fine-tuning. Despite this, you can see a lift on all the evals (most of which are saturated by the bigger models).
Datasets
-
Fable / GLM 5.2 traces -- about 10k traces of these more advanced models to function like a student/teacher (https://huggingface.co/datasets/DavidrPatton/Fable-5-GLM-5.2-Traces)
-
MBPP -- hillclimb, actual coding problems, about 500 (https://huggingface.co/datasets/google-research-datasets/mbpp)
SFT and GRPO
I wanted to give the feel of better coding LLMs, so for driving a harness, the hope was that SFT would begin to establish the right pattern for coding and driving a harness.
Both approaches helped the model establish better coding/harness-driving capabilities. Despite the change in quantization, this runs on essentially a consumer-level GPU that many folks have.
Next steps
Going to try to train it on a proper cluster and bring the evals up to par with the modern / non-satured coding evals.