The speaker reports that a model trained on data only through 1930, after fine-tuning on modern coding-agent data, outperformed Claude 3 Opus on SweepBench.

“model which is only trained on data up to 1930 on this like modern coding agent data and and it did better than Claude 3 Opus on on SweepBench so so this model that has like no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and like behave better as a coding agent than this much larger pre-trained model”