Alex A. (@alexatallah) is rolling out a new evaluation-writing approach and invited people to DM Jacky (@jjacky) to participate. Jacky reports running OpenRouter bakeoff evals to pick the best model for a specific use case and was surprised to find minimax m3 won, stressing that bakeoffs reveal unexpected top performers and that teams should always run model evaluations.
@alexatallah announced a new way to write evals and asked readers to DM Jacky (@jjacky) to try it out.
We're working on an exciting new way to write evals.
DM Jacky if you'd like to try it out!
jacky (@jjacky)
running openrouter evals to determine the best model for my exact use case
surprising results
minimax m3 wins out!
never would have been on my list if it weren't for the model bakeoff eval
always. run. model. evals.
— https://nitter.net/jjacky/status/2079299035257581655#m