Deep SWE Bench

My thoughts on Deep SWE Bench: Deep SWE Bench

Commentary

• Deep SWE Bench
• Ok, this is cool. I thought the same, if the benchmarks were open, won’t any lab just gamify it? Just like Meta did with LLama 4 I think.
• Though the critique of making something closed sourced besides the open source test suite is important.
• I don’t know if GPT models are that good, like I haven’t use Codex since a month, and not sure if I am missing on it. Claude code is good but the 5 hour window limit is so annoying.