About

TL;DR: I brought back 32 problems from a code-golf contest we played at Google in 2015 to see how frontier models compare with our old records. Each model gets 30 minutes per problem in each of eight languages to make its solution as short as it can. The leaderboard scores the byte counts, not just whether the code works.

Where the problems come from: go/codegolf

In 2015, when I was at Google, I spent a lot of time on an internal site called go/codegolf. There was one problem a week, and we competed to write the shortest program that solved it. Most of the regulars were code-golf enthusiasts.

Each problem went through three phases:

The rule was simple: the shortest program that passed every test case won. Undefined behavior, compiler quirks, non-standard library features, even hardcoding every test case were all allowed. Apart from passing the tests, the program only had to be deterministic.

The site supported JavaScript, Python, Go, C++ and Haskell in 2015. Rust was added after I left.

From a weekly contest to a benchmark

I kept a collection of problems from the 2015 contest and chose 32 of them for this benchmark. I wrote about half under the name @binjin. The Problems page credits the original author of each of the others.

I also archived the scoreboards, statements and test cases. For JavaScript, Python, C++ and Haskell, those scoreboards record the shortest solution any Googler submitted, including improvements made during the challenge phase. Those byte counts are the human records used here. The human solutions themselves are never shown on this site.

Eleven years later, I wanted to see how frontier models would do against the people I used to golf with.

The eight languages

There are no 2015 human records for the last four languages. I used an oracle run to get a reference size for each cell, though models have beaten those sizes in some cells. Later, I ran the oracle on the original four languages as well, this time starting from the human records. See Known Best and the oracle.

How models play

A cell is one problem in one language. Each gets a fresh session with a 30-minute wall-clock limit. The model receives the statement, the I/O specification, all the test cases and a short guide to the language. For GolfScript, it also gets the full language reference, since the language is small and not widely known.

There are two tools:

• submit runs a solution against all the test cases in the replicated sandbox.
  • A passing solution is saved only if it is strictly shorter than the model's previous best.
  • Wrong answers, runtime errors, timeouts and compile errors are returned with stderr.
  • Every response includes the current best known size. I want the model to know when someone has done better; without a target, it's easy to decide a solution is short enough and stop.
• playground runs a complete program in Python, Ruby, Raku or GolfScript, using the same sandbox as submissions.
  • It can preload the test cases as inputs and outputs, so the model can try its g on every case, compare the results with the expected outputs, or just examine the data.
  • It's useful for trying a language feature or checking what the interpreter does with an edge case.
  • It also lets the model write search programs: for example, to brute-force a magic number or find the shortest expression for a small formula. Human golfers do this all the time.

That's all the model gets. There is no internet, shell or access to other files.

For the four original languages, the sandbox reproduces the 2015 environment, down to the compiler or interpreter version, flags and quirks. A trick that worked in the old contest should work here; a feature added since then won't. The full runtime list is in Technical details.

Known Best and the oracle

Without human records for Ruby, Raku, Racket and GolfScript, I needed another way to set a target for the models and a reference for scoring. I ran GPT-6 Astra as an oracle, giving it every JavaScript, Python, C++ and Haskell solution in the benchmark: the shortest human solution for each cell and all the models' solutions. It also had internet access and no time or request limits.

I first used the oracle to find reference sizes for the four added languages. Later I ran it on the original four too, where it could work directly from the human records and try to improve them.

• The oracle is unranked; it only supplies best-known sizes.
• When it beats every competitor in a cell, the Problems page shows its byte count as Known Best, but never its code.
• Even the leading model scores below 100% in every language. The oracle had much more time and information to work with.

This matters when comparing Original and Extended scores. In the added languages, the oracle had no human solution in the same language to start from. In the original four, it could build on the best results of a week-long contest among golf enthusiasts, and it still found shorter solutions in many cells. That made the reference sizes harder to match, which is why the leading models score much lower in Original. It also means the human records score below 100% in any cell where the oracle or a model has beaten them.

Scoring

Each cell is scored by comparing its byte count with the best known:

score = (best known size ÷ your size)²

• Match the best known size and you get 100%. A solution 10% longer gets 82.6%; twice as long gets 25%.
• An unsolved cell gets 0.
• Original averages the 128 cells in JavaScript, Python, C++ and Haskell, the languages with human records. Extended averages the other 128 cells, in Ruby, Raku, Racket and GolfScript. Humans have no Extended score because they have no entries in those languages, so the leaderboard sorts by Original by default.
• The best known size can come from a human, a model or the oracle. If anyone finds a shorter solution, everyone else's score for that cell goes down.
• To score 100% across a whole run, you have to match the best known size in every cell.

A gold mark on a problem's scoreboard means that solution matches the best known size. A filled mark means no other competitor has matched it; an outlined mark means a tie. Humans count as competitors, but the oracle doesn't. If the oracle's Known Best is shorter than every competitor's solution, nobody gets a gold mark.

Technical details

• Function, not program. You define a function g. A wrapper calls it on every test case and prints the results; the wrapper doesn't count towards your size. For Haskell, it also supplies g's type signature. The solution is the body of a let binding, and mueval evaluates the whole thing as one expression.
• State between test cases. JavaScript, Python, Raku and Racket run every test case in one process, so global state persists from one case to the next (Haskell is pure). C++ compiles the solution inside a struct that is created fresh for each test case, so its "global" variables start at zero every time; only static ones persist. Ruby runs each test case in its own fresh environment. GolfScript runs the program's top level once and then each test case in a fresh copy of that state, so precomputed values are shared but nothing a case changes carries over.
• Bytes, not characters. The score uses the raw file size. Spaces, newlines and every byte of a multi-byte UTF-8 character count.
• Deterministic. Each submission runs twice, independently. Both stdout and stderr must match exactly before the output is judged.
• Shuffled inputs. Each run gets its own random ordering of the test cases. The judge puts the results back in the original order before checking them, so a solution can't rely on the order of the inputs.
• One time limit per run. Compilation and the full batch of test cases must fit within the same time limit.
• JavaScript. Problems 01–19 run on Rhino 1.7R4 (JavaScript 1.7, interpreted mode at optimization level -1, stack depth 4096). Problems 20–32 run on Rhino 1.7.7 (partial ES6, optimization level 9, stack depth 65535); on a stack overflow or an unknown exception it retries in interpreted mode within the same 20 s limit.
• No prime builtins. Primality and factorization builtins are withdrawn in C++, Ruby, Raku and Racket, so prime-flavored problems stay non-trivial. Modular exponentiation, gcd/lcm, factorials and the rest of the standard libraries remain available.
• C++ quirk. Clang 3.5 accepted the C++1z range-for shorthand for(x:a) (a proposal later dropped from the standard). The original site rejected it with -Werror=c++1z-extensions, and so does this replica; for(auto x:a) still works. An AI model found a way to switch that error off from inside its own solution and use the shorthand anyway. No Googler ever did.
• Isolation. Every execution starts fresh processes in an isolated container with no network access, and everything it started is cleaned up afterwards.

Why no Anthropic models?

With a 30-minute limit and no way for the model to see the clock, a session can end before the model expects it to. Only accepted submissions are saved. It makes sense to submit something that works early, then submit each improvement rather than save it for later. GPT models submit at least 10 times per cell, sometimes 30.

I tried Claude Opus 5 and 5.5, Claude Fable 5.1, GLM 5.3 and 5.3 Flash, and DeepSeek V4.1 Flash, all at their maximum thinking effort. I kept running into the same problem: they spent most of the session thinking, then submitted once or not at all. I tried changing the prompts to get them to submit early and often, without much success.

I don't know how much of this comes from the way maximum thinking effort is tuned, and how much is simply that golfing takes them that long. I considered two ways around it:

• Lower the thinking effort. Opus 5 at medium effort still submitted only a few times. On the small number of problems I tried, it couldn't match GPT-5.6 Luna. I don't think a leaderboard entry from that run would be a fair picture of what it can do.
• Remove the time limit. That would be unfair to the models already tested under the 30-minute rule, and would need a much larger budget.

Cost is another problem. Anthropic's terms don't allow a Claude subscription to be used in a third-party harness, so these runs would have to use the API. I estimate $200 to $500 for all 256 cells, depending on the model. It could cost more without the time limit. That's not a small amount.

For now, I'm leaving these models off the leaderboard. I'll try again if the situation changes.

Why I built this

I started this after OpenAI introduced GPT-6 Astra as "a new generation of intelligence", and its president Greg Brockman ended the launch briefing with "Welcome to the AGI era." It made me wonder: how would we judge whether frontier models have reached the limits of human intelligence?

My first thought was mathematics. Some of the sharpest human minds have worked on it for centuries, and Astra had already helped solve long-standing open problems. Then I thought of code golf. A golf score can't answer that question about intelligence, but it gives me a comparison I understand: models trying to beat people at something those people cared about and worked hard at.

I spent about three years in middle and high school training for math olympiads, and golfing often gives me the same feeling as working on a math problem. You can be stuck for ages until you find a different way to look at it. Then you still have to make the idea work. And as you get further, the next step tends to get harder, whether you're finishing a proof or trying to lose one more byte.

That was a big part of the appeal of go/codegolf. I'd guess 30 to 50 of us played the weekly problem. We'd spend lunch breaks trying to save a single byte. Sometimes an idea would come to you and you couldn't wait to try it; sometimes you'd go days without an improvement, while the scoreboard kept telling you a shorter solution existed. When the challenge phase opened and we could finally read each other's code, some of the shortest solutions were astonishing. I once tried to work out how many engineer hours we were all "wasting" on it, multiplied that by the average Googler's salary, and estimated its weekly "negative impact" on Google.

Thanks to @wangz for creating go/codegolf. Without it, neither that contest nor this benchmark would exist. And thanks to everyone who wrote a problem or competed: your records are representing the humans here, and you've made them hard to beat.

I'm impressed by how far language models have come, especially when I look at some of Astra's solutions. There's still plenty of room to improve. The best known size isn't a fixed answer key: each shorter solution changes the target for everyone. I'm looking forward to seeing what the next models find.

If you have thoughts about this project, I'd like to hear them. I'd also appreciate help benchmarking more models, or with anything else that could make this better. I'd like to let visitors test their own solutions here, try to beat the known best byte counts, and see the scores update as submissions are judged. We'd need a cloud-based sandbox to run those submissions safely. If you'd like to help with that, get in touch.

Why Code Golf

TL;DR: Every model with a complete run can solve these problems. They differ a lot in how short they can make their solutions, even with the same tools and time limit. That's what I want to measure: can a model keep finding better ideas after it has code that already works?

Previously unpublished problems

These problems were written by Googlers for an internal site. Until this site, neither the statements nor the test cases had been published. The human solutions are still private. There was no public set of answers for a model to memorize; it has to work out a solution for itself.

Getting a working solution

Frontier models don't have much trouble getting these problems right. Every model with a complete run has an accepted solution in all 256 cells. If I only counted passing solutions, they would all have the same result.

But their scores range from 56.646% to 85.764% in the original languages, and from 49.914% to 93.585% in the added ones. Once a solution works, there's still the question of how much shorter it could be. That's where the models start to differ.

Scoring by byte count

• Exact. Either the program passes or it doesn't. If it passes, we count its bytes. There's no human rater or LLM judge deciding whether it's good code, and no points for style.
• Fine-grained. Saving one byte always helps. The squared score makes the difference between a close solution and an even closer one matter.
• Open-ended. We don't know the shortest possible answers. "Best known" just means nobody has found anything shorter yet. Models have already beaten the 2015 human record in 42 cells. Getting every problem right doesn't end the competition, and a new record gives everyone else something to aim for.

Less boilerplate to golf

In some contests, you spend part of your byte count on getting the input in and the output out. I like go/codegolf's format because it leaves very little of that overhead. I kept the same format for the four added languages:

• A function, not a program. You define a function; the wrapper handles input and output. Top-level code still runs, so you can use global variables and precomputed tables, or write a recursive solution.
• A fixed name: g. The name costs one byte, and a recursive call is just g(…). Recursion is often the shortest approach, though getting a very short recursive function right can be difficult.
• Libraries already imported. The C++, Haskell, Racket and Ruby templates import most of the useful standard libraries. The Python template imports nothing, so using even re costs import re. Sometimes it's still worth it.
• Nearly every byte is part of the solution. With I/O, naming and most imports taken care of, you're mostly golfing the code that solves the problem, rather than the code needed to run it.

Knowing the language and the runtime

When you're trying to save a byte, details that rarely matter in ordinary code can become useful: operator precedence, implicit conversions, an obscure builtin, a non-standard extension, or what a particular compiler does with undefined behavior. The original languages use old runtimes, including a 2015 C++ toolchain, Python 2.7 and GHC 7.8. Knowing how a modern version behaves isn't enough. The model has to know, or try out, what works in the version it's actually using.

Trying things out

A lot of golfing involves questions you can answer by running a few lines of code:

1. Will this interpreter accept this syntax? Will the compiler let me leave that out?
2. Try it in the playground or make a submission.
3. If it works, see where else you can use it.

Sometimes the experiment needs a program of its own. Golfers write searches for a magic number or magic string that encodes a lookup table, or for the shortest expression that gives the right result using a few operators. A model that never runs these experiments will miss some of the same savings.

Finding a different approach

Often you can't get to the shortest solution by trimming the obvious one. You need a different algorithm, or a different way to represent the data. These are some examples I like from the models' solutions:

• Carries without carrying. Addition by hand asks how many carries occur when you add two numbers column by column. Each carry turns ten in one column into one in the next, reducing the digit sum by 9. So the answer is (digit sum of a + digit sum of b − digit sum of a+b) ÷ 9. Every GPT model uses this in Python and Haskell.

  The shortest Python solution avoids even the subtraction between digit sums. Instead of subtracting the digits of a+b, it adds the digits of 10⁵¹−1−(a+b). That's 51 nines minus a+b, so each digit is 9 minus the corresponding digit of a+b. Add up the digits in one string, divide by 9 and subtract 51. JavaScript and C++ don't have integers wide enough to add two 50-digit numbers here, so every GPT model still does the addition column by column in those languages.

• Letting the kingdoms fight. Bit Kingdoms asks which byte of a 32-bit integer has the most set bits, with ties going to the lowest byte. GPT-6 Astra's JavaScript doesn't count the bits in each byte. It removes one set bit from each byte in turn, starting at the top, until none are left. The byte that runs out last wins. Since each round goes from the top byte down, a lower byte outlasts a higher one in a tie.

  This fits neatly into a few bit operations: rotate by 8 bits to bring the next byte to the top, clear its lowest set bit with x & (x−1), and use the number of turns modulo 4 to identify the winner. The result is 50 bytes. The five GPT JavaScript solutions that count bits take 77 to 89. Astra's C++ does the reverse: each byte gains a bit in turn, and the first to fill up wins.

• Counting walls instead of paths. 2×n matrix asks how many blocked cells you have to clear to walk from the top-left to the bottom-right of a two-row grid. It looks like a shortest-path problem, but GPT-6 Astra's Haskell counts walls. A blocked cell in each row forms a wall if the two cells are at most one column apart. You have to clear at least one cell from every wall, and walls that don't share a cell need separate clearances.

  The answer is the largest number of walls that share no cell. A greedy left-to-right pairing finds it. Haskell's deleteFirstsBy does exactly that pairing if you define equality as "at most one column apart". It takes 71 bytes; the other GPT Haskell solutions work out the cheapest path column by column and take 80 or more.

• Queens in the complex plane. In Surviving queens, a queen survives if no other queen shares its row, column or diagonal. Three GPT models place the queen in row r and column c at the complex number r + ci. Then two queens attack each other exactly when the fourth power of their difference is real.

  The directions a queen can move are the eight multiples of 45°. Taking the fourth power multiplies angles by four, putting exactly those directions on the real axis, at 0° or 180°. GPT-5.5's version is the shortest Python solution, at 110 bytes.

• A table in one number. Parenthesis sequence (hard) counts the balanced subsequences in a string of parentheses. The usual approach keeps a table: for each depth, how many subsequences have we found so far that end there? Every GPT model's Python packs that table into one big integer, with one digit per depth.

  In base 9⁹, multiplying by 9⁹+1 keeps each count at its current depth and also copies it one level deeper. That's the whole update for (. For ), do the same multiplication and then divide by 9⁹, copying the counts one level up instead. Rounding down discards exactly the subsequences that would close more parentheses than they opened. The lowest digit holds the answer, apart from the empty subsequence, which we subtract: 61 bytes.

• One point instead of six faces. The Rolling Dice asks whether a die is back in its starting orientation after each roll. Most solutions keep track of the faces. GPT-6 Astra's Python and Haskell, the shortest in both languages, follow one point on a sphere around the die instead. Choose a point that no nontrivial rotation of the die leaves in place, and it returns to its starting position exactly when the die does.

  Stereographic projection maps the sphere to the complex plane. A roll in direction k, where k is one of 1, i, −1 or −i, becomes z → (z+k)/(1−z/k). Python can do this with ordinary complex arithmetic. The Haskell solution works modulo 37, using 6 for i because 6² = 36 ≡ −1, and multiplying by y³⁵ to divide by y. Every step is exact. Starting at 4, the point never visits a number below 4, so 4 ÷ z rounded down gives 1 when it returns to 4 and 0 otherwise. This takes 82 bytes; the other GPT Haskell solutions take 102 or more.

These solutions depend on finding and trying different ideas, then choosing the one that uses the fewest bytes. The fastest or most elegant algorithm isn't necessarily the one you want. Ordinary coding benchmarks rarely give a model a reason to look for this sort of solution.

Why one byte can be hard to remove

In a well-golfed program, the same piece of code often does several jobs. That's how it got so short. It's also why an apparently simple change can break something elsewhere.

Here is GPT-6 Astra's 81-byte JavaScript for Nim:

function g(a){for(i=d=-1,x=eval(a.join('^'));x*d<0;)d=2*(a[++i]&x)-x;return[i,d]}

The algorithm is the usual one. Let x be the XOR of all the heap sizes. If x is 0, there's no winning move. Otherwise, find the first heap a for which a − (a XOR x) is positive, and take that many stones. But fitting it into this code requires a few tricks:

• eval(a.join('^')) builds an expression such as 3^5^6 and evaluates it to get x.
• 2*(a[++i]&x)-x is equal to a − (a XOR x), but only needs to refer to the heap once. That saves a temporary variable. Unless x is 0, the expression is never 0, and it's negative exactly when we're looking at the wrong heap.
• x*d<0 keeps the loop running while d is negative. If x is 0, the product is 0 and we skip the loop entirely.
• i=d=-1 sets up three things at once: ++i will start at heap 0, the first loop test will pass whenever x isn't 0, and if the loop never runs, [i,d] already contains the required answer, [-1,-1].

GPT-5.5's JavaScript takes one more byte. It uses the same algorithm, eval and algebraic identity:

function g(a){x=eval(a.join('^'));for(i=~!x;(y=(a[++i]&x)*2-x)<0;);return[i,y-!y]}

The difference is where it handles the losing position. The search starts at ~!x, which is −2 when x is 0. The first heap it reads is then a[-1]. There is no such heap, so y comes out as 0 and the search stops. Finally, y-!y changes that 0 to −1, leaving any other value alone.

You can't turn one version into the other with a small local edit. The special case is handled in different places, and the surrounding code relies on those choices. That's common in golfed code: a variable has two meanings, or an expression both updates the state and checks whether we're done. Remove a byte in one place and you may break an assumption somewhere else.

The same thing happens in just 47 bytes of Python. GPT-5.5's solution for Drink Orders computes the binomial coefficient C(D+P−1, P) modulo 2³¹−1:

g=lambda a,b,c=2:b<1or a*g(a+1,b-1,9)/b%~-c**31

Each call multiplies by a and divides by b. To keep that division exact, we mustn't reduce modulo 2³¹−1 until the outermost call. The third parameter lets the code treat that call differently. ~-c**31 means c³¹−1: in the outermost call, c has its default value of 2, giving the required modulus. Recursive calls use c = 9, so their modulus is 9³¹−1, larger than anything they return. Taking the remainder there changes nothing. The base case, b<1or, returns True, which acts as 1 in the multiplication.

To make code like this shorter, the model has to understand why all the pieces work together. A change still has to leave the whole program correct.

Knowing when to keep trying

The first few cuts are usually easy. After that, you can spend a long time trying things without saving a byte. It's tempting to stop there, even when the scoreboard says a shorter solution exists.

Models get 30 minutes per cell and are always shown the best known size. They don't have to use the whole session: the instructions say to keep submitting while they have ideas for improvements, and to stop when they run out. Continuing to come up with things to try matters here.

GPT-6 Sol usually stops after 5 to 10 minutes, even at its highest reasoning effort. With exactly the same prompts, every GPT-5 model, including GPT-5.6 Luna, is still working at the 30-minute limit on most cells. I suspect this explains much of GPT-6 Sol's ranking.

Eight different languages

The language set includes imperative languages, Haskell's functional style, Racket's Lisp prefix notation and the stack-based esoteric language GolfScript, for which there's little training data. Golfing well across all of them takes more than repeating familiar patterns; a model has to apply what it knows in quite different settings.

Human records to compare against

These records came from Googlers who enjoyed code golf and spent a week competing on each problem, with the option to keep challenging afterwards. They put real time into finding shorter solutions. Beating those records means more to me than beating a reference answer nobody tried very hard to golf.

What the results show so far

• Humans still lead overall in the 2015 languages: 90.532%, compared with 85.764% for GPT-6 Astra.
• GPT-6 Astra has passed the humans in Haskell (88.861% vs 80.330%), but not in JavaScript, Python or C++.
• Models have beaten the 2015 records in 42 cells, including 22 in Haskell. That doesn't count the unranked oracle.
• When I later ran the oracle on the original languages, it could build on the human solutions and improve them. It has a shorter Known Best in 66 of the 128 original cells, compared with 48 of the 128 extended cells. That leaves a tougher target in Original: GPT-6 Astra, the leading model, scores 85.764% there and 93.585% in Extended. There's still plenty left to improve in both.