AL Runner: A Weekend of Autonomous Agents, a Windows Cross-Check, and 59% of Microsoft's Tests Passing

I last wrote about AL Runner on September 3, covering the test suite that verifies its behavior against real BC . Since then I got funded for an additional Claude Max 20x subscription, dedicated specifically to AL Runner work. Combined with the Max 20x subscription I already had, that gave me a Max 40x usage window, and a token usage reset on Friday, September 5 made the whole thing available fresh going into the weekend. I spent all of it running autonomous agent loops — implementation agents and review agents working inside a defined process — against AL Runner and its companion test suite continuously since Saturday.

What that looked like in numbers

On AL Runner itself: 394 commits, 351 merged pull requests, 401 closed issues, 483 opened, all of it shipped as v2.11.0 . On BusinessCentral.AL.Language.Tests , the test suite from three posts ago: 110 commits, 110 merged pull requests. All of that between the morning of September 3 and today.

The number that matters most for that repository is the test count, since every test in it pins one specific piece of AL behavior against a real, running BC service tier. On September 3 it had 2,346 tests across 566 AL files. It now has 3,178 tests across 847 files — 832 more confirmed, provable facts about how AL actually behaves, added in five days.

Watching that count climb that fast made me skeptical rather than confident. Every one of those tests pins its answer against a real BC service tier, but that service tier runs on BC on Linux , and I wanted to know whether what they prove on Linux actually holds on a real Microsoft sandbox too. So I stopped and checked, by hand, against a real Microsoft BC SaaS sandbox. 14 tests failed there that Linux CI had been reporting as passing the whole time.

A Windows cross-check catches what Linux CI missed

I didn’t want to have to remember to run that check again, so the fix was a permanent one: a new workflow, nightly-windows.yml, that runs the whole suite against a real Microsoft BC container on Windows automatically, every night, instead of only when I go looking by hand. The first automated run came back with 672 failures, 659 of them permission failures. The cause: 96 of 100 failing test codeunits were missing a TestPermissions property, so BC’s default restrictive permission mode locked them out of their own fixture tables even when running as SUPER. That’s a one-line fix repeated across 100 files. After it merged, the same container and commit went from 2,094 passing and 672 failing to 2,856 passing and 15 failing. The suite has grown since — the latest run, from today, passed 3,099 of 3,106 tests.

The issue thread where this got diagnosed is signed “Posted by an autonomous agent acting on the account holder’s behalf” — that’s the agent loop from the top of this post, working in public and correcting its own analysis twice before finding the actual cause.

That one manual spot-check turned into a permanent, systematic one, and the systematic version found more than the permission bug: 8 real differences between how BC behaves on Linux and how it behaves on Windows or in a real SaaS sandbox, filed one issue per finding against the Linux container image itself. A SingleInstance codeunit keeps its state across calls on real BC but loses it on Linux — confirmed independently against both the Windows container and the original SaaS sandbox. IsolatedStorage(Encrypted) silently does nothing under a Linux patch instead of encrypting. Media.ExportStream returns 31 fewer bytes on Linux than on real BC. A published application’s package ID and runtime package ID collapse into the same value on Linux, when they’re supposed to be distinct. Five of the eight are already closed; the remaining three are still being worked, including one that needs a design decision about how to represent a Linux-only limitation in a test at all.

A first honest number for Microsoft test coverage

The Windows cross-check settled one kind of doubt, but not the bigger one. I had two coordinator agents orchestrating a whole set of implementation and review agents over the weekend, spending some number of billions of tokens between them that I couldn’t tell you precisely, and at that pace I couldn’t watch closely enough to know whether the work was actually headed in the right direction, or moving at all. What I wanted was something objective: a number I could check myself, where tests turning from red to green would mean something on its own, not something I had to take the agents’ word for.

The obvious candidate was Microsoft’s own test suite. Running it correctly needs real BC data, and that only became possible with the Cronus backup support I wrote about a week earlier — before that, AL Runner’s in-memory database started empty, so most of Microsoft’s own base-app tests couldn’t get past their own setup, let alone prove anything.

A separate new workflow, ms-surface.yml, runs AL Runner against the entire Microsoft BaseApp test surface: 40,530 tests across 32 buckets, one run taking about four hours. The most recent complete run passed 23,985 of them — 59.2%.

That number needs two caveats to be useful. First, about 3,382 of those tests didn’t run at all rather than failing, and almost all of that is one open bug: a missing .NET type causes a compile failure that zeroes out two entire test buckets. Second, and more interesting: Microsoft doesn’t run all 40,530 of these tests itself. Reading Microsoft’s own list of disabled tests directly out of its BCApps repository turned up 16,141 tests that Microsoft has disabled and doesn’t execute in its own pipelines. So 59.2% is coverage of everything AL Runner can discover, not coverage of what Microsoft actually runs day to day. The adjusted number against Microsoft’s real test set is still being worked out, and I expect it to be higher once it lands.

The hang that turned out to be a report running 101,000 extra times

One run against a single Microsoft test library hung on CertificateOfSupplyPartiallyShippedOrder, a test in the ERM sales documents codeunit — it exceeded AL Runner’s 60-second watchdog timeout, hung again on the automatic retry, and the whole job eventually got killed by GitHub Actions.

The cause: when AL Runner rebuilds a precompiled report’s metadata from SymbolReference.json, it was dropping the report dataitem’s MaxIteration property. BC treats a missing or zero MaxIteration as unlimited, so a dataitem meant to run exactly once over the Integer virtual table instead ran 101,001 times. Once found, the same bug turned up at 5 separate call sites, not just the one that happened to hang first. It’s fixed now.

Where things stand

None of the three threads above are finished. Three of the eight Linux-versus-Windows differences are still open. The coverage number against what Microsoft actually runs isn’t computed yet. That’s the right state for a mechanism that’s supposed to keep finding things, not a checklist meant to reach zero.

What’s still open points straight at what’s next. Finishing the coverage number properly means reproducing Microsoft’s own recipe for running its tests — excluding the same tests Microsoft excludes, and setting up the same data the same way — so the percentage means coverage of what Microsoft actually runs, not just what AL Runner can discover. Separately, I’m looking at restructuring how AL Runner builds object metadata so it no longer has to rebuild it from SymbolReference.json at all — that reconstruction step is exactly what dropped MaxIteration and caused the hang above. If that works, the same change should also open the door to running tests against runtime packages, not just regular compiled apps.

What’s changed since September 3 is that AL Runner now has a working way to check itself against real BC on a real schedule, and in three days it has already found and fixed a permission bug affecting 96 test codeunits, 8 platform-level differences between Linux and Windows, and a report execution bug that made one specific test hang. That’s what’s worth writing down before any of it is finished.