Medium Rare Dev Log

devlog2: observability

It has been a while. Ups and downs, having and not having free time and energy for creative endeavors. While I took a break from this specific project I did the Factorio: Space Age speedrun achievement and upgraded my home network with a dedicated router appliance with OPNsense. I also consolidated multiple servers in my homelab into a single machine. Those two last points set me up for some of the whishlist items I wanted for this project: code hosting, doc hosting and a telemetry stack for observability.

Spring to Mid Summer 2026

This update focus on the second pillar of quality (the first being automation of menial tasks) for me: observability. Knowing the impact of new features on a frame budget, what takes time in the update loop and how much memory certain aspect of the game are taking is invaluable. Feels only get you so far. Since I dislike the shotgun approach of adding an observability library to project that hooks itself into your code (and that's also bad for performance critical software like games), I went with a very deliberate approach to it. The strategy is the same as the first pillar: solving the first use case while making a foundation solid enough that it's a no brainer to do things correctly instead of taking a shortcut.

Roselyn Generators and Interceptors

They suck to write, that's for sure. In typical Microsoft fashion, the abstractions are all over the place and are hard to read. I wouldn't recommend going for them as a solution to the vast majority of problems, but for this particular case they made the most sense. I wanted to be able to decorate a method with the kind of tracking I needed without sprinkling StopWatch.Start New() and Telemetry.Push() everywhere. With my system, the only instruction needed to track the execution time of a method, for all call sites, is shown below.

[MeteredExecTime(Instrument = Instruments.GRID_SYSTEM_UPDATE_TIME)]
public override void Update(TimeSpan delta)
{
    // do stuff
}

The Roslyn interceptors wrap the calls to this particular method with the necessary boilerplate to do the tracking. It's easy on the eye while reading the code, letting me concentrate on the core logic without noise. Added benefit that I can deactivate the telemetry at compile time with a single flag instead of sprinkling ifs where needed. The fastest code is the code that doesn't exist.

For the benefits of generators, you get to write, debug (poorly), and test (poorly) some ugly interceptor code like so.

var methodData = context.SyntaxProvider
    .ForAttributeWithMetadataName(
        "Core.Telemetry.MeteredExecTimeAttribute",
        (node, _) => node is MethodDeclarationSyntax,
        (ctx, ct) =>
        {
            IMethodSymbol? symbol = ctx.SemanticModel.GetDeclaredSymbol(ctx.TargetNode, ct) as IMethodSymbol;
            AttributeData attr = ctx.Attributes[0];

            // Pull named arguments from the attribute: [InjectTelemetry(Instrument = "nameOfInstrument")]
            string instrument = attr.NamedArguments
                                    .FirstOrDefault(a => a.Key == "Instrument").Value.Value as string
                                ?? symbol?.Name ?? "Unknown";


            return (symbol, instrument);
        })
    .Where(x => x.symbol is not null)
    .Collect()
    .Select((arr, _) =>
    {
        // Build a lookup dictionary keyed by method signature string
        var dict = new Dictionary<string, (IMethodSymbol symbol, string instrument)>();
        foreach ((IMethodSymbol? symbol, string instrument) item in arr) dict[item.symbol!.ToDisplayString()] = item;
        return dict;
    });

and it goes on for another hundred line of similar looking instructions to search for call sites, merge the data, etc. You haven't generated a single line of useful code yet.


For actually generating code once you have gathered the method and call site data you have two equally bad choices. You either do string templates with interpolation or get to play with the C# AST library to build the source code you need. I went with the templates since it's easier to write and easier to grok what is the general shape of the generated code you will end up with.

namespace UT.systems {

    public static class GridUpdateInterceptor
    {

        [global::System.Runtime.CompilerServices.InterceptsLocation(1, "FJ9WjSbOg/UdnaGMN3H3qaAEAABFeGFtcGxlR2FtZS5jcw==")] // /home/{$USER}/src/uniform-tango/Game/src/ExampleGame.cs(45,15)
        public static void Update(this Grid @this, System.TimeSpan delta)
        {
            Stopwatch watch = Stopwatch.StartNew();

            @this.Update(delta);

            Telemetry.Instance.Record("systems.grid.update_time", watch.Elapsed.Nanoseconds);
        }
    }
}

final, wrapped, code for the call sites of Grid.Update()

Result

I'm using OpenTelemetry library under the hood of my Telemetry service. The data is sent to a self hosted ClickStack on my homelab and I get some to do some fancy dashboard to track stuff. I can easily point to any OpenTelemetry compatible collector for ingestion in the future.

image

Next Steps

Bunch of Links