sao.dev / writing

Software Annealing

2026-08-08T00:00:00+00:00

One of the fundamental ideas behind the CPN project's</a> development is "Software Annealing". Annealing</a>, oversimplified, is the heating and cooling of a material to align molecular structure and merge crystaline regions within the material, generally hardening and improving other desirable features. Software annealing is philosophically similar, in higher dimension:

Establish or update the project's heading or scope.</li>

Establish applications/use-cases of the project that stretch its current state.</li>

Refactor or evolve the project into a new "simple" or "solid" state.</li>

Go to 1.</li> </ol>
It sounds simple; and at a high level, it is. The process is very Agile™, which gives it great optionality, and it aligns well with the naturally evolving process of discovering the new, valuable, insightful form for the software. One of the risks of software projects is that software can be whatever you make it, which tragically also means it can be whatever you imagined it maybe should be at 1 AM one fateful night.
This brings up one of the often unsung hero's of successful software: good applications of said software. This can be users, or other apps serving users, etc. These applications of the software-under-optimization are the neurons that propagate gradient from "the market" (the real world), and anchor the software in reality. It's the "personal software" workouts app</a> that motivates flexibility and client-server capabilities, it's the TPC-C benchmark that ensures performance, it's the scraper that motivates complex binding semantics, etc.
The power of these applications is even higher when driven by the project author. By looking at the high level goals of the project, and finding gaps with what is being tested now, the author can pattern match those gaps to applications that would truly test them. By doing continuous narrative building (it's all narrative building</a>), you can measure the delta between opportunities identified and recorded in the past and see how they align in total. These notes on opportunities and frictions from the past are the crystal dislocations</a> waiting to be annealed. This is why I love threads</a> so much as a human and machine-read planning and execution journal. They make execution easy and retros/insight mining almost free.
Oh yeah, LLMs! They make this whole process way faster. LLMs love a verifiable goal and a well described task. They are good foils for thinking through designs and problem framings (especially the latest models), and can then execute on the whole loop with some hand-holding. And maybe one of the easiest to overlook boons they bring is their surprisingly strong inductive capabilities. After enough "data" generated by way of applying the software-under-application, you can sometimes just point a sufficiently smart code agent at a set of threads with a vague "I get a vague sense that our migration story is complex" prompt and it will come back with some surprising insight.
Another practically valuable property of this strategy for solo developers is that it keeps the project interesting. It is very easy to park ideas that you capture, but don't want to act on yet, without it just going off into the ticket abyss. You are essentially maintaining a portfolio of options, of threads to pull when you are inspired or the time is right. I've got a thread on net change data capture (so postgres/etc can tail) on the shelf, while establishing identity capabilities that is more pressing. Very fun.

Appendix</h1>
Workouts App</h2>
Making software for yourself is great. The workouts app</a> is software I made to track workouts. Probably one of the most common personal projects, and honestly the second one I've made.
This one is pretty cool though, if you don't mind me bragging:

It implements petri nets for both the front and and backend, with semantics for uncertain network transmission</li>
It handles idempotence and reconciliation implicitly due to the foundational token movement concepts of CPNs

That means I don't have to wave my phone around in the gym where the wifi is always bad to be confident my workout gets submitted</li> </ul> </li> </ul> I was also inspred by a recent HN comment</a> about tracking strength training progress using "Pareto Fronts": strength training and physical fitness in general is multi-facted and can be hard to be certain if you're really progressing, and adding this to the workouts app should make it easier to suss out.
Beads, Gastown, Buzz, etc</h2>
The vibe that I get when I hear about software factories is that they are kinda missing the point. They're focused on "coding isn't the hard part", and betting that the thing to focus on is orchestration. Focusing on orchestration is kind of a bet against increasing LLM capabilities - if we start running Fable on ASICs, do we need things to be orchestrated? Or is a single threaded loop fine? It strikes me that these systems may become useful, but only briefly before "Fable on ASICs" makes them irrelevant (and probably destroys the fabric of capitalism).
Thinking about it through this lens, software annealing seems to be about enumerating and revising the bets of what is valuable over time. For things like the CPN project, its core bet is easy to describe (the arc graph can correctly encode all state transitions), but not quick to turn into complete software ready to replace postgres, temporal, or fastapi. You need to continuously think about what learnings to bake into the framework, about what requirements are ancillary. Claude would just love to add yet another concept to your codebase, and you have to have the taste to say no! No is the other side of the coin from deciding what is valuable.

A Vine in the Wind

2026-07-28T00:00:00+00:00

Evolution can surprise you sometimes.
As I looked out my window at my neighbors yard, I saw a vine, outstretched into nothingness.
So clear it was that it was not growing toward another anchor point.
And then, a gust - the vine's leaves, like sails, carried it up and around and the vine waved like an arm outstretched grasping for sustenance.
So clear it was that it could hook onto something nearby without even knowing it was there, or planning for any specific thing to be in any specific place or orientation.
Here was an arm grasping with no muscle, no sight, just environmental randomness doing the organism's work for it.
They say you need to be open to new opportunity to truly grow, and I think that's true.
Maybe you also need to be movable to a place or perspective of opportunity, a second order effect.
Maybe you need to be a vine in the wind.

Do we become our skills?

2026-07-25T00:00:00+00:00

As I've grown and aged, I've become more and more satisfied and self-assured. But in the wilds of modern life, is it justified? Or do we just become our skills, our opinions, our social groups, and the life arcs we paint in the moments of not knowing? And is that okay?
Fair warning, this post is probably mostly for my own benefit, but why not put it out on the internet for everyone :)
Here I am sitting in a café, studying Mandarin (traditional), when I'm struck by the thought (or maybe the vibe), am I just my skills? It was a moment where I felt the awareness of my current skillset, and thanks to the success of my company, awareness of the skill sets I don't have or which are nascent. I'm a great engineer, that I know; but I'm not a great sales person, nor a great org builder, nor am I a great manager (though I think I do pretty well). There was a time, common among engineers of the last thirty+ years I'm sure, that I thought "I don't need to sell, I don't need to market, I don't need to politic - I can just pursue the latent truth between technology and the market". ('"I can" and what?' being an obvious follow up)
The closer you are to these other roles, and the longer the arc of the Thing you are building and shepherding, the more you realize the incredible value of synthesis and alignment across these domains. I hate to give a point to Elon, but he was right about the value of capable generalists: often the framing of the problem is half the battle, and so many of my former peers saw everything that wasn't "the code / design" as a begrudging chore of the job. They liked the problem solving.
I love the problem solving too, but I was fortunate enough to learn early how flexible the problem definition really is in the real world. I was "lucky" to quit my first job at 25 to jump into a solo founded software startup. It was a terrible financial bet at the time, but it showed me that people pay for and use for the value, not the solution, and the value was a function of the problem for them. Don't mistake me - powerful solutions are also what reshape our value frontier, as they make previosuly ludicrous strategies or problem rotations "an insightful idea, actually". (why you don't settle for less than a 10x value increase when betting big on innovations)
I suppose you could say this gave me a small amount of "worldliness": I had to do more than pay lip service to the other components of building a business. I and to understand those domains, and to construct a believable strategy, which forced me to also see the bounds of my chosen domain (engineering) and its interactions with the others. Unfortunately, I was also taken in by things like "content marketing" as an answer for building commercial interest, and luckily have now learned that nothing beats pavement-pounding cold-calling sales blood sweat and tears. (most of why I'm not starting something solo any time soon)
All that to say, these domains forced me to construct a narrative space that my profession lived within, that connected it to "the way the world works". Now to land this plane: I think this largely explains why I am in the position I'm in now, compared to the positions of my peers at the time. I exposed myself to problems and growth-demands and perspectives that pointed the gradient of my growth here. Each challenge demanded growth, and the sum of those challenges plus "just quit my first job" me is now current me: my skills, my taste, my worldview, and yes, my shortcomings. And not in some "if you squint" way: my skillset and profession determine how I think about chores, and whether I hire cleaners or spend relationship conflict to solve it instead. It's how I think about what problems I should solve myself or delegate to others. It's how I prioritize what we plan to do, and what of those things later we defer based on new information. It's how I think about fixing the "god I need to get better at watering my plants" problem. It's what consumes my free time</a>. It's what whole regions of my brain have been optimized over years to understand, embed, and operate with. It's the topics of conversation that interest me most. You get the drift.
The easiest thing is to just keep going. There's an inconvenient version of it that is "go harder", leaning deeper into specific or more recent narratives. Something like "become a great leader", or "always do the right thing for your relationship and community" (though figuring that out is hard). Then there's the harder end of the space, truly questioning your mental model, incentives, and the opinions of those around you. There's a reason the latter is hard, and we mostly know why. This thought is freeing, in a way: once you have it, you can now strategically choose to walk that dimension of your growth/worldview. You can meta-policy your growth by picking a domain where you feel opportunity, and to decide to take more risks, inject more noise.
Oh, yeah - noise! Randomness. I should have mentioned that earlier. I think "fortunate noise" is what I was the beneficiary of in my "let's solo-found a company at 25!" adventure. At the time I did not understand the world enough to truly see the value of the experience it would give me. But it was an extremely high variance bet whose influence on the challenges and problems I saw day to day, month to month forced me to grow in a way I benefited from. Tangent: I want to be extremely clear that it isn't "the" way, and that there are likely many ways and directions that result in highly valuable growth. Almost the best thing you can have these days is "a really good problem".
I think this topic is on my mind these days because a big part of my identity is now under internal strain. Probably up until the pandemic, I held technology and software high: it was the literal lever we used to pull ourselves out of drudgery. I was ambivalent at the trash pile facebook had become, but software's (even commercial) value very much eclipsed its societal costs and risks. These days, that is obviously in question, and the arc of humanity looks to be headed for incredible tragedy or revolution (or both) because of the maladaptive incentives and circumstances it creates. (unfortunately "optimization" is at the heart of many of these systems) I now have to reconcile what I find to be beautiful from what I find to be horrific. I would never work for a weapons manufacturer because of the obvious direct harm. Now the same is true for Meta, and a growing list of other companies.
I think I need noise in a domain or on a dimension I can't yet predict. I fear naively becoming more "my skills". I yearn for inspiration in something that feels more true for humanity than what we have. Vive randomness, I suppose!

The Graph Structure of "Things Taking a Long Time"

2026-07-15T00:00:00+00:00

A recent project involved turning a manually bash-scripted loadsim into a CI-triggered deployment gate, which was parameterizable, handled system-under-test provisioning and cleanup, observable, and uploaded its results to block storage. You can probably tell where this is going: it took longer than expected. Why, though? By all accounts, the task is straight forward, and there are no "wow I'm not even sure how we'd do that" type subproblems involved. My take, though, is that graphs explain why, and particularly going from no graphs to many explicit graphs is the specific reason.
To set the stage, I'll provide the before and after:

Before

Just a bash script you run on your computer

seed data > launch SUT locally > seed data > warm up SUT > run the load test > tear down SUT > upload results</li> </ul> </li>
If something goes wrong you just debug and update the bash script</li>
You are responsible for interpreting the results</li>
No tests - "it continuing to work is the test"</li> </ul> </li>
After

This thing needs tests

This thing needs to be factored to allow DI to allow good testing

This thing needs to run as a set of state machines to truly factor per-stage responsibilities</li> </ul> </li> </ul> </li>
This thing needs to run in-cluster to point at the system-under-test-service directly

This thing needs to define its own k8s resources</li>
This thing needs to define the container it will run in and runtime deps that need to be packaged with it</li>
This thing needs to define the cluster resource relationships (taints etc) it relies upon to be isolated from other workloads</li> </ul> </li>
This thing needs to upload its results

This thing needs to be authenticated to block storage</li>
This thing needs to establish explicit storage structure for reports</li> </ul> </li>
This thing needs to gate PR approval based on specific configured performance expectations

This thing needs to log its status etc etc</li> </ul> </li> </ul> </li> </ul>
I'll stop there, but you can see how each of these things blows up. Were this "implicit graph" mode, we could just add lines to the bash script to do the marginal thing, and say "we aren't going to generalize this". It would be easy to break, as the dependencies between the concepts are completely implicit (manifest at runtime by what it does, not defined by and explicit model).
All of the pain and inaccuracy in estimation/planning came from the turning of implicit relationships into explicit graphs:

Creating the load sim state machine (and related concepts - oops we need a simulation state machine also!)</li>
Making the type/struct hierarchy that defines the config language of the loadsim process: before just a set of CLI args, now a modular collection of related config factors that couple loadsim profiles to literal application code interfaces. Higher optionality, but more complex also.</li>
TODO</li> </ul>
Why did we have to pay down this debt now? Well, we're going from "thing we run on our laptops sometimes" to "thing that runs every time we want to merge a related PR", where the latter has much higher quality expectations.

Structural Correctness

2026-06-27T00:00:00+00:00

Type systems are great: they catch huge swaths of bugs at compile time, and contribute to a grammar/vocab of forward development for your app. Like type systems, modern tools for verifying correctness leverage structural descriptions of the problem, via graph-structured model of the domain:

Type systems describe programs as a graph of types, functions, traits/etc, and their relationships. Types are nodes, with relationships like returns</code>, accepts</code>, implements</code>, etc as edges.</li>
Build systems like bazel have build targets as nodes and dependencies as edges between them. Without explicit deps stated, unstated packages are not provided to build targets even if they are built by other means, fulfilling hermiticity and trivial deployability.</li>
Interface description languages have messages types and services as nodes, with return types and parameter types as edges (a restricted type system).</li>
Databases use foreign keys (edges) to describe the relationship between related table rows (nodes). However, databases also use check contraints to implement correctness: not structural, but useful predicates to express aspects can't can't be described additively.</li> </ul>STRUCTURAL CORRECTNESS</div>
Model your domain as a graph of typed nodes and edges. The best of these graphs are definitional: the structure that describes the system is the system. No edge, no capability.</div>
Each domain has its own vocab of node and edge types. Bazel is a nice example of the richness via different edge types: deps means "link this code," srcs means "compile these files," data means "make available at runtime." Each is a different edge type in the target graph, and each constitutes a different capability. Elegant choice of node and edge types result in expressive and powerful tools. Many of these correctness tools leverage both structural correctness and predicates to describe valid configurations.
The best of these graph-structured systems are constitutive/definitional: they both define "what is valid" and "how outputs are computed". In bazel, a target can't import a module it doesn't dep</code>end on. This provides two valuable features: a) the verifiable invariant must be stated for the system to function, so you end up with a verifiable system by default, and b) it provides a brevity in specificaiton thanks to stated attributes and relationships doing double duty for behavior and validation. Building a bazel project forces you to describe the explicit structure of your product, and leaves you with a means to verify it as you continue to develop.
These two points about "each domain has its own nounds and verbs" and "definitional graph structured systems" are why bazel is such a good substrate for describing your whole product in: Bazel lets you define arbitrary node types for your domain concepts (packages, services, datasets), constrain their relationships via typed edges, and then derive your entire build/test/deploy pipeline from that single structural description. A service, its metrics endpoint, and the Prometheus deployment that scrapes it can all be nodes in the same graph — their relationships declared once and used to compile, deploy, and verify the whole system.
But what about state?</h2> I'm glad you asked! Databases define what can be stored via schema-specified tables, foreign keys, and check constraints. But outside of those they do not specify valid state transitions. This is often left to the application layer, where HTTP or RPC endpoints are relied on to faithfully implement the business logic. However, this is often where things go wrong: e.g. two endpoints that both modify order status, one forgetting to check inventory before marking it shipped</code>.Colored petri nets (CPNs) take the schema-specified collections (places/tokens</a>) and adds strict "next state" definition via transitions</a>. In CPNs, state only changes via a specific binding</a> being selected and its transition firing, consuming, mutating, and producing tokens in specific places. This is an applicaiton of graph-structured systems to state: places and transitions are nodes, and the arcs that connect them are edges. Like bazel, the concepts that define the functioning of CPN systems do double duty as means of verifying said system more completely. To demonstrate how CPNs do this, let's walk through a fairly complex example: a web scraper. I implemented the web scraper I described as an application of interest in the original CPN post</a>. In this example, the scraper needs to use a rotating set of proxies, and to implement "leasing" semantics for domains, to prevent overloading of a site, etc. Traditionally, building a stateful system like this means scattering coordination across multiple layers: a database handles resource state (via SELECT FOR UPDATE</code> and careful lock ordering), a task queue handles job scheduling, application code handles rate limiting and cooldown logic, and yet another component enforces domain concurrency limits. Each has its own incomplete view of correctness. And you increase reliability by writing check constraints, unit tests, and application-level predicates, and ensure they agree via trial and error. Proxy conservation isn't a property you test for, but a topological fact of the net. </blockquote> With a CPN, the scraper's entire coordination model is declared as tokens and transitions. Claiming a resource for scraping is a single transition (select_resource</code>) that binds across available resources, proxies, and domain states. It atomically leases a proxy, increments the domain's active count, and creates a session all in the same firing. There's no locking strategy to get right, because the binding system guarantees only valid combinations participate. Proxy conservation isn't a property you test for because it's a topological fact: a proxy token exists in exactly one place at any time (available_proxies</code> or leased_proxies</code>), and only declared transitions can move it. The resource lifecycle (available</code> → leased</code> → cooling</code> → available</code>) is the shape of the net itself, effectively a state machine. Also, it provides you with a cool animation of the net to show off (feed readers strip the embed below — view it full size</a>): </iframe> Full size</a> — the web scraper CPN, animated Personally, I'm bullish on this unification of state and transition: databases generally only handle the data + type side of this, plus predicate tools like check constraints. Where existing systems distribute state-related invariants across application code, unit tests, and database constraints (each with separate verification), CPNs unify these concerns and use them to define app function. As a new domain there's a lot of learning ahead about what kinds of bugs could emerge from these systems, or what kind of operations and performance trade-offs need to be made. Type systems are famously not-free. But I see an even larger amount of opportunity to go with it.
BECU OFX Connection 2026-02-01T00:00:00+00:00 (Written 2026-02-01) It sucks having to rely on Plaid or MasterCard Data Services to aggregate financial data, since they certainly sell this data in some way. I've been trying to get access to BECU programmatically, as they do expose an OFX API</a>, and after significant effort, I've recently succeeded in getting access! This little post tells you how to do it. The hardest part so far has been just getting BECU to respond in a non-generic error manner, and get it to initiate a connection as described in the link. You can initiate an app connection request using the python script below. Be aware that some details lie the version, app version, etc details may have changed since writing. And you'll need to populate env vars referenced at the start of the script - I generated a UUIDv4 for my BECU_CLIENTUID</code>. After running this and receiving a Please contact your Financial Institution. The CLIENTUID sent by the client was incorrect.</code> response, I was then able to find and authorize the OFX connection request in the BECU online banking connected services</a> page, where it showed up as Quicken for Windows</code> (wrong, but whatever). Now time to build out the full transaction and account state capture service! # main.py import urllib.request import urllib.error from datetime import datetime, timezone import os BECU_USERNAME = os.environ["BECU_USERNAME"] BECU_PASSWORD = os.environ["BECU_PASSWORD"] BECU_CLIENTUID = os.environ["BECU_CLIENTUID"] # Build raw OFX request now = datetime.now(timezone.utc).strftime("%Y%m%d%H%M%S") ofx_request = f"""OFXHEADER:100 DATA:OFXSGML VERSION:103 SECURITY:NONE ENCODING:USASCII CHARSET:1252 COMPRESSION:NONE OLDFILEUID:NONE NEWFILEUID:NONE <OFX> <SIGNONMSGSRQV1> <SONRQ> <DTCLIENT>{now} <USERID>{BECU_USERNAME} <USERPASS>{BECU_PASSWORD} <LANGUAGE>ENG <FI> <ORG>BECU <FID>3670 </FI> <APPID>QWIN <APPVER>2700 <CLIENTUID>{BECU_CLIENTUID} </SONRQ> </SIGNONMSGSRQV1> <SIGNUPMSGSRQV1> <ACCTINFOTRNRQ> <TRNUID>{now} <ACCTINFORQ> <DTACCTUP>19700101 </ACCTINFORQ> </ACCTINFOTRNRQ> </SIGNUPMSGSRQV1> </OFX>""" print("=== REQUEST ===") print(ofx_request) req = urllib.request.Request( "https://onlinebanking.becu.org/ofx/ofxprocessor.asp", data=ofx_request.encode("ascii"), headers={"Content-Type": "application/x-ofx"}, ) try: resp = urllib.request.urlopen(req, timeout=30) print("\n=== RESPONSE ===") print(resp.read().decode()) except urllib.error.HTTPError as e: print(f"\n=== ERROR {e.code} ===") print(e.read().decode()) </code></pre> The Everyday Salad 2026-01-08T00:00:00+00:00 I am a creature of habit and vegetarian, and needed a consistent and simple meal-prep option to have in the fridge for the week. Enter the Everyday Salad - a delicious and nutricious pile of beans and greens proven by years of experience to please at a party as easily as it does every lunch for a week. Ingredients</h2> Ingredient</th> Amount</th> Prep</th></tr></thead> Baby spinach</td> 5 oz</td> Chopped</td></tr> Baby arugula</td> 5 oz</td> Chopped</td></tr> Mikumame</td> 16 oz</td> </td></tr> Black beans</td> 15 oz</td> </td></tr> Corn</td> 15 oz</td> </td></tr> Grape tomatoes</td> 10 oz</td> Halved</td></tr> Red onion</td> 1/2 onion</td> Thin julienne or small dice</td></tr> Shredded carrot</td> 5 oz</td> </td></tr> Kalamata olive</td> 1 cup</td> Halved</td></tr> Mozerella</td> 9 oz</td> Cubed</td></tr> Sliced almonds</td> 2 oz</td> </td></tr> Craisins (optional)</td> Just a handful</td> </td></tr> Bell pepper (optional)</td> 1</td> Sliced</td></tr> </tbody></table> This should make about 8-10 individual servings, assuming servings are ~1/2 quart. Recommendations</h2> Dress with olive oil and salt.</li> Highly recommend chopping the greens, IMO this salad is best eaten with a spoon!</li> </ul> Nutrition</h2> Ingredient</th> Amount</th> Calories</th> Protein (g)</th> Net Carbs (g)</th> Fat (g)</th></tr></thead> Baby spinach</td> 5 oz</td> 42</td> 3</td> 2</td> 0</td></tr> Baby arugula</td> 5 oz</td> 30</td> 3</td> 3</td> 0</td></tr> Mikumame</td> 16 oz</td> 450</td> 50</td> 10</td> 16</td></tr> Black beans</td> 15 oz</td> 385</td> 25</td> 39</td> 0</td></tr> Corn</td> 15 oz</td> 245</td> 7</td> 35</td> 4</td></tr> Grape tomatoes</td> 10 oz</td> 47</td> 2</td> 7</td> 0</td></tr> Red onion</td> 1/2 onion</td> 32</td> 1</td> 7</td> 0</td></tr> Shredded carrot</td> 5 oz</td> 58</td> 2</td> 10</td> 0</td></tr> Kalamata olive</td> 1 cup</td> 490</td> 0</td> 0</td> 49</td></tr> Mozerella</td> 9 oz</td> 768</td> 63</td> 16</td> 50</td></tr> Sliced almonds</td> 2 oz</td> 360</td> 12</td> 3</td> 15</td></tr> </tbody></table> Ingredient</th> Calories</th> Protein (g)</th> Net Carbs (g)</th> Fat (g)</th></tr></thead> Total</td> 2907</td> 168</td> 131</td> 134</td></tr> Per Serving</td> 363</td> 21</td> 16</td> 17</td></tr> </tbody></table> Servings are 1/2 quart Threads - A context strategy for humans and LLMs 2025-12-29T00:00:00+00:00 You know what sucks? Coming back 2 months later to a project you started with a burning sense of purpose, and realizing you have no idea what the next step was, or even whether its still a good idea. (narratives are important!</a>) A similar need: capturing context, relevant links, and the "why" for enabling code-authoring agents to go solve a problem for you with high autonomy. Well, recently I've been reading Accelerando</a> (surprisingly relevant these days...) and it has an interseting through line: characters can fork instances of themselves to go do research, tasks, etc. It's a bit magical in the book, but very similar to typing claude</code> in your terminal and firing off a request. claude</code> is different from you in extremely important ways, but the analogy is good enough to be useful. They say that effective vibe coding/engineering is an exercise in context management. What I've found to be useful is using a document to manage this forking process, enabling re-entrance, and clear exit conditions - I call them threads</code>. Inside, you put the core bet (what's the hypothesis, the insight, inspiration, etc), relevant links, immediate next steps, the definition of done (to close the thread), and a log of significant decisions/actions. See the example template</a> below. I've found this is enough information to solve both needs from paragraph 1: I can cycle back onto this later and get the blurb that inspired 2-months-ago me, see what immediate next steps are, and regain momentum. (see It's all about momentum, innit?</a>)</li> I can essentially say "Hey, take a look at @threads/dev-box.md</a> - what solutions for declarative package installs should I consider?" to an LLM and get useful output.</li> </ol> This process/convention has proven useful in many situations: Peeling off threads when you're deep in another task, have discovered an important related task, and want to capture context you can come back to later without totally dumping your mental state.</li> Asking an LLM to capture significant decisions you've discussed (and their motivations) in its thread file during the agentic code authoring process.</li> As a place to accrue requirements for a soon-to-be-task to ensure you don't forget them later.</li> A place to refer to when you wonder why you made the decision you did.</li> </ul> In the code agent arena, this is a layer above the agent-generated plan. When done well, these should be high value density and be worth checking into your monorepo, allowing humans and code agents to utilize the narratives and decisions within as context. They seem to have a similar value to architectural decision records</a>, though I am admittedly no expert in those. Like all good things, the idea is simple and flexible: it's just a file containing the most important information regarding a task. And the information within is (mostly) all you need to know to close said thread. And it doesn't have to be coding related, I even have one for refactoring my physical desk to make it more modular! Appendix</h1> Example Template</h2> --- title: Thread Template created: 2025-12-15 --- # Bet REPLACE ME! See [motivating note](https://blog.sao.dev/2025-threads/). Establishes a format for encoding bets into threads that can be easily spawned and re-entered. This file acts as a template, with some self-reference for context. # Context REPLACE ME! - [Narratives and Better Bets](https://blog.sao.dev/narrative-and-better-bets/) - Tokens are expensive, keep this succinct # Next Steps REPLACE ME! This section is technically optional, but should exist as it contains the distillation of how to "get started" again. # Definition of Done REPLACE ME! This is more optional, but also important to add description of specific outcomes that represent payoff. Can be descriptions of test or use cases, or outcomes in the real world, etc. # Log ## 2025-12-29 - Drafted [motivating note](../dev/sao/notes/zt_2025-12-15_external-threads.md). - Implemented this template. </code></pre> CPNs, LLMs, and Distributed Applications 2025-12-19T00:00:00+00:00 A big theme in LLM-enabled software dev is that verifiable correctness makes it much easier to take bigger leaps with LLMs. E.g. tests, compilers, state machines, etc. While researching for databuild, I recently came across colored petri nets</a>, and instantly saw opportunity. Colored petri nets (CPNs) are an extension of petri nets. Petri nets are essentially directed bipartite graphs where places can contain tokens, and places are connected by transitions (where the side effects happen). In petri nets, a single token contains no data, and represents an identity-less tokens location in the net. Importantly, petri nets with transitions that have only one input and output and one token globally are equivalent to finite state machines. Colored petri nets extend this model, allowing individual tokens to have data associated with them. This allows CPNs to match closely to the Rust typestate pattern</a>, and suggests Rust may be able to implement CPN semantics easily. CPNs are of particular interest because a) it is still hard to write concurrent applications and b) provide the potential for formally verifying concurrent programs at build time. Also, if we can implement a high performance datastore underneath, a CPN framework might be able to handle the hard parts of concurrent applications: state synchronization, conflict detection, deadlock avoidance, and coordinating access to shared resources. These are enabled via two other features of CPNs: guards and multi-token consumption/production. A guard is a list of boolean conditions that may apply to a transition: things that must be true for a token to take that transition. For instance, there need to be more than 0 connections available in the connection pool to claim one. Multi-token consumption/production is just what it sounds like: a fork in the network via a transition, where P1 -> T1 -> (P2, P3), so T1 consuming a token from P1 produces a token in each of P2 and P3 simultaneously. Conversely, a join in the network via a transition, where (P1, P2) -> T1 -> P3, would require that both P1 and P2 to have tokens present that are able to transition via T1 into P3, which will also be simultaneous. One application that has light concurrency involved is web scraping with leased proxies and scrape targets. You have a limited number of proxies to use to proxy your requests, and need to rate limit your usage of all of them to ensure they don't over-request a given target. Also, you both want to avoid requesting the same target multiple times concurrently, and also avoid making requests to the domain too frequently, to be a responsible user. Traditionally, this is solved with leasing of resources via a central database, implemented via select for update</code> style semantics in the database. We can imagine implementing this with CPN semantics as having the scrape_target</code> transition being the join of the available_proxies</code> and prioritized_targets</code> places, with a scrape only starting when a proxy is available and a prioritized target is available. You can also imagine implementing other complex distributed scraper features with CPN semantics: Scrape target cooldowns: after being scraped, a target token transitions to a cool_down</code> state, where using a timed petri net allows us to delay transitioning back to prioritized_targets</code> after the delay period.</li> Domain-level rate limiting: add a separate domains</code> token, so that the scrape_target</code> transition is a 3-way join domains x available_proxies x prioritized_targets</code>.</li> Retry with backoff: Failed scrapes fork: one token to failed_log, one back to prioritized_targets with incremented retry count and a longer cooldown. Guard prevents retries beyond max attempts.</li> Result pipeline — Post-scrape, tokens flow through raw_html → parsed → validated → stored, each with its own concurrency limits, which naturally implements back pressure.</li> </ul> Another application is databuild, where the network is made up of partitions (which are effectively leased to job runs), wants, and job runs, and we would benefit from some self-organizing dynamic process to propagate the data deps and solve user stated partition wants in a safe, efficient, and fast manner. But more on that guy later. I'm still figuring this part out, but it seems like there are a few sensible or interesting strategies for implementing CPNs: As a traditional app + postgres combo, with transactions implementing simultaneous token movement and state storage, and select for update</code> enabling claiming of tokens for transitions to prevent double moves.</li> As a single-process Rust binary, with all CPN state stored in memory, enabling move semantics and the typestate pattern to implement the CPN semantics (this could be extremely fast, but need to figure out persistence - maybe snapshotted event log?)</li> </ol> One thing I'm trying to figure out is how to solve the partitiong problem: if app state grows to be too large for 1 server's memory, is there a way to automatically partition the network/tokens to enable fearless concurrency and horizontal scalability? So far the answers seem to be either a) solve in application with archive place/transition being part of the network or b) solving this at the database level. Or maybe something even crazier, where the whole network is made up of multiple CPN apps, which themselves expose query/consume interfaces? Tying it all back, if we can figure out a CPN app framework with sensible persistence, we could impose more compile-time constraints on agent-authored code (good for dev velocity) while also providing correctness guarantees and simulation capabilities out of the box that unconstrained computation apps don't provide. Exciting! Appendix</h1> Random Thoughts</h2> !! Is there an existing open source service that I can use the test suite from or benchmark against to validate this? Especially if it means just setting claude code off against it?</li> Timed petri nets - could we emit metrics by default and simulate the net impact of a change to the whole system?</li> Join transitions as literal joins in the database, with transition conditions implying indexes, and token transitions as inserts and deletes inside a transaction</li> What extent are we talking about just making a datastore or CPN database?</li> Distributed network of CPN apps? Or an app as a distributed network of CPN services? It's all about partitioning?</li> CPNs don't need explicit re-entrance? Every "step" is a movement of tokens, which is doable completely from the state of all tokens involved?</li> </ul> Potential Bets (LLM Written)</h2> To validate this, we need a real hill to climb: reimplement something with CPN semantics and compare. The core question isn't "can CPNs be fast/correct"—of course they can. It's: does the paradigm make writing simple, correct, and fast concurrent programs easier? Does the formalism pay for itself in reduced bugs and less bespoke coordination code? The bet: scraper scheduler (spider-rs) Implement the decision-making core of a scraper—URL prioritization, domain rate limiting, proxy assignment, retry scheduling—using CPNs. Mock the HTTP layer; benchmark decisions/sec and compare coordination code complexity against the original. Why scrapers? Traditional implementations are full of ad-hoc coordination: rate limit maps, cooldown trackers, retry counters, domain queues. This is exactly what CPNs formalize. Correctness matters (politeness violations get you banned), and timed transitions are central, not incidental. spider-rs is already Rust, so we can fork and compare directly. Why not the others first? Option</th> Why not</th></tr></thead> Job queue (Faktory)</td> Coordination is incidental; well-understood pattern where CPN overhead may not pay off</td></tr> Connection pooler (pgcat)</td> Too small—might not exercise enough CPN capabilities to be compelling</td></tr> CI runner (Buildkite)</td> Too large—incidental complexity (artifacts, logs, shell execution) obscures the coordination story</td></tr> </tbody></table> Implementation approach In-memory Rust with SQLite snapshots: single-threaded, move semantics apply, extremely fast. Partitioning = separate binaries with disjoint token ownership (enforceable at net-definition time via partition key requirements). Simulation = same CPN execution with mocked effects, enabling correctness testing, failure injection, and time fast-forwarding. Differentiable Systems 2025-11-11T00:00:00+00:00 Systems that are "differentiable": Can have performance or quality attributed to specific components, so that those components can be improved</li> Components have literal measures of quality so that progress in their quality is measurable</li> You can infer impact from one component onto others</li> </ul> OLD! "Kernel" Discussion</h1> What's a kernel? General: fundamental, irreducible core</li> Operating systems: core program that mediates access to hardware</li> Machine learning: a mapping from the domain space to an easily separable space</li> Linear algebra: the set of vectors that a matrix maps to zero</li> </ul> There is something tempting in this - to characterize a kernel as a mapping between domains or problem spaces. This is explicit in ML, but abstracted (or embedded in the domain) for software. In software, kernels mediate access to shared resources, and map the real constraints and interfaces of the hardware into a simpler interface and interaction model for applications and services. Zooming out a little, it is satisfying to imagine a complete application through this lens. For example, let's take Amazon retail: the incredible amounts of code that make it up are mappings of fractal-complex business situations and needs into interfaces that enable specific capabilities. At the highest level, this is turning the "purchase click" into a box on your doorstep. Breaking this down, its the sub components of the problem: checking that the item is still in stock, charging the card, routing delivery, etc. Individually this might not be too exciting. This is already explicit in ML, and a small hop from the "abstraction" concept already fundamental in software. I think the utility might be in the unification or synthesis - that we can consider different things (like Tension between general "fundamental core" definition and ML "useful mapping" definition?</li> The simpler your subspace, the simpler your kernel can be Example: library code vs application code</li> </ul> </li> What does "kernel of truth" mean? More similar to the math "maps to zero" definition? Neurons that fire together?</li> </ul> Japan Recs 2025-10-28T00:00:00+00:00 Meiji Jingu to Shibuya Walk</h1> Go to meiji jingu mae station</a> or Harajuku station</a></li> Optionally walk around Yoyogi park</a></li> Walk up to Meiji Jingu</a> (temple)</li> Explore temple and surrounding area</li> Walk over to takeshita street (this family mart https://maps.app.goo.gl/TECsJANVNe8eF1pYA)</li> Walk down takeshita street to this point and take a left on harajuku st https://maps.app.goo.gl/epknc7h4eyDWRumS8</li> Walk down to here https://maps.app.goo.gl/xgFcCJp73XpGAzCZ7</li> Take a right towards cat street</li> Take a right onto cat street here https://maps.app.goo.gl/f1KUiJkhyYrCJxhL8</li> Keep walking down cat street and sight seeing (good second hand stores!) https://maps.app.goo.gl/1vBgdoGBGZ1HYRjr5</li> Continue down to the north end of shibuya here https://maps.app.goo.gl/zLCUbuUftP8B5S9u8</li> Check out the view at shibuya sky https://maps.app.goo.gl/95hrd94RPgn4JsEv8</li> Go give the goodest boi hachiko a pet https://maps.app.goo.gl/2DXPfUKbc56dufsL8</li> Check out shopping around shibuya</li> Experience the shibuya scramble https://maps.app.goo.gl/JF55FkaG5ym1LTci8</li> Check out Loft in shibuya https://maps.app.goo.gl/heE9Bw83rneudqfu7</li> </ul> For google directions: Walking route</a> after exploring yoyogi / meiji jingu</li> </ul> Narrative and Better Bets 2025-06-22T00:00:00+00:00 One thing I love about Yobi</a> (my job) is one of its core values: make smart bets. It's one of those kinda reductive statements that requires a lot of context to be practically instructive, but fulfills its role in creating rhetorical focus. Everything we do is a bet. bets have risks, upsides, and downsides. Different bets interact and inform future bets. Given your portfolio of bets, what is your expected distribution of outcomes? Etc. These aren't the capped ludic bets of the casino, where all probabilities are known and there are no inter-bet interactions. They are the bets of the natural world, where bets build on each other and can lead to compounding results. Some bets have catastrophic downside (mostly bad bets), and some have incredible upside (some good bets). Narrative has a critical role in bet making: most good options are hidden to almost everyone - imagine the startup opportunity, which relies on key recent tech advancements, changes in market dynamics, a coalescing of talent, a unique framing of the problem, etc. Predicting success (binary or degree) for this venture relies on a lot of information that simply hasn't happened yet. However, narrative allows you to abstract and achieve much higher leverage over the problem/world, like our evolved brains allow pattern recognition in our senses 1</a>. Every step up the intelligence ladder represents another filter to remove irrelevant information, enabling efficient and effective planning. It allows you to take a hopelessly detailed and complex situation, and recognize and distill its core risks or opportunities. It helps you draw a (fuzzy) arc through time to the vague but potentially very beneficial outcome. And practically, it lets you recognize a good bet when you stumble upon it. Narratives and their pursuit are themselves bet generating processes, making the intentional cultivation of narrative a precursor for doing things of substantial significance. I think this lens is also useful for separating narrative from its close relative understanding. Understanding is also a noble pursuit, but appears more passive in its nature and effect. Narrative is about drawing the line into the unobserved space, and updating it when you get there. Narrative is instructive about where to focus, about what matters, and it is anti-fragile 2</a> to new information that disconfirms it, as we either discard it in favor of a new narrative, or build an even stronger narrative incorporating said new information. Narrative is required for rising above random chance, in terms of achieving significant outcomes. In fact, back at the start of 2023 before joining Yobi, many other companies wanted me to join to help them build LLM-powered applications. ChatGPT was starting to make real waves outside of "good enough" copy writing, and I even had significant experience in language modeling. But in the end I didn't, largely due to the narrative I had cultivated. The underlying technology was not differentiated, and my opportunity to differentiably impact any company that was more than a barnacle on the back of the LLM whales was minimal. Yobi, on the other hand, aligned almost completely with my views on ML and its relationship with the market, and was still early. Yobi bet that we could achieve privacy preserving personalization with manifold learning, allowing said information (not data) to be brought to bear in many applications currently served by fragmented offerings. Insightful, high leverage, and not compelling to those that aren't in the know. By comparison, LLMs and their derivatives? Obvious, high competition, with everyone and their mom's racing to the gold rush. ^1Max Bennett, A Brief History of Intelligence</a> - A retelling of the development of intelligence via critical capability steps bouncing between neuroscience and machine learning. </div> ^2Nassim Taleb, Antifragile</a> - Really incredible book. I have read it yearly ever since 2022. </div> Why Write 2025-06-21T00:00:00+00:00 Why Write</h1> Hey all, it's Stuart, probably showing up in your inbox. I'm back, writing my little thoughts, and would love if you'd lend an ear. What is it, a blog? A substack? Not exactly - I'm not building an audience, I'm not promoting anything - but I am hoping to inspire conversation and build meaning / crystalize ideas that are meaningful to me. Practically, I'll just be sending out emails whenever I write something or put together a photo album of some life experience, and want to use a direct channel to share it not mediated by capitalism. But why write? Well, it's obvious that being online isn't cool anymore. But really - there is a lot of meaning to make. You need to build narratives to understand, to make bets, and build community. I think a lot of the poverty of meaning we're experiencing in the last 5 years is from the systematic transfer of attention away to naively capitalist content platforms, away from flawed institutions (like journalists, movie makers, etc) that at least believed in their role in making meaning. This is a personal attempt to wrest that time and attention back, and if nothing else internalize responsibility for narrative building of my life and its relationship with others. And I hope you'll take part in this with me, please respond with your thoughts so we can build a narrative together. And be warned, it won't be polished - all that's real and natural isn't - but it will be true. In case you want to read the first real piece: [[zt_2025-06-22_narrative-and-better-bets]]</li> </ul>