The Graph Structure of "Things Taking a Long Time"

A recent project involved turning a manually bash-scripted loadsim into a CI-triggered deployment gate, which was parameterizable, handled system-under-test provisioning and cleanup, observable, and uploaded its results to block storage. You can probably tell where this is going: it took longer than expected. Why, though? By all accounts, the task is straight forward, and there are no "wow I'm not even sure how we'd do that" type subproblems involved. My take, though, is that graphs explain why, and particularly going from no graphs to many explicit graphs is the specific reason.

To set the stage, I'll provide the before and after:

• Before
  • Just a bash script you run on your computer
    • seed data > launch SUT locally > seed data > warm up SUT > run the load test > tear down SUT > upload results
  • If something goes wrong you just debug and update the bash script
  • You are responsible for interpreting the results
  • No tests - "it continuing to work is the test"
• After
  • This thing needs tests
    • This thing needs to be factored to allow DI to allow good testing
      • This thing needs to run as a set of state machines to truly factor per-stage responsibilities
  • This thing needs to run in-cluster to point at the system-under-test-service directly
    • This thing needs to define its own k8s resources
    • This thing needs to define the container it will run in and runtime deps that need to be packaged with it
    • This thing needs to define the cluster resource relationships (taints etc) it relies upon to be isolated from other workloads
  • This thing needs to upload its results
    • This thing needs to be authenticated to block storage
    • This thing needs to establish explicit storage structure for reports
  • This thing needs to gate PR approval based on specific configured performance expectations
    • This thing needs to log its status etc etc

I'll stop there, but you can see how each of these things blows up. Were this "implicit graph" mode, we could just add lines to the bash script to do the marginal thing, and say "we aren't going to generalize this". It would be easy to break, as the dependencies between the concepts are completely implicit (manifest at runtime by what it does, not defined by and explicit model).

All of the pain and inaccuracy in estimation/planning came from the turning of implicit relationships into explicit graphs:

• Creating the load sim state machine (and related concepts - oops we need a simulation state machine also!)
• Making the type/struct hierarchy that defines the config language of the loadsim process: before just a set of CLI args, now a modular collection of related config factors that couple loadsim profiles to literal application code interfaces. Higher optionality, but more complex also.
• TODO

Why did we have to pay down this debt now? Well, we're going from "thing we run on our laptops sometimes" to "thing that runs every time we want to merge a related PR", where the latter has much higher quality expectations.