Lab biologists should run the analysis

The fact is with today’s leading AI models you can do most any bioinformatic work without knowing a single thing about writing code. This is not a bad thing! It’s only been in the last few decades as the sheer scale of genome-wide measurments began to require dedicated skillsets in running computers to analyze them that scientists (like me) began to specialize. A divide formed between “dry” and “wet” lab roles. Wet lab biologists spent their time optimizing lab protocols, or designing experiments to generate data. When they needed data analyzed that couldn’t fit in your typical excel spreadsheet they turned to their dry lab counterparts who trained in things like multiple testing correction and analysis pipeline engineering. Today, AI’s ability to write working code in seconds has handed wet lab scientists back all the tools they need to analyze their largeest, most complex datasets from the very start to finished product. Perhaps someday robotic lab equipment and automation will do the same for us dry lab folks…but that’s for another post.

At the risk of alienating all my computational colleagues I’d argue that biologists thinking deeply about their scientific questions and collecting the data are in fact the best suited to be running the analysis through AI themselves. They have the deep domain knowledge that any generalist model will inevitably lack, and they can use it to cut through the professional-sounding noise that a model often produces when given an unfocused task or ill-formed question. Scientists are, in fact, speifically trained to ask strong, focused, and verifiable questions. Coincidentally, this is exactly what an AI needs to be given in order to help produce helpful results in the process of data analysis. What’s more, scientists know their data best. Giving them the tools to successfully analyze a dataset from start to finish avoids the dilution or “thrown over the wall” problem that plagues current collaborations where the norm is outsourcing this work to separate team members or worse…a core service group.

AI alone is a poor fit for science

But AI models with their stochastic, overly-confident nature and their transient lifespan can be unpredictable. They are by definition not reproducible which doesn’t sit well with how we do scientific work. Computational analysis, much like work in the lab, needs to be documented in a way that anyone could replicate it, it has to be reproducible. So what happens if your AI agent approaches a data anlysis problem you gave it in an unusual or unconventional way when faced with an initial roadblock, cutting a few corners but still producing something that looks and sounds like it could be correct? Or maybe an agent successfully processes your data, but then the session ends and all trace of how the outputs were generated is lost with the model’s death. Or worse, what if the agent asserts something about your data driven entirely by a hallucination and then builds off of that for the next few hours wasting everyone’s tokens, time, and grant money. You look at the spreadsheet it generated at the end and the numbers are completely wrong, or maybe you can’t even tell. At this point there is no way to answer the question: how did you do this? And without the knowledge of an experienced analyst it’s also very hard to answer the question why did you do it this way?

Scientists need a way to leverage the automation, the strong logical thinking and the significant programming ability AI brings to the table. But it needs to be in a way that is guided and constrained so that even those with no coding or technical knowledge of the underlying analysis work can drive the work forward in a responsible way.

The Commandments of Bioinformatic Agentic Pipelines

In the next few posts I’ll be introducing what I call the “Commandments of Bioinformatic Agentic Pipelines”. These are simple rules for how to use AI and autonomous agents in biological research and data analysis in a way that hopefully minimizes the risks I outlined above to reproducibility and reliability. My field is biology so I’ll be rooting things there, but much of what I’ll discuss is probably applicable to the wider research community.

Stringency is what I’m calling the engine that I’m building to force LLM models to hold to these principles. It’s a little like the belay system used in top rope climbing. In this case the agent is doing all the climbing, and you’re the one on the ground responsible for making sure it doesn’t fall off and break anything. It’s pretty hard to do that without a pulley and some rope.

We are all still learning how to use these new rapidly evolving tools, so stick around to hear more.