Written by Dan O'Day, Chief Executive Officer, ECFX
For context, I live at the intersection of two worlds. I am both a lawyer and a programmer. Long before artificial intelligence became dinner table conversation or venture capital's favorite buzzword, I was already fascinated by the question of whether machines could ever truly think.
In 1989, I wrote my college thesis on artificial intelligence and wrestled with a deceptively simple question: could a machine think? The first step in framing that question was defined by Alan Turing when he asked, "Could a machine ever pass for human?" This became known as the Turing Test: the idea that if a person could not reliably distinguish a computer from a human through conversation alone, the machine could reasonably be said to "think."
At the time, I predicted in my thesis that if AI ever crossed that threshold, it would not come from traditional programming. It would come from neural networks. That prediction turned out to be largely correct.
What is a Neural Network
For those without a background in AI, neural networks attempt to model, albeit in simplified form, how we believe the human brain works. Unlike traditional software, which follows explicit instructions written line by line, neural networks learn by adjusting internal weights through experience. Instead of telling the computer exactly what to do, you expose it to enormous amounts of data and train it with feedback, allowing patterns to emerge.
I also argued in that thesis, and this is critical to understanding the position I am taking here, that systems built this way would struggle with many of the things at which computers traditionally excel . Classical software is precise. It performs arithmetic flawlessly and executes logic exactly as written. Neural networks, by contrast, would make mistakes just like humans do. What I did not anticipate was that they would also make mistakes no human ever would, and only recently have I come to understand why.
It is worth remembering how early those days were. Backpropagation, which is the training method that ultimately unlocked modern deep learning, had only recently been popularized academically and was still largely impractical given the computing power available at the time. To understand neural networks, it helps to start with ourselves. Humans do not interact with the world through keyboards and mice. We exist inside a constant flood of sensory input, including sight, sound, touch, smell, taste, and many others we rarely think about consciously. Thousands of signals arrive simultaneously. Our brains weigh those signals against prior experience, reinforcing some pathways while suppressing others.
Learning is simply the adjustment of those weights over time. If enough signals reinforce a pathway, neurons continue firing. If they do not, the signal fades. Eventually, an output emerges, for example, words spoken in response to a question. Even speech itself is nothing more than coordinated muscle movements shaped by countless prior experiences.
A neural network works on a similar principle. Inputs arrive simultaneously. Internal pathways assign weights to those inputs. Through training, those weights are adjusted again and again until useful outputs begin to emerge.
An analogy I often use is a vertical machine filled with balls, like a more complex version of a pachinko machine. Imagine dropping balls into the top. Each ball has a different size and weight, and each has a word written on it. Inside the machine are countless channels, levers, and gates. Depending on their size and weight, the balls travel different paths. Some get stuck. Some trigger mechanisms release new balls. Others fall freely.
Every interaction changes which balls are released next. Eventually, a sequence of balls falls out of the bottom. To a human observer, those balls might form a sentence. Your job as the operator is not to handcraft every possible response. Instead, you adjust the internal mechanisms so that when a certain collection of balls, perhaps in the form of a question, enters the machine, a meaningful sequence emerges on the other side.
Modern AI works remarkably like that machine. And like the machine, it does not understand the sentence the way we do. It simply learned which paths tend to produce results that humans recognize as meaningful.
My Hypothesis
With this background as a foundation, I hypothesize that large language models (LLMs) work best when two conditions exist: first, a testing mechanism to validate the output, and second, a quick feedback loop to correct its behavior. Both of these conditions exist for writing code in a way that they simply do not for legal research.
Code must work in a way that is far more precise than human language. First, it must be successfully executed by running it through a compiler or processor that catches the slightest syntax error. Then it can be tested in ways that human-written words cannot because code has deterministic, verifiable outputs.[NQ2] Tests can be automated and repeated and the pass/fail result are objective. If it fails either of these steps, we can tell the model to try again. The feedback loop is tight, concrete, and nearly immediate.
Case law and legal arguments are far more subtle and sophisticated than any piece of code. In our common law system, they are about persuading a human that your position is correct based on the concept of precedent. This means researching prior cases and using them as the foundation for your argument. In contrast to a coder debugging his software as it is written, a legal brief is evaluated by a human judge, and that evaluation often comes months after the document was created. And even if the judge does not agree with your position, it may still have been a strong argument. As a practical matter, meaningful feedback on a legal document never comes back to the AI model in any useful timeframe.
Legal AI models are trained on what has worked in the past, such as strong briefs. This mirrors the early days of training AI to write code by training it with good examples. Back then, it made terrible mistakes of the type we rarely see in code today, but that we still see daily in legal research.
The two main sources of error are hallucinated cases: in other words, case law that simply does not exist or misinterpretation of what a case actually stands for. For non-lawyers, determining what a judge's written opinion stands for is genuinely complicated because only the language directly relevant to the final outcome carries legal weight. Additional language is called dicta (more formally, obiter dicta), and cannot be cited as legal precedent.
To illustrate, a judge may make a passing remark that landlords should be presumed at fault as background reasoning used to explain the defendant's state of mind. If the case has nothing to do with landlords, that statement is dicta. That case cannot be cited as having established that landlords are presumed at fault. AI does not understand this distinction, and while legal models are improving, they continue to struggle with this issue.
The legal research model is rewarded for finding what you are looking for. By "rewarded," I mean reinforced in the neural network sense. The channels that produce a positive result are strengthened, and those that produce a negative result are weakened. This creates a model that is eager to give you what you asked for, even when the underlying reasoning does not support it.
I have spoken with several people in the legal industry who use AI tools regularly for legal research. I work mostly with people in Managing Attorney, Docketing, Knowledge Management, and senior IT level positions. I have also spoken to friends who still practice law and are involved in litigation. Most use legal-specific platforms like Harvey and CoCounsel. A few use the more general tools like Chat GPT and Claude.
They all agree that, as a practical matter, legal research has become more difficult and time-consuming with AI. While AI has improved, and the citations are more accurate, the people I speak with say the interpretation of case holdings is often incorrect. So often, in fact, that the output cannot be trusted. Several are talking about giving up on the tools altogether. Others only use them when clients specifically require it, which ironically may be driving up billable hours on those very cases.
This stands in sharp contrast to my experience at ECFX and across the broader software community. AI makes coding dramatically more productive. Weeks become days, and days become hours. Our programmers are at least 50% more productive, and some are seeing gains of 200% or more. The reason we don’t see a 7X or 8X increase is that it takes time to write very clear prompts and the AI still makes mistakes. The contrast between writing code and legal work product could not be starker. In software, AI has become a genuine force multiplier, compressing months into weeks, weeks into days.
The compiler does not care about intent or effort. The code either runs or it does not, and that binary honesty has shaped these models into something truly useful. In law, there is no compiler. There is a judge, a courtroom, and a client whose life or livelihood may depend on the argument you put in front of them. The same feedback loop that makes AI trustworthy in code is the one that is almost entirely absent in law. A hallucinated case does not throw an error. A misread holding does not fail a test. It simply looks right, and by the time anyone discovers the problem, the brief has already been filed.
I have spent decades at the intersection of these two disciplines, and my conclusion is not that AI in legal is bad or that the law is somehow beyond the reach of technology. My conclusion is more specific than that: a tool is only as trustworthy as the mechanism we have to check it. We have that mechanism for code. We do not yet have it for law. And until we do, the cost of being wrong is not a failed build you fix before lunch. It is a failed client you can never make whole.