It is always helpful, when introducing something new, to be able to measure it against a familiar yardstick. When engines were first invented, their power was expressed as a multiple of the power of horses, and horses remain the comparator even now – highly sophisticated motorcars are still advertised by reference to the number of carthorses it would take to generate the same power output. We help each other to picture dimensions – height, length or area – by reference (in England at any rate) to Nelson’s Column, a London bus or a football pitch. I have heard document volumes expressed as “ESBs”, that is, the number of Empire State Buildings they would make if stacked (1 ESB = 7.57575758 Nelson’s Columns in case you wondered). We still refer to a “Gold standard”, although gold ceased to be the common medium of international exchange in 1971.
It is generally accepted by lawyers that the gold standard for accuracy of document review is reading by humans. For many lawyers, this is the standard to which they aspire and which they feel their duty requires of them. This is not the same as turning their backs on electronic review – they may be happy to conduct their review on the screen rather than on paper but are unwilling to delegate to a machine the task of deciding which documents must be reviewed and what decisions are made about them. It only when they get a case which cannot possibly be handled on this basis, that is, cannot be culled and filtered by humans, that they turn to technology.
This gives rise to the curious logical position (or, rather, the curiously illogical position) that you only apply your mind to more economic ways of working, and apply an allegedly “lesser” quality standard, for the biggest cases, and only because it is physically impossible to use the so-called gold standard. For other cases (if you follow this logic through) it is just fine to use uneconomic ways of working. The result, inevitably, is that there is not much mid-sized litigation going on. In any event, the idea that humans provide the gold standard for review depends on an assumption that the humans are of even quality, and remain so from Monday morning to Friday afternoon, whereas in fact (being only human) they are variable in quality as between each other and inconsistent in focus and application depending on whether they slept well last night, what they washed their lunches down with, and how their love lives are going.
If I had time (a week, or so) I would give you a survey of all the factors involved here. Instead, and focusing narrowly on the “man against the machine” point, I will refer you to a couple of papers which Equivio has produced reporting on tests which they have undertaken using their latest product, Equivio>Relevance. One is called How to jump start early case assessment: a case study based on Trec 2008. The other is a case study done jointly with Epiq Systems called Am Law 100 firm uses Equivio relevance to find more relevant documents and to find them faster. Both are to be found on Equivio’s downloads page .
The element common to both these papers is the mixture of human skill with technology’s speed. The Equivio>Relevance process begins with an assessment by a human “expert” as to the relevance of a sample – the Epiq Case study explains how this works and what follows. The trials involved a human review team and a human “Oracle” to resolve conflicting views (and thus assess the quality of the review). The horse power came from the Equivio application.
Henry Ford said that if he had asked his customers what they wanted they would have asked for a faster horse. Instead, he gave them the mass-produced motorcar, preserving the familiar idea of horsepower to root his new technology in what they were used to, but soon generating more applied power than could be achieved by harnessing any number of horses. All those horses would have needed a lot of stabling and feeding, in order to be available to provide the pulling power of the car. The parallel between that and a stable of manual reviewers holds good up to a point, but instead of developing it now, I will get out of the way (you have an ESB of documents to review) and let you read the Equivio papers.