An essential step to improving quality and efficiency of eDiscovery is to measure the effectiveness of the current process. Without measurement, you cannot evaluate the effect that new technology and process changes will deliver.
For years, the eDiscovery industry has focused on review speed (number of documents reviewed per hour). There is no doubt that the speed of review is important. But, the “responsive rate” is the most important metric when it comes to eDiscovery cost containment – and is commonly overlooked.
The “responsive rate” represents the percentage of the reviewed documents that are responsive to the discovery request. The simple formula for determining the responsive rate is:
Total Responsive Documents / Total Documents Reviewed
Assuming that the documents submitted for legal review are the result of search protocols, the responsive rate is essentially the same as the search “precision” measure referred to by search technologists.
While far from scientific, we have made a point over the last six months to ask people about their general experience in “responsive rates.” Lawyers consistently report responsive rates in the 20% range, with many attorneys noting that responsive rates are often much lower – even single digit responsive rates in some cases. Indeed, many firms typically assume a 20% responsive rate for reviews when they prepare internal project budgets.
A completed review with a responsive rate of 20% means that 80% of the total cost of review was spent paying lawyers to review irrelevant documents. If we were to accept the analysts’ $10 billion dollar estimate of the annual amount incurred in legal review costs (See source), and we assume that the average responsive rate is 20%, that means that companies are paying approximately $8 billion dollars a year for lawyers to review irrelevant documents. As a profession we can do better.
Low responsive rates commonly result from the ineffective search techniques that have become standard practice in e-discovery. In many cases, a list of terms, phrases and Boolean queries are drafted and submitted to vendors to serve as the filter to govern what documents are assigned for expensive review. The blunt search terms are often created without access to the collected data and without an understanding of how the search terms functions against the data set.
The search term list is also permeating into the enterprise as clients increasingly have the in-house capabilities of basic searching through their internal data stores. It is common for this approach to decrease the “responsive rate” because not only is the search simplistic, the search is often run against irrelevant custodians thereby introducing a whole new level of irrelevant data into the review stage.
There are three main points that counsel should keep in mind as they consider addressing responsive rates to control costs.
First, lawyers should always measure the responsive rate to understand the efficiency of the process. Taking a sample of documents returned by the search prior to full review will allow the legal team to estimate the “responsive rate” and improve the search protocol if the percentage of irrelevant documents is high. Every corporate counsel should ask their lawyers to report on the responsive rate after the completion of a review so that they can evaluate how much money was spent on the review of irrelevant documents.
Second, the search and analysis process must be iterative. Search is not a single event - it is a learning process. Few if any lawyers I know would conduct legal research by creating a single list of terms that may appear in the relevant case law and then read every opinion that contains any one of the terms. To the contrary, lawyers are trained to search iteratively, learning from the cases returned by the search results, refining the search and exploring the relationships among the legal opinions. E-discovery is no different. Lawyers should explore the search results to understand the context of search hits and intelligently refine the search. A little work will pay dividends in improving the responsive rate and controlling costs.
Third, the review must be connected with the search phase so that the legal team has access to the litigation hold data to intelligently refine and test the search protocols. A single unified platform that makes an iterative search and review process practical for diligent counsel is essential to achieving cost containment.
Servient’s Descriptive AI technology is a significant leap in the direction of improved responsive rates. Once the legal team has refined the search protocol, Servient’s AI technology kicks in and begins to further separate the relevant and irrelevant documents. By learning from the lawyer's work product early in the process, Servient’s AI technology can substantially improve the responsive rate and fundamentally change the economics of e-discovery. Measure it.
