Douglas Oard from the iSchool at the University of Maryland is speaking on "Evaluating E-Discovery Search: The TREC Legal Track".
Douglas Oard is an Associate Professor at the University of Maryland, College Park, with joint appointments in the College of Information Studies and the Institute for Advanced Computer Studies. He earned his Ph.D. in Electrical Engineering from the University of Maryland, and his research interests center around the use of emerging technologies to support information seeking by end users. Dr. Oard’s recent work has focused on interactive techniques for cross-language information retrieval and techniques for search and sense-making in conversational media. He serves as a track coordinator for the TREC Legal Track. Additional information is available at http://www.glue.umd.edu/~oard/.
Civil litigation relies on each side making relevant evidence available to the other, a process known in the USA as “discovery” (and in the U.K. as “disclosure”). The explosive growth of information in digital form has led to an increasing focus on how search technology can best be applied to balance costs and responsiveness in what has come to be known as “e-discovery.” This is now a $4 billion USD business, one in which new vendors are entering the market frequently, usually with impressive claims about the efficacy of their products or services. Courts, attorneys, and companies are actively looking to understand what should constitute best practice, both in the design of search technology and in how that technology is employed. In this talk I will provide an overview of the e-discovery process, and then I will use that background to motivate a discussion of which aspects of that process the TREC Legal Track is seeking to model. I will then spend most of the talk describing two novel aspects of evaluation design: (1) recall-focused evaluation in large collections, and (2) modeling an interactive process for “responsive review” with fairly high fidelity. Although I will draw on the results of participating teams to illustrate what we have learned, my principal focus will be on discussing what we presently understand to be the strengths and weaknesses of our evaluation designs.
Douglas Oard is an Associate Professor at the University of Maryland, College Park, with joint appointments in the College of Information Studies and the Institute for Advanced Computer Studies. He earned his Ph.D. in Electrical Engineering from the University of Maryland, and his research interests center around the use of emerging technologies to support information seeking by end users. Dr. Oard’s recent work has focused on interactive techniques for cross-language information retrieval and techniques for search and sense-making in conversational media. He serves as a track coordinator for the TREC Legal Track. Additional information is available at http://www.glue.umd.edu/~oard/.
Civil litigation relies on each side making relevant evidence available to the other, a process known in the USA as “discovery” (and in the U.K. as “disclosure”). The explosive growth of information in digital form has led to an increasing focus on how search technology can best be applied to balance costs and responsiveness in what has come to be known as “e-discovery.” This is now a $4 billion USD business, one in which new vendors are entering the market frequently, usually with impressive claims about the efficacy of their products or services. Courts, attorneys, and companies are actively looking to understand what should constitute best practice, both in the design of search technology and in how that technology is employed. In this talk I will provide an overview of the e-discovery process, and then I will use that background to motivate a discussion of which aspects of that process the TREC Legal Track is seeking to model. I will then spend most of the talk describing two novel aspects of evaluation design: (1) recall-focused evaluation in large collections, and (2) modeling an interactive process for “responsive review” with fairly high fidelity. Although I will draw on the results of participating teams to illustrate what we have learned, my principal focus will be on discussing what we presently understand to be the strengths and weaknesses of our evaluation designs.
Comments