Bad Science and the Law – Crime Labs Under Fire in National Study
May 26, 2009
A report issued in February by the National Academy of Sciences identifies bad science being used by the courts in criminal trials. The report says crime lab analyses of physical evidence is often handled by poorly trained technicians who then exaggerate the accuracy of their methods in court.
The report said that Ballastics, Fingerprint, Bite Mark, and Tool Mark evidence is not grounded in the kind of rigorous, peer-reviewed research that is the hallmark of classic science
The report recommends the establishment of a federal agency to finance research and training and promote universal standards in forensic science, a discipline that spans anthropology, biology, chemistry, physics, medicine and law. The report also calls for tougher regulation of crime laboratories.
Crime laboratories should be managed separately from police departments to ensure that their findings are protected from bias, the report said.
Judges, lawyers and juries are on notice that high-tech forensic perfection is a television fantasy, not a courtroom reality
The New York Times – Feb. 2009
Forensic evidence that has helped convict thousands of defendants for nearly a century is often the product of shoddy scientific practices that should be upgraded and standardized, according to accounts of a draft report by the nation’s pre-eminent scientific research group.
The report by the National Academy of Sciences is to be released this month. People who have seen it say it is a sweeping critique of many forensic methods that the police and prosecutors rely on, including fingerprinting, firearms identification and analysis of bite marks, blood spatter, hair and handwriting.
The report says such analyses are often handled by poorly trained technicians who then exaggerate the accuracy of their methods in court. It concludes that Congress should create a federal agency to guarantee the independence of the field, which has been dominated by law enforcement agencies, say forensic professionals, scholars and scientists who have seen review copies of the study. Early reviewers said the report was still subject to change.
The result of a two-year review, the report follows a series of widely publicized crime laboratory failures, including the case of Brandon Mayfield, a lawyer from Portland, Ore., and Muslim convert who was wrongly arrested in the 2004 terrorist train bombing in Madrid that killed 191 people and wounded 2,000.
American examiners matched Mr. Mayfield’s fingerprint to those found at the scene, although Spanish authorities eventually convinced the Federal Bureau of Investigation that its fingerprint identification methods were faulty. Mr. Mayfield was released, and the federal government settled with him for $2 million.
In 2005, Congress asked the National Academy to assess the state of the forensic techniques used in court proceedings. The report’s findings are not binding, but they are expected to be highly influential.
“This is not a judicial ruling; it is not a law,” said Michael J. Saks, a psychology and law professor at Arizona State University who presented fundamental weaknesses in forensic evidence to the academy. “But it will be used by others who will make law or will argue cases.”
Legal experts expect that the report will give ammunition to defense lawyers seeking to discredit forensic procedures and expert witnesses in court. Lawyers could also use the findings in their attempts to overturn convictions based on spurious evidence. Judges are likely to use the findings to raise the bar for admissibility of certain types of forensic evidence and to rein in exaggerated expert testimony.
The report may also drive federal legislation if Congress adopts its recommendations. Senator Richard C. Shelby, Republican of Alabama, who has pushed for forensic reform, said, “My hope is that this report will provide an objective and unbiased perspective of the critical needs of our crime labs.”
Forensics, which developed within law enforcement institutions — and have been mythologized on television shows from “Quincy , M.E.” to “CSI: Miami” — suffers from a lack of independence, the report found.
The report’s most controversial recommendation is the establishment of a federal agency to finance research and training and promote universal standards in forensic science, a discipline that spans anthropology, biology, chemistry, physics, medicine and law. The report also calls for tougher regulation of crime laboratories.
In an effort to mitigate law enforcement opposition to the report, which has already delayed its publication, the draft focuses on scientific shortcomings and policy changes that could improve forensics. It is largely silent on strictly legal issues to avoid overstepping its bounds.
Perhaps the most powerful example of the National Academy’s prior influence on forensic science was a 2004 report discrediting the F.B.I. technique of matching the chemical signatures of lead in bullets at a crime scene to similar bullets possessed by a suspect. As a result, the agency had to notify hundreds of people who potentially had been wrongfully convicted.
In its current draft report, the National Academy wrote that the field suffered from a reliance on outmoded and untested theories by analysts who often have no background in science, statistics or other empirical disciplines.
Although it is not subject to significant criticism in the report, the advent of DNA profiling clearly set the agenda. DNA evidence is presented in less than 10 percent of all violent crimes but has revolutionized the entire science of forensics. While DNA testing has helped to free more than 200 wrongfully convicted people, “DNA was a shock to police culture and created an alternative scientific model, which promoted standardization, transparency and a higher level of precision,” said Paul Giannelli, a forensic science expert at Case Western Reserve University School of Law who presented his research to the National Academy. Enforcement officials, Mr. Giannelli said, “chose to say they never make mistakes, but they have little scientific support, and this report could blow them out of the water.”
Peter J. Neufeld, a co-director of the Innocence Project, a nonprofit group that uses DNA evidence to exonerate the wrongfully convicted, presented to the academy a study of trial transcripts of 137 convictions that were overturned by DNA evidence and found that 60 percent included false or misleading statements regarding blood, hair, bite mark, shoe print, soil, fiber and fingerprint analyses.
The courts have long struggled with the proper role of scientific evidence. In a 1993 landmark decision, Daubert v. Merrell Dow Pharmaceuticals, the Supreme Court held that scientific testimony had to meet an objective standard. Federal courts have occasionally excluded evidence like handwriting and hair analysis.
Donald Kennedy, a Stanford scientist who helped select the report’s authors, said federal law enforcement agencies resented “intervention” of mainstream science — especially the National Academy — in the courts.
He said the National Institute of Justice, a research arm of the Justice Department, tried to derail the forensic study by refusing to finance it and demanding to review the findings before publication. A bipartisan vote in Congress in 2005 broke the impasse with a $1.5 million appropriation.
Mr. Shelby also accused the National Institute of Justice of trying to infiltrate the forensic study panel with lobbyists for private DNA analysis companies, who were seeking to limit the research to DNA studies.
The National Institute of Justice said it would not comment until the report was released. But a preview of potential turf wars played out in the presentations to the National Academy in December 2007. A forensic expert from the Secret Service blasted the F.B.I. for developing questionable techniques “on an ad-hoc basis, without proper research.”
He said the Secret Service wanted the National Academy “to send a message to the entire forensic science community that this type of method development is not acceptable practice.”
Everyone interviewed for this article agreed that the report would be a force of change in the forensics field.
One person who has reviewed the draft and who asked not to be identified because of promises to keep the contents confidential said: “I’m sure that every defense attorney in the country is waiting for this report to come out. There are going to be challenges to fingerprints and firearms evidence and the general lack of empirical grounding. It’s going to be big.”
Published: May 11, 2009
It was time, the panel of experts said, to put more science in forensic science.
A report in February by a committee of the National Academy of Sciences found “serious problems” with much of the work performed by crime laboratories in the United States. Recent incidents of faulty evidence analysis — including the case of an Oregon lawyer who was arrested by the F.B.I. after the 2004 Madrid terrorist bombings based on fingerprint identification that turned out to be wrong — were just high-profile examples of wider deficiencies, the committee said. Crime labs were overworked, there were few certification programs for investigators and technicians, and the entire field suffered from a lack of oversight.
But perhaps the most damning conclusion was that many forensic disciplines — including analysis of fingerprints, bite marks and the striations and indentations left by a pry bar or a gun’s firing mechanism — were not grounded in the kind of rigorous, peer-reviewed research that is the hallmark of classic science. DNA analysis was an exception, the report noted, in that it had been studied extensively. But many other investigative tests, the report said, “have never been exposed to stringent scientific scrutiny.”
While some forensic experts took issue with that conclusion, many welcomed it. And some scientists are working on just the kind of research necessary to improve the field. They are refining software and studying human decision-making to improve an important aspect of much forensic science — the ability to recognize and compare patterns.
The report was “basically saying what many of us have been saying for a long time,” said Lawrence Kobilinsky, chairman of the department of sciences at John Jay College of Criminal Justice in New York. “There are a lot of areas in forensics that need improvement.”
Barry Fisher, a past president of the American Academy of Forensic Sciences and a former director of the crime laboratory at the Los Angeles County Sheriff’s Department, said he and others had been pushing for this kind of independent assessment for years. “There needs to be a demonstration that this stuff is reliable,” he said.
It’s not that there hasn’t been any research in forensic science. But over the years much of it has been done in crime labs themselves. “It hasn’t gotten to the level where they can state findings in a rigorous scientific way,” said Constantine Gatsonis, director of the Center for Statistical Sciences at Brown University and co-chairman of the National Academy of Sciences committee. And rather than being teased out in academic papers and debated at scientific conferences, “a lot of this forensic stuff is being argued in the courtroom,” Mr. Fisher said. “That’s not the place to validate any kind of scientific information.”
Much forensic research has been geared to improving technologies and techniques. These studies can result in the kinds of gee-whiz advances that may show up in the next episode of the “C.S.I.” series — a technique to obtain fingerprints from a grocery bag or other unlikely source, for example, or equipment that enables analyses of the tiniest bits of evidence.
This kind of work is useful, Dr. Kobilinsky said, “but it doesn’t solve the basic problem.”
DNA analysis came out of the biological sciences, and much money and time has been spent developing the field, resulting in a large body of peer-reviewed research. So when a DNA expert testifies in court that there is a certain probability that a sample comes from a suspect, that claim is grounded in science.
As evidence to be analyzed, DNA has certain advantages. “DNA has a particular structure, and can be digitized,” Dr. Gatsonis said. So scientists can agree, for example, on how many loci on a DNA strand to use in their analyses, and computers can do the necessary computations of probability.
“Fingerprints are a lot more complicated,” Dr. Gatsonis said. “There are a lot of different ways you can select features and make comparisons.” A smudged print may have only a few ridge endings or other points for comparison, while a clear print may have many more. And other factors can affect prints, including the material they were found on and the pressure of the fingers in making them.
Sargur N. Srihari, an expert in pattern recognition at the University at Buffalo, part of the New York state university system, is trying to quantify the uncertainty. His group did much of the research that led to postal systems that can recognize handwritten addresses on envelopes, and he works with databases of fingerprints to derive probabilities of random correspondence between two prints.
Most features on a print are usually represented by X and Y coordinates and by an angle that represents the orientation of the particular ridge where the feature is located. A single print can have 40 or more comparable features.
Dr. Srihari uses relatively small databases, including an extreme one that contains fingerprints from dozens of identical twins (so the probability of matches is high), and employs the results to further refine mathematical tools for comparison that would work with larger populations.
These numbers are not easy to come by at this point,” he said. The goal is not individualization — matching two prints with absolute certainty — but coming up with firm probabilities that would be very useful in legal proceedings.
Nicholas D. K. Petraco, an assistant professor at John Jay College, is studying microscopic tool marks of the kind made by a screwdriver when a burglar jimmies a window. It has been hypothesized that no two screwdrivers leave exactly the same pattern of marks, although that has never been proved. So Dr. Petraco is systematically making marks in jeweler’s wax and other materials, creating images of them under a stereo microscope and quantifying the details, assembling a database that can eventually be mined to determine probabilities that a mark matches a certain tool.
Dr. Petraco, a chemist with a strong background in computer science, looks to industry for ideas about pattern recognition — the tools that a company like Netflix uses, for example, to classify people by the kinds of movies they like. “A lot of computational machinery goes into making those kinds of decisions,” he said.
He figures that if something works for industry, it will work for forensic science. “You don’t want to invent anything new,” he said, because that raises legal issues of admissibility of evidence.
The work takes time, but the good news is that the data stays around forever. So as software improves, the probabilities should get more accurate. “Algorithms and data comparison evolve over time,” Dr. Petraco said.
But it may not be possible to develop useful databases in some disciplines — bite mark analysis, for example. “Using a screwdriver, that’s very straightforward and simple,” said Ira Titunik, a forensic odontologist and adjunct professor at John Jay College. But bites involve numerous teeth, and there are other factors, including condition of the skin, that may make it difficult to quantify them for purposes of determining probabilities.
A few researchers are looking at how errors creep into forensic analysis. The National Institute of Standards and Technology recently established a working group on fingerprints, with statisticians, psychologists and others, “to try to understand the circumstances that lead to human error,” said Mark Stolorow, director of the Office of Law Enforcement Standards at the institute.
In Britain, Itiel Dror, a psychologist who studies decision-making processes, is already looking at human factors. “I like to say the mind is not a camera, objectively and passively recording information,” said Dr. Dror, who has a consulting firm and is affiliated with University College London. “The brain is an active and dynamic device.”
He has conducted studies that show that when working on an identification, fingerprint examiners can be influenced by what else they know about a case. In one experiment, he found that the same examiner can come to different conclusions about the same fingerprint, if the context is changed over time.
The same kinds of contextual biases arise with other decision-makers, said Dr. Dror, who works with the military and with financial and medical professionals. He thinks one reason forensic examiners often do not acknowledge that they make errors is that in these other fields, the mistakes are obvious. “In forensics, they don’t really see it,” he said. “People go to jail.”
Forensics experts say the need for research like Dr. Dror’s and Dr. Srihari’s does not mean that disciplines like fingerprint analysis will turn out to be invalid. “I have no doubt that fingerprint evidence and firearms evidence, once looked into by the appropriate research entities, are going to be shown to be very reliable and good,” said Mr. Fisher, the former American Academy of Forensic Sciences president.
Dr. Kobilinsky said people should not jump to the conclusion that forensic science is bad science. “There’s a lot of experience and knowledge that goes into somebody’s expertise,” he said.
“It’s not junk science. But that doesn’t mean it shouldn’t be improved.”
Crime Scene Imperfections
Published: February 21, 2009
Next time you see one of those television crime-scene investigators crack a case with high-tech analysis, better take it with a grain of salt. The National Academy of Sciences, the nation’s most prestigious scientific organization, has surveyed the field of forensic science and found it grossly deficient.
It’s not just that many forensic laboratories are poorly funded and staffed with “experts” who are poorly trained. The more fundamental problem, according to the study, is that there is little evidence of the accuracy and reliability of most forensic methods — especially those that rely on expert interpretation.
The most thoroughly validated technique is nuclear DNA analysis, which has a minuscule likelihood of error when done right. But other well-known methods that can supposedly identify a guilty person or link a weapon or other evidence to a particular crime have no rigorous scientific proof that they work consistently.
That goes for analyses of hair, bite marks, fibers, documents, tools, firearms, shoe impressions, tire tracks, handwriting and blood spatters, among others. The analyses can help focus an investigation but can seldom provide infallible evidence of guilt.
Even fingerprint analysis depends on a subjective judgment by experts as to how closely two prints match, a conclusion that can be biased by the examiner’s knowledge of the suspect or the case. Examiners have sometimes disagreed with their own past conclusions when viewing the same prints in a different context.
The academy’s panel makes sensible suggestions for improvement, such as certification of forensic professionals, accreditation of laboratories, uniform standards for analyzing evidence and independence of the laboratories from police and prosecutors who might bias judgments. In the long run, research is needed to determine the accuracy of forensic methods. For now, judges, lawyers and juries are on notice that high-tech forensic perfection is a television fantasy, not a courtroom reality.
Study Calls for Oversight of Forensics in Crime Labs
Published: February 18, 2009
Crime laboratories around the country are grossly underfunded, lack a scientific foundation and are compromised by critical delays in analyzing physical evidence, according to a broad study of forensic techniques published Wednesday by the National Academy of Sciences, the nation’s premier scientific body.
Skip to next paragraphAmong its many criticisms, the study counted a backlog of 359,000 requests for forensic analysis in 2005, a 24 percent increase in delays since 2002. A survey of crime laboratories found 80 percent of them to be understaffed.
A new federal agency is needed to regulate these laboratories, standardize forensic techniques and pay for research, according to the report, which was financed by Congress in 2005.
The study recommends that an agency, to be called the National Institute of Forensic Science, be created and be independent of the Justice Department, which has traditionally been the nation’s primary forensics research agency. Crime laboratories should be managed separately from police departments to ensure that their findings are protected from bias, the report said.
“The potential for conflicts of interest between the needs of law enforcement and the broader needs of forensic science are too great,” the authors wrote.
The report, the product of a two-year review by the academy, an independent body, is not legally binding. But legislators from both parties have already committed to holding hearings on the study, and officials from all the federal law enforcement agencies are reviewing the document in anticipation of possible policy changes.
“I am troubled by the report’s general finding that far too many forensic disciplines lack the standards necessary to ensure their scientific reliability in court,” said Senator Patrick J. Leahy, Democrat of Vermont and chairman of the Judiciary Committee.
Senator Arlen Specter of Pennsylvania, the ranking Republican on the committee, said he was also troubled by the findings.
The report calls into question the scientific merit of virtually every commonly used forensic method, including analysis of fingerprints, hair, fibers, blood spatters, ballistics and arson. Only DNA, which the panel said had benefited from rigorous scientific scrutiny and peer review outside of the forensics discipline, escaped significant criticism.
“The fact is that many forensic tests, such as those used to infer the source of tooth marks and bite marks, have never been exposed to stringent scientific scrutiny,” the report said. The report highlights crime laboratory scandals involving hundreds of tainted cases handled by police agencies in Michigan, Texas and West Virginia, and by the Federal Bureau of Investigation. At least 10 wrongly convicted men have been exonerated as a result of those laboratory investigations, and the cases of hundreds of other people convicted with the help of those facilities are under review.
The panel also found that most of the nation is served by death investigation offices that lack accreditation. It cited an 18-year-old high school student in Indiana who was recently elected deputy coroner after a short training course.
The academy said that in addition, judges and lawyers generally lacked the scientific expertise necessary to “comprehend and evaluate forensic evidence in an informed manner.”
Attorney General Eric H. Holder Jr. said his office would be reviewing the report over the next several days.
“We have the potential to solve a lot of crimes, to find people who are guilty and to absolve a lot of people who are not through the use of these great forensic techniques,” Mr. Holder said.
Justice Department officials declined to comment on specific recommendations, including the establishment of a National Institute of Forensic Science.