04.01.12
“There are three types of lies: lies, damned lies, and statistics” is a famous quote from Mark Twain—or is it?
The strategy of documenting this quote by typing “Twain” and “Statistics” into Google resulted in multiple instances of phrase use by Twain, however, he had attributed the phrase to Benjamin Disraeli, the 19th century British Prime Minister. There is no documentation to support crediting the quote to Disraeli. There are numerous hypothesized origins to the phrase or at least the concept the phrase embodies, and there are several prime author suspects, but there is no hard evidence that confirms any of the named individuals coined the phrase.
The search consumed more time and bandwidth than I’d care to admit, as I followed URLs through a series of twists and turns, each leading to a dead end. At a certain point I realized the irony of trying to trace the quote back to its point of origin after having written columns on raw material provenance and chain of custody. The sudden ironic insight led immediately to two thoughts. The first: “Rats! I’ve already done a supply chain column,” led immediately to a determination not to let a perfectly clever anecdote go to waste.
Good Manufacturing Practice (GMP) is about traceability in product manufacture and relies heavily on documentation. According to the 21 CFR Part 111 every step of the manufacturing, packaging, labeling and holding processes for dietary supplements must be documented. This translates to having standard operating procedures (SOPs) for all protocols, including laboratory test methods. In choosing a laboratory, you must make sure its operations, with respect to testing your products, are in compliance with cGMP. FDA has an expectation that laboratory work performed in the course of product manufacture, whether in-house or by a contracted laboratory, meets all the dietary supplement cGMP requirements. In January, Loren Israelsen of the United Natural Products Alliance (UNPA) noted that FDA reaffirmed its intention to begin Section 111 GMP inspections of contract analytical laboratories. Contracted laboratories are seen by FDA as an extension of the manufacturer and are thus within the scope of Section 111 inspection.
In general, contract labs have been at arms-length on the GMP front with inspectors focusing on paperwork rather than physical lab inspection to verify lab performance. The consequences of a decision to inspect are potentially far reaching for both laboratories and manufacturers.
By now you must know the drill: establish product specifications and tests to see that specifications are met, document everything, from written SOPs to equipment logs and proof of calibration, and have records that demonstrate the scientifically valid methods are suitable for your intended purpose. FDA has always had the authority to ask you for evidence that your contract lab is in compliance with GMP, but if the agency starts knocking on lab doors, you really could be in trouble. One of the biggest potential hazards is not poor method selection or even incomplete record-keeping; it is outsourcing your laboratory work to a “dry lab.”
Now, to be clear, there are many legitimate areas where the term “dry lab” is applied, such as computational biology or molecular modeling. But this not what we are discussing; we are talking about the act of “dry labbing,” which is described in Wikipedia as “…supplying fictional (yet plausible) results in lieu of performing an assigned experiment.” According to the Urban Dictionary, “to dry lab is to make up data in a scientific experiment, as opposed to observe or experiment in order to obtain it. Usually this is done in response to pressures to finish the experiment by unethical researchers.”
Dry-labbing is a phenomenon that’s been around for a while, but has not yet caught the attention of a majority of companies. A televised report on the subject by Dateline NBC, which aired in mid-March, will likely give the practice a higher profile. The dry-lab story starts when the manufacturer sends a sample and a sample submission form to a contract laboratory. In most dry-lab cases, it will request detailed information about the sample from the customer, including expected results. When the dry-lab receives the sample, it is not actually tested or an inexpensive generic method is employed that yields no real useable data. Nevertheless, the lab will generate a report that essentially affirms the expected result, as provided by the manufacturer.
Industry executive Frank Jaksch (Chromadex Inc.) is featured in the “Dateline” piece, and has agreed to share some of his insights with readers of this column. Mr. Jaksch notes that unscrupulous operations such as the one described can usually be spotted pretty easily if you are careful and attentive. He suggests that a company using a third party lab occasionally challenge the lab with “dummy” samples. Details for the samples should be blinded, meaning the exact expected result should not be provided. Occasionally, the same sample should be sent to a second laboratory for confirmation.
Mr. Jaksch also recommends that site visits, paper audits and even physical audits be conducted on a regular basis and notes (with a smile) that a facility that lacks visible analytical equipment might not be up to the task. Ask to see SOPs, in addition to method citations, date-stamped raw data and analytical reports. Certificates of analysis for reference materials and purchase orders for reagents, chromatographic columns and calibration standards should be reviewed. The contract lab should be able to walk its client through sample tracking procedures, including sample and equipment logs, laboratory notebooks and the sample holding room. Instrument equipment logs can also be very telling; they should include details of instrument usage and maintenance.
All of the information noted here, and more, is required for compliance with GMP. If the laboratory refuses to show any of it to you, there is no way you can assure FDA you are in compliance when the inspector shows up at your doorstep.
If you are permitted to see the documents, be prepared to evaluate them critically and carefully.
Unlabeled and undated chromatograms with no unique identifiers—such as your lot number and the date of analysis—can be recycled over and over again both for your samples and those of other manufacturers. You might want to see chromatograms for ingredients in different products. If the chromatograms look eerily similar, they might just be the same chromatogram. Photocopying a chromatogram is cheaper than buying and running an HPLC and doesn’t require trained operators.
Telltale clues that your contractor is dry labbing can also be found in the analytical test reports. The following is a list of warning signs or red flags:
• Analytical result reports that refer to modified official methods or modified USP methods;
• Citations of methods from peer reviewed scientific journals that are used to measure something “in urine” or “in serum;”
• There is mismatch between the equipment and/or detector employed by the contract lab when compared to that reported in the citation; and
• The order of magnitude for the measure being taken is different then that of the test method; for example, using a method designed to test purity of 100% pure compound for measuring the compound in a complex matrix, or vice versa.
If you have an in-house scientist, let him/her review the analytical test reports or contract an expert to review them. If a method citation looks like unintelligible gibberish to both you and the scientist, there’s a fair chance there is some wool-pulling going on. For more information on what should be in an analytical test report or certificate of analysis check out the Nutraceuticals World archives in print and online (www.nutraceuticalsworld.com) for the “Quality Focus” column titled “Certificates of Analysis” from October 2008.
Contract labs must modify methods often, whether it is to adapt to new technology or to expand the scope and applicability of an existing method. This is not unusual but when a method has been modified, insist that you are privy to the details of the modification, the method re-validation or verification SOP, and the data that proves to you (and FDA) that the method remains scientifically valid and is suitable for its intended use. That method suitability must extend to your product type. Don’t take anyone’s word for it; there must be documentation. After all, it is your company that is at risk. If a contract lab wants your business it should be willing to share information. While it may not want to send you a copy of its methods and data, it should certainly be wiling to undergo an audit at its facility to prove the documentation exists. For more guidance on selecting a contract analytical lab check out the “Quality Focus” column titled “Outside Analytical Labs” from January 2009 in the Nutraceuticals World archives.
For the future, remember to think GMP when you think of your off-site operations. Out of “site” should not be out of mind.
The strategy of documenting this quote by typing “Twain” and “Statistics” into Google resulted in multiple instances of phrase use by Twain, however, he had attributed the phrase to Benjamin Disraeli, the 19th century British Prime Minister. There is no documentation to support crediting the quote to Disraeli. There are numerous hypothesized origins to the phrase or at least the concept the phrase embodies, and there are several prime author suspects, but there is no hard evidence that confirms any of the named individuals coined the phrase.
The search consumed more time and bandwidth than I’d care to admit, as I followed URLs through a series of twists and turns, each leading to a dead end. At a certain point I realized the irony of trying to trace the quote back to its point of origin after having written columns on raw material provenance and chain of custody. The sudden ironic insight led immediately to two thoughts. The first: “Rats! I’ve already done a supply chain column,” led immediately to a determination not to let a perfectly clever anecdote go to waste.
Good Manufacturing Practice (GMP) is about traceability in product manufacture and relies heavily on documentation. According to the 21 CFR Part 111 every step of the manufacturing, packaging, labeling and holding processes for dietary supplements must be documented. This translates to having standard operating procedures (SOPs) for all protocols, including laboratory test methods. In choosing a laboratory, you must make sure its operations, with respect to testing your products, are in compliance with cGMP. FDA has an expectation that laboratory work performed in the course of product manufacture, whether in-house or by a contracted laboratory, meets all the dietary supplement cGMP requirements. In January, Loren Israelsen of the United Natural Products Alliance (UNPA) noted that FDA reaffirmed its intention to begin Section 111 GMP inspections of contract analytical laboratories. Contracted laboratories are seen by FDA as an extension of the manufacturer and are thus within the scope of Section 111 inspection.
In general, contract labs have been at arms-length on the GMP front with inspectors focusing on paperwork rather than physical lab inspection to verify lab performance. The consequences of a decision to inspect are potentially far reaching for both laboratories and manufacturers.
By now you must know the drill: establish product specifications and tests to see that specifications are met, document everything, from written SOPs to equipment logs and proof of calibration, and have records that demonstrate the scientifically valid methods are suitable for your intended purpose. FDA has always had the authority to ask you for evidence that your contract lab is in compliance with GMP, but if the agency starts knocking on lab doors, you really could be in trouble. One of the biggest potential hazards is not poor method selection or even incomplete record-keeping; it is outsourcing your laboratory work to a “dry lab.”
Now, to be clear, there are many legitimate areas where the term “dry lab” is applied, such as computational biology or molecular modeling. But this not what we are discussing; we are talking about the act of “dry labbing,” which is described in Wikipedia as “…supplying fictional (yet plausible) results in lieu of performing an assigned experiment.” According to the Urban Dictionary, “to dry lab is to make up data in a scientific experiment, as opposed to observe or experiment in order to obtain it. Usually this is done in response to pressures to finish the experiment by unethical researchers.”
Dry-labbing is a phenomenon that’s been around for a while, but has not yet caught the attention of a majority of companies. A televised report on the subject by Dateline NBC, which aired in mid-March, will likely give the practice a higher profile. The dry-lab story starts when the manufacturer sends a sample and a sample submission form to a contract laboratory. In most dry-lab cases, it will request detailed information about the sample from the customer, including expected results. When the dry-lab receives the sample, it is not actually tested or an inexpensive generic method is employed that yields no real useable data. Nevertheless, the lab will generate a report that essentially affirms the expected result, as provided by the manufacturer.
Industry executive Frank Jaksch (Chromadex Inc.) is featured in the “Dateline” piece, and has agreed to share some of his insights with readers of this column. Mr. Jaksch notes that unscrupulous operations such as the one described can usually be spotted pretty easily if you are careful and attentive. He suggests that a company using a third party lab occasionally challenge the lab with “dummy” samples. Details for the samples should be blinded, meaning the exact expected result should not be provided. Occasionally, the same sample should be sent to a second laboratory for confirmation.
Mr. Jaksch also recommends that site visits, paper audits and even physical audits be conducted on a regular basis and notes (with a smile) that a facility that lacks visible analytical equipment might not be up to the task. Ask to see SOPs, in addition to method citations, date-stamped raw data and analytical reports. Certificates of analysis for reference materials and purchase orders for reagents, chromatographic columns and calibration standards should be reviewed. The contract lab should be able to walk its client through sample tracking procedures, including sample and equipment logs, laboratory notebooks and the sample holding room. Instrument equipment logs can also be very telling; they should include details of instrument usage and maintenance.
All of the information noted here, and more, is required for compliance with GMP. If the laboratory refuses to show any of it to you, there is no way you can assure FDA you are in compliance when the inspector shows up at your doorstep.
If you are permitted to see the documents, be prepared to evaluate them critically and carefully.
Unlabeled and undated chromatograms with no unique identifiers—such as your lot number and the date of analysis—can be recycled over and over again both for your samples and those of other manufacturers. You might want to see chromatograms for ingredients in different products. If the chromatograms look eerily similar, they might just be the same chromatogram. Photocopying a chromatogram is cheaper than buying and running an HPLC and doesn’t require trained operators.
Telltale clues that your contractor is dry labbing can also be found in the analytical test reports. The following is a list of warning signs or red flags:
• Analytical result reports that refer to modified official methods or modified USP methods;
• Citations of methods from peer reviewed scientific journals that are used to measure something “in urine” or “in serum;”
• There is mismatch between the equipment and/or detector employed by the contract lab when compared to that reported in the citation; and
• The order of magnitude for the measure being taken is different then that of the test method; for example, using a method designed to test purity of 100% pure compound for measuring the compound in a complex matrix, or vice versa.
If you have an in-house scientist, let him/her review the analytical test reports or contract an expert to review them. If a method citation looks like unintelligible gibberish to both you and the scientist, there’s a fair chance there is some wool-pulling going on. For more information on what should be in an analytical test report or certificate of analysis check out the Nutraceuticals World archives in print and online (www.nutraceuticalsworld.com) for the “Quality Focus” column titled “Certificates of Analysis” from October 2008.
Contract labs must modify methods often, whether it is to adapt to new technology or to expand the scope and applicability of an existing method. This is not unusual but when a method has been modified, insist that you are privy to the details of the modification, the method re-validation or verification SOP, and the data that proves to you (and FDA) that the method remains scientifically valid and is suitable for its intended use. That method suitability must extend to your product type. Don’t take anyone’s word for it; there must be documentation. After all, it is your company that is at risk. If a contract lab wants your business it should be willing to share information. While it may not want to send you a copy of its methods and data, it should certainly be wiling to undergo an audit at its facility to prove the documentation exists. For more guidance on selecting a contract analytical lab check out the “Quality Focus” column titled “Outside Analytical Labs” from January 2009 in the Nutraceuticals World archives.
For the future, remember to think GMP when you think of your off-site operations. Out of “site” should not be out of mind.
