From time to time, you'll see us give a product our “Pants on Fire” rating.
This rating is reserved for situations where we feel that a manufacturer's marketing claims are so far off from what we've been able to validate in our hands-on tests, that we feel there is significant risk of consumers being misled by the claim.
For example, we gave this rating in the case of a baby monitor review where the manufacturer claimed a 2,000 foot range, which would be the longest manufacturer's claimed range of any of the products we reviewed, yet in our tests the product actually only delivered a fraction of their claimed distance, and offered less range capability in our tests than 9 of 20 monitors.
Our intention in giving such a rating is two-fold:
- To help alert consumers that the claim may not match their hands-on experience (since it certainly didn't match ours), and
- To alert the manufacturer's management team that their marketing claim may deserve re-consideration as it is potentially misleading to consumers.
Note that when we give such a rating we are not implying that the manufacturer has intentionally lied in their marketing. We assume that the manufacturer was able to do some test of their own to justify their marketing claims. Instead, we are raising a red-flag for consumers that our hands-on test results are substantially different (much worse) than what the manufacturer claims. For example, this may be due to the fact that we use a different test methodology than the manufacturer, or perhaps that the specific product we reviewed did not perform the same as the one the manufacturer used for their specifications. Regardless of the cause, we feel an obligation to report on such gaps between claimed performance and what we experience and observe in our hands-on tests since other consumers may face a similar situation.
Note also that before we rate a marketing claim as “Pants on Fire” we'll also attempt to research user-reviews at sites like Amazon and others to see if other users have had a similar experience to what we saw in our tests. We generally will not issue a “Pants on Fire” rating unless we see that our test findings are echoed in user reviews (the exception might be a new product, which has not yet received a statistically significant number of user reviews).