The Internet has enabled recruitment of large samples with specific characteristics. However, when researchers rely on participant self-report to determine eligibility, data quality depends on participant honesty. Across four studies on Amazon Mechanical Turk, we show that a substantial number of participants misrepresent theoretically relevant characteristics (e.g., demographics, product ownership) to meet eligibility criteria explicit in the studies or inferred by exclusion from the study on a first attempt or in previous experiences with similar studies. When recruiting rare populations, a large proportion of responses can be deceptive. We conclude with recommendations about how to ensure that ineligible participants are excluded that are applicable to a wide variety of data collection efforts that rely on self-report.