GO:0060695: negative regulation of cholesterol transporter activity
Any process that decreases the rate, frequency, or extent of cholesterol transporter activity.