Its not that it will lower your page rank, infact, no link can ever lower your PR. Its just some hosts have certain rules against gambling and adult content. Its also a bit unethical linking to gambling and porn websites knowing that you probably have kids browsing your website.

The only thing you need to worry about is linking to link farms and other websites that try to manipulate SERPs "illegally".