Is there any way to prevent a spider from grabbing URLs that you want to
keep off search engines?

Yeah, robots.txt

Oh, to block all bots from a certain area you do

That /directory/ could be warez.php or /diretory/warez.php

It can block any file.

If you didn't know. Just creat a file called robots.txt in your mai directory and add this information.

The reputable ones - robots.txt or the nofollow attribute... or put the content behind a password protected area.

Doesnt work.
I can configure (or used to be able to configure) my webbroser to behave as a googlebot, and I could access any webpage I want: password or no password.

Yeah, non-reputable bots wont listen to anything, just ignore them. You can find a list of bots that follow robots.txt, but are bad.

Browsers DON'T read the robots.txt!!!

the idea of the robots.txt protocol is not that it blocks useragents ad-hoc, but the useragents that ARE bots can read the text file for the directives that pertain to them specifically, and act on those directives.

If the text files specifies that a certain files or folders should not be indexed by any bots
or by a specific bot;
the robots that honour the protocol should NOT index the files contained in the folder.

__________________
And how can we win, when fools can be kings

Well chrishirst said it straight out. you should make that a sticky as a guide.