The great Google Sandbox-Penalty swindle

Ok now we all know about the sandbox effect => most people around here consider it a myth or a variant of the google dance.

Never the less most people consider there are penalties => google puts you a few places lower from 30 to 950

Now I dont say that there are such penalties but I dont admit they exist either.

Never the less google will “ban” you if other autority sites mark you as a spammer => you will become a so called spammer if you build a lot of low quality irelevant links.

Now after all that noncense I have a question:
If any of the above is true then you CAN take your competitor sites down but what if you have a white hat site and somebody spames it to death and complains to google about it- could you do anything about it?

People complain about there sites not ranking well anymore all the time I wonder what the real reason for this is.