Well... these are the rules. Read this to see how reputation works.
Also this post explains a bit why you get -1 rep when you downvote.
-2 to post owner
-1 to you
This is done to make sure downvotes are cast only when you feel
strongly that something is incorrect / wrong / dangerous / of low
quality.
And here is an other explanation:
Downvotes are important (very important!) but they are also
potentially divisive, so they are just slightly discourage with a
minor cost.
The idea is that you will think twice about them and only apply them
where they really belong.
'Course, once you have oodles of rep the one point cost doesn't mean
much and you can feel free to cast them whenever it seem like a good
idea...
I guess these are the rules and we have to play by them.
And just to make it clear...you get -1 rep only when you downvote an answer. Question downvotes are free of charge.