There is no hardware decoder support for 10 bit. MPlayer-VAAPI tries to decode 10 bit video as 8 bit, which results in artifacts. Just chatting at #mplayer @ freenode IRC and I got some information.
Maybe ffmpeg is too old or something. Or maybe that the MPlayer is too old or something. The MPlayer-VAAPI in Mageia is from 2011. Not the latest development snapshot of the source code. Maybe it is what causing MPlayer to display color incorrectly.
edit: sorry but currently there is no 10-bit video hardware decoding right now

I have compiled VLC from source with vaapi and avcodec enabled. Video playback is much better now because it's really working now. I'm sure because I got a message "[0x7f95ccc029d8] avcodec decoder: Using VA API version 0.32 for hardware decoding." in terminal. Another problem occurs though. The chipset really heating up and the fan moves at full speed because of this! Even after VLC is closed!