The only options I have on the Bluray player is PCM Downconversion on/off and Downmix surround either encoded or stereo.
Might that be your problem? You don't want the player to downconvert or downmix....you want the bitstream output for the processor to decode. Try setting PCM downconversion to Off.
Are you trying to get the BD hi-res soundtracks into the 861 as PCM? If so, the only way to do it is via HDMI from the player (not SPDIF) into a Meridian HD621, then via 3x SPDIF into the processor, either MC1 or MC2 on the IE42 (whichever one your 800v3 isn't using)
Cheers
George