Meridian Unplugged - Powered by MrTechGuy.co.uk Forums Discussion Meridian - Streaming How to check if your Sooloos network is 'bit perfect'

Jack

I have not tried with Control:PC but fairly sure it will be same. Apologies if I made wrong assumption. An alternative filter that may be worth trying is the second filter option that I gave with the addition of IP of audio endpoint but less the 'data contains " etc"' bit. I can't remember if this will also pick up handshaking between endpoint and core. To watch packets only for the destination (i.e. audio endpoint as receiver) use IP.dst == 192. etc. Also apologies that UBB is converting case of some of my instructions - IP.dst is all lower case here.

Alternatively, if you are trying to look at MS600 audio, you will need a switch that supports port mirroring/monitoring (or hub) as your PC/Mac won't be able to see the data between core and audio endpoint as the switch will isolate these ports from the PC running Wireshark sniffer.

Not to worry. I saw your post and thought it was an interesting idea. I tried it quickly while in the middle of doing three other things. I will have to sit down and pay a little closer attention when I get a minute.

Have you ever seen situations where the audio portions of the streams are not identical?

This is a neat way to prove that the data is identical, but expecting it to be different is more akin to debugging the Meridian Sooloos software than debugging your network.

First, UDP has checksumming, so the UDP packet would actually get thrown out by the IP stack before it makes it up to the receiving application, which would result in a lost packet, not a malformed packet.

Next, given that packet loss happens even in common cases, including corruption, there must be a simple acking or nacking scheme here, or else we would hear choppiness often (802.3 and 802.11 is lossy, IP is lossy, UDP is lossy).

The key to IP networking is that they don't have to be bit perfect. They just have to be decently good. TCP ensures that, and UDP helps against corruption, and the application is required to deal with reliability (usually much more lightweight than TCP's solution).

Even if you do see errors on the wire, the assembled audio stream should be perfect, because of retransmissions and sequencing.

Hi

I have undertaken a handful of tests and not seen situations where audio portions of the stream are not identical between multiple runs.

However this is between an MC200 and Control:Mac as I do not have any other Sooloos equipment. My tests also were on an audio track truncated manually to approximate 40 second track - I think a perfectly valid amount of time for people to hear differences. There is potentially a huge amount of data that can be generated and keeping this to reasonable quantity that can be fully verified is IMHO more valuable than looking for a needle in a haystack.

I put together a (relatively) simple test to prove whether or not Sooloos audio passing over the network is consistent across multiple identical runs. Some people are reporting hearing differences due to changes in their Sooloos setup. I wanted to create a test to prove whether these differences were data related or not.

There are other better tests that could be performed, but I wanted a test that was suitable for as wider range of audience as possible using a basic network setup. This test is analogous to an A/B listening test, except that proof of differences (if any) can be verified analytically.

The test as presented does however rely on a number of basic, but I believe valid, assumptions. The most important is that undesirable differences will be random and this can be shown as differences between multiple runs. A random difference indicates a likelihood that something is not right (with the network or Sooloos equipment for example) rather than a change by design which would likely be repeatable across consecutive runs.

Another assumption is that to achieve consistent results across multiple runs, the network should be robust and 100% error free. A hardwired home network should IMHO be 100% error free. Add wireless or power line and all bets are off as these are highly dependant on environment conditions.

Wireshark is capable of testing the validity of both of these assumptions but this test does not make full use of Wiresharks ability to help resolve issues, merely to identify whether one exists or not. The Wireshark statistics/compare function allows you to take two files captured from different ends of a network that have been merged and analyse them for dropped or mis timed packets amongst other tests, but this requires the ability to simultaneously sniff two network ports at different network locations. Wireshark does allow packets with errors to be easily detected - these are highlighted in red on the main display (I believe by default) so it is very easy to spot packets that have errors detected with them. Unfortunately the type of error checking used in IP network packets is not 100% reliable in some circumstances so a very very small percentage of errors may not get reported but I believe that the chances of this happening on the same packet across multiple runs is tiny compared to consistent differences that may be heard by someone noticeable audio differences in a listening test.

At the end of the day, I would expect multiple runs to be 100% consistent if used in a hardwired network, except in the IP and UDP headers and what appears to be a byte (or two?) immediately following the Sooloos AESPDATA. The IP differences are to do with sequential ID and header checksum and UDP destination port (only in packet containing Sooloos AESPDATA).

Well, what you say is valid, and it may be a decent test to prove a network is clean while the test is running. However, your test can produce false positive errors.

As you have shown, all Sooloos audio still sits on top of UDP. As I stated in my per my previous post, you can see that corruption will never make past the kernel into the application because of UDP's checksumming feature.

That is your first false positive. Even a corrupt packet shown by Wireshark, but will never be played by Sooloos because Wireshark has access to packets at a much lower level than Sooloos does, and Sooloos will never see those corrupt packets.

Now that corruption is not an issue, what happens with packet loss.

The packets in your traces show sequencing, which is in the application portion of the UDP packet (UDP has no built-in sequencing), which leads us to believe that Sooloos is doing its own packet loss detection as well as out-of-order delivery detection. You can easily test this by pulling out your ethernet cable during playback, and seeing how a few UDP packets will flow the other way, clearly re-requesting the lost packets from the sender. During this time, Sooloos will continue playing out it's buffered packets. If the sender resends the lost packets in a timely fashion, you will hear zero loss in the audio signal because the original data stream was reassembled properly.

That is your second false positive. Your UDP streams will differ in order of data, and Wireshark will be unable to put the packets into the right order because it knows nothing about Sooloos's protocols on top of UDP.

-

Another common piece of incorrect information floating around often is that wireless can somehow corrupt audio. 802.11 replaces 802.3, but both are unreliable. IP/UDP/Sooloos all together ensure that the data stream is perfect assembled. Even if the wireless signals were interfered with, the Sooloos would never see that corrupt data, although your Wireshark may. A bad ethernet cable or wireless networking will make zero difference to the corrupted packets getting to Sooloos. Wireless may be more prone to corruption, but that should make no difference.

-

So the question then, is why wireless or a flakey cable can make the sound very poor, even if we can prove no corruption is possible.

Well, above, there was a critical thing I stated: "If the sender resends in a timely fashion, you will hear zero loss in the audio signal"

If the sender does not resend in a timely fashion, then the Sooloos player's audio buffer will empty and it will have no data to play. As soon as this happens, it must stop playing audio, or it may start playing the start of the buffer over again.

A bit of a tangent here, but the reason scratched CD's loop the same part over and over is because of the same premise. The buffer emptied while the laser was trying to read data off the CD, and the CD player just continued to play the buffer over and over again. The corrupt part of the CD is the part directly after the looping audio.

Either way, playing nothing or playing the buffer over again for a moment, will sound terrible. That's where all the problems with digital audio come from.

-

The real problem with the Wireshark method is that it only looks at the data in the packets, and not the timing for the assembly of those packets, thus the possibility for false positive errors.

The absolute best test you can conduct for the digital domain is this:

1) Take your audio file, decode to PCM data. WAV will do fine if you chop off the headers.
2) Take your Sooloos's SPDIF output and record it using a PC, saving that output to a WAV file. Chop off the headers from the WAV.
3) The data from the SPDIF will have a ton of zeros before it since you had no audio playing when you started recording, so chop those out.
4) do a byte for byte comparison between the two sets of data. Any difference is a reproduction error. Perfect match here will prove that Sooloos's SPDIF output is producing exactly what the original source material data contained.

After that, it's all in the DAC, and that becomes much more subjective.

Hi

It's always good to have discussion but I feel that the discussion has gone well beyond the intention of the test as devised.

This test was designed to show in a scientific way whether or not some of the changes that people have recently discussed about hearing audible differences when changing 'variables' in their Sooloos setups could be due to changes in underlying data - bit for bit rather than timing or other nasties that may or may not influenced the audio during audio replay.

By measuring and comparing multiple 'auditions' of the same track data, it can be shown whether each rendition produces exactly (excluding the bytes expected to change) the same packet data as previous auditions. If the results are exactly the same, then it can be proven that the change to a 'variable' can not have resulted in an audible difference unless the change was due to jitter or other (cable borne or otherwise) external influences, but not in the data as some have implied.

I fully appreciate that a difference may not necessarily result in an audible difference for whatever reason such as redundancy. What is important is that no difference cannot result in an audible change unless that change came from somewhere other than the data.

What the test also aims to show, is that by using a real time packet analyser such as Wireshark, people will see how reliable a wired computer network should be ie so accurate that it may as well be thought of as 100% accurate. By leaving the filtered window running, there should never be any reason to see red marked packets indicating an error. The scenario of packet errors in a hard wired home network should lead the user to question the integrity of their network and do a bit of trouble shooting to sort the problem out.

By running the test multiple times, the chances of two random errors masking each other out becomes smaller and smaller with each run. On the 40 second clip that captured and compared, approximately 3600 packets were collected. If a packet was dropped and thus not captured by WireShark, the chance of a second run having exactly the same error would be 1 in 3600. Even with that, I think it would be fair to say that 1 random error in 3600 samples is not going to result in some of the changes that people are hearing. You can do your own sums what a third run would yield.

I do fully understand that packet errors may not yield audible differences due to error detection and recovery by the Sooloos applications. But, just like error correction when reading from a CD, it cannot be proven whether the error recovery process may or may not have induced audible anomalies.

I just wanted to keep it simple and beyond doubt. 100% consistent data resulted will result in Sooloos having exactly the same audio information to process. It's up to Sooloos to deal with timing and external influences.

NB. The topic of recording and comparing SPDIF output has been discussed multiple times on HH forum.

I'm pretty sure that all current Meridian devices employ undefeatable noise shaping on their SPDIF outputs anyway, so a comparison of WAV to SPDIF capture isn't meaningful.

I did the SPDIF/WAV comparison just now. When cross fading and normalization are off, there was no difference.

This was on a Control:15.