View RSS Feed

mitchco

FLAC vs WAV Part 2 Final Results

Rating: 7 votes, 2.71 average.
In part 1, I used a null test technique to show that both FLAC and WAV (lossless) file formats are identical. In this post, I have expanded the null test to cover off playing the same FLAC and WAV files dynamically from JRiver and capturing the audio waveform after the Digital to Analog conversion and analog line output stage. Here is a high level block diagram of my test setup:



For playback, I am using the exact same original FLAC and converted (by JRiver) WAV file I used in Part 1. It is Tom Petty and Heartbreakers Refugee at 24/96. JRiver is set up for bit perfect playback with no DSP, resampling, or anything else in the signal chain. I used the native Lynx ASIO driver to communicate between the sound card and JRiver. All sample rates for the tests are at 24/96.

My Win7 64 Bit HTPC build is nothing special. No special power supply or SSD or interconnects.



Side note, for Windows users, always invaluable to check your PC for latency with http://www.thesycon.de/deu/latency_check.shtml

I have tested the frequency response of my Lynx L22 sound card using REW http://www.hometheatershack.com/roomeq/ and noise levels, distortion, etc., using RightMark Audio Analyzer http://audio.rightmark.org/index_new.shtml





For capturing (i.e. recording) the audio waveforms, I used a Dell M4600 latptop and the onboard HD audio chip and driver. Here is the noise measurement of the on board sound chip. Not as good as my Lynx card above, but a check to see that everything is in working order.



I used Audio Diffmaker http://www.libinst.com/Audio%20DiffMaker.htm for recording the waveforms that were coming off the analog outputs of my playback PC. Here is the process used by Audio DiffMaker:



As an aside, I should point out that you can use this software to objectively measure anything in your audio playback chain that you have changed. Whether that be power supply, DAC, interconnects, music players, SSD, VST plugins, or whatever.

Remember, if you are audibly hearing a difference when you change something in your audio system (ABX testing), the audio waveform must have changed, and if it has changed, it can be objectively measured. I find there is a direct correlation between what I hear and what I measure. For me, to form any valid opinion about audio reproduction, I want to correlate my subjective results with my objective results and vice versa. I want a balanced view.

In the Audio DiffMaker help file, the software program is able to line up the waveforms if the program material is within 1 second of each other (protip).

Here I am capturing the first 40 seconds of TP’s Refugee in Audio DiffMaker:



I did this twice, once playing the FLAC and then the WAV, without making any changes on either computer.

To test the DiffMaker software (and everything else) is working correctly, I took the FLAC recording and compared it to itself. Theoretically, it should null itself out completely.



And it does. Ok so now let’s compare the two recordings, one FLAC and the other WAV:



What the result is saying is that the difference signal is almost -90 dB. I repeated the test ten times and obtained the same results.

You can listen to the difference track for yourself as it is attached to this post. PLEASE BE CAREFUL as you will need to turn up the volume (likely to max) to hear anything. I suggest doing this in volume level stages so you can verify there are no other artificats while listening.

As you can hear for yourself, a faint ghost track of the music, that nulls itself out completely halfway through the track and slowly drifts back into being barely audible at the end.

According to the DiffMaker documentation, this is sample rate drift and there is a checkbox in the settings to compensate for this drift:

“Any test in which the signal rate (such as clock speed for a digital source, or tape speed or turntable speed for an analog source) is not constant can result in a large and audible residual level in the Difference track. This is usually heard as a weak version of the Reference track that is present over only a portion of the Difference track, normally dropping into silence midway through the track, then becoming perceptible again toward the end. When severe, it can sound like a "flanging" effect in the high frequencies over the length of the track. For this reason, it is best to allow DiffMaker to compensate for sample rate drift. The default setting is to allow this compensation, with an accuracy level of "4".”

Of course this makes sense given that I used a different computer to record on versus the playback computer and I did not have the two sample rate clocks synched together. The DiffMaker software recommends this approach, but I have no way of synching the sample rate clock on the Dell to my Lynx card.

So when this is not possible, the DiffMaker documentation indicates to use the sample rate compensation.

However, when I tried the sample rate compensation, the DiffMaker program thru the following error:



I sent an email to the software manufacture and will follow up once I hear back.

Given that the signal is almost -90 dB from the reference and that the noise level of my Dell sound card is -86 dB, we are definitely nearing the limits of my gear. Also, given that the dynamic range of most music material we listen to is less than 20dB http://en.wikipedia.org/wiki/Dynamic_range#Audio it seems unlikely that I could hear the difference track, relative to the reference level – that’s a 90 dB difference.

Subjective Listening Tests

In JRiver, I played the FLAC and WAV (and vice versa) several times through headphones and speakers. I did this sighted and blind. I also played back the recorded reference and compare files in Audio DiffMaker using headphones. Finally, I played back the Reference + Difference track.

In my subjective listening tests, I could not hear any differences between the FLAC and WAV files in any combination of the above. Not only from the playback machine but also the recorded tracks. They all sounded identical to me. There seems to be good correlation between objective and subjective results.

As a side note, I have been into audio and music for over 40 years. For 8 of those years I was a recording/mixing engineer where I was trained and relied upon to note very small audible changes. http://www.thepikes.com/bio The reason I am saying this is because of psychoacoustic http://en.wikipedia.org/wiki/Psychoacoustics effects, our ears can be easily fooled http://en.wikipedia.org/wiki/Auditory_illusion or put in a positive way, our ears adapt to changes very quickly.

In fact, most recording, mixing, and mastering engineers use these psychoacoustic effects on purpose. For example, the HAAS effect http://en.wikipedia.org/wiki/Haas_ef...s_and_findings to make the sound more full, wider, sense of air, etc. All tricks played on our ears: http://www.algorithmix.com/en/kstereo.htm including some remastered material we download from HDTracks.

So do we not trust our ears? I am not saying that. What I am doing is bringing a balance of both subjective and objective thoughts together so we can correlate what we hear with what we measure and vice versa. Again, when performing ABX listening tests, if you are hearing an audible difference, then the waveform must have changed. If the waveform has changed then we can measure the difference.

Btw, all of the software used in these tests is free. I would encourage you to download the software’s and try this out for yourself as it does not require any special equipment. Further, you can objectively quantify any differences throughout the audio chain in your playback system.

In conclusion, using my ears and measurement software, on my system, I cannot hear or (significantly) measure any difference between FLAC and WAV. Not only just file formats, but the rest of the audio playback chain as well.

Happy Listening!
Attached Thumbnails Attached Files

Updated 11-12-2012 at 11:32 PM by mitchco

Categories
Personal Blogs

Comments

  1. 1audio's Avatar
    I'm glad you got diffmaker working for you. The results are pretty clear and you indicate that they are repeatable.



    I think your results are exceptional since there is no evidence even of hum. A sign of very good system setup.



    What do we do now? You took all the fun out of this.
  2. losingmyreligion's Avatar
    Nice work.



    "What do we do now? You took all the fun out of this."



    The fun might only be beginning.



    Any chance you'll compare JPlay to Jriver?
  3. Paul.Raulerson's Avatar
    Really nice work, and it looks like your system is well setup too. Any chance you could repeat the recording with other software? Non-windows software, like Macs and Linux.



    -Paul
  4. new_media's Avatar
    Very nice work and a very clear write up as well.



    Thanks very much for your efforts and for reporting your results.
  5. One and a half's Avatar
    well thought out and a rational approach. This will finally put an end to endless debates about which format is better played on the SAME machine.



    Now, the door is open to test the same files on different OS from the same machine.. :) or use a SSD versus a rotating drive. If there are differences I wonder how they will sound or be just a set of beeps and clicks, more clicks mean more differences, but in which way?



    Much to learn young grasshopper :)
  6. Mark Powell's Avatar
    It is indeed. Great that someone bothered to do it.



    "This will finally put an end"



    NOT A CHANCE.
  7. 1audio's Avatar
    The next step is to try to validate that the tool will show an audible difference of the sort we are interested in. Its pretty important to verify that it does detect significant differences. Sometimes the simple difference will not be sensitive to something we may hear. It may need some sort of frequency, amplitude and timing weighting to highlight things we may hear, but this is all speculation.



    I think comparing a .wav with a 64K mp3 would be the next clear step to confirm that a well accepted sonic difference is easily detectable and at what level. My guess is that the difference will be on the order of 20 dB below program level with a really annoying sounding "residue". And perhaps see what you get with a small eq dialed in. We know that a 1/2 dB shift in a fairly broad band will produce a subtle but significant difference in the sound. I'm not sure how the difference would show up in this test.
  8. hamildad's Avatar
    Thanks for this, although you might be taking all the fun out of arguing on message boards ;)



    this was a valid methodology and you explained strengths and weaknesses in your chain all through the process.



    if any one asks about wav vs. FLAC, I'll now point them to your thread.



    well done and thanks for doing what loads of us have wanted to do but never been able to.
  9. Julf's Avatar
    very thorough and rational study. Beats the hundreds and hundreds of messages with wild speculation.
  10. losingmyreligion's Avatar
    "The next step is to try to validate that the tool will show an audible difference of the sort we are interested in. Its pretty important to verify that it does detect significant differences. Sometimes the simple difference will not be sensitive to something we may hear."



    Agreed.



    How audible do you think that the small signal differences (that comprise the reported differences) between players, cables, etc. are, relative to using something like Audio Diffmaker as a test apparatus?



    I'll reach out to you offline if you don't mind, I have some questions about possibly testing using my LIO-8.



    clay
  11. mitchco's Avatar
    I think comparing a .wav with a 64K mp3 would be the next clear step to confirm that a well accepted sonic difference is easily detectable and at what level. My guess is that the difference will be on the order of 20 dB below program level with a really annoying sounding "residue".



    Agreed. I performed a file null test with FLAC vs MP3 at: http://www.computeraudiophile.com/en...M4A-Experiment I have attached the difference file to that blog post and sounds exactly as you describe. It will be interestng to see if DiffMaker correlates the result.



    On another note, I was asked to perform the same DiffMaker test on the digital output of the Lynx L22 sound card, but not including the digital to analog conversion and analog output stage.



    The Lynx L22 card supports loopback mode: http://www.lynxstudio.com/support_faq_result.asp?c=32



    I used a music player to play TP Refugee 24/96 FLAC/WAV and in real-time record the waveforms in Audio DiffMaker on my computer. One pass for FLAC and one for WAV.



    Here is the difference result:







    Inaudible.



    http://en.wikipedia.org/wiki/Audio_file_format



    "Lossless compression formats enable the original uncompressed data to be recreated exactly."



    With respect to Audio DiffMaker and how it works:



    AES Paper: http://www.libinst.com/AES%20Audio%2...ng%20Paper.pdf



    Slides: http://www.libinst.com/Detecting%20D...20(slides).pdf



    Cheers
    Updated 05-09-2012 at 10:27 AM by mitchco
  12. Paul.Raulerson's Avatar
    You refer to m4a in your tests. If downloaded from iTunes, that would be a 256k AAC file. A lot of folks, me included, have copies of our libraries in ALAC, which should compare exactly to FLAC.



    Would you mind transcoding one of your WAV or FLAC file to ALAC and running the result?



    Paul
  13. Talos2000's Avatar
    Together with the "FLAC vs WAV vs MP3 vs M4A Experiment" you posted last year, you've demonstrated a compelling capability where something we all (well, most of us anyway) agree is audible can be measured, and something we all (well, most of us anyway) agree is inaudible is shown to be not measurable.



    I agree with the direction 1audio suggests going in. It would be real nice to map out what differences this technique can quantify vs differences that a critical mass of listeners can agree are audible.



    I dunno, but how about the good old "do USB cables make an audible difference" debate? Do you find that there is an audible difference on your system? If so, that might be a good place to start. If you don't claim to hear a difference, then maybe you could pick something similarly contentious where you do. This is where the rubber of your work starts to meet the road.



    Thanks for all your hard work, and for presenting your results in a format that serves as a model of clarity for everyone.
  14. Talos2000's Avatar
    I just clicked the SAVE button, and then a good idea (I think) struck me. How about you take a reference track (maybe the Tom Petty 'refugee' track you mentioned). Up-convert it to 24/192, and then down-convert it back to 24/96. Most people would agree that the difference between the two would be marginally audible. See what your measurement makes of it.



    I think that would make for another awesome post!
  15. Mike27's Avatar
    I've been a firm believer in the null test for many years. Already did my own mp3 comparison a while back (it can be done solely in Audacity by importing the file to be compared as a second pair of tracks, aligning, inverting, and using "mix and render.") So I already know the diff signal sounds really yucky.



    Not necessarily relevant, but my Tom Petty "Playback" box set is HDCD encoded (http://en.wikipedia.org/wiki/HDCD.). Since this process is essentially a digitally controlled companding scheme, I wonder if the iTunes version may have been encoded from these "masters" (which would be a travesty, yet another in the plethora of bad choices through the years... and utterly absurd as the AAC encoding would of course destroy the control bits.) In any case, the waveform view shows none of the modern "loudness war" clippy-mastering so prevalent these days...



    Thanks for an informative and enjoyable post.
  16. mitchco's Avatar
    from 1audio: The next step is to try to validate that the tool will show an audible difference of the sort we are interested in. Its pretty important to verify that it does detect significant differences.



    I think comparing a .wav with a 64K mp3 would be the next clear step to confirm that a well accepted sonic difference is easily detectable and at what level.



    I compared the Tom Petty original FLAC to a LAME encoded 320kbps MP3 so I can correlate that with the work I already performed here: http://www.computeraudiophile.com/en...M4A-Experiment The difference being that was a null test on the file format. I have attached the results of that test (FLAC vs MP3 File Null test.zip) so it can be sonically compared with the results of the Audio DiffMaker test.



    Here is the result of the Audio DiffMaker test:







    The difference is about -40 dB, quite the difference from the -90 dB when comparing FLAC vs WAV in the same test configuration.



    I have attached the MP3 difference file (FLAC vs MP3 Audio DiffMaker test.zip)



    If you listen to both difference files, one using the file null test and the other DiffMaker null test (at the analog outs of my gear), they correlate extremely well. You can hear the same annoying high frequency residue that is left over as a result from comparing a lossless file format to a lossy file format (using the highest quality MP3 encoder and settings).



    How close is the correlation? You can listen with your ears and you can measure it as well.



    Here is the Audacity frequency spectrum of the null "file" test result:







    And frequency spectrum of the "DiffMaker" test result:







    You can hear the correlation with your ears and see the correlation with your eyes. As mentioned before, I like a "balanced view" between objective measures and subjective listening tests.



    Just to call out what is happening. The first "file" null test was comparing FLAC vs MP3 at the file format level. The Audio DiffMaker test is comparing the same, but includes the computer signal chain of music player -> ASIO driver -> Lynx L22 DAC -> Lynx analog outputs, and in real-time being recorded to a different computer using Audio DiffMaker



    If you think about it, one test is comparing just the file formats, the other is comparing the file formats and the rest of the playback chain to the analog outs. Look how close the waveforms match (and by listening!) even when introducing the playback chain into the measurement equation.



    Given what I hear and what I measure, and the direct correlation to another null test approach, this validates that Audio DiffMaker is a sensitive and accurate measurement tool.
    Updated 05-09-2012 at 10:28 AM by mitchco
  17. vortecjr's Avatar
    I haven't finished reading the post, but why the need for a second computer to capture the signal? It's been a while since I used the L22 on Windows, but you can pipe the signals in the mixer from play to record and capture the playback directly in software. You should also be able to plug the analog output of the card into the analog input of the card (same applies for the digital stream) and capture it that way. Anyway, just a thought...



    Jesus R
  18. Audio_ELF's Avatar
    Jesus... I would say that using a second computer for measuring allows a lot more scope for confirming the independence of the recording from the playback...



    Eloise
  19. 1audio's Avatar
    If you use the same clock for source and capture you will hide any jitter and time based issues, they become invisible. Second any ground noise issues may become invisible in a loopback. Finally I would prefer different sample rates for source and capture incase some almost synchronous thing is happening. 44.1 as a source and 192 as capture would also show supersonic stuff that may be interesting. . .
  20. vortecjr's Avatar
    the focus should be the what is being compared. The setup caused problems and this could have been avoided without introducing an unnecessary fix. Either way, it's just an observation.



    Demian, understood and agree in full. The card's header has master clock input/output and the cable comes with the card so it would shave been easy to incorporate..



    Jesus R
  21. esldude's Avatar
    Superbly done Mitchco. Hadn't looked in on the blogs lately. The null test is quite powerful of course. It should end some debates or at least allow people to test them like you have. It probably won't. Still excellent job by you here.
  22. Alkan's Avatar
    mitchco,
    Congratulations! Your work illuminated mine darks doubts about FLAC versus WAVE.
    Thank you very much.
    Dalton