Jump to content
IGNORED

Software for audio testing and evaluation


Recommended Posts

This is a spin off thread from the Blue or red pill?  thread to discuss the use of Audio DiffMaker and similar software in audio testing. 

 

I've tried to use the software to detect some differences in analog captures with varying amount of success. Some things worked, some didn't, and it seems I got different results than others using the same software on the same audio files.

 

What software do you use to help pick out differences in equipment? What settings are best? What works what doesn't?  Do you have ideas for new software or for new methods of comparison?

 

 

 

 

Link to comment

it would help to briefly note what the software does too - e.g. does DiffMocker subtract one input from the other for instrument analysis?

 

------

 

My idea for software would be a randomized tester A/B/X box - this would allow 1 person to do DBTs, instead of dragging in a gf, spouse, etc. who might be more usefully employed pumping the clutch & brake pedals on your sports car while you flush the hydraulic fluid... :/

Link to comment

You might want to look at the slides in the AES presentation for Diffmaker.

 

Especially page 15 and 16 for the formulae about how gain mismatch and phase mismatch will effect null depth. 

 

http://www.libinst.com/Detecting Differences (slides).pdf

 

The paper itself is here.

 

http://www.libinst.com/AES Audio Differencing Paper.pdf

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

I tried DiffMaker some years ago, separated by some time between 2 efforts to try and get something useful from the program - both times I gave up in disgust; the software as a program just has too many bugs and inadequacies to be able to rely upon; and when fed deliberately hand tainted tracks to compare it failed miserably to find what the true differences were ... a poor tool is worse than useless.

 

Foobar for doing ABX is highly flawed - it has similar issues to the blue or red pill experiment: what you are listening to in the ABX are not the true A and B, it fails at the first hurdle.

 

At one stage I was motivated to develop my own DiffMaker, using the Octave package for manipulating the data - I ran out of steam, and it sits there as an idea only.

 

Otherwise, I use Audacity, with the option to call in SoX if I need more muscle.

Link to comment
6 hours ago, Ralf11 said:

it would help to briefly note what the software does too - e.g. does DiffMocker subtract one input from the other for instrument analysis?

 

------

 

My idea for software would be a randomized tester A/B/X box - this would allow 1 person to do DBTs, instead of dragging in a gf, spouse, etc. who might be more usefully employed pumping the clutch & brake pedals on your sports car while you flush the hydraulic fluid... :/

At least a couple of those were made for a time.  One was a semi-DIY situation where someone had the design worked out for you to copy it along with purchasing from them key components.  Another was offered for sale complete.  They aren't in real big demand I don't suppose.  

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

If the first couple seconds are low in level and also noisy it will be hard to get a matched peak and gain.  Imagine doing this yourself.  If you see the first couple seconds in a sound editor are too noisy and variable you just skip to a louder portion.  Diffmaker apparently doesn't do this.  So noisy low level initial seconds can cause it to go haywire.  So you can help it by slicing off silent portions near the beginning. Also at the end.  It uses the last couple seconds to adjust sample rate drift.  If I am making signals to run thru some gear I will sometimes append a simple -3db tone the first and last 5 seconds to make Diffmaker's job easy at the beginning and end.  Or sometimes I'll put a steady mid-frequency constant tone in one channel and music in the other channel. 

 

So files need to have some good signal in them. What happens if they don't? Sometimes it just blows up and the program becomes non-responsive or shows an error.  Or it may appear to work just fine, but the result makes no sense.  Like a pair of signals you can tell yourself are very similar and it will give a null depth of -3 db.  Noise might almost give that result. 

 

If you try and do too large a file with sample rate drift enabled it may just blow up.  I rarely get it to work for more than 30 second files.  Sample rate drift correction in general may make it blow up.  If you select for increased accuracy in the settings for timing correction or drift correction it becomes almost sure to blow up.  What it is doing is incredibly intensive computationally and some situations just leave the algorithm likely to malfunction.  

 

You also can let Diffmaker do a comparison and then tighten up on some parameters like time accuracy and sample rate drift.  Leave the settings for lower accuracy and it is more likely to work.  Diffmaker leaves the corrected file in the folder where the music came from.  You can take this corrected file and use it to compare to the reference a second time with the timing settings tightened up and sometimes it will work then where it would blow up before.  You've essentially let it do half the work and then do more work with less variability the second time. 

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment

So, keep your files compared short, 30 seconds or less. 

 

Make sure there is some signal in the first and last few seconds. 

 

Don't use sample rate drift unless you have to do so.  

 

Why does someone put up with such a unreliable thing?  Because when it works it can compare things to a level of accuracy that is difficult for a normal person to even approach any other way.  Because it is so unreliable some discount the results completely and won't use it.  My opinion is if you get a deep null (what is deep.........-60 db or more) it worked correctly and is not in error.  It usually blows up or has errors showing very shallow low value nulls when it has messed up.  So in my mind when it works you can count on it, if it doesn't work all you can say is it didn't tell you anything useful.  

 

How can you check on the results?  Depends upon the particulars of the situation.  One I used to do was using software to speed up one file and compare to another in Audacity.  If you were using the same gear with the same clocks changing something else the speed was right, but the exact sample times didn't match.  In Audacity I would load both files, and use Change Speed to speed one up slightly.  Then "invert" and "mix and render" them together.  At first they don't cancel out, but as one file drifted in time relative to the other there is some point where they do cancel out very deeply.  If you see that the files are very similar they just aren't lined up in time. Diffmaker can usually line that up and get a good deep null. 

 

But I only use Diffmaker when I don't have any other options.  It just makes you want to pull your hair out the way it is so flaky sometimes. 

 

I think a development team of programmers could enhance it to check on itself and prevent these blow ups.  It would just be an iterative approach fixing one obvious gotcha and then another.  Waslo even did some of that as it will sometimes tell you, the files are too different to compare.  So it is a shame some group of people aren't interested enough to shape it up. 

 

Oh it one last thing.  It runs differently on some hardware than others.  By that I mean it blows up differently. In one case it would run the same test again and give different results.  Though it doesn't usually do that.  I've found it most reliable on a Linux machine run via WINE.  It might work well in an XP emulation of Virtual machines.  

 

Oh and you should have read the links I posted above.  Was easier than my explanations. I expect lots of people won't do that however. 

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
13 hours ago, Ralf11 said:

it would help to briefly note what the software does too - e.g. does DiffMocker subtract one input from the other for instrument analysis?

 

------

 

My idea for software would be a randomized tester A/B/X box - this would allow 1 person to do DBTs, instead of dragging in a gf, spouse, etc. who might be more usefully employed pumping the clutch & brake pedals on your sports car while you flush the hydraulic fluid... :/

 

DiffMaker first matches up the waveforms from the two recordings, ensuring they are as close as possible in phase and amplitude, and then subtracts them, letting you listen to the differences. Very nice idea, except that it appears to be not maintained and crashes and misbehaves frequently.

 

Doesn't Foobar have an ABX comparator tool? I've written a simple utility for me to do this with HQPlayer -- basically randomizes tracks, and allows me to randomize HQPlayer settings. I did this primarily to evaluate various filters and upsampling rates HQPlayer provides, as well as various impulse response files for the convolver. 

 

 

Link to comment
11 hours ago, esldude said:

You might want to look at the slides in the AES presentation for Diffmaker.

 

Especially page 15 and 16 for the formulae about how gain mismatch and phase mismatch will effect null depth. 

 

http://www.libinst.com/Detecting Differences (slides).pdf

 

The paper itself is here.

 

http://www.libinst.com/AES Audio Differencing Paper.pdf


I've read the paper. I've actually spent some time in the past life doing something similar for a very different problem (astronomical telescope periodic error correction). Matching up two noisy periodic signals exactly is not simple. I found Diffmaker frustrating because it really didn't seem to produce a consistent result and crashed often. You mentioned on another thread that it's open source, but I was unable to find the sources. Do you have a reference? I'm in the process of writing my own version, but if I can instead fix Diffmaker, it might be a lot quicker :)

Link to comment
4 minutes ago, pkane2001 said:

 

Looks nice, but what do you do with it? 

 

 

I only use that for stereo width and depth. The stereo meter is very useful for speakers setup and comparing the recorded acoustic  sound from the speakers. You can download the trial version and compare mani’s two analogue ( I trimmed to the first 18 seconds) and it should show the difference. 

Link to comment
11 hours ago, fas42 said:

I tried DiffMaker some years ago, separated by some time between 2 efforts to try and get something useful from the program - both times I gave up in disgust; the software as a program just has too many bugs and inadequacies to be able to rely upon; and when fed deliberately hand tainted tracks to compare it failed miserably to find what the true differences were ... a poor tool is worse than useless.

 

Foobar for doing ABX is highly flawed - it has similar issues to the blue or red pill experiment: what you are listening to in the ABX are not the true A and B, it fails at the first hurdle.

 

At one stage I was motivated to develop my own DiffMaker, using the Octave package for manipulating the data - I ran out of steam, and it sits there as an idea only.

 

Otherwise, I use Audacity, with the option to call in SoX if I need more muscle.

 

Thanks, Frank. I'm in a similar situation, with similar tools. Diffmaker -- frustrating. SoX is great for complex manipulation. Audacity is  pretty good as a general purpose audio examination and simple editing tool.

 

Link to comment
5 hours ago, esldude said:

So, keep your files compared short, 30 seconds or less. 

 

Make sure there is some signal in the first and last few seconds. 

 

Don't use sample rate drift unless you have to do so.  

 

Why does someone put up with such a unreliable thing?  Because when it works it can compare things to a level of accuracy that is difficult for a normal person to even approach any other way.  Because it is so unreliable some discount the results completely and won't use it.  My opinion is if you get a deep null (what is deep.........-60 db or more) it worked correctly and is not in error.  It usually blows up or has errors showing very shallow low value nulls when it has messed up.  So in my mind when it works you can count on it, if it doesn't work all you can say is it didn't tell you anything useful.  

 

How can you check on the results?  Depends upon the particulars of the situation.  One I used to do was using software to speed up one file and compare to another in Audacity.  If you were using the same gear with the same clocks changing something else the speed was right, but the exact sample times didn't match.  In Audacity I would load both files, and use Change Speed to speed one up slightly.  Then "invert" and "mix and render" them together.  At first they don't cancel out, but as one file drifted in time relative to the other there is some point where they do cancel out very deeply.  If you see that the files are very similar they just aren't lined up in time. Diffmaker can usually line that up and get a good deep null. 

 

But I only use Diffmaker when I don't have any other options.  It just makes you want to pull your hair out the way it is so flaky sometimes. 

 

I think a development team of programmers could enhance it to check on itself and prevent these blow ups.  It would just be an iterative approach fixing one obvious gotcha and then another.  Waslo even did some of that as it will sometimes tell you, the files are too different to compare.  So it is a shame some group of people aren't interested enough to shape it up. 

 

Oh it one last thing.  It runs differently on some hardware than others.  By that I mean it blows up differently. In one case it would run the same test again and give different results.  Though it doesn't usually do that.  I've found it most reliable on a Linux machine run via WINE.  It might work well in an XP emulation of Virtual machines.  

 

Oh and you should have read the links I posted above.  Was easier than my explanations. I expect lots of people won't do that however. 

 

Thanks, for the detailed explanation, Dennis. I'm seeing if I can rewrite at least some of the core functionality of Diffmaker. Doing coarse matching, then aligning and computing energy differences, aligning to reduce differences. Shifting data by a fraction of a sample isn't hard in frequency domain, so yes, lots of huge FFTs/IFFTs required :) I'm planning on also doing the timing drift correction.  I believe Diffmaker only does a linear correction. I guess I'll start with that, but I may also test correcting for non-linear drift.

 

Link to comment
10 minutes ago, STC said:

 

I only use that for stereo width and depth. The stereo meter is very useful for speakers setup and comparing the recorded acoustic  sound from the speakers. You can download the trial version and compare mani’s two analogue ( I trimmed to the first 18 seconds) and it should show the difference. 

 

Interesting. Does this mean it measures phase differences between two recordings? That would be the opposite of what Diffmaker does, as it tries to adjust all the phase differences out.

Link to comment
1 minute ago, pkane2001 said:

 

Interesting. Does this mean it measures phase differences between two recordings? That would be the opposite of what Diffmaker does, as it tries to adjust all the phase differences out.

 

It should be as phase is one of the factor that affects soundstage. I think someone with recording background can provide a better answer. 

Link to comment

not related to abx or nullification, but I'm looking for a way to capture the FFT graphs for various filter settings of upsampling software.  

 

I have a mac and a Logic Pro 10 (which is a recording software), but I don't think it has these type of tools.  

 

Can anyone point me in a better direction?  

Link to comment
3 hours ago, buonassi said:

not related to abx or nullification, but I'm looking for a way to capture the FFT graphs for various filter settings of upsampling software.  

 

I have a mac and a Logic Pro 10 (which is a recording software), but I don't think it has these type of tools.  

 

Can anyone point me in a better direction?  

Audacity is available for Mac.  It has FFT.

And always keep in mind: Cognitive biases, like seeing optical illusions are a sign of a normally functioning brain. We all have them, it’s nothing to be ashamed about, but it is something that affects our objective evaluation of reality. 

Link to comment
  • 2 weeks later...

@pkane2001

Reading the "pill" thread, it looks like you are making good progress on your software.  Kudos!

 

One thing I have been wondering... if gain adjustment is required even for recordings that are done one right after the other, this implies that there is some gain drift in the equipment.  Do you think there would be any improvement by adding a gain drift correction, instead of simply correcting with a constant like Audio Diffmaker does (at least as I understand it)?

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...