Results 26 to 50 of 103
Thread: What is wrong with null testing?
-
06-06-2012, 12:47 PM #26
In my experience, null testing is wonderful and, performed properly, is 100% reliable....
if it is used to see if two things are identical or not.
As I mentioned in the other thread, something either nulls, i.e., cancels completely, or it doesn't. Used for comparing digital files, the null test is flawless. I used the words "performed properly" above. The reason for this is the files being compared must be synchronized *perfectly* lest the results turn out to be a false negative. There are no false positives in a null test.
By definition, a null = nothing, zero. Hence, I don't consider it the best approach to use a null test to examine *degree of cancellation*. Partial cancellation is not a null; anything that is not a null is, in my view, a non-null. (Again, by definition, there is no such thing as a partial null. There is a null or there isn't a null.)
I believe the trouble comes from using a null test when the results cannot possibly show a true null. This is compounded by the assumption that the degree of cancellation, quantified in dB, is translatable to the degree of audibility. This is why I repeatedly asked in the other thread, what sort of numbers one would *expect* from two very different sounding cables playing the *exact same program*. If the recording is recognizably the same, I personally would expect the cancellation to be *very* deep.
So for me, the question isn't really "what is wrong with null testing?" because there is nothing at all wrong with null testing - provided it is used to assess the presence or absence of a null. That is all it is supposed to do and it does it perfectly.
In my view, any assumptions based on a non-null result, other than the result is a non-null, are not logical. If we want to "objectify" cable comparisons, I believe we need to think of another approach.
I use the term "objectify" loosely as I don't subscribe to the objective-subjective dichotomy because it is arbitrarily exclusive. First, it excludes *observation*, which I do not see as limited to "subjective". To borrow from Pirsig, one does not need a thermometer to know a stove is hot enough to burn. To borrow from Dylan, one does not need a weatherman to know which way the wind blows. These are observations, separate from how one might *feel* about them (the latter being where I would use the term "subjective").
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 01:12 PM #27Junior Member
- Join Date
- Jan 2011
- Location
- Dallas, TX
- Posts
- 794
But "observations" can easily be fooled - examples abound. There's the school bus that rolls "uphill" over a set of railroad tracks - paranormal researchers claim it must be ghosts pushing the bus up and over the tracks, because your eyes show you very clearly the slope is uphill . . . except it's actually downhill, which can only be determined by *measurements*. There are both visual and aural illusions, and no one, not matter how experienced or golden-eared is immune, thus the need for objective measurements.
John Walker
Senior IT / Infrastructure Project Manager, Fortune 20 technology company
2x2TB HD > generic firewire cable > Mac Mini running iTunes + Audirvana Plus > HDMI > Onkyo TX-NR809 receiver > MartinLogan Motion series home theater speakers + M&K subwoofer
-
06-06-2012, 01:16 PM #28Propeller headed robotic parody of someone's idea of an inhuman objectivist
- Join Date
- Oct 2011
- Location
- Amsterdam
- Posts
- 3,129
- Blog Entries
- 3
Barry, did you read the Bill Waslo paper that mitchco linked to? Especially the "compensating for uninteresting differences"?
If you compare analog quantities you will never get a pure zero cancellation.
What is a "positive" and what is "negative" in a null test? To quote Wasio: "Sound left in Difference track might be real. Silence almost certainly is real."There are no false positives in a null test.
Well, maybe. Or maybe not. Let me quote from one definition: "In radio electronics, a null is an area or vector in an antenna's radiation pattern where the signal cancels out almost entirely."By definition, a null = nothing, zero. Hence, I don't consider it the best approach to use a null test to examine *degree of cancellation*. Partial cancellation is not a null; anything that is not a null is, in my view, a non-null. (Again, by definition, there is no such thing as a partial null. There is a null or there isn't a null.)
I know we disagree on interpreting Pirsig, but I again point out a definition: "a subject's personal perspective, feelings, beliefs, desires or discovery, as opposed to those made from an independent, objective, point of view"I use the term "objectify" loosely as I don't subscribe to the objective-subjective dichotomy because it is arbitrarily exclusive. First, it excludes *observation*, which I do not see as limited to "subjective". To borrow from Pirsig, one does not need a thermometer to know a stove is hot enough to burn. To borrow from Dylan, one does not need a weatherman to know which way the wind blows. These are observations, separate from how one might *feel* about them (the latter being where I would use the term "subjective").
So yes, you know the stove is hot. But there is no way for me to independently verify if you find the stove hot or not (apart from watching you jump and yell).Julf
"To try to judge the real from the false will always be hard. In this fast-growing art of 'high fidelity' the quackery will bear a solid gilt edge that will fool many people" - Paul W Klipsch, 1953
-
06-06-2012, 01:26 PM #29
You know one could apply the acoustic version of a complete null.
One gets complete nulls in digital files when nothing is there looking at the info.
One gets an acoustic complete null when playing back the result at the same original volume and hearing nothing.
I see no reason that wouldn't be a complete null. You are simply using your ears for the nulling rather than instruments.Computer>pre-amp>amp>speakers.
-
06-06-2012, 01:26 PM #30
Maybe there is confusion with terminology. I prefer the term difference testing as that is what is what is being discussed here and how Audio DiffMaker works. Digital differencing changes the problem from “can you hear a difference?” to “can you hear anything?” The result is the absolute signal difference in which you can use your ears to listen and draw your own conclusions.
In my own personal experience, what I have measured and what I have heard (and vice versa) correlate 1:1. I have shared those experiences on my blog and encourage others, like esldude has done, to try it out yourself as the software is free.
-
06-06-2012, 01:29 PM #31
Hi John,
I don't disagree. However, because something *can* be, does not mean it always *is*.
In the case of a phenomenon like listening to music, an *experiential* phenomenon, observation by an *experienced listener* is, I believe, the most reliable means of evaluation we have.
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 01:38 PM #32Propeller headed robotic parody of someone's idea of an inhuman objectivist
- Join Date
- Oct 2011
- Location
- Amsterdam
- Posts
- 3,129
- Blog Entries
- 3
Julf
"To try to judge the real from the false will always be hard. In this fast-growing art of 'high fidelity' the quackery will bear a solid gilt edge that will fool many people" - Paul W Klipsch, 1953
-
06-06-2012, 01:45 PM #33
Hi Julf,
Yes I did. Mr. Waslo contacted me early on in the history of his Diffmaker app, to ask my permission to use a sample from one of my Soundkeeper Recordings on his site (which I gave him).
Agreed. That is why I said I don't believe a null test is the correct approach for this.
A positive is a null. A negative is no null.
Yes, how one *feels* about what is observed is subjective. (I said this in my post.)
What is observed, in my view, is neither subjective or objective. I can observe, by approaching or touching that a stove is dangerously hot. This I can communicate to you by saying "Julf, be careful. That stove is dangerously hot."
As I see it, you would then have three possible courses of action:
1.You can either take my word for it
2.You can take my word enough to be cautious
3. You can say "that is 'subjective' and in the absence of 'confirmation', I don't believe it" and go ahead and place your hand on the stove.
You would have the same three options if you saw me jump and yell. ;-}
I can think of all the great audio gear I've had the good fortune to experience. In all the decades I've been interested in this subject, never once have I found something special from looking at its spec sheet. It has always, without exception, been because another listener (whose ears I trust, a point I consider important) has reported the item as being something special. This leads me to investigate for myself. Sometimes, I agree, other times I don't.
To be clear, I applaud the search for numeric quantification of what we experience. I just think we have a long, long way to go before most of (let alone everything) what an experienced listener will hear can be measured (let alone quantified). Right now, I see our measurements as being considerably more useful in pointing out flaws (and not all flaws, just the bigger, more obvious ones) than they are in pointing out assets.
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 01:48 PM #34
Hi Dennis,
To play devil's advocate here, isn't that introducing a "subjective" element to an "objective" test?
Why is that any more "valid" than just listening to the cables play music (a more representative, realistic test in my view) and making your determination that way?
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 01:58 PM #35
Hi Julf,
Okay. If you prefer, a phenomenon like listening to our equipment reproduce music is an *experiential* phenomenon. It is still listening.
The reproduction is a technical process, agreed. It can be analyzed (to a limited degree), agreed. But the evaluation is, in my experience, at this time in history, hopelessly inadequate.
Would you purchase a set of speakers or any other component based solely on the spec sheet? If so, have you?
I have yet to see a measurement that tells me about the complete dynamic behavior of say, a loudspeaker. On overall dynamic range within a certain degree of harmonic distortion can be quantified. But where is the differentiation across the frequency band? (Some gear is more dynamic in some areas of the spectrum than elsewhere.) Where is the differentiation for different dynamic levels? (Some gear is more dynamic within some parts of the overall dynamic range than it is in other parts. Some does macro dynamics but is not so good with micro dynamics.) Where is the quantification of how clearly images on the soundstage are differentiated and of how "solid" those images appear to the listener? What is the number that tells me about soundstage depth or of how far back into the soundstage width is maintained? These are just a start on a long list of items I've not seen a specification for. But a short conversation with an experienced listener, whose ears I've come to trust, can give me at the least, a pretty good picture of all of these. Certainly enough to know if I'm interested in an audition of the product (or not).
In the end, it can be said that *all* processes are "technical". Human orgasm is a technical process too and muscle contractions and heart rates can be measured. Do we need to be hooked up to an EKG to know if the experience was satisfying? ;-}
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 02:05 PM #36Propeller headed robotic parody of someone's idea of an inhuman objectivist
- Join Date
- Oct 2011
- Location
- Amsterdam
- Posts
- 3,129
- Blog Entries
- 3
Hi, Barry,
Excellent!
I think I would go for a variation of 3. - before touching the stove, I would quickly point an IR thermometer towards it to check how hot it actually is.What is observed, in my view, is neither subjective or objective. I can observe, by approaching or touching that a stove is dangerously hot. This I can communicate to you by saying "Julf, be careful. That stove is dangerously hot."
As I see it, you would then have three possible courses of action:
1.You can either take my word for it
2.You can take my word enough to be cautious
3. You can say "that is 'subjective' and in the absence of 'confirmation', I don't believe it" and go ahead and place your hand on the stove.
In your example, I would of course not have any problem taking your word for it. But imagine that instead of you, there was someone selling special insulated oven mitts for $1200. Would you take his word for it that they work OK, or would you like to have some objective verification before touching the stove?
That I can definitely agree with. My problem is with people who say "but you don''t even need to measure - I can reassure you there are no flaws".Right now, I see our measurements as being considerably more useful in pointing out flaws (and not all flaws, just the bigger, more obvious ones) than they are in pointing out assets.Julf
"To try to judge the real from the false will always be hard. In this fast-growing art of 'high fidelity' the quackery will bear a solid gilt edge that will fool many people" - Paul W Klipsch, 1953
-
06-06-2012, 02:13 PM #37Propeller headed robotic parody of someone's idea of an inhuman objectivist
- Join Date
- Oct 2011
- Location
- Amsterdam
- Posts
- 3,129
- Blog Entries
- 3
Yes, on both counts.
I think a waterfall graph comes pretty close.I have yet to see a measurement that tells me about the complete dynamic behavior of say, a loudspeaker
You might be faking it...Human orgasm is a technical process too and muscle contractions and heart rates can be measured. Do we need to be hooked up to an EKG to know if the experience was satisfying? ;-}
Julf
"To try to judge the real from the false will always be hard. In this fast-growing art of 'high fidelity' the quackery will bear a solid gilt edge that will fool many people" - Paul W Klipsch, 1953
-
06-06-2012, 02:18 PM #38
Well it is an observation using a different method than putting on music and listening to it.
Putting on music and listening to it is an observation. The description of what is observed this way is subjective.
In the acoustical null test, you either hear something or you don't. You have no need to describe it beyond yes I hear something or no I hear nothing.
Because description isn't necessary and other reasons I described in the opening three posts on this thread, I believe such difference testing can be more discriminating than just listening to the music to hear differences. Not that differences cannot be heard both ways. Just that one way shows smaller differences than another.Computer>pre-amp>amp>speakers.
-
06-06-2012, 02:56 PM #39
Hi Julf,
Different example. I'm not talking about what someone selling a product tells me, I'm talking about an experienced user whom I trust. If an experienced chef I trusted told me about the oven mitts, I'd ask them if they offered anything over the $2 mitts that already provide complete protection. ;-}
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 03:00 PM #40
Hi Julf,
Yikes! I wouldn't consider it. Just want speakers are those?
To describing the complete dynamic behavior of a loudspeaker?!? I would not agree.
What part of the waterfall graph, for example, is telling you about the microdynamic performance on the attack of piano keys on a Steinway, while the drummer is going full tilt on the drums?
I was talking about *you*, not your partner. Even in the case of the latter, well... I'll just say some aspects of the thing cannot be faked (except perhaps by a Zen master in complete control of their autonomic reflexes). ;-}
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 03:15 PM #41
Hi Dennis,
Agreed.
Here we must agree to disagree. I would say how one *feels* about the observation is the subjective part. As I see it, the observation itself is no more "subjective" than the belief that measurements (the map) can be fully equivocated with the listening experience (the territory) is "objective".
Here again, we must agree to disagree.
I've often found what is inaudible to one listener is plain to the other.
Folks have often described hearing things that, try as I might (and I do), I simply don't hear.
Conversely, I remember an experience at Atlantic when there was a rush job and in listening to the source tapes, I heard what to me sounded like quite obvious and annoying clicks. I went to management to say we needed to delay the job and request another source tape. The studio manager came into the studio to hear what I was describing. I had to play the same section of the song for him three times, on the third, visually cueing him by pointing to the speaker at the instances of the clicks.
He said words my mother would not be pleased to hear anyone say. Then he asked me to play the section again. Then he delayed the job and sent the request for a replacement tape.
As to whether difference testing can be more discriminating than just listening to the music, I would cite the example of your hearing differences between the cables, then upon interpreting the numbers you got from the partial cancellation as meaning such differences are not audible (or was it not significant), you questioned what you heard.
To me, a subjective interpretation of what the numbers say influenced the conclusion, despite what your ears tried to tell you.
Or maybe I have it wrong (in which case please accept my apologies), so I'll speak in terms of myself. If I question whether I'm hearing a difference or not, I enlist the aid of my assistant to do the switching while I listen, without knowing which I'm hearing at a given moment. (I did this a few weeks ago in comparing two different music server apps where I sometimes thought one *might* be more "open" sounding but wasn't sure. In the end, with my assistant's help, I concluded I do not hear any difference between those two apps.)
In the case of cables, I've listened to a lot of them over the years and have yet to hear two that sound the same. Enlisting my assistant's aid, the results don't change; I can pick the cable. I say this not to convince anyone but merely to report my own experience. I'll add that in some cases, the differences are considerably more pronounced than in others. Further, all but one in my experience, has changed its sound (for the better) after having played music for ~50 hours. This was confirmed (for me) by having an identical but unused set to listen to after the first set had been used for a few weeks; with the unused set, all the refinements attained in use on the first set disappeared.
I think we need another sort of test -- for those who seek a test.
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 04:37 PM #42
Barry
I can think of one that would be extremely informative. Due to your knowledge and experience in this area you would seem almost the perfect choice for a properly conducted DBT. You say you can pick cables with the help of your assistant. Can you do it without the assistant? How about subjecting yourself to a DBT where you are happy with the protocol/design and the result can be independently verified and report back to us. For me, get them right >75% of the time in 20 attempts and you win, game over.
This is the incredibly frustrating thing about this whole saga. For decades now we have listeners like yourself saying you can do these things, yet none of you will front up and prove it. Get in touch with Randi and there might even be a million bucks in for you.
-
06-06-2012, 04:47 PM #43Main Music: AIFF Library -> Mac Mini i5 (Late 2012) -> MacOS 10.8.3 -> JRMC 18 -> Siltech Optical -> Jolida Tube DAC II -> Parasound M2100 Preamp -> Outlaw Audio M2200 Monos -> Nodost Flatline MKII Speaker cables -> PSB Synchrony 1Bs on 36" stands
Vinyl -> Audio Technica LP120 w/ AT440MLa cart installed -> Phono input on Parasound M2100
Video -> NAD 557 Bluray + Apple TV 3g -> NAD T747 -> Preouts -> Parasound M2100 HT Bypass -> same as music
Bedroom -> Macbook Pro -> JRMC18 -> Peachtree DAC*IT -> NAD B33326 -> PSB Imagine Bs
Office -> Mac Mini i5 -> Amarra -> Kimber USB -> Wavelength Proton -> Creek e50 -> Maggie MMGs
-
06-06-2012, 04:59 PM #44
Hi prufrock,
A few points if I may...
1. Do you hear differences between cables?
If you do, my results don't matter. If you don't, my results don't matter.
I could really end this post here but...
2. I undertake the tests in order to convince myself. I have no interest in convincing anyone else.
I could end it here too. But...
3. In fact, I *have* been in touch with the hack who refers to himself as Randi.
Well, I've responded to his empty "challenge" and did not hear back from him or any of his representatives. My name was also submitted by other folks, who also received no reply.
He is a showman, for those who would find his type of showmanship entertaining, nothing more. He boldly talks the talk and then disappears when it come time to do the walk.
All that said, if you really think it so important that I engage in a double blind test, please arrange it but also consider that I must be paid my hourly rate if I'm to turn away work in order to spend the time participating. Perhaps you can write to Randi and see if he responds to you. ;-}
Really, I say again, what I hear or do not impacts no one but myself. It will have zero effect on what you hear or don't hear. When I referred to needing another test, I'm talking about a means of measuring/quantifying, as Dennis sought to do in his null test.
Best regards,
Barry
Soundkeeper Recordings
Barry Diament Audio
-
06-06-2012, 05:05 PM #45
I hear your saying we need another test. I just don't know what it would be. Don't know what would fail to be uncovered in this type test. That is one of the things I thought someone might comment upon and point out.
Computer>pre-amp>amp>speakers.
-
06-06-2012, 05:19 PM #46
Paul
I dont find Randi annoying. I find him a valuable counterbalance in a world full of woolly thinkers and bulldust.
Barry,
you say: "When I referred to needing another test, I'm talking about a means of measuring/quantifying, as Dennis sought to do in his null test." Whats the point? You will just say oh, sorry, that test must be missing something too, as I can still hear the difference.
This thing will only ever get settled when listeners like yourself and Paul front up and show you can do it in an independently verifiable test.
-
06-06-2012, 05:31 PM #47Sophomore Member
- Join Date
- Mar 2012
- Location
- Laramie, Wyoming USA
- Posts
- 103
Some of the properties Barry mentioned seem to be things there should be some science about and some things to be measured. A brief google search suggests that there is research on how humans perceive distance to sound sources, for example. I'm guessing this would be related to depth of sound stage and imaging. I'm thinking there must at least be some theories about what causes these perceptions in humans. Anyone have more info on this?
JohnMH
-
06-06-2012, 05:51 PM #48
Just some anecdotal info on sound stage and imaging. Messing about with EQ one can push things forward or back, widen or narrow the apparent width. All very educational. Try bumping up and knocking down the 1500 to 4000 hz range as a start by one decibel each way. The effect on better recordings is usually pretty obvious. Something you can easily do with the EQ in Audacity. Just make copies of files to protect your originals and play with them some.
Computer>pre-amp>amp>speakers.
-
06-07-2012, 01:17 AM #49
Hey esldude, that is because this is the right test :-)
To answer your question about what is wrong with null testing, the answer is nothing at all. It is a proven, valid, mathematical technique to measure the absolute signal difference (in the case of DiffMaker, 300 dB range of correlation) between a reference signal and a comparison signal.
Do we have what it takes to measure everything audio. Absolutely, most of it was figured out over 50 years ago with the 4 audio parameters from The Audio Myths Workshop (about 21 minutes in): Audio Myths Workshop - YouTube

Today, our computers are so powerful and DSP software so advanced that we can easily reproduce any sonic signature we want.
Take this quote from Ethan Winer’s article on, "Why We Believe – A common sense explanation of audiophile beliefs”
“Audio is a science, and most aspects of the science of audio have been understood fully for 50 years or more. Yet audiophiles sometimes report being able to hear things that defy what is known about the perception of sound. For example, some people claim to hear a difference between speaker cables even when both cables have very similar resistance and other properties. When pressed, they often say they believe they can hear things that science has not yet learned how to measure. Is this really possible? Modern audio test equipment is capable of measuring everything known to affect sound quality over a range exceeding 120 dB, yet nobody I know of can hear artifacts even 80 dB below the music while it is playing. In my experience, the top 20 to 30 dB matters the most.”
Ethan believes comb filtering is the root of all evil. I generally agree, it is a matter of degree, and has been consistent with my listening and measurement tests.
Another article from Ethan that is worth reading is the Artifact Audibility Comparisons:
“Debate rages in both audiophile and professional audio circles about the importance of low-level artifacts such as distortion, jitter, quantization noise, and summing errors in DAW software. This article addresses the audibility of very soft artifacts, and includes Wave files you can download to discover for yourself at what volume level these artifacts can be heard.”.
Ours ears (read: brain) are easily fooled. Spend 10 minutes listening to the beginning of the Audio Myths Workshop and you can see (and hear!) for yourself. This is called psychoacoustics.
One such psychoacoustic effect is the Hass effect. As an ex-recording/mixing engineer, the Hass Effect is used extensively to fool people’s ears in believing sound is coming from a specific location when adding digital delay to one channel. Hey, how do you think we create stereo when not using stereo mics! You can hear it for yourself in this video:
What’s my point. If folks reading this are really interested in the science of audio, then here are a few books I would recommend as I feel they capture and represent the science in a practical light
Principles of Digital Audio
Anything and everything digital audio – the reference.
Mastering Audio – the art and the science – Bob Katz does an amazing job of capturing from the mastering chair to your listening chair, the science, the process and the art. Excellent read.
Acoustics – well how about a master list of acoustic links
Given that the speaker to room interface is the weakest link in the audio reproduction chain, it would be cool to see more discussion (and pictures!) of peoples listening environments, and what they have done in the way of improving acoustics.
Btw, awesome job esldude – I really appreciate your efforts as I understand how much effort it took ;-) Cheers! Mitch
-
06-07-2012, 05:58 AM #50
Well, proof of the pudding and all that....
"We don't know everything yet," "We should pay attention to these other measurements as well," and "Even low level artifacts affect the illusion of reality" gave us async USB DACs, Spectral equipment, etc.
"We know everything that matters" and "Low level artifacts are masked and inaudible" gave us "Perfect [CD] sound forever" and mp3.
So whatever you prefer to listen to....One never knows, do one? - Fats Waller
WD MyBook FW -> MacBook Pro w/SSD (Audirvana Plus) -> Mapleshade Clearlink USB (Plus version) -> Semi-customized DAC (plays DSD natively; any necessary oversampling done prior to DAC in software; for more detail see blog) -> Spectral DMC-12 & DMA-150 -> Vandersteen 2Ce. Other cabling and power strip Omega Mikro/Mapleshade. Also MIT Z-Stabilizer.



LinkBack URL
About LinkBacks
Reply With Quote





