A/B testing favors B over A

[mmerrill99]

10 minutes ago, pkane2001 said:

 

While I've encountered (many times) differences in sighted tests that I could swear were very obvious, I frequently could not make the same distinction in a blind test. To me, this is an indication of a failure of the sighted test to control for subjective variables, and not a failure of a blind test to discover differences.

 

So in your deflecting, politician's answer, you are saying that you never encountered a situation that I asked you about - is this correct - yes or no will do this time?

 

21 minutes ago, mmerrill99 said:

But this short term A/B listening does not always result in identifiable differences & yet the 'feel' is different between two devices. Have you encountered this?

 

[pkane2001]

Just now, mmerrill99 said:

So in your deflecting, politician's answer, you are saying that you never encountered a situation that I asked you about - is this correct - yes or no will do this time?

 

I think I answered very clearly.

[mmerrill99]

9 minutes ago, Daudio said:

 

Merrill,

Thanks, I think that is the best, simple, 10,000 ft up, analysis of this whole fervor over audiophile testing methods I've yet seen, cutting through the bias, obscuring details, and bullshit ! It should be copied to a pinned thread to guide all of us.

 

And if one isn't in this hobby to chase better sound, then what the hell are they doing it for ? Waste money, exercise their oscilloscopes, or Online Armored Combat ?

 

Thanks, yes, this comparison of flaws is like playground bickering - whose father drives the biggest car!

[mmerrill99]

3 minutes ago, pkane2001 said:

 

I think I answered very clearly.

A typical politican's avoidance tactic - well done for sticking so avidly to the stereotype!

[pkane2001]

Just now, mmerrill99 said:

A typical politican's avoidance tactic - well done for following the stereotype!

 

Appreciate the feedback. So long, @mmerrill99.

[mmerrill99]

3 minutes ago, pkane2001 said:

 

Appreciate the feedback. So long, @mmerrill99.

Your welcome!

[mansr]

22 minutes ago, Daudio said:

And if one isn't in this hobby to chase better sound, then what the hell are they doing it for ? Waste money, exercise their oscilloscopes, or Online Armored Combat ?

I'd rather spend $5k on an oscilloscope than a power cord. Others would, apparently, get the power cord. I have yet to meet anyone who'd get a power cord for the oscilloscope.

[Jud]

47 minutes ago, jabbr said:

 

Yes but... when I compare two sounds, be they components or instruments or recording I "feature extract" that is commit my impression to memory be that smoothness of a string in a certain octave or extension of bass or position of instruments on soundstage, and these features can be remembered longer than 10sec

 

How does this feature extraction capability relate to the results of this experiment?

 

http://deutsch.ucsd.edu/psychology/pages.php?i=209

[Daudio]

1 hour ago, Jud said:

The "less than a minute" time frame isn't nearly short enough.  Scientific research shows echoic memory for everything except loudness lasts maybe 4-10 seconds.

 

I think this whole 'short echoic memory' thing is a red herring. Obviously we remember sounds for far longer then a few seconds. How long have you known the sound of your mother voice, in all it's different volumes, moods, health, age, emotion, inflection, etc. ? How can one hear the differences in a new remaster of a familiar album, if one can't recall sound beyond a few seconds ? How can a mother pick out the sound of only her baby from a group of infants ?

 

I think that this 'echoic memory' thing is a myth build up from agenda driven extrapolation of the only formal study of anything having to do with audio 'memory. But it just doesn't pass a smell test, of course we remember sounds for long periods, some most of our lives !

 

 

1 hour ago, Jud said:

"[V]ery specific in a familiar sound track" is a really good thing.  It means you may not actually be trying to compare your memories of the two sounds (see the first point; after a few seconds this is fruitless, *especially* if there's intervening music

 

Another thing that bothers me about this 'short A/B test' thing involves the 'test signals', which I find extremely limited, and possibly of little to no use in showing any difference between UUTs. What I mean is that in the other, longer term, types of testing we can expose our ears to many different stimuli (albums, tracks, clips) to compare lots of effects for differences, which may only be exhibited in a small set of recorded musical moments. Dynamic range, dynamics across frequency, noise levels and character, the multiple aspects of 'soundstage', subtle musical cues, details of breath, bowing, plucking, etc. as well as specific cuts that may help some uncover subtle points of tonality. Then you can add in the subtle, longer term reactions, like emotion, interest, boredom and fatigue.

 

The point is that musical test signals that can show differences are not particularly common, or even known in advance, and are quite diverse, and often individualistic, so how can one, or a very few, preselected, very short clips, be expected to excite the differences we are interested in teasing out of our audio testing ?

 

Doesn't make any sense to me

 

 

P.S. Not specificly directed at you Jud, just a vehicle to express these thoughts.

 

[Jud]

3 minutes ago, Daudio said:

 

I think this whole 'short echoic memory' thing is a red herring. Obviously we remember sounds for far longer then a few seconds. How long have you known the sound of your mother voice, in all it's different volumes, moods, health, age, emotion, inflection, etc. ? How can one hear the differences in a new remaster of a familiar album, if one can't recall sound beyond a few seconds ? How can a mother pick out the sound of only her baby from a group ?

 

I think that this 'echoic memory' thing is a myth build up from agenda driven extrapolation of the only formal study of anything having to do with audio 'memory. But it just doesn't pass the simplest smell test, of course we remember sounds for long periods, some most of our lives !

 

 

 

Another thing that bothers me about this 'short A/B test' thing involves the 'test signals', which I find extremely limited, and possibly of little to no use in showing any difference between UUTs. What I mean is that in the other, longer term, types of testing we can expose our ears to many different stimuli (albums, tracks, clips) to compare lots of effects for differences, which may only be exhibited in a small set of recorded musical moments. Dynamic range, dynamics across frequency, noise levels and character, the multiple aspects of 'soundstage', subtle musical cues, details of breath, bowing, plucking, etc. as well as specific cuts that may help some uncover subtle points of tonality.

 

The point is that musical test signals that can show differences are not particularly common, or even known in advance, and are quite diverse, so how can one, or a few, preselected, very short clips, be expected to excite the differences we are interested in teasing out of our testing ?

 

Doesn't make any sense to me

 

 

 

I think you're half right.

 

The short duration of echoic memory is an experimentally verified reality.  The things you are talking about (sound of mother's or child's voices) fall into the category of "pattern matching."  The fact that humans do this sensory "pattern matching" is another known, experimentally verified reality.

 

With regard to the "race" between echoic memory and the types of sonic phenomena we are thinking of when we talk about high end audio - fine discrimination of timbre, soundstaging, etc. - I completely agree.  It's why I ran a little test of timbre discrimination on the forum (in which 50+ people participated) that did not depend on echoic memory, but did test for ability to make that timbre discrimination over a short versus a longer period of time:

 

https://www.computeraudiophile.com/forums/topic/22709-heres-a-little-test-you-can-all-join-in/#comment-390666

[mmerrill99]

20 minutes ago, Jud said:

t's why I ran a little test of timbre discrimination on the forum (in which 50+ people participated) that did not depend on echoic memory, but did test for ability to make that timbre discrimination over a short versus a longer period of time:

 

https://www.computeraudiophile.com/forums/topic/22709-heres-a-little-test-you-can-all-join-in/#comment-390666

Can you summarize the results of this test please as 10 pages of posts is a lot to read through ?

[Daudio]

23 minutes ago, Jud said:

The things you are talking about (sound of mother's or child's voices) fall into the category of "pattern matching."

 

I have said before that people are using this "pattern matching" term as a kind of catch-all buzzword, that obfuscates much more then illuminates. Pattern matching is a generic term, and doesn't really have anything to do with audio memory, except, perhaps for some low level machinery. It does NOT help us understand how we remember sounds for more then a few seconds.

 

I also think that there is a subtle bias in many true and quasi audiophiles, that says we can't remember sound quality factors, yet it's fine that we can remember words, notes, timbre, instrumentation, recording effects, whatever. We remember what we are interested in !

 

So the musician will recall note sequences, the songwriter recalls lyrics, the technician recalls test results and white papers, the business man recalls income & expense numbers, while the audiophile (listening, not armchair) recalls sound quality.

 

(edit) Oh, and BTW, The nerves from the cochlea project completely in number and morphology into the cortex, so all the original information is present there for processing and memory formation. Unlike what some CA wag postulated was a "compressed version, like MP3"

 

 

23 minutes ago, Jud said:

https://www.computeraudiophile.com/forums/topic/22709-heres-a-little-test-you-can-all-join-in/#comment-390666

 

Yes I recall that thread, but I don't recall a resolution ! And I didn't find one by going to the last page of the thread either, just promises. So the next time you want to link to that idea, perhaps you could link to where you summarize the results of your test, so there might be something worthwhile to discuss about it.

 

P.S  (to be fair, I'm not too good about finishing things either)

 

[mmerrill99]

1 hour ago, Daudio said:

I have said before that people are using this "pattern matching" term as a kind of catch-all buzzword, that obfuscates much more then illuminates.

Not really!  'pattern matching' is an essential part of auditory processing - we do it all the time - we are always analyzing sound over time & thus analyzing the pattern of the sound envelopes . For instance, at the finer temporal level, the development of the sound envelope over time identifies the difference between instruments - the attack portion being a very significant factor in this difference. Beat & rhythm identification are examples of pattern formation over longer time frames.

Even longer term matching of such well established patterns are what Jud & I are suggesting is the underlying mechanism playing an important role in how certain types of audible differences are perceived 

[mmerrill99]

1 hour ago, Daudio said:

perhaps you could link to where you summarize the results of your test, so there might be something worthwhile to discuss about it.

Agreed, I too would like a summary of the 'results' or discussion of the test.

[Daudio]

4 minutes ago, mmerrill99 said:

Not really!  'pattern matching' is an essential part of auditory processing - we do it all the time - we are always analyzing sound over time & thus analyzing the pattern of the sound envelopes .

 

I agree, but see the generic 'pattern matching' as a lower level function, unrelated to audio alone, distributed all over the cortex (and beyond). And I'm trying to prevent it from growing, here on CA, as a meaningless buzzword that doesn't have much of any explanatory power, thus I downplay the phrase until it can be seen in a better context.

 

[mmerrill99]

5 minutes ago, Daudio said:

 

I agree, but see the generic 'pattern matching' as a lower level function, unrelated to audio alone, distributed all over the cortex (and beyond). And I'm trying to prevent it from growing, here on CA, as a meaningless buzzword that doesn't have much of any explanatory power, thus I downplay the phrase until it can be seen in a better context.

 

Maybe delving into exactly what pattern matching means in an audio context is a better way to approach it as it is such a fundamental & core issue in auditory perception?

[Jud]

2 hours ago, Daudio said:

Yes I recall that thread, but I don't recall a resolution ! And I didn't find one by going to the last page of the thread either, just promises. So the next time you want to link to that idea, perhaps you could link to where you summarize the results of your test, so there might be something worthwhile to discuss about it.

 

P.S (to be fair, I'm not too good about finishing things either)

 

 

You're correct, so I've attached an Excel spreadsheet with a statistical analysis of the results.  (I sure as heck didn't do the analysis - statistics is one of the many subjects I regret not learning much of anything about in my life.)  First a very brief recapitulation of the test, then an equally brief summary of what little I gathered from the attached statistical analysis.

 

The test: I showed photos of two different acoustic guitars and asked people to tell me which was primarily in the left and which primarily in the right channel in a musical selection.  One selection was 30 seconds of a track where these two guitars were being played.  The other was a 2 second selection from those 30 seconds, then two seconds of silence, then the 2-second selection, then two seconds of silence, etc., for a total of 30 seconds.  (So: On (2 sec), Off (2 sec) On (2 sec), until 30 seconds has elapsed.)  This was intended to simulate the timbre information about the guitars that you might get in a rapid switching A/B test.  Anyone who said he was an objectivist got the 2-second sequence, and anyone who said he was a subjectivist got the 30-second passage.  This was so subjectivists wouldn't give up on the 2-second sequence without really trying and screw up the results.

 

I counted an answer as correct if someone could clearly describe to me the sonic differences between the two guitars (the Epiphone archtop with the f-holes sounded very nearly like an electric guitar with much less body resonance than the classic Gibson with the round sound hole), even if they got the channels wrong (i.e., if they had their "wires crossed").  One person did get the 30-second passage wrong.  He said using his iPhone and earbuds that both guitars sounded like they were somewhere in the middle.  I counted it as an incorrect answer (the only one) for the 30-second passage.

 

The statistics:  While 1 person of 40 got the 30-second passage wrong, 3 of 14 got the 2-second sequence wrong.  The  attached statistical analysis as I understand it says this equates to a non-chance probability of about 94% that people were better able to identify the separate timbres of the two guitars with the longer passage.

Survey results analysis for Jud.xlsx

[mmerrill99]

Thanks, Jud, maybe we should take discussion of those results back to the test thread although they are relevant here too?

[Jud]

6 minutes ago, mmerrill99 said:

Thanks, Jud, maybe we should take discussion of those results back to the test thread although they are relevant here too?

 

I don’t mind either or both, but whether we discuss it further in this thread is up to the OP.

[Daudio]

24 minutes ago, Jud said:

One selection was 30 seconds of a track where these two guitars were being played.

 

Thanks, but I don't follow exactly. If two instruments are playing, is that together, separate tracks, separate channels, or what ???

 

Were these 'objective' & 'subjective' subjects self classified ? Why the 40 vs 14 subject tests ?

 

 

28 minutes ago, Jud said:

The  attached statistical analysis as I understand it says this equates to a non-chance probability of about 94% that people were better able to identify the separate timbres of the two guitars with the longer passage.

 

That prompts me to ask why then are you still pushing the whole short echoic memory thing ?

 

And makes me think that your test could be extended to longer time periods 

 

 

[mmerrill99]

3 minutes ago, Jud said:

 

I don’t mind either or both, but whether we discuss it further in this thread is up to the OP.

Ok, we'll take his lead

[Daudio]

53 minutes ago, mmerrill99 said:

Maybe delving into exactly what pattern matching means in an audio context is a better way to approach it as it is such a fundamental & core issue in auditory perception?

 

Maybe...  so:

https://en.wikipedia.org/wiki/Category:Pattern_matching, and: https://en.wikipedia.org/wiki/Pattern_matching

 

Whoops, they’re only about machines, not brains - no help

But, by changing the phrase to “Pattern recognition”, and disambiguating, we find
https://en.wikipedia.org/wiki/Pattern_recognition_(psychology)

 

Still sounds awfully generic to me with no differentiation or detail by modality, so I still don’t see where you want to go with this.

 

Perhaps our different perspectives have to do with where we started out in this investigation. I was already very interested and knowledgeable about the brain and mind before I became an audiophile, then developed a more specific interest, and study, of the audio senses. That is my context: the brain first, then the auditory subsystems.

 

I don’t know how you came to this investigation, but I strongly suspect it was a different path

 

 

 

[Jud]

19 minutes ago, Daudio said:

 

Thanks, but I don't follow exactly. If two instruments are playing, is that together, separate tracks, separate channels, or what ???

 

Same song, two acoustic guitars playing at the same time, one primarily (close to exclusively) in the left channel, one primarily (close to exclusively) in the right channel.

 

Quote

 

Were these 'objective' & 'subjective' subjects self classified ? Why the 40 vs 14 subject tests ?

 

 

Self classified.  That's how many people who took the test called themselves objectivists (14) and subjectivists (40).

 

Quote

 

That prompts me to ask why then are you still pushing the whole short echoic memory thing ?

 

And makes me think that your test could be extended to longer time periods 

 

This doesn't test echoic memory.  The two guitars are playing at the same time.  The test is to distinguish them from each other by timbre.

 

For a test of echoic memory, see this: http://deutsch.ucsd.edu/psychology/pages.php?i=209

[Daudio]

9 minutes ago, Jud said:

Self classified.  That's how many people who took the test called themselves objectivists (14) and subjectivists (40).

 

I find that interesting in itself 

 

Got it, on the rest (except the last - agree to disagree, for now)

Is that the same link you posted earlier ?

Wish I had 'strike-through' available on this damn post editor !

 

Nope, on further thought... I think that what you are actually testing is how long it takes to make an accurate discrimination, where the timbres and channels were the vehicle for the test, not what got measured. That's my quick take...

[Jud]

16 minutes ago, Daudio said:

I think that what you are actually testing is how long it takes to make an accurate discrimination

 

Of at least this degree of timbre variation, yes.

 

If we were talking about a loudness difference of say 1dB, the time needed to make an accurate discrimination would I am guessing be less.

 

A smaller timbre difference might take longer to discriminate.  That's the point about A/B comparisons as they are usually done, in sequence: You are in a race between echoic memory and the time it takes to discriminate what might be fairly small sonic differences.  (Yes, it's the same link, but that link summarizes one of a very large number of experiments by various people over decades all coming to the conclusion that echoic memory lasts perhaps 4-10 seconds; and hearing more notes, as you do when you play two pieces of music in sequence, unavoidably degrades that memory.)