A/B testing favors B over A

[Teresa]

On 6/17/2017 at 3:55 AM, pkane2001 said:

You are really determined to make a blind A/B test into a bad thing.

 

In audio when was it ever a good thing?

With a photograph you can see two pictures at the same time side by side.
With video you can see two images at the same time side by side with two monitors.
With audio you cannot hear two samples at the same time, instead you must switch between them.

 

Only very large audio differences are heard AB’ing. When I do hear a difference A/B’ing I always pick the worst sounding one as the best. I believe I know why, even with levels matched there can be differences in levels of different frequency ranges, especially the bass and mid-bass. And an item with a restricted dynamic range compared to a wide dynamic range will sound louder during much of the music. In A/B’ing humans select whatever is loudest as the best thus the item with a bass hump or compressed dynamics will be wrongfully chosen as best.

 

Ears can be tricked and the number one way to do that is A/B’ing. That is why I spend a lot of time auditioning and only purchase with a 30-day money-back guarantee. With long-term listening using my favorite music I can then determine if the item under audition makes my music sound and more importantly feel closer to real live acoustic music I hear in a good concert hall or club, or moves it further away, or offers no change at all.

 

I use to let salespeople A/B me, not anymore I now do single item demos and compare how close it gets me to the feeling of real music, not one item against another. At a store or audio show I bring my own SACDs for demo, I select something I might like. My number priority is how the item looks and will it match well with my decor. Second is friendly ergonomics, will it be easy to use. This usually eliminates the majority of items an audio store has for sale. I then pick out what I want to hear and ask the salesperson to turn out the lights in the sound room, if not possible, to turn them to the lowest setting, and to leave the room for about 10 minutes while I listen with the music I brought. If they refuse to do this, I leave.

 

I'm fine with blind listening as long as it is under normal long-term listening to music for pleasure. I'm totally against AB'ing, sighted or blind as it fails to reveal important sonic differences IMHO.

 

Why sighted or blind A/B testing fails to reveal statistical differences between nearly everything:

• Cognitive bias - your brain will fill in missing information thus making both samples sound the same on repeated listening.
• Listener Fatigue - switch back and forth too often and both music files will sound like crap.
• Accumulative effects are hidden - Accumulative effects on sound quality increase over time and remain hidden when switching back and forth between two music files, especially things such as strident/smooth, cold/warm sound, etc.
• Soundstage and instrument placement - it takes anywhere between 30 seconds to several minutes for my brain to map the soundstage and hear the instrument and vocal placement before I can judge anything. A/B'ing insures this never happens.
• Confirmation Bias - In addition sighted A/B testing has to fight confirmation bias, as some people think the major brand or more expensive item must sound better. This is not always true as sometimes the unknown brand or the least expensive item sounds the best.
• A/B testing favors B over A

On 6/17/2017 at 3:55 AM, pkane2001 said:

Cognitive bias is exactly what you get in a long term listening test. In such a test you are not evaluating the equipment, you are evaluating the ability of your brain to adjust to your equipment. If you conduct it sighted, then you are also adding in the much more powerful selection bias into your test.

 

While I may be adjusting to whatever I’m trying out, I prefer to say I am learning its strengths and weaknesses as I play a wide variety of music I love. Thus I can actually hear what it does well and what it does poorly with long-term listening, something not possible with A/B’ing. A long-term listening audition can go one of three ways, it makes my music sound more real, it does nothing positive or it makes my music less enjoyable. Sadly, the third option is what happens to me the most as I have returned more audio products than I have kept. As you can see nothing whatsoever to do with cognitive bias. Cognitive bias is your brain doing pattern matching and that is what makes everything sound the same when A/B'ing just after a couple of switches back and forth.

 

On 6/17/2017 at 3:55 AM, pkane2001 said:

Listener fatigue is something that might occur after a prolonged exposure to some sound. Show me a study that demonstrates listener fatigue after a total of a few minutes of repeated listening to the same track in an A/B test.

 

I don't read studies, I am more interested in music. I can only relate my nearly 50 years as a music loving audiophile.

[pkane2001]

1 hour ago, Teresa said:

I don't read studies, I am more interested in music. I can only relate my nearly 50 years as a music loving audiophile.

 

Teresa, I don't doubt you hear differences when you do sighted, long-term or even short-term evaluation. This doesn't mean the differences are there.  I've been listening to music for nearly as long as you, but I've also been an engineer and a scientist. In science, an objective approach that eliminates real biases or other errors is critical to any experiment. An experiment that does not account for these will produce wildly varying results that cannot be substantiated by others and will prove nothing.

 

On the other hand, just stating that something is a bias in an experiment without having (yes, an objective!) evidence to back it up is also insufficient and is not an acceptable scientific practice.

 

We are arguing on two different levels: you are coming from your own personal experience and I'm arguing from a scientific, objective testing perspective. We are basing our arguments on incompatible principles of how to study the physical world. With that, I suggest that we agree to disagree, as I don't think either one of us will convince the other.

[AJ Soundfield]

4 hours ago, Teresa said:

In audio when was it ever a good thing?

Since around the 1920's-30's, Bell Labs, etc.

About 30 year after audiophiles though horses could count.

The "Fletcher" in Fletcher-Munson, etc, used to develop (real) audio to this day.

 

Quote

I don't read studies

Yes, that shows in the level of misinformation filling your post.

But even Wikipedia could help. YMMV.

 

 

[AJ Soundfield]

Btw, is everyone clear this was not a "test" of the amplifiers at all, but of people?

[audiventory]

To serious considering of results of either A/B or ABX or any same test need detailed protocols of the measurements:

1. Methodics;

2. List of participants and their skill level;

3. Schemes of measurements and listening room(s);

4. List of used equipment;

5. Thousands of measurements;

6. Etc.

 

Otherwise, we should remember that other conditions or participants or numbers can give other results.

[Ralf11]

Don't conflate A/B testing with short-term testing

[jabbr]

2 hours ago, Ralf11 said:

Don't conflate A/B testing with short-term testing

Yeah. Unfortunately A/B testing might mean, to many people, what salespeople do on listening rooms. There seems, unfortunately, a preconception that there is only one way to compare two things.

[WMW]

Sorry for my absence ("Life is what happens to you when you're busy planning for something else.").

 

Teresa:  Many drug trials concern psychotropic medicines.  A patient taking an antidepressant (or placebo) must decide whether he or she "feels better".  Certainly this at least approaches (far surpasses IMO) the difficulty in choosing whether an audio component "sounds better".

 

Can we agree that a transistor radio will never be mistaken for a quality mono system playing the identical, familiar, volume-compensated audio example?  IMO this is fact.  Given this, aren't we really debating subtle differences between audio products, nuances that some hear and some do not?  If folks aren't willing to accept/admit that unblinding sonic sources (or medicinal identities) doesn't confound any/every audio 'preference' (or medicinal efficacy), I conclude the emperor is, indeed, unclothed.

 

No offense meant to anyone as there are some extremely knowledgeable and forthright individuals on both sides of this issue.  I don't and never have regarded myself as an audiophile.   I think I know a good sound system when I hear one.  I admit I cannot rank quality systems until I continuously listen to each for at least 90 minutes.  If my ears do not fatigue, THAT is the 'best' system IMO.  Perhaps there is something of that in this debate.

 

Ultimately, I offer the following observation:  Shopping for  quality TV's (they're all pretty damn good actually) in an appliance store, it's easy to compare picture differences and besides price and size this becomes the focus in product selection.  At home that new TV's resolution, brightness, color accuracy, contrast, etc., matters FAR LESS than what that TV is playing.  Put another way, a great song on a good system trumps a great system playing a good song.

 

Enjoy the MUSIC!!

 

Bill Walker

 

PS  Events beyond my control may limit my ability to participate in the near future.

 

[fas42]

13 minutes ago, WMW said:

Ultimately, I offer the following observation:  Shopping for  quality TV's (they're all pretty damn good actually) in an appliance store, it's easy to compare picture differences and besides price and size this becomes the focus in product selection.  At home that new TV's resolution, brightness, color accuracy, contrast, etc., matters FAR LESS than what that TV is playing.  Put another way, a great song on a good system trumps a great system playing a good song.

 

Ummm - what I feel you've left out there is that at home one can tweak the settings of all the TV's controls to optimise the picture quality, for the particular set. And, that's exactly what I have done with our now somewhat outdated LCD flat screen - years ago, I spent days fine tuning all the colour controls, those magic behind the scenes adjusters, until it gave the best possible results. And it has always been worth it - even the most tedious rubbish looks good! The visual "punch" occurs not because the colours have been hyped, but because they always feel right, on every broadcast - hyped colour that the channels add for show previews is obviously such, and nature docos are fabulous.

 

Which means I have zero new set envy - I go into a store now, and the huge size, and showroom demo mode don't do a thing for me - they look 'wrong', and would be irritating to view, as is.

 

And that's exactly how I approach sound reproduction - get it 'right', and everything else falls into place ...

[WMW]

Dear fas42,

 

Perhaps I should clarify:  When shopping/comparing TV's (or audio components/systems) the focus is on evaluating DIFFERENCE(S) which are frequently subtle. Ultimately, the enjoyment of the TV (or sound system) is predicated on WHAT the TV (or sound system) is playing.  I still have analog TV's (long story).  When viewing excellent, involving programming (e.g. The Godfather), I'm lost in the programming and never, ever think "This really needs more resolution!".  As I scribe this seems a ridiculous, obvious point.  Doesn't 'audiophilia' take quality content as a given and seek to discover equipment/technology which most faithfully recreates this content? (The question is rhetorical).  IMO the technology/equipment is and has been pretty darn good for some time.  Great audio or video enjoyment can be had without breaking the bank to be sure.  And to your point fine tuning components to your environment can enhance the enjoyment which will ultimately be maximal once differentiation is forgotten and content is the focus.

 

IMO true A/B testing in audio is possible, but very difficult to achieve (read "Costly!").  Perhaps there isn't enough profit to spark rigorous experimental conditions.  But "Take it home and try it!" testing reminds me of the "Series of one." in clinical medicine which means a clinician had a single experience with a patient, drug  or clinical event and this forever influences practice.  

 

In my experience when reasonable, intelligent people line up on opposite sides of a question it's because both sides are partially correct , the question is improperly phrased/irrelevant or all the above.

 

You say "colour"; I say "color".  I'm not arguing with you at all as I agree with your post as it stands, just not as it relates to my previous post.

 

Cheers and thanks!

 

Bill Walker