Bob Stern

A/B testing favors B over A

Rate this topic

157 posts in this topic

10 minutes ago, jabbr said:

See your thought experiment through and propose an actual study design.

Thought experiments don't need to be converted into actual experiments - they exist as pure logic.

 

If you can point out the flaw in logic of my thought experiment then that is a valid objection.

 

Stating that it needs to be an actual physical experiment misses the whole purpose & concept of thought experiments in the scientific discipline.

1

Share this post


Link to post
Share on other sites
27 minutes ago, mmerrill99 said:

The argument was & still is being made that listening order bias is eliminated by randomization & I'm using a thought experiment to show that it couldn't be. A perfectly valid & scientific approach even though some jump on a bandwagon claiming I don't know what I'm talking about - it would seem they belie their own lack of ability to think logically

 

You don't understand. I am not asking you for a thought experiment "proving" that listening order bias can't be eliminated by randomization.

 

I am asking you for a thought experiment which a) eliminates listening order bias and b) doesn't employ randomization?

 

Do you understand the difference?

 

A disproves B is not equivalent to

C proves D

 

0

Share this post


Link to post
Share on other sites
3 minutes ago, jabbr said:

 

You don't understand. I am not asking you for a thought experiment "proving" that listening order bias can't be eliminated by randomization.

 

I am asking you for a thought experiment which a) eliminates listening order bias and b) doesn't employ randomization?

 

Do you understand the difference?

Well you can't have another thought experiment - I only do them on Thursdays but I'll be here every week :)

 

Instead of looking for another logic experiment,  let me ask you about this thought experiment - do you agree - randomization does not eliminate listening order bias?

1

Share this post


Link to post
Share on other sites
7 minutes ago, jabbr said:

You don't understand. I am not asking you for a thought experiment "proving" that listening order bias can't be eliminated by randomization.

Your posts are confused & confusing

What I believe you mean to say is "I am not asking you for a ACTUAL experiment " as I already gave you the thought experiment " "proving" that listening order bias can't be eliminated by randomization."

1

Share this post


Link to post
Share on other sites
11 minutes ago, mmerrill99 said:

Instead of looking for another logic experiment,  let me ask you about this thought experiment - do you agree - randomization does not eliminate listening order bias?

 

a) that's not a thought experiment

b) "randomization does not eliminate listening order bias" is a non-sequitur. Randomization is a technique which is employed in studies in order to reduce bias.(its not the only thing that defines a study) A study designed to eliminate listening order bias would employ randomization and other techniques.

0

Share this post


Link to post
Share on other sites
2 minutes ago, jabbr said:

 

a) that's not a thought experiment

b) "randomization does not eliminate listening order bias" is a non-sequitur. Randomization is a technique which is employed in studies in order to reduce bias.(its not the only thing that defines a study) A study designed to eliminate listening order bias would employ randomization and other techniques.

And another failure to address the logic I gave which showed your statement was false - that randomization eliminates listening order bias.

 

I'm done with these incessant OT counterpoints.

1

Share this post


Link to post
Share on other sites
17 minutes ago, mmerrill99 said:
35 minutes ago, Ralf11 said:

Sad.  There are science courses in every high school, much less college.

 

Pity you never have anything of worth to contribute - why do you participate here?

 

I think the gadfly only has cut-and-paste skills to/from the first hit of a Google search, and a sad need for attention, or a love of littering. Lurking and learning is a much better strategy for him.

 

0

Share this post


Link to post
Share on other sites
8 minutes ago, jabbr said:

Randomization is a technique which is employed in studies in order to reduce bias.(its not the only thing that defines a study) A study designed to eliminate listening order bias would employ randomization and other techniques.

 

Doesn't it kind of depend on what you randomize ?

 

1

Share this post


Link to post
Share on other sites
11 hours ago, Daudio said:

And if one isn't in this hobby to chase better sound, then what the hell are they doing it for ? Waste money, exercise their oscilloscopes, or Online Armored Combat ?

 

 

I agree, it's all about enjoying the music, test equipment doesn't do it for me!O.o

 

10 hours ago, mansr said:

I'd rather spend $5k on an oscilloscope than a power cord. Others would, apparently, get the power cord. I have yet to meet anyone who'd get a power cord for the oscilloscope.

 

I vote neither!


$5k is twice the cost of my entire audio/video system including my computer, all my upgraded cabling and my HDTV. I prefer to spend my money on music.


Test equipment cannot play music, nor make my music sound better! Thus I see no need for expensive test equipment since I don't design my own equipment, I buy good equipment already assembled at the best price I can find, on sale, clearance, demo or used. And I usually wait for it to die or be too expensive to repair before I replace it.

 

If I were an audio designer I could justify the cost of expensive test equipment to verify my designs meet specifications, but I am not an audio designer.

 

0

Share this post


Link to post
Share on other sites
24 minutes ago, jabbr said:

Randomization is a technique which entirely depends on what is being randomized.

 

Ok, so if we randomize the aforementioned bias, so that sometimes the test is A->B, and other times B->A, we haven't eliminated the bias, it is still there, but it's effect is masked by the randomization, and the bias itself masks differences smaller then the bias effect.

 

Seems to me a comedy of errors, best left in the dust bin  :S

 

0

Share this post


Link to post
Share on other sites
15 minutes ago, Daudio said:

Seems to me a comedy of errors, best left in the dust bin  :S

 

No really not. Bias isn't just "error", it is a systemic error as if the baseline is adjusted +x. Randomization eliminates this by statistically averaging +x and -x such that it approaches zero. Not for each sample : systemically.

0

Share this post


Link to post
Share on other sites
1 minute ago, jabbr said:

Bias isn't just "error", it is a systemic error as if the baseline is adjusted +x. Randomization eliminates this by statistically averaging +x and -x such that it approaches zero

 

Sorry, but I'm an empirisist, not a theorist, and the above just doesn't resonate with me, while I think what I described was pretty clear.  Guess we are at an 'agree to disagree' point.

 

Have a fine weekend :)

1

Share this post


Link to post
Share on other sites
19 minutes ago, Daudio said:

we haven't eliminated the bias, it is still there, but it's effect is masked by the randomization, and the bias itself masks differences smaller then the bias effect.

Actually you've eliminated the bias. If and only if the individual sample effect of the bias masks other differences then the "internal validity" of the experiment is not sufficient to detect these small differences -- you then need to use other techniques to improve the resolution of your experimental design. But masking isn't something which is automatic. It means that the variables aren't independent. Or you might need to use more individual subjects.

0

Share this post


Link to post
Share on other sites
3 minutes ago, The Computer Audiophile said:

you can always use the "ignore user" feature of the site if someone gets on your nerves or makes your time here unenjoyable.

 

I make extensive use of it, to make the best of... this place.

 

But occasionaly one will see their words in quotes, then ??

 

0

Share this post


Link to post
Share on other sites

Thank you, Teresa - it's good to see my thoughts/posts actually make sense & people 'get it' - I don't find it a difficult concept to understand & I'm glad to see others also can understand the point.

 

Sometimes there are so many red herrings thrown into a thread that the smell of fish dominates rather than the real, valid points being posted.

 

A lot of wasted time is spent trying to clear out these red herrings

1

Share this post


Link to post
Share on other sites
15 minutes ago, Teresa said:

Let's assume that Amp A sounds better than Amp B when actually playing music in a relaxed manner. So, there is an equal number of A’s and B’s randomly selected and the last sample played is always selected as best despite them sounding different, then one would only get a null result no matter how large the sample size is. When Amp A is played last people chose it, when Amp B is played last people chose it. Thus it doesn't matter how random the switching occurs if there is an equal number of A's and B's being last, one would get a null (50%) result.

 

Right Teresa. So this wouldn't be the best way to do the study. This is what we call a "Type II" error.

 

Suppose each track is played 3 times and each time randomized to A/B options would be:

 

A-A-A

A-A-B

A-B-A

A-B-B

...

B-B-B

 

and instead of picking "best" between A and B, perhaps just rank the listening experience from 1 (bad) - 10 (best possible)

 

and then average the scores for A vs B for each time and for all together -- perhaps just use the middle repeat.

If this isn't good enough you could do, say 6 repeats and multiple listening sessions and with enough samples the statistics would show the difference.

 

Now this isn't so simple and I hope my example isn't too hard to understand but that is an example of how to reduce listening order bias.

0

Share this post


Link to post
Share on other sites
8 minutes ago, jabbr said:

 

Right Teresa. So this wouldn't be the best way to do the study. This is what we call a "Type II" error.

 

Suppose each track is played 3 times and each time randomized to A/B options would be:

 

A-A-A

A-A-B

A-B-A

A-B-B

...

B-B-B

 

and instead of picking "best" between A and B, perhaps just rank the listening experience from 1 (bad) - 10 (best possible)

 

and then average the scores for A vs B for each time and for all together -- perhaps just use the middle repeat.

If this isn't good enough you could do, say 6 repeats and multiple listening sessions and with enough samples the statistics would show the difference.

The thread is about A/B blind testing in which two tracks are played A & then B or B & the A - a preference is made between the two tracks heard - what is your list about?

0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now