A/B testing favors B over A

[andifor]

6 hours ago, pkane2001 said:

 

There's a huge difference between a sighted, long term A/B comparison with one or two attempts to switch components and a blind A/B test, repeated sufficient number of times to achieve statistical significance. While both can contain biases and other flaws, the blind test controls for many more variables and is much more objective and reproducible by others.

 

Last time I managed a small A/B blind test, I told the audience I'll switch speakers but didn't. They found out I was cheating within seconds. I guess, the sound of the loudspeakers was to different, but at least I was sure, they really listened.

[Bob Stern]

3 hours ago, Jud said:

I don’t mind either or both, but whether we discuss it further in this thread is up to the OP.

 

I hereby relinquish my prerogatives and status as OP.

[Jud]

32 minutes ago, Bob Stern said:

 

I hereby relinquish my prerogatives and status as OP.

 

OK, well I don’t care either, so whatever anyone else prefers.

[mmerrill99]

4 hours ago, Daudio said:

I don’t know how you came to this investigation, but I strongly suspect it was a different path

Research directly into the question of why we hear what we hear - some relevant paper on auditory pattern analysis
"Modelling the emergence and dynamics of perceptual organisation in auditory streaming"

[mmerrill99]

42 minutes ago, Jud said:

 

OK, well I don’t care either, so whatever anyone else prefers.

Seems like continuing here makes sense?

[mmerrill99]

A video worth watching which again addresses auditory pattern recognition

 

[Jud]

11 minutes ago, mmerrill99 said:

Seems like continuing here makes sense?

 

Sure.

[jabbr]

9 hours ago, mmerrill99 said:

Yes & I was hopefully careful to just point out that randomization was not the answer to the listening order bias - it seemed to be suggested by you & others that this was the case until I pointed out the flaw in your logic.

 

Flaw in logic? If you think I suggested that randomization was the only thing important in designing a valid study, I never said that nor would I ever. I could easily design a study aimed at reducing listening order bias (this is similar to other ordering biases and the study techniques I would use are widely employed), but since you are questioning logic, let's turn this around: why don't you tell us how you would eliminate listening order bias without using randomization in a way that is necessarily better than one using randomization (or pseudorandomization).

[mmerrill99]

2 minutes ago, jabbr said:

 

Flaw in logic? If you think I suggested that randomization was the only thing important in designing a valid study, I never said that nor would I ever. I could easily design a study aimed at reducing listening order bias (this is similar to other ordering biases and the study techniques I would use are widely employed), but since you are questioning logic, let's turn this around: why don't you tell us how you would eliminate listening order bias without using randomization in a way that is necessarily better than one using randomization (or pseudorandomization).

We've already had & settled that discussion - randomization only hides the listening order bias, doesn't eliminate it - actually it makes things worse by the fact that it is no longer an obvious bias. Why the need to revisit this?

[jabbr]

4 minutes ago, mmerrill99 said:

We've already had & settled that discussion - randomization only hides the listening order bias, doesn't eliminate it - actually it makes things worse by the fact that it is no longer an obvious bias. Why the need to revisit this?

I'm not buying this assertion. Prove it. Randomization doesn't hide listening order bias. Where does this concept come from? Are you pulling this out of your hat? Reference? As I asked, show me a study design that eliminates listening order bias without using randomization -- and then I'll show you one that eliminates listening order bias using randomization ... go for it.

[mmerrill99]

4 minutes ago, jabbr said:

I'm not buying this assertion. Prove it. Randomization doesn't hide listening order bias. Where does this concept come from?

 

So you want to go back 22 hours to reenact the whole debate we had?

I'm not that interested in this reenactment.

[jabbr]

6 minutes ago, mmerrill99 said:

 

So you want to go back 22 hours to reenact the whole debate we had?

I'm not that interested in this reenactment.

No, I want you to pesent a study design that eliminates listening order bias that doesn't employ randomization. Fine to be a critic, harder to design a study. I've got one btw (that uses randomization). So if what you say is true, cough it up.

[mansr]

3 minutes ago, jabbr said:

No, I want you to pesent a study design that eliminates listening order bias that doesn't employ randomization.

Why not use randomisation? It's an effective tool to remove that kind of bias. In some ways it is analogous to how dither turns distortion into random noise.

[jabbr]

4 minutes ago, mansr said:

Why not use randomisation? It's an effective tool to remove that kind of bias. In some ways it is analogous to how dither turns distortion into random noise.

@mmerrill99 is asserting that randomization is worse, and that my logic is flawed -- if so, a study design which doesn't employ randomization awaits us (and then I'll post my relatively simple design which does use randomization) ... the analogy to dither is more than an analogy -- pseudorandomization is fine)

[Ralf11]

It's pretty clear that you are arguing with someone who doesn't understand things.

 

(or maybe thousands of published studies are wrong and will have to be done over)

[mmerrill99]

19 minutes ago, jabbr said:

No, I want you to pesent a study design that eliminates listening order bias that doesn't employ randomization. Fine to be a critic, harder to design a study. I've got one btw (that uses randomization). So if what you say is true, cough it up.

We don't need a study when a thought experiment will do - I already gave this.

In an A/B test, if instead of listening order bias we use volume bias. Let's say that every second sample is played at a higher SPL, just enough to bias the listener's preference but not to be obviously louder.

 

How does randomization remove the effect of this bias? It will make the comparison of two samples for subtle differences, impossible & as a side note it will hide the bias i.e the results won't obviously show that every second sample is preferred

[Ralf11]

I think we all agree that there are non-random ways to deliberately overwhelm the effects of a sampling design that is random with respect to other variables.  But, so what?

[mmerrill99]

3 minutes ago, Ralf11 said:

I think we all agree that there are non-random ways to deliberately overwhelm the effects of a sampling design that is random with respect to other variables.  But, so what?

You might need to read the discussion from the start instead of trying to score guerilla sniping points

The discussion is about listening order bias - have a read & come back with something cogent & related to the topic

[jabbr]

8 minutes ago, mmerrill99 said:

We don't need a study when a thought experiment will do - I already gave this.

In an A/B test, if instead of listening order bias we use volume bias. Let's say that every second sample is played at a higher SPL, just enough to bias the listener's preference but not to be obviously louder.

It would be absurd to deliberately introduce another bias to counteract an entirely different bias. SPL bias is entirely different than sample order bias. 

 

At this point we are discussing basic study design. There are known techniques.

 

But let's get this straight. You are proposing always to play Amp A first and B second, but play B slightly louder? (no randomization)

[Daudio]

2 minutes ago, jabbr said:

It would be absurd to deliberately introduce another bias to counteract an entirely different bias. SPL bias is entirely different than sample order bias. 

 

He's not talking about trying to counteract another bias, but posing an example to show how this different embedded bias would not be removed by randomization.

 

[mmerrill99]

2 minutes ago, jabbr said:

It would be absurd to deliberately introduce another bias to counteract an entirely different bias. SPL bias is entirely different than sample order bias. 

Are you purposefully trying to misunderstand the analogy I made?

 

Instead of listening order bias I substituted an SPL bias which always occurs on the second sample listened to - it doesn't really matter what bias we use here to illustrate that randomization is not the answer to reducing to eliminating a bias which occurs as a result of the order of listening.

 

I'm not going to explain this over & over - people are either going to think about what I say & see the logic in it or they are going to argue OT points like the above.

 

I'm not interested in wasting my time on this sort of tire kicking - either buy the car or move along

[mmerrill99]

4 minutes ago, Daudio said:

 

He's not talking about trying to counteract another bias, but posing an example to show how this different embedded bias would not be removed by randomization.

 

Exactly!

The argument was & still is being made that listening order bias is eliminated by randomization & I'm using a thought experiment to show that it couldn't be. A perfectly valid & scientific approach even though some jump on a bandwagon claiming I don't know what I'm talking about - it would seem they belie their own lack of ability to think logically

[jabbr]

2 minutes ago, Daudio said:

 

He's not talking about trying to counteract another bias, but posing an example to show how this different embedded bias would not be removed by randomization.

 

I'm asking for a simple study design that eliminates order bias without using randomization. For SPL bias I would equalize the levels, who would use randomization for that?

[jabbr]

1 minute ago, mmerrill99 said:

Exactly!

The argument was & still is made that listening order bias is eliminated by randomization & I'm using a thought experiment to show that it couldn't be. A perfectly valid & scientific approach

See your thought experiment through and propose an actual study design.

[Daudio]

1 hour ago, mmerrill99 said:

Research directly into the question of why we hear what we hear - some relevant paper on auditory pattern analysis
"Modelling the emergence and dynamics of perceptual organisation in auditory streaming"

 

Thanks for the link and video ! Still processing...