If you're using itunes then that only has two channel output, so even using an Aggregate Device you just have two channels to be sent out to the four or more outputs in your Aggregate Device, which doesn't work.
I've been struggling with the same problem - HDMI to my HDTV and also to my Weiss DAC2. In the end I downloaded JackOSX, which is a software audio router.
To set it up you need to make JackOSX the output device, give it two inputs and four outputs, then once JackOSX and itunes are running open the JackOSX routing window and direct channel one input to channels one and three output of the Aggregate Device and channel two input to channels two and four output of the Aggregate Device and it should produce correct audio on all four hardware output channels.
I ran the Weiss Bit Transparency Check after setting this up and the output to the Weiss is still bit accurate through JackOSX.