Making 24-bit samples have the same endianness as the CPU is pretty non-sensical. There is nothing to gain from doing so (as there is no native 24-bit integer format in the first place), and only serves to make your code more complex than if you had picked an arbitary endianness and stuck with it.