In this case, I decided the program would assume the input samples have the same endianness as the CPU. Thus in order to extend 24-bit integers to 32-bit, it needs to know which endianness that is.

Ultimately I might make the data endianness an option and do like you say. Thanks for the comment.