We can make use of if constexpr in several scenarios here to allow
compilers to exise the relevant code paths out.
Technically a decent compiler would do this already, but now we can give
compilers a little more nudging here in the event that isn't the case.