Many papers claim exotic algebras (Quaternion, etc.) are parameter-efficient. However, they often fail to compare against real-valued networks with equivalent structural constraints. When properly matched controls are introduced, the supposed advantage from the exotic algebra evaporates, revealing the benefit was simply from the imposed structure itself, not the algebra.
A quaternion network showed superior extrapolation, suggesting unique representational power. However, analysis revealed the equivalent real-valued network could find the solution; the quaternion parameterization just created an "optimization basin" that SGD found more reliably. The advantage was in optimization ease, not fundamental capability.
After spending two months and 35 experiments disproving a popular research direction, the authors published their negative results to save others from the same fate. They advocate for simple, rigorous controls that would have prevented the false positives they chased, highlighting the immense value of transparently sharing what doesn't work.
Even for tasks designed to require specific algebraic properties like non-associativity, a general-purpose real-valued network performs just as well. This is because any fixed algebra's product is just a fixed real bilinear tensor, which a universal approximator can learn and absorb, negating the need for a specialized, hard-coded architecture.