We show how particle-vortex duality implies the existence of a large non-abelian discrete symmetry group which relates the electromagnetic response for dual two-dimensional systems in a magnetic field. For conductors with charge carriers satisfying Fermi statistics (or those related to fermions by the action of the group), the resulting group is known to imply many, if not all, of the remarkable features of Quantum Hall systems. For conductors with boson charge carriers (modulo group transformations) a different group is predicted, implying equally striking implications for the conductivities of these systems, including a super-universality of the critical exponents for conductor/insulator and superconductor/insulator transitions in two dimensions and a hierarchical structure, analogous to that of the quantum Hall effect but different in its details. Our derivation shows how this symmetry emerges at low energies, depending only weakly on the details of dynamics of the underlying systems.