Initially cold and spherically symmetric self-gravitating systems may give rise to a virial equilibrium state which is far from spherically symmetric, and typically triaxial. We focus here on how the degree of symmetry breaking in the final state depends on the initial density profile. We note that the most asymmetric structures result when, during the collapse phase, there is a strong injection of energy preferentially into the particles which are localized initially in the outer shells. These particles are still collapsing when the others, initially located in the inner part, are already re-expanding; the motion of particles in a time varying potential allow them to gain kinetic energy — in some cases enough to be ejected from the system. We show that this mechanism of energy gain amplifies the initial small deviations from perfect spherical symmetry due to finite N fluctuations. This amplification is more efficient when the initial density profile depends on radius, because particles have a greater spread of fall times compared to a uniform density profile, for which very close to symmetric final states are obtained}. These effects lead to a distinctive correlation of the orientation of the final structure with the distribution of ejected mass, and also with the initial (very small) angular fluctuations.