During the violent relaxation of a self-gravitating system a significant fraction of its mass may be ejected. If the time varying gravitational field also breaks spherical symmetry this mass can potentially carry angular momentum. Thus starting initial configurations with zero angular momentum can in principle lead to a bound virialized system with non-zero angular momentum. We explore here, using numerical simulations, how much angular momentum can be generated in a virialized structure in this way, starting from configurations of cold particles which are very close to spherically symmetric. For initial configurations in which spherical symmetry is broken only by the Poissonian fluctuations associated with the finite particle number N, with N in range 1000 to 100000, we find that the relaxed structures have standard “spin” parameters λ∼0,001 and decreasing slowly with N. For slightly ellipsoidal initial conditions, in which the finite-N fluctuations break the residual reflection symmetries, we observe values λ∼0,01 i.e. of the same order of magnitude as those reported for elliptical galaxies and dark matter halos in cosmological simulations. The net angular momentum vector is typically aligned close to normal to the major semi-axis of the triaxial relaxed structure, and also with that of the ejected mass. This simple mechanism may provide an alternative, or complement, to “tidal torque theory” for understanding the origin of angular momentum in astrophysical structures.