The evolution of self-gravitating systems, and long-range interacting systems more generally, from initial configurations far from dynamical equilibrium is often described as a simple two phase process: a first phase of violent relaxation bringing it to a quasi-stationary state in a few dynamical times, followed by a slow adiabatic evolution driven by collisional processes. In this context the complex spatial structure evident, e.g., in spiral galaxies is understood either in terms of instabilities of quasi-stationary states, or a result of dissipative non-gravitational interactions. We illustrate here, using numerical simulations, that purely self-gravitating systems evolving from quite simple initial configurations can in fact give rise easily to structures of this kind of which the lifetime can be large compared to the dynamical characteristic time, but short compared to the collisional relaxation time scale. More specifically, for a broad range of non-spherical and non-uniform rotating initial conditions, gravitational relaxation gives rise quite generically to long-lived non-stationary structures of a rich variety, characterized by spiral-like arms, bars and even ring-like structures in special cases. These structures are a feature of the intrinsically out-of-equilibrium nature of the system’s collapse, associated with a part of the system’s mass while the bulk is well virialized. They are characterized by predominantly radial motions in their outermost parts, but also incorporate an extended flattened region which rotates coherently about a well virialized core of triaxial shape with an approximately isotropic velocity dispersion. We discuss the possible relevance of these simple toy models to the observed structure of real galaxies emphasizing the difference between dissipative and dissipationless disc formation.