Frustration, or the competition between interacting components of a network, is often responsible for the complexity of many body systems, from social and neural networks to protein folding and magnetism. In quantum magnetic systems, frustration arises naturally from competing spin-spin interactions given by the geometry of the spin lattice or by the presence of long-range antiferromagnetic couplings. Frustrated magnetism is a hallmark of poorly understood systems such as quantum spin liquids, spin glasses and spin ices, whose ground states are massively degenerate and can carry high degrees of quantum entanglement. The controlled study of frustrated magnetism in materials is hampered by short dynamical time scales and the presence of impurities, while numerical modeling is generally intractable when dealing with dynamics beyond N~30 particles. Alternatively, a quantum simulator can be exploited to directly engineer prescribed frustrated interactions between controlled quantum systems, and several small-scale experiments have moved in this direction. In this article, we perform a quantum simulation of a long-range antiferromagnetic quantum Ising model with a transverse field, on a crystal of up to N = 16 trapped Yb+ atoms. We directly control the amount of frustration by continuously tuning the range of interaction and directly measure spin correlation functions and their dynamics through spatially-resolved spin detection. We find a pronounced dependence of the magnetic order on the amount of frustration, and extract signatures of quantum coherence in the resulting phases.