An axial MRI image of the lumbar spine generally contains multiple spinal structures and their simultaneous segmentation will help analyze the pathogenesis of the spinal disease, generate the spinal medical report, and make a clinical surgery plan for the treatment of the spinal disease. However, it is still a challenging issue that multiple spinal structures are segmented simultaneously and accurately because of the large diversities of the same spinal structure in intensity, resolution, position, shape, and size, the implicit borders between different structures, and the overfitting problem caused by the insufficient training data. In this paper, we propose a novel network framework ResAttenGAN to address these challenges and achieve the simultaneous and accurate segmentation of disc, neural foramina, thecal sac, and posterior arch. ResAttenGAN comprises three modules, i.e. full feature fusion (FFF) module, residual refinement attention (RRA) module, and adversarial learning (AL) module. The FFF module captures multi-scale feature information and fully fuse the features at all hierarchies for generating the discriminative feature representation. The RRA module is made up of a local position attention block and a residual border refinement block to accurately locate the implicit borders and refine their pixel-wise classification. The AL module smooths and strengthens the higher-order spatial consistency to solve the overfitting problem. Experimental results show that the three integrated modules in ResAttenGAN have advantages in tackling the above challenges and ResAttenGAN outperforms the existing segmentation methods under evaluation metrics.