Reinforcement Learning is a reasonable strategy to address object manipulation with long-horizon operation tasks. However, due to limitations in network capac- ity and exploration efficiency, it is challenging to achieve high success rates in dealing with such complex tasks. In this paper, we propose a Multiple Goal- conditioned Hierarchical Learning (MGHL) framework to tackle long-horizon task manipulated by redundant robotic arms. A long-horizon operation task is viewed as a sequential composition of multiple subtasks. By combining Multi- Task Learning and Hierarchical Reinforcement Learning, MGHL simultaneously learns multiple subtasks with sparse rewards under the same policy, and gradu- ally completes the goals of each subtask. During the pre-training of subtasks, a Mid-goal Guided Exploration Strategy is proposed to improve the sample qual- ity by establishing mid-goals to guide the exploration of the policies. In addition, MGHL implements a joint impedance controller that enables the robotic arm to take advantage of the flexibility of each joint to handle complex operations in a limited action space of the end effector. Three types of long-horizon manipulation tasks are introduced for training subtasks. The experimental results show that our proposed MGHL framework can learn multiple subtasks in the same policy with high success rate. By completing pre-trained subtasks one by one, MGHL outperforms several state-of-the-art algorithms on long-horizon tasks.