Network attacks are occurring continuously day after day. The researchers are expected to find the solution by identifying the address of source. We propose the IP traceback ant colony system (ITACS) algorithm to solve the IP traceback of denial of service (DoS) problem. The ITACS is novel attempted to apply in solving the problem. It is a meta-heuristic algorithm, which is a technique applies so that attack detection and attack identification can be implemented. The proposed algorithm has improved by the previous one to conquer this problem successfully. We obtained the data set of topology from one of famous research organizations for the experiment. The parameters of algorithm are considered by packet contents in topology. In the meanwhile, we discussed the increment of traffic condition. By the experiment, the examples of increment of traffic are above average 70%. The results show that the performance of ITACS algorithm is efficient and accurate. Furthermore, the proposed algorithm has also nature of robust for the problem. Future work may even be extended to study the other behaviors of organisms from derivations of meta-heuristic algorithm.