This work presents a novel neural network-based approach to detect structural damage. The proposed approach comprises two steps. The first step, system identification, involves using neural system identification networks (NSINs) to identify the undamaged and damaged states of a structural system. The partial derivatives of the outputs with respect to the inputs of the NSIN, which identifies the system in a certain undamaged or damaged state, have a negligible variation with different system errors. This loosely defined unique property enables these partial derivatives to quantitatively indicate system damage from the model parameters. The second step, structural damage detection, involves using the neural damage detection network (NDDN) to detect the location and extent of the structural damage. The input to the NDDN is taken as the aforementioned partial derivatives of NSIN, and the output of the NDDN identifies the damage level for each member in the structure. Moreover, SDOF and MDOF examples are presented to demonstrate the feasibility of using the proposed method for damage detection of linear structures.