Graphene and its nanohybrid photodetectors (PDs) offer unique interfacial and synergistic properties in contrast to conventional semiconductor-based PDs. In this work, we compare the performance of identically fabricated PDs based on pure chemical vapor deposition-grown monolayer graphene, demonstrating a positive photoresponse, that exhibits a gain in photoresponsivity (R) with the addition of two-dimensional MoS2 (chemically exfoliated). However, further addition of plasmonic gold nanorods (AuNRs), on the MoS2/graphene PD, results in the flipping of the photoconductivity to negative. The negative photoresponse is explained by an increased dark current via the complex conductive network model and a lower photocurrent due to the photogenerated electron capture by the AuNRs in the AuNRs/MoS2/graphene PD. The contrasting shape and nature of the dynamic photoresponses are modeled using theoretical fitting and explained based on carrier transport in individual components. The highest R in the moderately fast graphene PD device is ∼2 × 104 A W-1 under 808 nm which could be slightly improved to ∼3 × 104 A W-1 by introducing MoS2 nanoflakes, albeit at the cost of the response time which slows down from ∼90 ms (in the former) to 558 ms in the latter. The introduction of AuNRs did speed up the response time in the AuNRs/MoS2/graphene PD to 20.5 ms but produced a strikingly different negative photoresponse attributed to a higher dark current in the material. We proposed a single mechanism to explain the positive and negative photoresponses reported so far for the hybrid PDs.