This paper presents a novel lane detection scheme for detecting and tracking various lanes directly from videos in real time. The key contribution of this paper is to propose a new edge labeling scheme for labeling each edge point to different types. Even under different lighting and weather conditions, different lane segments still can be effectively extracted from the labeling result. Only the subtraction and counting operators are involved in the labeling process. It is also flexible for detecting lanes with different types especially from an interchange area. To filter out false lane segments, this paper uses a pinhole camera model to derive a geometrical constraint for lane verification. The constraint is invariant to shadows and lighting changes. Thus, each lane line can be verified and then detected more accurately from roads. Since lanes seldom change their colors during two adjacent frames, we propose a kernel-based technique for tracking them even fragmented into pieces of segments. Then, different lanes can be more efficiently detected and tracked from videos captured under various lighting and weather conditions. With the lane information, different dangerous driving events like lane departure can be easily analyzed for driver assistances. All the involved operations are very simple and effective for hardware implementation. Extensive experimental results reveal the feasibility and superiority of the proposed approach in lane detection.