Background removal is often used for segmentation of the main subject from a photograph. This paper proposes a new method of background removal for a single image. The proposed method uses Shannon entropy to quantify the texture complexity of background and foreground areas. A normalized entropy filter is applied to compute the entropy of each pixel. The pixels can be classified effectively if the entropy distributions of the background and foreground can be distinguished. To optimize performance, the proposed method constructs an image pyramid such that most background pixels can be labeled in a low-resolution image; thus, the computational cost of entropy calculation can be reduced in the image with the original resolution. Connected component labeling is also adopted for denoising to retain the main subject area completely.