Passive walk-assist robots play an important role in providing safe mobility to the elderly. They use the brake torques on the wheels to assist the user walking and differentially steer the vehicle, while the user pushes the walker. It is important to appropriately select the brake torques in accordance with user-applied forces for controlling the robot. In this paper, a receding horizon passive control method is presented to calculate the brake torques so that the robot can perform point-to-point guidance. The control scheme first designs a smooth path to be followed by the robot. The terminal-state penalty is added to the cost function for path following, while the end point equality constraint is used to achieve the goal when the robot approaches the end point. The passivity of the system is guaranteed by adding the brake torque constraints into the formulated optimization problem. To be efficient, we look for a suboptimal solution of the optimization problem by using an analytical method. Simulation results show that the passive walk-assist robot, along with the proposed control scheme, can guide the user to a goal effectively.