This article presents a novel approach for a real-time person tracking system based on particle filters that use different visual streams. Due to the difficulty of detecting a person from a top view, a new architecture is presented that integrates different vision streams by means of a Sigma-Pi network. A short-term memory mechanism enhances the tracking robustness. Experimental results show that robust real-time person tracking can be achieved.