Peer Review is a process that journals use to ensure the articles they publish represent the best scholarship currently available, and articles from peer reviewed journal are often grounded in empirical research. When an article is submitted to a peer reviewed journal, the editors send it out to other scholars in the same field (the author's peers) to get their opinion on the quality of the scholarship, its relevance to the field, its appropriateness for the journal, etc. Sometimes, you'll see this referred to as "refereed" as well.
Publications that don't use peer review (Time, Cosmo, Salon) just rely on an editor to determine the value of an article and their goal is mainly to entertain and educate the general public. That's why they're often not the best source for your academic papers.
Most library databases will have a search feature that allows you to limit your results to peer reviewed or scholarly sources.
Usually, you can tell just by looking. A scholarly journal is visibly different from other magazines, but occasionally it can be hard to tell, or you just want to be extra-certain. In that case, you turn to Ulrich's Periodical Directory Online. Just type the journal's title into the text box, hit "submit," and you'll get back a report that will tell you (among other things) whether the journal contains articles that are peer reviewed, or, as Ulrich's calls it, Refereed.
They even use a cute little referee's jersey icon:
This work is licensed under a Creative Commons Attribution-NonCommercial 2.0 Generic License.