You can't weatherstrip the damper (that would be a fire hazard), so build a rustic reclaimed wood fireplace insert to stop drafts. Easy to follow step-by-step tutorial.
Build a Fireplace Insert Draft Stopper - Pretty Handy Girl
Fireplace Accessories » Full Service Chimney™
Portfolio - Home Office - Engineer Your Space
Fireplace Draft Stopper: An Easy DIY Sewing Tutorial
Tutorial for Painting Dandelion Wall Graphic
Pretty Handy Girl
Build a Fireplace Insert Draft Stopper - Pretty Handy Girl
Keep the Cold Air Out — No more shivering through movie nights because of a chilly draft. Get cozy in seconds while saving valuable energy year-round
Choice Home Goods Magnetic Fireplace Cover - Cozy Wood Design Fireplace Draft Blocker - Fire Place Cover for the Living Room - Fireplace Insulation
Build a Fireplace Insert Draft Stopper {a Lowe's Creator Idea}, Pretty Handy Girl
Cheapest way of making a Chimney Draught Stopper
Build a Fireplace Insert Draft Stopper - Pretty Handy Girl
My Fireplace is Too Drafty