Life requires fun.
This is coming from a person who has always taken her pursuits seriously. From studies, to athletics, to dating (I had a checklist), I made it a point to do everything within my power to succeed and excell.
When I'm in this mode, I easily put off those things which seem frivolous: playing games with my children, exercise, girls' night, mystery novels... and other things which aren't "productive" or helping me meet my goals.
As I've come to grudgingly accept, doing fun or leisure things is part of being a healthy, well-rounded person. Having something fun to look forward to lightens the load and helps me press forward. It's like shaking out an old rug, letting all the dirt and dust fly- allowing me to be refreshed for the next challenge ahead.
Go ahead. Schedule some fun into your day and your week, and don't feel guilty about it. It will make you a better person.