I hear this all the time. Heck, I think this all the time. But for me, it mostly vaporizes. For you, perhaps it keeps re-surfacing. Do you want to pursue your idea? Here's a couple of suggestions to start. This might seem like “duh” but it's really not.
• Write it down. A one-sentence description.
• Research it. Has someone else done a similar book? Is there room for another?
• If you keep thinking about it, make an outline.
• Keep thinking about your book project and refine your outline. The more you refine it, the better it should get.
• Try not to talk about it too much at the beginning. Work it out in your own head before you tell anyone because people are going to want to contribute their own suggestions.