Fiction or non-fiction--can you just not call it?

I've written several manuscripts that are an exaggerated mix of actual incidents that I've linked together to create a story. The Red Scarf, published in 2007 is one of those. It's a nostalgic look back to the fall of 1944. A funny and sometimes touching story told in the first person by the small town's paper boy. I can't tell you how many times I've been asked, "Now, who was Rosalie?" Or, "Did you really make just $3.50 a week delivering papers?" Of course, the book is a novel, and it's very clear that, even though it's set in a real town with real characters, it's fiction. However, I've found out, that even if you stand before a group and tell them in no uncertain terms, it's fiction, many members of the audience will invariable ask those same questions. So why not just let the reader guess. Don't label your book. I have a real peeve about labeling anyway. The Red Scarf on Amazon.com is categorized as 9-12 reading level, and I've tried to get that removed, because it's a turn off for adults. The novel has sold around 9500 copies and the vast majority of sales are to adults. It is truly a book that can be enjoyed by all ages.
But is the reader smart enough to distinguish between fiction and non-fiction? Yes, and most of the time they don't care. They're interested in a good story. And as for reading levels--get rid of them!