xkcd on the use of made-up words in fiction.
He’s basically right, too. Unless you’re J. R. R Tolkien, and marinated so thoroughly in philology, literature, and Indo-European linguistics that you might as well be writing your novel in Elvish or Anglo-Saxon and translating it into standard English as you go along . . . think twice before adding neologisms to your story’s vocabulary.
But if you have to do it —
Make certain that your invented words can be read and pronounced by an English speaker (if you’re writing in English for an English-speaking audience) with no more than a typical grade-school acquaintance with phonics. If you’re unsure about any of your words, get somebody else to tackle them cold and listen for what works and what doesn’t.
Compounding your new terms from Greek and Latin roots can provide your story with an erudite or technical flavor. If you don’t want overtones of the lab or the library hanging about your epic tale, consider making your new words by compounding English terms instead. And needless to say, if you’re bound and determined to use Greek and Latin elements, take the time to get them right.
And in this age of easy internet searches, it wouldn’t hurt to put each of your invented terms through Google Translate and a couple of search engines, just to make certain that you haven’t independently recreated a thundering obscenity in some language you’ve never even heard of (but which will, if you let it stand, turn out to be the native tongue of your most keen-eyed reader.)