I think it is definitely a good thing that you are conscientious of cultural appropriation. It is good that you are trying to work through the contradiction that you are feeling (“I still wish to go ahead with this book, knowing full well that I am appropriating a culture while simultaneously loathing people who appropriate cultures.”) This kind of self awareness is important. It is also good to see authors want to feature characters of color and to create fantasy worlds that are not based on medieval Europe.
http://racebending.tumblr.com/post/30462111875/attn-the-person-who-submitted-the-post-about-being