Would you refactor your app as you go or focus on completing app 1st. Refactoring will mean progress of app app will slow down. Completing app will mean you get a possibly very hard to maintain app later on?
Make it work. Then make it fast. Finally, make it beautiful.
If you have good separation between your code (presentation, business, and data layers) using interfaces, and it's not a monolithic design, then refactoring should not be that difficult.
If you're having that much difficulty refactoring, that's probably a code-smell -- I suggest you look at Solid PrinciplesTweet