I have come an conclusion, if you are making game than you are going to need a set of tools, and an engine (its going to make your life a heck of a lot easier). Using a pre-existing game engine will save you alot of time and hassle but might cost you later if you hit its limitations. At the beginning project, everyone in the group will be really excited (but this excitement only lasts a short time) if you squander this time working on the engine (some optimization, or file loading) rather than the game you will find that the rest of the team will lose interest and you will a series of lulls in production. That is why I advise you to already have your engine finished before starting your project (or using a preexisting engine).

A design document is actually very important, as it will allow to ensure that the game has enough gameplay to warrant the project, it is also helpful to convey to the rest of your team a clear and coherent vision of what you are trying to achieve. Without a design document you are going to be lost.

At the beginning of a project you need to be able to prototype your idea to see if it should be scrapped or continued (the quicker you can do this the better off you are). It is imperative to ensure that your team members that you explain that you are in the prototyping phase. For example you should have your modelers create a low poly version of your assets, so that you can get a feel for how it should look but ensure that they are not spending massive amounts of time (otherwise they will not be too happy to spend 3 weeks on something only to scrap it).

Finally you are going to need deadlines, this might seem like a bad idea (especially when the project members are doing in there free time and not getting paid) but its essential or you will soon find yourself drowning in a pool of inaction.