Permits to have them or not? That is not the question. They really are for your protection, but there are a number of drawbacks to getting them. Typically, they require plans or drawings that the contractor or some other professional needs to create. This can add to the cost.
Additionally, they cost money for the actual permit. In Berkeley and other locals, they are divided into the planning portion and the building portion. They each charge fees. With Berkeley, the building permit also has sub-categories of mechanical, electrical and plumbing categories. Oftentimes the cost can range dramatically up to 12% of the contract amount.
Inspections are necessary for each category; each typically have a rough and a final inspection. A job card is issued for the permit and building inspectors need to initial and date each completed category. Giving the inspectors donuts doesn't work, but it is great to have a good cordial relationship with them. They are there to protect both the client and the contractor. This can slow down the contract completion, but offers peace of mind that another professional is ensuring code compliance.
The upside to having a permit is that everything on the contract is allowed by the city, it is ensures safety for the client and the property value increases. You might have a little higher tax bill, but you get to claim that value at the sale of the house.