Few supporters back building size change