I'm conflicted about this idea, on one hand, it most likely would reduce lot griefing. However I feel that users should also have complete control over how many people they want to allow on their lot. And as Nos said, sometimes a lot of people might be needed for a big project.
Overall though I think a warning like Nos suggested might be best.
Here's another idea I had, but would take a bit of work to implement:
If an user has a history of griefing (this would require a new system to label known griefers) users would get a warning allowing them.