It is a natural impossibility that a government should have a right to punish men for their vices; because it is impossible that a government should have any rights, except such as the individuals composing it previously had, as individuals. They could not delegate to a government any rights which they did not themselves possess. They could not contribute to the government any rights, except such as they themselves possessed as individuals. Now, nobody but a fool or an impostor pretends that he, as an individual, has a right to punish other men for their vices. But anybody and everybody have a natural right, as individuals, to punish other men for their crimes; for everybody has a natural right, not only to defend his own person and property against aggressors, but also to go to the assistance and defense of everybody else, whose person or property is invaded. The natural right of each individual to defend his own person and property against an aggressor, and to go to the assistance and defense of every one else whose person or property is invaded, is a right without which men could not exist on the earth. And government has no rightful existence, except in so far as it embodies, and is limited by, this natural right of individuals.
From: Vices Are Not Crimes