Really good question, and it’s one of several potential problems with Libertarianism.
So, at face value, you’re right - you do. But then, you can also form organisations within your community to do the same. So then your whole village or street enforces the NAP.
Besides which, it’s not really about other people, it’s about you.
I won’t hurt you or nick your stuff.
Expand your definition. It’s Authoritarianism you should really be afraid of.
When someone has their jackboot on your neck, what does it matter if it’s left or right?