They can live everywhere. They prefer to live under human houses, because they can "borrow" food en masse.
The population of Borrowers is widely dispersed. Arrietty's parents didn't know other Borrowers exist outside of their home. Pod Clock get know about other of his kind, when he meets Spiller in the woods. In the woods, many Borrowers live together to protect themselves, because life in the woods is more dangerous than under human houses.
From the Borrowers' perspective, the borrowing is not stealing, because it is necessary for their survival. They never bring the borrowed things back, of course, though.
Because of their size, they have many enemies of bigger size. Haru tries to chase them off. But not all humans are their enemies. Sho protects them from crows and Shawn's grandfather built a Dollhouse for them.