I'm not 100% sure but I think a house of refuge was a "poor house", meaning people who had run out of money and had no relatives living in the area that would take them in. If they were not able to find a job or make a living they went to stay in the poor house. It was in the days before the government welfare system. A genealogy/history librarian might be able to confirm this.