1 Book Found

Based on your description:

"a skinny guy shows up on an isolated island with a wooden raft, the island is occupied by guys significantly larger than him. The people of the island don't know what to do with him, so they put him in a goat's pen because no one is willing to give him work. When the man exits the goat's pen because of how hungry he is, he gets chased down by the island people with weapons of all sorts. The fisherman also tries to help him and plead his case."