4 Books Found

Based on your description:

"science fiction book about a man exiled to time in the far future in a space ship with an AI and they find a potato-shaped asteroid and set up an interstellar bank a series with multiple characters derived from the exile. they build a family on the asteroid and periodically travel back to earth to an earlier time"