Following the interstellar magnetic field from the heliosphere into space with polarized starlight