They stopped inviting him to the gatherings, and he really got bent out of shape about it.
Portions of those letters quoted by Bruccoli indicate that though Hemingway could be sympathetic, he used a lot of ink telling Fitzgerald to shape up or ship out.
It's ok, don't worry about returning the books. I don't get bent out of shape about things like that.