What reasons does Holden give for being "sort of sorry" for visiting Mr. Spencer in Catcher in the Rye?

There are two main reasons that Holden gives as to why he is "sort of sorry" for visiting Mr. Spencer.  The first reason is because Mr. Spencer has been sick, and Holden has to visit him in his room. 

Holden comments,

"He was reading the Atlantic Monthly, and there were pills and medicine all over the place, and everything smelled like Vicks Nose Drops.  It was pretty depressing.  I'm not too crazy about sick people, anyway.  What made it even more depressing, old Spencer had on this very sad, ratty old bathrobe that he was probably born in or something.  I don't much like to see old guys in their pajamas and bathrobes anyway.  Their bumpy old chests are always showing.  And their legs...always look so white and unhairy".

It is uncomfortable for Holden to see his history teacher looking sick and old.  He is used to interacting with him in the more professional environment of the classroom.  To see his teacher as vulnerable, elderly, and mired in all the unpleasantness of illness is hard for Holden to take.

After the two have engaged in a bit of small talk, Holden begins to "feel a terrific lecture coming on", which is the second reason why he regrets having come to visit Mr. Spencer.  Mr. Spencer knows that Holden has been kicked out of Pencey, and begins to repeat the same admonitions Holden had already received from Dr. Thurmer, about "Life being a game and all...and how you should play it according to the rules".  Mr. Spencer than brings up Holden's parents, what "grand people" they are, and how they will react when they hear the news of their son's expulsion, before proceeding to the topic of the reasons why Holden has been expelled in the first place. He actually confronts Holden with a sample of his poor work, which embarrasses Holden and makes him feel resentful.

Mr. Spencer ends his lecture by telling Holden that he'd "like to put some sense in that head of (his)", and that he is only trying to help him.  When Holden finally leaves, he is irritated, focusing on the "phoniness" of Mr. Spencer's sincere concern, and he feels "sad as hell" (Chapter 2).