fucthest8 - you aren't going to get a fully rounded unbiased viewpoint from me about this, and I consciously make an effort usually NOT to tell people whether their cats should be indoors or allowed out, it's not something where I think there is a wrong or a right of it, I do it one way myself but I am aware that is controversial and it certainly isn't a hill I want to die on.
For me, personally, my cats have always been 100% indoor cats. Only have 1 left now, but they all had really happy lives, I don't think anyone who met them would have said they were missing out on anything fundamental to their wellbeing. I have a lot of vertical space for them indoors (ceiling high bookcases lining every room) which is something they enjoy.
I've never kept in a cat that was used to going out however, that is a bit different territory for me and I can't comment directly on that. Except if he had contracted FIV or FELV or had lost his sight you'd keep him in for the sake of his health and wellbeing, and however difficult, he would adjust to that.
So it's up to you really, if you feel that being out is having a bad impact on him, you may well decide that he needs to stay in. I can't tell you whether that is right for him or you, but I can say with some confidence that indoor cats that have never been out can have the most wonderful lives. With cats that have been allowed out, it is going to be an adjustment for them and one that I cannot give advice about, other than to see how it goes and wish you luck