While primarily considered a mental game, clearly there is a major physical component to the game. One need look to further than the PGA tour where 20 years ago you would see guys like John Daley on the course, to todays lean and mean Professional Golfers that more resemble their professional athlete peers from other sports. That said, there are still clearly many outliers on the tour, and one need venture no further than the local golf and country club to see out of shape men and women play excellent golf.
Rather than requiring significant speed or strength, the golf swing requires hand eye coordination and flexibility. Sure, more muscle will allow you to generate more clubhead speed which will allow you to hit the ball further, however your average weekend warrior would experience vastly greater improvements in their games if able to more consistently hit fairways than add 20 yards to their drives.
I am an avid fitness enthusiast and consider myself to be in good shape. That said, I am going to change my normal gym routine this summer from powerlifting and HIIT cardo, to include more golf-appropriate workouts that will maximize flexibility and range of movement. I have traded in my barbells for a yoga mat, and for golf training, on top of hitting balls 5 to 6 days per week, will be engaging in calisthenics, bodyweight movements, hot yoga and lots of stretching on the same frequency, taking minimal days off each week. The ideal outcome will be to maintain my strength while gaining flexibility and range of movement.