Ben was a right-handed center fielder and pitcher who played in the farm system of the Philadelphia Phillies in 1981. He was on the Bend Phillies in the Northwest League in Bend, Oregon. Future major leaguer Rick Schu was also on that team.

Rufkahr signed with the Phillies right out of high school. He starred at the plate and on the mound for Duchesne High School from 1977-80. He was named the St. Louis Globe-Democrat Player of the Year in 1980 after compiling a .406 batting average. That year, he received honorable mention as an outfielder on the All-Metro team.

Ben played American Legion ball for St. Charles VFW Post 2866. He played for the Cottleville and Harvester Athletic Association for 16 years. In 1981-82, he was with the C&H Travelers in the Metro Collegiate League. Rufkahr was named to the All-star team in 1982. From 1985 to 1998, he was a center fielder and pitcher for Cottleville and Harvester in the Eastern Missouri Baseball Association.

From 1981-84, he played for O’Fallon in the EMBA. He was on several EMBA championship teams and was selected to the league’s all-star team six times. In 1992, Rufkahr was inducted into the St. Charles County Amateur Sports Hall of Fame.