Walker played high school ball at Highland Park with Bobby Layne. Together they made it to the state championship. Walker was a 5 sport star in high school. After some time in the Marines, he went on to play college ball. He then went on to SMU where he had a stellar career and won the 1948 Heisman Trophy. He is considered by some as the best college running back ever to play.
Walker was drafted third in the 1949 NFL Draft by the Boston Yanks but was traded to the Detroit Lions. He was reunited with fellow Texan Bobby Layne. He put up over 1,200 total yards as a rookie and was First Team All-Pro. He was all-pro in his second season too. In his third and fourth seasons, he helped lead Detroit to back-to-back NFL Championships. Walker played only 5 seasons in the NFL because he had multiple business interests that paid significantly more. He made the pro-bowl in all 5 seasons and was first-team all-pro 4 times. When he retired, he was third all-time in points scored, despite only playing 5 years.