Kelly was born in Paris, Kentucky. He began his athletic career in high school, competing successfully in basketball, football, and track and field. He attended the University of Louisville and left during his freshman year to begin studying Shorin-ryu karate. Additionally, he received his black belt in Okinawa-te Karate under the direction of Shihan Gordon Doversola. After winning the 1971 International Middleweight Karate Championship, he opened his own martial arts school.
As an actor, Kelly is best known for co-starring alongside Bruce Lee in Enter the Dragon. This appearance led to starring roles in a string of martial arts-themed Blaxploitation films, among them Melinda and Black Belt Jones. Most of Kelly’s film roles played up the (then) novelty of an African-American martial arts master. After his appearance in 1982’s One Down, Two to Go, Kelly appeared in movies only rarely.