Harvey Grimsley was born in Haleburg, Alabama in 1922. His family fled the racial oppression and violence of the Jim Crow-era South and moved to New Jersey during his childhood.
Grimsley attended schools in Bloomfield and then Orange, where his relative Monte Irvin also grew up. Irvin went on to play professional baseball and became a member of the Baseball Hall of Fame. Grimsley graduated from Orange High School in 1942.
During World War II, Grimsley was drafted. He served overseas in Europe in the segregated U.S. Army in an all-Black transportation unit. He and his unit partook in D-Day, the Allied invasion of German-occupied Normandy on June 6, 1944, and landed on Utah Beach. In 1945-'46, Grimsley attended Biarritz American University in Europe and played on the university's integrated basketball team.
After being discharged from the Army, Grimsley was recruited to play football at Rutgers, which he attended on the GI Bill. Between '46 and '49, Grimsley distinguished himself as the Scarlet Knight's leading scorer, despite never starting a game under coach Harvey Harman.
After graduating in the Rutgers College Class of 1950, Grimsley spent his career working as a coach, including being a high school coach in Newark and Piscataway and working as a recruiter for Governors State University in Chicago. He was inducted into the Rutgers Athletic Hall of Fame in 1993.