Smith was born in Humphreys County, Mississippi, to Ioda and Willie Warren Smith, who divorced when he was young. He was raised by his maternal grandparents in Louise, Mississippi, where they had a small farm and dry goods store.
Smith took up the guitar to while away his evenings while in the United States Air Force stationed in San Antonio, Texas. By the time of his discharge from the service, he had decided to make a career of music. He moved to West Memphis, Arkansas, and auditioned, successfully, to play the Cotton Club, a local nightclub. The steel guitarist Stan Kesler, who was playing there with the Snearly Ranch Boys, immediately spotted Smith's potential and took him to Sun Records to audition for Sam Phillips, with the Snearly Ranch Boys providing backup.
Phillips liked what he heard and decided that "Rock 'n' Roll Ruby", a song credited to Johnny Cash, would be Smith's first record. (Smith later claimed that the song was actually written by George Jones and sold to Cash for $40.)[citation needed] Smith recorded it on February 5, 1956. Phillips, playing it safe in case rock and roll did not maintain its popularity, released it with a country crooner, aptly named "I'd Rather Be Safe Than Sorry", on the flip side. By May 26, "Rock 'n' Roll Ruby" had reached number 1 on the local pop chart; this record, his first for Sun, went on to outsell the first Sun releases by Elvis Presley, Johnny Cash and Carl Perkins.
In August 1956, Smith went back to the Sun Records studio to record his second release, "Ubangi Stomp", an infectious rocker with crude lyrics and vocals suggesting an African chief with the syntax of a movie Indian. For the B-side, he recorded the classic ballad "Black Jack David", a song that originated in early 18th-century Britain and survived in various forms in the mountains of the American South; it may be the oldest song ever recorded by a rock-and-roll performer. Although a resounding artistic success, this record did not sell as well as Smith's debut.