Rhino started as a record shop on Westwood Boulevard, Los Angeles, during 1973, run by Richard Foos. It became a record distributor five years later thanks to the effort of then-store manager Harold Bronson. Their early releases were mostly novelty records (with their first single being Wild Man Fischer's "Go To Rhino Records") and some punk rock singles; the difficulties involved in getting airplay and distribution for such material eventually caused Foos and Bronson to take the label in other directions. One of their earliest reissues was achieved by acquiring the rights to the White Whale label. By the mid-1980s most of their releases were reissues of previously released recordings licensed from other companies. Superior sound quality (remastering of the original tapes was done under the direction of Bill Inglot) and creative packaging made Rhino one of the most respected reissue record labels, receiving rave reviews from music collectors, fans, and historians. Rhino was quick to get into the compact disc market, releasing dozens of oldies CDs at the dawn of the CD age in 1984. Their retrospective compact disc releases are often remastered to restore or improve upon the original analog release's audio quality.

The company also continued to produce new music, with releases on subsidiary labels such as RNA (Rhino New Artists) and Forward, as well as the main Rhino label. However, the company's entertainers tended to generate more critical acclaim than public interest. One of Rhino's early artists was The Twisters whose Los Angeles popularity far exceeded their album sales. For the most part, sales totals in the low five figures or less were routine for Rhino-produced albums, and the less costly, less risky reissue business remained the company’s primary revenue stream. One exception was the late-1986 success "At This Moment" by Billy Vera & the Beaters, a 1981 song that unexpectedly made it to the top of the U.S. Billboard charts after being featured in a 1986 episode of the hit NBC series Family Ties.

In 1985, Rhino signed a six-year distribution agreement with Capitol Records. During 1989 Rhino and Capitol’s parent EMI made a deal to jointly acquire Roulette Records; Rhino received the US rights to Roulette's catalog, excluding jazz. When the distribution deal with Capitol ended in 1992, Rhino signed a new distribution deal with Atlantic Records, and in turn Time Warner bought a 50% stake in the record company. In 1998, Time Warner bought the other half of Rhino; thus the company became a wholly owned unit of Time Warner. The Rhino Records retail store, which was not part of the sale, closed in 2005. In 1999, Rhino started the Rhino Handmade division of limited edition releases available primarily from their website. All of these Handmade reissues are limited to about 3000 copies or less and once sold out are not re-pressed. (Since Rhino redid their website in early 2013, the Handmade division appears to have been discontinued.)

It is through this merger that the label has released remastered editions of collections from such musicians as Eric Burdon, Fanny, Dannii Minogue, The Ramones, The Grateful Dead, Emerson, Lake & Palmer, The Beach Boys, Yes, The Doobie Brothers, The Cars, Chicago, Tom Paxton, Third Eye Blind, The Doors, War, Spirit of the West and most recently The Bee Gees; as well as soundtracks spanning the Turner-owned pre-1986 MGM and pre-1950 Warner Bros. periods, in addition to WB's own post-1949 period. Rhino's soundtrack releases include Gone with the Wind, The Wizard of Oz, Easter Parade, North by Northwest, Casablanca, King Kong, Doctor Zhivago, Superman, and Finian's Rainbow.

Rhino also owns the rights to The Monkees’ master tapes and film footage (which they acquired from Raybert Productions in 1994), and thus has released home videos and CDs from the group.

In 2003, co-founders and longtime executives Richard Foos and Harold Bronson left Rhino, reportedly due to frustration at being unable to release compilation albums in an increasingly competitive market. In fact, Time Warner's final vesting of its 100% ownership of the label, and its subsequent 'reorganization' of label staff, which did not stop at the former owners, were the major factors in their exits. Soon after, Foos inaugurated a new label, Shout! Factory, which began releasing dozens of CDs and videos mirroring the original early-1990s Rhino philosophy.

In 2004, Time Warner spun off its music divisions and today Rhino is part of the newly organized Warner Music Group.

In addition to dealing with archive material, the label also manages the US distribution or worldwide production of compilations for more recent Warner acts, including still-active artists such as Enya, New Order, and Chicago.

In June 2006, Warner Strategic Marketing in the UK was dissolved and Rhino Records UK was formed. The division has two main factions - TV advertised compilations (for example Pure Garage Rewind Back to the Old School) and catalog material from the Warner vaults. Led Zeppelin's 2007 release Mothership and the soundtrack to the film Juno serve as the label's most recent successes.

Similar music companies include Collectables Records, Hip-O Records, One Way Records, Beat Goes On Records, Varèse Sarabande, and Sony Music's Legacy Recordings, as well as Ace Records of Britain and Bear Family of Germany.