Brought up within the USSR's system for developing musical talent, Järvi studied percussion and conducting at the Tallinn Music School. He made his debut as a conductor at age 18. From 1955 to 1960 he pursued further studies at the Leningrad Conservatory, where his principal teachers were Nikolaï Rabinovich and Yevgeny Mravinsky.

From the early '60s, Järvi took a leading role in the musical life of his homeland. In 1963 he assumed the directorship of the Estonian Radio & Television Orchestra, his first important post. He also founded the Tallinn Chamber Orchestra, and for 13 years, was the chief conductor of Opera House Estonia in Tallinn. From 1976 to 1980 he was chief conductor and artistic director of the Estonian State Symphony Orchestra, then in its infancy. By the late 1970s his fame had spread throughout the Soviet Union and Eastern Europe, and he received favorable notices for his appearances in the West. He made history by leading the first performances of Strauss' Der Rosenkavalier and Gershwin's Porgy and Bess ever given in the USSR.

While with the ESSO, Järvi developed a particular interest in unearthing and performing neglected repertory by both little-known and important composers. He was a particular champion of the Estonian composers Eduard Tubin and Arvo Pärt. In 1979 he premiered Pärt's Credo, a work that represents a turning point in that composer's stylistic evolution. Järvi, recognizing the importance of Credo (which incorporates biblical texts), presented it without first navigating through the usual channels of the Communist Party or the Composers' Union. The resulting controversy and official disfavor induced Järvi to emigrate. He was permitted to leave Estonia in 1980; within a month of his departure, he made his debut performances with the Boston Symphony, the Philadelphia Orchestra, and the New York Philharmonic. He quickly received important appointments: principal guest conductor of the City of Birmingham Symphony Orchestra in England (1981-1983), music director of the Royal Scottish Orchestra (1984-1988), music director of the Gothenburg Symphony Orchestra (Sweden, 1982-2004), and principal guest conductor of the Japan Philharmonic.

In 1990, he assumed the post of music director of the Detroit Symphony Orchestra. With that ensemble, he made 30 of some 150 recordings on the Chandos label. Järvi has also recorded for BIS, Deutsche Grammophon, and Orfeo; his various recording projects include cycles of orchestral music by Sibelius, Prokofiev, Shostakovich, Tubin, Brahms, Schumann, Shostakovich, and others. Järvi announced his decision to step down from his Detroit post in 2005. He has also served as principal conductor of the New Jersey Symphony, and accepted music director positions with the Resident Orchestra of the Hague and the Orchestre de la Suisse Romande for 2012.

Järvi's children have made their mark on the musical world as well: son Paavo is gaining an international reputation as a conductor and holds posts with the Orchestre de Paris and the Frankfurt Radio Symphony Orchestra; Kristjan is the founder and conductor of the Absolut Ensemble of New York City, and daughter Maarika was principal flutist with the RTVE Symphony Orchestra in Madrid.