As a child he undertook both piano and violin; he was so accomplished on the violin that at the age of ten he played Mendelssohn's second violin concerto at a Cincinnati Symphony Orchestra youth concert. He studied piano at various summer music festivals before enrolling at New York's Juilliard School, where he took conducting courses with Jean Morel and continued piano studies with Rosina Lhevinne.

George Szell of the Cleveland Orchestra then invited him to become an apprentice. At the conclusion of this term, he became an assistant conductor of the orchestra from 1965 to 1970. In 1973 he became the music director of both the Ravinia Festival (the Chicago Symphony's summer series) and of the Cincinnati May Festival. His debut with the Metropolitan Opera Company of New York also came in a summer festival appearance; on June 5, 1971, he conducted Puccini's Tosca.

In 1973, following the sudden departure of music director Rafael Kubelik over artistic differences, the Metropolitan Opera appointed Levine its principal conductor. Although Kubelik was the first conductor ever to have been given that title at the Metropolitan, he did not remain there long enough to actually exercise the wide responsibilities it implies; Levine was effectively the first music director of the Met.

Levine's tenure at the Metropolitan has been one of the longest such associations between an opera company and its chief conductor, providing the company with one of its greatest and most stable periods. His season planning has been outstanding, and the Metropolitan Opera Orchestra owes its reputation as a world-class ensemble to his leadership.

He began in 1975 to appear regularly at the Salzburg Festival in Austria, where his performances of Mozart were highly esteemed, and proved himself an excellent Wagnerian in his appearances at the Bayreuth Festival beginning in 1982. When time allows, he performs as a pianist in chamber music and as an accompanist for singers.

Levine commands a wide repertoire in opera and also in the concert hall, ranging from Bach, Haydn, and Mozart to Iannis Xenakis. He is known for an open and engaging personality, which appears in his rehearsals as well, but this does not prevent him from conducting them with businesslike efficiency. In 1996 the Metropolitan conducted a televised celebration of 25th anniversary of Levine's first performance there in the form of a gala. So many singers agreed to appear that the event lasted over eight hours, conducted entirely by Levine. He has produced a large number of recordings with the Metropolitan Opera and other opera houses, as well as orchestral recordings with the Chicago Symphony and other great orchestras. He is also noted as a pianist who has accompanied many of the world's top vocalists and participated in several chamber music recordings. In 2004, Levine became music director of the Boston Symphony Orchestra.