Brown broke the single-season rushing record in 1958, gaining 1,527 yards in the 12-game season, shattering the previous NFL mark of 1,146 yards set by Steve Van Buren in 1949. In this MVP season, Brown led all players with a staggering 17 touchdowns scored, besting his nearest rival, Baltimore Colts wide receiver Raymond Berry, by 8.
Perhaps more impressive was his success as a multisport athlete. In addition to his football accomplishments, he excelled in basketball, track, and especially lacrosse. As a sophomore, he was the second-leading scorer for the basketball team (15 ppg), and earned a letter on the track team. In 1955, he finished in fifth place in the Nation Championship decathlon. His junior year, he averaged 11.3 points in basketball, and was named a second-team All-American in lacrosse. His senior year, he was named a first-team All-American in lacrosse (43 goals in 10 games to rank second in scoring nationally). He is in the Lacrosse Hall of Fame. The Carrier Dome has an 800 square-foot tapestry depicting Brown in football and lacrosse uniforms with the words "Greatest Player Ever".
As a sophomore at Syracuse University (1954), Brown was the second-leading rusher on the team. As a junior, he rushed for 666 yards (5.2 per carry). In his senior year in 1956, Brown was a consensus first-team All-American. He finished fifth in the Heisman Trophy voting and set school records for highest season rush average (6.2) and most rushing touchdowns in a single game (6). He ran for 986 yards—third-most in the country despite Syracuse playing only eight games—and scored 14 touchdowns. In the regular-season finale, a 61–7 rout of Colgate, he rushed for 197 yards, scored six touchdowns, and kicked seven extra points for a school-record 43 points. Then in the Cotton Bowl, he rushed for 132 yards, scored three touchdowns, and kicked three extra points, but a blocked extra point after Syracuse's third touchdown was the difference as TCU won 28–27.
During Brown's career, Cleveland won the NFL championship in 1964 and were runners-up in 1957 and 1965, his rookie and final season, respectively.
After nine years in the NFL, he departed as the league's record holder for both single-season (1,863 in 1963) and career rushing (12,312 yards), as well as the all-time leader in rushing touchdowns (106), total touchdowns (126), and all-purpose yards (15,549). He was the first player ever to reach the 100-rushing-touchdowns milestone, and only a few others have done so since, despite the league's expansion to a 16-game season in 1978 (Brown's first four seasons were only 12 games, and his last five were 14 games).
In 1965, Brown was arrested in his hotel room for assault and battery against an 18-year-old named Brenda Ayres; he was later acquitted of those charges. A year later, he fought paternity allegations that he fathered Brenda Ayres' child. In 1968, Brown was charged with assault with intent to commit murder after model Eva Bohn-Chin was found beneath the balcony of Brown's second-floor apartment. The charges were later dismissed after Bohn-Chin refused to cooperate with the prosecutor's office. Brown was also ordered to pay a $300 fine for striking a deputy sheriff involved in the investigation during the incident. In Brown's autobiography, he stated that Bohn-Chin was angry and jealous over an affair he had been having with Gloria Steinem, and this argument is what led to the "misunderstanding with the police".
In 1965, Brown was the first African-American to announce a televised boxing match in the United States, for the Terrell-Chuvalo fight, and is also credited with then first suggesting a career in boxing promotion to Bob Arum.
In early 1966, Brown was shooting his second film in London. MGM's The Dirty Dozen cast Brown as Robert Jefferson, one of 12 convicts sent to France during World War II to assassinate German officers meeting at a castle near Rennes in Brittany before the D-Day invasion. Production delays due to bad weather meant he missed at least the first part of training camp on the campus of Hiram College, which annoyed Cleveland Browns owner Art Modell, who threatened to fine Brown $1,500 ($11,877.69 in 2019 dollars) for every week of camp he missed. Brown, who had previously said that 1966 would be his last season, the final year of a three-year contract, announced his retirement, instead.
Brown retired in July 1966, after nine seasons, as the NFL's all-time leading rusher. He held the record of 12,312 yards until it was broken by Walter Payton on October 7, 1984, during Payton's 10th NFL season. Brown is still the Browns' all-time leading rusher. As of 2018 Brown is 11th on the all-time rushing list.
In 1970, Brown was found not guilty of assault and battery, the charges stemming from a road-rage incident that had occurred in 1969. In 1975, Brown was sentenced to one day in jail and two years' probation and ordered to pay a fine of $500 for beating and choking his golfing partner, Frank Snow. In 1985, Brown was charged with raping a 33-year-old woman. The charges were later dismissed. In 1986, Brown was arrested for assaulting his girlfriend Debra Clark. Clark refused to press charges, though, and Brown was released. In 1999, Brown was arrested and charged with making terrorist threats toward his wife. Later that year, he was found guilty of vandalism for smashing his wife's car with a shovel. He was sentenced to three years' probation, one year of domestic violence counseling, and 400 hours of community service or 40 hours on a work crew along with a $1,800 fine. Brown ignored the terms of his sentence and in 2000 was sentenced to six months in jail for refusing the court-ordered counseling and community service. He was released after 3 months.
Brown's memorable professional career led to his induction into the Pro Football Hall of Fame in 1971. His football accomplishments at Syracuse garnered him a berth in the College Football Hall of Fame.
Brown served as a color analyst on NFL telecasts for CBS in 1978, teaming with Vin Scully and George Allen.
In 1983, 17 years after retiring from professional football, Brown mused about coming out of retirement to play for the Los Angeles Raiders when it appeared that Pittsburgh Steelers running back Franco Harris would break Brown's all-time rushing record. Brown disliked Harris' style of running, criticizing the Steelers' running back's tendency to run out of bounds, a marked contrast to Brown's approach of fighting for every yard and taking on the approaching tackler. Eventually, Walter Payton of the Chicago Bears broke the record on October 7, 1984, with Brown having ended thoughts of a comeback. Harris himself, who retired after the 1984 season after playing eight games with the Seattle Seahawks, fell short of Brown's mark. Following Harris's last season, in that January, a challenge between Brown and Harris in a 40-yard dash was nationally televised. Brown, at 48 years old, was certain he could beat Harris, though Harris was only 34 years old and just ending his elite career. Harris clocked in at 5.16 seconds, and Brown in at 5.72 seconds.
He was in 1987's The Running Man, an adaptation of a Stephen King story, as Fireball and had a cameo in the blaxploitation spoof I'm Gonna Git You Sucka (1988).
In 1988, Brown founded the Amer-I-Can Program. He currently works with juveniles caught up in the gang scene in Los Angeles and Cleveland through this Amer-I-Can program. It is a life-management skills organization that operates in inner cities and prisons.
Brown's autobiography, published in 1989 by Zebra Books, was titled Out of Bounds and was co-written with Steve Delsohn. He was a subject of the book Jim: The Author's Self-Centered Memoir of the Great Jim Brown, by James Toback.
In 1993, Brown was hired as a color commentator for the Ultimate Fighting Championship, a role he occupied for the first six pay-per-view events.
In 2002, The Sporting News selected him as the greatest football player of all time, as did the New York Daily News in 2014. On November 4, 2010, Brown was chosen by NFL Network's NFL Films production The Top 100: NFL's Greatest Players as the second-greatest player in NFL history, behind only Jerry Rice. In November 2019, he was selected as one of the twelve running backs on the NFL 100 All Time Team.
In 2002, film director Spike Lee released the film Jim Brown: All-American, a retrospective on Brown's professional career and personal life.
In 2008, Brown initiated a lawsuit against Sony and EA Sports for using his likeness in the Madden NFL video game series. He claimed that he "never signed away any rights that would allow his likeness to be used".
Brown is also a part owner of the New York Lizards of Major League Lacrosse, joining a group of investors in the purchase of the team in 2012.
On May 29, 2013, Brown was named a special adviser to the Browns.
On October 11, 2018, Brown along with Kanye West met with president Donald Trump to discuss the state of America among other topics.
On January 13, 2020, Brown was named the greatest college football player of all time by ESPN, during a ceremony at the College Football Playoff National Championship Game celebrating the 150th anniversary of college football.