He lives in an enormous brick house near where he grew up, in Mountain Brook Estates, the most exclusive of Birmingham’s suburbs. He doesn’t have an engineering degree, like his father. He studied business at Auburn University—a respectable state school—and went straight to work for the family firm. It is difficult to pinpoint his net worth, but it is safe to say that he has done very well.

Harbert tries hard not to draw attention to himself or his money. He rarely talks to reporters and declined to be interviewed for this article. But his money has spoken for him. He donated $5 million to his alma mater, just as his father did (although Raymond’s appli­cation to join the school’s board of trustees was sidelined by the Alabama state senate). His political contributions suggest that he’s a devoted Republican, and a report by the watchdog organization Public Citizen has identified him as a low-profile contributor of funds to a group lobbying to permanently eliminate the estate tax—which makes sense if he’s to inherit a chunk of the estimated $1.6 billion fortune his mother, Marguerite “Wita” Jones Harbert, now 85, will someday bequeath her children. (According to Harbert’s spokesman, he is no longer involved with the estate-tax group.)

When I ask people about Harbert, they don’t have much more to add, although they all seem to remember his father. It’s impossible to understand Raymond and the way he operates, they say, without first knowing John. One close watcher of the Harbert family remarks that “it’s difficult to follow a very strong-willed, successful, and dynamic father and become very successful in your own way, and that’s what Raymond’s done.”

John Harbert III was born in Greenville, Mississippi, and made his life in Birmingham. His legacy looms—literally—over the city. He built one of its tallest buildings and blasted his way through a mountain to lay a highway that runs through town. He was the kind of man who always carried a pocketknife to cut his apples with. “They don’t make ’em like that anymore,” says William Powers, who worked with John from 1968 to 1990. Referring to John’s legendary powers of persuasion, Powers says, “He could sit you down and talk with you, and you’d whip your weight in wildcats.”

After returning home from W.W. II, John started Harbert Construction using cash from his war bonds and winnings from shooting craps, according to a biography commissioned by Wita after his death in 1995. His first project was building a bridge over a creek in Autauga County, Alabama. The company soon expanded across the South and then to South America and the Middle East. In 1969, John began buying leases for about 240,000 acres of coal mines in Kentucky, eventually spending a total of $150 million, according to Forbes. The venture didn’t come without challenges: He was vehemently antiunion, and during a United Mine Workers strike in 1971 he led an armed convoy of trucks across the picket line to prevent the union from organizing his men.

In 1981 came a watershed event in the life of the company, now called Harbert Corp.: John sold the firm’s Kentucky coal mines to Standard Oil. Three years later, he was named to the Forbes 400 list for the first time, with an estimated net worth of $500 million.

In a 1975 address at Birmingham’s Samford University, John shed some light on his attitude toward home life. “In many cases, a man’s wife and children must take second place in receiving his attention,” he said. “The life of a construction man means a sacrifice by his entire family.”

The family had no shortage of material comforts. They lived on an estate that covered 13 wooded acres on a Mountain Brook hill, where there was plenty of space, even with the staff Wita employed. The house was so isolated that Raymond’s sister, Margie, 47, could remember only one visit from a trick-or-treater. John acquired a 2,000-acre farm in St. Clair County, Alabama; a mountain retreat in North Carolina; and Pinebloom, a 10,000-acre hunting plantation in Georgia where he and Wita shot quail, which was the inspiration for the plantation in Tom Wolfe’s A Man in Full.

The three Harbert children were known as the little princes and princess of Mountain Brook, members of Birmingham’s first family. When I ask Margie what it was like growing up privileged in the civil-rights-era South, she says she was only vaguely aware of her family’s status, because money was never discussed. She remembers asking a childhood friend to explain what “keeping up with the Joneses” meant. “You are the Joneses,” her friend replied.

Presiding over the family then and now is Wita, the matriarch. Since John’s death from pancreatitis in 1995, his widow has become a target of gossip and speculation around town—the rich old lady sequestered in a mansion on a hill. She still lives in the family’s Mountain Brook compound with her son Jay, 55, and the staff of attendants—including her beloved Jamaican manservant, Neville—who look after her.