The Economy's Hidden Problem: We're Out of Big Ideas The Wall Street Journal. Greg Ip

By all appearances, we’re in a golden age of innovation. Every month sees new advances in artificial intelligence, gene therapy, robotics and software apps. Research and development as a share of gross domestic product is near an all-time high. There are more scientists and engineers in the U.S. than ever before.

None of this has translated into meaningful advances in Americans’ standard of living.

Economies grow by equipping an expanding workforce with more capital such as equipment, software and buildings, then combining capital and labor more creatively. This last element, called “total factor productivity,” captures the contribution of innovation. Its growth peaked in the 1950s at 3.4% a year as prior breakthroughs such as electricity, aviation and antibiotics reached their maximum impact. It has steadily slowed since and averaged a pathetic 0.5% for the current decade.

Outside of personal technology, improvements in everyday life have been incremental, not revolutionary. Houses, appliances and cars look much like they did a generation ago. Airplanes fly no faster than in the 1960s. None of the 20 most-prescribed drugs in the U.S. came to market in the past decade.

The innovation slump is a key reason the American standards of living have stagnated since 2000. Indeed, absent a turnaround, that stagnation is likely to continue, deepening the malaise that has left the middle class so dissatisfied.

Economists hotly debate the reasons, but there are several clear forces at play. The hurdles for transforming ideas into commercially successful products have grown. The low-hanging fruit in science, medicine and technology has been harvested and new advances are costlier, more complex and more prone to failure. Innovation comes through trial and error, but society has grown less tolerant of risk.

Regulations have raised the bar for commercializing new ideas while directing a growing share of innovative effort toward goals with benefits, such as cleaner air, that don’t translate into gross domestic product. Meanwhile, a trend toward industry concentration may have made it harder for upstart innovators to gain a toehold.

The innovation drought isn’t insoluble. Capital is plentiful, and some of the hype is valid: Old-line companies and upstart entrepreneurs alike are making high-risk bets on cars, space travel and drones, and some policy makers are trying to ​tolerate more risk so that these bets succeed.

More-optimistic economists note it can take years for breakthrough innovations to transform the economy. Some 40 years elapsed after the introduction of the electric lightbulb in 1879 before electricity had a measurable impact on national growth. It took some 20 years after the introduction of the personal computer in the 1970s for information technology to lift productivity.

“There has been a burst of innovations recently, especially in artificial intelligence, ​that we will see come to fruition in the next five to 15 years,” predicts Erik Brynjolfsson, an economist at the Massachusetts Institute of Technology. “You can easily imagine that as these come to maturity and pervade the economy, the effects will be staggering.”

Still, apart from information technology, the hurdles to innovation are getting higher, not lower, and nowhere more acutely than in medicine.

In the past century, vaccines, antibiotics and clean water vanquished humanity’s biggest killers. Early researchers were aided by reliable theories for how to attack common diseases, which made it easy to figure out which compounds might yield a cure. Most of those diseases now have therapies.

“There is no longer either a commercial or scientific reason to search for any more anti-stomach-ulcer drugs,” says Jack Scannell of Oxford University’s Center for the Advancement of Sustainable Medical Innovation.

What’s left, he says, are diseases such as Alzheimer’s for which scientists lack a useful theory of treatment, leaving them with multiple dead ends so far. Mr. Scannell and several co-authors estimate the number of new drugs approved in the U.S. per dollar of research and development has fallen by half every nine years between 1950 and 2010. Approvals have risen since, though 40% are for “orphan” drugs which address diseases that afflict fewer than 200,000 people.

The declining payoff to medical research is starkly illustrated by a new study by Charles Jones of Stanford University and three co-authors. It found that in the decades before 1985, years of life saved through breast cancer treatment rose steadily each year, along with the volume of research. But since 1985, improvements in mortality slowed. They calculate that each new published trial added 16 years of life per 100,000 people in 1985, and that fell to less than one year by 2006. They found the same pattern across agriculture and semiconductors: steadily declining productivity per researcher.

Drugs are symptomatic of the rising value affluent societies place on human life. In 1960, 7% of U.S. R&D was devoted to health care. By 2007, it was 25%, according to another study by Stanford’s Mr. Jones. Thus, health research is displacing R&D that could have gone toward more mundane consumer products. Indeed, Mr. Jones predicts the rising value of human life virtually dictates slower growth in regular consumer goods and services—and they constitute the bulk of measured GDP.

Undoing the damage that past innovations—the burning of fossil fuels, for example—have done to the environment and human health is also gobbling up more innovative effort. This directly eats into the consumer’s pocketbook. The portion of a car’s price that pays to meet federal safety and fuel efficiency mandates has gone from zero in 1967 to 22% now, or $5,500 on a $25,000 car, according to Sean McAlinden, an economist at the Center for Automotive Research, an industry-supported think tank.

These have delivered genuine benefits: Highway fatalities fell from the late 1960s until recently, and the air is cleaner. Mr. McAlinden notes consumers may not have bought those features if given the choice.

A California mandate first introduced in 1990 now aims to make one in seven cars in the state emit zero emissions, which means powered by hydrogen or electricity. ​So while the purpose of the mandate, less pollution, is broadly shared, it achieves it by forcing car makers to favor certain technologies ​over others that may be commercially more viable.

Electric cars, for example, cost more and perform worse than equivalent gasoline cars; thebatteries subtract space and add weight, and mileage is limited especially in extreme temperatures. Even with significant federal subsidies, sales have been hammered by low gasoline prices. Electric and hybrid vehicles together made up 1.9% of national sales so far this year, the lowest since 2006, according to Edmunds.com.

Electric cars don’t yet offer a “value proposition that resonates with the mainstream customer,” says John Viera, head of sustainability at Ford Motor Co. He contrasts that with EcoBoost, a Ford-developed gasoline injection technology that achieves the same power with fewer cylinders. “The beauty is you get the fuel economy improvement with no loss in performance,” he says. “It does add cost, but the customer is willing to pay for that technology unlike with the electrified vehicle.”

Innovation proceeds by trial and error, and errors sometimes kill people. Plane crashes, toxic waste spills and financial crises routinely lead to new regulations that make the world safer, but raise the bar for future innovation. The postcrisis imperative to prevent another has led to toughened financial regulation that has limited the supply of home-equity loans, credit cards and business loans that are often how new businesses finance themselves.

Joel Mokyr, a technology historian at Northwestern University, says innovation is “a messy process inevitably with some negative bite-back. But I have this sense we have become more risk-averse; we are less willing to accept the fact that things can go wrong.”

Hobbyists and the military have operated drones for years, but drones didn’t offer much commercial advantage over manned aircraft. Then, in the past decade the cost of one critical component, the gyroscope that keeps the vehicle level, plunged as the devices were developed for smartphones. Yet commercial drone operation was illegal, with some exemptions, because it required Federal Aviation Administration approval, which is designed for manned aircraft and requires a licensed pilot.

The FAA, at Congress’ request, introduced new rules last year that still restrict the operation of drones. They must generally stay within sight of the operator, below a certain altitude, to avoid collision with manned aircraft. Eli Dourado, a scholar at the Mercatus Institute, a free-market think tank, thinks that is overkill. Birds vastly outnumber drones. Yet in 25 years there have been just 12 fatal collisions between an aircraft and wildlife, and the only one involving a commercial airliner didn’t involve birds: it struck a pair of deer while landing.

The continued restrictions on commercial drones not only limit their use for consumer delivery by the likes of Amazon, but potentially lifesaving roles. U.S. railroads must regularly inspect their track, tunnels, bridges and signals, often in remote territory, usually from the ground, notes the Association of American Railroads. It is labor-intensive and sometimes dangerous. In remote regions, manned aircraft can’t fly low enough to spot problems.Drones would thus be ideal. The association notes such use is limited by the requirement that drones remain in the operator’s line of sight, stay below 500 feet, and not fly over people: “At a major derailment, for example, there potentially will be numerous railroad personnel on the scene.”

Despite these burdens, innovation is continuing, and in some fields, at an astonishing pace, nowhere more so than on the internet and on smartphones.

Amazon.com Inc. is almost single-handedly lifting productivity in retailing. J.P. Morgan notes the average internet retailer generates $1.3 million in sales per employee, compared with the average brick-and-mortar retailer’s $279,000. As Amazon’s market share has grown, that has lifted the entire industry’s productivity performance. Retail output per hour rose 3% in the past year, compared with 0.8% for business as a whole.

Yet there is a less encouraging side to this boom. A study by the Organization for Economic Cooperation and Development found that productivity growth has accelerated at “frontier” companies, which use the most efficient processes and technology, while slowing at the remainder of firms. In other words, productivity is being held back by the inability of competitors to the likes of Amazon, Facebook and Google to catch up.

The authors speculate this may be because new technologies are in fact amalgams of technologies and business processes that are difficult to replicate and often patent protected. Many digital companies are “platforms” that invest heavily in proprietary algorithms to more efficiently match customers to what they need, whether it is a Google search, an Uber car, or an eBay auction. Platforms in turn enjoy network effects: The more users join, the more useful they are to each user.

So long as the frontier firm continues to innovate, that doesn’t hold back productivity. The risk is that once a firm becomes dominant, no competitor can match its network and innovation is less necessary to retain customers.

“The market position came from some early innovative activity,” says Fiona Scott Morton, an economist at Yale University and former federal antitrust official. “But it need not continue if my network effects are very strong. Everyone is on Facebook. What will cause us to leave Facebook? It would have to get dramatically worse.”

Faced with these barriers, what’s the solution? Mr. Jones of Stanford University says it takes more researchers today to produce an innovation of equivalent benefit than in the past. This means society will have to devote a bigger share of its people and resources to R&D just to maintain the same growth rate.

One way is to better exploit knowledge in other countries. Historically, poor countries caught up to rich ones by copying their ideas, as China has done so effectively. Now, the flow of ideas can run in the opposite direction as the volume of research in India and China explodes.

Regulators may have to become more tolerant of risk. Self-driving automobiles offer one bit of proof that they are.

Car manufacturers are eager to develop autonomous driving technology which, unlike alternative fuel technology, is being driven by market demand, not government mandate. Indeed, patents for autonomous driving technology took off rather suddenly in 2012 and already features such as automatic emergency braking and adaptive cruise control are standard on many models. Truly driverless cars are still years away, but that hasn’t stopped many companies from pouring billions into the pursuit of them.

Then last May, Joshua Brown, a 40-year-old Ohio resident, was killed when his Tesla hit a tractor trailer while operating under Tesla’s “autopilot” mode. The incident could have triggered a regulatory crackdown that brought deployment of the technology to a halt. Instead, the National Highway Traffic Safety Administration in September announced nonbinding guidance on how manufacturers should ensure their systems are as safe as possible.

“It’s very different from our usual model,” says Anthony Foxx, the Secretary of Transportation. Auto safety standards are typically quite prescriptive, he said. This time, “we’re leaving room for the industry to establish approaches to safety we may not have thought about.”