Underarmed and politically fractious, the continent worries it is becoming the biggest loser in the new era of transactional politics

BRUSSELS—In the accelerating contest between great powers, Europe is struggling to keep up.

The continent’s leaders have long worried they will be left behind as the U.S., China and Russia vie for economic, technological and military dominance.

Officials now fear they have reached that point.

Their mood darkened over the summer as Europe was left on the sidelines as the U.S. and China sought to reset the rules of global trade.

It became bleak when the White House presented a plan for ending the war between Russia and Ukraine this month without consulting European leaders.

In response, the European Union crafted a counterproposal more acceptable to Ukraine, and its member states are rushing to rearm as the bloc looks for ways to break its institutional gridlock.

Change will be hard and take time, something many European officials worry the continent doesn’t have.

“Battle lines for a new world order, based on power, are being drawn right now,” European Commission President Ursula von der Leyen said in her annual address to EU lawmakers in September. “A new Europe must emerge.”

How to make that metamorphosis happen is concentrating minds in Europe, where the escalating fear among current and former officials is that the EU’s structure and procedures will leave it among the biggest losers in the new geopolitical pecking order.

European officials are warming up to harnessing smaller groups of countries to make the whole bloc militarily and economically fitter.

Mario Draghi , a former European Central Bank president who was asked last year to design a plan to make Europe more competitive, is pushing for groups of countries to conduct joint defense research and procurement, and to design common rules allowing European tech companies to scale up quickly. Draghi, a former Italian prime minister, wants European industrial giants to pool investments in strategic sectors such as semiconductors to help the continent regain an edge.

It is an approach that is gradually winning support.

“I think that we are finally getting realistic,” said Latvian President Edgars Rinkevics . “You can’t change the dynamics if you don’t have real power—be it political, military or diplomatic.”

For Germany, Europe’s longtime engine of growth, global shifts have dislodged the tentpoles of its economic success: cheap gas from Russia, booming export markets in China and the U.S. defense umbrella.

In response, Berlin has eased its debt brake, allowing it to pour 500 billion euros, equivalent to around $580 billion, into a decadelong rearmament program.

A rearmed Germany combined with the toughened up militaries of Poland, Scandinavian and Baltic states, and the extra layer of defense offered by nuclear-armed Britain and France, could create a coalition to check Russian expansionism, says Nico Lange, a former chief of staff at the German Defense Ministry.

Yet obstacles to wholesale change abound.

Defense ministries won’t easily surrender control over plans and procurement, nor will Europe’s big industrial players easily pivot from competition to collaboration.

The need for consensus, which defines the 27-member EU, often leaves it flat-footed in response to unreliable actors and fast-changing circumstances although the EU surprised many with its swift and resilient response to Russia’s invasion of Ukraine.

“I think we are in the most difficult and dangerous situation since the end of the Second World War,” Danish Prime Minister Mette Frederiksen said at a Copenhagen summit last month.

Events over the past few months have hammered those anxieties home.

In July, the EU had to swallow a lopsided trade deal with the U.S. that allowed Washington to impose 15% tariffs without blowback.

President Trump ignored European calls to pressure Moscow and rolled out the red carpet for his Russian counterpart, Vladimir Putin , at a summit in Alaska in August. “This is not to do with Europe, Europe’s not telling me what to do,” Trump said on the way to the summit. Then he sidelined them in drawing up his Ukraine cease-fire plan.

The trade clash between Washington and Beijing threatened Western access to rare earths, which are critical to Europe’s defense and green transition. When a meeting between Chinese leader Xi Jinping and Trump brought about a temporary truce, it demonstrated to European officials that the continent isn’t the master of its own destiny.

French President Emmanuel Macron in 2017 called for the EU to bolster its military, economic and industrial independence. Last year, in a follow-up speech, Macron warned the European project could die.

“It all depends on the choices we make and these choices need to be made now,” he said.

Pierre Vimont, a former senior EU and French diplomat now at Carnegie Europe, said the EU’s institutions are struggling but that there was little will in capitals to spend years arguing over the bloc’s future setup.

“The whole Brussels institutional framework, its methods, its mindset were not at all tailored” for the current period of “power politics, confrontation, highly brutal competition,” Vimont said.

Looming over it all is the U.S.’s hardening stance toward Europe. Trump has forced European NATO members to pay more for their defense and that of Ukraine. The EU has set a 2030 rearmament goal. This year, the region is set to spend more than $560 billion on defense, according to analysts at Bernstein, double what it spent a decade ago.

Still, Europeans remain nervous about Trump’s allegiance to the North Atlantic Treaty Organization. His covetous comments about Greenland , an autonomous Danish territory, deepened concerns.

Europe had always believed access to its wealthy single market gave it real trade clout. But July’s trade negotiations with Washington punctured that belief, showing that the U.S. would wield security leverage over Europe to win a trade clash.

Europe had sought to avoid confrontation with China.

But Beijing continues to flood Europe with cheap imports as its own domestic economy slows, while China’s technological edge and mass market have seen it pull ahead of European competition in industries such as electric vehicles, forcing significant job losses in Germany.

Washington has vacillated between pressing Europe to hit China with tariffs to cutting its own deals with Beijing.

The coming years will tell “whether Europe will remain an independent economic power…or whether we will become a pawn of the major economic centers in Asia or America,” German Chancellor Friedrich Merz said last month.

When Josep Borrell became EU foreign policy chief in 2019, he received an intelligence document spelling out threats. It listed the risk that Russia would invade Ukraine, fresh violence would erupt between Palestinians and Israelis and migration flows would increase. It warned of trade friction between China and Europe, and Europe and the U.S.

For Borrell, who stepped down last year, it justified his earlier warning that Europe “must learn to speak the language of power.”

“I produced hundreds of EU statements asking other people to behave,” Borrell said. “The problem is behind me: there are 27 which are completely divided,” he added, referring to the EU’s member states.

Write to Laurence Norman at laurence.norman@wsj.com