Along Texas’ 1,200-mile-long border with Mexico, state troopers routinely arrest migrants for trespassing. Texas National Guard troops unspool razor wire along the banks of the Rio Grande. State game wardens patrol the river in fast-moving boats.

For more than two years, Gov. Greg Abbott has been testing the legal limits of what a state can do to enforce immigration law. The effort, known as Operation Lone Star, has been broadly popular in Texas, including among many Democrats, while its cost, already more than $4 billion, was expected to top $9 billion by the end of next year.

Even as the number of migrants has gone down in recent months, Mr. Abbott, a Republican, has pushed the boundaries further. He has overseen aggressive deterrence by state police officers at the border and mounted a brazen challenge to federal authority by placing a floating barrier in the middle of the Rio Grande. In the small border city of Eagle Pass, the state police bulldozed vegetation from a sandbar in the middle of the river last month to create a new security outpost.

The moves met their first significant challenge from the federal government, which sued the state of Texas on Monday, saying that the floating barrier in the river has “flouted federal law” and “risks damaging U.S. foreign policy.”

The newly aggressive tactics have tested the support that the governor’s immigration policies have often enjoyed in recent years in Texas. As recently as last month, 59 percent of Texans polled backed the law enforcement deployments and border spending, including 30 percent of Democrats.

But the stepped-up tactics employed in recent weeks, including reports of harsh treatment of migrants by Texas officers and injuries to people encountering the razor wire, have soured some of that support.

“It worked for the first year, year and a half,” said State Representative Eddie Morales, a conservative Democratic member of the Texas House who represents Eagle Pass and voted to authorize spending on Operation Lone Star back in 2021. “But I think he’s gone overboard,” he said of Mr. Abbott. “It’s time that somebody put a check on our state government.”

In recent years, residents of border communities and their elected leaders, mostly Hispanic and mostly Democrats, have complained about the strain put on local services by so many arriving migrants, whose numbers increased sharply during the Biden administration. They welcomed the influx of state money and law enforcement support.

But over its first two years, Operation Lone Star appeared to have had little impact on the program’s primary goal of deterring migrants from crossing illegally. The state has claimed credit for more than 381,000 apprehensions of people who had crossed the border illegally. Most of them were turned over to U.S. Border Patrol agents, who in many cases released them into the country to await immigration hearings, and the overall number of arrivals has mostly remained above the levels measured when the program began.

Mr. Abbott has stressed other metrics, including more than 29,000 criminal arrests and an estimated 418 million lethal doses of fentanyl recovered by state troopers on the border — mostly seized at legal crossing points.

The governor touted his border campaign during his 2022 re-election battle with Beto O’Rourke, a former Democratic congressman who had criticized Operation Lone Star, including Mr. Abbott’s orders to bus migrants to Democratic-led cities across the country. The governor won a third term by double digits.

“Had the state done nothing, I think the voters and the citizens would have demanded change,” David Carney, the top political adviser to Mr. Abbott’s re-election campaign, said. “This is really not a Republican-Democratic issue,” Mr. Carney added. “Hispanics support it. African Americans support it. This is not a cutting issue.”

Mr. Carney said the migrant busing program had been unfairly criticized as a political stunt but was, in fact, a robust effort that has so far moved over 25,000 migrants to cities where they wanted to go.

Recent polling has suggested a slight increase in the number of Texans, including a “sizable minority” of Democrats, who would support more punitive immigration measures, such as deporting undocumented migrants already in the United States, said James Henson, director of the Texas Politics Project at the University of Texas at Austin. Only about a quarter of those surveyed in the June poll thought the state was spending too much on border security.

In the spring, the Republican-dominated Texas Legislature authorized another $5 billion in spending on Operation Lone Star through next year.

Far fewer migrants have been arriving in recent weeks, since the ending in late May of a policy known as Title 42, under which migrants who illegally crossed were swiftly removed.

The Biden administration has credited its own policy changes — including more restrictive rules for seeking asylum — with the drop in illegal crossings, a theory that could soon be put to the test. On Tuesday, a federal judge blocked the new asylum policy in a suit brought by the American Civil Liberties Union.

But conservatives have credited Mr. Abbott with the lower numbers. Since late May, his administration has directed state troopers and National Guard members to enhance existing measures, which have been in place in various areas of high migrant traffic along the banks of the Rio Grande.

Officials have called this a “hold the line” operation, and it has been particularly pronounced in Eagle Pass, recently a main arrival point for migrants crossing without authorization and seeking asylum. The operation has included shouting at arriving migrants to return to Mexico, denying water to some despite searing heat and rolling out physical barriers such as concertina wire.

Local officials there said the state has also recently filled in a section of the river that separated the shore from a sandy island in an apparent attempt to intercept migrants who might use the island as a staging point.

A spokesman for the Texas Department of Public Safety disputed that soil had been moved into the river, saying a natural sandbar “has always been there but is now visible because we cleared all the vegetation” in order to prevent smuggling in that area.

The placement of the floating barrier — about 1,000 feet of large round buoys about two miles south of an international bridge in Eagle Pass — appeared to challenge international rules and a federal law regulating construction in navigable rivers.

In a letter on Monday to President Biden, Mr. Abbott indicated that he may invoke a section of the U.S. Constitution that expands the powers that states have during a foreign invasion — a legal tactic that conservatives have been pushing for years.

Almost from the start of Operation Lone Star, far right Republicans have criticized Mr. Abbott for not claiming state power under that clause to send migrants directly back to Mexico.

Mr. Abbott has invoked the invasion clause — he first did so last year — but has stopped short of deportations.

“He’s not doing the one thing that needs to be done to stem the flow of illegal migration,” said Mark Morgan, a visiting fellow at the Heritage Foundation and a former acting commissioner of U.S. Customs and Border Protection during the Trump administration.

Legal experts said that Mr. Abbott’s actions already have challenged the limits of state authority on immigration and may have crossed the boundaries in certain instances.

“Stringing up prison wire along the banks of the Rio Grande is not only blatantly illegal but also appallingly cruel,” said Jeffrey B. Abramson, emeritus professor of government and law at the University of Texas at Austin.

There have been some legal challenges to the use of state troopers to arrest migrants who are found trespassing on private ranch land. The program originally involved arrests only of men; a state judge found last year that it was discriminatory and could violate the Constitution.

But the federal government has not appeared eager to challenge Mr. Abbott in court over the primacy of federal immigration authority, perhaps, legal experts said, out of a concern that a direct confrontation could end with a strongly conservative U.S. Supreme Court siding with Mr. Abbott.

In the politically diverse, mostly Hispanic border city of McAllen, the mayor, Javier Villalobos, said that Operation Lone Star had been a success in his area of the Rio Grande Valley. “We don’t have the issues that we used to have before,” he said.

Mr. Villalobos, a registered Republican, said he hoped some of the jarring images of buoys and barbed wire would deter people from crossing the river. “I always say, we need immigrants, people who are going to be productive in this country,” he said. “But we need to do it the right way.”

Edgar Sandoval contributed reporting from McAllen, Texas.