Starting with the title question, US States are bound by the federal constitution, which explicitly denies certain powers to the States, found mostly in Article I Section 10. The first clause even starts with foreign policy:

No State shall enter into any Treaty, Alliance, or Confederation; grant Letters of Marque and Reprisal; coin Money; emit Bills of Credit; make any Thing but gold and silver Coin a Tender in Payment of Debts; pass any Bill of Attainder, ex post facto Law, or Law impairing the Obligation of Contracts, or grant any Title of Nobility.

In this context, the terms “treaty, alliance, or confederation” are understood to mean some organization which would compete with the union that is the United States of America. That is to say, a US State cannot join the United Kingdom as a fifth country, for example. Whereas agreements between states – the normal meaning of “treaty” – is controlled by the third clause, which refers to such agreements as “compacts”.

No State shall, without the Consent of Congress, lay any Duty of Tonnage, keep Troops, or Ships of War in time of Peace, enter into any Agreement or Compact with another State, or with a foreign Power, or engage in War, unless actually invaded, or in such imminent Danger as will not admit of delay.

Compacts are only allowed if the US Congress also approves. This is what allows the western US States, the federal government, and Mexico to all agree on how to (badly) divide the water of the Colorado River.

So if foreign policy is meant to include diplomatic relationships, military exercises, setting tariffs, and things like that, then no, the US States are severely constrained in doing foreign policy. The diplomatic relations part is doable, where state-elected officials can go to foreign countries to advocate for trade and tourism. But those officials must not violate the federal Logan Act, which prohibits mediating an active dispute involving the USA, since that’s the US Secretary of State’s job. For example, it would be unlawful if a US State governor tried to mediate a prisoner exchange with a country that the USA has engaged the military against.

For your other question about what US States are, the answer to the question changed significantly in the 1860s. During that decade, the federal constitution gained three amendments, with the 14th Amendment being the most significantly for the notion of statehood. That Amendment’s Equal Protection and Due Process Clauses gave life to the notion of “incorporation”, which is that the US Constitution’s limits on the federal government also applied to the several States.

Before the 1860s, US States were indeed closer to countries in a trading, monetary, and foreign policy alliance. Some US States even had official religions, since the First Amendment’s prohibition on endorsing religion only applied to the federal government. But post 1860s, it was firmly established that the federal government isn’t just some economic committee, but an actual representative body, one whose laws will trounce state laws.

The best example I can point to is how broad the federal government exercises its “interstate commerce” powers. Basically, if something has anything remotely to do with crossing a state border, the feds can write laws on that topic. That was extremely rare pre 1860s, and now it’s basically the norm. The postal service is one such activity which is explicitly and wholly a federal matter, written into the initial Constitution. But now, airspace, telecoms, and railroads are all matters which the federal government asserts its authority via the “interstate commerce” powers, and if US States were countries, they might object to the feds. But they’re not countries, so they don’t wield that power.

Constitutionally, it is spelled out in Article I, Section 10, Clause 1:

No State shall enter into any Treaty, Alliance, or Confederation; grant Letters of Marque and Reprisal; coin Money; emit Bills of Credit; make any Thing but gold and silver Coin a Tender in Payment of Debts; pass any Bill of Attainder, ex post facto Law, or Law impairing the Obligation of Contracts, or grant any Title of Nobility.

So foreign policy is principally a federal government domain, established by cases heard by the Supreme Court. https://constitution.congress.gov/browse/essay/artI-S10-C1-1/ALDE_00001097/

There is a somewhat gray zone where neighbors can talk to each other on the state level, e.g. Maine and Quebec. But they will find themselves restricted by what the governments one step up have decided. I think certain states also are on friendly terms with other nations, probably to deepen economic ties. But that’s more on the level of a city friendship than actual foreign policy.