Backport of backend/http: Refresh state earlier (align with other backends) into v1.14

[github-actions]

Backport

This PR is auto-generated from #38033 to be assessed for backporting due to the inclusion of the label 1.14-backport.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@radeksimko
This person should resolve the merge-conflict(s) by either:

• Manually completing the cherry picks into this branch
• Creating a new branch and manually cherry-picking all commits being backported

merge conflict error: POST https://api.github.com/repos/hashicorp/terraform/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

This updates the http backend's behaviour such that it refreshes state prior to returning the StateMgr. In practice this should not have any impact on the end user, perhaps just that errors will be raised a little bit earlier.

This change however enables us to now use that backend to test edge cases and prevent bugs such as #38027

Prior to this patch, the http backend is not actually impacted by the crash reported in #38027

Some backends (azure, gcs, kubernetes) are assumed to be impacted as they implement the exact same behaviour I'm trying to mimic here:

terraform/internal/backend/remote-state/azure/backend_state.go

Lines 107 to 110 in 193867f

	// Grab the value
	if err := stateMgr.RefreshState(); err != nil {
	return nil, diags.Append(err)
	}

terraform/internal/backend/remote-state/gcs/backend_state.go

Lines 108 to 111 in 193867f

	// Grab the value
	if err := st.RefreshState(); err != nil {
	return nil, diags.Append(err)
	}

terraform/internal/backend/remote-state/kubernetes/backend_state.go

Lines 94 to 97 in 193867f

	// Grab the value
	if err := stateMgr.RefreshState(); err != nil {
	return nil, diags.Append(err)
	}

Other backends implement the same logic conditionally (only refresh once a previously not-existing state was auto-created) but the bug likely impacts them in a similar way anyway:

terraform/internal/backend/remote-state/consul/backend_state.go

Lines 120 to 124 in 193867f

	// Grab the value
	if err := stateMgr.RefreshState(); err != nil {
	err = lockUnlock(err)
	return nil, diags.Append(err)
	}

terraform/internal/backend/remote-state/cos/backend_state.go

Lines 126 to 130 in 193867f

	// Grab the value
	if err := stateMgr.RefreshState(); err != nil {
	err = lockUnlock(err)
	return nil, diags.Append(err)
	}

terraform/internal/backend/remote-state/oci/backend_state.go

Lines 88 to 91 in 193867f

	if err := stateMgr.RefreshState(); err != nil {
	err = lockUnlock(err)
	return nil, diags.Append(err)
	}

terraform/internal/backend/remote-state/oss/backend_state.go

Lines 162 to 166 in 193867f

	// Grab the value
	if err := stateMgr.RefreshState(); err != nil {
	err = lockUnlock(err)
	return nil, diags.Append(err)
	}

terraform/internal/backend/remote-state/s3/backend_state.go

Lines 231 to 234 in 193867f

	if err := stateMgr.RefreshState(); err != nil {
	err = lockUnlock(err)
	return nil, diags.Append(err)
	}

---

I'm open to breaking away the last commit and reviewing it alongside #38028 - we cannot really merge this PR on its own without #38028 anyway but having it crash like this in isolation hopefully demonstrates the value.

---

Overview of commits

• 9880dfe

• 18ea052

• 61ee0ba

• 0d0cfb5

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[hashicorp-cla-app]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA.
If you have already a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[radeksimko]

Addressed in #38061

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.