Using access token returned from federated access gives unauthorized error in API Gateway

We are in the process of adding Single Sign On capability to an existing serverless app using Azure Entra ID as the identity provider for a Cognito driven application.

The SSO part seems to work using the Cognito Hosted UI. Where we're having problems is with authenticating the logged in user's access to API Gateway HTTP endpoints.

We're using user_pool authorization on API Gateway.

In the frontend code (VueJs) we're using Amplify to help with the management of the Auth and API calls. At the same time, we're migrating from Amplify v5 to v6 as the TypeScript support is much better which we thought would help with the addition of SSO.

We are retrieving the accessToken by doing:

const authSession = await fetchAuthSession();
const token = authSession.tokens?.acccessToken?.toString() || null;

We have also tried using the idToken

const authSession = await fetchAuthSession();
const token = authSession.tokens?.idToken?.toString() || null;

However, when we provide either of these as a Bearer token via the Authorization header on the API request we get Unauthorized

I can see that our PreTokenGenerationTriggerHandler lambda fires to set the custom claims as expected, I also see that the idToken available in the frontend includes these claims so that side of things is working properly. It's just that, when using this token (via the toString() method), it is rejected as Unauthorized.

What is it that we're doing wrong?