How to create an OpenSearch (Vector Database) On Demand instance for Bedrock Knowledge Base

I am trying to create a Knowledge base to be used with AWS Bedrock for a very Proof of Concept solution. The knowledge base needs some form of Vector Database to run off of, which one of the options is "Vector engine for Amazon OpenSearch Serverless"

I already setup using the default serverless option and ended up incurring high billing costs, but the representative told me that there is an option when creating your OpenSearch Vector DB to run it in full On Demand mode rather than serverless. I have tried searching all across AWS for how to specifically do this in console and cannot find anything.

Could someone give instructions for how to setup an OpenSearch Vector engine in On Demand mode? Since I am doing testing I do not want to have a server that will be always running from the moment I start it up, as I will only need to run it for 1-2 hour periods per week as I am testing and trying to build out my solution. Thank you! And if