AWS glue job parameters error

Question

I am trying to use chromadb in AWS Glue, it's throwing out the below error,

RuntimeError: [91mYour system has an unsupported version of sqlite3. Chroma requires sqlite3 >= 3.35.0.

I tried to add to both chromadb and sqlite3 modules in job parameters, as chromadb==0.4.13,sqlite3==3.35.0. And then i get this error-

Some pip modules could not be installed. This may mean that you have requested a module that does not exist or produced an error: sqlite3==3.35.0,chromadb==0.4.13.
Any suggestions on how to resolve would be helpful.

Accepted Answer

Hi Tej,

To resolve the SQLite version issue, please follow the troubleshooting steps provided by Chroma in their official docs. [1]

```
Chroma requires SQLite > 3.35, if you encounter issues with having too low of a SQLite version please try the following.

1. Install the latest version of Python 3.10, sometimes lower versions of python are bundled with older versions of SQLite.
2. If you are on a Linux system, you can install pysqlite3-binary, pip install pysqlite3-binary and then override the default sqlite3 library before running Chroma with the steps here. Alternatively you can compile SQLite from scratch and replace the library in your python installation with the latest version as documented here.
3. If you are on Windows, you can manually download the latest version of SQLite from https://www.sqlite.org/download.html and replace the DLL in your python installation's DLLs folder with the latest version. You can find your python installation path by running os.path.dirname(sys.executable) in python.
4. If you are using a Debian based Docker container, older Debian versions do not have an up to date SQLite, please use bookworm or higher.
```

Most likely, it should be resolved by using "pysqlit3-binary" which should override the default sqlite3 library. [2]

References: \
[1] https://docs.trychroma.com/troubleshooting#sqlite \
[2] https://www.pythonanywhere.com/forums/topic/33348/

Please let me know if this resolves the issue.

Thanks, \
Atul

AWS glue job parameters error

Relevant content