Chat and search read an index of your public records, kept in your Turso database. Chatbot > Index shows its state and holds the tools to rebuild and clean it.
What is indexed
| Kind | Text indexed | Linked to |
|---|---|---|
| Items | Every metadata field with its name (Title: …, Creator: …), the item type, the collection's title and the tags | The item's page |
| Files | Their own metadata, when they have more than their item's title, and their item's title | Their item's page |
| Collections | Their metadata | The collection's page |
| Simple pages | Title and text (without markup and shortcodes) | The page |
| Exhibit pages | Title, the text and captions of their blocks, and the exhibit's title | The exhibit page |
Only what visitors may see is indexed: public items and their files, public collections, published simple pages, pages of public exhibits. Simple pages and exhibit pages are indexed while their plugin is active.
Long texts are cut into chunks (see Chunk size in the settings); each chunk is embedded by OpenAI once and stored with its vector.
Keeping it current
- Saving a record queues it. One background job indexes the queue, so an import of thousands of items starts one job, not thousands. The Index tab and the top of every Chatbot page show how many records are waiting.
- Making an item private, unpublishing a page or making an exhibit private removes its chunks when it is indexed again (at once, in the background). Making an item public again indexes its files too.
- Deleting a record removes its chunks at once, even with automatic indexing off.
- With Index automatically off, saved records are not queued: re-index when you are done.
Re-indexing
Re-index now on the Index tab deletes the chunks of the kinds you tick and indexes every public record of those kinds again, in the background. Use it after setting Chatbot up, after changing the collections indexed or the chunk size, or after a large import with automatic indexing off. It embeds everything again: with OpenAI's prices, re-indexing a few thousand records costs cents with text-embedding-3-small and a little more with -large.
To re-index some items, select them in Items, click Edit (batch edit), and tick Chatbot: Re-index. They and their files are queued.
Test search
Test search embeds a query and shows the chunks the index returns, with their distance (smaller is closer), as visitors would get them. Chunks of records that are no longer public are counted as hidden.
Cleaning up
| Tool | What it does |
|---|---|
| Remove non-public chunks | Deletes the chunks of records that became private, unpublished or were deleted while the index was not told (a record changed with automatic indexing off, or directly in the database). Chat and search skip them anyway. No OpenAI calls. |
| Purge the index | Deletes every chunk and recreates the index: after changing Embedding dimensions, or when the Index tab reports a damaged index. Re-index afterwards. |