Manticore search

Manticore Search is an Open Source database that was created in 2017 as a continuation of Sphinx Search engine (Started in 2001). It's a modern, fast, light-weight and full-featured database with outstanding full-text search capabilities. https://manticoresearch.com/

Manticore Search is very appreciated by its community.

It is a Unified Index option in Tiki. As of 2022-11-16, the results between the various engines are not identical in all cases. We wrote a script to compare the results of MariaDB/MySQL vs Elasticsearch vs Manticore Search, and review all the discrepancies. Also, different engines have different features. If Manticore doesn't work for you, you can revert to your previous engine.

Status:

• Experimental in Tiki 25.0
• Beta in Tiki 25.2
• Stable in Tiki 26+
• Intensely tested in Tiki 27LTS+

Performance

Manticore Search uses way less RAM than Elasticsearch. One reason is that that Manticore Search is in C++ while Elasticsearch is in Java.

https://db-benchmarks.com/ show that Manticore is faster than Elasticsearch. However, unexpectedly, as of 2024-08-21, the result of Manticore-Tiki is slower than Elasticsearch-Tiki and this will be worked on.

Installation

See instructions on how to install it server side:

• https://manticoresearch.com/install/
• https://wikisuite.org/Manticore-Search
• How to Install Tiki on Ubuntu GNU Linux

How to indicate to Tiki to use Manticore Search

Once Manticore is installed and running:

1. Pick Manticore as Unified search engine at tiki-admin.php?page=search
2. Rebuild index
   • Small sites, via GUI
   • Medium and large sites, via CLI php console.php index:rebuild --log
3. Enjoy!

Please see development information: Manticore Search and https://wikisuite.org/Manticore-Search

It has many features including:

• Faceted search
• Stored Search
• Federated Search

See also

• Manticore Buddy

Dev notes from Victor

In short:

1. Introducing a new setting in manticore search settings - you can enumerate the fields you want to be always indexed - e.g. important fields or fields that search too slowly with regex. Default here is title and contents - you will always want contents be in the full-text index for the whole site search.
2. Keep the search setting of the tracker fields as it works until now - work in combination with the preference to exclude non-searchable fields from the index.
3. When building the index, we try to fit all text fields as full-text but if this is more than 255, we prefer the preferenced ones and then start with the rest until we fill the limit. If we go over the limit, the rest are indexed as string columns and allow regex search.
4. Changed multivalue attributes to MVA native manticore implementation to spare some fulltext fields.

So, most of the sites shouldn't notice this change. Bigger sites will probably need some configuration to allow faster lookups on big trackers - ensure the searched fields end up in the preference.

Move dev notes from Victor

https://gitlab.com/tikiwiki/tiki/-/merge_requests/2735

Had to unify the way we gather and store stats as well as building it for reporting. Now each reporting place uses a single stats builder, so they should be in sync (those are: rebuild console command output, rebuild log file output, last stats displayed via module).

Due to the way we are indexing wiki pages and other wiki content, the content is parsed but there is no easy way to gather the fields used in that specific place, so I added a catch-all type of a rule: manticore query builder logs all field usages in queries (either fulltext or not) and stores these. Then, after rebuild finishes, we display the following additional stats:

indexed fulltext fields
text fields converted to strings (those are supposed to be indexed but the 256 was limit hit)
indexed fields that are not used in fulltext queries (note that this might not be a complete picture as we only take the wiki texts; you can possibly do other search queries even with API or customsearch templates residing in themes that are not taken in to account)
non-indexed fields that are used in queries

If you want a more sophisticated report of where each indexed/non-indexed field is used, then I think a console command is needed with proper table display or might be part of the admin UI?

Mac OS Notes

The only way I could get this to run locally on macOS (on Apple Silicon) was by using Docker, and i needed to add these config items, pasted here for reference)

Manticore Federation

We had a federation of two servers (each with Manticore) which were handling multiple sites. A third server was created and one of the sites was moved there.

Moving a site from a Manticore federation to a new server has several side effects which all need to be fixed. What I did for dev.tiki.org and the new dogfood6.evoludata.com server:

1. Mirror manticore config from dogfood5 server. Most importantly, make it listen on the external inet interface - namely, dev.tiki.org, so it is accessible from the outside.
2. Open ports 9306 and 9312 to all other hosts from the federation. dogfood servers use nftables, so I did this in /etc/nftables.conf and restarted the service.
3. Change search config in the new server to use the same hostname as manticore config - dev.tiki.org - this was already done.
4. Go to Admin->Tools->external wikis and update the federation index locations. We were using local index locations for doc and themes and I had to change this to remote for dev.tiki.org.
5. Go to external wikis of ALL other sites participating in the federation and update dev.tiki.org index to the new correct remote location.
6. Open any ports on the remote hosts to the new host. I sent an email to Oliver to do this for tiki.org server. Also opened the ports for dogfood5.evoludata.com server in /etc/nftables.conf

After all this is done, all federated indices need to be rebuilt to re-configure each other.

Running the Docker Image of Manticore

For some Linux distributions, such as OpenSUSE, there's no Manticore Search package. I (Volker Wysk, 2025-03-11) have managed to run it in the Docker container provided for Manticore instead.

Tiki appears to communicate, amongst other ways, over the file system with Manticore. So some preparations are necessary, so Manticore can access Tiki's "temp" directory. See below.

Use the following commands to setup the Manticore Docker container. Execute them as the root user.

1. Install Docker
   Under OpenSUSE, you can use these commands:
   
   Copy to clipboard

   zypper install docker docker-compose docker-compose-switch
   systemctl enable docker
   systemctl start docker

2. Add your ordinary Unix user to the "docker" group, so it can use Docker. (The USER variable is set to the ordinary user.)
   
   Copy to clipboard

   usermod -G docker -a $USER

3. Download the Manticore image
   
   Copy to clipboard

   docker pull manticoresearch/manticore

   
   The morphology data (lemmatizers and stemmer) are included in the image. They don't need to be installed in addition. (That's different from Ubuntu.)
4. Create a directory for the Manticore data. This is inside the ordinary file system, not inside the Docker container. It is needed so the data isn't lost accross reboots.
   
   Copy to clipboard

   sudo mkdir /var/lib/manticore

5. Transfer ownership of it to Manticore. The Manticore user and group, as used inside the Docker container, are both 999. You need the same two numbers in the host system.
   
   Copy to clipboard

   groupadd --gid 999 manticore
   useradd --uid 999 --gid 999 manticore
   chown manticore:manticore /var/lib/manticore/

6. Prepare the Tiki "temp" directory.
   This directory must be made accessible by Manticore, because there are things like the stopword lists inside, which Manticore needs to access. I've chosen to use the group ownership as the way to grant it access.
   
   Copy to clipboard

   chgrp manticore /srv/www/htdocs/tiki/temp

   
   The "temp" directory will be bind-mounted into the Docker container in the next step.
7. Start Manticore as a Docker container.
   Starting the Docker container with the --restart=... option makes it to be restarted when it exits or when the system is rebooted. This is the way Docker images are "installed".
   
   Copy to clipboard

   docker run -e EXTRA=1 --name manticore --restart=always  --detach -v /var/lib/manticore:/var/lib/manticore -v /srv/www/htdocs/tiki/temp:/srv/www/htdocs/tiki/temp -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308 manticoresearch/manticore

   
   The parts have the following meaning:
   -e EXTRA=1: This sets the environment variable "EXTRA". See here for what it does.
   --name manticore: This defines a handy name "manticore" for the container.
   --restart=always: This permanently installs the container.
   --detach: Makes the container run in the background.
   -v /var/lib/manticore:/var/lib/manticore: This is the bind mount of the Manticore data directory. See above.
   -v /srv/www/htdocs/tiki/temp:/srv/www/htdocs/tiki/temp: This is the bind mount of the Tiki "temp" directory. See above.
   -p 127.0.0.1:9306:9306 -p 127.0.0.1:9308:9308: This makes the docker image listen at the default ports. The "127.0.0.1:" part says that only connections from localhost are allowed. If you want to run Manticore in another machine, leave it out.
   manticoresearch/manticore: This is the docker image to run.
8. Rebuild the Tiki search index and enjoy.