New in Sugar 6.5 – Full Text Search thru elasticsearch

sugarcrmdevelopers —  July 3, 2012 — 18 Comments

If there’s two things that have defined the improved user experience in Sugar 6.5, it would be the refreshed navigation bar and the addition of full text search engine support.

We added this for the commercial versions of Sugar leveraging the elasticsearch FTS engine, which is a new component to platform stack for Sugar administrators. There’s been lots of questions around this, so I thought I would address them in this blog post.

Q. How do I install elasticsearch?

There’s a great guide for this on the elasticsearch site.

In additon, we have some RPMs we’ve built against CentOS 5.x on SugarForge; these are provided for reference only and not officially supported by us. There are 3 RPMs there, one for the elasticsearch itself and two others which provide additional functionality for elasticsearch. Bigdesk provides better charts and statistics and the head plugin provides a web front end for browsing the Elastic Search index.

Q. How do I enable using elasticsearch with Sugar?

There’s a full guide on how to do this in the Sugar 6.5 Adminstrator’s Guide. Do note that you will need to have the Scheduler inside Sugar setup and running in order to have the FTS work as expected.

Q. Any tips to configuring elasticsearch to work optimally?

Generally speaking, if you run your database server and web server on different machines you should probably do the same for elasticsearch. Here’s a few other tips that come from our Ops team:

In on-demand, we tweak /etc/sysconfig/elasticsearch to look like the following:

ES_PATH_LOG=/var/log/elasticsearch
ES_PATH_DATA=/var/lib/elasticsearch
ES_PATH_WORK=/tmp/elasticsearch
ES_PATH_CONF=/etc/elasticsearch
ES_CONFIG=/etc/elasticsearch/elasticsearch.yml
ES_INCLUDE=/usr/share/java/elasticsearch/bin/elasticsearch.in.sh
ES_MIN_MEM=3000M
ES_MAX_MEM=9600M
NFILES=65536

The most important line for us in this file is NFILES which allows elastic search to have a larger number of file handles available for use.  That number should be tweaked upwards if the user ever sees messages such as the following in the logs:

[2012-06-12 11:53:50,815][WARN ][gateway.local            ] [fts-test1] failed to write local gateway shards state to /var/lib/elasticsearch/fts-test/node/0/_state/shards-66102 java.io.FileNotFoundException: /var/lib/elasticsearch/fts-test/nodes/0/_state/shards-66102 (Too many open files)

Elastic search logs to /var/log/elasticsearch

In on-demand we configure elastic search in two nodes clusters.  We do this by making adjustments to /etc/elasticsearch/elasticsearch.yml

It is important that both nodes have the same cluster.name parameter in this file and that they have unique node.name parameters.

On both nodes, we set the following:

node.master: true
node.data: true

We set the number of shards as follows in our two node cluster:
index.number_of_shards: 2

Q: What happens if the elasticsearch server goes down or becomes unresponsive?

Starting in 6.5.1, this will be auto detected and any searches performed until the server full comes back online will fallback to the previous global search functionality. When the server is back up and running it will automatically switch back to searching via elasticsearch.

Let us know in the comments on any other questions you have around elasticsearch, and be sure to report any issues you are seeing with it to our bug tracker.

18 responses to New in Sugar 6.5 – Full Text Search thru elasticsearch

  1. 
    Stuart Gilberd July 3, 2012 at 2:59 pm

    Is there support for sophisticated searches such as using syntax to search for documents where a custom field has a certain value?  

  2. 

    Pro/Ent only?

  3. 
    Stewart Osborne August 16, 2013 at 8:47 am

    I realize this is a bit old, but I’m trying to understand why deleted beans show up in the fts_queue table after you schedule a full index for a module. They seem to get stuck in the table and hose things up.

  4. 
    Stewart Osborne August 29, 2013 at 8:09 pm

    John, I’m sorry to flame your post, but I run into this post a lot when I’m searching so I need to air some things that go unaddressed by SugarCRM. Ok, so, searching in SugarCRM is abysmal. First, let’s say you have 20 Chris Smiths at different accounts, and you also have accounts called Chris Smith in addition to the contacts. Index them using the FTS feature, and do a search for chris smith. You can’t 1) sort by module 2) see any popup information about details, etc. You have to click into each record (and keep track of which ones you’ve already clicked by divination because the account name isn’t available when lead or contact records are returned). It is an arduous process to find anything in a large SugarCRM database. Also, if Chris Smith has several different email addresses, forget about finding the second or third. Furthermore, if you’re looking for specific words in an email from Chris Smith, you’d best have some consulting money available, because searching through the texts of emails is not an intrinsic feature of SugarCRM 6.5.x. Another annoyance is if you delete a record, the $bean->mark_deleted() routine does not remove it from the FTS results so users will see a blank return record in the search results until you do a full index. Let’s also say you have a custom customer ID field you’ve added and added to the dictionary to be indexed by the FTS search. It will not be indexed. You won’t be able to search for records by customer ID either, unless you also copy that field into a standard field that is indexed by default with the FTS search. I can’t tell you how upset users are about this.

    If you’re reading this and have a moment, please vote up this bug: http://www.sugarcrm.com/support/bugs.html#issue_54529

    • 

      Thanks for your feedback. I’m passing along your comments to our PM team to review.

      • 
        Stewart Osborne August 29, 2013 at 9:47 pm

        Thanks, I appreciate it. I think SugarCRM is a great platform, but this has gotten less development effort than the 7.0 interface. Interface is great, but in the end, if you can’t find what you’re looking for…

        • 

          Hello Stewart. Thank you for the feedback. We have not yet fully utilized the power of FTS in Sugar apps. We are looking to enhance our FTS uptake in the next release we are planning as we speak. Once we have design ideas to share, I will reach out to you to validate the concepts.

          • 

            I just went through the beta search in 7.7. THANK YOU FOR LISTENING!!! The search is so amazingly better than it used to be. This is a much needed improvement. We’re so excited to roll it out.

  5. 

    Hi Guys, Is there some proper documentation on configuring elasticsearch in a production environment? With say 300 users?

  6. 

    Hi there,

    I have an issue in Sugar version 6.5.18 about general search, whenever i save an email under contacts with first letter capital it’s not available for global search but if i convert it to small letter it works. Is there anything i missed with the settings for elasticsearch?

    Thank you

Trackbacks and Pingbacks:

  1. After Upgradation to 6.5.2, the global search is coming still and the features of 6.2 - September 25, 2012

    […] […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s