An Introduction to Semantic Search

If you use any type of database to help run your business, you already know about a traditional database search; you type in a few key words, hit enter, and get some results. Your results will typically include records that match the keywords or word fragments that you entered. This works great if (for example) you’re looking for a person named Charlie; you’d likely enter ‘charlie’ into a search field and get a set of records where the person’s name includes Charlie. You might get fancy and use some type of wildcard search like ‘charl*’ if you wanted to get all the Charlie, Charles, (and Charlene) records. With a traditional search like this, you get what you ask for, and this can be an appropriate and powerful way to mine your database for information. With FileMaker, we get a great “Find Mode”, and even beginner users can perform powerful searches with this simple tool. And users that have truly mastered Find Mode can do seemingly magical things with their database.

Whether or not you’re already a wizard with Find Mode, semantic search gives you a whole new way to get meaningful information out of your database. Semantic search uses artificial intelligence and natural language processing to understand the context and intent behind a user’s search query, rather than just focusing on literal keywords. This type of search delivers more accurate and relevant results by understanding the relationships between different words and phrases, relating query terms to concepts, and even considering user behaviors and trends. Essentially, semantic search understands what the user is really looking for, almost as if you were having a conversation with the search engine itself.

A Semantic Search Example

If this sounds a little abstract, let’s look at a specific example. Let’s say you’ve got a database of Airbnb reviews and you’re interested in finding a positively reviewed place in the Boston area that’s pet friendly. With a traditional search, you’d likely use structured fields like the city and state of the listing, the average customer rating, and perhaps a checkbox for “pet friendly”. You’d probably be able to get what you’re looking for pretty quickly, but it would be hard to go beyond the search options that the database already has in place.

Imagine you have severe allergies and were looking for a place where fellow allergy sufferers had a great experience? If you were able to talk to your database, you might say something like ”dog fur makes me sneeze and get watery eyes. Is there a place in the Boston area where I’d be comfortable?” This is where semantic search shines – you want listings that match the intent of your query, not necessarily the specific words used. We’ll be doing what’s called a vector search to find records where our query’s semantic meaning matches the semantic meaning of the content in the customer review. Wow!

Take this example database of ~10k hotel reviews. These reviews include structured data like the hotel name and address, the review’s numeric rating, as well as unstructured data, like the review itself. With structured data, like city and state, a FileMaker find is the way to go; we can search on `San Francisco` in the city field and `CA` in the state field and get a found set of records that’s predictably accurate. It’s trivial to perform a Find on these fields using Find Mode or a Quick Find, and that can be quick and efficient (truly, it’s one of FileMaker’s best features!). 

But with unstructured text like the narrative comments we have for these reviews, that type of find may not get us what we need. Sure, we can probably find places that have a swimming pool by performing a find for ‘pool’ in the review field; but what we really want are places that have awesome water attractions for the kids. That’s where a semantic search is really what we’re looking for.

Now look at this next example where we’re searching in plain english and getting a set of records that match the intent of what we’re looking for. Note that the prompt is just written in natural language, not a set of key words or a bunch of complicated find requests. FileMaker’s new features make it just as easy to perform this kind of natural language search on the unstructured Review text as it is to find a city or state in their structured fields.

Why is this so Cool?

It’s no surprise that we are collecting data faster than we can get meaning from it. Much of this data is unstructured and often requires human intervention to pluck out the important bits. Think about our Airbnb review database; you have all these customer reviews with all sorts of great information locked away in this unstructured data, and that data is getting generated at an increasingly faster pace. We can certainly hire an army of workers (Amazon Mechanical Turk ?) to parse through these reviews and (for example) tag the reviews with a set of pre-defined key words or concepts to facilitate more accurate traditional searches. But that can be an overwhelming, time consuming, (and perhaps expensive) task that’s getting harder and harder to accomplish given the rate of new reviews arriving every day. You’ve probably got a similar situation with your CRM; maybe you’re logging all your email communication with customers (and if you’re not, you should be – hit me up, I have ideas!) and generating wonderful and lengthy narratives with your customers and potential customers. But what do you do with all this information?

Semantic search just might be the hook that you need to extract insight from this type of unstructured data. And with FileMaker’s new semantic search tools, it’s easy to find the right information using just a natural language prompt.

Beyond Database Fields

With a little work, you can go beyond the unstructured data that’s just in your database’s text fields. This technique can be adapted to search content from a library of PDF manuals, conversations from video or phone calls, or any other kind of unstructured data that your business generates every day. Using AI tools like Chat GPT can help you open up brand new ways to explore and leverage your existing data. What ideas do you have? Give us a shout and let us know!