In this article we will setup a Weaviate Cluster and load the same books data used in the article IOS Intro - Search with Benefits.

You can download the resources used in this blog here.

Quckstart​

How to create a Weaviate account and cluster

Running Python on your Mac​

How to run Python applications on a Mac.

External Builder (Xcode) for Python​

Using Xcode to run Python applications

Create a Cluster​

Follow account setup and cluster creation as explained in the Quickstart Tutorial and take note of the following.

1. REST Endpoint URL

2. Weaviate API Key

Create XCode Project​

Create an Xcode Project using External Builder as described in External Builder (Xcode) for Python.

Install Weaviate Client​

From the root directory of your Xcode Project, install the Weaviate Client.

pip install -U weaviate-client

Refer to the section Install Python Packages and Setup a Virtual Environment in the article External Builder (Xcode) for Python for guidance

OpenAI Embeddings Access​

Create an OpenAI API Key and note the number created.

https://www.howtogeek.com/885918/how-to-get-an-openai-api-key/

OpenAI is needed to translate text to vector embeddings for book data.

As of this article, it will cost less than 10 cents to use this one-time (create vector input to be loaded into the cluster) OpenAI feature.

Enter Run Arguments​

As described in the article External Builder (Xcode) for Python enter the Run Arguments.

The Cohere key can remain blank.

Support Files​

What is included: A python script main.py to define the cluster and load book data, and book data 7k-books-kaggle.csv.

These files were copied from Adam’s (of Weaviate) GitHub project

The python script was modified slightly for clarity.

https://github.com/weaviate/BookRecs

Copy these files to your Xcode project.

Book Data​

A CSV of approximately 7,000 books.

Python Script to Create and Load Cluster​

The main.py python script will do the following

Define the schema of a Books Custer Create vector embeddings of Book Data Load embeddings into the Books Cluster

Each step is logged in the console.

The longest step to run is the “Load Data” step which may take a couple of minutes.

Don’t forget to make sure that the virtual environment is activated before running.

source venv/bin/activate

main.py: Startup​

API Keys are specified.

main.py: Cluster Defined​

Defines the Weaviate Books cluster.

The text2vec-openai parameter defines a vectorizer to convert text into numerical vectors.

The specific OpenAI model used for embedding is ADA Version 2.

https://openai.com/index/new-and-improved-embedding-model

main.py: Read Book Data​

Read the file of Book Data.

Make sure to define your directory path.

main.py: Load Data​

Properties are mapped from the data file to the Weaviate cluster.

For each book record, text is converted to a vector and loaded into the Weaviate cluster.

main.py: Clean-up​

Close the book data file and end the script.

Stay connected​

Thank you so much for reading! If you would like to talk to us more about this topic, please connect with us:

• Community forum
• GitHub Don’t forget to give us a ⭐️!
• X (Twitter)
• LinkedIn

Convert your structured data to a Weaviate Vector Database and unlock powerful search features. A vector DB understands the meaning behind your data, enabling you to find similar items and make connections you might have missed with traditional searches.

In this article we will talk about some of these important features.

• This article includes a ready-to-use Swift app and all the code. Download it from the Git repo here.
• You can run the code in a Swift Playground, which is an interactive learning environment for Swift.
• By running the code in the Playground, you can experiment with Swift and try out different ideas.

What you need

1. OpenAI API Key

   • https://www.howtogeek.com/885918/how-to-get-an-openai-api-key/

2. Apple’s Playgrounds App

   • https://www.apple.com/sg/swift/playgrounds/

3. Either a Mac or iPad to run the Playgrounds App

Everything else is already included such as the “Search with Benefits” app, Swift Code and a Weaviate Vector Database of Books.

Search with Benefits

Searching structured data is limited to the exact matching of keywords to values within a schema. To access a set of records a language such as SQL needs to be coded by a developer. Each search requires unique code for each type of query.

With Weaviate only one statement is needed.

Vector search is done by inferring the meaning of what you are trying to find and reasoning what scenarios are similar.

Let’s compare Structured Search and Vector Search in more detail.

Structured Search

Using a conventional search on structured data you are limited to schema and logic constraints.

Exact matches on structured data is required.

In addition, for each type of search a developer needs to provide SQL code to account for each and every scenario.

Examples​

1. Find all books by a specific author:

   author = "J.R.R. Tolkien"

2. Find books with a title containing a keyword:

   title LIKE "%Pride and Prejudice%"

3. Find books with descriptions mentioning a specific topic:

   description LIKE "%artificial intelligence%"

4. Find books with titles starting with a specific letter:

   title LIKE "A%"

5. Find books with a specific number of words in the title:

   LENGTH(title) = 3

6. Find all books by an author and sort by title alphabetically:

   author = "Agatha Christie" ORDER BY title ASC

7. Find books with descriptions longer than a specific word count:

   LENGTH(description) > 500

8. Find books with a title or description containing a specific phrase:

   (title LIKE "%Lord of the Rings%" OR description LIKE "%Lord of the Rings%")

9. Find all books except those by a specific author:

   author != "Stephen King"

10. Find books that haven't been assigned a description yet:

    description IS NULL

11. Find books with titles shorter than 10 characters:

    LENGTH(title) < 10

12. Find all books with a specific ID:

    ID = 123

13. Find a random sample of 10 books:

    RANDOM() LIMIT 10

14. Find all books with descriptions containing at least two keywords (combine with OR):

    description LIKE "%machine learning%" OR description LIKE "%data science%"

15. Find all books with descriptions containing keywords but exclude books by author:

    (description LIKE "%machine learning%" OR description LIKE "%data science%") AND author != "Ian Goodfellow"

Vector Search

With Vector Search you state your meaning in a prompt. Using one reusable statement (varying only by the prompt entered) a search will explore a Vector Database and return a set of results.

Examples​

1. Find similar books to Pride and Prejudice by Jane Austen.
2. Find science fiction books similar to Dune by Frank Herbert.
3. Find fantasy novels with a coming-of-age theme.
4. Recommend dramas similar to Wizard of Oz.
5. Search for mysteries set in Western Europe.
6. Discover historical books similar to The Audacity of Hope: Thoughts on Reclaiming the American Dream.
7. Personalize search results based on a college aged woman from Boston majoring in fine arts who enjoys vacations in France.
8. Find books with a similar writing style to a JR Tolkien.
9. Search for light-hearted comedies similar to a Tom Hanks movie like Forrest Gump.
10. Recommend books with a similar emotional impact to Danielle Steel's The Butler.
11. Assist book clubs by finding thematically linked books to The Anxious Generation: How the Great Rewiring of Childhood Is Causing an Epidemic of Mental Illness.
12. Personalize children's book recommendations based on a kindergarten reading level.

Running Search with Benefits App

Open App​

Download the app and open it in Swift Playgrounds.

Enter OpenAI API Key​

Open the SearchWithBenefitsViewModel swift file and enter your key.

Run the App​

Open the ContentView swift file and navigate towards the bottom where there are some suggested prompts.

Copy one of the prompts onto the search field of the app.

Click on the magnifying glass icon to perform a search. Or, enter any search prompt you can think of!

Or, before doing the above - run the app by clicking on this button below:

About the Code​

Weaviate related code can be found in the SearchWithBenefitsViewModel swift file.

Weaviate API Connection​

urlString

The web address (endpoint) used to connect to the Weaviate API.

weaviateApiKey

Special code (token) that grants you access.

For this demonstration, the Weaviate DB was already set up for you.

Leave the predefined values as they are.

We'll cover how to set up your own vector database in detail in future articles.

Weaviate Query​

Search Query Example:

Input

"Search for mysteries set in Western Europe”

Output

This query retrieves information about up to 10 mystery novels set in Western Europe. It returns both the title and a brief description of each book.

Using Postman​

Stay connected​

Thank you so much for reading! If you would like to talk to us more about this topic, please connect with us:

• Community forum
• GitHub Don’t forget to give us a ⭐️!
• X (Twitter)
• LinkedIn

Xcode itself isn't built for directly developing and building Python projects. However, you can leverage external builders to integrate Python development into your Xcode workflow.

Using Xcode to run python scripts is an alternative to using a native to python playground like Jupyter while allowing you to use a development tool you are familiar with.

Set up requires using Xcode and the terminal command line.

External Builder​

The External Builder is created in Xcode by selecting File>New>Project followed by selecting the External Build System icon in the Other tab.

Define Build Tool Path​

The Build Tool parameter defines a path to the python build executable.

Enter a path.

The build executable is a Python interpreter that runs scripts and imports modules (pre-written Python code) from various libraries.

The path to the build executable can be found by entering which python on the terminal command line while in your project directory

This path points to an alias .

/opt/anaconda3/bin/python

The alias points to an executable file.

/opt/anaconda3/bin/python3.11

Refer to the article Running Python on your Mac, for more details, if this sounds confusing.

Install Python Packages and Setup a Virtual Environment​

See the article Running Python on your Mac for instructions.

Install Python packages into a venv directory within your project to create a virtual environment.

In this example numpy and pandas are the external packages have been loaded.

Remember: If you close your terminal window, you'll need to reactivate the virtual environment using the source venv/bin/activate command before working on your project again.

Edit Scheme​

Define a Run Script and Executable in your scheme.

After completing all of the steps that follow, the scheme will look something like this.

Run Script (Pre-action)​

Enter a run script to give your project access to the python packages implemented in the virtual environment.

source ~/.bash_profile
workon venv

The script is modified through the scheme Run > Pre-actions.

Executable (Run Info)

Enter the path to the executable used to run python scripts.

Have the path you already defined for the build tool handy.

In this example it was: /opt/anaconda3/bin/python3.11.

The executable parameter is modified through the scheme Run > Run.

1. Click on Executable list that is currently set to None.

2. Select the Other option.

3. Ignore the Choose the executable to launch directory presented.

4. Instead choose a directory by entering Shift + Command + G and paste the path in the prompt. Click and select the file in the Go to section and press enter.

   Your Run Info has now been defined!

5. Make sure the Debug executable is NOT marked as checked.

Entry Point (Run Arguments) Arguments Passed on Launch

Define the entry point python script.

It is the script to be launched.

In the Run > Arguments > Arguments Passed On Launch section

Press the + icon and add the python script file name. For this example enter: main.py.

Entry Point (Run Arguments) Environment Variables - Optional

Include Environment Variables as needed (if any) by the application.

In the example below, Weaviate and OpenAi parameters are required.

Note: Environment Variables are not required to run our example app.

Run Options (Working Directory)

Define the working directory in which the python script and other resources related to the application reside.

This is the directory in which the project resides.

Run > Options > Working Directory section

Check the Use custom working directory and click on the arrow to pop-up a directory for selection.

Run Python Script​

Press ▶️ to run the scheme.

Stay connected​

Thank you so much for reading! If you would like to talk to us more about this topic, please connect with us:

• Community forum
• GitHub Don’t forget to give us a ⭐️!
• X (Twitter)
• LinkedIn

Weaviate and other data science frameworks are making Python a powerful tool for data analysis on the Mac. Get ready to use them in your workflow.

Python Installed with Mac by Apple​

Your Mac actually comes with a pre-installed version of Python built-in. This is typically an older version to ensure compatibility with tasks or programs that rely on how Python functioned in earlier macOS versions.

This implementation of Python is maintained by Apple, only.

Python Installed by User​

Before diving into Python frameworks, you'll need to ensure you have a user installed version of Python installed on your Mac. This acts as the foundation for running these powerful tools.

Multiple Installation Options:

There are several ways to get Python up and running on your Mac:

1. Official Python Installer: Download the latest version directly from the Python website. This is a straightforward approach.

2. Anaconda: This popular distribution includes Python along with a vast collection of scientific computing and data science packages. This method simplifies installation and is recommended.

3. Homebrew: This package manager allows you to install various software on your Mac, including Python. It offers flexibility for managing multiple Python versions.

Virtual Environment​

Isolating Your Projects with Virtual Environments:

When working on Python projects, each one should have its own virtual environment activated within the project directory. This provides several benefits:

Dependency Management: Virtual environments keep package dependencies for each project separate. This prevents conflicts that might arise if different projects require different versions of the same package. Imagine your project01 using the numpy package for calculations, while another project, project02, might use a different library for a different purpose. Virtual environments ensure each project has the exact versions it needs without affecting others.

Organization: Virtual environments help keep your development environment organized. Packages installed for a specific project are contained within its own virtual environment, making it clear which packages belong to which project.

Example: Virtual Environment for project01​

Consider having a project01 directory that includes a volume calculator application. You would need to activate a virtual environment within this directory and install a numpy package specifically for this project. This ensures numpy is available for project01 calculations without affecting other projects on your system.

Creating a Virtual Environment for Your Project:​

Let's walk through setting up a virtual environment for your Python project, assuming you already have user installed version of Python in place. We'll use the project01 directory as an example:

Navigate to the project directory: Open your terminal and use the cd command to switch to the directory containing your project files. For instance, if project01 is on your desktop, you might type...

cd ~/Desktop/project01.

Create the virtual environment: Use the following command in your terminal, replacing the last venv parameter with your desired virtual environment name:

python3 -m venv venv

This creates a new directory named venv in your project directory. It contains all the necessary files for a separate Python environment.

Activate the virtual environment: Tell your terminal to use the newly created virtual environment.

source venv/bin/activate

Your terminal prompt will likely change to indicate you're now working within the virtual environment.

Install packages within the environment: Now you can install Python packages specifically for your project. For example, to install numpy:

pip install numpy

This installs numpy within your virtual environment's isolated space, ensuring it doesn't interfere with other Python projects on your system.

Your project is now ready! You can proceed with your Python development in project01, confident that the environment has the correct numpy version and any other packages you install specifically for this project.

Remember: If you close your terminal window, you'll need to reactivate the virtual environment using the source venv/bin/activate command before working on your project again.

Helpful Commands​

Checking Your Python Environment:

These commands help you verify your Python setup:

Version of Python:

This will display the version of Python currently in use by your terminal.

On my Mac, it currently shows Python 3.11.7.

python --version

Path to Python Executable:

This will show the location of the Python executable (via an alias) that your terminal is using.

which python

This path points to an alias rather than an executable, the actual path can be found by right clicking on the alias and selecting Show Original.

The output from these commands helps you understand which Python installation your terminal is currently set up to use. This can be important, especially when managing multiple Python versions.

The commands we just reviewed (checking Python version and path) are helpful when we explore using Xcode to run Python scripts in a future article. Understanding your Python environment is crucial for configuring Xcode to work seamlessly with Python.

Example​

Here is an example of creating a simple python script and going through the steps of running it on your Mac.

1. Create a Python script in your project directory (project01 in this example)

2. Open a Terminal console using the Project Directory by right clicking the project in Finder.

3. Create a Virtual Environment within your Project Directory

   python -m venv venv

4. Activate the Virtual Environment

   source venv/bin/activate

5. Install Python Package(s)

   Install package(s) as needed.

   pip install numpy

5. Download the calcVolume.py Script from here. Run script.

   python calcVolume.py

Stay connected​

Thank you so much for reading! If you would like to talk to us more about this topic, please connect with us:

• Community forum
• GitHub Don’t forget to give us a ⭐️!
• X (Twitter)
• LinkedIn

Let's unpack Weaviate's search functionalities.

• Vector (similarity)
• Keyword (exact match)
• Hybrid (mix of vector and keyword)

We will discuss various search methods, from vector similarity for similar searches to keyword-based lookups. We'll even explore how to combine these methods for even more precise results.

In addition, we will go over features like filters and aggregation to fine-tune your searches and get the most out of your Weaviate data.

Included is a Postman collection that can be run to help you understand each of these concepts. Download that from here.

What you need

1. OpenAI API Key

https://www.howtogeek.com/885918/how-to-get-an-openai-api-key/

2. Postman

Download the Mac version or run from the Postman website.

3. A Mac to run the Postman App

A Weaviate Vector Database of Books is already included to run your Postman queries on.

Search Features

Weaviate offers various search features.

Vector search finds similar items based on vector representations.

Best Match (BM25) search takes into account the frequency in which a keyword appears in a text.

Conditional Filters look for an exact match for a given property.

Hybrid search is a combination of vector and keyword searches.

There are miscellaneous search features that can be applied, too. Such as sorting records and tallying of a count according to a property value grouping.

A Postman query has been created for each of these scenarios for you to try out as shown in the outline on the right.

Vector Similarity Search​

Imagine you have a giant library with books described by numbers instead of words. A vector similarity search is like a special tool that helps you find books similar to the one you're holding by comparing their numbers. It's faster than reading every book description and works well for things like finding similar images or recommending products.

Let's say you're in the library and love a particular mystery novel. This book can be represented by a vector, a fancy way of saying it's a bunch of numbers that capture information about the book. These numbers might encode things like genre, character count, or how many times certain words appear.

Now, you want to find other similar mystery novels. The library (or the Weaviate platform in this case) uses the same kind of number code (vector) for all the books. The search tool then compares the vector of your favorite book to all the others. Books with similar vectors, meaning their numbers are close together, are likely similar mystery novels you might enjoy. This way, you can find new reads without sifting through every single book description.

https://docs.weaviate.io/weaviate/search/similarity

Best Match (BM25) Search​

When a user submits a BM25 query, Weaviate will look for objects that contain the search terms in the text properties of the objects. Then, it will rank the results based on how many times the search terms appear in the text properties of the objects.

https://docs.weaviate.io/weaviate/search/bm25

Conditional Exact Match Filter Search​

An exact match search filter consists of one or more conditions, which are combined using the And or Or operators. Each condition consists of a property path, an operator, and a value.

Another variation of this feature is applying a condition to a list of properties, rather than a single property. This is done through the ContainsAll and ContainsAny operators.

https://docs.weaviate.io/weaviate/api/graphql/filters

Hybrid Search​

A hybrid search combines two different search techniques: keyword-based search and semantic vector search.

https://docs.weaviate.io/weaviate/search/hybrid

These are the two major concepts.

• Keyword-based search is the traditional method used by most search engines. It works by matching keywords in your query to keywords in the documents or data being searched. This can be effective for finding exact matches,but it can also lead to irrelevant results if the query uses different words or phrases than the documents.
• Semantic search tries to understand the meaning of your query and the meaning of the documents or data being searched. It does this by using techniques like natural language processing and machine learning. This can help to find more relevant results, even if the query doesn't use the same words or phrases as the documents.

Hybrid search combines the strengths of both of these techniques. It can find exact matches using keyword search, and it can also find relevant results that use different words or phrases using semantic search. This can lead to more accurate and relevant search results overall.

Here are some of the advantages of hybrid search:

• More relevant results: Hybrid search can help to surface more relevant results, even if the query doesn't use the same words or phrases as the documents.
• Improved accuracy: By combining keyword-based and semantic search, hybrid search can provide more accurate results than either technique alone.
• Better user experience: Hybrid search can lead to a better user experience by providing more relevant and accurate results.

Hybrid search is becoming increasingly popular in a variety of applications, including:

• E-commerce search: Hybrid search can help shoppers find the products they're looking for, even if they don't know the exact names of the products.
• Enterprise search: Hybrid search can help employees find the information they need more quickly and easily.
• Customer service: Hybrid search can help customer service representatives find the answers to customer questions more quickly.

Miscellaneous​

Various scenarios and use cases.

• Sort
• Aggregate Count by Grouping

https://docs.weaviate.io/weaviate/search/aggregate

Tips on Working with Queries

When constructing queries there are some things to keep in mind.

Formatting queries can can result in frustrating errors when constructing them by hand.

Invisible escape characters and formatting gotchas can ruin what looks to be a syntactically correct query.

Query Advice​

1. Quotation marks (“) need to be proceeded by a backslash () when placed within a query parameter.eg. concepts:

   ["Search for mysteries set in Western Europe.\”]

2. Use an online formatter such as the GraphQL Formatter to make reading a query easier and to catch syntax issues. Remove any backslashes(“\”) beforehand.

3. Before running a query, line breaks need to be removed.

   The Remove Line Breaks online tool helps resolve this problem.

   Removing extraneous spaces are also recommended. Sometimes they include hidden format characters.

   eg. ”{Get {Book(limit: 10 nearText: {concepts: ["Search for mysteries set in Western Europe."]} ) {title description _additional {certainty distance}}}}"

4. Don’t create prompts using ONLY stopwords.

   Stopwords are common words like "the", "a", "an", "is", “in", “of”, etc. Search algorithms typically filter out stopwords because they don't provide much meaning in the context of a search.

   For example, searching for "the cat" would likely filter out "the" as a stopword and focus on finding results related to "cat".

Helpful Online Tools

There are a couple of free online formatting tools to help you construct as well as inspect queries.

GraphQL Formatter​

Inspects queries and formats them into a more readable format.

Also checks for GraphQL format completeness.

https://jsonformatter.org/graphql-formatter

Remove Line Breaks​

Prepares a query to be run by removing line breaks.

https://www.textfixer.com/tools/remove-line-breaks.php

Postman Examples​

Use the included Postman Workspace to run common Weaviate GraphQL queries against a live database.

Only the OpenAi needs to be entered, if using the database in the example.

If you are using you own cluster, both Weaviate parameters need to be entered, also.

Weaviate Parameters​

From the Weaviate Cloud Console dashboard find the associated parameters for the cluster.

WEAVIATE_API_KEY

The cluster API Key.

URL_TO_WEAVIATE

The Cluster REST endpoint, plus v1/graphql added to the endpoint.

Enter your Open AI Key​

This key is needed to access Open AI to make sense of your prompt.

Here is how you can set one up.

https://www.howtogeek.com/885918/how-to-get-an-openai-api-key/

Enter the key value in the variable section of Postman for your collection.

Queries​

A list of queries are listed to the right.

Each is fully documented and can be run as is after entering your OpenAI Key.

Query Documentation​

Each query includes an explanation of what the high level (numbered 1-5) search feature does as well as a formatted example of the query to read.

A link to the corresponding Weaviate API documentation from which this information is mostly borrowed from is included.

Postman Query​

Each query runs as is.

Don’t forget to enter your OpenAI Key for your collection beforehand!

They also work great as templates should you want to create a query for you own project, too.

Stay connected​

Thank you so much for reading! If you would like to talk to us more about this topic, please connect with us:

• Community forum
• GitHub Don’t forget to give us a ⭐️!
• X (Twitter)
• LinkedIn