★ wanayoo — archive 1999 https://developers.google.com/appengine/docs/python/search/overviewNouvelle recherche | Portail wanayoo

Google App Engine

Search Overview (Python)

Experimental!

Search is an experimental, innovative, and rapidly changing new feature for Google App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Search. We will inform the community when this feature is no longer experimental.
 


The Search API allows your application to perform Google-like searches over structured data. You can search across several different types of data (plain text, HTML, atom, numbers, dates, and geographic locations). Searches return a sorted list of matching text. You can customize the sorting and presentation of results.

This overview explains how to:

  • Build documents structured with fields to describe the data you wish to search
  • Create an index of documents you wish to search
  • Construct queries to search the index
  • Build search requests to run queries against the documents in your application
  • Score results and customize their presentation to the user

It has the following sections:

Conceptual Overview

The Search API provides a simple model for indexing and searching data, and organizing and presenting search results. Search can query any data, as long as that data is described by a document. Documents consist of any number of fields, and the fields can contain a variety of different content types described by Field classes (plain text, HTML, atom, numbers, dates, and locations).

After structuring searchable data into document fields, you can structure documents into an index using the Index class. This class contains functions to add, remove, or list documents in an index, and also to search an index with a query. Search queries are based on a comprehensive query language that supports numeric and text expressions. The API also allows you to specify query options, which provide fine control over the results returned in a search. You can specify how many results to return, whether to return result snippets, how to rank and order results, and more.

Matching search results are returned to the application in a SearchResults object. This object includes the number of results found, the actual results returned, and an optional cursor object allowing you to page through results. You can also control the presentation of search results using a number of Field expressions and snippeted fields. These tools allow you to compute customized content snippets for any result returned by a search. This object also provides a starting cursor position or offset allowing you to present multiple pages of results.

Working with Documents

Search uses the Document class to build a document object describing a specific type of data, such as greetings added to a guestbook or metadata attached to an image. Documents contain fields that describe a particular type of data (text, HTML, atom, number, date, or location). The data type is designated by a Field class specified when structuring the document. Each document is uniquely identifiable by a document id (doc_id) assigned when you create the document.

To make your data searchable, you first create documents to describe it, then add those documents to the index. You can remove or list the index contents programmatically. You can also re-index existing documents, by adding a document with the same document ID of an existing document.

Describing Data with Documents

In order to make your data searchable, you need to describe it using a Document object.

At a minimum, each document has a document identifier and a list of fields.

Document Identifiers

The document identifier (doc_id) is used to identify a document in searches. You do not need to specify the document identifier explicitly. However, specifying your own identifier can be useful for implementing search with other storage mechanisms, such as Blobstore or Google Storage. In the case of Blobstore, for example, you can set the document identifier to the BlobKey in order to associate that document with a specific blob. The ID must be visible, printable ASCII, must not begin with "!", and cannot contain spaces.

Other situations may also call for you to create your own document identifier. For example, retailers often have a product ID code for each of their products. This retailer could create a document for each product, explicitly setting the product ID to be the doc_id. This approach would allow the retailer to to easily update the index as data about products changes, simply by uploading a new document with the same ID.

Document Fields

Each document contains fields that will contain a particular type of content. The content type is identified by one of the following concrete Field classes

  • TextField : plain text
  • HtmlField : HTML formatted text
  • AtomField : a string which is treated as a single token
  • NumberField : a numeric which supports int, long and float values
  • DateField : a date with no time component
  • GeoField : a locale based on latitude and longitude

Each field has a name, which allows you to reference it in search queries, and a value that indicates the type of content in the field.

The following code sample builds a document that might be used to index user comments in a bulletin board. The entry includes the user's nickname and comment, and a timestamp provided from the Python datetime module:

from datetime import datetime
from google.appengine.api import search

def CreateDocument(nickname, content):
    return search.Document(
        fields=[search.TextField(name='author', value=nickname),
                search.HtmlField(name='comment', value=content),
                search.DateField(name='date', value=datetime.now().date())])

For the full context of this code sample, check out the full application available on the Google App Engine samples page.

Once you have created documents describing your data, the next step is to add the documents to the index, described in the following section.

Adding Documents to an Index

Before you can perform queries on a document, you need to index it to make the data it describes searchable. You add documents to an index by calling the add() method on an Index object.

The following code sample creates an index called comment and adds a document to it:

from google.appengine.api import search
...
_INDEX_NAME="comment"

try:
    search.Index(name=_INDEX_NAME).add(document)
except search.Error:
    logging.exception('Add failed')
...

Consistency

An Index object requires a name and a consistency mode, either PER_DOCUMENT_CONSISTENT or GLOBALLY_CONSISTENT . You set the consistency mode when you get the index:

index = search.Index(name='index-name',
                     consistency=search.Index.PER_DOCUMENT_CONSISTENT)

Once set, that consistency mode is used from that point forward. The default is PER_DOCUMENT_CONSISTENT.

The following table describes the differences between these two consistency modes:

  Per-document consistency Global consistency
Throughput of indexed documents Provides relatively high throughput of indexed documents, enabling faster searches. Provides relatively low throughput of indexed documents (around 1 write per second), causing slower searches.
Visibility of recent changes Changes to any document in the index may not be applied in the correct chronological order. Changes to any document stored in the index are applied in the correct chronological order.
Freshness of results Results may not contain all changes committed before the search request. Results contain all changes committed before the search request.
Sample use case Documents that amalgamate information from multiple sources, where no single user is aware of all collected data. Documents that organize personal user information, to reflect all changes known to the user in any search results.

Text Parsing

The Search API parses content and special characters stored in your indexes according to a specific algorithm. This algorithm identifies sequences of text and parses them based on the characters present. Strings of letters interpreted as an acronym (such as a b c, a.b.c, or a-b-c) are interpreted without spaces or punctuation. Other special characters are interpreted based on the context in which they appear.

You can also use the ~ search operator to search for comment variants and synonyms of a term.

Removing Documents from an Index

You can remove documents in an index by specifying the doc_id of one or more documents you wish to remove to the Index.remove() method. To remove a collection of documents, specify the ids_only argument to the Index.list_documents() method . When you use this argument, the API returns document objects populated only with the document ID. You can then delete the documents by passing those Document objects to the remove() method:

from google.appengine.api import search
...
def deleteAllInIndex(index_name):
    """Delete all the docs in the given index."""
    doc_index = search.Index(name=index_name)

    while True:
        # Get a list of documents populating only the doc_id field and extract the ids.
        document_ids = [document.doc_id
                        for document in doc_index.list_documents(ids_only=True)]
        if not document_ids:
            break
        # Remove the documents for the given ids from the Index.
        doc_index.remove(document_ids)

Viewing Index Schemas

Each index has an associated schema used to interpret queries on that index. This schema is created dynamically from the field names and types present in all of the documents in the index. Viewing an index schema allows you to see the structure of the data in that index so you can determine the structured queries you can perform on the index.

There is currently a limit of 1000 named fields for each a given index schema.

There is currently no way to remove fields from a schema, nor to delete a schema, but you can view the schemas for your indexes like this:

from google.appengine.api import search
...
for index in search.list_indexes(fetch_schema=True):
    logging.info("index %s", index.name)
    logging.info("schema: %s", index.schema)

A simple schema for an index might look like this:

{'comment': ['TEXT'], 'date': ['DATE'], 'author': ['TEXT'], 'count': ['NUMBER']}

Each key in this dict is the name of a document field that has been added to the index. The key's value is a list of the field types used with that field name. If you have used the used the same field name with different field types, (for example, in different documents), the schema will list more than one field type for a field name, like this:

{'field1': ['TEXT', 'NUMBER']}

Building Queries

The Search API provides a number of query operators that you can use to query data stored in a document's fields. You can filter on a field's values using a field operator value expression such as:

created >= 2011-1-1

The field identifies the name of a property defined as part of a document. The value is a word, string, number, or other text that restricts the query on the specified field.

Search supports all space-delimited languages. Search supports some languages not segmented by spaces (specifically, Chinese, Japanese, Korean, and Thai). For these languages, Search segments the text automatically.

Simple Queries

The simplest type of query consists only of a word, which the API matches only against a document's text, HTML, or atom fields. Words are not case sensitive. In the case of atom fields, the word must match the entire field value.

For example, to find all documents with the word "rose" (regardless of case) in any string or text field in the document, use a query like the following:

rose

This simple query matches against any supported string or text field in the document. If your documents are "Comments" as defined in the Creating Documents section, the query matches if the word "rose" appears in the author or body fields. If the schema defines additional text or HTML fields, such as a subject or email, "rose" also matches the contents of those fields.

To match a phrase, surround the query in quotes as follows:

"any other name"

To search for plural variants of an exact query, use the ~ operator:

~"car"  # searches for "car" and "cars"

Note: Simple query strings match only against text, HTML, or Atom fields. The next section describes how to query against specific fields.

Queries on Fields

To build queries that reference specific fields, use both field and value in your query, separated by a colon:

field:value
field:"value as a string"

This syntax allows you to reference any supported field by name, assuming the field has been defined in the document. For example, to search for the word "rose" only within the author field of a comment document as defined in Creating Documents, use the following query:

author:rose

To search the body field for the phrase "any other name", use the following query:

body:"any other name"

To match against multiple fields at the same time, list a series of field:value pairs together with a space between them as follows:

author:"Rose Jones" body:rose

Boolean Operators and Grouping

The query language supports some Boolean operators as well as parentheses for grouping parts of the query together. The supported Boolean operators are AND, OR, and NOT. Always use uppercase for Boolean operators. Lowercase words are treated as part of the field or value portions of the query.

By default, when you create queries that match multiple fields at the same time, each value is combined with a Boolean AND. For the query as a whole to match, all the specified values must match.

You can also explicitly specify this by using the AND Boolean operator. The following two queries are equivalent:

author:rose body:"any other name"
author:rose AND body:"any other name"

Use the OR operator if you only want to know if any of the two values matches. You can use more than one OR in a query. For example:

author:(bob OR ((rose OR tom) AND jones))

This example matches any document whose author field contains the set of words in no particular order or vicinity ({rose, jones}, {tom, jones}, or {bob}).

The following example of the Boolean NOT operator matches any document whose author field contains "rose" but whose body field does not contain the word "filligree":

author:rose NOT body:filigree

Use parentheses to create more complex queries combining supported Boolean operators.

For example:

(author:Thomas OR author:Jones) AND (NOT body:rose)

This example matches documents with author "Thomas" or "Jones", but whose body field does not contain the word "rose".

Numeric Operators

Numeric operators only match against numeric and date fields. Query values containing a numeric operator must be quoted.

Search supports the following numeric operators:

  • :
  • =
  • >
  • >=
  • <
  • <=

For example, assuming your documents contain a votes field, to find all documents with more than ten votes:

votes > 10

For "not equal to", use the Boolean NOT with a numeric field name such as length. For example:

NOT length = 15

This example returns documents whose length is not 15.

You can combine numeric operators with text and Boolean operators. For example:

author:"Rose Jones" length > 15

This query matches comments whose length field is longer than 15 characters in length and whose author field is "Rose Jones".

You can also use numeric operators to query for dates. For example:

published > 2011-1-1

This query searches for documents published after January 1, 2011.

Expressions

In addition to the search queries described above, the API also supports query expressions that you can use in the computation of search results. You can use these expressions with the expression parameter of both the FieldExpression and SortExpression classes, as well as some search queries. The available expressions are:

Operator Type Description Example
geopoint geographic Degrees of latitude and longitude.

To return the map location of the specified coordinates, use:
geopoint(-31.3, 151.4)
distance geographic The distance between two locations in meters. To determine the distance between the specified geopoint and the value of the location field, use:
distance(geopoint(23, 134), location)

May be combined with numeric operators. For example, this query returns all matching locations within 10 meters of the geopoint:

distance(geopoint(23, 134), location) < 10
< binary, numeric Matches if the field contains a number less than the value specified in the query.

To return documents whose price field value is less than 100:
price < 100

+ binary, numeric Adds the provided numeric values. To add the value of a document's price and tax fields, use:
price + tax
– binary, numeric Subtracts the second numeric value from the first. To subtract the document's predefined _score attribute from the value of the rank field, use:
rank – _score
* binary, numeric Multiplies the arguments. To multiply the document's predefined _score attribute by the value of the rank field, use:
_score * rating
/ binary, numeric Divides the first value by the second. To divide the _score by the rank, use:
_score / rating
max N-ary

Returns the larger of the two arguments.

max(recommended_retail_price, discount_price)
min N-ary

Returns the smaller of the two arguments.

max(recommended_retail_price, discount_price)
snippet string

Returns a snippet for the given query. Returned snippets are in HTML, with the query terms in bold. This operator is constructed as follows:
snippet("query", body, [max_chars_per_snippet], [max_num_snippets], [separator]). The arguments are defined as follows:

  • query: The search query. Must be wrapped in quotation marks.
  • body: The name of the field.
  • [max_chars_per_snippet]: The maximum number of characters to return in a snippet. Defaults to 160 characters. Must be used in conjunction with [max_num_snippets].
  • [max_num_snippets]: The maximum number of snippets to return. Defaults to 3 snippets. Must be used in conjunction with [max_chars_per_snippet].
  • [separator]: markup to separate each snippet. Defaults to \n.
To query for rose in the field author, returning a maximum of 4 snippets that are each 200 characters, use:
snippet("rose", author, 200, 4)

Similarly, the following sample creates snippet for the phrase "few good" on a description field. snippet("price < 10 \"few good\"", description)
_score Predefined The score of the document returned in results, based on how well that document matches the query. To multiply the _score by the value of the rank field, use:
_score * rank
count Unary Returns the total number of attribute values for this document. count(size)

Constructing Search Requests

Constructing the query is only one part of performing a search. Once you have the query, you need to pass it to the API either as a simple string or as part of a Query object.

Queries passed as a string match all document fields and allow you to specify other details, such as the number of results to return, the point from which to continue the search (using a cursor), and even specific text fields to return in results.

Queries passed to the Index.search() method must follow a specific query language. Simple queries can simply pass a query string, but you can issue more complex queries using the Query and QueryOptions classes.

This section describes how to construct queries and use them to search the content of documents.

Passing Queries in a String

The simplest means of executing a query is to pass it to the Index.search() method, taking the query as an argument to the method and returning results:

from google.appengine.api import search
...
try:
    results = search.Index(name=_INDEX_NAME).search(query_string)
    for scored_document in results:
        # process scored_document
except search.Error:
    logging.exception('Search failed')

When you execute a search, the API returns a SearchResults object representing the results of the search request. This object describes whether the query succeeded, providing a list of matching documents ranked based on their degree of match. You can choose to include a cursor in results, allowing you to continue the search from the end of the current result set.

Passing Queries in a Query Object

Passing a query in a Query object allows you to define more complex search requests, including details such as:

  • A limit on the number of results to return
  • An offset defining a number of matched documents to skip when returning results
  • Specific scoring criteria for search results based on the accuracy of the matches
  • A cursor to return with results that allows you to get the next set of results
  • Sort criteria for results
  • Names of specific fields to return from matched documents
  • A Field Expressions object allowing you to display snippets of a document's field instead of their full contents.

When constructing searches using the Query class, you first construct the QueryOptions:

# Set query options
options = search.QueryOptions(
    limit=limit,  # the number of results to return
    cursor=cursor,
    sort_options=sort,
    returned_fields=['author', 'subject', 'summary'],
    snippeted_fields=['content'])

Then pass this object as the options argument to the Query class:

query = search.Query(query_string=query_string, options=options)

As was the case with simple searches, the API returns a SearchResults object representing the results of the search request.

The following code sample demonstrates a full search request, starting the search from the cursor location of the previous set of results, returning another single cursor for the response, sorting by "subject" in descending order, and returning the author, subject, and summary fields with snippeted field content.

from google.appengine.api import search
...
def FindDocuments(query_string, limit, cursor):
    try:
        subject_desc = search.SortExpression(
            expression='subject',
            direction=search.SortExpression.DESCENDING,
            default_value='')

        # Sort up to 1000 matching results by subject in descending order
        sort = search.SortOptions(expressions=[subject_desc], limit=1000)

        # Set query options
        options = search.QueryOptions(
            limit=limit,  # the number of results to return
            cursor=cursor,
            sort_options=sort,
            returned_fields=['author', 'subject', 'summary'],
            snippeted_fields=['content'])

        query = search.Query(query_string=query_string, options=options)

        index = search.Index(name=_INDEX_NAME)

        # Execute the query
        return index.search(query)
    except search.Error:
        logging.exception('Search failed')
    return None

Then call the above function to get the cursor for the next set of results:

results = FindDocuments(query_string, 20, search.Cursor())
if results:
    for scored_document in results:
        # process scored_document

next_cursor = results.cursor

You can then specify a query using FindDocuments:

next_cursor = results.cursor

next_cursor_urlsafe = next_cursor.web_safe_string
# save next_cursor_urlsafe
...
# restore next_cursor_urlsafe

results = FindDocuments(query_string, 20,
                        search.Cursor(web_safe_string=next_cursor_urlsafe))

Performing Location-Based Searches

You can use the Search API to include location search in your application. Locations to be searched must be stored within document fields in your application, while the browser typically provides the user's point of origin and search radius. With this information, the API can search for locations within the user's specified radius, and return results sorted by distance from that origin.

In order to perform location-based searches, your app first needs to store locations within document fields, as described in the next section.

Constructing GeoPoint Fields

To build locations into documents, first construct the GeoPoint object and then set that object as the value of the of the GeoField in your document:

from google.appengine.api import search
...
# Construct the GeoPoint class
geopoint = search.GeoPoint(latitude, longitude)

fields = [search.TextField(name='name', value=store_name),
          search.TextField(name='address', value=store_address),
          # Construct a GeoField passing geopoint as the value of that field
          search.GeoField(name='store_location', value=geopoint)
         ]

After constructing and indexing these documents, you can run search queries against them, as described in the following section.

Searching for Locations within a Specified Distance

Once you have stored a set of locations in your application, you can allow users to search for locations within a specified radius from a point of origin (for example, 1500 meters from 345 Spear Street in San Francisco). The user typically provides this information via the browser.

Locations are expressed in latitude and longitude coordinates, and distances are expressed in meters. Applications must supply the logic to translate addresses into lat/long pairs (using the Google Maps API, for example) and to convert metric measurements into the preferred system.

A geosearch query has the following general format:

distance(<field-name>, geopoint(lat, lng)) < <distance-in-meters>

The <field-name> is the name of the GeoField you wish to search (store_location in the above example). The place is a GeoPoint representing the location from which we are searching. The radius of the search is specified in meters. For example, the following query locates every store_location within 4.5km of the Sydney Opera House:

distance(store_location, geopoint(-33.857, 151.215)) < 4500

Running this query against the index returns all documents containing store_location fields less than 4500m away from the specified latitude and longitude. The following code snippet demonstrates such a query:

from google.appengine.api import search

index = search.Index(STORE_INDEX_NAME)
query = "distance(store_location, geopoint(-33.857, 151.215)) < 4500"
try:
  search_results = index.search(query)
  for doc in search_results:
    # process doc ...
except search.Error:
  # ...

Sorting Results by Distance

Once you have run the query and generated search results, you can sort them by distance by defining the SortOptions class before displaying them to the user:

index = search.Index(config.STORE_INDEX)
loc = (-33.857, 151.215)
query = "distance(store_location, geopoint(-33.857, 151.215)) < 4500"
loc_expr = "distance(store_location, geopoint(-33.857, 151.215))"
sortexpr = search.SortExpression(
    expression=loc_expr,
    direction=search.SortExpression.ASCENDING, default_value=4501)
search_query = search.Query(
    query_string=query,
    options=search.QueryOptions(
        sort_options=search.SortOptions(expressions=[sortexpr])))
results = index.search(search_query)

Quotas

The Full Text Search API is currently available with limited free quotas suitable for experimental purposes:

Limit Amount
Total Index Size* 250 MB
API Calls** 20,000/day
Document Size 1MB
Schema Size 1000 fields per schema
Numeric Field Value Must be between –231–1 and 231–1
Query Limit 1000 bytes
Maximum Search Offset 1000
Limit for list_indexes limit and offset 1000 indexes

*The index size limit is not per-index, but aggregated across all indexes.

**When indexing multiple documents in a single call, the call count is increased by the number of documents indexed.

If you require additional quota please contact us. The free quota and pricing are subject to change in further releases, and we expect that our free quota will cover about 1,000 searches per day once the feature has graduated from experimental.

Authentication required

You need to be signed in with Google+ to do that.

Signing you in...

Google Developers needs your permission to do that.