| ★ wanayoo — archive 1999 https://developers.google.com/appengine/docs/python/search/overview | Nouvelle recherche | Portail wanayoo |
Search is an experimental, innovative, and rapidly changing new feature for Google App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes to Search. We will inform the community when this feature is no longer experimental.
The Search API allows your application to perform Google-like searches over structured data. You can search across several different types of data (plain text, HTML, atom, numbers, dates, and geographic locations). Searches return a sorted list of matching text. You can customize the sorting and presentation of results.
This overview explains how to:
It has the following sections:
The Search API provides a simple model for indexing and searching data, and organizing and presenting search results. Search can query any data, as long as that data is described by a document. Documents consist of any number of fields, and the fields can contain a variety of different content types described by Field classes (plain text, HTML, atom, numbers, dates, and locations).
After structuring searchable data into document fields, you can structure documents into an index using the
Index class.
This class contains functions to add, remove, or list documents in an index, and also to search an index with a query. Search queries are based on a comprehensive query language that supports numeric and text expressions. The API also allows you to specify
query options,
which provide fine control over the results returned in a search. You can specify how many results to return, whether to return result snippets, how to rank and order results, and more.
Matching search results are returned to the application in a
SearchResults
object. This object includes the number of results found, the actual results returned, and an optional
cursor
object allowing you to page through results. You can also control the presentation of search results using a number of Field expressions and snippeted fields. These tools allow you to compute customized content snippets for any result returned by a search. This object also provides a starting cursor position or offset allowing you to present multiple pages of results.
Search uses the
Document class
to build a document object describing a specific type of data, such as greetings added to a guestbook or metadata attached to an image. Documents contain fields that describe a particular type of data (text, HTML, atom, number, date, or location). The data type is designated by a
Field class
specified when structuring the document. Each document is uniquely identifiable by a document id (doc_id) assigned when you create the document.
To make your data searchable, you first create documents to describe it, then add those documents to the index. You can remove or list the index contents programmatically. You can also re-index existing documents, by adding a document with the same document ID of an existing document.
In order to make your data searchable, you need to describe it using a
Document object.
At a minimum, each document has a document identifier and a list of fields.
The document identifier (doc_id) is used to identify a document in searches. You do not need to specify the document identifier explicitly. However, specifying your own identifier can be useful for implementing search with other storage mechanisms, such as Blobstore or Google Storage. In the case of Blobstore, for example, you can set the document identifier to the
BlobKey
in order to associate that document with a specific blob. The ID must be visible, printable ASCII, must not begin with "!", and cannot contain spaces.
Other situations may also call for you to create your own document identifier. For example, retailers often have a product ID code for each of their products. This retailer could create a document for each product, explicitly setting the product ID to be the doc_id. This approach would allow the retailer to to easily update the index as data about products changes, simply by uploading a new document with the same ID.
Each document contains fields that will contain a particular type of content. The content type is identified by one of the following concrete Field classes
TextField
: plain textHtmlField
: HTML formatted textAtomField
: a string which is treated as a single tokenNumberField
: a numeric which supports int, long and float valuesDateField
: a date with no time componentGeoField
: a locale based on latitude and longitudeEach field has a name, which allows you to reference it in search queries, and a value that indicates the type of content in the field.
The following code sample builds a document that might be used to index user comments in a bulletin board. The entry includes the user's nickname and comment, and a timestamp provided from the Python datetime module:
from datetime import datetime
from google.appengine.api import search
def CreateDocument(nickname, content):
return search.Document(
fields=[search.TextField(name='author', value=nickname),
search.HtmlField(name='comment', value=content),
search.DateField(name='date', value=datetime.now().date())])
For the full context of this code sample, check out the full application available on the Google App Engine samples page.
Once you have created documents describing your data, the next step is to add the documents to the index, described in the following section.
Before you can perform queries on a document, you need to index it to make the data it describes searchable. You add documents to an index by calling the add() method on an Index object.
The following code sample creates an index called comment and adds a document to it:
from google.appengine.api import search
...
_INDEX_NAME="comment"
try:
search.Index(name=_INDEX_NAME).add(document)
except search.Error:
logging.exception('Add failed')
...
An Index object requires a name and a consistency mode, either
PER_DOCUMENT_CONSISTENT
or
GLOBALLY_CONSISTENT
. You set the consistency mode when you get the index:
index = search.Index(name='index-name',
consistency=search.Index.PER_DOCUMENT_CONSISTENT)
Once set, that consistency mode is used from that point forward. The default is
PER_DOCUMENT_CONSISTENT.
The following table describes the differences between these two consistency modes:
| Per-document consistency | Global consistency | |
|---|---|---|
| Throughput of indexed documents | Provides relatively high throughput of indexed documents, enabling faster searches. | Provides relatively low throughput of indexed documents (around 1 write per second), causing slower searches. |
| Visibility of recent changes | Changes to any document in the index may not be applied in the correct chronological order. | Changes to any document stored in the index are applied in the correct chronological order. |
| Freshness of results | Results may not contain all changes committed before the search request. | Results contain all changes committed before the search request. |
| Sample use case | Documents that amalgamate information from multiple sources, where no single user is aware of all collected data. | Documents that organize personal user information, to reflect all changes known to the user in any search results. |
The Search API parses content and special characters stored in your indexes according to a specific algorithm. This algorithm identifies sequences of text and parses them based on the characters present. Strings of letters interpreted as an acronym (such as a b c, a.b.c, or a-b-c) are interpreted without spaces or punctuation. Other special characters are interpreted based on the context in which they appear.
You can also use the ~ search operator to search for comment variants and synonyms of a term.
You can remove documents in an index by specifying the
doc_id
of one or more documents you wish to remove to the
Index.remove()
method. To remove a collection of documents, specify
the ids_only argument to the Index.list_documents() method
. When you use this argument, the API returns document objects populated only with the document ID. You can then delete the documents by passing those Document objects to the remove() method:
from google.appengine.api import search
...
def deleteAllInIndex(index_name):
"""Delete all the docs in the given index."""
doc_index = search.Index(name=index_name)
while True:
# Get a list of documents populating only the doc_id field and extract the ids.
document_ids = [document.doc_id
for document in doc_index.list_documents(ids_only=True)]
if not document_ids:
break
# Remove the documents for the given ids from the Index.
doc_index.remove(document_ids)
Each index has an associated schema used to interpret queries on that index. This schema is created dynamically from the field names and types present in all of the documents in the index. Viewing an index schema allows you to see the structure of the data in that index so you can determine the structured queries you can perform on the index.
There is currently a limit of 1000 named fields for each a given index schema.
There is currently no way to remove fields from a schema, nor to delete a schema, but you can view the schemas for your indexes like this:
from google.appengine.api import search
...
for index in search.list_indexes(fetch_schema=True):
logging.info("index %s", index.name)
logging.info("schema: %s", index.schema)
A simple schema for an index might look like this:
{'comment': ['TEXT'], 'date': ['DATE'], 'author': ['TEXT'], 'count': ['NUMBER']}
Each key in this dict is the name of a document field that has been added to the index. The key's value is a list of the field types used with that field name. If you have used the used the same field name with different field types, (for example, in different documents), the schema will list more than one field type for a field name, like this:
{'field1': ['TEXT', 'NUMBER']}
The Search API provides a number of query operators that you can use to query data stored in a document's fields. You can filter on a field's values using a field operator value expression such as:
created >= 2011-1-1
The field identifies the name of a property defined as part of a document. The value is a word, string, number, or other text that restricts the query on the specified field.
Search supports all space-delimited languages. Search supports some languages not segmented by spaces (specifically, Chinese, Japanese, Korean, and Thai). For these languages, Search segments the text automatically.
The simplest type of query consists only of a word, which the API matches only against a document's text, HTML, or atom fields. Words are not case sensitive. In the case of atom fields, the word must match the entire field value.
For example, to find all documents with the word "rose" (regardless of case) in any string or text field in the document, use a query like the following:
rose
This simple query matches against any supported string or text field in the document. If your documents are "Comments" as defined in the Creating Documents section, the query matches if the word "rose" appears in the author or body fields. If the schema defines additional text or HTML fields, such as a subject or email, "rose" also matches the contents of those fields.
To match a phrase, surround the query in quotes as follows:
"any other name"
To search for plural variants of an exact query, use the ~ operator:
~"car" # searches for "car" and "cars"
Note: Simple query strings match only against text, HTML, or Atom fields. The next section describes how to query against specific fields.
To build queries that reference specific fields, use both field and value in your query, separated by a colon:
field:value field:"value as a string"
This syntax allows you to reference any supported field by name, assuming the field has been defined in the document. For example, to search for the word "rose" only within the author field of a comment document as defined in Creating Documents, use the following query:
author:rose
To search the body field for the phrase "any other name", use the following query:
body:"any other name"
To match against multiple fields at the same time, list a series of field:value pairs together with a space between them as follows:
author:"Rose Jones" body:rose
The query language supports some Boolean operators as well as parentheses for grouping parts of the query together. The supported Boolean operators are AND, OR, and NOT. Always use uppercase for Boolean operators. Lowercase words are treated as part of the field or value portions of the query.
By default, when you create queries that match multiple fields at the same time, each value is combined with a Boolean AND. For the query as a whole to match, all the specified values must match.
You can also explicitly specify this by using the AND Boolean operator. The following two queries are equivalent:
author:rose body:"any other name" author:rose AND body:"any other name"
Use the OR operator if you only want to know if any of the two values matches. You can use more than one OR in a query. For example:
author:(bob OR ((rose OR tom) AND jones))
This example matches any document whose author field contains the set of words in no particular order or vicinity ({rose, jones}, {tom, jones}, or {bob}).
The following example of the Boolean NOT operator matches any document whose author field contains "rose" but whose body field does not contain the word "filligree":
author:rose NOT body:filigree
Use parentheses to create more complex queries combining supported Boolean operators.
For example:
(author:Thomas OR author:Jones) AND (NOT body:rose)
This example matches documents with author "Thomas" or "Jones", but whose body field does not contain the word "rose".
Numeric operators only match against numeric and date fields. Query values containing a numeric operator must be quoted.
Search supports the following numeric operators:
:=>>=<<=For example, assuming your documents contain a votes field, to find all documents with more than ten votes:
votes > 10
For "not equal to", use the Boolean NOT with a numeric field name such as length. For example:
NOT length = 15
This example returns documents whose length is not 15.
You can combine numeric operators with text and Boolean operators. For example:
author:"Rose Jones" length > 15
This query matches comments whose length field is longer than 15 characters in length and whose author field is "Rose Jones".
You can also use numeric operators to query for dates. For example:
published > 2011-1-1
This query searches for documents published after January 1, 2011.
In addition to the search queries described above, the API also supports query expressions that you can use in the computation of search results. You can use these expressions with the expression parameter of both the FieldExpression and SortExpression classes, as well as some search queries. The available expressions are:
| Operator | Type | Description | Example |
|---|---|---|---|
geopoint |
geographic | Degrees of latitude and longitude. | To return the map location of the specified coordinates, use:
geopoint(-31.3, 151.4) |
distance |
geographic | The distance between two locations in meters. | To determine the distance between the specified geopoint and the value of the location field, use:
distance(geopoint(23, 134), location)
May be combined with numeric operators. For example, this query returns all matching locations within 10 meters of the distance(geopoint(23, 134), location) < 10 |
< |
binary, numeric | Matches if the field contains a number less than the value specified in the query. | To return documents whose |
+ |
binary, numeric | Adds the provided numeric values. | To add the value of a document's price and tax fields, use:
price + tax
|
– |
binary, numeric | Subtracts the second numeric value from the first. | To subtract the document's predefined _score attribute from the value of the rank field, use: rank – _score
|
* |
binary, numeric | Multiplies the arguments. | To multiply the document's predefined _score attribute by the value of the rank field, use:
_score * rating |
/ |
binary, numeric | Divides the first value by the second. | To divide the _score by the rank, use:
_score / rating |
max |
N-ary | Returns the larger of the two arguments. |
max(recommended_retail_price, discount_price) |
min |
N-ary | Returns the smaller of the two arguments. |
max(recommended_retail_price, discount_price) |
snippet |
string | Returns a snippet for the given query. Returned snippets are in HTML, with the query terms in bold. This operator is constructed as follows:
|
To query for rose in the field author, returning a maximum of 4 snippets that are each 200 characters, use:
snippet("rose", author, 200, 4)Similarly, the following sample creates snippet for the phrase "few good" on a description field. snippet("price < 10 \"few good\"", description) |
_score |
Predefined | The score of the document returned in results, based on how well that document matches the query. | To multiply the _score by the value of the rank field, use:
_score * rank |
count |
Unary | Returns the total number of attribute values for this document. | count(size) |
Constructing the query is only one part of performing a search. Once you have the query, you need to pass it to the API either as a simple string or as part of a Query object.
Queries passed as a string match all document fields and allow you to specify other details, such as the number of results to return, the point from which to continue the search (using a cursor), and even specific text fields to return in results.
Queries passed to the Index.search() method must follow a specific query language. Simple queries can simply pass a query string, but you can issue more complex queries
using the Query and QueryOptions classes.
This section describes how to construct queries and use them to search the content of documents.
The simplest means of executing a query is to pass it to the
Index.search()
method, taking the query as an argument to the method and returning results:
from google.appengine.api import search
...
try:
results = search.Index(name=_INDEX_NAME).search(query_string)
for scored_document in results:
# process scored_document
except search.Error:
logging.exception('Search failed')
When you execute a search, the API returns a
SearchResults
object representing the results of the search request. This object describes whether the query succeeded, providing a list of matching documents ranked based on their degree of match. You can choose to include a cursor in results, allowing you to continue the search from the end of the current result set.
Passing a query in a Query object allows you to define more complex search requests, including details such as:
limit on the number of results to returnoffset defining a number of matched documents to skip when returning resultscursor to return with results that allows you to get the next set of resultsWhen constructing searches using the Query class, you first construct the
QueryOptions:
# Set query options
options = search.QueryOptions(
limit=limit, # the number of results to return
cursor=cursor,
sort_options=sort,
returned_fields=['author', 'subject', 'summary'],
snippeted_fields=['content'])
Then pass this object as the options argument to the Query class:
query = search.Query(query_string=query_string, options=options)
As was the case with simple searches, the API returns a
SearchResults
object representing the results of the search request.
The following code sample demonstrates a full search request, starting the search from the cursor location of the previous set of results, returning another single cursor for the response, sorting by "subject" in descending order, and returning the author, subject, and summary fields with snippeted field content.
from google.appengine.api import search
...
def FindDocuments(query_string, limit, cursor):
try:
subject_desc = search.SortExpression(
expression='subject',
direction=search.SortExpression.DESCENDING,
default_value='')
# Sort up to 1000 matching results by subject in descending order
sort = search.SortOptions(expressions=[subject_desc], limit=1000)
# Set query options
options = search.QueryOptions(
limit=limit, # the number of results to return
cursor=cursor,
sort_options=sort,
returned_fields=['author', 'subject', 'summary'],
snippeted_fields=['content'])
query = search.Query(query_string=query_string, options=options)
index = search.Index(name=_INDEX_NAME)
# Execute the query
return index.search(query)
except search.Error:
logging.exception('Search failed')
return None
Then call the above function to get the cursor for the next set of results:
results = FindDocuments(query_string, 20, search.Cursor())
if results:
for scored_document in results:
# process scored_document
next_cursor = results.cursor
You can then specify a query using FindDocuments:
next_cursor = results.cursor
next_cursor_urlsafe = next_cursor.web_safe_string
# save next_cursor_urlsafe
...
# restore next_cursor_urlsafe
results = FindDocuments(query_string, 20,
search.Cursor(web_safe_string=next_cursor_urlsafe))
You can use the Search API to include location search in your application. Locations to be searched must be stored within document fields in your application, while the browser typically provides the user's point of origin and search radius. With this information, the API can search for locations within the user's specified radius, and return results sorted by distance from that origin.
In order to perform location-based searches, your app first needs to store locations within document fields, as described in the next section.
To build locations into documents, first construct the
GeoPoint
object
and then set that object as the value of the of the GeoField in your document:
from google.appengine.api import search
...
# Construct the GeoPoint class
geopoint = search.GeoPoint(latitude, longitude)
fields = [search.TextField(name='name', value=store_name),
search.TextField(name='address', value=store_address),
# Construct a GeoField passing geopoint as the value of that field
search.GeoField(name='store_location', value=geopoint)
]
After constructing and indexing these documents, you can run search queries against them, as described in the following section.
Once you have stored a set of locations in your application, you can allow users to search for locations within a specified radius from a point of origin (for example, 1500 meters from 345 Spear Street in San Francisco). The user typically provides this information via the browser.
Locations are expressed in latitude and longitude coordinates, and distances are expressed in meters. Applications must supply the logic to translate addresses into lat/long pairs (using the Google Maps API, for example) and to convert metric measurements into the preferred system.
A geosearch query has the following general format:
distance(<field-name>, geopoint(lat, lng)) < <distance-in-meters>
The <field-name> is the name of the GeoField you wish to search (store_location in the above example). The place is a GeoPoint representing the location from which we are searching. The radius of the search is specified in meters. For example, the following query locates every store_location within 4.5km of the Sydney Opera House:
distance(store_location, geopoint(-33.857, 151.215)) < 4500
Running this query against the index returns all documents containing store_location fields less than 4500m away from the specified latitude and longitude. The following code snippet demonstrates such a query:
from google.appengine.api import search
index = search.Index(STORE_INDEX_NAME)
query = "distance(store_location, geopoint(-33.857, 151.215)) < 4500"
try:
search_results = index.search(query)
for doc in search_results:
# process doc ...
except search.Error:
# ...
Once you have run the query and generated search results, you can sort them by distance by defining the
SortOptions class before displaying them to the user:
index = search.Index(config.STORE_INDEX)
loc = (-33.857, 151.215)
query = "distance(store_location, geopoint(-33.857, 151.215)) < 4500"
loc_expr = "distance(store_location, geopoint(-33.857, 151.215))"
sortexpr = search.SortExpression(
expression=loc_expr,
direction=search.SortExpression.ASCENDING, default_value=4501)
search_query = search.Query(
query_string=query,
options=search.QueryOptions(
sort_options=search.SortOptions(expressions=[sortexpr])))
results = index.search(search_query)
The Full Text Search API is currently available with limited free quotas suitable for experimental purposes:
| Limit | Amount |
|---|---|
| Total Index Size* | 250 MB |
| API Calls** | 20,000/day |
| Document Size | 1MB |
| Schema Size | 1000 fields per schema |
| Numeric Field Value | Must be between –231–1 and 231–1 |
| Query Limit | 1000 bytes |
| Maximum Search Offset | 1000 |
Limit for
list_indexes limit and offset
|
1000 indexes |
*The index size limit is not per-index, but aggregated across all indexes.
**When indexing multiple documents in a single call, the call count is increased by the number of documents indexed.
If you require additional quota please contact us. The free quota and pricing are subject to change in further releases, and we expect that our free quota will cover about 1,000 searches per day once the feature has graduated from experimental.