-
-
26 Jun 2020 11:22:42 UTC
- Distribution: Search-Elasticsearch-Client-1_0-Async
- Module version: 6.81
- Source (raw)
- Browse (raw)
- Changes
- Homepage
- How to Contribute
- Repository
- Issues (20)
- Testers (13 / 0 / 0)
- Kwalitee
Bus factor: 1- % Coverage
- License: apache_2_0
- Activity
24 month- Tools
- Download (28.41KB)
- MetaCPAN Explorer
- Permissions
- Subscribe to distribution
- Permalinks
- This version
- Latest version
and 1 contributors-
Enrico Zimuel
- Dependencies
- Moo
- Promises
- Scalar::Util
- Search::Elasticsearch::Async::Util
- Search::Elasticsearch::Client::1_0
- Search::Elasticsearch::Client::1_0::Role::Bulk
- Search::Elasticsearch::Client::1_0::Role::Scroll
- Search::Elasticsearch::Role::Is_Async
- Search::Elasticsearch::Util
- Try::Tiny
- namespace::clean
- Reverse dependencies
- CPAN Testers List
- Dependency graph
- NAME
- VERSION
- SYNOPSIS
- DESCRIPTION
- USE CASES
- DEEP SCROLLING
- METHODS
- INFO ACCESSORS
- SEE ALSO
- AUTHOR
- COPYRIGHT AND LICENSE
NAME
Search::Elasticsearch::Client::1_0::Async::Scroll - A helper module for scrolled searches
VERSION
version 6.81
SYNOPSIS
use Search::Elasticsearch::Async; my $es = Search::Elasticsearch::Async->new; my $scroll = $es->scroll_helper index => 'my_index', search_type => 'scan', size => 500, on_start => \&on_start, on_result => \&on_result, | on_results => \&on_results, on_error => \&on_error ); $scroll->start->then( sub {say "Done"}, sub { warn @_ } ); sub on_start { my $scroll = shift; say "Total hits: ". $scroll->total; } sub on_result { my $doc = shift; do_something($doc); } sub on_results { for my $doc (@_) { do_something($doc) } } sub on_error { my $error = shift; warn "$error"; }
DESCRIPTION
A scrolled search is a search that allows you to keep pulling results until there are no more matching results, much like a cursor in an SQL database.
Unlike paginating through results (with the
from
parameter in search()), scrolled searches take a snapshot of the current state of the index. Even if you keep adding new documents to the index or updating existing documents, a scrolled search will only see the index as it was when the search began.This module is a helper utility that wraps the functionality of the search() and scroll() methods to make them easier to use.
IMPORTANT: Deep scrolling can be expensive. See "DEEP SCROLLING" for more.
This class does Search::Elasticsearch::Client::1_0::Role::Scroll and Search::Elasticsearch::Role::Is_Async.
USE CASES
There are two primary use cases:
Pulling enough results
Perhaps you want to group your results by some field, and you don't know exactly how many results you will need in order to return 10 grouped results. With a scrolled search you can keep pulling more results until you have enough. For instance, you can search emails in a mailing list, and return results grouped by
thread_id
:use Promises qw(deferred); sub find_email_threads { my (%groups,@results,$scroll); my $d = deferred; $scroll = $es->scroll_helper( index => 'my_emails', type => 'email', body => { query => {... some query ... }}, on_result => sub { my $doc = shift; my $thread = $doc->{_source}{thread_id}; unless ($groups{$thread}) { $groups{$thread} = []; push @results, $groups{$thread}; } push @{$groups{$thread}},$doc; # stop collecting if we have 10 results if (@results == 10) { $scroll->finish; } } ); $scroll->start->then( # resolve with results if completed successfully sub { $d->resolve(@results) }, # reject with error if failed sub { $d->reject(@_) } ); return $d->promise; }
Extracting all documents
Often you will want to extract all (or a subset of) documents in an index. If you want to change your type mappings, you will need to reindex all of your data. Or perhaps you want to move a subset of the data in one index into a new dedicated index. In these cases, you don't care about sort order, you just want to retrieve all documents which match a query, and do something with them. For instance, to retrieve all the docs for a particular
client_id
:$es->scroll_helper( index => 'my_index', search_type => 'scan', # important! size => 500, body => { query => { match => { client_id => 123 } } }, on_result => sub { do_something(@_) } )->start;
Very often the something that you will want to do with these results involves bulk-indexing them into a new index. The easiest way to marry a scrolled search with bulk indexing is to use the "reindex()" in Search::Elasticsearch::Client::1_0::Async::Bulk method.
DEEP SCROLLING
Deep scrolling (and deep pagination) are very expensive in a distributed environment, and the reason they are expensive is that results need to be sorted in a global order.
For example, if we have an index with 5 shards, and we request the first 10 results, each shard has to return its top 10, and then the requesting node (the node that is handling the search request) has to resort these 50 results to return a global top 10. Now, if we request page 1,000 (ie results 10,001 .. 10,010), then each shard has to return 10,010 results, and the requesting node has to sort through 50,050 results just to return 10 of them!
You can see how this can get very heavy very quickly. This is the reason that web search engines never return more than 1,000 results.
Disable sorting for efficient scrolling
The problem with deep scrolling is the sorting phase. If we disable sorting, then we can happily scroll through millions of documents efficiently. The way to do this is to set
search_type
toscan
:$es->scroll_helper( search_type => 'scan', size => 500, )->start;
Scanning disables sorting and will just return
size
results from each shard until there are no more results to return. Note: this means that, when querying an index with 5 shards, the scrolled search will pullsize * 5
results at a time. If you have large documents or are memory constrained, you will need to take this into account.METHODS
new()
use Search::Elasticsearch::Async; my $es = Search::Elasticsearch::Async->new(...); my $scroll = $es->scroll_helper( scroll => '1m', # optional scroll_in_qs => 0|1, # optional on_result => sub {...} # required | on_results => sub {...} # required on_start => sub {...} # optional on_error => sub {...} # optional %search_params, ); $scroll->start;
The "scroll_helper()" in Search::Elasticsearch::Client::1_0::Direct method loads Search::Elasticsearch::Client::1_0::Async::Scroll class and calls "new()", passing in any arguments.
You can specify a
scroll
duration (which defaults to"1m"
),scroll_in_qs
(which defaults tofalse
), and any of the listed callbacks. Any other parameters are passed directly to "search()" in Search::Elasticsearch::Client::1_0::Direct.The
scroll
duration tells Elasticearch how long it should keep the scroll alive. Note: this duration doesn't need to be long enough to process all results, just long enough to process a single batch of results. The expiry gets renewed for anotherscroll
period every time new a new batch of results is retrieved from the cluster.By default, the
scroll_id
is passed as thebody
to the scroll request. To send it in the query string instead, setscroll_in_qs
to a true value, but be aware: when querying very many indices, the scroll ID can become too long for intervening proxies.The
scroll
request usesGET
by default. To usePOST
instead, set send_get_body_as toPOST
.Callbacks
You must specify either an
on_result
callback or anon_results
callback.on_result
andon_results
The
on_result
callback is called once for every result that is received.sub on_result { my $doc = shift; do_something($doc); }
Alternatively, you can specify an
on_results
callback which is called once for every set of results returned by Elasticsearch:sub on_results { for my $doc (@_) { do_something($doc) } }
If either
on_result
oron_results
returns a new Promise, processing of further results will be paused until the promise has been rejected or resolved.on_start
The
on_start
callback is called after the first request has completed, at which stage the properties liketotal()
,aggregations()
, etc will have been populated.on_error
The
on_error
callback is called if any error occurs. The default implementation warns about the error, and rethrows it.sub on_error { warn "Scroll error: @_"; die @_ }
If you wish to handle (and surpress) certain errors, then don't call
die()
, eg:sub on_error { my $error = shift; if ($error =~/SomeCatchableError/) { # do something to handle error } else { # rethrow error die $error; } }
start()
$scroll->start ->then( \&success, \&failure );
The
start()
method starts the scroll and returns a Promise which will be resolved when the scroll completes (or "finish()" is called), or rejected if any errors remain unhandled.finish()
$scroll->finish;
The
finish()
method clears out the buffer, sets "is_finished()" totrue
and tries to clear thescroll_id
on Elasticsearch. This API is only supported since v0.90.5, but the call toclear_scroll
is wrapped in aneval
so thefinish()
method can be safely called with any version of Elasticsearch.When the
$scroll
instance goes out of scope, "finish()" is called automatically if required.is_finished()
$bool = $scroll->is_finished;
A flag which returns
true
if all results have been processed or "finish()" has been called.INFO ACCESSORS
The information from the original search is returned via the accessors below. These values can be accessed in the
on_start
callback:total
The total number of documents that matched your query.
max_score
The maximum score of any documents in your query.
aggregations
Any aggregations that were specified, or
undef
facets
Any facets that were specified, or
undef
suggest
Any suggestions that were specified, or
undef
took
How long the original search took, in milliseconds
took_total
How long the original search plus all subsequent batches took, in milliseconds. This value can only be checked once the scroll has completed.
SEE ALSO
AUTHOR
Enrico Zimuel <enrico.zimuel@elastic.co>
COPYRIGHT AND LICENSE
This software is Copyright (c) 2020 by Elasticsearch BV.
This is free software, licensed under:
The Apache License, Version 2.0, January 2004
Module Install Instructions
To install Search::Elasticsearch::Client::1_0::Async, copy and paste the appropriate command in to your terminal.
cpanm Search::Elasticsearch::Client::1_0::Async
perl -MCPAN -e shell install Search::Elasticsearch::Client::1_0::Async
For more information on module installation, please visit the detailed CPAN module installation guide.