Search::Elasticsearch::Client::6_0::Async::Scroll - A helper module for scrolled searches


version 7.717


    use Search::Elasticsearch::Async;

    my $es = Search::Elasticsearch::Async->new;

    my $scroll = $es->scroll_helper
        index       => 'my_index',
        body => {
            size    => 1000,
            sort    => '_doc',
            query   => {...}
        on_start    => \&on_start,
        on_result   => \&on_result,
      | on_results  => \&on_results,
        on_error    => \&on_error

    $scroll->start->then( sub {say "Done"}, sub { warn @_ } );

    sub on_start {
        my $scroll = shift;
        say "Total hits: ". $scroll->total;

    sub on_result {
        my $doc = shift;

    sub on_results {
        for my $doc (@_) {

    sub on_error {
        my $error = shift;
        warn "$error";


A scrolled search is a search that allows you to keep pulling results until there are no more matching results, much like a cursor in an SQL database.

Unlike paginating through results (with the from parameter in search()), scrolled searches take a snapshot of the current state of the index. Even if you keep adding new documents to the index or updating existing documents, a scrolled search will only see the index as it was when the search began.

This module is a helper utility that wraps the functionality of the search() and scroll() methods to make them easier to use.

This class does Search::Elasticsearch::Client::6_0::Role::Scroll and Search::Elasticsearch::Role::Is_Async.


There are two primary use cases:

Pulling enough results

Perhaps you want to group your results by some field, and you don't know exactly how many results you will need in order to return 10 grouped results. With a scrolled search you can keep pulling more results until you have enough. For instance, you can search emails in a mailing list, and return results grouped by thread_id:

    use Promises qw(deferred);

    sub find_email_threads {
        my (%groups,@results,$scroll);

        my $d = deferred;

        $scroll = $es->scroll_helper(
            index     => 'my_emails',
            type      => 'email',
            body      => { query => {... some query ... }},
            on_result => sub {
                my $doc = shift;
                my $thread = $doc->{_source}{thread_id};
                unless ($groups{$thread}) {
                    $groups{$thread} = [];
                    push @results, $groups{$thread};
                push @{$groups{$thread}},$doc;

                # stop collecting if we have 10 results
                if (@results == 10) {

            # resolve with results if completed successfully
            sub { $d->resolve(@results) },

            # reject with error if failed
            sub { $d->reject(@_) }

        return $d->promise;

Extracting all documents

Often you will want to extract all (or a subset of) documents in an index. If you want to change your type mappings, you will need to reindex all of your data. Or perhaps you want to move a subset of the data in one index into a new dedicated index. In these cases, you don't care about sort order, you just want to retrieve all documents which match a query, and do something with them. For instance, to retrieve all the docs for a particular client_id:

        index       => 'my_index',
        size        => 1000,
        body        => {
            query => {
                match => {
                    client_id => 123
            sort => '_doc'
        on_result => sub { do_something(@_) }

Very often the something that you will want to do with these results involves bulk-indexing them into a new index. The easiest way to do this is to use the built-in "reindex()" in Search::Elasticsearch::Client::6_0::Direct functionality provided by Elasticsearch.



    use Search::Elasticsearch::Async;

    my $es = Search::Elasticsearch::Async->new(...);
    my $scroll = $es->scroll_helper(
        scroll             => '1m',            # optional
        scroll_in_qs       => 0|1,             # optional

        on_result          => sub {...}        # required
      | on_results         => sub {...}        # required

        on_start           => sub {...}        # optional
        on_error           => sub {...}        # optional

The "scroll_helper()" in Search::Elasticsearch::Client::6_0::Direct method loads Search::Elasticsearch::Client::6_0::Async::Scroll class and calls "new()", passing in any arguments.

You can specify a scroll duration (which defaults to "1m"), scroll_in_qs (which defaults to false), and any of the listed callbacks. Any other parameters are passed directly to "search()" in Search::Elasticsearch::Client::6_0::Direct.

The scroll duration tells Elasticearch how long it should keep the scroll alive. Note: this duration doesn't need to be long enough to process all results, just long enough to process a single batch of results. The expiry gets renewed for another scroll period every time new a new batch of results is retrieved from the cluster.

By default, the scroll_id is passed as the body to the scroll request. To send it in the query string instead, set scroll_in_qs to a true value, but be aware: when querying very many indices, the scroll ID can become too long for intervening proxies.

The scroll request uses GET by default. To use POST instead, set send_get_body_as to POST.


You must specify either an on_result callback or an on_results callback.

on_result and on_results

The on_result callback is called once for every result that is received.

    sub on_result {
        my $doc = shift;

Alternatively, you can specify an on_results callback which is called once for every set of results returned by Elasticsearch:

    sub on_results {
        for my $doc (@_) {

If either on_result or on_results returns a new Promise, processing of further results will be paused until the promise has been rejected or resolved.


The on_start callback is called after the first request has completed, at which stage the properties like total(), aggregations(), etc will have been populated.


The on_error callback is called if any error occurs. The default implementation warns about the error, and rethrows it.

    sub on_error { warn "Scroll error: @_"; die @_ }

If you wish to handle (and surpress) certain errors, then don't call die(), eg:

    sub on_error {
        my $error = shift;
        if ($error =~/SomeCatchableError/) {
            # do something to handle error
        else {
            # rethrow error
            die $error;


           ->then( \&success, \&failure );

The start() method starts the scroll and returns a Promise which will be resolved when the scroll completes (or "finish()" is called), or rejected if any errors remain unhandled.



The finish() method clears out the buffer, sets "is_finished()" to true and tries to clear the scroll_id on Elasticsearch. This API is only supported since v0.90.6, but the call to clear_scroll is wrapped in an eval so the finish() method can be safely called with any version of Elasticsearch.

When the $scroll instance goes out of scope, "finish()" is called automatically if required.


    $bool = $scroll->is_finished;

A flag which returns true if all results have been processed or "finish()" has been called.


The information from the original search is returned via the accessors below. These values can be accessed in the on_start callback:


The total number of documents that matched your query.


The maximum score of any documents in your query.


Any aggregations that were specified, or undef


Any facets that were specified, or undef


Any suggestions that were specified, or undef


How long the original search took, in milliseconds


How long the original search plus all subsequent batches took, in milliseconds. This value can only be checked once the scroll has completed.



Enrico Zimuel <>


This software is Copyright (c) 2022 by Elasticsearch BV.

This is free software, licensed under:

  The Apache License, Version 2.0, January 2004