Recursive Pagination

When the content of a data structure is monotonically increasing or decreasing at a manageable rate through the control of a single or small set of variables, and the encapsulating procedure can be recursively called within the limits of the implementing language and machine, this work is conducive to recursion.

A good example is retrieving data from a paginated REST API. Each time the client makes a GET request to the server, a successful response will provide a segment of the desired data, and the client is expected to update their passed pagination variable in order to receive the next segment of data.

    getAllData():
        data = newDataStructure()
        url = "http://example.com/data?page="
        page = 1

        getData(data, url, page)

        return data

    getData(data, url, page):
        response = httpGET(url + string(page))
        if response.success == false
            return

        body = response.body

        data.append(body.data)

        if body.hasNext == true
            getData(data, url, page + 1)
        return
                

The above pseudocode defines the function getAllData and its recursive helper function getData. getAllData instantiates a new data structure to hold and operate on the retrieved API data, the url string of the endpoint (which follows the query parameter pattern), and an integer to pass as the query parameter that tells the endpoint which segment of paginated data to return. getData is then called with the data, url and page arguments. A side-effect of getData is to populate data by reference.

getData first makes a GET request to the endpoint, the returned result being some API response data structure. The response body data is appended to data, and if there is more data to paginate through, the page variable is incremented (in this case offset pagination is used) and a recursive call is made. Otherwise, all the data has been paginated through, and recursion ends.