Introduction

If you’re anything like me, then sometimes you get the urge to really just scrape over API responses and re-format the returned information in a particular way - my favourite evening activity. But API providers don’t tend to enjoy people like me returning huge amounts of data in single calls, and so they page their responses; limiting the number of results and providing a method to return the next result set until there are no more results.

Sounds great! But how do I, the intrepid Powershell scripter, utilise all of these results? If this is a question you’ve been asking, then below you’ll (hopefully) find an answer.

“How do I make one of those API calls, anyway?”

As good a place as any to start, right at the beginning!

For the purposes of this post, I’m going to use an API from Microsoft that returns retail price information for their Azure platform - it’s worth noting at this stage that this particular API doesn’t require authentication, and so I’ll add a section at the end of the post about bearer authentication (one of the most common methods) if you’re interested.

The API in question is the Azure Retail Prices API, and I’ve used this in the past in conjunction with the Resources - List API to merge the two datasets for a single view of technical and pricing information. All the information you could need to get started with the API is available at the link above

I’m going to use a built-in Powershell cmdlet, Invoke-RestMethod - if you’re familiar with Invoke-WebRequest, then this is the new standard approach to making HTTP requests from Powershell. As mentioned, this is an unauthenticated API so we don’t need to worry about that and we therefore don’t need to build a header object. This request expects a GET method passed to it, and that’s everything! Therefore the API call will look something like this:

$uri = "https://prices.azure.com/api/retail/prices"
$method = "GET"

$response = Invoke-RestMethod -Method $method -Uri $uri

Et voila! The $response object will contain the entire return from the call and will look like this:

$response output

We’ve got it!

Almost! You can already see some important information here:

  • items is the body of the response, with everything we will want to collect
  • NextPageLink is the link for the next set of results

So we need to more variables to our script to store only the information we care about:

$uri = "https://prices.azure.com/api/retail/prices"
$method = "GET"

$response = Invoke-RestMethod -Method $method -Uri $uri
$results = $response.Items
$nextLink = $response.NextPageLink

We could then run another Invoke-RestMethod using $nextLink instead of $uri to get the next set of results. But we’d need to add those results to our existing $results object, and we don’t know how many times we might need to do this to get the full set (spoiler - hundreds if not thousands of times!) so we need a way of iterating through the Next Link’s without knowing how many we might need to make.

The Do While Loop

Anyone who has done any amount of programming will be familiar with Do While (Finally) loops, but if you’re not it’s a useful tool to perform an activity until a condition changes (and finally, which I don’t use here, allows us to perform a last iteration which might perform a slightly different action than the rest of the loop, like finalising the output into a file) - we set that condition, and while that condition exists the code within the loop will run on repeat. We can therefore use this to:

  • Run the first API call
  • Check for the existence of a NextPageLink
  • If it exists, set the $uri variable to the NextPageLink and restart the loop
  • If it doesn’t exist, finish the loop!

Below is the loop, with comments to explain each section:


$uri = "https://prices.azure.com/api/retail/prices"
$method = "GET"

#Use $output to collect all of the results - this is more efficient than using the += operator

$output = do {

    #Make the API call and store the response in $results
    $results = Invoke-RestMethod -Method $method -Uri $uri

    #pipe each individual object in items to stdout, storing it in $output
    $results.items | foreach-object {
        $_
    }

    #creates a boolean variable that is true if nextPageLink has a value
    $nextLinkExists = [bool]($($results.nextPageLink -ne $null))

    #if $nextLinkExists is true, change the value of $uri and sleep for 30 seconds to prevent rate limiting
    if ($nextLinkExists) {
        $uri = $results.NextPageLink
        Start-Sleep -Seconds 30
    }

#stays within the loop only if $nextLinkExists is true
} while ($nextLinkExists)

And that really is it! The resultant $output object will contain every single record from all of the API calls, without needing to know in advance how many times you need to run and increment the API parameters.

Hopefully this can help you - note that other APIs might have different requirements around how to use the paginated results, but this should give you a head start.

You mentioned something about authentication?

If your API requires authentication, or perhaps expects a specific content-type, then you’ll need to build a headers object for the Invoke-RestMethod invocation. Without going into detail, that can be as simple as this:


$authToken = "your-token"

$headers = @{
    "authorization" = "bearer $authToken"
    "content-type" = "application/json"
}

$response = Invoke-RestMethod -Method $method -Uri $uri -Headers $headers