Building An End-to-end Application with LoopBack & React.js — Part 3: Pagination in GitHub API Results

Published in

ITNEXT

3 min readJun 29, 2021

Previously, we created the service proxy that connects to the GitHub APIs in a LoopBack application. The result contains 30 items per page by default. We can increase this number but only to a maximum of 100. In this article, I’m going to show you how to traverse the pages to get all the items we need.

If you want to see how the completed application looks like, you can go here.

Basic thing about GitHub API pagination

Before we begin, let’s take a look at how pagination works in GitHub API.

According to this doc about GitHub API pagination https://docs.github.com/en/rest/guides/traversing-with-pagination, the pagination information is in the Link header. For example

Link: <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=2>; rel="next",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=34>; rel="last"

That means, if a Link header exists, we can use the url for rel="next" as the next API call to get the next set of results. To get all the results, we are going to keep inspecting the Link header on the response until there is no more “next” link.

Recap on what we’ve created for the DataSource

There are 2 things to note here when reviewing the DataSource we created earlier:

fullResponse:true : this is needed because the pagination information is in the response header
getIssuesByURL function: since the API endpoint for the next set of results is already available, it might be easier if we have this extra function.

Adding new functions in GHQueryController

We’re going to make some changes in the GHQueryController so that we can keep evaluating the Linkheader to get the next page of results.

We’ll create the following 3 functions:

addToResult : this is not really related to pagination, but rather tidying up the code we created previously because this will be called in multiple places.
getNextLink : this new function inspects the Link header and find the url associated with rel="next" . When we reach the last page, this function returns null.
getIssueByURL : this function maps to the service proxy we’ve created previously. After we get the “next” link, this function will be called to get more results.

Putting things together

Now that we have everything in place, let’s make some more changes in the getIssuesByLabel function to make use of the functions we’ve created in the above section.


  @get('/issues/repo/{repo}/label/{label}')
  async getIssuesByLabel(
    @param.path.string('repo') repo: string,
    @param.path.string('label') label:string): Promise<QueryResult> {
    let result:QueryResponse = await this.queryService.getIssuesByLabel(repo, label);
    let queryResult = new QueryResult();
    queryResult.items = [];
    queryResult.total_count = result.body.total_count;
    result.body.items.forEach(issue => {
      this.addToResult(issue, queryResult);
    });// check if there is next page of the results
    const nextLink = this.getNextLink(result.headers.link);
    if (nextLink == null) return queryResult;
    await this.getIssueByURL(nextLink, this.queryService, queryResult);
    return queryResult;
  }

You can find the completed controller: https://github.com/dhmlau/loopback4-example-github/blob/main/src/controllers/gh-query.controller.ts.

Let’s test it out!

Now if we call the endpoint with the same values (like the one I have below), you’ll get the similar results as before, just that if the total_count is more than 30, the items size you get in the result will be the same as the total_count .

Conclusion

What we’ve created so far is a REST API that takes the GitHub org+repo name and the label as path parameter and returns a list of issues (and pull requests) that match the criteria. In this article, we traverse the result pages to get all the items instead of the first 30 items.

This API is a bit primitive but I think it’s a good starting point to build a frontend application that talks to that API. We can always continuously improve the application.

References

Previous posts in this blog series:

Part 1 — Creating DataSource to GitHub API: https://mobilediana.medium.com/building-an-end-to-end-application-with-loopback-react-js-7a22d726c35d
Part 2 — Creating Service Proxy: https://mobilediana.medium.com/building-an-end-to-end-application-with-loopback-react-js-part-2-creating-service-proxy-7ffac2bd7980