Saving External API Data To Your Own Rails API

YOU MAY FIND THIS BLOG HELPFUL IF YOU ARE TRYING TO PERSIST EXTERNAL API DATA TO YOUR OWN BACK-END.

Here’s the demo video of the project I based this off of.

If you are coming to this article, you probably already understand what a fetch request is and it’s significance. If you do not, you may want to stop here and quickly read up on using fetch.

Now, since you are familiar with fetch requests, you know that they are used to get, post, put, and delete data on a front-end JavaScript interface. Something that you may not have known and the reason you are here is that there is also a way to do this on the back-end of a Ruby on Rails application.

Cue RestClient Requests!!!

There are a few reasons you might want to utilize RestClient calls.

1. The external API you are using has rotating data and you want to save it all to look back on it (or conduct a longitudinal study)
2. You have a limited number of API calls on a certain external API and you want to limit that number of calls
3. You are using too many external APIs to keep your code clean
4. All of the above and more

In my case and the example I will be using, I worked with RestClient in order to save the data from 5 external APIs to my own Rails API. I chose to utilize RestClient for every reason above.

Initially, I was making these calls from the front-end using fetch requests, but this was a time consuming process that was slowing my application and leaving my code long and confusing. In my case, RestClient calls kept all of my data in my database, making it easy to reach with one single fetch request from my front-end to my own personal Rails API.

Project Background

I decided to run a study where I sentimentally analyzed articles from The New York Times, Fox News, ABC, BBC, and CNN. Using a React front-end and a Rails backend, I did this because there has clearly been a great amount of turmoil and disconnect from major news stations and the general populous. I wanted to see if the sentiment behind their articles told any stories when I displayed the data side by side. For this project, I used the Indico Sentimental Analysis API. It was incredibly easy to work with and I would recommend it for getting surface level sentiment. I would be hesitant to say that you can truly trust the results as it has trouble understanding context (A major flaw in my study). I also began building my own sentimental analysis AI, but it was taking my old MacBook Pro hours to train my model, so I scrapped that for the sake of time.

Now, here is what one of my back-end RestClient files looked like. I was pulling in the data from The New York Times API all while running each article through a sentimental analysis AND saving it to my own database. I have commented the code for reference (everything following a “#”). If you are looking to do something similar, you should be able to copy a similar structure. Take special note of the gems that are required at the top of the page, the data that is being parsed so that I am able to iterate over each individual result or article, the measures I took to ensure there are no duplicates in my database, and how I saved the article (output can be seen in the GIF towards the bottom of the page).

require 'rest-client'
require 'indico'
require 'rails/configuration'
#New York Times API endpoint
  nyt_url = "https://api.nytimes.com/svc/topstories/v2/home.json?&api-key="
#RestClient get request to the NYT endpoint with my API Key being parsed into a JS object
  data = JSON.parse( RestClient.get("#{nyt_url}#{ENV["NYT_API_KEY"]}") )
#Iterating through each result/article of the NYT
  data["results"].each do |article, index|
    #finding existing articles
    existing_article = Article.find_by(headline: article["title"])
    #checking if the articles abstract is there and if the article doesnt exist
    if article["abstract"].length > 0
      if !existing_article
        config = {api_key: "#{ENV["INDICO_API_KEY"]}"}
        #Running sentimental analysis on the article's abstract
        emotion = (Indico.emotion(article["abstract"], config))
#Creating a new article in my database and assigning it's properties
        news = Article.new do |key|
          key.headline = article["title"]
          key.news_station = "New York Times"
          key.abstract = article["abstract"]
          key.category = article["section"]
          key.url = article["url"]
          #Some of the articles didnt have pictures so I made a default one if there was no pic
          if article["multimedia"].length === 0
            key.image = "http://www.nytimes.com/services/mobile/img/ios-newsreader-icon.png"
          else
            key.image = article["multimedia"][0]["url"]
          end
          #Saving sentiments
          key.anger = emotion["anger"]
          key.joy = emotion["joy"]
          key.fear = emotion["fear"]
          key.sadness = emotion["sadness"]
          key.surprise = emotion["surprise"]
          key.date = Time.now
        end
        #Saving articles
        if news.save
          puts "saved nyt"
        else
          puts "not saved"
        end
      end
    end
  end

Okay, so the code makes sense… but how do I get it to pull in articles to my database and where do I write this code?

There are a few different ways you can do this. In my case, I decided to do it upon initialization of my server. At the time of this project, I was starting up my server multiple times a day, so I was pulling in all of the articles that were being published. However, I would not say this is the best way. It works for me because I chose not to take my project live; my database was simply too big to host on Heroku.

If you want to do it like I did, I will leave my file structure for the Rails API here.

-app
-bin
-config
--environments
--initializers
---<RestClient file as seen above here>
--locales
-db
-lib
-log
-public
-tmp
-vendor

Here is what my Rails server looks like when it starts up. As you can see it is outputting “saved <news station>” for each article being saved as it initializes the server.

If you wanted to take your project live, I would recommend something other than updating on initialization. Here are two ways for you to update your server every time a new article is uploaded or on set time increments.

Cron Jobs

javan/whenever

Scheduled Jobs on Heroku

Scheduled Jobs and Custom Clock Processes | Heroku Dev Center

Once I figured out RestClient requests for this project, it was smooth sailing from there. Good luck!