Pandas for Data Analysis: How to get Data from APIs

It is much easier to create Data Frame objects from data we have from files such as csv, excel, list or JSON, but how do we obtain data from online sources such as APIs? That is what we will learn from this short blog.

Requirements

We need “requests” and “pandas” packages to pull data from APIs sources. The requests package will be used to make API request while pandas is used to create Data Frame.

On our Jupiter notebook, we add the following imports

To test the aforementioned imports and how it pulls data from API source, we will request data from the USGS API. These are data recorded every day that relates to earthquake. We will pull data for the last 30 days up to yesterday. As such, we’ll need datetime package. Thus, all imports are as follows:

Now that we’ve got all the imports ready, we can start pulling data from USGS API. With requests package, we will be able to make GET request to the USGSAPI and obtain JSON payload. The data we are trying to obtain were recorded in the last 30 days beginning from yesterday. Because we need data beginning from yesterday, we have to specify that in the code that tries to retrieve data – this can be done with the help of timedelta from datetime package. Here is the code:

Now that we have the payload and api ready, we can pass the two objects to GET HTTP method to inform the server what we want.

Before creating the DataFrame from response object obtained from the server, we have to check the response to confirm if the server agrees to give use information. This can be done with “status_code” attribute as follows;

A 200-response seen above shows that the server (USGSAPI) has agreed to give us the information we have requested.

Now, it’s time to check the data obtained – the data is obtained in JSON format because that’s what we asked for it (‘format’: ‘geojson’,). It comes formatted in a dictionary. Python dictionary data structure provides a ton of methods that can be used to check data.

Obtain JSON Payload

We can isolated data from the JSON response as follows:

The earthq_json above is a dictionary that carries data. We can confirm the type and check the keys using dictionary function "keys":

For all the keys in the list shown above, we can inspect each one to confirm the data they have. Based on my analysis, 'features' is likely to carry the data we need. Let's check check the type using type function:

As seen above, 'features' key contains a list of information. We can pick a small section of data using index.

We are almost done. From the screenshot above, need to pick the properties section out of every entry in the features JSON array to create our dataframe. This can only be achieved using List Comprehension.

Finally, we can create a dataframe using using what we have obtained above

We can inspect the first 3 rows of the dataframe generated using dataframe head function as follows:

Comments

Post a Comment

Popular posts from this blog

Hangman Game With Python

Image
  Problem and Solution  Here are the instructions of what is expected of this game: You have been assigned a task to implement the Hangman game. In hangman, one player chooses a secret word and the other player guesses letters. Each time the second player guesses a correct letter, all instances of that letter in the secret word are revealed. If the second player guesses a letter that is not in the word, then a portion of the hangman is drawn. The game ends once the entire hangman is drawn or the second player guesses all the letters that are in the word. NOTE: The second player has 7 guesses to guess the correct word After each incorrect guess, the pieces of the hangman are drawn in the following order; Rope, Head, Torso, Left arm, Right arm, Left leg, Right leg At the beginning of the game, the first player should be asked to enter the secret word The letters in the secret word should be displayed as '?' until the user guesses them  At the beginning of each round you should di
Read more

Build a Shopping List Application with ArrayList in Java

Image
ArrayList What’s an ArrayList? Oracle documentation defines an ArrayList as a resizable array that implements List interface. ArrayList comes with a ton of methods that allow manipulation of the array. Some of the methods applicable in ArrayLize include, size, add, get, set, isEmpty and more. I will show how some of the methods work with using code. When defining an ArrayList, it is important to specify the type because ArrayList holds objects. Above,we have specified the type as String. As such, the ArrayList defined will only be able to hold elements of type String. Adding Items to ArrayList ArrayList provides add method that can be used to add items to ArrayList. The method can be called using ArrayList variable defined earlier. In our case, we have created a method and passed an item that will be added to ArrayList using add method. See the following screenshot: : Print items in an ArrayList I mentioned earlier that ArrayList provides several methods to manipulate d
Read more

File handling with C++ Programming

Image
Learn File handling with the following sample code You are to write a program that reads in lines of input using scanf or cin, and file redirection, process the data, and then output the results using printf or cout. This means you can use C or C++ style I/O as is your personal preference. Name your file lab1assignment.cpp. Failure to name the file correctly may lead to a grade of zero. The data file used in the example is available at: ~dmk0080/public/1040/labs/one.or in the Cavans assignment. Use the file named "transactiondata Use command line redirection to execute as in ./a.out angle bracket transactiondata" to compile use g++ labassignment1.cpp to create the executable file a.out The Data: Each line of th named "transactione data file begins with an 'e', 'c', or 't'. Lines beginning with 'e' will have an integer employee ID and an employee name. Lines beginning with 'c' will have an integer custo
Read more