Before we integrate the requests package in our Behave Tests, I'd like to do a quick overview on how the HTTP Protocol works specifically the structure of an HTTP Message. Most of the testers that I know do not really understand the underlying protocol behind REST API calls. Thus it's very common in Facebook  Groups to see inquiries like...

How do I use POSTMAN to do API Tests?
Can I use JMETER to do API Tests?
Our Devs are using Charles Proxy, how do I use it for API tests?

How to do API Tests with <insert programming language here>?

What's disturbing is that most of the replies from the inquiries will give them a very specific instruction on how to operate the tool or they will recommend a different tool that they use on their company. But... it doesn't really solve the underlying problem that most of the testers do not understand the technology behind it.

To cut it short, Testers who have zero experience in doing API Tests should first learn the HTTP Protocol specifically the HTTP Messages. Once they've learned the basics of it then everything will be easier to understand.

HTTP - A Quick Overview

Request Response Model

One bullet point that i need to highlight is that HTTP  operates in a Request - Response Model. It means that when a client (you) requests for something then the server is obliged to respond back. You're probably scratching your head right now but you've been doing HTTP requests without the help of a tool (PostMan, ReadyAPI, JMeter) without realizing it.

If you've logged in to your Facebook Account then you've successfully made an HTTP request. And when Facebook prompted that you've successfully logged in then you've successfully received an HTTP Response. These Requests and Responses follows particular set of rules or format that was defined by the HTTP Protocol. Once you know the proper format in sending HTTP Messages then learning new tools or using different programming languages to automate testing of REST APIs is easy.

HTTP Messages

HTTP Messages have two types the Request Message and the Response Message. The client sends the Request Message and the Server sends the Response Message.

Request Message

Here's an Example of a Request Message

POST https://www.reddit.com/timings/perf HTTP/2.0
Host: www.reddit.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.reddit.com/explore
Content-Type: application/json
Content-Length: 759
Connection: keep-alive
{"time":"djsleee89392873hfksnnv0usjoqwuururur239309309u3u3="}

The request message is divided into three parts.

  • Starting Line
  • Headers
  • Body

The Starting Line is the first line of the code which is the following

POST https://www.reddit.com/timings/perf HTTP/2.0

The first part of the starting line is called the method. If you've been testing REST APIS before then you must have heard of GET, DELETE, PATCH and PUT. They're examples of HTTP methods and each one has a particular purpose. Followed by the HTTP Method is the URL which is the https://www.reddit.com/timings/perf it is sometimes called Endpoints. And lastly the HTTP Version and on this case it's HTTP/2.0. It's important to remember that the Starting Line is only the mandatory part of an HTTP Request. without the Starting Line it's impossible to make a connection to the server.

The Next part of the Request Message is called the Headers. It's easy to spot them because they immediately follow the Starting Line and each Header is separated by new line following a Key Value Pair Format. To point out here's the Headers of the HTTP Requests that I've made above.

Host: www.reddit.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://www.reddit.com/explore
Content-Type: application/json
Content-Length: 759
Connection: keep-alive

Headers are optional and you can send a request without them but in most cases web services and websites in the modern web require mandatory information that you can find in headers. A good example is the Content-Type Header with the value application/json. RESTFUL web services consumes information in JSON format and without that header, depending on how the web application was programmed you might encounter some issues if you did not supply the correct Content-Type.

The last part of the Request Message is the Body. Similar to Headers, it's also optional. You would usually supply JSON data or sometimes  in the body if you are going to make a POST request because POST requests is usually associated with creation of new Data. From the example above you'll see at the bottom this part which is an example of a request body. {"time":"djsleee89392873hfksnnv0usjoqwuururur239309309u3u3="}

Response Message

The response message is almost identical to the Request Message. An example of a Response message is what you see below.

HTTP/2.0 200 OK
content-type: application/json
accept-ranges: bytes
date: Fri, 02 Nov 2018 05:16:52 GMT
via: 1.1 varnish
x-served-by: cache-sin18025-SIN
x-cache: MISS
x-cache-hits: 0
x-timer: S1541135812.959098,VS0,VE231
cache-control: private, max-age=3600
server: snooserv
content-length: 2
X-Firefox-Spdy: h2

Response Message has similar Structure to Request Headers the only difference is that instead of the Starting Line, Response message has Status Line. The first line of the status message is called the Status Line and it includes the HTTP Version the Status Code and the Status Description respectively.

HTTP/2.0 200 OK

The two remaining parts  would be the Response Headers and the Response Body which is similar in structure and function to a Request Body and Request header. Just remember that the function of a Response Message caters the client that uses it, client's can be a web browser, another web service that integrates with it or anything that calls the endpoint.

Percent Encoding and Query Parameters

Now that you've got some basic understanding with Requests Messages I'd like to add some additional pointers that you'll surely encounter when we're developing our automated tests. One thing that I did not mention on the Request/Response Message section is Percent Encoding. So what about percent encoding???

Well, percent encoding is the representation that the HTTP protocol use when reading the URLs. What that means is that given a string of text we translate that text again to another form so that the protocol will not encounter errors or to avoid miss representation. Percent encoding is used because there are characters that would cause confusion when processed by servers. A good example are spaces where in they are converted to + when you percent encode a URL. That's why you'll see URLS that looks like the one below.

https://www.google.com/search?hl=en&source=hp&ei=_CrdW6PHDNTk-AaegqrABg&q=What+is+an+Encoding&oq=What+is+an+Encoding&gs_l=psy-ab.3..35i39k1j0l6j0i22i30k1l3.1832.4391.0.4576.20.19.0.0.0.0.172.2281.2j17.19.0....0...1c.1.64.psy-ab..1.19.2274.0..0i67k1j0i131k1

Another thing that I want to point out is Query Parameters. Think of it as passing data through variables in the url.  If you'll notice closely on the URL above you'll see a question mark ? which means that you'll be using query parameters and that you'll want to assign data on some variables. On the case of the above example hl=en means that you want to assign en to hl. If you want to assign to multiple variables you need to use the & sign. Which is why you'll see hl=en&source=hp&ei=_CrdW6PHDNTk-AaegqrABg& and that means assigning values to hl, source and ei.

Sessions

HTTP is stateless. It doesn't natively track your status, it doesn't know your transactions and it just processes the request and returns the response. It actually sucks because programmers have to find ways to implement a solution of tracking your state through the use of sessions. Sessions can be stored via cookies and it's a way for developers to validate if you're logged in or if you are allowed to do a certain action. Sessions are kinda complicated because developers have different implementations on how to track state. The important thing to remember is that HTTP is stateless by design and Developers tracks your state through the use of sessions.

Next Steps...

Now that you have a rough understanding about HTTP Messages let's go to the real deal and tackle how to implement REST Calls using Python's Request Package. I'll be posting a new article about that a few days from now. For the mean time you can review on how we did our setup and of course a nice introduction to Feature Files and Steps Implementation