I’m messing around with using python for testing REST web services.

Python 2

I quickly found that Python 2 has a messy tangle of URL and HTTP libraries

There are several low-level libraries:

1. urllib
2. urllib2
3. httlib
4. httplib2

You’ll probably end up using all or most of them together. There is some overlapping functionality, and several bugs & unimplemented features. So while you might be able to do some things with urllib.urlopen() you won’t be able to do others unless you use urllib2.urlopen(). But you still need urllib for urlencode() for instance.

See this post on stackoverflow.com about urllib vs urllib2.

Here’s an example GET request for JSON with Basic Auth over HTTPS using urllib2:

import urllib
import urllib2
import base64

url = "https://example.com/rest/resource"
authKey = base64.b64encode("username:password")
headers = {"Content-Type":"application/json", "Authorization":"Basic " + authKey}
data = { "param":"value"}

request = urllib2.Request(url)

# post form data
# request.add_data(urllib.urlencode(data))

for key,value in headers.items():
  request.add_header(key,value)

response = urllib2.urlopen(request)

print response.info().headers
print response.read()

Here’s an example using httplib:

import httplib
import urlparse

domain = urlparse.urlparse(url).netloc
connection = httplib.HTTPSConnection(domain)
connection.request("GET", url, headers=headers)
response = connection.getresponse()

print "status: " + str(response.status), response.reason
print response.getheaders()
print response.read()

And here’s an example using httplib2:

import httplib2</code>

http = httplib2.Http(disable_ssl_certificate_validation=True)
headers, content = http.request(url, "GET", headers=headers)

print headers
print content

httplib2 is actually a third party module, and on Python 2.6 & 2.7 you need to manually upgrade it to get it to work. If you get the message “TypeError: a float is required” — see the comments in the examples for the fix. Full traceback is included below. I was able to get it working with a simple pip install –upgrade httplib2 though.

An issue you might run into with httplib2 when testing is an invalid ssl cert (or perfectly valid self-signed cert.) You need to set OpenSSL::SSL::VERIFY_NONE. You can do this by specifying in the constructor:

http = httplib2.Http(disable_ssl_certificate_validation=True)

See the following for reference:

http://forums.ocsinventory-ng.org/viewtopic.php?id=980
http://viraj-workstuff.blogspot.com/2011/07/python-httplib2-certificate-verify.html

Python 3

Python3, in an attempt to fix this has created a new module http.client that implements httplib and urllib combines features urllib and urllib2 in python2 and uses http.client, but chances are you’ll still need to use the old ones. (See “Dive Into Python” Appendix A)

• http.client
• urllib.request

diveintopytyhon3.org also has good documentation for using web services

But then Python3 goes and completely screws up strings all in the name of Unicode political correctness.

Here’s the example using python 3 and http.client:

import base64
from urllib.parse import urlparse</code>

authKey = base64.b64encode("username:password".encode('utf-8')).decode('utf-8')
headers = {"Content-Type":"application/json", "Authorization":"Basic " + authKey}
domain = urlparse(url).netloc

import http.client

connection = http.client.HTTPSConnection(domain)
connection.request("GET", url, headers=headers)
response = connection.getresponse()

print("status: " + str(response.status), response.reason)
print(response.getheaders())
print(response.read().decode())

And with urllib.request in python3:

import urllib.request

request = urllib.request.Request(url)

for key, value in headers.items():
  request.add_header(key, value)

response = urllib.request.urlopen(request)

print(response.getheaders())
print(response.read().decode())

finally, with httplib2 on python3:

import httplib2

http = httplib2.Http( disable_ssl_certificate_validation=True)
http.add_credentials(username, password)
response, content = http.request(url, "GET", headers=headers)

print(response)
print(content)

There is actually a bug (logged only a few days ago) that prevents httplib2 from working using HTTPS with a self signed cert

There are several “REST” libraries worth looking at that I will be looking at next:

1. python-rest-client
2. siesta
3. requests

python-rest-client uses urllib2 and httpclient2 and has good examples of how to use them. It also includes a Google App Engine version which wraps the GAE url fetch functionality.

You can download the full source code for my python REST client examples and the python 3 examples at

http://www.one-shore.com/aaron/examples/python/webservices

Error Traces

Error in httplib2 for Python 2.6 & 2.7:

Traceback (most recent call last):
File "C:\dev\projects\pythonwebservices\CheckFlightStatus.py", line 21, in
main()
File "C:\dev\projects\pythonwebservices\CheckFlightStatus.py", line 17, in main
fs.check("AS", "859", str(date.today()))
File "C:\dev\projects\pythonwebservices\FlightStatus.py", line 36, in check
self.getResponseUsingHttplib2(url)
File "C:\dev\projects\pythonwebservices\FlightStatus.py", line 68, in getResponseUsingHttplib2
headers, content = h.request(url, "GET", headers=self.headers)
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 1050, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 854, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 823, in _conn_request
conn.request(method, request_uri, body, headers)
File "C:\dev\Python\2.7.2\lib\httplib.py", line 955, in request self._send_request(method, url, body, headers)
File "C:\dev\Python\2.7.2\lib\httplib.py", line 989, in _send_request
self.endheaders(body)
File "C:\dev\Python\2.7.2\lib\httplib.py", line 951, in endheaders
self._send_output(message_body)
File "C:\dev\Python\2.7.2\lib\httplib.py", line 811, in _send_output
self.send(msg)
File "C:\dev\Python\2.7.2\lib\httplib.py", line 773, in send
self.connect()
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 736, in connect
sock.settimeout(self.timeout)
File "C:\dev\Python\2.7.2\lib\socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
TypeError: a float is required

Self-signed SSL certificate validation problem in httplib2:

Traceback (most recent call last):
File "C:\dev\projects\pythonwebservices\CheckFlightStatus.py", line 35, in
response, content = http.request(url, "GET", headers={"Authorization":token})
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 1436, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 1188, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 1123, in _conn_request
conn.connect()
File "C:\dev\Python\2.7.2\lib\site-packages\httplib2\__init__.py", line 911, in connect
raise SSLHandshakeError(e)
SSLHandshakeError: [Errno 1] _ssl.c:503: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

Error using httplib2 on Python 3 with self signed certificate:

    File "C:\dev\projects\python3webservices\src\example\usinghttplib2.py", line 28, in
    response, content = http.request(url, "GET", headers=headers)
  File "C:\dev\Python\3.2.2\lib\site-packages\httplib2\__init__.py", line 1059, in request
    self.disable_ssl_certificate_validation)
  File "C:\dev\Python\3.2.2\lib\site-packages\httplib2\__init__.py", line 775, in __init__
    check_hostname=True)
  File "C:\dev\Python\3.2.2\lib\http\client.py", line 1086, in __init__
    raise ValueError("check_hostname needs a SSL context with "
ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED