2008 Summer School

Calling Web Services: RESTfully and SOAPfully

As more is done in a distributed fashion, it is important to learn how to interact with services using standard protocols. In this exercise, we will explore interaction with REST-type services and SOAP-type services through the Python language. The NVOSS software package contains all the infrastructure we require.

REST Service Access

REST (Representational State Transfer) services access endpoints using defined HTTP methods (GET, POST, PUT, DELETE). These methods are then mapped by the server to specific actions. Arguments can be sent either as part of the URL in the GET and DELETE cases or as part of the body of the message in teh POST and PUT cases. The response to the request will contain a status code and optionally content in the body of the message which can be parsed for useful information. The NVOSS software package contains an example REST service written in python that we will use to examine the use of python in REST type communication.

Access the code

Demonstration code has been writen and resides in the python branch of the NVOSS software distribution. Follow Listing 1 to import the code. The Listings are written in a *nix style and may need modification on other operating systems.

> source $NVOSS_HOME/bin/setup.csh
> cd $NVOSS_HOME/python/src/webservice/
*Edit ImageAnnotationService.py.  Set imgdir and commentdir to point to valid paths on your filesystem.
> python ImageAnnotationService.py
[19/Aug/2008:16:20:30] ENGINE Listening for SIGHUP.
[19/Aug/2008:16:20:30] ENGINE Listening for SIGTERM.
[19/Aug/2008:16:20:30] ENGINE Listening for SIGUSR1.
[19/Aug/2008:16:20:30] ENGINE Bus STARTING
[19/Aug/2008:16:20:30] ENGINE Started monitor thread '_TimeoutMonitor'.
[19/Aug/2008:16:20:30] ENGINE Started monitor thread 'Autoreloader'.
[19/Aug/2008:16:20:30] ENGINE Serving on 127.0.0.1:8080
[19/Aug/2008:16:20:30] ENGINE Bus STARTED

> cd http_client
> python HTTP_client.py
200 OK
200 OK
200 OK
200 OK
200 OK
My distance is(GET method): 74.343850 Mpc/h
200 OK
My distance is(POST method): 74.343850 Mpc/h
Listing 1
For interactive access to the methods in HTTP_client.py:
> python
>>> import HTTP_client as hc

Try it out!
>>> file = "/home/simon/nvoss/python/bin/whirlpool.jpg"
>>> uri = "/whirlpool"  
Note:  The leading slash in the previous line is important
>>> hc.PUTImage(file, uri)
200 OK
>>> hc.POSTComment("This is a pretty picture of M51", uri)
200 OK

Listing 2

Your browser does a GET call by default so we can use python calls or our browser to look at the results of Listing 2. If you used the same identifier as listed, you can go to http://localhost:8080/whirlpool to see the uploaded image and comment.  

How does it work

Python has built in methods to take care of construction of HTTP headers. POST and PUT can contain bodies in which case the Content-length keyword is automatically set in the header. Get and Delete don't have bodies. The demonstration code uses the built in httplib python package. Let's look at the code to "PUT" an image.

from httplib import *
baseurl = "localhost:8080"                          Location of the Service
def PUTImage (file, uri):                           This takes the path to the file to upload and some identifier for it
  binarr = open(file, 'rb').read()                  Read the file into a binary array
  connection = HTTPConnection(baseurl)              Open a connection to the service
  connection.request("PUT", uri, binarr)            Use PUT method to path set by uri and set the body to contain binarr
  response = connection.getresponse()               Get the response so that the status can be checked
  print response.status, response.reason            Print out the response status (should be 200 OK)

Listing 3

That's all there is to it. The request method can be set to any of the defined HTTP request methods. As shown in the GETPage method, if the service returns useful information other than the status, the read() method can be called on the response to get the content.

What about other REST service?

Johns Hopkins University (voservices.net) has made available several services that are accesible via several protocols (GET, POST, or SOAP). Let's look at the luminosity distance calculator service. Unlike most of the methods in our image annotator, the distance calculator takes several arguments and returns information in the body of the response. See http://voservices.net/Cosmology/ws_v1_0/Distance.asmx?op=Luminosity for descriptions of the messages.

The GET case

Most REST services that use the GET method use the convention that the service location is defined by the path following the base URL with arguments following a ? separated by &. The luminosity distance method takes four arguments: redshift, hubble parameter, Ωm, and ΩΛ. These arguments will be appended in name/value pairs to the path to the services. For example, try this call to the distance calculator:
http://voservices.net//Cosmology/ws_v1_0/Distance.asmx/Luminosity?z=0.5&hubble=0.70&omega=0.3&lambda=0.7

We can do this programmatically in python using httplib. We will use the GET method and construct the URL by hand.

>>> args = hc.urlencode({'z':'0.5','hubble':'0.7','omega':'0.3','lambda':'0.7'})     urlencode automatically adds the & and = characters
>>> print args
hubble=0.7&z=0.5&omega=0.3&lambda=0.7
>>> ldist = hc.GETLdist(args)
200 OK
>>> print ldist
2832.93823

Listing 3
Let's look inside the method that calls the calculator:
from httplib import *
from urllib import *
import elementtree.ElementTree as ET                        We need ElementTree to parse the returned XML
distanceurl = "voservices.net"                              The base URl of the service
serviceuri = "/Cosmology/ws_v1_0/Distance.asmx/Luminosity"  Path to the method of interest
def GETLdist (args): 
  connection = HTTPConnection(distanceurl)                  Open a connection to the server
  uri = serviceuri + "?" + args                             Concatenate the service path and arguments with a ?
  connection.request("GET", uri)                            Issue the GET request
  response = connection.getresponse()                       Get the response so that status can be checked and output parsed
  print response.status, response.reason
  str = response.read()                                     Get a string representation of the response
  doc = ET.fromstring(str)                                  Parse the XML into a tree
  return float(doc.text)                                    In our case the result is in the root element, so get text.  More parsing can be done for more complicated returns.

Listing 4

The POST case

This service also allows for communication via the POST method. This is useful if the inputs are coming from a web form. The difference is that instead of encoding the arguments in the URL, we send the agruments in the body of the POST request.

Try the example code:
>>> args = hc.urlencode({'z':'0.5','hubble':'0.7','omega':'0.3','lambda':'0.7'})
>>> print args
hubble=0.7&z=0.5&omega=0.3&lambda=0.7
>>> ldist = hc.GETLdist(args)
200 OK
>>> print ldist
2832.93823

Listing 5

Good, it worked. How does the code for calling via POST differ from that for calling via GET?

from httplib import *
from urllib import *
import elementtree.ElementTree as ET                               Once again we need ElementTree to parse the output
distanceurl = "voservices.net"                                     The server url
serviceuri = "/Cosmology/ws_v1_0/Distance.asmx/Luminosity"         Path to the service
def POSTLdist (args):
  connection = HTTPConnection(distanceurl)                         Get connection to the service
  headers = {"Content-type": "application/x-www-form-urlencoded"}  We need to set the Content-type header keyword correctly.  See message description.
  connection.request("POST", serviceuri, args, headers)            Issue the POST request with the path to the service, set args to the body, and add the extra header value
  response = connection.getresponse()                              Get the response
  print response.status, response.reason                           Check status
  str = response.read()                                            Read response body
  doc = ET.fromstring(str)                                         Parse XML return
  return float(doc.text)                                           Return value of root node

Listing 6

SOAP Service Access

SOAP web services use a transport protocol (usually HTTP in our case) to send verbose XML messages adhering to the SOAP specification. Both request and response are sent as SOAP XML documents. As such, SOAP is inherently platform and language independent and lends itself to being extensible. One attractive aspect of SOAP web services is that they can be described using WSDL (Web Service Description Language). Using the WSDL, many language can auto generate the code for writting and reading SOAP messages from a particular service. The NVOSS software package contains a demonstration server and client.

Access the code

Follow Listing 7 to set up the server and run the test client. Once again, the examples are done in a *nix way asuming tcsh as the shell.

> source $NVOSS_HOME/bin/setup.csh
> cd $NVOSS_HOME/python/src/webservice
> python AbsoluteMagnitudeService.py
[20/Aug/2008:11:35:34] ENGINE Listening for SIGHUP.
[20/Aug/2008:11:35:34] ENGINE Listening for SIGTERM.
[20/Aug/2008:11:35:34] ENGINE Listening for SIGUSR1.
[20/Aug/2008:11:35:34] ENGINE Bus STARTING
[20/Aug/2008:11:35:34] ENGINE Started monitor thread '_TimeoutMonitor'.
[20/Aug/2008:11:35:34] ENGINE Started monitor thread 'Autoreloader'.
[20/Aug/2008:11:35:34] ENGINE Serving on 127.0.0.1:8080
[20/Aug/2008:11:35:34] ENGINE Bus STARTED

> cd webservice_client
> wsdl2py --lazy -b "http://localhost:8080/AbsMagService?wsdl"
> python AbsMagClient.py
Hipp_ID Vmag Plx V_absmag 
1168 4.79 10.01 -0.207829612603 
71939 8.81 14.31 4.5881981688 
70890 11.01 772.33 15.4490145234 

Listing 7

How does it work?

SOAP interaction is sometimes slightly more complicated that REST type because, in general, complex data types are being used in both the request and response phase. Luckily, most code generation tools will generate the code to manipulate the complex types. In the case of this absolute magnitude calculator, both the input and output are in VOTable format. This means that we must build a VOTable to send and parse the VOTable that is returned. The methods for dealing with the VOTable are attached to the Request and Response objects.

Let's get started...
from AbsMagService_client import *
import sys

def callSVC():
  loc = AbsMagServiceLocator()                   First get the location of the service
  tracefile = open('message.soap', 'w')          Optionally, set a file to contain the response and request
  kw = { 'tracefile' : tracefile }               
  svc = loc.getAbsMagService_PortType(**kw)      Get instances of the generated client code
  wsreq = calcAbsMagRequest()                    Get an instance of the Request object
  vot = wsreq.new_VOTABLE()                      Get instance of the VOTABLE object
  .
  .                                              Read values into a VOTABLE object
  .
  wsreq.set_element_VOTABLE(vot)                 Assign the VOTABLE to the Request object
  response = svc.calcAbsMag(wsreq)               Call the service and get the response back
  vot_ret = response.get_element_VOTABLE()       Get the VOTABLE from the response and operate on it.

Listing 8

That's really all there is to it. The methods exposed by SOAP services are generally wrapped to look like regular method calls. In this aspect, SOAP interaction is generally more transparent than REST calls in that it is not usually necessary to compose the message as this is done by the generated code.


The NVO Summer School is made possible through the support of the National Science Foundation.