Building the Igor REST API

This weekend marks the end of my GSoC with OSUOSL. We now have a full-fledged REST API, a hierarchical CLI and more than a couple of patches to pyipmi implementing IPMI commands. I have already talked about the CLI before the midterm evaluation. In this post, I will focus on the Igor REST API that the CLI delegates all the heavy lifting to.

Design

The decision to split the overall project into two pieces (the REST API and the CLI) was motivated by the need of having multiple possible clients: Android applications, web applications and curses interfaces, to name a few. Having this server-client dichotomy helps us write the core logic, error handling and tests once for the server, and have our clients be really thin. This is also the philosophy embodied by the Heroku CLI.

The REST API serves a dual purpose: to manage machines, users and regulate their interactions, and to call the actual IPMI commands. The following diagram depicts a bird’s-eye view of the architecture:

Igor Architecture

Diagram source

Machines and users are standard resources in this architecture, accessible via endpoints that accept the GET, PUT, POST and DELETE verbs. In the back-end, both machines and users are represented by their own database models.

The API regulates interactions between the two via permissions: a many-to-many relationship between users and machines. An IPMI command can be initiated by a user targeting a machine only if such a permission exists between the user and that machine. Permissions can be granted and revoked via PUT and DELETE requests to the permissions endpoint.

The API is implemented as a Flask application using flask-restful. flask-restful wraps Flask’s method-based dispatching and pluggable views, providing convenient abstractions for REST applications.

Users, Machines, Authentication and Authorization

Access regulations are of two kinds: authentication and authorization. Authentication performs endpoint-wide blocking and ensures the user has valid credentials. Authorization checks if users are permitted to access the specific machine they are attempting management or IPMI operations on.

Both authentication and authorization are implemented as decorators. Protecting an endpoint then simply involves decorating the flask-restful resource class for that endpoint. The only tricky bit is the order of applied decorators: we want authentication to take place before authorization, and must hence place authorization before authentication in the decorators array (decorators are applied outward-in).

Separate modules each implement the user management endpoints, machine management endpoints and permissions endpoints. The routes module collects these and the IPMI operation endpoints into a list and registers them with the application. Having this module separate helps us look at a map of all the available endpoints. The endpoint list also helps us in testing, as we will see in an upcoming section.

IPMI Operations

IPMI operations are implemented via my fork of pyipmi. pyipmi wraps calls to ipmitool and freeipmi, parses their response and returns them as Python data structures.

Each command is called within the methods of a resource, which is a subclass of IPMIResource. IPMIResource obtains the target hostname that is set by the authorization decorator and creates the necessary pyipmi objects to access that machine. Each command is called via a wrapper that makes the call, obtains the response and returns a HTTP 400 (bad request) if the command fails, otherwise passes through to the rest of the method body. Since most error-handling is delegated to ipmitool, the calls are thin and easily adaptable in case the underlying command changes. All the IPMI operation endpoints are implemented in a single module.

Unfortunately, the company maintaining pyipmi shut down and the project is now unmaintained. Nevertheless, it appears to be the most full-fledged Python-IPMI interface available right now. The IPMI lan alert command, sel delete command and sensor commands were contributed back to pyipmi during the course of this work.

Tests

Tests were implemented with nose as the test collector/runner, and flask-testing providing a nice abstraction over Python’s built-in unittest module that makes it convenient to test Flask applications. A pull request contributing optional test-failure messages was merged into flask-testing during the course of this work.

Each test is set up by creating a new database instance with a single root user. The current bunch of tests consists of four modules: authentication tests, user management tests, machine management tests and permission tests.

The authorization tests take advantage of the resources list in the routes module described earlier. Each endpoint (except the root at /) is verified to be inaccessible without authentication. The user and machine management tests verify CRUD operations on the respective endpoints, in addition to additional cases such as adding users/machines that already exist. The permission tests verify the addition and removal of users-machines relationships via the API endpoints.

Future

The most important extension to this work is closely tied to the development of one of the following pure Python implementations of the IPMI protocol: xcat python-ipmi, kontron python-ipmi or pyghmi. Once these libraries reasonably cover the complete IPMI protocol, they can be integrated into the Igor REST API. Both the API and CLI architecture would remain the same with this change.

This can be followed by a good implementation of serial-on-LAN (SOL), which was planned for this project but abandoned due to poor fit with the REST protocol, since implementing SOL over REST would require maintaining server-side state for each user. The SOL is asynchronous and full-duplex by nature, and per-user queues could (not very elegantly) replicate this functionality over REST. The proposed architecture was as depicted below:

Igor SOL Design