pwrkap -- Energy Use Monitor and Power Cap Enforcement Tools
version 7.30 (2009-01-12)

Written by Darrick J. Wong.
(C) Copyright IBM Corp. 2008-2009

This software is covered under the GNU GPL v2; see COPYING for details.

Overview
--------
This document attempts to describe the structure of the the pwrkap software.
There are two big parts to this program--device and power meter enumeration
and the grouping of those devices into power domains; and the part that builds
a table describing the effects of device power-state changes on the power
consumption of that power domain.

Power Meters
------------
All power meter objects must implement the methods described in the power_meter
class in pwrkap_data.py.  For a reference implementation, see syfs_meter.py.
Obviously, power meter driver implementations will vary.

Managed Devices
---------------
All devices that are to be managed by pwrkap must implement the methods of
the device class in pwrkap_data.py.  For a reference implementation, see
the file cpu_device.py.  Obviously, device driver code will vary.

Discovery
---------
Code module, power meter, device and power domain discovery are all handled
through discovery.py.  Discovery for pwrkap is done in a much different
way than it has been done in the past.  All code modules to be loaded should
be listed in PWRKAP_DRIVERS; every module listed in that array is imported when
the program starts.  Each code module should have a non-indented code snippet
that adds discovery functions to the lists PWRKAP_DEVICE_DISCOVERY,
PWRKAP_METER_DISCOVERY, or PWRKAP_POWER_DOMAIN_DISCOVERY.

When discovery is done, all functions registered in those lists will be called
in succession.  First devices are discovered, then power meters, and finally
power domains are created to map power meters to the devices that the meter
measures.

Device discovery has two parts.  First, the devices should be enumerated and
put into PWRKAP_DEVICES.  Second, device control domain information must be
discovered.  This means that all devices with their own set of controls should
be put into a device_domain, one domain per device.  For sets of devices that
share the same controls, all the devices should be put into one device_domain.
Finally, all device_domain objects should be put into the PWRKAP_DOMAIN_DEVICES
list.

Power meters are discovered and should be put into PWRKAP_POWER_METERS.
Energy meters are discovered and should be put into PWRKAP_ENERGY_METERS.

Finally, meters and device control domains are united in the power domain
discovery function.  Each power domain is created with links to three data
structures--the list of device_domain objects that are bound to the power
meter, something called an identical device profile domain, and the power meter
itself.  The fourth argument is the initial power cap for the domain.  All
power domains should be put in PWRKAP_POWER_DOMAINS; all devices, device control
domains, and meters claimed by the power domain should be removed from
PWRKAP_DEVICES, PWRKAP_DEVICE_DOMAINS, PWRKAP_ENERGY_METERS, and PWRKAP_POWER_METERS,
respectively.

The identical device profile domain, or idomain, identifies devices that have
identical power use profiles.  This enables some extra flexibility in both the
training program as well as the cap enforcement algorithm.  Instead of having
to iterate through all power states of all devices in a domain for training,
the program uses the idomain data to test one device in the idomain and apply
its observations to all other devices in the idomain.  This drastically reduces
training time as well as shrinking the size of the power use effects table.
Please also note that even a device with a unique profile needs to be placed
in an idomain by itself.

The power domain discovery code are critical to correct operation of pwrkap!
Currently, there are two discovery paths--one for certain IBM systems, and a
simple one that lumps everything it finds into one power domain provided there
is only one power meter.  Systems with multiple domains will need to provide
their own power domain discovery logic.

Inventories and Snapshots
-------------------------
Nearly all the data objects involved with pwrkap have two methods that have
not yet been discussed--inventory() and snapshot().  These two functions are
described below.  However, a discussion of call flow may be useful.

Generically, the inventory() function returns a description of the static
characteristics of the system--a hardware identifier of the power meter, the
list of supported CPU frequency states, etc.  It is assumed that changes in
inventory do not happen within the life of the daemon; thus, a change of this
sort should result in the daemon discarding all training data and starting
anew.  Typically this data are used to estabish machine capabilities.

The snapshot() function, then, returns a picture of dynamic system state at
the time of invocation.  These items should be fairly volatile and are used to
compute the current state of the machine and where it should go next.

pwrkap's controller object (discussed later) will call the inventory() or
snapshot() methods of power domain objects; it is the duty of these objects to
make the appropriate calls to the power meter and the control domain objects;
the control domain objects will (eventually) call the methods in the device
drivers.  After that, the power domain object will compile the subordinate
objects' inventory or snapshot data into a single report and return it.

Relating Power Use to Device Utilization and Power States
---------------------------------------------------------
The heart of pwrkap is a four-dimensional table that enables pwrkap to
guess what kind of impact a change in a device's power state will have on the
power domain's power use.  This large-ish array is indexed like this:

transition_table[idomain][dev_use][p_state][p_state] = power_change

The idomain field describes an identical device profile domain as outlined
above.  The dev_use index cuts the rest of the table into utilization buckets,
because changes between power states of certain devices (CPUs in particular)
have different effects on power use depending on the device's utilization.
The last two fields are used to index the current power state of the device
and a proposed new power state.  The value stored in the table is the average
impact on power use given the four indexing factors.

This table can contain empty cells.  By convention, the two p-state indices
are always ordered with the lower of the two coming first, as it is assumed
that transitions are commutative, i.e. a->b => c and b->a => -c.

Training
--------
In the event that pwrkap finds itself lacking data relating power use to
device utilization and power state, a training algorithm is needed to observe
the necessary data to enforce the cap effectively.  The code to do this can be
found in transitions.py.

The training algorithm is quite simple:

1. Load down the system so that the "100% Utilization" buckets are filled.
   This probably requires outside attention.
2. Set all devices to their lowest power state.  This is done partly to avoid
   overloading the power mains and partly to work around cpufreq bugs in Linux.
3. In each power domain, identify one device from each idomain.
4. For all possible combinations of device and power state, take a snapshot
   of the power domain and process the snapshot.  See Processing a Snapshot
   below for details.

When the daemon is offline, the power domains and their transition tables are
written to disk via python's cPickle mechanism.  When the program loads, the
inventory of the saved data is compared to what is discovered; if there is a
mismatch, the saved data are discarded and the training algorithm is started.

Processing a Snapshot
---------------------
When snapshots are taken, either during training or during normal operation of
the pwrkap daemon, it is useful to augment the transition database with the
data that are being collected, thereby enabling the software to adapt to an
environment that changes over time.  Each power domain remembers the past few
snapshots that were taken of that domain.  When a new snapshot is taken, it is
compared to every snapshot currently in the retention buffer.  If it can be
shown that the only difference between the old and new snapshots is a change
in power state one device in an idomain, the differences in states and in power
use are noted in the transition table.

Enforcing Caps
--------------
Each power domain gets its own thread to run a control loop.  The operation of
this control loop is as follows:

1.  No more than once every MEASUREMENT_PERIOD seconds, take a snapshot of the
    power domain.
2.  If less than ENFORCEMENT_INTERVAL seconds have passed since the last
    enforcement attempt, go to sleep and restart at step 1.
3.  Compute the difference between the domain's power cap and power use.
4.  If use is less than cap, find all transitions that increase performance.
    If use is more than cap, find all transitions that decrease use.
    if they are equal, go back to step 1.
5.  Find the transition with the most positive performance increase for the
    most negative increase in power use.
6.  Implement the power state change specified by the transition and return
    to step 3.

Talking to Clients
------------------
Communication with the pwrkap client is achieved through three components--
controller, sockmux and lazy_log.  The controller accepts incoming commands
from sockmux and modifies the power domains as necessary.  The lazy log receives
new snapshots from the domains as they run through their control loops.  While
forwarding new log entries to the sockmux, the lazy log also retains the last
few log entries, which are sent to new clients.  Finally, the sockmux dispatches
incoming commands to the controller, sends data from the lazy log to all
clients, and deals with the underlying socket plumbing.

Communications across sockets are done with Python objects in cPickle format.
When a client first connects to a pwrkap server, it is sent a list of
power domain inventories and the log entries retained in the lazy log.

The pwrkap Client
-------------------
The pwrkap client connects to the server and receives a list of power domain
inventories and recent log entries.  After that, the client should listen for
incoming power domain snapshots; it may also send commands to the pwrkap
daemon.

The CLI client dumps all incoming data to the console in raw format.  Anything
typed into it is cPickle'd and send to the pwrkap server.

The GTK client is a bit more sophisticated.  In addition to being able to
watch multiple pwrkap daemons simultaneously, it employs GTK controls to
switch the view between power domains (or aggregates of power domains).  It
also interfaces with matplotlib to draw a graph of the past few minutes' of
power cap, power use, and overall domain utilization.  Most of the client code
is fairly straightforward and not worth mentioning here, with the exception of
the domain aggregator.

Domain Aggregator
-----------------
The domain aggregator (found in ui_data.py) is used by the GTK client to roll
the power cap, use and domain utilization data of many domains into a single
report.  It can also be used to set the power cap of a large number of
machines; in that case, the power cap of each machine is scaled up or down
to maintain the same proportion between domain power cap and overall power cap.

To aggregate data for graphing, the historical data of several power domains
must be blended together.  Individual samples are taken in timestamp order from
each domain; the graphing interval is then split into small ranges of time.
For each time range, the power use and cap of each domain are added and the
utilizations are averaged to create the aggregate's profile for that time stamp.
This much smaller set of data is then graphed.

Wire Protocols
--------------
The wire protocol in use between the server and client software is a simple one;
both encode python objects in pickle format and send that across the wire.  The
handshake protocol looks approximately like this:

1. Management client connects to pwrkap daemon.
2. pwrkap daemon sends a list of tuples of the format (domain name, inventory).
3. pwrkap daemon replays the last few log entries one by one.  Log entries
   are of the format (UTC timestamp, (domain name, snapshot))
4. pwrkap daemon sends (UTC timestamp, "live") to indicate that all data after
   this point are live.  The server's UTC timestamp can be used to synchronize
   with the client's clock, since all log entries have UTC timestamps.
5. pwrkap daemon sends live snapshots of the form (UTC timestamp, (domain name,
   snapshot))
6. Client software can send commands as an array of strings at any time.
