Tabling of the 2007-2008 Departmental Performance Reports and Canada's Performance Report

The 2007-2008 Departmental Performance Reports (DPR)and the 2007-08 Canada's Performance Report were tabled on February 5, 2009.

Departmental Performance Reports are reports written by departments and agencies at the end of the fiscal year. They describe what the organization has achieved (it's performance) and how it performed compared to its plans and goals.

Canada's Performance Report is basically a chapeau piece to the Departmental Performance Reports, and tries to combine the performance of all the departments and agencies to create a performance report for the government as a whole. Where DPRs are historically based mostly around what the department achieved, Canada's Performance Report gives a much more social perspective to the results of government spending.

2007-2008 Departmental Performance Reports: http://www.tbs-sct.gc.ca/dpr-rmr/2007-2008/index-eng.asp

Canada's Performance Report 2007-08: http://www.tbs-sct.gc.ca/reports-rapports/cp-rc/2007-2008/cp-rctb-eng.asp

Updated MRRS Policy

There has been an update to the Policy on Management, Resources and Results Structures (MRRS). The updated policy took effect on December 20, 2008.

The updated policy is available here:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=14252§ion=text

The old policy is available here:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12412§ion=text

Overall, I would say the new update improves the policy. What I consider the core of the policy, the core requirements which used to be under 7.1, and are now under 6.1.1 haven't changed in the essence of their meaning.

The update adds more responsibilities for Deputy Heads related to the implementation of the policy, keeping the MRRS up-to-date, following proper procedures for updates, etc.

The updated policy also has a new section, 7. Consequences, describing consequences for untimely or unsatisfactory implementation of the policy.

Aggregating Indicator Scores

To measure the performance of your organization in a certain area, you will typically use a set of indicators. These indicators may or may not cover the entire area you are trying to measure, may contain indicators of short-term or long-term progress, etc. Anyhow, you'll have a set of indicators that you have chosen to represent an area of management, a process, an activity, etc.

So indicators are a set of metrics. You may have something like this to measure client service:

Indicator	Actual Value	Target
Percentage of pizzas delivered within 30 minutes	90%	100%
Percentage of calls answered within 2 minutes of entering the queue	80%	100%

Now, to get an aggregate score for client service, you could just take the average of the 2 indicators, that would give you (90+80)/2=85. However, you may decide that the indicators don't all have the same importance, the so they shouldn't all have the same weight. Let's say people hate waiting in a telephone queue, but won't notice if there pizza is 2 minutes late. In that case, the indicator for call wait time is more important, so we'll give it a weight of 70%, and we'll give a weight of 30% to the pizza delivery time. That would give us a score of (90*0.3)+(80*0.7)=27+56=83.

A few notes on this:

be careful of the units you use, in the example, we used 2 percentages with the same target, so we know they'll be fairly close and that they are fairly comparable. But if you were measuring something like the number of units sold and average call wait time in minutes, your units would be too different to be compared directly. What can you do? Use the target, and compare the result to the target. That will give you 2 results in "percentage of target achieved", which can than be directly compared to one another. If you use that method, setting meaningful targets becomes essential if you want your aggregate indicator score to be meaningful and useful.

In the example, the weights used add up to 1. It doesn't necessarily have to. But having a score that has an understandable maximum (100 in this case) makes it more understandable and intuitive. The resulting aggregate indicator score in the example, is not in a particular unit: all we know is that it's maximum is 100. There are times when, because of either your indicator or target your result may exceed 100. There is nothing wrong with that, but it highlights the importance of explaining how you go about measuring your performance, and how your data should be interpreted.

Finally, defining weights is a tricky exercise, and some managers may abuse this system by assigning low weights to indicators on which they know they will perform poorly. Another aspect to consider is that you may want to assign low weights to indicators for which the results are not very reliable.

Speech from the Throne

The Speech from the Throne and the Prime Minister's speech provide the foundation for the government's legislative priorities and agenda for the current session of Parliament. These priorities should be reflected, where applicable, in a department's Report on Plans and Priorities (RPP).

Speech from the Throne: http://www.sft.gc.ca/eng/media.asp?id=1383

Address by the Prime Minister in Reply to the Speech from the Throne: http://www.pm.gc.ca/eng/media.asp?id=2318

Correlation and Causality

Correlation is not causality, they are two different concepts.

Correlation

Correlation is a relationship between variables. When the value of X goes up (or down), the value of Y goes up (or down) in a predictable way. The height and weight of a person are correlated. Their eye color and their weight is not.

Causality

Causality is a cause-effect relationship between variables. A change in the value of X is the cause of a change in the value of Y. For example, viruses make you sick. Be careful not to confuse the cause and the effect: you sneeze because you have a cold, but you don’t have a cold because you sneeze.

Proving a cause and effect relationship is difficult, as all other variables must be controlled. It is also possible for an observation to have more than a single cause; the change of the price of a stock is an example. Normally, the change in the variable causing a change in the other is observed before the change of the value of the dependent variable.

The Management Accountability Framework (MAF)

The Management Accountability Framework (MAF) is a framework used by the Treasury board Portfolio to assess the quality of management in departments. It is structured around 10 elements: Public Service Values, Governance and Strategic Directions, Policy and Programs, Results and Performance, Learning, Innovation and Change Management, Risk Management, People, Stewardship, Citizen-focused Service, and Accountability. Indicators are defined for each element and are used to measure performance in each area.

More information on the MAF is available on the Treasury Board of Canada Secretariat's website: http://www.tbs-sct.gc.ca/maf-crg/index-eng.asp

Dimensions of an Outcome

Outcomes are normally part of a performance measurement framework of one type or another. Most often, they will be used in the public sector or the non-profit sector to explain how their activities’ contribution to society. The might also be used at a lower level to measure the implementation of a strategy in the private sector. In that particular case an organization would be measuring the outcome of a strategy, although in the terminology generally used in public sector performance measurement; this would be closer to an expected result.

Regardless, the purpose of this post is to clarify a perceived ambiguity surrounding outcome levels. In performance measurement literature, different levels of outcomes are often mentioned, such as immediate, intermediate, long-term and final outcomes. The descriptions given usually revolve around time and impact on society.

However, to clearly define outcomes, they need to be perceived through at least 3 dimensions:

1. reach or societal impact,
2. time (frame, lag, or delay) and
3. attributability or responsibility

The reach or societal impact can be generally conceived of as the “societal importance or value” of the outcome. For example, “reducing the number of sick Canadians” may be an outcome, but “healthy Canadians” is a broader, and further reaching one.

The time dimension is a little more complex, because more things can be measured here. For example, an outcome could be defined as a desired end-state. In that context the time dimension would refer to the time required to bridge the gap between the current state and the desired end-state. The time dimension can also be important in a context where an organization’s action will only have an impact on the outcome after a period of time.

The attributability of an outcome for the organization or the responsibility or the organization for the outcome are also to be considered. Attributability can be defined as the amount of “credit” an organization can take for the change in the outcome. Most often, not all change in an outcome can be attributed to the actions of an organization. The concept of attributability is closely linked to the concept of causality. A change in the outcome is attributable to the organization if the organization’s actions are the cause of the change. Responsibility, however, is a different concept. Where attribution is when an organization appropriates changes in an outcome, responsibility is when an organization is made responsible for an outcome, or if you prefer, is mandated to have an impact on the outcome. However, attributability of the change in the outcome still remains to be proven of organizations with clear responsibilities. For example, the Bank of Canada has an agreement with the Government of Canada regarding target inflation rates. To a certain extent, it is responsible for the rate of inflation. The question in that case is, what level of change (or lack of) in the inflation rate can the Bank take credit for?

Although it has not been included with the 3 other dimensions, the measurability of an outcome should always be considered. It is hard to measure the performance of a set of actions if the change in the outcome itself is not measurable. An unmeasurable outcome will also lead to questions and debates about methods and approaches, and may lead to questioning of the value the organization brings to society.