Guide to the Approval of Strategic Outcomes and Program Activity Architectures

I didn't think this document was available to the general public, but it is, and here is the link:

http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/mrrsp-psgrr/guide/guide01-eng.asp

This is the guide on the approval process for Strategic Outcomes (SOs) and Program Activity Architecture (PAA). At least, this is the guide for the process for the 2010-2011 PAA.

For those who don't know, the SOs and PAA are the skeleton of the framework used by departments of the Government of Canada for public reporting.

Measuring Efficiency

As discussed in an earlier post, efficiency can generally be conceived of as the ratio of the output to the input of a system. So how do you measure that, and why would you want to do so?

Basically, the way to measure efficiency is to link resources to results, whether that result is an output or an outcome. Obviously, if you start linking final outcomes directly to resources, your model is going to be somewhat questionable, but what you may want to try to do is measure how much resources it takes to maintain a certain level of achievement of a lower level outcome. What am I talking about?

Take for example the outcome of client satisfaction. A lower-level outcome could be customer satisfaction with telephone support. An indicator of that could be call wait time. Ok, now, say you set a target of 2 minutes (this could be a service standard), how many employees do you need on the lines to keep call wait times within that target? So in the end, to link the result to the resources in this case, you’ll need to measure at least 3 things in order for it to be meaningful: the call wait time, the number of employees answering calls, and the number of calls. At some point, by looking at historical data, finding trends and building forecasts, you should be able to get a pretty good idea of how many employees you need to answer calls at different times of the year, to stay within your target call wait time. Ok, that was a pretty complicated example, and the analysis should go further because there are costs involved in adding and removing staff.

But here are more simple examples of efficiency indicators:

• Average number of hours per file
• Average number of days to staff a position
• Average cost of a staffing process
• Average cost per unit (i.e., production lines)
• Total value of sales per month per salesperson
• Server up time
• Number of units produced by machine 4 per week

As you can see, you can link different types of outputs or outcomes to different types of resources (time, employees, funds, etc.). However, one of the main weaknesses of efficiency measures is that they do generally assess quality. For example, if it takes 5 hours on average to review a file, but a lot of mistakes are made, or steps are skipped to reduce the time required, then you just created a perverse indicator, or perverse incentive. The lesson? Balance efficiency measures with measures of quality.

Why would you want to measure efficiency? To optimize the use of limited resources.

Service Standards and Year End Reporting

Let’s assume we are in April 2009 and take the following example of some applications:

Receive Date	Processing Finish Date	Number of Days to Process
February 1, 2009	February 15, 2009	14
February 15, 2009	February 28, 2009	13
February 15, 2009	March 1, 2009	14
February 15, 2009	March 15, 2009	28
March 1, 2009	March 15, 2009	14
March 15, 2009	March 31, 2009	16
March 31, 2009	TBD

Now, let’s assume that no applications were received in January and that the service standard for this type of application is 15 days (applications are processed in 15 days).

There are two ways of looking at this. The typical view is to measure from the receive date. That would measure the number of days it took to process an application from the day on which it is received. The other possibility is to measure based on the date on which the application’s processing was finished. This method is less common.

Here is a tricky question: “What was the average processing time for applications in February?” The question is tricky because it doesn’t give you a reference for which date to use as a base, are we talking about the applications received in February, or the applications processed (finished) in February? Here are the options:

Average Processing Time of Applications by Received Date
February	17.25
March	TBD

Average Processing Time of Applications by Processing Finish Date
February	13.5
March	18

According to the received date base, 75% of applications received in February were processed within the service standard (15 days). But of the applications processed in February, 100% were processed within the service standard (if we consider that no applications were received in January, which we do in this example).

Normally, I expect most organizations to use the first method, based on the date on which the application is received.

So the fiscal year is over (ends March 31), and it’s now April 7, 2009. Can you produce accurate statements on your performance against the service standard for applications received in March, using the first method (based on the date the application is received)? The answer is no, because your service standard is 15 days, and only 7 days have passed since the last day of March. April 15 is the last day for which the applications received March 31 will still be processed within the service standard, so you would only be able to know how many of the applications received in March were processed within the service standard at the end of the business day on April 15. That’s assuming you have instant access to up-to-date information, which is not always the case. If there is a delay between the time an application is processed (processing finish) and when you know about it, then you also need to take that into account. This is often the case for electronic systems, there is often a delay between the data entry into the application, and the availability of the data in data marts or cubes used for reporting.

In conclusion, be aware of your service standards, of how your performance is calculated, and of the delays in the availability of data when you are doing your year end reporting, or you could end up with inaccurate performance statements.

Efficiency and Effectiveness

Efficiency and effectiveness are two closely linked concepts that are, unfortunately, often misunderstood. So what are efficiency and effectiveness?

Efficiency

Efficiency relates to the amount of resources (input) used to achieve a goal (output). Efficiency can generally be conceived of as the ratio of the output to the input of any system. An efficient system would have a high output to input ratio, that is, it would produce a lot of the output for little of the input. There are different situations that can describe a gain in efficiency:

1. producing more output with a given amount of input
2. producing a given amount of output with a reduced amount of input

The other 2 situations representing possible efficiency gains,

3. producing more output with more input
4. producing less output with less input

depend on the measure of the ratio of output to input. In the first case, an additional amount of input must lead to the creation of more additional units of output than the current value of the ratio for the situation to represent a gain in efficiency. In the second, a reduction of one unit of input must be accompanied by a reduction of less units of output than the current value of the ratio. In other words, the output to input ratio must increase.

Effectiveness

Effectiveness relates to whether the means used lead to the end. In other words, whether the action has the intended result. In the business context, it most often refers to the extent to which a program or service is meeting its stated goals and objectives (or outcomes). Improving the effectiveness usually means changing something (normally the action) that will increase the extent to which the goal is met. For example, improving the effectiveness of an anti-smoking program would mean changing something that would increase the percentage of people who smoke (if that’s the indicator you decide to use to measure the achievement of the goal).

It should be noted that a program’s effectiveness can be increased by changes outside the scope of influence of the program. Changes external to a program can impact the effectiveness of a program, both positively and negatively. That is part of the reason of environmental scanning. Effectiveness is one of those concepts where it is important to understand the difference between correlation and causality.

Tabling of the 2007-2008 Departmental Performance Reports and Canada's Performance Report

The 2007-2008 Departmental Performance Reports (DPR)and the 2007-08 Canada's Performance Report were tabled on February 5, 2009.

Departmental Performance Reports are reports written by departments and agencies at the end of the fiscal year. They describe what the organization has achieved (it's performance) and how it performed compared to its plans and goals.

Canada's Performance Report is basically a chapeau piece to the Departmental Performance Reports, and tries to combine the performance of all the departments and agencies to create a performance report for the government as a whole. Where DPRs are historically based mostly around what the department achieved, Canada's Performance Report gives a much more social perspective to the results of government spending.

2007-2008 Departmental Performance Reports: http://www.tbs-sct.gc.ca/dpr-rmr/2007-2008/index-eng.asp

Canada's Performance Report 2007-08: http://www.tbs-sct.gc.ca/reports-rapports/cp-rc/2007-2008/cp-rctb-eng.asp

Updated MRRS Policy

There has been an update to the Policy on Management, Resources and Results Structures (MRRS). The updated policy took effect on December 20, 2008.

The updated policy is available here:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=14252§ion=text

The old policy is available here:
http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12412§ion=text

Overall, I would say the new update improves the policy. What I consider the core of the policy, the core requirements which used to be under 7.1, and are now under 6.1.1 haven't changed in the essence of their meaning.

The update adds more responsibilities for Deputy Heads related to the implementation of the policy, keeping the MRRS up-to-date, following proper procedures for updates, etc.

The updated policy also has a new section, 7. Consequences, describing consequences for untimely or unsatisfactory implementation of the policy.

Aggregating Indicator Scores

To measure the performance of your organization in a certain area, you will typically use a set of indicators. These indicators may or may not cover the entire area you are trying to measure, may contain indicators of short-term or long-term progress, etc. Anyhow, you'll have a set of indicators that you have chosen to represent an area of management, a process, an activity, etc.

So indicators are a set of metrics. You may have something like this to measure client service:

Indicator	Actual Value	Target
Percentage of pizzas delivered within 30 minutes	90%	100%
Percentage of calls answered within 2 minutes of entering the queue	80%	100%

Now, to get an aggregate score for client service, you could just take the average of the 2 indicators, that would give you (90+80)/2=85. However, you may decide that the indicators don't all have the same importance, the so they shouldn't all have the same weight. Let's say people hate waiting in a telephone queue, but won't notice if there pizza is 2 minutes late. In that case, the indicator for call wait time is more important, so we'll give it a weight of 70%, and we'll give a weight of 30% to the pizza delivery time. That would give us a score of (90*0.3)+(80*0.7)=27+56=83.

A few notes on this:

be careful of the units you use, in the example, we used 2 percentages with the same target, so we know they'll be fairly close and that they are fairly comparable. But if you were measuring something like the number of units sold and average call wait time in minutes, your units would be too different to be compared directly. What can you do? Use the target, and compare the result to the target. That will give you 2 results in "percentage of target achieved", which can than be directly compared to one another. If you use that method, setting meaningful targets becomes essential if you want your aggregate indicator score to be meaningful and useful.

In the example, the weights used add up to 1. It doesn't necessarily have to. But having a score that has an understandable maximum (100 in this case) makes it more understandable and intuitive. The resulting aggregate indicator score in the example, is not in a particular unit: all we know is that it's maximum is 100. There are times when, because of either your indicator or target your result may exceed 100. There is nothing wrong with that, but it highlights the importance of explaining how you go about measuring your performance, and how your data should be interpreted.

Finally, defining weights is a tricky exercise, and some managers may abuse this system by assigning low weights to indicators on which they know they will perform poorly. Another aspect to consider is that you may want to assign low weights to indicators for which the results are not very reliable.

Speech from the Throne

The Speech from the Throne and the Prime Minister's speech provide the foundation for the government's legislative priorities and agenda for the current session of Parliament. These priorities should be reflected, where applicable, in a department's Report on Plans and Priorities (RPP).

Speech from the Throne: http://www.sft.gc.ca/eng/media.asp?id=1383

Address by the Prime Minister in Reply to the Speech from the Throne: http://www.pm.gc.ca/eng/media.asp?id=2318

Correlation and Causality

Correlation is not causality, they are two different concepts.

Correlation

Correlation is a relationship between variables. When the value of X goes up (or down), the value of Y goes up (or down) in a predictable way. The height and weight of a person are correlated. Their eye color and their weight is not.

Causality

Causality is a cause-effect relationship between variables. A change in the value of X is the cause of a change in the value of Y. For example, viruses make you sick. Be careful not to confuse the cause and the effect: you sneeze because you have a cold, but you don’t have a cold because you sneeze.

Proving a cause and effect relationship is difficult, as all other variables must be controlled. It is also possible for an observation to have more than a single cause; the change of the price of a stock is an example. Normally, the change in the variable causing a change in the other is observed before the change of the value of the dependent variable.

The Management Accountability Framework (MAF)

The Management Accountability Framework (MAF) is a framework used by the Treasury board Portfolio to assess the quality of management in departments. It is structured around 10 elements: Public Service Values, Governance and Strategic Directions, Policy and Programs, Results and Performance, Learning, Innovation and Change Management, Risk Management, People, Stewardship, Citizen-focused Service, and Accountability. Indicators are defined for each element and are used to measure performance in each area.

More information on the MAF is available on the Treasury Board of Canada Secretariat's website: http://www.tbs-sct.gc.ca/maf-crg/index-eng.asp

Dimensions of an Outcome

Outcomes are normally part of a performance measurement framework of one type or another. Most often, they will be used in the public sector or the non-profit sector to explain how their activities’ contribution to society. The might also be used at a lower level to measure the implementation of a strategy in the private sector. In that particular case an organization would be measuring the outcome of a strategy, although in the terminology generally used in public sector performance measurement; this would be closer to an expected result.

Regardless, the purpose of this post is to clarify a perceived ambiguity surrounding outcome levels. In performance measurement literature, different levels of outcomes are often mentioned, such as immediate, intermediate, long-term and final outcomes. The descriptions given usually revolve around time and impact on society.

However, to clearly define outcomes, they need to be perceived through at least 3 dimensions:

1. reach or societal impact,
2. time (frame, lag, or delay) and
3. attributability or responsibility

The reach or societal impact can be generally conceived of as the “societal importance or value” of the outcome. For example, “reducing the number of sick Canadians” may be an outcome, but “healthy Canadians” is a broader, and further reaching one.

The time dimension is a little more complex, because more things can be measured here. For example, an outcome could be defined as a desired end-state. In that context the time dimension would refer to the time required to bridge the gap between the current state and the desired end-state. The time dimension can also be important in a context where an organization’s action will only have an impact on the outcome after a period of time.

The attributability of an outcome for the organization or the responsibility or the organization for the outcome are also to be considered. Attributability can be defined as the amount of “credit” an organization can take for the change in the outcome. Most often, not all change in an outcome can be attributed to the actions of an organization. The concept of attributability is closely linked to the concept of causality. A change in the outcome is attributable to the organization if the organization’s actions are the cause of the change. Responsibility, however, is a different concept. Where attribution is when an organization appropriates changes in an outcome, responsibility is when an organization is made responsible for an outcome, or if you prefer, is mandated to have an impact on the outcome. However, attributability of the change in the outcome still remains to be proven of organizations with clear responsibilities. For example, the Bank of Canada has an agreement with the Government of Canada regarding target inflation rates. To a certain extent, it is responsible for the rate of inflation. The question in that case is, what level of change (or lack of) in the inflation rate can the Bank take credit for?

Although it has not been included with the 3 other dimensions, the measurability of an outcome should always be considered. It is hard to measure the performance of a set of actions if the change in the outcome itself is not measurable. An unmeasurable outcome will also lead to questions and debates about methods and approaches, and may lead to questioning of the value the organization brings to society.

Cognos Help Resources

For those of you interested in Business Intelligence (BI) software, here are a few links relevant for IBM Cognos 8 BI:


Cognos

Most of the support or help information on the Cognos site requires a login and password.

Supportlink is published frequently and includes some interesting tips and techniques
http://support.cognos.com/supportlink/

The main Cognos support site, the Knowledge Base is a useful tool
http://support.cognos.com/en/support/index.html

Customer Resource Center - Report Author Section contains more detailed documents on different subjects and techniques
http://support.cognos.com/en/resources/roles/gcs_3.html


COGNOISe

Cognos Centered community, here's the link to the forums:

http://www.cognoise.com/community/


ITtoolbox

Some Cognos related forums, take a look at the forum list for other products

http://businessintelligence.ittoolbox.com/groups/technical-functional/cognos8-l
http://businessintelligence.ittoolbox.com/groups/technical-functional/cognos-l


Tek-Tips

Another Cognos related forum

http://www.tek-tips.com/threadminder.cfm?pid=401

DPR Requirements Relating to Government of Canada Outcome Areas

When writing the Departmental Performance Report, a link between the department’s strategic outcomes and the Whole-of-Government Framework must be established. An explanation of how the department’s strategic outcomes are aligned with Government of Canada outcome areas. This should be done in Section 1: Overview under the Summary Information heading.

From the Template Instructions for Departmental Performance Reports (http://www.tbs-sct.gc.ca/rma/dpr3/06-07/instructions/instructions_e.asp):

“a summary status on the department’s performance in achieving their strategic outcome(s) and program activity expected results. The Summary Information table is mandatory and must be followed by a narrative section. The narrative section is to provide an overall description of the department’s performance for 2007–08. All key elements provided in the summary table must be explained. This section should provide the department’s overall performance in relation to the previously set priorities; indicate the progress made towards departmental strategic outcomes and how it is supported by the program activities; and outline how the departmental strategic outcomes contribute to broader government-wide objectives.”

“the description of the departmental context must also include a discussion of how departmental strategic outcomes are aligned with Government of Canada outcome areas. For more information on current outcome areas or Canada’s Performance and the RPP Overview for Parliamentarians website, departments can consult the “Whole of Government Framework” instructions online at http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/mrrsp-psgrr/siglist_e.asp (see the contact list at the end of the Guide to the Preparation of Part III of the 2007–08 Estimates).”

Here is a link to the Template Instructions for Departmental Performance Reports (PDF and RTF version available by clicking on the links at the bottom of the menu on the left side): http://www.tbs-sct.gc.ca/dpr-rmr/2007-2008/instructions/instructions00-eng.asp

Whole-of-Government Framework

The Whole-of-Government Framework can be found here:

http://www.tbs-sct.gc.ca/ppg-cpr/framework-cadre-eng.aspx?Rt=1037

It looks something like this:




I'm no expert, but it doesn't look like it's accessible for the visually impaired.

The following page explains in more detail how it works/how it is used:

http://www.tbs-sct.gc.ca/reports-rapports/cp-rc/2006-2007/cp-rc02-eng.asp#Introduction

The Government’s Priorities

The Government’s 5 priorities are:

A Proud and Sovereign Canada

There is nothing more fundamental than the protection of our nation’s sovereignty and security. The Government will rigorously defend Canada’s place in the world including through the realization of our strong Arctic vision and a responsible, effective path forward in Afghanistan.

A Strong Federation

Canada is more united today than it has been in 40 years. The Government will continue to strengthen the federation – and modernize its democratic institutions – through measures including formal limits on federal spending power and long-overdue reform of the Senate.

A Prosperous Future

Canada cannot be complacent about the continued growth of its economy. The Government will provide effective economic leadership and a prosperous future by aggressively moving forward with broad tax relief that includes a further promised reduction in the GST.

A Safe and Secure Canada

Canadians want their safe streets and communities back. The Government will continue to tackle crime and strengthen the security of Canadians by reintroducing important crime legislation with the new a Tackling Violent Crime Bill, and by putting a strong focus on safe communities and youth and property crime.

A Healthy Environment for Canadians

Canada’s environmental and health standards are already among the highest on Earth. The Government will continue to improve the environment and health of Canadians by delivering realistic and achievable results in areas such as environmental enforcement and product and food safety.

Source: http://www.pm.gc.ca/eng/feature.asp?featureId=5

Instructions (Guide) for Developing a Management, Resources, and Result Structure

This document is a set of instructions, although I see it more as a guide, to develop (or if you prefer, implement) Program Activity Architectures (PAA), Performance Measurement Frameworks (PMF) and Management, Resources, and Results Structure (MRRS).

I found it useful. Constructive criticism: the pages showing the PMF tables didn't print out well on letter size paper in portrait layout. If I remember correctly, those pages are towards the end and print out well on landscape legal. I wish they would have made a pdf file, probably would have avoided this type of problem, and it would have made the document more portable and sharable.

Link: http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/mrrsp-psgrr/id-cm/id-cm_e.asp

RCMP Environmental Scan

A very nice document prepared by the RCMP, and publicly available.

It covers demographics, society, economy, politics & government, science & technology, environment and public safety & security at both the global and Canadian levels.

Link: http://www.rcmp.gc.ca/enviro/2007/index_e.htm

Kurtosis

Kurtosis is the degree to which the frequency distribution is concentrated around a peak, that is, it describes the sharpness of the central peak of the curve, usually as compared with the normal distribution.

Higher kurtosis means more of the variance is due to infrequent extreme deviations (more variance, less concentrated around the mean), as opposed to frequent modestly-sized deviations (less variance, more concentration around the mean)

The normal distribution is mesokurtic; the curve with a higher degree of kurtosis (peakedness) is leptokurtic; and the curve with the flat top (compared to the normal curve) is platykurtic.

Links:

http://www.riskglossary.com/articles/kurtosis.htm

http://mvpprograms.com/help/mvpstats/distributions/SkewnessKurtosis

http://www.statistics4u.info/fundstat_eng/cc_kurtosis.html

http://www.almprofessional.com/Articles/skew.pdf

http://www.ats.ucla.edu/stat/spss/faq/kurtosis.htm

Mean Absolute Deviation

The mean deviation is the mean of the absolute deviations of a set of data about the data's mean.

Mean deviation is an important descriptive statistic that is not frequently encountered in mathematical statistics. The mean deviation has a natural intuitive definition as the "mean deviation from the mean".

The average absolute deviation from the mean is less than or equal to the standard deviation.

When applied to time series, the mean absolute deviation becomes a measure of volatility.

Standard Deviation

Standard deviation is a measure of dispersion. It is defined as the square root of the variance. It measures how widely spread the values in a data set are. If many data points are close to the mean, then the standard deviation is small; if many data points are far from the mean, then the standard deviation is large. In a loose sense, the standard deviation tells us how far from the mean the data points tend to be. The standard deviation has the same units as the data points themselves.

When applied to time series, standard deviation becomes a measure of volatility.

See also: http://www.childrensmercy.org/stats/definitions/stdev.htm
http://www.quickmba.com/stats/standard-deviation/