Evaluation and Auditing

When I explain to people what a performance audit is, I usually refer to evaluation practice. Most people understand what an evaluation is while when they hear the word “audit” they immediately think about the IRS. I just finished reading the results of an evaluation of an area that I had wanted to audit. The evaluation was successful in determining the “condition” and the “effect.” In describing the “condition”, it also was able to allude to the “cause.” The recommendations are also on point; however, without addressing the “cause” head on, I wonder how easily they will be to implement.

Performance audits, to be successful, need to get to root causes. In this evaluation, the root cause is the organization. The evaluation looked at similar programs across departments and made recommendations to improve at the program level. However, without effective organization, clear lines of authority and responsibility, the road to improvement will be slow and possibly difficult. In this case, there should be a recommendation to consolidate leadership of programs.

I also was reminded of another difference between evaluation and auditing – the report. In this evaluation, the report language was very academic. Prior to getting to the findings, you had to wade through the methodology of the evaluation and the analysis. Usually the first draft of an audit will also contain writing about the methodology and the analysis, but by the time we get to a draft for our audiences (program, Council, public) that information is removed. We try to describe in simple and concise terms what we found and what it caused.

Believing in the tooth fairy

I just finished The Return of Depression Economics by Paul Krugman. The first edition of the book was written in the 1990's and published in 1999. It was updated in the midst of the current crisis. It is very readable and very instructive. Mr. Krugman reviews the previous crises back to the Depression in an attempt to understand and provide analysis and recommendations for the present. He makes the conclusion that this crisis is like everything we've seen before, all at once. Trading one bubble for another and ignoring the risk of acting outside of the regulated banking system, "the shadow banking system," only led to the failure that had been seen many times before in Asia, Japan, South America, and here.

Elements of an Audit

Several times I've written about how the work that auditors and journalists do is sometimes similar. Here is a news story by Henry Esteve that begins with a simple questions. Were the estimates of the cost of a tax credit accurate? Estimates are just that, estimates. Costs may turn out to be lower than higher than estimated. But it is good management to check and adjust a program depending upon what unfolds.

It's called closing the loop. Many audits address this management area. In my auditing experience, government service providers intensely pay attention to the design and organization of services. Just as much work and effort should also be placed in monitoring the service and evaluating whether it was successful.

This lengthy unfolding news story also illustrates a difference between journalism and auditing. News stories are short, not comprehensive, and accumulate over time. Turns out that state energy subsidies have been an ongoing story in 2009 in the Oregonian. The most recent story linked above that caught my eye was followed by at least 10 more stories and commentary including a story that the Governor would roll back subsidies.

Journalism is designed to foster accountability and transparency by shining a light on government practices. So is auditing, but audits are also intended to analyze the source of the problem and offer recommendations for change. Audits have a dual purpose - accountability and improvement of government services. The article cited a potential cause for the under estimated costs - that they had been low balled to gain legislative approval. If I had been conducting an audit I would have wanted to look at management practices that allowed estimates to be manipulated, if that is indeed what happened. What an auditor wants is to shine a light on the problem but also to have management rebuild a system that will protect the accuracy of data and prevent the problem in the future.

How do you turn things around?

I was at a conference last week that included auditors from federal, state, and local governments. As I listened to presentations and participated in discussions, it became clear to me that many times local government is far ahead of the federal level. I think that it is because government at the local level is so immediate and near. It doesn't take long to hear the echo from your constituency when they are upset.

Even comparing the federal and state audit organizations to local auditor shops, I could see the difference. There are very few levels, if any, of hierarchy at the local level. The audit director knows almost immediately what direction an audit is taking and has a close eye on audit quality and standards. At the state and local level there are levels of review and supervision before an audit is issued. This takes time and money.

So how do you turn things around in a large organization or how do you ensure that what you want the direction to be is what is occurring on the ground? It takes a concerted and dedicated effort. There are many analogies that I could think of. Imagine grabbing a 20 foot long rope and twirling around so that all of the rope was eventually off the ground. Just think how fast you would have to twirl before that final foot of rope was elevated. In large bureaucracies it may take years before the direction that you envisioned actually is reached.

Auditors can have a role in making the vision happen. Audits look at weaknesses in design and direction. As management twirls to lift that rope up, auditors can find places where weights drag the rope down.

Just a note. I am not saying that local efforts are more important. Many times those barriers to success at a local level occur because of state or federal action. So it takes us all to improve government.