Auditing by Processing

My thought process too. Like pulling a string or auditing.

Words

Words are very important in an audit. Auditors cannot afford to be sloppy. At the beginning of an audit when the scope is a concept and lacks specificity, words are important to help narrow the scope. Words used in the audit objectives help focus an auditor's work, keep her on track, and increase efficiency when followed. At the end of the analytical stage of an audit, words become very important in crafting the overarching message. What is it we are really trying to say? What is important and what is not?

The words we use effect the evidence that we need to support the words. If we do not have adequate evidence to support the words, for example "some" as opposed to "few," then we have to change the words. And, finally, when the auditee reads the draft, words are critical. Words that we have used to describe the condition or the effect may have a different meaning to the auditee. We have to be open to substituting words to reach agreement on the recommendations of the audit. As long as we do not change the facts, then changing the words to describe the facts, is acceptable. At the end of the day, these words express our findings to the public and the decision-makers. If we have not written clearly and exactly, our message may not be as effective.

Citizen Point of View "Audit"?

Here's a news story about a Housing Program in the Bay Area. The Program had people pose as persons seeking housing to gauge the response. I did a brief search of the web site to view the report but was unsuccessful. Probably not an audit according to standards but still an approach that could be worth using. Reminds me of the joint housing audit conducted by Multnomah County and the City of Portland where we had students posing as homeless call emergency housing services. The results were amazing and powerful.

Elements of an Audit

Several times I've written about how the work that auditors and journalists do is sometimes similar. Here is a news story by Henry Esteve that begins with a simple questions. Were the estimates of the cost of a tax credit accurate? Estimates are just that, estimates. Costs may turn out to be lower than higher than estimated. But it is good management to check and adjust a program depending upon what unfolds.

It's called closing the loop. Many audits address this management area. In my auditing experience, government service providers intensely pay attention to the design and organization of services. Just as much work and effort should also be placed in monitoring the service and evaluating whether it was successful.

This lengthy unfolding news story also illustrates a difference between journalism and auditing. News stories are short, not comprehensive, and accumulate over time. Turns out that state energy subsidies have been an ongoing story in 2009 in the Oregonian. The most recent story linked above that caught my eye was followed by at least 10 more stories and commentary including a story that the Governor would roll back subsidies.

Journalism is designed to foster accountability and transparency by shining a light on government practices. So is auditing, but audits are also intended to analyze the source of the problem and offer recommendations for change. Audits have a dual purpose - accountability and improvement of government services. The article cited a potential cause for the under estimated costs - that they had been low balled to gain legislative approval. If I had been conducting an audit I would have wanted to look at management practices that allowed estimates to be manipulated, if that is indeed what happened. What an auditor wants is to shine a light on the problem but also to have management rebuild a system that will protect the accuracy of data and prevent the problem in the future.

Curiousity

I've started reading Freakonomics. I thought since I found the blog so interesting that I should read the book that started it all. The drive behind this economist is similar to what I find fascinating about auditing. Many times as you start an audit you hear from the program or service some "truth" that influences operations. You test that "truth" and find out how things really work. That is what Freakonomics is about - how things really work. Sometimes it is just more detail behind the scenes that, although the "truth" still stands, what makes it true is different then you thought. But sometimes the "truth" is really wrong. Things are not as they seem. Whatever the finding, the audit is unraveling the ball of string and then trying to craft recommendations to make the program or service better.

Finding data

Auditors look for data to explain audit findings and make them meaningful, e.g. illustrate effect. Sometimes you have to create your own data. Here is an interesting example of how to think creatively and find new data in the NY Times. They used Twitter chatter to capture the conversation during the Super Bowl. (H/T Community Indicators)

Auditing process and ignoring science

I was led to this report after reading a post by Kari Chisholm on the BlueOregon blog. The post describes the political battle that occurred over a nomination to the Department of the Interior and a hold put on that nomination by Senator Ron Wyden who asked for a report by the Office of the Inspector General (OIG). A year later in mid-December 2008 the report was issued. While the politics were interesting; from an auditor's perspective I found the methodology used to complete the investigation just as interesting.

Many times in audits, we struggle with how to audit something that is less than concrete. This audit reminds me how that is done-through the elements of a finding. The objective of the investigation was to determine if endangered species decisions had been improperly influenced by the Deputy Assistant Secretary for Fish, Wildlife, and Parks.

Criteria: The methodology was to see if policies and procedures were followed. The OIG reviewed 20 Endangered Species Act decisions. The OIG conducted 89 interviews with employees and reviewed over 20,000 emails and other documents and sent out a standardized questionnaire to all of the US Fish and Wildlife Services regions.

Condition: The investigation found a policy void and lack of guidance. One interviewee stated that there was way too much "informal policy" and that he would wake up in the morning and think, "Okay, what's the agency doing today?" The investigation also noted that this intentional failure to clarify policy had been ongoing through several administrations in order to allow the policy du jour.

Effect: As a result, the Deputy Assistant Secretary in this current administration was able to influence decisions and steer the best available science to pre-ordained ends. She caused significant harm to the integrity of the decision-making process and and its reputation with state and local agencies. The report also stated that there will be considerable cost in undoing some of the decisions and that the actions also caused considerable expenditure in taxpayer dollars for litigation.

The emotions behind words

There is an interesting post on Freakonomics with many comments. It reminds me how words can evoke emotions. Justin Wolfers is commenting on an Op-Ed piece that concludes that despite the economic crisis. people are still happy. Using daily survey data on life satisfaction based on the Gallup-Healthways Well-Being Index, Mr. Wolfers attempts to show factually that people are not happy. He bases his argument on the results of questions about thriving, struggling, and suffering. The comments are a very interesting read. At one point it devolves into a discussion about Communism.

Several years ago in my career I was part of a team that audited Multnomah County's adult foster care regulatory program. We designed a checklist and performed surprise visits on the foster care homes to determine the quality of the care. One of the criteria we measured was the "home-like" environment. One of the strongest responses I ever got to an audit was because of our finding that some of the homes were not "home-like." It was a very emotional discussion. The manager who objected to us trying to measure that quality felt that we were applying middle-class values and could not adequately understand or measure what home might be like for other cultures.

What is the goal?

Many times in performance auditing you try to analyze whether a program has accomplished what it set out to do. Defining the goal that a program is trying to achieve is very important because the measure used to determine success depends on the goal. Here's an interesting story about how the number of Olympic medals that a country wins is reported differently. Is the goal to win more medals overall no matter their color (gold, silver, or bronze) or is the goal to be first?

Good advice for auditors

...there is always more than meets the camera's eye. Subject everything to a reality check. (From Bill Moyers Journal, hat tip to kidfurbo)