Social Media Performance Measures

Here is an interesting story about the Washington Post trying to adjust to the digital age. "Employees are schooled in the lingo of web traffic."  (p.2)  Interesting things to note:

• 35 different daily reports track traffic to different parts of the web site
• Editors receive a performance alert each day telling them whether the site is on track to meet its traffic goals
• Even if traffic is low to some pages, if the online visitor is from a government suffix web site, who they consider a main audience, they will support those pages

Makes me wonder how government should be looking at traffic.

Love/hate relationship with numbers

Recently my daughter told me she hated numbers and math. We were talking about being a financial auditor. I had to agree with her. It feels like my head will explode when I sit for very long in front of a spreadsheet ensuring that all the numbers are accurate and add up.

But I love numbers when they can illustrate a story or a concept. And that's why I like performance auditing. In performance auditing, the goal isn't to just opine on the accuracy of numbers (although we do make sure of reliability before we conclude), the numbers tell you how important a finding is or communicate new perspectives.

Here's an example - Don't jump to conclusions about the census.

Not statistically significant

The Fair Housing Council of Oregon conducted audits from the citizens' point of view to determine if housing discrimination existed. In Ashland and Beaverton, when African American and white testers were sent to the same location, two-thirds of the black testers faced discrimination in Ashland and 78 percent in Beaverton. Management response in Beaverton was that the testing sample was too small to be statistically significant. But one wonders what the tolerance level is. Even though it was not a large enough sample to generalize to the full population, is it then okay to have some discrimination?

Back in the game

I have been gone for a while. Some of you know the reason. I had many other things on my mind unrelated to auditing. Although, now that I think about it, it was related to outliers. When you have cancer you can look at statistics and play the odds, the outliers. Auditing posts to follow.

Curiousity

I've started reading Freakonomics. I thought since I found the blog so interesting that I should read the book that started it all. The drive behind this economist is similar to what I find fascinating about auditing. Many times as you start an audit you hear from the program or service some "truth" that influences operations. You test that "truth" and find out how things really work. That is what Freakonomics is about - how things really work. Sometimes it is just more detail behind the scenes that, although the "truth" still stands, what makes it true is different then you thought. But sometimes the "truth" is really wrong. Things are not as they seem. Whatever the finding, the audit is unraveling the ball of string and then trying to craft recommendations to make the program or service better.

Finding data

Auditors look for data to explain audit findings and make them meaningful, e.g. illustrate effect. Sometimes you have to create your own data. Here is an interesting example of how to think creatively and find new data in the NY Times. They used Twitter chatter to capture the conversation during the Super Bowl. (H/T Community Indicators)

Interesting statistics web sites

Here is a web site that compiles information for comparison of state data. One statistic is called the "penny pincher" index or the ability to build and preserve savings. Oregon is 25th, right in the middle. Another online source is via Governing.com called the state and local sourcebook. Unfortunately the Portland region isn't reported separately from Oregon unlike many other metropolitan areas.

Storytelling

There's a blog post that I just have to tap into. It's from one of my favorite blogs, Community Indicators. So thanks for pulling together these thoughts. The post references other bloggers and writers who are discussing how statistics do not always evoke a response as strongly as emotions might and also that a good story is more memorable than data. When I followed the link and read about the finding that when statistics were part of an appeal for donations the amount given dropped, I wondered - is an audit report not as effective as it could be because we lay out the factual, statistical evidence? But the second reference to A Case for Web Storytelling reminded me of the work that we put into a report so that it has a cogent message. Yes, we rely on statistics and fact, but we also try to put it into a story that management and the public can relate to.

Interesting graphics

Audits can be very dense. It is difficult to simply describe complex ideas and findings. I am always on the look out for new ways to visually communicate ideas and reduce the reliance on writing. In a GAO audit that was issued recently I found some very interesting examples of visual communication of complex ideas. See the audit, pages 3, 11, 14, and 18. The graphics show the U.S. Department of Transportation's efforts over time to reduce traffic fatalities, illustrate the consequences of falling behind a damaging trend that with early action could have been reduced, and the evolution and spread of portable electronic devices in the United States to illustrate how this trend may need to be addressed for accident prevention. All very dense concepts but effectively communicated in visual diagrams.

More about Olympic Medals

I have blogged here and here about the quandary on how to measure success at the Olympic games. I have now found an interesting blog site and here is a map that shows Olympic success on a world scale at the 2004 games in Athens. It'll be interesting to compare to the updated 2008 games.

Why it's important to create good measures

Great guest post on NYTimes Feakonomics blog. Not only does it talk about performance measures that cause public policy to misdirect resources and efforts, but also touches on how systems can be just as important as building infrastructure.

The challenge

One of the reasons I enjoy auditing is that it allows you to challenge perceptions. Most organizations have a story that has become reality. Management often doesn't have time to step back and review landscape changes. What may have been true in the past may no longer be true. One prime example I remember from an audit long ago was library circulation. Library book circulation was reported to be steadily increasing and seemed to support management's request for increased funding. However, when auditors reviewed the statistics we found two things. Part of the increase was due to an increased ability to renew books. So the statistics didn't reflect just new books checked out but also books that were kept longer through renewal. Also, when we looked at circulation trends by branch library, we found significant differences. Both of these findings didn't necessarily suggest that the library shouldn't receive increased funding, but did suggest that the current level of funding could be managed better. It also didn't necessarily mean that management was deceptive or inept. Auditors have more time to dig into the data and find subtleties.

Just as an aside, I was reading another blog today and was directed to the Vote Studies Workbook that looks at voting history and how congressmen and women votes are aligned with their party. One of the things you can do is sort the data by various categories. I sorted by the "Member Since" column which shows the year they came to office. The oldest is from 1955, so 53 years in office. Can that be right? It would have been cool to examine the data to see how many had been in office since 2000 or to aggregate by 10 year periods. Makes you wonder how the voting history looks based upon time in office.

Data normalization

Now it's not just a discussion on how to count Olympic medals, either total of all types or total of gold only, to determine which country was the leader. On Freakanomics there is a discussion about making the data comparable or data normalization. If you divide the number of medals by million inhabitants of the country, then Jamaica is the winner with Bahrain second. The US becomes 33rd and China 47th. The discussion goes on. If sheep are added to the denominator then Australia zooms ahead. Scroll down to the comments and you'll find other suggestions. LOL.

What is the goal?

Many times in performance auditing you try to analyze whether a program has accomplished what it set out to do. Defining the goal that a program is trying to achieve is very important because the measure used to determine success depends on the goal. Here's an interesting story about how the number of Olympic medals that a country wins is reported differently. Is the goal to win more medals overall no matter their color (gold, silver, or bronze) or is the goal to be first?

Interesting blogs

I did some web searching this morning and found these interesting blogs.

• Community Indicators
• Reporting Statistics

A Pattern

On a personal note, it's been quite a 24 hours for me. When we were out to dinner last night with my son and his girlfriend, they announced they were engaged. Now that's a pattern as you watch your children go through the life stages that you have passed through. My daughter is off to college this fall and now the engagement. Then this morning I took my husband to the airport to head out to Beijing. That's an outlier.