By Charles Thibault on December 1, 2009

Forbes Magazine released an article of "America's Fastest-Recovering Cities" on November 19th, also producing a ranked list of America's 100 fastest recovering cities.

Unfortunately for Forbes readers, the list contains methodological issues which limit its usefulness in determining the "fastest recovering cities".

The most outstanding issue is the use of "levels" of economic indicators to measure recovery, as opposed to their changes. For example, the author uses the unemployment rate in the ranking system. The current unemployment rate has nothing to do with how quickly the labor market situation has improved, which is more accurately reflected by changes in the unemployment rate. Low unemployment rates might be a good measure of the "most stable cities", but not the "fastest recovering".

Let's compare two cities to see how using "levels" of economic variables is not that informative. Omaha's "Unemployment Rank" was No.1 in Forbes' list, with a 5.3% unemployment rate, while Wichita's "Unemployment Rank" was No.42, with an 8.6% unemployment rate. This indicates that Omaha's economy was not wrecked by the recession, which is great for Omaha. But that has nothing to do with how quickly the economy has improved, as suggested by article's "Fastest-Recovering" title.

You can see from the table below that Wichita, KS had higher unemployment than Omaha, NE before the recession even started, but its labor market situation has improved more dramatically since – in both absolute terms and relative terms. Wichita's unemployment rate is down 1.4 percentage points since July, while Omaha's unemployment rate is only down 0.5 percentage points. Relative to their respective July unemployment rates, Wichita's drop represents a 14% improvement, but Omaha improved only 9.4%. So, which city improved the fastest?

What matters is: what's the unemployment rate in Omaha now, compared to before the recession; what's the unemployment rate in Wichita now, compared to before the recession; and how do those two differences compare to each other? In statistics, this method is called the "difference-in-difference estimator", which factors out local market particularities, for example the fact that Wichita had systematically higher unemployment than Omaha before the recession even started.

We use our own proprietary Hiring Demand Indicators, which is the count of new online job ads posted during a month, to rank the US cities based on their "actual recovery rates". The "recovery rate" is the average monthly growth in the number of online job ads, over the past 9 months. WANTED Technologies covers more than 1,000 employment-specific job boards.

Our rankings are very different than Forbes'. Forbes' No.1 city, Omaha, falls to the 71^st position on our list. San Antonio, Forbes' No.2 city is ranked 39^th on our list, and Austin, TX, which was ranked No.3 on the Forbes List, is ranked 75^th on our list, with a Hiring Demand growth of 0.81%. New York City, ranked 64^th by Forbes, is ranked 7^th here. Other Florida cities, which were hit hard by the real estate crisis, climb in our rankings with solid growth in labor demand.

There are other methodological issues that we can highlight in the Forbes Ranking:

• The author blends two different types of economic measurements – levels and changes – in the construction of the index. The composite ranking is an average of other economic indicator rankings. Three measurements are "current levels" and two are "changes":

• • Unemployment Rate: current level
  • Foreclosures: current level
  • Property time-on-market: current level
  • Gross Metropolitan  Product (similar to Gross Domestic Product): year-over-year change (Q2)
  • Home Prices: year-over-year change (Q2)

It's not appropriate to mix rankings of "levels" and rankings of "changes" together in such a way.

• Rankings are not as useful as "indexes" or actual growth rates. For example, how much better off is No.11 compared to No.12, and is that relative difference the same between No.64 and No.65? No.11 is probably much better off compared to No.12, than No.64 is compared to No.65. Rankings are useful for lists, but they do not convey any information on "how much better" one city is relative to another like an index can.

• We like that the index combines real estate data, labor market data, and a measure of economic output (Gross Metro Product). However, the national rankings are heavily weighted towards real estate data (3 indicators for that general economic segment, versus only 1 for labor dynamics and 1 for general economic health).

In fact, running a linear regression of the composite rank on the 5 economic variables allows us to see how Forbes weighted each one of those individual pieces. The following table below shows these weightings. (The modeled weights total 104). You can see that real estate metrics have a combined weight of 61%. The one variable that we like the best, changes in Gross Metropolitan Product, carries the lowest weight (6%).

What's more, the correlation between the Foreclosing Rank and the Home Price Rank is 0.65, which is fairly tight (correlations can be between -1 and 1, with values closer to -1 or +1 indicating a tighter relationship). In other words, the more expensive houses are, the more likely homeowners are to foreclose. Either one of these variables would have sufficed for the index, not both, since they are highly related to begin with.

• Inter-temporal analysis should compare the current quarter relative to Q1 2009. Comparing Q2 2009 to Q2 2008 doesn't capture the most recent "recovery trends". The trough (bottom) of the recession was in Q1 2009, and the recovery started in March. It would be better to capture monthly growth rates starting in March, or simply compare the progress of Q2, Q3 and Q4 2009 data compared to the "end" of the recession which was Q1 2009. In other words, the recession hadn't fully hit the economy in Q2 2008, and doing year-over-year comparisons to pre-recession levels doesn't quite capture "the recovery", which technically starts at the bottom of the recession.

Comments (1)

• Rich Maloy says:

  December 2, 2009 at 9:06 am

  What a great article. I religiously follow your blog and I'd have to say this is by far the best example of why I read Hiring Demand: well articulated critical thinking and statistical analysis. One has to wonder who comes up with these major publication "Top/Best" lists and why they choose their methodology. I learned quite a bit from this article, not only about the analysis of the article and the recovery, but also about how to think critically about other people’s suppositions. Thanks for the great post!

Links to this article (4)

• December 3, 2009 - www.johnsumser.com
  091203 In The Know | johnsumser.com
  
• December 7, 2009 - idaho-homes.net
  Is Forbes’ List of Fast-Recovering Cities Incorrect? « Idaho-Homes.Net
  
• December 21, 2009 - www.hrexaminer.com
  091203 In The Know v1.04 | HR Examiner with John Sumser
  
• January 14, 2010 - jefferson-county.nycountyrealestate.com
  Forbes List of “America's Fastest-Recovering Cities” – Not a Good … | Jefferson County NY Real Estate
  

Comments are closed.