In 2015, exchange traded funds (ETFs) celebrated a 25-year anniversary by commemorating the 1990 launch of the Toronto 35 Index Participation Units. 

Thousands of ETFs have since been introduced—and many subsequently retired. Lost among the multitude of index trackers, however, are a handful of late-entry actively managed products. 

It’s fair to ask how good these active ETFs are as alpha producers, but finding the right answer isn’t easy. First, there’s the novelty of the product line. Few active ETFs have the three-year track record required by outfits like Morningstar for detailed analysis. And then there’s the problem with the analysis itself.

Morningstar assigns a so-called “standard index,” such as the S&P 500 Index or the Barclays Capital U.S. Aggregate Bond Index, as a benchmark for each ETF. This index, in Morningstar’s own words “has the highest correlation to the funds in the asset class based on at least three years’ worth of return history.” 

That’s rather hard to believe if you look at the numbers. 

Take the case of the PowerShares Active U.S. Real Estate ETF (NYSE Arca: PSR) as an example. Why would Morningstar think the MSCI All Country World Index is a suitable benchmark for PSR when the fund’s r-squared coefficient is only 15.01 against the global equity index? After all, the fund’s “best-fit” index, the S&P U.S. REIT Index, explains nearly 99 percent of PSR’s movements.

Here’s the rub: If you go into a search for ETFs with the highest alpha, the output is most likely to be based on Morningstar’s standard index, not the best-fit one. That could lead you to attribute unwarranted skill (or ineptitude) to a fund manager. Case in point: PSR sits at the top of the alpha search list with a 6.49 coefficient against the MSCI ACWI benchmark. The fund, however, earns a -0.10 alpha versus the S&P U.S. REIT Index. Yes, PSR indeed earns alpha. The problem is, against a real benchmark for its asset class, it’s negative alpha.

Caveat emptor.

Seeing this, we decided to test the top 10 alpha crankers in the Morningstar active ETF universe. To better simulate real-world conditions, we relied on daily, not monthly, returns for the past three years. This gives us hundreds of data points to study rather than the dozens used by Morningstar. Daily data yields more granularity, so the resultant coefficients can differ widely from those derived monthly. We think the larger data field produces metrics that are more predictive of day-to-day performance.