Business is Personal (Finance)

Patriot Pyro 120G SSD RAID

2011-10-06T01:11:00Z

While I periodically write about technology, I try do so in a business context. I will make an exception for this entry though as I want to document the gotchas we've ran into for others to benefit from our experiences. I myself Google'd and Google'd to no avail trying to find answers to these questions while researching what to buy and analyzing rest results.

SSD

Let's first start with this premise. If you want fast storage, you want SSDs. They will wear out fast from over-use but to be honest, that's not much different than hard drives dying from mechanical failure from over-use. And after burning out our previous SSDs, we picked 5 of the following drive as replacements:

Patriot Pyro 120G SSD
SandForce SF2281 controller
SATA 3.0 6Gbps
Sequential Read Speed: 550MB/s
Sequential Write Speed: 515MB/s
Random Write IOPS: 85,000
Read Latency: 0.05ms
Write Latency: 0.25ms

By comparison, the latest spinning magnetic hard can transfer data at 150MB/s when on the outer edge (new drive), 100MB/s on the inner edge (filled up hard drive) with a drive latency of 5ms (15K RPM) to 12ms (7200RPM). In short, SSDs of this class are mind-bogglingly faster. But if 500MB/s is not fast enough for you, you can put 2 drives together in a RAID0 array for 1000MB/s transfer rates.

And since we process a lot of data every week, we aimed high. The data processing server motherboard has 6 x SATA3 (6Gbps) connectors -- allocate the 1st for the boot OS leaving 5 for data drives. 5 500MB/s SSDs striped together? 2500MB/s! Woohoo!

CPU & Motherboard

After all the hardware arrived, we built our replacement data processing server on the following platform:

AMD Phenom II X6 1100T
3.2GHz / 3.7GHz turbo
6-core
ASUS M4A87TD
AMD 870 northbridge
AMD SB850 southbridge

Simple SSD Bandwidth Tests

While there are far more comprehensive disk benchmarking tools, I started with HDPARM for simplicity purposes to test the upper bounds. The command used was "hdparm -t /dev/sdX" producing the following results:

1 x Patriot Pyro: 520 MB/s
2 x Patriot Pyro RAID0: 1040 MB/s
3 x Patriot Pyro RAID0: 1120 MB/s
4 x Patriot Pyro RAID0: 1120 MB/s
5 x Patriot Pyro RAID0: 1120 MB/s

It doesn't take much analysis to see what we ran into. My initial guess was you simply cannot shovel more than 1100 MB/s from either the SATA subsystem or the overall southbridge. Now to confirm my guess, I read through a ton of reviews for hardware using the SB850 chipset but could not find anybody who was mad enough to stack 2.5GB/s worth of SSDs into a system. I did run into the following image though as a common graphics filler material:

The key part of the above diagram is the connection between the 890GX and SB850 chips where it says Alink Express III. So there we have the limitation -- 2 GB/s. And because the arrow is a 2-headed arrow, that number says you can push 1 GB/s one way and 1 GB/s the other way at the same time for an aggregate total of 2 GB/s. 1 GB/s is 1024 MB/s which is close enough for government work to our 1120MB/s number if the latest SB850 revision is now 10% faster that originally depicted in this diagram.

Highpoint Rocket 620

As it seems like a waste to run our SSDs at less than half speed, we want to squeeze more speed out. And from looking at that diagram, I noticed the PCIe connections to the northbridge have 16 GB/s bandwidth. What if we put SATA3 6Gbps adaptors there and moved 3 of the SSDs over? This way the SB850 could handle 2xSSDs for 1000 MB/s with the 1500 MB/s of SSDs connected to the northbridge. We pulled a Highpoint Rocket 620 card from another server and hooked up 2 SSDs it. The results:

Highpoint R620, 1 x Patriot Pyro: 200 MB/s
Highpoint R620, 2 x Patriot Pyro RAID0: 400 MB/s

Rather underwhelming. Even though the R620 is a SATA3 6Gbps card, the R620's Marvel chipset is just not fast enough to drive latest, greatest SSDs. I'm sure this card works perfectly fine for SATA3 hard drives though.

PostgreSQL SSD Bandwidth Tests

But since our old SSDs were now unsable, we had to go live with the new hardware regardless of sub-optimal performance. After copying over the databases, I started with real world testing running PostgreSQL queries while tuning disk configuration options. What I found was the maximum data we could push through any single query was about 800 MB/s on a Phenom II X6 3.3GHz. As PostgreSQL cannot use use multiple processors/cores to speed up a single query, it turns out the SB850 1100 MB/s limit is not as great as it seems.

Hardware Upgrade Options

PostgreSQL can assign different queries to different processors/cores so if I had 2 queries hitting 2 different giant tables, that would in theory need 1600 MB/s for maximum speed -- 2400 MB/s for 3 queries. For these scenarios, it is "possible" high-end 6Gbps SAS/SATA RAID cards that can push all these SSDs. However, I'm beyond trusting specifications only and buying $1000 adaptors sight unseen for us to be the guinea pigs is not very appealing.

The best option appears to be a different CPU/MB platform not only to increase total disk I/O bandwidth but to supply a faster CPU to drive single-threaded PostgreSQL queries beyond 800 MB/s. The options I discarded after doing research:

Intel i7 Sandy Bridge: 2 GB/s to P55 SB, no ECC memory support
Intel i7 Nehalem: 2 GB/s to ICH10 SB, no ECC memory support
Intel i7 Xeon: seems like 2 GB/s non-video to H55 SB, nosebleed pricing for 6-core
AMD Opteron C32: 6Gbps SATA support only through LSI SAS, 2.8GHz max speed
AMD Opteron G34: 6Gbps SATA support only through LSI SAS, 2.4GHz max speed

After tossing out everything else, the only option left was the AMD SB950 chipset. It's interesting how the SB850 diagram was plastered in almost every review but I had a hard time finding a similar one for the SB950. I finally found it after clicking through a lot of links for motherboard reviews.

Woo! 4 GB/s for SB950 Alink Express III. That could mean 2.5 mega PostgreSQL queries could run at maximum speed. Of course, this is just a diagram -- who knows if this is true or not in the real world.

Conclusions

Beware if you plan on creating a mega SSD RAID0 array in hopes of going above 1 GB/s transfer rates. This is the bleeding edge as system platforms were mostly designed for the previous generation of SSDs where you were expected to stuff at most 4 x 300 MB/s SATA2 SSDs. Luckily, there is not much price difference between using a 2 x 240G SSD array versus a 4 X 120G SSD array. What you gain in possible future speed enhancements as ecosystems evolve to catch up, you do lose in possible future disk space expansion (by going from 2 x 240G to 4 x 240G).

Our plan for now is to live with the mere 1100 MB/s disk subsystem limit and wait for the FX-8170 3.9GHz (4.5GHz turbo) to be released. At that point, we'll upgrade the CPU and motherboard -- if the disk performance doesn't pan out, at least we might get a faster processor to push a single-threaded PostgreSQL query closer to the limit.

Equipment guaranteed to break

2011-10-01T11:43:58Z

We upgraded our main data processing server with 4x120GB Indilinx SSDs a little over 2 years ago. It was quite an astounding leap in performance going from hard drives to 1000MB/s reads, 800MB/s writes and 0 latency.

This week though, we started seeing SSD degradation from too many writes. The SQL software would crash under unexplainable conditions. After testing every possible factor (CPU, memory, OS, software), I finally narrowed it down to SSD burnout after examining a backup dump. Out of 5000 tables, there were 2 tables showing blocks of '0's which sounds like what happens when memory cells can no longer accept writes -- they revert to the default state of 0's.

On average, we did 100GB of writes per week which calculates out to 3TB of writes per 120GB drive. At a ratio of 3000:120, that's an average of 25 writes per location which is a magnitude less than the claimed 5000 write life for MLC SSDs. However, these are 1st generation drives so my guess is the write levelling algorithms are not very good and distribution of write are concentrated. To back that up, having ~10 bad blocks out of 350GB of data is a very low amount -- just about what you get for hard drives with bad sectors here and there. The problem is the remapping/detection of bad blocks for these 1st generation SSDs is just as bad as their write levelling. Hence having 0.01% of cells wear out due to concentrated writes is just as bad as the entire drive going bad if there's no way to avoid that 0.01%.

We will be replacing these SSDs with 5x Sandforce SATA 6Gbps SSDs next week so our server will soon have 2500MB/s reads and 2000MB/s writes. Sandforce controllers have built-in compression and error correction so it's possible these drives will come much closer to the rated 5000 write life for MLC memory. If not, it's just the cost of doing business and it's impossible to return to regular hard drives after using SSDs.

Rebalancing Commodities & Treasuries

2011-09-17T16:33:19Z

Earlier this month, I wrote about my silly investment strategy for my Roth IRA. I switched to this strategy mostly from the frustration of managing an integrated portfolio. I missed a lot of rebalancing opportunities due to the hassle of updating a daunting spreadsheet to calculate which accounts to buy/sell investments and which accounts needed changes in automatic contribution amounts. To solve for this problem, I am moving my tax-free/tax-deferred accounts to simple percentage holdings that can be easily rebalanced after a big market swing. (2 Vanguard fund accounts left to do -- just waiting for better market conditions.) For my taxable accounts, I will continue to update that complex spreadsheet once a year to determine contribution amounts for the year.

In my Roth IRA, I chose to rebalance commodities against long-term treasuries. I did not look at any data before picking this strategy as I was influenced by the core idea of Harry Browne's Permanent Portfolio to understand how/when/why asset classes move during different economic conditions versus looking at price data. Long-term treasuries do well in 2 different scenarios: deflation and flight to safety. Commodities do well in unexpected inflation and hot economic growth that increases demand for commodities. So we have 2 asset classes that have some overlap -- economic growth during times of decreasing interest rates -- but mostly protects against different conditions.

---

But now that I have been using this strategy for 2 years, I guess I could drum up some past performance numbers. Using Yahoo's historic prices, I was able to get daily dividend-adjusted prices since inception (12/8/2002) for PCRIX (Pimco Commodity Return) and use the same beginning date for VUSTX (Vanguard Long-Term Treasuries). The following graph shows what daily rebalancing looks like compared to buying 50:50 and forgetting about it.

Commodities (PCRIX)
Annualized Return: 11.66%
Daily Volatility (Standard Deviation): 1.34%
Biggest 1-day Loss: -9.04%

Long-Term Treasuries (VUSTX)
Annualized Return: 10.88%
Daily Volatility (Standard Deviation): 0.71%
Biggest 1-day Loss: -3.95%

50:50 Set & Forget
Annualized Return: 11.27%
Daily Volatility (Standard Deviation): 0.82%
Biggest 1-day Loss: -5.11%

50:50 Daily Rebalancing
Annualized Return: 12.41%
Daily Volatility (Standard Deviation): 0.77%
Biggest 1-day Loss: -4.83%

---

Obviously, daily rebalancing is beyond the ability for small investors. The tax drag and brokerage costs would be too great. How do the numbers look for annual rebalancing? For this scenario, I grabbed the performance numbers from Simba's backtest spreadsheet and set the return scale to logarithmic.

Commodities (PCRIX)
Annualized Return: 10.58%
Annual Volatility (Standard Deviation): 20.44%
Biggest 1-year Loss: -43.33%

Long-Term Treasuries (VUSTX)
Annualized Return: 8.24%
Annual Volatility (Standard Deviation): 11.35%
Biggest 1-year Loss: -12.05%

50:50 Set & Forget
Annualized Return: 9.65%
Annual Volatility (Standard Deviation): 12.97%
Biggest 1-year Loss: -26.92%

50:50 Yearly Rebalancing
Annualized Return: 10.19%
Annual Volatility (Standard Deviation): 10.37%
Biggest 1-year Loss: -10.44%

Holding 100% would have produced the highest returns but what a ride it was. The largest commodity futures loss was -43.33% in 2008 -- yikes! Starting out with 50% Treasuries helps but by 2007 comes around, Commodities grows to 75% of the total balance which meant you still saw a -26.92% loss in 2008. The smoothest strategy would definitely have been annual 50:50 rebalancing producing lower volatility and smaller losses than holding 100% Treasuries.

401k Lineup Tweaks

2011-09-08T02:27:00Z

Our balance in Vanguard Inflation Protected Securities went over $100K in August so I sent off a note to our contact at Employee Fiduciary to switch from Investor to Admiral class. When that gets settled, this is how our fund lineup will look like:

                                        Class    E/R   Alloc
--------------------------------------- -------- ----- -----
Vanguard Prime Money Market             Investor 0.23%    1%
Vanguard Intermediate Government Bond   Signal   0.15%   17%
Vanguard Inflation Protected Securities Admiral  0.22%   14%
Vanguard Value                          Signal   0.12%    9%
Vanguard Growth                         Signal   0.12%    5%
Vanguard Small Cap Value                Investor 0.37%    8%
Vanguard Small Cap Growth               Investor 0.26%    5%
Vanguard European Stock                 Signal   0.14%    5%
Vanguard Pacific Stock                  Signal   0.15%    6%
Vanguard Emerging Markets               Signal   0.22%    6%
Vanguard REIT                           Signal   0.12%    6%
Harbor Commodity Real Return            Instl    0.94%    7%
American Century Global Gold            Investor 0.69%   10%
------------------------------------------------------------
Dollar-weighted E/R                              0.29%

Out of the 13 funds, 7 have been either upgraded to a lower-cost share class or switched to an alternative fund tracking the same index/strategy. The asset classes with no changes:

Money Market -- There are no other optiions for a small 401K plan. There are institutional MMFs with lower E/Rs but with a $5M minimum, the Roman Empire (Planet of the Apes version) will return first.

Gold -- Vanguard has a lower E/R precious metals fund but American Century's fund tracks gold more closely as Vanguard invests in a wider range of metals. Plus Vanguard has a 1 year early redemption fee which makes it costlier to rebalance and with how volatile gold/gold miners are, rebalancing is good.

Small Cap Value, Small Cap Growth -- Vanguard just announced Admiral classes for these funds but I have yet to hear of a similar announcement for Signal classes. Obviously we would switch to Signal the moment Vanguard offered it. But if we had to wait for Admiral, our current allocation percentages would see us meeting the Admiral $100K minimum at $1.2M plan balance for Small Cap Value and $2M for Small Cap Growth.

With these changes, our overall plan E/R has gone from 0.38% at inception to 0.29% which comes out to about a $50 savings per year per plan participant. When all Vanguard funds are either Signal or Admiral, plan E/R would drop to 0.27%. Unfortunately, we would probably be above the $1M threshold for waiving pass-through trust fees.

Now for kicks, let's project how our plan might look if we were still using our previous 401K vendor with an average expense ratio of 1.25%. Note that I make no assumptions about rate of return as the majority of plan balance growth is from employee contributions through employee hiring and employee salary increase.

   Balance   EFC ER   EFC Exp   1.25%   Savings
   ---------   ------   -----   --------   -------
1      150,000   0.38%      570    1,875    1,305
2      300,000   0.38%    1,140    3,750    2,610
3      500,000   0.38%    1,900    6,250    4,350
4      700,000   0.29%    2,030    8,750    6,720
5      900,000   0.29%    2,610   11,250    8,640
6    1,100,000   0.35%    3,850   13,750    9,900
7    1,400,000   0.34%    4,760   17,500   12,740
8    1,700,000   0.34%    5,780   21,250   15,470
9    2,000,000   0.33%    6,600   25,000   18,400
10   2,300,000   0.33%    7,590   28,750   21,160
11   2,700,000   0.33%    8,910   33,750   24,840
12   3,100,000   0.32%    9,920   38,750   28,830
13   3,500,000   0.32%   11,200   43,750   32,550
14   4,000,000   0.32%   12,800   50,000   37,200
15   4,500,000   0.31%   13,950   56,250   42,300
16   5,000,000   0.31%   15,500   62,500   47,000
17   5,500,000   0.31%   17,050   68,750   51,700
18   6,000,000   0.30%   18,000   75,000   57,000
19   7,000,000   0.30%   21,000   87,500   66,500
20   7,500,000   0.30%   22,500   93,750   71,250
                        -------  -------  -------
                        187,660  748,125  560,465

Uhhh, OMFG? Total projected expenses paid over 20 years under EFC 401K is about $200K. Under the average 401K plan with 1.25% E/R (whether direct fees, wrap fees, 12b1 commmissions or hidden in reduced fund dividends), $7.5M.

UPDATE: Ooops, somebody on Bogleheads pointed out a spreadsheet error. Stupid OpenOffice auto-formatting decided to guess the % I entered were a magnitude larger. Actual difference is $560K over 20 years.

Stupid Roth IRA tricks

2011-09-04T17:16:00Z

It's been several years since I've been able to make a Roth IRA contribution and the back door conversion is too painful to consider since I have a hefty balance in a traditional IRA. As a practical matter, not being able to add more money has made integration with my overall portfolio a tougher challenge, especially with auto-rebalancing enabled for my 401K.

After a few years of treating my Roth IRA as part of my overall portfolio, I decided it was not working very well. The amounts in my other accounts were growing too fast and I was losing rebalancing opportunities by having just 1 holding in my Roth IRA. I puzzled over various strategies but the relatively low balance limited the options I could effectively pursue. I came up with the either brilliant or stupid idea of having 2 volatile but counterbalancing asset classes I would continually rebalance against each other. The 2 classes:

Commodities to capitalize on unexpected price inflation and soaring costs of commodities due to economic growth, especially in emerging market countries. My choice in this category is/was DBC PowerShares Commodity Index.
Long Term Treasuries to capitalize on deflation and flights to safety during big market drops. My preferred choice in this category always was EDV Vanguard Extended Duration Treasuries (30 yr) as the longest duration is the most volatile holding. However, trading volumes were not high enough in 2009 so I started with TLT iShares 20+ Year Treasuries. By the end of 2010, EDV trading volumes were high enough for me to comfortably use market orders. (I also waited until EDV's gain over TLT since my strategy start point had disappeared.)

Performance for this strategy for the past 2 years:

Commodities:

17.32% annualized return

Long Term Treasuries

19.80% annualized return

Overall Strategy

19.09% annualized return

So during this period, this rebalancing strategy has produced returns as high as holding either asset class in isolation but with a much smoother growth line. Looking at the graph, I missed out on a few rebalancing opportunities (01/10, 02/10, 07/10, 08/10) due work/travel/whatever that would have boosted my return over 20%. Otherwise, the 15 rebalancing transactions (purple dots) turned 2 volatile classes into a much straighter line. The green dot is a pending transaction for Tuesday after Friday's dismal job reports sent EDV up 6.25%.

While past performance -- especially just 2 years -- is no guarantee of the future, I am relatively confident in this strategy since it depends on market volatility and it is easy to implement in an "isolated" account. I don't have to calculate out 100% in Roth IRA = 3.46% and then propagate that number out against the rest of my portfolio. I don't have to worry about "interest rates are too low, don't buy bonds" or "market PE is too high, don't buy stocks" or "gold/oil/commodities are in a bubble". I just periodically monitor my balances and if it's not 50:50, I sell one and buy the other.

Unexpected costs of vacations

2011-03-01T23:06:25Z

After vacationing in Hong Kong and Hainan, I completely lost focus on my regular schedule -- no blog updates, no exercising. This happened also last year when I came back from the U.S. and Singapore. We really are creatures of habits and it's easy to throw ourselves off.

I'll try to get back into the groove of things ... after my trip to Guangzhou to add more pages to my passport.

Hainan vacation notes

2011-01-26T17:57:00Z

For the past month, the temperature in Taishan has been in the low 40s during nights and when lucky, up to the high 50s during the day time. Hence, we decided to take a trip to warm Hainan where it's 85 afternoon and 70 at night for a few days of beaches and swimming.

One night while taking a break from expensive tourist fare in a pulled noddle shop, the shopkeepers brought in a bunch of coconuts. In Hainan, almost every store has 10-15 coconuts sitting outside and for about 5rmb (cheaper for the green ones, more expensive for the orange ones), you can sip on refreshing coconut juice. Now this noodle shop is in the middle of a city just on the edge of a newly developed hotel resort district. There are no coconut tree farms nearby and I certainly didn't see a delivery truck drop anything off. So we asked the shopkeepers about it ... they gave a little laugh and said everybody just cuts fruits down from beaches/parks/resort areas when it gets dark. Nothing better than letting nature be your supply.

Hainan is a tropical destination in Asia -- especially for Russians and northern Chinese fleeing the cold winters. This means tourist traps are everywhere. At one restaurant, customers sign the order before it goes to the kitchen for cooking. The prices were twice as high as what we pay in Taishan but we held our noses and shrugged "oh well". But when the bill came, an extra line item appeared as "cooking service charge". What? This make sense at Chunyuan where you first buy your seafood at the market and then bring it over to the restaurants for cooking. No way you should pay a cooking fee when the restaurant is already charging nosebleed prices for their inventory. It didn't take much arguing for the fee to be waived which meant it was just their standard practice to sock it to unsuspecting tourists.

Another trap we ran into was rigged scales. Hainan is famous for fruits so as we passed through the small town outside Nanwan Monkey Island, our mouths were watering to buy the delicious fruits. Except when they filled up a bag of small mangos, it came up to 1.1lbs (the standard measure is "gun" or a half-KG). Our friend said "no way this is 1.5" and considering the shopkeeper didn't even try to argue about it, it seems likely their electronic scales were purposely set to over weigh. We did get a cheap glass of coffee for 3rmb though in this town (coffee also is grown locally in Hainan).

In any case, we finally hit Fruit Street in Hainan and loaded up on all sorts of exotic fruit. Hell, I don't even know the names of most of this stuff. There was spiny red fruit with insides like lychees. Purple skinned fruits with white insides that tasted like oranges. Greenish-grey pear-shapes with insides like kiwis (but sweeter and smoother). Small, round, green pears with an apple taste. And of course mangoes -- we bought a full case for 100rmb, brought them back and I'm now eating one a day. (Here's a link to some of Hainan's fruits.)

Another good option to avoid tourist prices are street BBQs. Almost anywhere in Sanya, you can find a vendor grilling food on the sidewalk with simple tables and chairs. Now street BBQs are also common in Taishan but the variety of food is much narrower. In Hainan, they also grill 10+ different types of fish, cuttlefish, peppers, mushrooms, corn, seaweed, rice bread and beef. The last two were fantastic!

Did I mention beef? In the U.S., beef is the #1 meat but in mainland China, pork is #1 -- Hainan is the big exception though. There is a huge culture of raising cattle in Hainan and herds of cows walking along/crossing roads is a common sight in rural areas. In Haikou, I had a delicious tenderloin steak (plus soup and sides) for 58rmb ($8.85 USD). The western-style restaurants here in Taishan commonly charge 88rmb-128rmb for a steak neither as large nor as good.

So these would be my takeaways for keeping costs down on a Hainan vacation: (1) Fruit Street in Sanya, (2) street BBQs, (3) steaks at western-style restaurants. Only eat seafood if you have a local guiding you to local dives.

Oh yes, one final note. Hainanese chicken is not available in Hainan. Strange right? You can find Wenchang chicken though and Hainanese chicken is what it evolved into after Hainan locals brought it to Malaysia and Singapore. Wenchang chicken has a similar cooking style but the bird is larger (hence not as tender) and the dipping sauce is sweet pepper instead of salt+oil+ginger.

Historic Portfolio Performance

2011-01-13T16:31:46Z

A few days ago, I mentioned my online tool to inspect historic returns. The inspiration for the tool was a NY Times article -- however it's chart only showed stock returns while my tool allows you to choose any combination of stock, bond, money market and gold.

After coming back from a quick trip to Hong Kong (bought an Olympus Micro 4/3 DSLR), I puzzled over the idea of integrating contributions and withdrawals into this chart. I first attempted to make it part of the same script but the input parameters and output data were too dis-similar. While it feels hackish (duplicate code in multiple files), copying the code into a totally new script allowed me to rework the internal logic without breaking the old one.

And here is the result:

http://personalbizfinance.com/pbf/data/portfolio_lifespan_start_end.pl

For those interested in the source code:

http://personalbizfinance.com/pbf/data/portfolio_lifespan_start_end.zip

What this dynamic chart effectively does reproduce my numbers from Savings Rates Analysis in a historical context. Enter years of working, savings rate (by using the savings rate -> withdrawal table), portfolio asset allocation -- and the red/green will tell you how often your scenario played out poorly or favorably since 1928. If you see too much red, tweak the withdrawal amount down (in effect increasing the savings rate) to find the threshold that gives you a high probability of success.

Rebalancing Issue with Employee Fiduciary

2011-01-08T16:33:00Z

Originally posted December 29, 2010

I just ran into a rebalancing issue with Employee Fiduciary's 401k platform. My personal setup has automatic rebalancing enabled for every 3 months on the 25th. Easy breezy -- except we had a fund change from VFITX to VSIGX on the 22nd. While the Signal share upgrades within the same funds (e.g. Value Investor to Value Signal) happened immediately, VFITX to VSIGX required a full sell + buy and didn't complete until the 27th.

Well you can guess what happened. During this time from the 22nd to the 27th, my existing balancing in VSIGX showed a pending status with 0% as the balance. So when my rebalancing transaction happened on the 25th, a big chunk from every other fund was sold in order to bring up this balance to 20%. And when the pending transfer finalized, the total jumped up to 36%. Ooops?

I tried to immediately reallocate holdings back to my original 20% but the system's market timing rules blocked my transactions. (You cannot manually buy back into Vanguard funds until 60 days after a sale.) For now, I've changed my new contributions to ignore VSIGX and will attempt another manual rebalancing in February. I also fired off an email to EFC to see if these limits can't be lifted for my case and am awaiting a response.

Luckily, it is only Treasuries -- the least volatile option on our plan. Unluckily, everybody is on the edge about interest rate increases.

Updated January 8, 2011

I didn't have time to update this entry before my trip to Hong Kong. EFC emailed me saying their operations group reviewed my bug report and rebalanced my funds to my target allocations as of the 25th. I logged into my account and saw the transfer transactions with settlement dates as of 1/3/2011. The markets didn't do much in that period so no harm, no foul.

Luck of the Market

2011-01-08T06:49:00Z

Originally posted January 4, 2011

Here's an article in NY times about the luck of investing. The graph can take a while to soak in so you will need to stare at it for a while.

In Investing, It's When You Start And When You Finish

After looking at this, I decided to create the same graphics for bonds. I found data T-Bond returns for 1928-2006. From looking at the more recent years, they correspond closely to Intermedate Treasuries so I filled in the missing years with VFIUX's returns.

TBonds_Real_Return_In_Out.html

Updated January 7, 2011

Updated chart link:

http://personalbizfinance.com/pbf/data/real_return_start_end.pl

It's now a dynamic script where you can choose allocations between 4 categories (stock, bond, money market, gold). I don't validate the percentages since most users should be able to do the math. The open TITLE tag is fixed for Chrome and added NOWRAP for Safari.

You can also download the source code at:

http://personalbizfinance.com/pbf/data/real_return_start_end.zip

2010 Year End Graphs

2011-01-01T06:24:00Z

It's the year-end graph blow-out sale! There will be no pretense of analysis. Instead, I will simply bombard you with pretty colors.

Portfolio Return vs Stocks -- 2010

Portfolio Return vs Bonds -- 2010

Portfolio Return vs Commodities -- 2010

Portfolio Return vs Balanced Funds -- 2010

Portfolio Return vs Managed Payout Fund (VPDFX) -- 2010

Asset Allocation Totals -- 2006-2010

Asset Allocation Percentages -- 2006-2010

Asset Allocation Returns -- 1996-2010

Asset Allocation Gains Percentage -- 1996-2010

Asset Allocation Losses Percentage -- 1996-2010

Portfolio Contributions vs Returns -- 1996-2010

Portfolio Contributions vs Returns Percentage -- 1996-2010

Savings:Return Ratio -- 1996-2010
positive off-the-charts = negative returns
negative off-the-charts = debt incurred

Net Worth Growth -- 1996-2010

Tax, Spending, Saving Rates -- 1989-2010

Net Income: Spending, Saving -- 1989-2010

Gross Income: Tax, Spending, Saving -- 1989-2010

Retail Admiral Minimums Reduction

2010-12-31T13:48:00Z

Originally Published 12/24/2010, Updated 12/31/2010

About a month before Vanguard eliminated Signal share minimum requirements, Vanguard also changed the Admiral requirements for retail investors . From $100K, the requirements were changed to $10K for index funds and $50K for actively-managed funds. (Note this change only applies to those investing directly with Vanguard -- those invested via 401K, SIMPLE-IRA, investment advisor omnibus accounts still have the $100K requirement.) Of course, I immediately upgraded all my possible funds the moment this news came out -- nice fee reduction for me.

But while poring through annual reports , another thought came to mind. Are these minimum reductions good for previous Admiral holders? Keep in mind the structure of Vanguard. The investors of each fund are indirectly the shareholders of the parent Vanguard company. Hence, all costs are simply pass-through -- there is no parent company subsidizing expense ratios for other funds. (If you browse through other mutual fund companies, you often run across "gross E/R, net E/R" where the difference is subsidized.) What this means is if somebody in the Investor class is incurring $100 in annual fees from transactions, paper statements, paper prospectuses, phone calls to customer service and so on -- moving this person to Admiral class will also move these expenses there.

To illustrate, I looked through the annual report for Vanguard Total Bond Market. It is an index fund so it is part of the new $10K retail Admiral minimum. It also has all the possible share classes of ETF, Investor, Admiral, Signal and Institutional. Here's how the existing expenses break out:

The annual reports have some rounding in them but the calculated E/R is close enough to reported values to work with. For our purposes, we will use +/- against the calculated expenses.

Shift 100% Assets / 100% Expenses

Let's imagine the most extreme scenario. Everybody in Investor class qualifies for Admiral and hence all the costs move over.

With all Investor costs bundled up with Admiral, the expense ratio is +0.06% higher (0.16% versus 0.10%). So for reported numbers, expect 0.18% -- this hurts existing Admiral holders as it's closer to Investor 0.22% than Admiral 0.12%.

Now most expenses are roughly fixed. Somebody with $100K requesting a paper statement incurs the same cost as a $10K account. That means as the dollar threshold for conversion goes up, the percentage of expenses shifting will decrease faster than the assets shifting.

Shift 75% Assets / 70% Expenses

Admiral cost go up by +0.05% E/R, those left in Investor +0.04%.

Shift 50% Assets / 40% Expenses

Admiral +0.03% E/R, Investor +0.04%.

Shift 25% Assets / 15% Expenses

Admiral +0.01% E/R, Investor +0.03%.

From looking at these numbers, my guess is somewhere between the last 2 scenarios are the likely outcome of the new $10K minimums. This does not factor in the impact of new money from outside Vanguard. If the $10K limit attracts money that would have gone to Fidelity Spartan funds or somewhere else similar, it probably will offset the additional costs for Admiral. We'll have to wait until the next-next annual report to look through the numbers again. (The next annual report won't show a full year of the new minimums.)

Update (12/31/2010)

While discussing this possibility at Bogleheads, another thought struck me. Seems rather arbitrary that Admiral requirements were reduced to $10K for index funds but only $50K for actively-managed funds. Now there something "special" about index funds at Vanguard and that is they can offer ETF shares of those funds. And from looking through all Vanguard's funds, Admiral+ETF+Signal shares always have the same expense ratio even though the above calcs show they are not exactly the same. That seems to imply those 3 classes calculate their expenses in a pool and perhaps what is making the lower Admiral minimums possible is the additional assets growing on the ETF side. Let's recalculate pooled expenses in the 50/40 scenario:

Using the pooled calculation decreases the change from +.03% to +.02%. In addition, Vanguard ETFs have been rapidly gaining customers and the latest quotes on BND show assets have grown by another 3B since the last annual report.

China's Housing Bubble and Divorce

2010-12-26T08:06:00Z

When the housing bubble crashed in the U.S., it sent the economy in a big tailspin putting many families under stress. By comparison, the flush of money flooding into China's housing sector is generating a wave of divorces... What? You are probably asking if I forget a "not" somewhere.

Let's go back to the beginning. China's economy is capitalism on steroids but the country is still run by a totalitarian government who tries to rule from top-down. The government knows the dangers of a housing bubble and beyond the interest rate hikes, they have many internal roadblocks that would be foreign to those used to freer markets. For those buying their 1st property, there is a 25% downpayment requirement for mortgages. For a 2nd property, DP jumps to 50%. (I *think* these numbers just went up to 35%/60% but we don't need to know the exact number to understand the purpose of the difference.) For a 3rd property, you cannot get a mortgage. In addition, the house transfer fees go from 1% to 10% if a property was not previously held for 5 years.

China applies their limits by household registry. A married couple is interpreted as a 1 entity and if they already own property, they cannot buy another one without running into the 2nd property penalties -- even if they only put the 1st property under one of their names. Of course, it doesn't take a rocket scientist to see how to get around this problem. Couples all over China are divorcing in order to buy 2nd houses/condos to bypass these rules and then remarry afterwards. Of course, this only works if the parents' household registry a divorcee returns to doesn't already have siblings pulling the same stunt.

Treasury vs Government Bond

2010-12-22T04:25:00Z

Not too long ago, Vanguard eliminated the minimums for Signal class shares. This effectively means all 401K plans could in theory get a price break ... if the people in charge weren't more interested in ripping people off. For our plan, Employee Fiduciary just made the switch -- the change happened immediately after I emailed them the final go ahead. What took me some time was deciding whether to wait for Vanguard Intermediate Treasury Admiral (VFIUX) or go immediately with Vanguard Intermediate Government Bond Signal (VSIGX). From looking at the growth graphs, the 2 funds track each other pretty well.

From a quick glance, it seems like VFIUX at 0.12% ER is a slightly cheaper option than VSIGX at 0.15% ER. But then my mind starting pondering deeper. Vanguard does not arbitrarily set an expense ratio for funds. Instead, they report what last year's E/R was and it's just assumed that number will be in the same range in the future. So when Vanguard launched VSIGX just a short time ago, there was no "last year" to base the E/R numbers -- where did it come from?

First step was to look at the annual reports. Here are the numbers for Vanguard Intermediate Treasury:

Next for Vanguard Intermediate Government Bond Index:

From looking at 9 months, it appears the 0.15% is mostly an estimate based on previous fund launches. The actual expense ratio is unknown until Vanguard does another annual audit.

Can we project what it might look like after more net assets accumulate? (With Signal share minimums reduced to $0, the puny $3M in Signal should increase a lot.) To estimate costs, I dwelved into a lot of annual reports and compared funds with Investor/Admiral against those funds I compared funds with Investor/Admiral shares against those with just Signal/Institutional shares. The following table is a rough estimate of costs breakdown at higher asset values:

My educated guess is VSIGX will have about the same 0.12% E/R as VFIUX very soon if it isn't at that level already. For this reason, I decided to make the switch to VSIGX immediately in our 401K pan instead of waiting a few more months for VFIUX. Less confusion, less changes is better when dealing with a general population of 401K participants.

Lessons in TIPS Market Timing

2010-12-19T22:56:00Z

I've mentioned in the past that a strategy I like to use is TIPS duration shifting based on real rates. The quick summary here is when real rates go down, sell longer duration TIPS and buy shorter ones -- do the opposite when real rates go up. On first impressions, it seems to have worked decently for me for me as I've made 3 roundtrips since 2007. But perhaps I should quantify the performance to see whether it's worth my effort?

Period 1

Eyeballing this graph, I scooped up 10% on this transaction. By comparison, if I had stayed in Vanguard Short Term Treasury (VFISX), my return would have been 8%. About an extra +2% bonus.

Period 2

5% on this transaction compared to 2.5% in VFISX for +2.5% bonus.

Period 3

Technically this is not officially part of the strategy. I had been slowly getting rid of European holdings and made a slight bet on U.S. TIPS.

4% on this transaction compared to 1% in SHY for a +3% bonus.

Overall

If we looked at just those periods, increasing bond returns by an average of 2.5% over 6 months is a pretty nice deal. With bonds paying out less than 1% on the short term and less than 3% for intermediates, 5% annualized is a whopping jump. But let's zoom out and compare just holding everything indefinitely? (I hand drew the purple dotted line in the following graph to estimate my transactions.)

Holding TIPS 100% = +23%
Holding Short Term 100% = +17%
Shifting = +40%

From the eyeball numbers, the difference calculates out to +5.8% annualized over the course of 2 1/2 years. Not bad at all for bonds -- however, my inclination is to phase out this strategy for the following reasons:

1) When you sell TIPS, you have no idea when you will get back in. And if your asset allocation calls for an inflation-protection component, luck may not favor you if you are out during a critical period.

2) You have to keep up with the markets. I've gotten lazy lately and on my last roundtrip, I missed out on another 3%-4%. But even if I was watching the markets, perhaps I might have been fooled by the pre-QE2 rumors thinking there was another bump to come. Instead, the direction went the opposite direction after QE2 with rates going up and prices going down.

3) My tax-deferred play space (IRA/Roth IRA at Wells Fargo Brokerage) is decreasing percentage-wise every year as I can no longer contribute to either account. An extra +10% on this sliver available comes out to +0.36% and at some point, it will not be worth my effort.

With the availability of short duration TIPS ETFs (STPZ and STIP), I've moved 100% of my play space bonds into STPZ as rates still "feel" too low but I still want some inflation protection. I'll probably check real rates once in a blue moon when the impulse hits and decide then whether to give this another try.