Wednesday, March 27, 2013

“Building ractives is so addictive it should be illegal!”

clickme is an amazing R package. I was not sure what to expect when I first saw Nacho Caballero's announcement. I actually was both skeptical and intimidated, but neither reaction was justified. The examples prove its power, and his wiki tutorials ease the noobie difficulties. Very similar to shiny, clickme serves as an integration point for html, javascript (especially d3), and R. While clickme does not allow the R websocket interactivity that shiny does, its more concentrated focus on quick reproducibility and sharing makes it a very useful tool. This is very much in the spirit of Reusable Charts. ractives defined as

(short for interactives-a hat tip to Neal Stephenson), which are simple folder structures that contain a template file used to populate the JS code with R input data

provide the structure for clickme to produce an html file from

  1. a template in R markdown (template.rmd)
  2. a translator R script (translator.r)
  3. a data source
  4. external scripts (probably javascript) and styles (.css).

Inspired by the clickme longitudinal heatmap example, I just had to try to create my own ractive. I thought Mike Bostock's line chart example would serve as a nice template for my first ractive. The data not surprisingly will come from the R finance package PerformanceAnalytics dataset named managers. With very minor modifications to the Bostock source and a simple custom R script translator (translator.R shown below), we have everything we need for this ractive, which I will call multiline.

#' Translate the data object to the format expected by current template
#' @param data input data object
#' @param opts options of current template
#' @return The opts variable with the opts$data variable filled in
translate <- function(data, opts = NULL) {

# I would like to generalize this to handle both price and return right
# now just handles return clickme template.Rmd javascript can handle
# prices or cumulative so we will send cumulative which can serve as price

# remove na
data[] <- 0
# get cumulative growth
data <- cumprod(1 + data)

# convert to data frame
data.df <- data.frame(cbind(format(index(data), "%Y-%m-%d"), coredata(data)))
colnames(data.df)[1] = "date"
# melt the data frame so we have our data in long form
data.melt <- melt(data.df, id.vars = 1, stringsAsFactors = FALSE)
colnames(data.melt) <- c("date", "indexname", "price")
# remove periods from indexnames to prevent javascript confusion these .
# usually come from spaces in the colnames when melted
data.melt[, "indexname"] <- apply(matrix(data.melt[, "indexname"]), 2, gsub,
pattern = "[.]", replacement = "")
opts$data <- df2json(data.melt)

Now to create our first clickme html page, we just need a couple lines of code in R.

# if not already installed, uncomment the tow lines below
# library(devtools) install_github('clickme', 'nachocab')

# set location where you put your multiline ractive
data(managers) #although I use managers, really any xts series of returns will work
clickme(managers, "multiline")

Then, we have a web page that will create an interactive d3 line chart using the cumulative growth of the managers return series. If you do not see the embed below, then please follow the link.

Eventually, it will be very nice to have an entire gallery of amazing ractives.

git repo

Friday, March 22, 2013

Production Quality Report with R and knitr on Yen

Sometimes I actually use my experiments for real work.  For example, I wanted to send an update  on the Japanese Yen.  This was a great opportunity to use the chart created in Shading and Points with xtsExtra plot.xts.

I was fairly pleased with the finished product (click here if embed does not show).

R sweave file from GIST:

Thursday, March 21, 2013

Dust off 130 Year Old Gold Books on Google Bookshelf

The very fine paper

The Golden Dilemma (January 8, 2013)
Erb, Claude B. and Harvey, Campbell R.
Available at SSRN:

discussed in this post How Did I Miss “The Golden Dilemma”? motivated me to revisit some really old books on gold that I had added to my Google bookshelf. I am always surprised by how centuries old finance and economics conversations still sound so relevant. Even more interesting is how many of the questions remain unanswered. My new favorite discussion on gold and deflation occurs in Gold and Prices Since 1873 by James Laurence Laughlin and starts like this (emphasis and links mine):

§ I. Much of the difference of opinion as to the significance of recent movements of prices is due to the fact that the value of gold is a ratio which varies with a variation in either of its terms. Whether commodities fall in relation to gold or gold rises in relation to commodities, in either case the value of gold has risen. The same phenomena, therefore, may be due to radically different causes. So that, admitting the fall of prices, it is said, on the one hand, that the rise in the value of gold is due to some cause affecting gold itself, such as scarcity; and, on the other hand, it is claimed that the fall in prices is due to causes connected solely with commodities, and not with gold.

The believers in the scarcity value of gold substantiate their position by reference to the falling off in the annual production of gold; the unusual demands for gold since 1873, by Germany, Italy, and the United States; stringencies in the money market; the increased use of gold in the arts; the claim that the fall of prices is general; the exceptional character of the depression of trade since 1873; the general existence of low wages, profits, and rents; and the absence of any progress since 1873 in the means of economizing gold and silver. These opinions have been prominently associated with Mr. Robert Giffen, the statistician of the English Board of Trade, and Mr. Goschen, the present Chancellor of the Exchequtr; while the evident connection of the main proposition with bimetallism has given it a semi-political character, and many supporters in both Europe and America.

§ II. Inasmuch as the rise in the value of gold since 1873 is in proportion to the fall of prices, it is a matter of some importance to look critically at the facts in regard to prices. With this object in view, the more important tables of prices since 1850 have been collected in the Appendix, with explanations as to the methods of computation, sources, and reliability. It is hoped that a comparison of the diverse methods and results of these tables will serve a useful purpose.

Hitherto, the figures of the London Economist for twenty-two articles have been almost universally used as evidence in regard to the movement of prices; but it is time that the worship of this fetich should cease.* Of late, much more trustworthy tables have been published.

I very much like the author's approach, which is to consider both the numerator and denominator in a price fraction and to gather as much data as possible rather than rely on one potentially flawed dataset. By page 3 we are already presented with this wonderful chart on prices/inflation in Great Britain, France, Germany, and the United States. These extra sources of price data allow us to frame the discussion much more objectively.

[Chart 1 from the Table]


In the spirit of the author's approach, I wanted to reproduce the chart presented above in R and also compare the US Burchard's table with CPI and GDP deflator data from MeasuringWorth. I tried to take the easy route with OCR, but eventually manually entered the price tables from the appendix.

plot of chunk unnamed-chunk-1

The chart was well done and really could not be improved beyond color and labels.

As I have mentioned before, MeasuringWorth is a wonderful source of long-term economic and financial data. I wanted to compare Burchard's US table of prices to MeasuringWorth's US GDP deflator and CPI. The Civil War significantly destabilized prices. The effect is much more pronounced in the MeasuringWorth data.

plot of chunk unnamed-chunk-2

The author discusses the shortcomings of the Burchard prices in Appendix Table M.

An examination of the Finance Report tables indicates that they were not compiled with great care. The price of an article will run along without change from month to month; then, suddenly, it will rise or fall sharply. The prices of articles that normally would fluctuate together (pig and bar iron, butter and cheese) show a very loose correspondence. Moreover, the same articles do not appear from year to year. An article will be quoted for a number of years, will then disappear, and later, perhaps, will reappear…Notwithstanding these serious imperfections, we reprint Mr. Burchard's figures for the years 1850-84, since they are the only continuous figures of average prices in the United States.

The author organizes his remaining arguments into 8 additional sections, which I have titled. Also, I will give my favorite passage from each section. Interestingly, many of these same arguments apply directly to the MMT world of today with the only exception being the absence of a gold standard.


Section 3: The Supply of Both Money and Credit Must Be Considered

In truth, in society as it exists to-day, the general level of prices for considerable periods (sufficiently long to permit the effect of changes in the business habits of the community, or changes in the existing stock of gold, to be felt) must depend upon a combination of the quantity of money with the various forms of credit. The two are inextricably bound together. So, therefore, the level of prices (so far as it is affected by the offer of purchasing power) depends on the expansion or contraction of two factors, quantity of money and credit, each of which may change to a considerable extent independently of each other. Both may increase or diminish together, or the gain of one may offset the loss of the other.

Section 4: Other Events Affecting Prices

The extraordinary and exceptional demand for commodities in periods of war, at the very time of the great destruction of wealth, produced an unhealthy state of affairs; but on the outside all seemed fair, and men had begun to believe that prices were fated always to rise. The speculation in metals (see Chart II.) in 1873 was of an unparalleled kind…There were all the evidences of an unhealthy and abnormal condition of affairs. But the unchecked demand, when the actual power to buy had been greatly impaired, could not go on forever. When it was once found that men had been creating liabilities beyond their means to meet them, the end had come. The crisis of 1873 was the painful return to a consciousness of the real situation, after a prolonged fever of speculation for nearly twenty years, which had spread over many countries. The effects were the more serious because the disease had got such great headway.

Section 5: Improvements in Industrial Process and New Supplies of Commodities

The period following a great financial upheaval is naturally crowded with improvements in processes and in methods of lowering the cost of production. Necessity becomes the mother of invention. The extent to which producers have been driven by the fierce competition since 1873 to cheapen production leads to the inquiry how far the fall of prices can be accounted for by influences connected solely with commodities, and not with gold. If these influences have been widely extended, it will be strong evidence that the scarcity of gold has had less effect than some suppose.

Section 6: Analysis of Prices by Commodity Type

Whether to draw inferences as to a scarcity of gold from forty-nine articles, or to infer that gold was abundant, according to the prices of fifty-one articles, is an awkward dilemma for those who think that prices give direct evidence as to the quantity of money. As Forsell remarks, the theory of a scarcity of gold is incompatible with the rise * in price of so many commodities.

Section 7: Supply of Gold

From these figures, it will be seen that the reserves in the banks of the civilized world show a very remarkable increase in gold. Although the total note circulation was increased 29 per cent., the gold in the reserves was increased 75 per cent., while the silver was also increased 25 per cent. In 1870-74, the gold reserves amounted to 28 per cent. of the total note circulation, and constituted 64 per cent. of all the specie reserves. In 1885, the gold bore a larger ratio to a larger issue of paper, or 41 per cent. of the total note circulation; and, in spite of unusual accumulations of silver (in the Bank of France, for example), the gold formed 71 per cent. of the specie reserves. This is a very significant showing. What it means, without a shadow of doubt, is that the supply of gold is so abundant that the character and safety of the note circulation have been improved in a signal manner. In 1871-74 there was $1 of gold for every $3.60 of paper circulation.* In 1885 there was 11 of gold for every $2.40.

Section 8: Supply of Credit

Specie to the amount of 1800,000,000 has gone into circulation in the form of note issues, representing an equivalent amount of specie; but gold has not been economized by the use of credit in the form of notes. While the total circulation of these countries has increased 35 per cent., the paper has been much better protected…The paper currency of every country except Russia has gained in security, together with a large increase in many of the countries. The gold supplies have not merely permitted an enlarged note circulation, but have furnished a much better protection to that increased issue.

Section 9: Price of Land Labor

It will be recalled at once, in regard to rents, that a marked characteristic of the period since 1873 has been the opening up of new and fertile lands, whose products have been transported at a greatly diminished rate. But this in itself is a reason why lands in the older countries should be thrown out of cultivation, and why rents should be lowered. This phenomenon, then, can be accounted for on other grounds than the scarcity of gold…The fact that wages have risen tends to confirm the belief that the fall of prices is due chiefly to the introduction of improvements.

Section 10: Conclusion

To assume that because the fall of prices coincided with the demonetization of silver it was due to an appreciation of gold, without considering whether the coincident phenomena were traceable to entirely distinct causes, is to fall into the fallacy of post hoc propter hoc. The forces which fix the level of prices at any time, moreover, are far too complex to admit of the inference that, because prices have fallen seriously, gold has become scarce.

Gold and Prices Since 1873 by James Laurence Laughlin is a reminder of how our current is wedded to our past. I often feel like the unprecedented actions of the world's central banks have broken the link to centuries of financial history. Rather, it seems they have simply temporarily suspended the immutable laws of humans and money and maybe “The effects were the more serious because the disease had got such great headway.” will haunt us.

I ran out of time and could not do the visualization in d3. Anybody that is motivated, here is a way to get the JSON:

# for the ambitious and more motivated than me, here is how to easily get
# the JSON to translate the graph to an interactive
toJSON(list(Date = format(index(priceTables), "%Y-%m-%d"), as.list(priceTables)))

Post entirely generated from R markdown for reproducibility:

Monday, March 18, 2013

One Pager Performance Report with knitr, R, and a Different Font

Although I suffer from complete ignorance of typography, with a little help from a post from Hyndsight and post from mages' blog, I wanted to try a different font on the one-pager performance report that we created in Onepager Now with knitR. I do not think Open Sans Light is the best choice for this report, but since it is the most popular font on Google I figured I could not be criticized too heavily by those more knowledgeable than me in typography.

options(tikzDefaultEngine = "xetex")


knit2pdf("onepager_text_graphics-knitr.rnw", compiler = "xelatex")

And the result (go to Box if the embed does not appear below):

R code from GIST:

Friday, March 15, 2013

How Did I Miss “The Golden Dilemma”?

I am ashamed to admit that I am way behind (about 10,127 downloads) in discovering this wonderful paper:

The Golden Dilemma (January 8, 2013)
Erb, Claude B. and Harvey, Campbell R.
Available at SSRN:


Here are the authors presenting the concept in July 2012 if you prefer slideshow format (thanks

A presentation on “The Gold Dilemma” by Claude Erb and Campbell Harvey. (Duke)

I have struggled with the various reasons I have heard for owning gold, which often conflict with each other, and how gold cannot universally work in all environments.  The authors enumerate and explore 6 arguments for owning gold:

    1. “gold provides an inflation hedge”
    2. “gold serves as a currency hedge”
    3. “gold is an attractive alternative to assets with low real returns”
    4. “gold a safe haven in times of stress”
    5. “gold should be held because we are returning to a de facto world gold standard”
    6. “gold is underowned”

In previous posts from 2010 and 2011, I focused on the gold as a very expensive inflation hedge, but I was nowhere near as exhaustive as the authors on this point and did not even touch on the five others.

I very much liked Exhibit 6 from the paper outlining scenarios based on varying 10 year forward inflation and a gold/CPI constant (3.2 average since 1975 and 7.1 as March 2012).


Although the gold/CPI ratio averaged 3.2 1975-2012, the authors extend the analysis to 1791 using the very fine data gathered at MeasuringWorth (more later in the post on importing this data to R for the R geeks).  The gold/CPI ratio has averaged 2 since 1791, which is clearly much lower than the almost current 8.  Using the table above would suggest a negative nominal return unless annualized inflation exceeds 15% for the next ten years.  Just think what happens to other markets in this environment, and gold might not even be the worse.


I was also delighted to see the authors reflect my concern that argument 2 could easily be the same as argument 1 in a debasement environment.

“A second way to interpret the “gold as a currency hedge” argument sees “gold as a hedge of my own currency, spent in my own country, when the local government is printing money with abandon”. This is also sometimes referred to as “currency debasement”. If this debasement is a result of inflation, then this interpretation is just another version of the “gold as an inflation hedge” argument.”

I highly recommend readers work through the other arguments with the authors by reading the original paper, but (spoiler alert) if you do not want to do that, I think the conclusion very simply is that after a mammoth increase in gold prices, none of the 6 arguments can justify the current price.  The “gold is underowned” argument is the strongest but even then requires a unique marginal buyer “[who] are not focused on ‘valuation’.”  Of course, we do have a lot of central bank buyers now of every financial asset “not focused on ‘valuation’.”

Now almost for the R as promised, but first the data at MeasuringWorth is truly special, and I would liken it to the other unbelievably generous set of financial data at  Although no one has asked that I do this, please consider making a donation at to insure the continuation of this data and hopefully maybe the collection of more.  Non-US financial data prior to 1900 is nearly impossible to find.

MeasuringWorth data is extremely easy to access in R. As a simple example, I wanted to pull in the price of gold since the 1700s.  I did it in 2 lines of R.  In 5 I got this graph.


R code from GIST:

Thursday, March 14, 2013

Doing Things the Hard Way with d3

Although embarrassing, I thought this might help some folks learning d3.js.  Most of my struggles with d3.js center around getting the data in proper form.  I think most of it is mindset.  d3.js offers a full range of helper functions to easily accomplish this task, but it seems my brain has been conditioned to think a different way.  In some ways, I liken it to my initial difficulty with R’s set of apply functions.  With a lot of practice, I think once you get it you get it.

Take for example a recent experiment with zui53 and d3 on index data.  I spent more than an hour coming up with this mess:

var indexkeys = d3.keys(d3.nest().key(function (d) { return d.indexname; }).map(data))
indexdata = indexkeys
.map(function (name) {
return {
name: name,
values: data.filter(function (d) { return d.indexname == name; })

The original data is a csv in long format

which I would like to look like.
Although my ugly mess works, after a couple days away from the code, I had a thought after reading D3 Nest Tutorial and examples for a fifth time.  I could have just simply done this.
indexdata = d3.nest().key(function(d) {return d.indexname;}).entries(data)

I thought I had it all figured out after!topic/d3-js/biALMQnYUgo, but as you can see not even close.