• A choice between N options is modelled as a multinomial distribution with parameters θ₁, θ₂, θ₃ … θ_N, where the thetas also represent the probabilities of each option being chosen. For example we might model votes cast in an election as draws from a multinomial distribution with parameters θ₁=0.7, θ₂=0.2, θ₃=0.1.
• However the multinomial distribution by itself is likely to be a poor model of the choices made within a population as it assumes all individuals select options with the same probabilities. It would be more realistic to say that the thetas themselves vary over the population. This gives us what is known as an over-dispersed distribution: the parameters for one distribution are modelled by another distribution. In this case we use a Dirichlet distribution, which is the multivariate version of a Beta distribution, to model the distribution of the thetas.

As we’ll be using it a lot here’s the probability density function for the Dirichlet distribution.

Where the normalising constant is:

One of the powerful things about the Dirichlet distribution as a modelling tool is that it allows us to capture not only the proportions of the population that opt for each choice but also the amount of switching between the choices from one draw to the next. To take our election example again, an election result of 70%, 20%, 10% for three parties could be modelled by Dirichlet distribution with alphas of 0.1, 0.029 and 0.014 or with alphas of 20, 5.71 and 2.86. In fact there are infinitely many possible settings of the alpha parameters that will produce this result. The difference between them is stability. If two successive elections produce the same result then this could be because the same people are voting for the same parties or, less likely but equally possible, people are switching their votes but in such a way that the net result is the same. Different settings of the alpha parameters produce different levels of switching.

A natural question is then: given a particular parametrisation of the Dirchlet distribution, what is the expected percentage of individuals that will switch category from one draw of the multinomial distribution to another?

I’m sure this has been worked out before somewhere but after a quick and fruitless trawl through the online literature I decided to do it myself, helped a lot by a great post from Leo Alekseyev who demonstrates a clever way of integrating over Dirichlet distributions. All I’ve done is adapt his technique.

(By the way to convert the latex in this post into a form easily used in wordpress I used an excellent python package from Luca Trevisan)

So let’s say we have N choices. For individual i the probability of picking choice j is θ_ij. What then is the probability that a randomly selected individual will make the same choice in two successive draws from a multinomial distribution? The individual could either select the first option twice or the second option twice or the third option twice etc. In other words the probability we are interested in is

We will call L the loyalty and work out expected switching as 1-E[L].

Leo Alekseyev has created an excellent video on you tube talking through his technique. I would recommend watching it if you would like to follow the arguments below. If you’re just interested in the end result then scroll to the end of the post.

The quantity we are interested in is where the come from a Dirichlet distribution with parameters To get the expected value of L we can use a generalised version of the Law of the Unconscious Statistician to adapt the proof given by Leo Alekseyev. As a reminder, the Law of the Unconscious Statistician is:

where is the probability distribution of the random variable X.

This will give us the following integral

So how do we evaluate this integral? It’s domain D is not straightforward as it is constrained by (i.e. the probabilities must sum to zero).

Leo Alekseyev shows us a trick using the Dirac delta function:

This (generalised) function, the limit of increasingly concentrated distributions, has an area of one beneath the curve (if you can call it that) at x=0 and an area of zero everywhere else – a slightly odd concept sometimes thought of as infinitely tall spike above the origin. The helpful thing for us is if we set and multiply the contents of the integral by the delta function then this is equivalent to evaluating the integral over D.

Since…

… the integral can be rewritten as:

If we group together like terms we reach:

Note the pattern in the exponents!

Continuing along the lines described by Leo Alekseyev we use the substitutions and to set us up for the Laplace transform and evalute the integral at to set us up for the inverse Laplace transform.

As a reminder the Laplace transform is:

So we can substitute it in giving us:

The Laplace transformation evaluates as:

Which we can substitute back into our integral:

Since we get:

Some rearranging gives us:

Now we use the inverse Laplace transform to evaluate as giving us:

Bringing the normalisation constant back in and cancelling out we get

which is our expected loyalty .

All we need to do now is check it’s right by comparing with a simulated result in R

comp<-NULL

for (i in 1:100){

alphas<-runif(3, 0.01, 10)

#Simulate
sample.d1<-rdirichlet(10000, alphas)
purchases<-t(mapply(function(x, y, z) rmultinom(1, size = 2, prob=c(x,y,z)), sample.d1[,1], sample.d1[,2], sample.d1[,3]))
loyal<-sum(purchases[,1]==2)+sum(purchases[,2]==2)+sum(purchases[,3]==2)
switching.sim<-(10000-loyal)/10000

#Derived

switching.dir<-1-(sum(alphas*(alphas+1))/(sum(alphas)*(1+sum(alphas))))

comp<-rbind(comp, c(switching.sim, switching.dir))

}

colnames(comp)<-c("Simulated", "Derived")
plot(comp)
x<-seq(0,1,0.1)
lines(x,x)

A plot of the simulated as against the derived results shows that, as we would hope, they are approximately equal (the line is x=y)

His principal concept is the distinction between exploit and drudgery.

The institution of a leisure class is the outgrowth of an early discrimination between employments, according to which some employments are worthy and others unworthy. Under this ancient distinction the worthy employments are those which may be classed as exploit; unworthy are those necessary employments into which no appreciable element of exploit enters.

He sees this division arising early in history as honour and status are assigned to those are successful in making others do what they want and “at the same time, employment in industry becomes correspondingly odious, and, in the common-sense apprehension, the handling of the tools and implements of industry falls beneath the dignity of able-bodied men. Labour becomes irksome.”

We’re no longer hairy barbarians using the vanquished for foot-stools but, as Veblen points out, the distinction persists no matter how illogical and unprofitable. Coding is still somehow viewed as necessarily inferior to the boardroom meeting even if it is the genius piece of code that makes or breaks a business.

Attitudes are changing, but still so slowly that there is a desperate shortage of people with both the experience and the hands-on skills to get things done.

So if you are a good analyst, and you love what you do, then this is my advice to you: when they come to lure you from your lovely pristine scripts, resist. Stay where you are. The 21st century is going to need you!

Ubuntu on AWS (50 mins)

Was going to buy a new laptop but it made more sense to set up a linux instance on AWS and remote in (a quarter of the price and more interesting). Here’s my recipe

1. As in Mark’s earlier post set yourself up with an AWS account and a key pair by following this tutorial
2. Launch an Ubuntu instance using the EC2 management console and select memory and processing power to suit.
3. Start up the instance then connect to it by using Mindterm (very useful alternative to SSHing in with putty). To do this just select the instance in the terminal. Select Actions and then Connect. (You’ll need to provide the path to your saved key)
4. Now you probably want to remote into your machine. Do this by setting up NoMachineNX following steps 2 to 4 in the following post

However when you execute the last line of step 2 you’ll find that nxsetup is not found. To fix this switch to this post and follow steps 6-7 (life’s so complicated)

5. Change password authentication to yes in  /etc/ssh/sshd_config
6. Add gnome fall back

sudo apt-get install gnome-session-fallback

1. Restart the instance and log in

Just remember to keep an eye on the charges!

Single Cluster Hadoop on Ubuntu (20 mins)

Of course you can run Hadoop directly on Amazon’s EMR platform but if you want to get more of a feel for how it works in a familiar environment you can set it up on a single instance.

1. Follow the instructions in this post substituting in the latest hadoop stable release
2. Install the latest JDK sudo apt-get install openjdk-7-jdk
3. Set the JAVA_HOME path variable export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-amd64 Substituting in the path to the JDK binaries
4. From the Hadoop quick start guide follow the instructions in the “Prepare to start the Hadoop Cluster” and “Stand Alone Operations” sections. If this all works you should be ready to go.

A Reingold–Tilford Tree of the IVOA Astronomical Object Ontology using D3

The images above and below are visualisations using D3 of a classification hierarchy for astronomical objects provided by the IVOA (International Virtual Observatory Alliance). I take no credit for the layout. The designs are taken straight from the D3 examples gallery but I will show you how I got the environment set up and my data into the graphs. The process should be replicable for any hierarchical dataset stored in a similar fashion.

A dendrogram of the same data

Even better than the static images are various interactive versions such as the rotating Reingold–Tilford Tree, the collapsible dendrogram and collapsible indented tree . These were all created fairly easily by substituting the astronomical object data for the data in the original examples. (I say fairly easily as you need to get the hierarchy into the right format but more on that later.)

First the environment.

If you are an R user with little experience of building web pages then you’ll probably find yourself squinting at the D3 documentation wondering how you get from the code to the output. With just a browser to read the JavaScript and an editor (notepad if you like but preferably a specialist HTML editor) you can get through the most of the tutorials referred to in Mark’s previous post . Doing this locally on your own pc works ok because the data visualised in the tutorials is mostly hard coded into the JavaScript. However once you want to start referring to data contained in external files some sensible security restrictions on what files your browser can access will block your attempts. The options are to turn these off (not advised) or switch to working on a webserver.

You can either opt for one of the many free hosting services or if you are feeling more adventurous you can follow Mark’s posts on setting up a Linux instance on Amazon Web Services and then follow Libby Hemphill’s instructions for starting up Apache and opening the port. Finally, going with the latter, I use FileZilla to transfer over any data I need to my Linux instance. See this post for getting it to work with your authentication key.

This should leave you in the following situation:

• You have a working web server and an IP address that will take you to an HTML index page from which you can provide links to your D3 documents.
• You have a way of transferring work that you do locally over to your web server.
• You have a whole gallery of examples to play around with

Rather than creating my own D3 scripts I’m just going too substitute my own data into the examples. The issue here is that the hierarchical examples take as an input a nested JSON object. This means something like:

{
 "name": "Astronomical Object",
 "children": [
  {
   "name": "Planet",
   "children": [
    {
     "name": "Type",
     "children": [
      {"name": "Terrestrial"},
      {"name": "Gas Giant"}
     ]
    }

The issue is that our data looks like this:

1. Planet
1.1. [Type]
1.1.1. Terrestrial
1.1.2. Gas Giant
1.2.  [Feature]
1.2.1. Surface
1.2.1.1. Mountain
1.2.1.2. Canyon
1.2.1.3. Volcanic
1.2.1.4. Impact
1.2.1.5. Erosion
1.2.1.6. Liquid
1.2.1.7. Ice
1.2.2. Atmosphere
1.2.2.1. Cloud
1.2.2.2. Storm
1.2.2.3. Belt
1.2.2.4. Aurora

To put this into the right format I’ve used Python to read in the file as csv (with dot as a delimiter) and construct the nested JSON object. Here’s the full python code:

import csv, re

#Open the file and read as dot delimited

def readLev(row):
    level=0
    pattern = re.compile("[a-zA-Z]")
    for col in row:
        if pattern.search(col) !=None:
            level=row.index(col)
    return level
    

with open('B:\PythonFiles\PythonInOut\AstroObject\Astro.csv', 'rb') as csvfile:
    astroReader = csv.reader(csvfile, delimiter='.')
    astroJson='{"name": "Astronomical Object"'
    prevLev=0
    for row in astroReader:
        #Identify the depth
        currentLev=readLev(row)
        if currentLev>prevLev:
            astroJson=astroJson+', "children": [{"name": "' + row[currentLev]+'"'
        elif currentLev<prevLev:
            jump=prevLev-currentLev
            astroJson=astroJson+', "size": 1}'+']}'*jump+', {"name": "' + row[currentLev]+'"'
        elif currentLev==prevLev:
            astroJson=astroJson+', "size": 1},{"name": "' + row[currentLev]+'"'
        prevLev=readLev(row)
    astroJson=astroJson+ '}]}]}'
    print astroJson

Since only the level of indentation matters to this process it could be repeated with on any data that has the form

* Planet
** [Type]
*** Terrestrial
*** Gas Giant

So that’s it. You’ll see that it in the source code for the hierarchical examples there is a reference to flare.json. Substitute in a reference to your own file containing the outputted JSON object and be sure to include that file in the directory on your web server.

Of course it’s a poor substitute for learning the language itself as that enables you to construct your own innovative visualisations but it gets you started.

Tonight, I thought I’d try and apply the theme and styling to some real word and slightly XKCD content: UK politics. Two of the biggest stories of the month in the UK have been Nick Clegg’s apology over reneging on the Liberal Democrat’s tuition fees pledge and Andrew Mitchell’s “incident” trying to cycle out of Downing Street, the so-called GateGate.

Using the newly redesigned Google Insights for Search, I looked at searches for clegg and pleb over the last 30 days. A quick manipulation into a csv and applying the XKCD theme and some geom_smoothing gives this:

Looks like Andrew Mitchell might be Nick Clegg’s new best friend in terms of deflecting some of the attention away from the sorry song…

And here’s the code (note that you need to have installed the Humor Sans font using install_fonts() ):

library(ggplot2)
library(extrafont)

### Already have read in fonts (see previous answer on how to do this)
loadfonts()

### Clegg and Pleb data
pleb.clegg <- read.csv("pleb and clegg.csv")
pleb.clegg$Date <- as.Date(pleb.clegg$Date, format="%d/%m/%Y")
pleb.clegg$xaxis <- -4

### XKCD theme
theme_xkcd <- theme(
	panel.background = element_rect(fill="white"),
	axis.ticks = element_line(colour=NA),
	panel.grid = element_line(colour="white"),
	axis.text.y = element_text(colour=NA),
	axis.text.x = element_text(colour="black"),
	text = element_text(size=16, family="Humor Sans")
	)

### Plot the chart
p <- ggplot(data=pleb.clegg, aes(x=Date, y=Pleb))+
geom_smooth(aes(y=Clegg), colour="gold", size=1, position="jitter", fill=NA)+
geom_smooth(colour="white", size=3, position="jitter", fill=NA)+
geom_smooth(colour="dark blue", size=1, position="jitter", fill=NA)+
geom_text(data=pleb.clegg[10, ], family="Humor Sans", aes(x=Date), colour="gold", y=20, label="Searches for clegg")+
geom_text(data=pleb.clegg[22, ], family="Humor Sans", aes(x=Date), colour="dark blue", y=4, label="Searches for pleb")+
geom_line(aes(y=xaxis), position = position_jitter(h = 0.1), colour="black")+
coord_cartesian(ylim=c(-5, 40))+
labs(x="", y="", title="Pleb vs Clegg: Google Keyword Volumes")+
theme_xkcd

ggsave("xkcd_cleggpleb.jpg", plot=p, width=8, height=5)

I recently stumbled across a series of blog posts from the folks at IDV that visualised the archive of recorded pirate attacks which has been collected by the US National Geospatial-Intelligence Agency. It’s a dataset of 6000+ pirate attacks which have been recorded over the last 30 or so years.

This first map shows where the attacks have been recorded, with four clear areas standing out when the data is aggregated into hexagon bins:

Map showing areas where pirate attacks have been recorded

Zooming in on the area around Yemen, there’s a clear ramp up in the number of attacks since 2008, which saw a 570% increase compared to the previous year.  As noted by the IDV analysis, most attacks take place on Wednesday and during the spring and autumn.

Number of attacks recorded in the Aden region by year

The reaction to the massive increase in attacks in 2008 seems to have been ships not travelling as close to the shore in 2009, leading to more attacks happening further out to sea. This can clearly be seen by looking only at the attacks in 2008 and 2009:

Number of Pirates attacks in the Aden area in 2008 and 2009 (Distance is in degrees)

Within the dataset, as well as information around the location of the attack, there are also descriptions of the attacks, which lends itself well to some text analysis to understand where there have been changes in the nature of the attacks above and beyond their distance.

Some analysis of the descriptions of the attacks reveals that the nature of the attacks also changed in 2010, with more featuring terms such as security and speedboats (full details of how these topic groups have been created is below). The analysis was used to identify five different types of attacks.

From the chart below, Topics 4 and 5came to prominence in 2008, with Topic 5 maintaining it’s share in 2010 before Topic 2 then increasing in number in 2011 and 2012. This is just scratching the surface with what can be done with topic analysis and given that all the documents are related to pirate attacks, there’s not the variation compared to what you would see in say, news articles about many subjects. There’s a good walk through of using the TopicModels package here.

Attacks in the Yemen region classified into one of five topics based on description

And what are these topics? The table below shows the top 10 terms for each of the 5 topics – they’re not as clear cut as you’d hope (mainly because there are a fair few verbs and numbers in there at the moment), but give an idea of some differences – skiffs versus speedboats, topic 2 featuring “security”, the numbers involved and months of the year all hit at different aspects of the attacks which have been picked out.

     Topic 1  Topic 2  Topic 3    Topic 4    Topic 5
1  attempted     were hijacked       boat      boats
2        six     fire      are      white       four
3       took security attacked       port        men
4  increased   skiffs    miles      about    persons
5      board     team  vessels      small      three
6     skiffs     when     this        sep        may
7     alarm,      jan  advised        apr speedboats
8        for    seven merchant    general       five
9   chemical      had  boarded reportedly    reports
10      guns    which exercise  speedboat       each

And the reason why pirates are called pirates? Because they Argghhhhhhhhhh.

NB. I haven’t had a chance to check on the copyright, etc. for hosting the pirate dataset, so please download it from here.

Reading the data into R.

library(maps)
library(sp)
library(maptools)
library(ggplot2)
library(spatstat)
gpclibPermit()
library(topicmodels)

pirates.data <- readShapePoints("C:\\ASAM 05 SEP 12")
pirates.data.2 <- as.data.frame(pirates.data)

How far to the shore?

Next is to turn the data into a Planar Point Pattern in order to allow for calculation of nearest coastal point for each attack. The same technique is then used to create a similar file for the coastline. Use the nncross function to find the distance from each attack to the nearest point on the coast (and identify which point that is).

bb <- c(40, 56, 7, 17)
pirates.ppp <- as.ppp(pirates.data.2[,13:14], bb)

worldmap <- map_data("world")
land.ppp <- as.ppp(worldmap[, 1:2], bb)
land.df <- as.data.frame(cbind(land.ppp$x, land.ppp$y))

reg <- as.data.frame(map("world", xlim = c(40, 56), ylim = c(7, 17), plot = FALSE)[c("x", "y")])

nearest.land <- nncross(pirates.ppp, land.ppp)
pirates.nearest.land <- as.data.frame(cbind(as.numeric(pirates.ppp$x), as.numeric(pirates.ppp$y), as.numeric(nearest.land$dist)))

pirates.data.aden <- merge(pirates.nearest.land, pirates.data.2, by.x=c("V1", "V2"), by.y = c("coords.x1", "coords.x2"))

Calculate various extra columns such as year of date and number of attacks by year

pirates.data.aden$year <- 1900 + as.POSIXlt(pirates.data.aden$DateOfOcc)$year
pirates.data.aden$month <- 1900 + as.POSIXlt(pirates.data.aden$DateOfOcc)$year
pirates.data.aden$year <- 1900 + as.POSIXlt(pirates.data.aden$DateOfOcc)$year

year.stats <- ddply(pirates.data.aden, .(year), summarise, attacks= length(year))
year.stats$Delt <- Delt(year.stats$attacks)

Plot the world map showing attacks binned into hexagons

ggplot()+
	stat_summary_hex(fun="length", data=pirates.data.2, aes(x=coords.x1, y=coords.x2, z=coords.x2)) +
	scale_fill_gradient(low="white", high="red", "Pirate Attacks recorded") +
	geom_path(aes(x=x, y=y), data = world) +
	mb.theme +
	labs(x="", y="") +
	theme(panel.background = element_rect(fill="white"),
		axis.ticks = element_line(colour="white"),
		axis.text = element_text(colour="white"),
		axis.line = element_line(colour="white"),
		panel.grid = element_line(colour=NA)) +
		scale_x_continuous(breaks=NA)+
		scale_y_continuous(breaks=NA)

Number of attacks by year near Aden

ggplot(data=pirates.data.aden, aes(x=year))+
		geom_histogram(binwidth=1, colour="white", fill="dark blue")+
		mb.theme +
		labs(x="Year", y="Number of attacks recorded")

Attacks by distance from Shore as a histogram

ggplot(data=subset(pirates.data.aden, year%in%c(2008, 2009)), aes(x=V3))+geom_histogram(fill="dark blue", colour="white")+
		mb.theme+
		facet_wrap(~year, ncol=1) +
		labs(x="Distance from shore", y="Number of attacks")

Topic Models analysis of attack descriptions near Aden

corpus <- Corpus(VectorSource(pirates.data.aden$Desc1))

dtm <- DocumentTermMatrix(corpus)

term_tfidf <- tapply(dtm$v/row_sums(dtm)[dtm$i], dtm$j, mean) * log2(nDocs(dtm)/col_sums(dtm > 0))
dtm <- dtm[, term_tfidf >= 0.1]
dtm <- dtm[row_sums(dtm) > 0,]

k=5
SEED=2012

TM <- list(VEM = LDA(dtm, k = k, control = list(seed = SEED)),
		VEM_fixed = LDA(dtm, k = k, control = list(estimate.alpha = FALSE, seed = SEED)),
		Gibbs = LDA(dtm, k = k, method = "Gibbs", control = list(seed = SEED, burnin = 1000, thin = 100, iter = 1000)),
		CTM = CTM(dtm, k = k, control = list(seed = SEED, var = list(tol = 10^-4), em = list(tol = 10^-3))))

pirates.data.aden$Topic <- topics(TM[["Gibbs"]], 1)

ggplot(data=pirates.data.aden, aes(x=year, fill=as.factor(Topic), group=as.factor(Topic)))+
		geom_histogram(binwidth=1, colour="white")+
		scale_fill_brewer(palette="Set3", "Topic Group") +
		mb.theme +
		labs(x="", y="")

One of the things I find great about R is the rapidly evolving ecosystem where new packages are being constantly created and others are being updated.

Up until now, I’ve found this to be a very good thing, but experienced the other side this week, where an upgrade to a package broke a pretty big script that I’d been working on.   

The “quick” solution in my case was to use the CRAN archive to download the source for earlier versions of the package (and it’s dependencies) that was causing issue and then build and install them to overwrite the upgrade – knowledge of how to build a package from source in Windows came in very handy and you can read more about how to do it in a previous D&L post. 

The longer term solution is a lot trickier, particularly where R is being used in a collaborative environment and reproducibility is important, either between machines or over time.

I suspect one solution is to have a common library that is centrally maintained and then changing the default location that R looks for installed packages (which is also a handy solution when dealing with not having to download all packages against once you’ve upgraded base R). 

On top of this, I suspect there would then need to be some type of test suite which, once updates to packages were available, checked in a development environment that existing scripts and processes still worked. None of this is new to software and IT folk, but it’s a novel issue at the moment for the analyst community I suspect.

The likes of 4Dpiecharts offer some good introductions to various aspects of the tool, but one thing which I struggled to find was a way of creating an icon based way of launching the GUI – i.e. click on an icon on the desktop and have the GUI launch in an application like way.

At first, I assumed that this could be done via a batch file, but found that this didn’t seem to work when the script should display a GUI as the output.

I then came across this very handy stackoverflow post. Although, slightly enigmatic, the method outlined by Greg Snow worked and gave me what I wanted.

1. Create a new folder at a convenient place on your drive.

2. Create a shortcut to RGui.exe or Rterm.exe – these will likely be located in a folder like C:\program files\R\R-version\bin\i386. If the shortcut is to RGui then the full R GUI will be brought up, whilst R term will bring up the console version (I went with this option so that the focus would stay with the GUI).

3. Create an R function called .First which includes the various functions, data, etc. to launch your code.

 .First <- function() {  insert code here } 

4. Ensure that your R session is clean and only contains the functions (particularly .First) and data that you want to launch when you double click the shortcut. Save this workspace (which will be called “.rData”) to the folder that you created in Step 1.

5. Right click on the Shortcut that you created in Step 2 and change the Start In property (under the “Shortcut” tab) to the name of the folder that you created in Step 1.

A close up of ancient and medieval philosophy ending at Descartes and Leibniz

This one came about because I was searching for a data set on horror films (don’t ask) and ended up with one describing the links between philosophers.

To cut a long story very short I’ve extracted the information in the influenced by section for every philosopher on Wikipedia and used it to construct a network which I’ve then visualised using gephi

It’s an easy process to repeat. It could be done for any area within Wikipedia where the information forms a network. I chose philosophy because firstly the influences section is very well maintained and secondly I know a little bit about it. At the bottom of this post I’ve described how I got there.

First I’ll show why I think it’s worked as a visualisation. Here’s the whole graph.

Each philosopher is a node in the network and the lines between them (or edges in the terminology of graph theory) represents lines of influence. The node and text are sized according to the number of connections (both in and out). The algorithm that visualises the graph also tends to put the better connected nodes in the centre of the diagram so we see the most influential philosophers, in large text, clustered in the centre. It all seems about right with the major figures in the western philosophical tradition taking the centre stage. (I need to also add the direction of influence with a arrow head – something I’ve not got round to yet.) A shortcoming however is that this evaluation only takes into account direct lines of influence. Indirect influence via another person in the network does not enter into it. This probably explains why Descartes is smaller than you’d think. It would also be better if the nodes were sized only by the number of outward connections although I think overall the differences would be slight. I’ll get round to that.

It gets more interesting when we use Gephi to identify communities (or modules) within the network. Roughly speaking it identifies groups of nodes which are more connected with each other than with nodes in other groups. Philosophy has many traditions and schools so a good test would be whether the algorithm picks them out.

It has been fairly successful. Below we can see the so called continental tradition picked out in green, stemming from Hegel and Nietzsche, leading into Heidegger and Sartre and ending in the isms of the twentieth century. It’s interesting that there is separate subgroup, in purple, influenced mainly by Schopenhauer (out of shot) and Freud.

The Continental Tradition

And this close up is of the analytical tradition emerging out of Frege, Russell and Wittgenstein. At the top and to the left you can see the British empirical school and the American pragmatists.

British Empiricism, American Pragmatism and the Analytical Tradition

It would be interesting to play with the number of groups picked out by the algorithm. It would hopefully identify sub groups within these overarching traditions.

The graph is probably most insightful when you zoom in close. Gephi produces a vector graphic output so if you’re interested you can download it here and explore it yourself.

Now for how you do it.

The first stop is dbpedia. This is a fantastic resource which stores structured information extracted from wikepdia in a database that accessible through the web. Among other things it stores all of the information you see in an infobox on a Wikipedia page. For example I was after the influenced and influenced by fields that you find on the infobox on the page for Plato.

The next step is to extract this information. For this we need two things: a SPARQL endpoint (try snorql), which is an online interface to submit our queries and little knowledge of SPARQL a specialist language for querying the semantic web. This is a big (and exciting) area that has to do with querying information that is structured in triples (subject-relationship-object). I assume it has its roots in predicate logic so the analytical philosophers would have been pleased. However the downside is that the language itself a lot more difficult to learn than say SQL and to complicate things still further you need to know the ontological structure of the resource you are querying. I probably wouldn’t have got anywhere at all were it not for a great blog post by Bob DuCharme which is a simple guide for getting the information out of wikipedia.

In the end the query I needed was very simple. You can test it by submitting it in snorql.

SELECT *
WHERE {
?p a
<http://dbpedia.org/ontology/Philosopher&gt; .
?p <http://dbpedia.org/ontology/influenced&gt; ?influenced.
}

It then needed a bit of cleaning as the punctuation was coded for URLS. For this I used the following online URL decoder.

After a bit more simple manipulation in excel I had a finished csv file that consisted of three columns

Philosopher A
Philosopher B
Weight


Each row in the data set represented a line of influence from philosopher A to philosopher B. The weight column contained a dummy entry of 1 because in our graph we do not want any one link to matter more than any other.

Gephi is the tool I used to create the visualisation of the graph. It’s both fantastic and open source. You can download it and set it up in minutes. For a quick tutorial see this link. There are many settings you can use to change the way your graph looks. I used a combination of the force atlas and the fruchterman-reingold layout algorithms. I then scaled text and node size by node degree (number of connections) and suppressed all nodes with less than four connections (it was overwhelming otherwise). The partition tool is used to create the communities. Full instructions are in the tutorial. I also found this blog entry very useful as a guide.

I hope that helps anyone who is trying to do something similar. If anyone does has a data set on horror films tagged with keywords please let me know!

If you liked this post and would like to see more like it then please subscribe by email (see the link in the side bar ) or sign up to our RSS feed.

Simon

Update Griff at Griff’s graphs has used the instructions above to create a fantastic visualization of the influence network of everyone on Wikipedia. It’s well worth a look.

Graphing the history of philosophy by Simon Raper is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Based on a work at drunksandlampposts.files.wordpress.com.
Permissions beyond the scope of this license may be available at http://drunks-and-lampposts.com/.

D3

For anyone coming from a ggplot2 background, there are a lot of similarities with certain parts of D3 and the attributes that can be given to objects. There is a fair amount more code to write, but there are a lot of great tutorials out there which give very good introductions to creating good looking visualisations. There’s a bit of a learning curve to using it, but there are lots of tutorials out there which make it easy to get started, for example, here, here and here.

The general idea behind the language is to encode objects, for example, a circle, with various attributes like colour, size and  location. These could either be hard coded, based on a function (e.g. the sine function) or based on a dataset. The additional nice feature of the software is the built in ability to transition between states (e.g. moving a circle from location A to location B) which would normally be difficult to do, and probably appear quite jerky.

For anyone coming from a Tableau background, D3 provides a lot more flexibility in what can be achieved, but is a very code based solution and not a BI solution.

From my experience so far, a specialist HTML editor like Astana Studio is a good place to start for a few reasons, including colour coding for the HTML and also having a built in local web server so that you can quickly preview the results of your code writing.

As an aside, you’ll need a web server of some sort for any D3 code to work (It took me a frustrating hour to figure this out). The alternative to using the built in Astana server is something like using the python simple http server, but I find this a lot more cumbersome.

Oh, and sorry for not having any examples built into the post – WordPress.com doesn’t like having custom JavaScript in posts.