Tuesday, May 10, 2016

Apache BigTop

Apparently, Apache has the BigTop project going, the idea of which is to have a single repository for linux distributions to pick up hadoop from.


I first came across this tutorial on how to install hadoop using BigTop:


 There are a couple of gotchas:
(1) you want a newer version of BigTop, considering, as of now 1.1.0 is released
(2) For Ubuntu, `lsb_release --codename --short` gives "wily" (on 15.10) whereas the specified BigTop repo only has "trusty" - so I had to replace that with trusty

This seemed to work. However, this is tying you to BigTop version 1.1.0

Thereafter, I found this blog post:
That post was written when big top was still incubating. So the url has changed. Also that post refers to an "ubuntu" folder within the repo for bigtop.list. That folder doesn't exist, and so based on my above experience, I used the "trusty" folder.

So the steps are:
wget -O- http://www.apache.org/dist/bigtop/stable/repos/GPG-KEY-bigtop | sudo apt-key add -
sudo wget -O /etc/apt/sources.list.d/bigtop.list http://www.apache.org/dist/bigtop/stable/repos/trusty/bigtop.list

At this point, edit the /etc/apt/sources.list.d/bigtop.list file and verify that there is only one deb and one (or zero) deb-src line each. If there are multiple, only one should be uncommented (comments prefixed with #)

Now you can apt update and be on your merry way to use say synaptic to install whatever you want. Follow the apache wiki link above to get hadoop up and running.

PS: this doesn't work for a raspberry pi, or at least not currently, since the above bigtop.list specifies an amd_64 architecture alone. So at some point, I'll write some blog posts on installing the hadoop ecosystem on the Raspberry Pi, especially since I now have a BitScope BP10A rack (see: http://my.bitscope.com/store/?p=list&a=list&i=cat+3 ) on order, which I posted about in my previous blog post.


No comments: