Big Data Cookbook

How-to: Configure Eclipse for Hadoop Contributions

Mon, 20 May 2013 07:20:41 -0400

How-to: Configure Eclipse for Hadoop Contributions:

Useful how-to on setting up Hadoop in Eclipse to enable dev and contribution.

—Jason

Really useful EC2 Pricing Breakdown

Thu, 16 May 2013 07:02:17 -0400

The guys at Infochimps published this pricing breakdown for the various EC2 instances.

Seems like it will come in handy one day…

(via @rjurney)

—Jason

Machine Learning over Storm

Mon, 13 May 2013 06:38:38 -0400

trident-ml

Trident-ML is a realtime online machine learning library. It allows you to build real time predictive features using scalable online algorihms. This library is built on top of Storm, a distributed stream processing framework which runs on a cluster of machines and supports horizontal scaling. The packaged algorithms are designed to fit into limited memory and processing time but they don’t work in a distributed way.

storm-r

Storm-R uses the multilang protocol to integrate R function calls with a trident Function.

storm-pattern

Based on the cascading.pattern project. The pattern sub-project for http://Cascading.org/ which uses flows as containers for machine learning models, importing PMML model descriptions from R, SAS, Weka, RapidMiner, KNIME, SQL Server, etc.

—Jason

Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node

Wed, 20 Mar 2013 13:15:45 -0400

Running a Multi-Broker Apache Kafka 0.8 Cluster on a Single Node:

This is an excellent howto by Michael Nole on setting up a multi broker Kafka cluster.

—Jason

Parquet columnar storage format for Hadoop released

Tue, 12 Mar 2013 15:31:49 -0400

Parquet is a columnar storage format for Hadoop.

Format and MapReduce code on GH. Includes a Loader and Storer for pig.

Released under the Apache 2.0 License.

Spatial Hadoop: MapReduce framework for spatial data

Tue, 12 Mar 2013 14:45:09 -0400

Spatial Hadoop is a MapReduce framework designed specifically to handle huge datasets of spatial data. SpatialHadoop is shipped with built-in spatial high level language, spatial data types, spatial indexes and efficient spatial operations.

Code on GH.

—Jason

NetflixGraph: Compact in-memory representation of directed graph data

Sun, 10 Mar 2013 14:34:40 -0400

Great write-up and release of an open source (Apache 2) project from Netflix. Code on GH.

—Jason

Fnordmetric: Nice metrics visualization

Sun, 10 Mar 2013 14:19:52 -0400

I found this recently and it looks pretty nice. Check out the screenshots.

Code on GH.

—Jason

Write-up on Clustering Graphite

Sun, 10 Mar 2013 14:04:20 -0400

Short write-up on clustering Graphite.

—Jason

[Video] ElasticSearch: The Missing Intro

Mon, 04 Mar 2013 06:37:54 -0500

[Video] ElasticSearch: The Missing Intro:

A nice intro video on ElasticSearch from Air Mozilla. It is ~56min.

—Jason

Spark and Shark Tutorial from Stratconf 2013

Sun, 03 Mar 2013 15:21:50 -0500

Spark and Shark tutorial/course given at Strata, materials online.

—Jason

Graph Based Recommendations using "How-To" Guides Dataset

Sun, 03 Mar 2013 13:37:51 -0500

This is a great post using Python, Neo4j, and Bulbflow to build a recommendation system using a graph database. It looks like they crawled SnapGuide to get their data for this.

The code for bulbflow is on GH.

Here is another post on Graph Recommendation Systems using Gremlin.

—Jason

Using SOLR Cloud as a NoSQL database for low latency analytics

Sat, 02 Mar 2013 15:56:39 -0500

Article on using SolrCloud for low latency analytics. Example configs on GH.

—Jason

Agile Analytics Applications [Presentation]

Mon, 18 Feb 2013 07:50:00 -0500

This is an awesome presentation by Russel Jurney on building data driven applications that use big data.

Russel is in the midst of writing a book on this topic and the book is currently available for review on O’reilly’s Open Feedback Publishing System

—Jason

Computing Rollups on Impression Logs using Storm

Mon, 18 Feb 2013 07:39:49 -0500

Great blog post at Chimpler on using Storm to compute rollups on impression logs and storing the results in MongoDB. Code in GH

—Jason

Habakkuk: Finding Religious Tweets using Storm

Mon, 18 Feb 2013 06:39:26 -0500

Great blog post on mining the Twitter stream for religious tweets using Storm. Code on GH.

—Jason

Storing Time Series Metrics With Cassandra and Composite Columns

Sat, 16 Feb 2013 14:04:01 -0500

Great (short) presentation on “Storing Time Series Metrics With Cassandra and Composite Columns” by Joe Stein from Medialets. With code on GH.

—Jason

Polyglot Persistence and Query with Gremlin

Wed, 13 Feb 2013 12:21:39 -0500

Polyglot Persistence and Query with Gremlin:

Great article on using Gremlin to query graph data in various datastores.

—Jason

A Tutorial on Writing Hive UDFs

Wed, 13 Feb 2013 12:01:41 -0500

A Tutorial on Writing Hive UDFs:

Nice tutorial on writing UDFs for Hive.

—Jason

Flatten entire HBase column families with Pig and Python UDFs

Wed, 13 Feb 2013 11:58:55 -0500

Flatten entire HBase column families with Pig and Python UDFs:

Some nice Pig and HBase hackery.

—Jason