Welcome to Brickhouse Confessions

Hey, Folks, welcome to the new blog “Brickhouse Confessions”.  The purpose is mainly to provide guidance using the new Brickhouse library of Hive extensions, but it will also be a forum to discuss big data and new technologies.

Advertisements
This entry was posted in Uncategorized. Bookmark the permalink.

2 Responses to Welcome to Brickhouse Confessions

  1. RAVI PARAMASIVAM says:

    Hi,
    Do you have the below UDF’s converted into Spark SQL UDF’S(BY Implementing UDF1,UDF2,…)

    https://github.com/klout/brickhouse

    I know we can access hive udf functions from Spark SQL. However we need these functions re-written in default Spark SQL native API’s

    • jeromebanks says:

      Ravi,
      You can access the UDF’s from Spark, simply by running hiveContext.sql(” CREATE TEMPORARY FUNCTION collect AS ….”);
      There were issues with earlier versions of Spark (before 1.5) accessing UDF’s, but now it should work fine.

      I don’t believe there is any intrinsic need to port to Spark Native API’s (the Java Hive versions work fine; there aren’t any major performance problems.) I have been porting some of Brickhouse to Spark and Scala, (mostly around XUnits and Sketch sets), but this is more to support Scala’s functional paradigm. Unfortunately, this is not currently open-sourced

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s