Data science -- the cool scientist without white gowns

"Scientist" is a cool word during my school days, wanna become one but donno on what. All I see as scientists wore white gown with colorful liquids around. But later during college and working days scientists seemed to be boring people with no personal life, accumulated in educational institutes with college kids helping around.

Recently there is a profile called "Data scientist" all over the BI market and started appearing in every article where Hadoop / Big data is there. It must be some part what a BI/DW person is doing with some specialization. Yes it is the formula,as for me..

BI + big-data + statistics + scripting + visualization = Data scientist

Ok can be scientist and work for a corporate or invent something new for you name? possibly...

But seems like a lot to cover , learn and experience at work. Maybe not really if we are in right job. Im just listing the very higher level outline..(and it is not limited to..) and my intention is not to oversimplify, but certainly to simplify the puzzle..

BI

- DW work like ETL , databases, SQL with exposure to enterprise setup. Collecting data from heterogenous data sources. Log analysis. Dimensional modelling, DW architecture. DB performance.

Big-data

Hadoop is the first thing comes to mind for Big-data.. but good to know about noSQL dbs.

Mapreduce - shared nothing architecture - need for MR - use cases - tools available - pros and cons

Statistics

Basics - application of statistics in real-world - R programming

Scripting

Perl, Python, Java

Visualization

Reporting (I like Tableau), Complex SQLs , Ability to tell a story with data - by whatever way you effectively deliver..

I would like to list some of the coolest learning materials available for above topics..

I believe the thirst for discovery, admiring the hidden secret in the boring pile of data would make a Data Scientist..

Dual table in Hive

Since there is no Dual table in Hive..

I have created a dummy dual with dummy value ‘X’.

hive> CREATE TABLE dual (dummy STRING);

hive> load data local inpath '/local/user/dw/hive/dual.txt' overwrite into table dual;

Now we can use this table to select string values. Like

hive> select 'name','place','age' from dual;

to get current timestamp

hive> select unix_timestamp() from dual;

Talent is overrated - Geoff Colvin

Simply to say this book is good and can be life changing for you or may be to your kid.

I will get some new idea everyday, while taking bath or watching some movie or during half-sleep @ office or even at workout. Some of them are great, turned out to be great things which are implemented by someone, some are already existing and I'm ignorant of it. Many self analyzing thought made what I'm now. But the point is I'm not great, maybe by the scale of this society. Then how to define greatness - a person achieved enviable status in a field. The word field is the key here.

Being a Software Engineer I cannot become maddeningly wealthy like Buffet just by reading 'the intelligent investor'. Im already a hard-wired employee in a field who's time is controlled by someone else. And if you are married with kids personal time is a joke. There are success stories everywhere in Tennis, basketball, investing and especially entrepreneurship. Just by looking at it or researching will not make us the same. You might have known this but yet human greed takeover sometimes and we will start day-dreaming. Mostly the end result is disappointment.

If I have to become a cricketer I should have that thought by at least age 10 and played relentlessly until my skin is totally tanned. At this point this book makes a clear the 10000 hours point which Malcolm Gladwell's 'Outliers' is also talking about. Reading "Talent is overrated" book I can recall Arnold Schwarzenegger's quote ..

"Number one, come to America. Number two, work your butt off. And number three, marry a Kennedy."

The book says these very clearly

1. Choose your field, if you are already in a field and earning from it, its hard to leave and try new. But children has the advantage to excel enormously.

2. Deliberate practice .. it must be conscious, measurable and improving

3. Dont stick in the OK plateau if you want to become world-class

4. Never look around to excel in other area which you cannot afford to spend time on.

5. Get family support for what you are doing

Its a revealing read and can be read again if you fail to succeed in some area. The world is so competitive that only the extraordinary can win. (even in a job interview ) Can be a great book if you are a parent which can change your kids life all together.

Shell script - snippets

Loop through dates

startdate=`/bin/date --date="2007-07-01" +%Y-%m-%d`

enddate=`/bin/date --date="2011-07-01" +%Y-%m-%d`

foldate="$startdate"

until [ "$foldate" == "$enddate" ]

do

echo $foldate

foldate=`/bin/date --date="$foldate 1 month" +%Y-%m-%d`

done

Slowly changing dimensions in Hive

If you keep you Datawarehouse in Hive,How to implement slowly changing dimensions?

http://en.wikipedia.org/wiki/Slowly_changing_dimension

This is not a solution blog, actually Im looking for an effective solution to implement SCD in Hive.

Since all the types uses Update one way or another. for eg Type 2 can be a good candidate but it requires to update the enddate of history record. The only option I can think off is to insert the rows into the dimension and query on the maxid or maxdate to get the latest dimensions on the analytics layer.. (seems like an overhead on mapreduce)

I posted this question to Cloudera professionals group at LinkedIn. And Jasper recommended an idea which is not using Hive queries, moreover he prefers mapreduce scripts to handle SCD situations. I modified the flow to suit Hive tables.

Idea is this

1. Open the underlying HDFS file or select all rows using Hive-ql

2. Pass the data through a mapper as key,value pair

3. End-date the old record and leave the new record with NULL end_date (for type2) through script

4. Overwrite the Hive table

But the process could be tricky if the data is present in different partitions.

tipping point - Book journal

Malcolm Gladwell is an expert in writing book about the things we know already (very vaguly) and make an compelling book with storytelling and statistics.. he must be the perfect author for the current information overloaded culture.

Tipping point said to be his best, but for me listening in audio format felt 'outliers' was interesting and stories remained in memory for relatively more time.

the law of few - it is true that few people make a big difference and also some critical piece of work make a big difference in a big project. Here the point is not about innovation but marketing. I feel we play all these roles sometimes in different context. Gladwell categorizes these people, so it makes it easier to find who is who is future...

Stickiness factor - we are living in a era of short memory loss.. stickiness could be a day to a week. In the case of viral videos it could be an hour of fame. There is a war out there in advertising, web and entertainment to make a thing sticky. the point to make your stuff sticky specifically to your target audience. Sometimes the stickiness can come from a funny context or intelligence or experience.

new york crime example - this is a fascinating story of how new york subway handled crime by simply clearing the cars from graffiti everyday. You can make a big difference by making small spirited changes. Like if you allow small mistakes in a project to go by and didnt appreciate the people taking care of those stuff..then the manager is setting a wrong example. Its all about defined values. And always values are simple and to the core.

Connectivity - You are connected to anyone in this world in a factor of six. this may be reduced after facebook and linkedin. And 150 is the number where a group or organization can work effectively. I see the reason why my company split organizations without any reason and give meaningless names to those groups.

Teenage smoking issue - simple in one line "experimenting may not become a habit". We can protect our kids only to a certain extent from smoking. But anyway they are going to try at least once, but that doesn't means that they are hooked forever. More than 80% of kids who tried smoking quit is along the way. Chippers is the name for occasional and non-addictive smokers.

Prelude - Identifying mavens.. identifying the right context.. identifying minor signs before they tip..

How small things can bring big impact.. small things mostly are in our control..

Hive outer join issue

Recently we faced a memory issue (java heap space) when joining two partitioned tables.

Both the tables are huge in millions of rows and each partitioned on hourly basis.

select a.*,b.*

from a left outer join b

on a.id = b.id

where

a.part > '2012-01-01'

and b.part > '2012-01-01';

this fails during peak times.

issue:

when we checked the logs we found that the mapper is trying to access all the partitions of table b. Since the system is scarce in resources the job failed due to heap space issue.

resolution:

Change the right table to a inline..

select a.*,b.*

from a left outer join (select * from b where b.part > '2012-01-01') c

on a.id = c.id

where

a.part > '2012-01-01';

the number of mappers reduced dramatically and execution is scanning only the required partitions from table b.

Learning:

Always check the number of mappers while running. If the numbers are not realistic take a peak.

Divide and conquer approach when writing complex hive queries.

Since Hive is not yet a completely matured SQL, we need to put extra effort to examine the execution plan.