Skip to content

pig script to convert snappy into gzip

by on March 3, 2015

My first ever pig script:

set output.compression.enabled true;
set output.compression.codec org.apache.hadoop.io.compress.GzipCodec;
A = load '/path/to/snappy/dir/part*' using PigStorage();
store A into '/path/to/gzip/dir' USING PigStorage();

It’s that simple :)

I don’t know why we chose snappy for compression as we’ve found out that many third party could not reach out and read this type of data.
The compression is also nto as strong.
e.g. from pig output:

Successfully read 6041224101 records (205720552522 bytes) from: "/path/to/snappy/dir/part*"
Successfully stored 6041224101 records (117690493503 bytes) in: "/path/to/gzip/dir"

and from hdfs du -s -h command:

> hdfs dfs -du -s -h "/path/to/snappy/dir/"
191.6 G  574.8 G  /path/to/snappy/dir/
> hdfs dfs -du -s -h "/path/to/gzip/dir"
109.6 G  328.8 G  /path/to/gzip/dir
Advertisements
Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: