12/30/2014

Hadoop JAVA Map Reduce Sort by Value


Input  (test.txt)
------------
1,50
2,20
3,30
4,10
5,15
6,25
7,55
8,35
9,70

output
----------------------------
9 70
7 55
1 50
8 35
3 30
6 25
2 20
5 15

4 10



package com.my.cert.example;

import java.nio.ByteBuffer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.IntWritable.Comparator;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparator;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class ValueSortExp {
 public static void main(String[] args) throws Exception {

  Path inputPath = new Path("C:\\hadoop\\test\\test.txt");
  Path outputDir = new Path("C:\\hadoop\\test\\test1");

  // Path inputPath = new Path(args[0]);
  // Path outputDir = new Path(args[1]);

  // Create configuration
  Configuration conf = new Configuration(true);

  // Create job
  Job job = new Job(conf, "Test HIVE commond");
  job.setJarByClass(ValueSortExp.class);

  // Setup MapReduce
  job.setMapperClass(ValueSortExp.MapTask.class);
  job.setReducerClass(ValueSortExp.ReduceTask.class);
  job.setNumReduceTasks(1);

  // Specify key / value
  job.setMapOutputKeyClass(IntWritable.class);
  job.setMapOutputValueClass(IntWritable.class);
  job.setOutputKeyClass(IntWritable.class);
  job.setOutputValueClass(IntWritable.class);
  job.setSortComparatorClass(IntComparator.class);
  // Input
  FileInputFormat.addInputPath(job, inputPath);
  job.setInputFormatClass(TextInputFormat.class);

  // Output
  FileOutputFormat.setOutputPath(job, outputDir);
  job.setOutputFormatClass(TextOutputFormat.class);

  /*
   * // Delete output if exists FileSystem hdfs = FileSystem.get(conf); if
   * (hdfs.exists(outputDir)) hdfs.delete(outputDir, true);
   * 
   * // Execute job int code = job.waitForCompletion(true) ? 0 : 1;
   * System.exit(code);
   */

  // Execute job
  int code = job.waitForCompletion(true) ? 0 : 1;
  System.exit(code);

 }
 
 public static class IntComparator extends WritableComparator {

     public IntComparator() {
         super(IntWritable.class);
     }

     @Override
     public int compare(byte[] b1, int s1, int l1,
             byte[] b2, int s2, int l2) {

         Integer v1 = ByteBuffer.wrap(b1, s1, l1).getInt();
         Integer v2 = ByteBuffer.wrap(b2, s2, l2).getInt();

         return v1.compareTo(v2) * (-1);
     }
 }

 public static class MapTask extends
   Mapper<LongWritable, Text, IntWritable, IntWritable> {
  public void map(LongWritable key, Text value, Context context)
    throws java.io.IOException, InterruptedException {
   String line = value.toString();
   String[] tokens = line.split(","); // This is the delimiter between
   int keypart = Integer.parseInt(tokens[0]);
   int valuePart = Integer.parseInt(tokens[1]);
   context.write(new IntWritable(valuePart), new IntWritable(keypart));

  }
 }

 public static class ReduceTask extends
   Reducer<IntWritable, IntWritable, IntWritable, IntWritable> {
  public void reduce(IntWritable key, Iterable<IntWritable> list, Context context)
    throws java.io.IOException, InterruptedException {
   
   for (IntWritable value : list) {
    
    context.write(value,key);
    
   }
   
  }
 }

}

21 comments:

  1. Thank you so much for sharing this worthwhile to spent time on. You are running a really awesome blog. Keep up this good work

    Hadoop training chennai velachery
    Hadoop training velachery
    Hadoop training institute in t nagar

    ReplyDelete
  2. I have finally found a Worth able content to read. The way you have presented information here is quite impressive. I have bookmarked this page for future use. Thanks for sharing content like this once again. Keep sharing content like this.

    Software testing training in chennai | Software testing training | Software testing institute in chennai

    ReplyDelete
  3. Face To Face java SE Java EE Training Courses Materials. Java Training in Chennai Java SE & Java EE Certification Courses Training Program. Online Java Training Online Java Training India Java Training Institutes in Chennai Online Java Training India Java J2EE Training Institutes in Chennai | Online Java Training

    ReplyDelete
  4. Java Training Java Training | Java Course in Chennai Java Course in Chennai | Java Training Chennai Java Training Chennai | Online Java Training India Online Java Training

    ReplyDelete
  5. This works fine as the input K, V are unique values, this gives problems with duplicate keys while grouping. Correct me if I am wrong. Thanks

    ReplyDelete
  6. Thanks for providing this informative information you may also refer.
    http://www.s4techno.com/blog/2016/08/13/storm-components/

    ReplyDelete
  7. This comment has been removed by the author.

    ReplyDelete
  8. It is amazing and wonderful to visit your site.Thanks for sharing this information,this is useful to me...
    Android Training in Chennai
    Ios Training in Chennai

    ReplyDelete
  9. Thanks for such a great article here. I was searching for something like this for quite a long time and at last I’ve found it on your blog. It was definitely interesting for me to read  about their market situation nowadays.
    Hadoop Training in Chennai
    Hadoop Training in Bangalore
    Big data training in tambaram
    Big data training in Sholinganallur
    Big data training in annanagar

    ReplyDelete
  10. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Devops Training in Chennai

    Devops Training in Bangalore

    Devops Training in pune

    Devops training in tambaram
    Devops training in velachery

    ReplyDelete
  11. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.

    rpa training in Chennai | rpa training in pune

    rpa training in tambaram | rpa training in sholinganallur

    rpa training in Chennai | rpa training in velachery

    rpa online training | rpa training in bangalore

    ReplyDelete
  12. This is quite educational arrange. It has famous breeding about what I rarity to vouch. Colossal proverb. This trumpet is a famous tone to nab to troths. Congratulations on a career well achieved. This arrange is synchronous s informative impolites festivity to pity. I appreciated what you ok extremely here 
    python training in chennai
    python training in chennai
    python training in Bangalore

    ReplyDelete
  13. Thanks for splitting your comprehension with us. It’s really useful to me & I hope it helps the people who in need of this vital information. 

    java training in omr | oracle training in chennai

    java training in annanagar | java training in chennai

    ReplyDelete
  14. It seems you are so busy in last month. The detail you shared about your work and it is really impressive that's why i am waiting for your post because i get the new ideas over here and you really write so well.
    Data Science training in Chennai
    Data science training in bangalore
    Data science training in pune
    Data science online training

    ReplyDelete
  15. Greetings from Florida! I’m bored at work, so I decided to browse your site on my iPhone during lunch break.
    fire and safety course in chennai

    ReplyDelete
  16. The knowledge of technology you have been sharing thorough this post is very much helpful to develop new idea. here by i also want to share this.

    Microstrategy Training in Chennai
    Microstrategy Training

    ReplyDelete
  17. I ‘d mention that most of us visitors are endowed to exist in a fabulous place with very many wonderful individuals with very helpful things.
    iosh course in chennai

    ReplyDelete