Java执行hadoop的基本操作实例代码
Java执行hadoop的基本操作实例代码
在观山湖等地区,都构建了全面的区域性战略布局,加强发展的系统性、市场前瞻性、产品创新能力,以专注、极致的服务理念,为客户提供做网站、成都网站建设 网站设计制作按需策划,公司网站建设,企业网站建设,品牌网站设计,营销型网站,外贸营销网站建设,观山湖网站建设费用合理。
向HDFS上传本地文件
public static void uploadInputFile(String localFile) throws IOException{ Configuration conf = new Configuration(); String hdfsPath = "hdfs://localhost:9000/"; String hdfsInput = "hdfs://localhost:9000/user/hadoop/input"; FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf); fs.copyFromLocalFile(new Path(localFile), new Path(hdfsInput)); fs.close(); System.out.println("已经上传文件到input文件夹啦"); }
将output文件下载到本地
public static void getOutput(String outputfile) throws IOException{ String remoteFile = "hdfs://localhost:9000/user/hadoop/output/part-r-00000"; Path path = new Path(remoteFile); Configuration conf = new Configuration(); String hdfsPath = "hdfs://localhost:9000/"; FileSystem fs = FileSystem.get(URI.create(hdfsPath),conf); fs.copyToLocalFile(path, new Path(outputfile)); System.out.println("已经将输出文件保留到本地文件"); fs.close(); }
删除hdfs中的文件
public static void deleteOutput() throws IOException{ Configuration conf = new Configuration(); String hdfsOutput = "hdfs://localhost:9000/user/hadoop/output"; String hdfsPath = "hdfs://localhost:9000/"; Path path = new Path(hdfsOutput); FileSystem fs = FileSystem.get(URI.create(hdfsPath), conf); fs.deleteOnExit(path); fs.close(); System.out.println("output文件已经删除"); }
执行mapReduce程序
创建Mapper类和Reducer类
public static class TokenizerMapper extends Mapper
执行mapReduce程序
public static void runMapReduce(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if(otherArgs.length != 2){ System.err.println("Usage: wordcount"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.out.println("mapReduce 执行完毕!"); System.exit(job.waitForCompletion(true)?0:1); }
感谢阅读,希望能帮助到大家,谢谢大家对本站的支持!
本文题目:Java执行hadoop的基本操作实例代码
本文链接:http://myzitong.com/article/jhidgg.html