本篇文章主要介绍了" 如何利用MapReduce的分治策略提高KNN算法的运行速度"，主要涉及到方面的内容，对于软件工程感兴趣的同学可以参考一下：
集群环境介绍：hadoop2.4.1 64位
6台服务器：
hadoop11 NameNode 、SecondaryNameNode
hadoop22 ...
集群环境介绍：
hadoop2.4.1 64位
6台服务器：
hadoop11 NameNode 、SecondaryNameNode
hadoop22 ResourceManager
hadoop33 DataNode、NodeManager
hadoop44 DataNode、NodeManager
hadoop55 DataNode、NodeManager
hadoop66 DataNode、NodeManager
实验1：训练集train.txt样例个数为245057（3.24M） 测试集test.txt样例个数为51444（640kb），并将全部测试集都存放在test.txt中
[root@hadoop11local]# hadoop fs -lsr /dir6/
-rw-r--r-- 3 root supergroup 34008162016-07-1719:28 /dir6/test.txt
注意：此时所有的测试集都在一个文本中(test.txt)存放，作为输入路径
ＫＮＮ算法运行日志：
16/07/1719:32:24 INFO client.RMProxy: Connecting to ResourceManager at hadoop22/10.187.84.51:803216/07/1719:32:25 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.16/07/1719:32:25 INFO input.FileInputFormat: Total input paths to process : 116/07/1719:32:25 INFO mapreduce.JobSubmitter: number of splits:116/07/1719:32:26 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1468752229715_0016
16/07/1719:32:26 INFO impl.YarnClientImpl: Submitted application application_1468752229715_0016
16/07/1719:32:26 INFO mapreduce.Job: The url to track the job: http://hadoop22:8088/proxy/application_1468752229715_0016/16/07/1719:32:26 INFO mapreduce.Job: Running job: job_1468752229715_0016
16/07/1719:32:32 INFO mapreduce.Job: Job job_1468752229715_0016 running in uber mode : false16/07/1719:32:32 INFO mapreduce.Job: map0% reduce 0%16/07/1719:32:49 INFO mapreduce.Job: map1% reduce 0%16/07/1719:33:05 INFO mapreduce.Job: map2% reduce 0%16/07/1719:33:20 INFO mapreduce.Job: map3% reduce 0%16/07/1719:33:35 INFO mapreduce.Job: map4% reduce 0%16/07/1719:33:50 INFO mapreduce.Job: map5% reduce 0%16/07/1719:34:02 INFO mapreduce.Job: map6% reduce 0%16/07/1719:34:17 INFO mapreduce.Job: map7% reduce 0%16/07/1719:34:32 INFO mapreduce.Job: map8% reduce 0%16/07/1719:34:47 INFO mapreduce.Job: map9% reduce 0%16/07/1719:35:02 INFO mapreduce.Job: map10% reduce 0%16/07/1719:35:14 INFO mapreduce.Job: map11% reduce 0%16/07/1719:35:29 INFO mapreduce.Job: map12% reduce 0%16/07/1719:35:44 INFO mapreduce.Job: map13% reduce 0%16/07/1719:35:59 INFO mapreduce.Job: map14% reduce 0%16/07/1719:36:12 INFO mapreduce.Job: map15% reduce 0%16/07/1719:36:27 INFO mapreduce.Job: map16% reduce 0%16/07/1719:36:42 INFO mapreduce.Job: map17% reduce 0%16/07/1719:36:57 INFO mapreduce.Job: map18% reduce 0%16/07/1719:37:12 INFO mapreduce.Job: map19% reduce 0%16/07/1719:37:27 INFO mapreduce.Job: map20% reduce 0%16/07/1719:37:39 INFO mapreduce.Job: map21% reduce 0%16/07/1719:37:54 INFO mapreduce.Job: map22% reduce 0%16/07/1719:38:09 INFO mapreduce.Job: map23% reduce 0%16/07/1719:38:24 INFO mapreduce.Job: map24% reduce 0%16/07/1719:38:39 INFO mapreduce.Job: map25% reduce 0%16/07/1719:38:51 INFO mapreduce.Job: map26% reduce 0%16/07/1719:39:06 INFO mapreduce.Job: map27% reduce 0%16/07/1719:39:22 INFO mapreduce.Job: map28% reduce 0%16/07/1719:39:37 INFO mapreduce.Job: map29% reduce 0%16/07/1719:39:52 INFO mapreduce.Job: map30% reduce 0%16/07/1719:40:07 INFO mapreduce.Job: map31% reduce 0%16/07/1719:40:22 INFO mapreduce.Job: map32% reduce 0%16/07/1719:40:37 INFO mapreduce.Job: map33% reduce 0%16/07/1719:40:52 INFO mapreduce.Job: map34% reduce 0%16/07/1719:41:04 INFO mapreduce.Job: map35% reduce 0%16/07/1719:41:22 INFO mapreduce.Job: map36% reduce 0%16/07/1719:41:37 INFO mapreduce.Job: map37% reduce 0%16/07/1719:41:52 INFO mapreduce.Job: map38% reduce 0%16/07/1719:42:07 INFO mapreduce.Job: map39% reduce 0%16/07/1719:42:22 INFO mapreduce.Job: map40% reduce 0%16/07/1719:42:37 INFO mapreduce.Job: map41% reduce 0%16/07/1719:42:53 INFO mapreduce.Job: map42% reduce 0%16/07/1719:43:08 INFO mapreduce.Job: map43% reduce 0%16/07/1719:43:23 INFO mapreduce.Job: map44% reduce 0%16/07/1719:43:41 INFO mapreduce.Job: map45% reduce 0%16/07/1719:43:56 INFO mapreduce.Job: map46% reduce 0%16/07/1719:44:12 INFO mapreduce.Job: map47% reduce 0%16/07/1719:44:30 INFO mapreduce.Job: map48% reduce 0%16/07/1719:44:45 INFO mapreduce.Job: map49% reduce 0%16/07/1719:45:00 INFO mapreduce.Job: map50% reduce 0%16/07/1719:45:15 INFO mapreduce.Job: map51% reduce 0%16/07/1719:45:30 INFO mapreduce.Job: map52% reduce 0%16/07/1719:45:48 INFO mapreduce.Job: map53% reduce 0%16/07/1719:46:03 INFO mapreduce.Job: map54% reduce 0%16/07/1719:46:18 INFO mapreduce.Job: map55% reduce 0%16/07/1719:46:33 INFO mapreduce.Job: map56% reduce 0%16/07/1719:46:49 INFO mapreduce.Job: map57% reduce 0%16/07/1719:47:07 INFO mapreduce.Job: map58% reduce 0%16/07/1719:47:22 INFO mapreduce.Job: map59% reduce 0%16/07/1719:47:37 INFO mapreduce.Job: map60% reduce 0%16/07/1719:47:55 INFO mapreduce.Job: map61% reduce 0%16/07/1719:48:10 INFO mapreduce.Job: map62% reduce 0%16/07/1719:48:25 INFO mapreduce.Job: map63% reduce 0%16/07/1719:48:43 INFO mapreduce.Job: map64% reduce 0%16/07/1719:48:58 INFO mapreduce.Job: map65% reduce 0%16/07/1719:49:13 INFO mapreduce.Job: map66% reduce 0%16/07/1719:49:28 INFO mapreduce.Job: map67% reduce 0%16/07/1719:49:30 INFO mapreduce.Job: map100% reduce 0%16/07/1719:49:37 INFO mapreduce.Job: map100% reduce 100%16/07/1719:49:38 INFO mapreduce.Job: Job job_1468752229715_0016 completed successfully
16/07/1719:49:39 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2892255
FILE: Number of bytes written=5971253
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=4056338
HDFS: Number of bytes written=861195
HDFS: Number of read operations=7
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1Data-localmap tasks=1
Total time spent byall maps in occupied slots (ms)=1016177
Total time spent byall reduces in occupied slots (ms)=4948
Total time spent byallmap tasks (ms)=1016177
Total time spent byall reduce tasks (ms)=4948
Total vcore-seconds taken byallmap tasks=1016177
Total vcore-seconds taken byall reduce tasks=4948
Total megabyte-seconds taken byallmap tasks=1040565248
Total megabyte-seconds taken byall reduce tasks=5066752Map-Reduce Framework
Map input records=51444Map output records=154332Map output bytes=2583585Map output materialized bytes=2892255
Input split bytes=103
Combine input records=0
Combine output records=0
Reduce input groups=51444
Reduce shuffle bytes=2892255
Reduce input records=154332
Reduce output records=51444
Spilled Records=308664
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=5836
CPU time spent (ms)=1033510
Physical memory (bytes) snapshot=517627904
Virtual memory (bytes) snapshot=1786634240
Total committed heap usage (bytes)=306774016
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=655419
File Output Format Counters
Bytes Written=861195
统计：
精确度：5144451367
CPU time spent (ms)=1033510map tasks=1
实验2：训练集train.txt样例个数为245057不变 测试集test.txt样例个数为51444，并将全部测试集存放在
test1.txt（25568）和test2.txt（25857）中
[root@hadoop11local]#hadoopfs-lsr/dir6/-rw-r--r--3rootsupergroup3687742016-07-1720:15/dir6/test1.txt-rw-r--r--3rootsupergroup3122102016-07-1720:15/dir6/test2.txt
ＫＮＮ算法运行日志：
先看进程日志：
[root@hadoop66 ~]# jps24659YarnChild （mapper任务）
22777DataNode25592Jps24660YarnChild （mapper任务）
24557MRAppMaster22622NodeManager
计数器日志：
