技术标签： jvm hadoop 集群 大数据平台开发笔记（hadoop|storm|spark）
1、刚在公司搭建好的一个集群，然后运行wordcount测试看是否能正常使用，发现报如下错误（我在自己电脑上也是用同一版本，并没有报错）
[root@S1PA124 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output
14/08/20 09:51:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/08/20 09:51:35 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/08/20 09:51:36 INFO input.FileInputFormat: Total input paths to process : 1
14/08/20 09:51:36 INFO mapreduce.JobSubmitter: number of splits:1
14/08/20 09:51:36 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/08/20 09:51:36 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/08/20 09:51:36 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/08/20 09:51:37 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/08/20 09:51:37 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/08/20 09:51:37 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1408499127545_0001
14/08/20 09:51:37 INFO impl.YarnClientImpl: Submitted application application_1408499127545_0001 to ResourceManager at /0.0.0.0:8032
14/08/20 09:51:37 INFO mapreduce.Job: The url to track the job: http://S1PA124:8088/proxy/application_1408499127545_0001/
14/08/20 09:51:37 INFO mapreduce.Job: Running job: job_1408499127545_0001
14/08/20 09:51:44 INFO mapreduce.Job: Job job_1408499127545_0001 running in uber mode : false
14/08/20 09:51:44 INFO mapreduce.Job: map 0% reduce 0%
14/08/20 09:51:49 INFO mapreduce.Job: map 100% reduce 0%
14/08/20 09:51:54 INFO mapreduce.Job: Task Id : attempt_1408499127545_0001_r_000000_0, Status : FAILED
Container [pid=26042,containerID=container_1408499127545_0001_01_000003] is running beyond virtual memory limits. Current usage: 35.5 MB of 1 GB physical memory used; 16.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1408499127545_0001_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 26047 26042 26042 26042 (java) 36 3 17963216896 8801 /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_0 3
|- 26042 25026 26042 26042 (bash) 0 0 65409024 276 /bin/bash -c /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_0 3 1>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003/stdout 2>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000003/stderr
Container killed on request. Exit code is 143
14/08/20 09:52:00 INFO mapreduce.Job: Task Id : attempt_1408499127545_0001_r_000000_1, Status : FAILED
Container [pid=26111,containerID=container_1408499127545_0001_01_000004] is running beyond virtual memory limits. Current usage: 100.3 MB of 1 GB physical memory used; 16.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1408499127545_0001_01_000004 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 26116 26111 26111 26111 (java) 275 8 18016677888 25393 /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_1 4
|- 26111 25026 26111 26111 (bash) 0 0 65409024 275 /bin/bash -c /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_1 4 1>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004/stdout 2>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000004/stderr
Container killed on request. Exit code is 143
14/08/20 09:52:06 INFO mapreduce.Job: Task Id : attempt_1408499127545_0001_r_000000_2, Status : FAILED
Container [pid=26185,containerID=container_1408499127545_0001_01_000005] is running beyond virtual memory limits. Current usage: 100.4 MB of 1 GB physical memory used; 16.8 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1408499127545_0001_01_000005 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 26190 26185 26185 26185 (java) 271 7 18025807872 25414 /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_2 5
|- 26185 25026 26185 26185 (bash) 0 0 65409024 276 /bin/bash -c /opt/lxx/jdk1.7.0_51/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Djava.awt.headless=true -Djava.io.tmpdir=/root/install/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1408499127545_0001/container_1408499127545_0001_01_000005/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA org.apache.hadoop.mapred.YarnChild 10.58.22.221 10301 attempt_1408499127545_0001_r_000000_2 5 1>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005/stdout 2>/root/install/hadoop-2.2.0/logs/userlogs/application_1408499127545_0001/container_1408499127545_0001_01_000005/stderr
Container killed on request. Exit code is 143
14/08/20 09:52:13 INFO mapreduce.Job: map 100% reduce 100%
14/08/20 09:52:13 INFO mapreduce.Job: Job job_1408499127545_0001 failed with state FAILED due to: Task failed task_1408499127545_0001_r_000000
Job failed as tasks failed. failedMaps:0 failedReduces:1
14/08/20 09:52:13 INFO mapreduce.Job: Counters: 32
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=80425
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=895
HDFS: Number of bytes written=0
HDFS: Number of read operations=3
HDFS: Number of large read operations=0
HDFS: Number of write operations=0
Job Counters
Failed reduce tasks=4
Launched map tasks=1
Launched reduce tasks=4
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3082
Total time spent by all reduces in occupied slots (ms)=11065
Map-Reduce Framework
Map input records=56
Map output records=56
Map output bytes=1023
Map output materialized bytes=1141
Input split bytes=96
Combine input records=56
Combine output records=56
Spilled Records=56
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=25
CPU time spent (ms)=680
Physical memory (bytes) snapshot=253157376
Virtual memory (bytes) snapshot=18103181312
Total committed heap usage (bytes)=1011875840
File Input Format Counters
Bytes Read=799<configuration>
<property>
<name>mapreduce.cluster.local.dir</name>
<value>/root/install/hadoop/mapred/local</value>
</property>
<property>
<name>mapreduce.cluster.system.dir</name>
<value>/root/install/hadoop/mapred/system</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>S1PA124:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>S1PA124:19888</value>
</property>
<!--
<property>
<name>mapred.child.java.opts</name>
<value>-Djava.awt.headless=true</value>
</property>
<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Djava.awt.headless=true -Xmx1024m</value>
</property>
<property>
<name>yarn.app.mapreduce.am.admin-command-opts</name>
<value>-Djava.awt.headless=true</value>
</property>
-->
</configuration>我把mapred-site.xml配置文件里配置与JVM运行内存空间的那几行配置注释掉，然后重新启动集群就解决了。具体原因暂时还没有时间来研究，大概知道是与机器JVM的分配情况有关。
文章浏览阅读1.1k次。一、选择题1． 串行接口是指( )。A. 接口与系统总线之间串行传送，接口与I/0设备之间串行传送B. 接口与系统总线之间串行传送，接口与1/0设备之间并行传送C. 接口与系统总线之间并行传送，接口与I/0设备之间串行传送D. 接口与系统总线之间并行传送，接口与I/0设备之间并行传送【答案】C2． 最容易造成很多小碎片的可变分区分配算法是( )。A. 首次适应算法B. 最佳适应算法..._874 计算机科学专业基础综合题型
文章浏览阅读9.7k次，点赞5次，收藏15次。连接xshell失败，报错如下图，怎么解决呢。1、通过ps -e|grep ssh命令判断是否安装ssh服务2、如果只有客户端安装了，服务器没有安装，则需要安装ssh服务器，命令：apt-get install openssh-server3、安装成功之后，启动ssh服务，命令：/etc/init.d/ssh start4、通过ps -e|grep ssh命令再次判断是否正确启动..._could not connect to '192.168.17.128' (port 22): connection failed.
文章浏览阅读209次。00000000_杰理 空白芯片 烧入key文件
文章浏览阅读475次。2023年初，“ChatGPT”一词在社交媒体上引起了热议，人们纷纷探讨它的本质和对社会的影响。就连央视新闻也对此进行了报道。作为新传专业的前沿人士，我们当然不能忽视这一热点。本文将全面解析ChatGPT，打开“技术黑箱”，探讨它对新闻与传播领域的影响。_引发对chatgpt兴趣的表述
文章浏览阅读259次。用Python数据分析方法进行汉字声调频率统计分析木合塔尔·沙地克;布合力齐姑丽·瓦斯力【期刊名称】《电脑知识与技术》【年(卷),期】2017(013)035【摘要】该文首先用Python程序,自动获取基本汉字字符集中的所有汉字,然后用汉字拼音转换工具pypinyin把所有汉字转换成拼音,最后根据所有汉字的拼音声调,统计并可视化拼音声调的占比.【总页数】2页(13-14)【关键词】数据分析;数据可..._汉字声调频率统计
文章浏览阅读64次。最近在做一个android系统移植的项目，所使用的开发板com1是调试串口，就是说会有uboot和kernel的调试信息打印在com1上（ttySAC0）。因为后期要使用ttySAC0作为上层应用通信串口，所以要把所有的调试信息都给去掉。参考网上的几篇文章，自己做了如下修改，终于把调试信息重定向到ttySAC1上了，在这做下记录。参考文章有：http://blog.csdn.net/longt..._嵌入式rootfs 输出重定向到/dev/console
文章浏览阅读1.2k次，点赞4次，收藏12次。1，先去iconfont登录，然后选择图标加入购物车 2，点击又上角车车添加进入项目我的项目中就会出现选择的图标 3，点击下载至本地，然后解压文件夹，然后切换到uniapp打开终端运行注：要保证自己电脑有安装node(没有安装node可以去官网下载Node.js 中文网)npm i -g iconfont-tools（mac用户失败的话在前面加个sudo，password就是自己的开机密码吧）4,终端切换到上面解压的文件夹里面，运行iconfont-tools 这些可以默认也可以自己命名（我是自己命名的_uniapp symbol图标
文章浏览阅读1.2w次，点赞25次，收藏192次。char*和char[]都是指针，指向第一个字符所在的地址，但char*是常量的指针，char[]是指针的常量_c++ char*
文章浏览阅读930次。代码编辑器或者文本编辑器，对于程序员来说，就像剑与战士一样，谁都想拥有一把可以随心驾驭且锋利无比的宝剑，而每一位程序员，同样会去追求最适合自己的强大、灵活的编辑器，相信你和我一样，都不会例外。我用过的编辑器不少，真不少～ 但却没有哪款让我特别心仪的，直到我遇到了 Sublime Text 2 ！如果说“神器”是我能给予一款软件最高的评价，那么我很乐意为它封上这么一个称号。它小巧绿色且速度非
文章浏览阅读4.1k次。一、选择法这是每一个数出来跟后面所有的进行比较。2.冒泡排序法，是两个相邻的进行对比。_对十个数进行大小排序java
文章浏览阅读2.9k次。物联网开发笔记——使用网络调试助手连接阿里云物联网平台（基于MQTT协议）其实作者本意是使用4G模块来实现与阿里云物联网平台的连接过程，但是由于自己用的4G模块自身的限制，使得阿里云连接总是无法建立，已经联系客服返厂检修了，于是我在此使用网络调试助手来演示如何与阿里云物联网平台建立连接。一.准备工作1.MQTT协议说明文档（3.1.1版本）2.网络调试助手（可使用域名与服务器建立连接）PS：与阿里云建立连解释，最好使用域名来完成连接过程，而不是使用IP号。这里我跟阿里云的售后工程师咨询过，表示对应_网络调试助手连接阿里云连不上
文章浏览阅读544次，点赞5次，收藏6次。运算符与表达式任何高级程序设计语言中，表达式都是最基本的组成部分，可以说C++中的大部分语句都是由表达式构成的。_无c语言基础c++期末速成