侯上校
侯上校 · 共 170 篇文章
tensorflow安装
首先, 安装所有必备工具: # 在 Mac 上: $ sudo easy_install pip # 如果还没有安装 pip $ sudo pip install --upgrade virtualenv 接下来, 建立一个全新的 virtualenv 环境. 为了将环境建在 ~/tensorflow 目录下, 执行: $ virtualenv --system-site-packages ~/tensorflow $ cd ~/tensorflow 然后, 激活 virtualenv: [colonelhou@localhost tensorflow]$ source bin/activa…
2018-04-10 10:38 · AI · 评论(0)
树霉派控制电灯
VCC=>3.3v GND=>地 IN=>GPIO 01 常开、公共端当开关 import com.pi4j.io.gpio.GpioController; import com.pi4j.io.gpio.GpioFactory; import com.pi4j.io.gpio.GpioPinDigitalOutput; import com.pi4j.io.gpio.PinState; import com.pi4j.io.gpio.RaspiPin; public class Test{ public static void main(String args[]) throws Inte…
2017-11-19 22:16 · 树霉派 · 评论(0)
解决数据倾斜
import org.apache.spark._ import scala.collection.mutable._ import scala.collection.mutable.HashMap import org.apache.spark.Partitioner import java.net.URL val sc = SparkContext.getOrCreate val rdd1 = sc.textFile("/home/hadoop/my.data") val rdd2 = rdd1.map(x => { val url = x.split(",")(0) val urls …
2017-11-17 19:39 · spark · 评论(0)
小度小度,你几岁了?
2017-08-10 15:43 · AI · 评论(0)
SparkStreaming定时器生成Job
/** * An interface to represent clocks, so that they can be mocked out in unit tests. */ private[spark] trait Clock { def getTimeMillis(): Long def waitTillTime(targetTime: Long): Long } /** * A clock backed by the actual time from the OS as reported by the `System` API. */ private[spark] class Sys…
2016-11-15 18:47 · spark · 评论(0)
Hive的Base64使用
public static void main(String[] args) throws IOException { BASE64Encoder base64=new BASE64Encoder(); base64.encode("http://www.xnhzj.com/".getBytes("utf-8"), System.out); System.out.println(); String apache = new String(org.apache.commons.codec.binary.Base64.encodeBase64("http://www.xnhzj.com/".ge…
2016-01-13 10:29 · hive · 评论(0)
编译基于Hadoop2.6的Spark1.6源码
...... [INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar. [WARNING] hadoop-yarn-common-2.6.0.jar, hadoop-yarn-api-2.6.0.jar define 3 overlapping classes: [WARNING] - org.apache.hadoop.yarn.factories.package-info [WARNING] - org.apache.hadoop.yarn.util.package-info [WARNING…
2016-01-12 14:50 · spark · 评论(0)
【转】Scala的协变和逆变
转自:http://my.oschina.net/scipio/blog/282794 一、概念 协变和逆变是描述在集合中的多态关系与原来类型多态关系之间的关系。如果T1是T的子类,那么Container[T1]是不是Container[T]的子类呢?Variance注释允许你在类继承和多态类型之间表达下面的这些关系: * covariant(协变),[+T],C[T1]是 C[T]的子类 * contravariant(逆变),[-T],C[T] 是 C[T1]的子类 * invariant(不变),[T], C[T] 与 C[T1]不相关 ●协变:covariance,将子类实例的容器赋…
2016-01-11 20:56 · spark · 评论(0)
编译spark-project的hive
mvn clean compile package install -Phadoop-2 -DskipTests main: [delete] Deleting directory /usr/local/src/spark_hive/hive-release-1.2.1-spark/packaging/target/tmp [delete] Deleting directory /usr/local/src/spark_hive/hive-release-1.2.1-spark/packaging/target/warehouse [mkdir] Created dir: /usr/loca…
2015-12-17 17:18 · hive · 评论(0)
编译Spark1.5.1源码
[root@cluster spark-1.5.1]# mvn -Dhadoop.version=2.6.0-cdh5.4.0 -Dscala-2.10.4 -Phive -Dmaven.test.skip=true clean package [INFO] Replacing original artifact with shaded artifact. [INFO] Replacing /usr/local/src/spark-1.5.1/external/kafka-assembly/target/spark-streaming-kafka-assembly_2.10-1.5.1.ja…
2015-12-15 14:51 · spark · 评论(0)
编译Hive1.2.1
[root@cluster apache-hive-1.2.1-src]# mvn clean package -Pdist -Phadoop-2 -Dmaven.test.skip=true .... Downloaded: https://repo1.maven.org/maven2/org/codehaus/plexus/plexus-interpolation/1.15/plexus-interpolation-1.15.jar (60 KB at 123.5 KB/sec) Downloaded: https://repo1.maven.org/maven2/org/apache/…
2015-12-15 14:22 · hive · 评论(0)
Hadooop序列化框架使用
第一步:实现Writable接口(TestGenericWritable.java) public static class Baz extends Bar { @Override public void readFields(DataInput in) throws IOException { super.readFields(in); //needs a configuration parameter assertEquals("Configuration is not set for the wrapped object", CONF_TEST_VALUE, getConf().get…
2015-07-03 11:03 · hadoop · 评论(0)
Flume写数据换行
在Flume写入HDFS的时候会自动换行问题, 项目要求写入HDFS中数据不需要换行, 查看源码及配置如下: BodyTextEventSerializer.java @Override public void write(Event e) throws IOException { out.write(e.getBody()); if (appendNewline) { out.write('\n'); } } // for legacy reasons, by default, append a newline to each event written out private final…
2015-05-18 10:43 · flume · 评论(0)