The inscription is of the first level :

The fifth chapter : Construction of actual combat environment

Spark Source code compilation command :
./dev/make-distribution.sh \
--name 2.6.0-cdh5.7.0 \
--tgz \
-Pyarn -Phadoop-2.6 \
-Phive -Phive-thriftserver \
-Dhadoop.version=2.6.0-cdh5.7.0

Inscription Level 2 :

The fifth chapter : Construction of actual combat environment ( All are configured to environment variables )

1、Scala Installation :Download->previous releases  // The course uses 2.11.8

2、Maven Installation :Download->archives->binaries   // The course uses 3.3.9

modify settings.xml Default warehouse address for

3、 download hadoop Installation package (CDH5 There is... In my address )

Generate ssh Public key instructions :ssh-keygen -t rsa

Copy public key :cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys

decompression hadoop Installation package :

etc/hadoop Under the table of contents 6 Configuration files need to be modified (*#*  // Corresponding name And value pairing ):

The first part of the configuration file (4 individual ):

hadoop.env.sh // modify java_home Address

core-site.xml /*     fs.defaultFS#hdfs://hadoop000:8020     hadoop.tmp.dir#/home/hadoop/app/tmp      */

hdfs-site.xml // The copy factor is set by itself : dfs.replication#1

slaves   // Modify host localhost by hadoop000

After modification, these four files can be formatted hadoop:

Get into hadoop The directory bin Catalog , perform :./hadoop namenode -format

Configure to environment variables

To sbin Under the table of contents :./start-dfs.sh     //hadoop000:50070

The second part of the configuration file (2 individual ):

mapred-site.xml //cp A template comes out Running on the yarn On :mapreduce.framework.name#yarn

yarn-site.xml // Integrate mapreduce:yarn.nodemanager.aux-services#mapredue-shuffle

To sbin Under the table of contents :./start-yarn.sh     //hadoop000:8088

hadoop fs -ls /

hadoop fs -mkdir /data

hadoop fs -put $FILE /data/

hadoop fs -test $FILE // View details

Run a built-in example test hadoop Environment installed successfully =>

hadoop Directory is share/hadoop/mapreduce( Run in this directory ? Yes )

hadoop jar $example that jar package pi 2 3

Refresh yarn In the interface, you can see that there is content submitted

4、Hbase Installation (CDH5 The website can be downloaded 1.2.0 edition )

Three configuration files need to be modified :

hbase.env.sh //A.java_home、B. Untie ZK, And set to false, no need hbase default zk To manage

hbase-site.xml // And hadoop-site.xml compare

A.hbase.rootdir#hdfs://hadoop000:8020/hbase

B.hbase.cluster.distributed#true

C.hbase.zookeeper.quorum#hadoop000:2181

regionservers   // It is amended as follows hadoop000, And hadoop Of slaves similar

start-up : Start... First zookeeper And hadoop Of dfs, Restart hbase,bin Under the table of contents : ./start-hbase.sh

hadoop000:60010 As the default UI port

Use ( If you can't or forget a word, you can press tab Key tip ):./hbase shell

View version :version

Check the status :status

Build table :create 'member','info','address'

View all tables :list

Look at the details of the table :describe 'member'

5、Spark Environment building , See Note 1's inscription 2 for details 、 Inscription four or :https://www.imooc.com/article/18419

bin Under the table of contents ./spark-shell --master local[2]    // Default UI Port is :hadoop000:4040

IDEA build maven Of Spark Streaming Environmental Science , On the basis of the original :

1. Add three version:

spark.version:2.2.0

hadoop:2.6.0-cdh5.7.0

hbase:1.2.0-cdh5.7.0

2. Because use cdh5, So introduce cdh5 Warehouse <repositories><repository>:

id# You can take it yourself

url#http://repository.cloudera.com/artifactory/cloudera-repos

3. Add four dependency:// A single dependency actually contains a lot of dependencies , Otherwise, it will be very troublesome to introduce one by one

groupId#org.apache.*

artifactId#hadoop-client、hbase-client、hbase-server、spark-streaming_2.11

version#//${ Relative to the introduction of }

ps:spark-streaming_2.11 It can be found on the official website

Programming Guides->Spark Streaming-> to pull down :Basic Concepts notice , Directly introduced

In the end rebuild once maven project ( Menu bar or right click )

【 MOOCS real combat 】Spark Streaming Real time stream processing project actual combat notes 7 inscription upgrade more related articles

  1. 【 MOOCS real combat 】Spark Streaming Real time streaming project combat notes 10 inscription upgrade

    The inscription is of the first level : Chapter viii. :Spark Streaming Advance and case combat updateStateByKey Operator requirements : Count the number of words accumulated so far ( You need to stay the same as before ) java.lang.Illega ...

  2. 【 MOOCS real combat 】Spark Streaming Real time stream processing project actual combat notes 14 inscription upgrade

    The inscription is of the first level : The first 11 Chapter Spark Streaming Integrate Flume&Kafka Build the foundation of General stream processing streaming.conf agent1.sources=avro-sourceagent1 ...

  3. 【 MOOCS real combat 】Spark Streaming Real time stream processing project combat notes 2: updated version of the inscription

    The inscription is of the first level : Chapter two : Real time stream processing demand : Statistics master station each ( Appoint ) The client of course access . Geographical information distribution regional :ip transformation Spark SQL Project practice client :useragent obtain Hadoop basic course ==&g ...

  4. 【 MOOCS real combat 】Spark Streaming Real time streaming project combat notes 16 inscription upgrade

    The inscription is of the first level : linux crontab Website :http://tool.lu/crontab Executed every minute crontab expression : */1 * * * * crontab -e */1 * * * ...

  5. 【 MOOCS real combat 】Spark Streaming Real time streaming project actual combat notes 15 inscription upgrade

    The inscription is of the first level :[ Wood has notes ] Inscription Level 2 : The first 12 Chapter Spark Streaming Project practice Behavior log analysis : 1. Statistics of visits 2. Website stickiness 3. recommend Python Real time data generation visit URL->IP Information - ...

  6. 【 MOOCS real combat 】Spark Streaming Real time streaming project actual combat notes 12 inscription upgrade

    The inscription is of the first level : ======Pull Mode integration Flume Agent Compiling : flume_pull_streaming.conf simple-agent.sources = netcat-sources ...

  7. 【 MOOCS real combat 】Spark Streaming Real time streaming project actual combat notes 11 inscription upgrade

    The inscription is of the first level : The first 8 Chapter Spark Streaming Advance and case combat Blacklist filtering Access log ==> DStream20180808,zs20180808,ls20180808,ww ==> ( ...

  8. 【 MOOCS real combat 】Spark Streaming Real time stream processing project actual combat notes 9 inscription upgrade

    The inscription is of the first level : The core concept :StreamingContext def this(sparkContext: SparkContext, batchDuration: Duration) = { this(s ...

  9. 【 MOOCS real combat 】Spark Streaming Real time stream processing project combat notes 8 inscription upgrade

    The inscription is of the first level : Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, ...

Random recommendation

  1. Mysql The string field determines whether a string is included 3 Methods

    Method 1 : SELECT * FROM users WHERE emails like "%b@email.com%"; Method 2 : utilize MySQL  String function find_in_set ...

  2. 01Spring_ basic jia Package import andSpring The overall structure of and How to add log function

    1. What is? Spring : v\:* {behavior:url(#default#VML);} o\:* {behavior:url(#default#VML);} w\:* {behavior:u ...

  3. 【Android】Android Open source project classification summary

    The first part Personalized controls (View) Mainly introduce those good personalized View, Include ListView.ActionBar.Menu.ViewPager.Gallery.GridView.ImageView.Pro ...

  4. establish TextView The location section can be clicked , Different colors

    String url=" Register on behalf of treasure drive , You have allowed members to < Terms of service >, Please read... Carefully . "; SpannableStringBuilder style = new Spa ...

  5. kvm Dynamic mount hard disk

    According to the latest demand, we need to give kvm Under the windows Virtual machine mount hard disk , I checked a lot of information on the Internet and finally tried to get through , Record here for your convenience , Colleagues can give you a reference , Make complaints about Tucao if you have any questions Environmental Science : Let's talk about the environment I use first , The environment is to use ...

  6. Machine learning based web Anomaly detection

    Machine learning based web Anomaly detection Web Firewall is the first line of defense of information security . With the rapid update of network technology , New hacking technologies are emerging one after another , It brings challenges to the traditional rule firewall . Tradition web Intrusion detection technology intercepts intrusion access by maintaining rule set . One ...

  7. Problem solving :HNOI 2013 Cards

    Topic Apart from not shuffling , The color in each cycle of each shuffle must be the same , And then just pack it vigorously . Finally, remember to include the plan of no shuffling #include<cstdio> #include<cstrin ...

  8. &lt; Seven years to be a millionaire : A practical course for Europe's most famous rich coaches &gt; Reading notes

    If optimism shows you the positive side of things , So confidence also gives you the confidence to deal with the dark side Only the weak can shirk responsibility Treat problems as opportunities , Ask yourself : How can I create an environment , Make the problem never come back Ask the right questions Just you yourself? , instead of ...

  9. HashMap HashTable ConcurrentHashMap

    1. Hashtable and HashMap (1) difference , These two classes are different in the following aspects :Hashtable and HashMap It's all done Map Interface , however Hashtable The implementation is based on Dictionary smoke ...

  10. turn :HTML5 Medium element.dataset

    Use HTML5 Medium element.dataset Operation customization data-* data : Not long ago, I showed you something very useful classList API, It is a kind of HTML5 The native page elements provided in CSS Class into ...