(1) establish Scala project

File->new->Project, Here's the picture  

choice Scala 

then next 

among Project SDK Specify the installed JDK,Scala SDK Specify the installed Scala( What we use here is IDEA Self contained scala SDK), Here, change the project name to SparkWordCount, then finish 

stay IDEA When developing applications in , Often need to organize the source code through a certain file directory , For example, the source file directory 、 Test source directory , Here's a demonstration in Intellij IDEA Of src Create under directory main/scala Source directory . 
Directly by F4 Or right click on the project file  

Re selection open module setting, Open project configuration , Click on src Catalog , Then right click to create main/scala Folder , Click again scala The folder is sources, As shown in the figure below  

### (2) Import Spark 1.5.0 Dependency package  
direct F4 open Project Structure, And then choose libraries 

Click on the image above + Add external dependency package , choice ”java”, Then choose spark-assembly-1.5.0-hadoop2.4.0.jar 

After success, it is as follows  

thus Spark Development environment configuration complete

2. Spark Application development

(1) Local execution Spark WordCount Program

stay src/main/scala Create a SparkWordCount Application object , The content is as follows :

import org.apache.spark.SparkContext._
import org.apache.spark.{SparkConf, SparkContext} object SparkWordCount{
def main(args: Array[String]) {
// The input file can be local linux System files , It can also be from other sources , for example HDFS
if (args.length == 0) {
System.err.println("Usage: SparkWordCount <inputfile>")
System.exit(1)
}
// Run as a local thread , You can specify the number of threads ,
// Such as .setMaster("local[2]"), Two thread execution
// Here's a single thread execution
val conf = new SparkConf().setAppName("SparkWordCount").setMaster("local")
val sc = new SparkContext(conf) //wordcount operation , The calculation file contains Spark The number of rows
val count=sc.textFile(args(0)).filter(line => line.contains("Spark")).count()
// Print the results
println("count="+count)
sc.stop()
}
}

Compile code , direct Build->Make Project 

Then program the operating parameters ,Run->Edit Configurations 

Main Class Input :SparkWordCount 
Program arguments Input :/hadoopLearning/spark-1.5.0-bin-hadoop2.4/README.md 
Here's the picture : 

When it's done, just Run->Run or Alt+Shift+F10 Run the program , The results are shown in the following figure : 

(2) Spark Execute on cluster Spark WordCount Program

take SparkWordCount Pack it up Jar file

Modify the program as follows :

import org.apache.spark.SparkContext._
import org.apache.spark.{SparkConf, SparkContext} object SparkWordCount{
def main(args: Array[String]) {
// The input file can be local linux System files , It can also be from other sources , for example HDFS
if (args.length == 0) {
System.err.println("Usage: SparkWordCount <inputfile> <outputfile>")
System.exit(1)
}
// When you submit a cluster , Local threads don't work
val conf = new SparkConf().setAppName("SparkWordCount").setMaster("local")
val sc = new SparkContext(conf) //rdd2 For all including Spark The line of
val rdd2=sc.textFile(args(0)).filter(line => line.contains("Spark"))
// Save the content , In this case, it's stored in HDFS On
rdd2.saveAsTextFile(args(1))
sc.stop()
}
}

Click Project SparkWordCount, Then press F4 To make a Project Structure And select Artifacts, Here's the picture  

choice Jar->form modules with dependencies, Here's the picture  

Enter the following screen
stay main class in , choice SparkWordCount, Here's the picture  

Click OK to get the following interface  

Because it will be submitted to the cluster to run later , So it's relevant jar All the bags are there , To reduce jar Volume of package , take spark-assembly-1.5.0-hadoop2.4.0.jar etc. jar Just delete the package , Here's the picture  

after , Click again Build->Build Artifacts 

Generated jar The file is kept in root@sparkmaster:~/IdeaProjects/SparkWordCount/out/artifacts/SparkWordCount_jar# Directory , Here's the picture : 

Submit the cluster to run

./spark-submit --master spark://sparkmaster:7077 --class SparkWordCount 
--executor-memory 1g 
/root/IdeaProjects/SparkWordCount/out/artifacts/SparkWordCount_jar/SparkWordCount.jar 
hdfs://ns1/README.md 
hdfs://ns1/SparkWordCountResult

Execution results : 

HDFS The file has been generated SparkWordCountResult 

Use

root@sparkmaster:/hadoopLearning/spark-1.5.0-bin-hadoop2.4/bin# hadoop dfs -ls

/SparkWordCountResult

root@sparkmaster:/hadoopLearning/spark-1.5.0-bin-hadoop2.4/bin# hadoop dfs -cat

/SparkWordCountResult/part-00000

View directory contents , The results are shown in the figure below :

spark Intellij IDEA More related articles on the construction of development environment

  1. IntelliJ OpenCV Development environment construction

    Windows Under the IntelliJ + OpenCV Development environment construction be based on IntelliJ IDEA 15 and OpenCV 3.1.0 1. stay OpenCV Download from the official website OpenCV Erection sequence , Double click to unzip to the target ...

  2. Spark Idea Maven Development environment construction

    One . install jdk jdk The best version is 1.7 above , Set the environment variables , setup script , A little . Two . install Maven I choose the Maven The version is 3.3.3, setup script , A little . edit Maven The installation directory conf/settings.x ...

  3. Liferay7 Intellij IDEA Development environment construction

    One . install Liferay plug-in unit The installation process is not repeated , Two installation methods are recommended : adopt Intellij Plug in market installation By downloading the plug-in zip Package installation After installation , Right click in the project section , There will be Liferay menu . Two . install L ...

  4. IntelliJ IDEA Installation and jsp Development environment construction

    One . Preface Now? .net The domestic market is not very good , The whole company .net Group technology transformation , Personally, I prefer .net, After all, not every company is as rich as Microsoft VS This cosmic level IDE For developers , double-click sln You can open the project , ...

  5. Spark+ECLIPSE+JAVA+MAVEN windows Development environment construction and entry examples 【 With detailed code 】

    http://blog.csdn.net/xiefu5hh/article/details/51707529 Spark+ECLIPSE+JAVA+MAVEN windows Development environment construction and entry examples [ attach ...

  6. spark JAVA Development environment construction and remote debugging

    spark JAVA Development environment construction and remote debugging It will be used in the project in the future Spark Do clustering analysis on user nickname text , Find out some illegal nicknames . I've been tossing about before Hadoop, So I took a look Spark Documents on the official website and github ...

  7. HBase、Hive、MapReduce、Hadoop、Spark Some steps after setting up the development environment (export export jar package or Ant The way )

    Step one if , Can't HBase Blog posts built in the development environment , See my blog below . HBase Development environment construction (Eclipse\MyEclipse + Maven) In step one , What needs to be added . as follows : In the project name , Right click , ...

  8. Apache Spark1.1.0 Deployment and development environment construction

    Spark yes Apache The company launched a new product based on Hadoop Distributed File System(HDFS) Parallel computing architecture of . And MapReduce Different ,Spark It's not limited to writing map and reduce ...

  9. Scala In simple terms, the actual combat classic ---001-Scala Development environment construction and HelloWorld analysis

    001-Scala Development environment construction and HelloWorld analysis Scala Functional programming and object-oriented language Functional programming is good at numerical calculation Object oriented is good at organization of large projects and projects and division and cooperation of teams Scala It will be the next great ...

Random recommendation

  1. tomcat To configure

    Modify the available memory size D:\escloud\apache-tomcat-7.0.63\bin Modify the catalina.bat set "JAVA_OPTS=-Xms1024m -Xmx1024m ...

  2. ViewManager

    We often need to add and delete views dynamically , It needs to be used ViewManager Interface : public interface ViewManager{ public void addView(View view ...

  3. Qt And VC Programming is dirty

    For historical reasons , The interface of a software project adopts QT, And backstage uses VC, Interface static call VC Generated dll, I have been able to cooperate happily all the time , But recently there have been two small problems , I think the cooperation between the two is not as good as I think . stay VC Next use multimedia timing ...

  4. PHP Functions that continue to execute after closing the browser

    The name of the function :ignore_user_abort  This function configures or obtains the user end after the connection is interrupted ,PHP Whether the program continues to run . The default value is to stop execution after disconnection . stay PHP In profile (php3.ini/php.ini) ...

  5. POJ 2553 The Bottom of a Graph TarJan Algorithm problem solution

    This problem is divided into two steps : 1 Use Tarjan Algorithm for all maximal sub strongly connected graphs . And it's marked 2 Then traverse these nodes to see if there are outgoing edges , All the points of a sub strongly connected graph where there are no vertices , They are all solution sets . Tarjan The algorithm is the template algorithm . This is where ...

  6. canvas( Two ) lineCap demo

    var dom = document.getElementById('clock'), ctx = dom.getContext('2d'); ctx.beginPath(); ctx.moveTo( ...

  7. Hive error :java.net.ConnectException: Connection refused: connect

    modify hive The configuration file hive-site.xml, Add the following configuration items : <property> <name>hive.server2.thrift.port</name> ...

  8. mac install thrift

    One . install brew Package management tools Don't want to Ubuntu Bring their own apt,redhat Bring their own yum,mac There is no package management tool . You need to install it yourself , The most common is brew,Homebrew abbreviation brew,OSX Package management tools on , stay ...

  9. iOS Development - Development documentation installation

    iOS Development is definitely inseparable from development documents , Apple has online help documents ,xCode You can actually download the simulator documentation and iOS8.1 Document , But the speed of the download is not flattering , And the headache is not to display the download progress bar , Apple's development documents are all in )/ Applications ...

  10. UI Introduction to design concept

    UI Introduction to design concept http://www.slideshare.net/tedzhaoxa/ui-and-ue-design-basic