Setting up Scala sbt, Spark, Hadoop and IntelliJIdea in Windows
source link: https://adaickalavan.github.io/spark/scala/setting-up-scala-spark-hadoop-intelliJIdea/#gsc.tab=0
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Setting up Scala sbt, Spark, Hadoop and IntelliJIdea in Windows
1 minute read
We briefly describe the steps to setup Scala sbt, Spark, Hadoop, and IntelliJ Idea in Windows computer.
First, please follow the tools setup instructions to install the necessary JDK and Scala sbt. Then, follow this IntelliJIdea tutorial to setup the IDE.
Next, we focus on Spark and Hadoop.
- Download the latest version of Spark from Spark website.
- Choose a Spark release: choose your desired version (I chose
spark-2.2.1) - Choose a package type: select
Pre-built for Apache Hadoop 2.7 and later - Then, click on download
spark-x.x.x-bin-hadoop2.7.tgz
- Choose a Spark release: choose your desired version (I chose
-
Unzip the Spark into your C drive, for example unzip to:
C:\spark-2.2.1-bin-hadoop2.7\ -
Download the entire winutils distribution from this repository. Form the downloaded repo, copy the folder corresponding to your installed Hadoop version to a location on your C drive. In my case I copied
hadoop-2.7.1folder toC:\. Then, rename thehadoop-x.x.xfolder towinutil. -
Go to your Environment Variables setting and set the following
System Variables:Variable Value HADOOP_HOME C:\winutil\ SPARK_HOME C:\spark-2.2.1-bin-hadoop2.7\ Path C:\spark-2.2.1-bin-hadoop2.7\bin Path C:\winutil\bin
To verify installation of Spark, open a comannd prompt and navigate to your SPARK_HOME directory, for example go to C:\spark-2.2.1-bin-hadoop2.7\. Run bin\spark-shell which should start up Scala console with a Spark session as sc. Exit the shell via :quit command.
That is it, we are done. Yay!
Recommend
-
94
在Maven核心概念一文中提到，最近的项目中使用了akka与scala，因此选择sbt（Simple Build Tool）做为项目的构建工具。在sbt之前也使用过很多的构建工具，但是sbt是我至今遇到的最复杂的一个，这和它名字里的Simple完全搭不上边。在我看来，sbt的复杂主要在两个方面...
-
33
本来不太想使用sbt，但是公司这边普遍要求使用sbt来进行部署。所以，so！ sbt的语法什么的还好，唯一让人无法忍受的是sbt下载依赖的速度 —— 在国内大环境下实在是慢到了让人抓狂的程度。 要提升sbt下载依赖的速度，方法...
-
11
Dummy Guide to sbt Table of Contents Why sbt In Scala Survey 2019 there was a question: What are the *other pai...
-
9
Provisioning a VM with Scala and SBT So I’ve been giving a little more time and attention to both Scala and Vagrant. Vagrant is a nice way to manage development environments via the use of VM’s and Scala, of course, is a great hy...
-
12
晚上不用加班了，推荐十款精选IntelliJIdea插件，效率提升N倍 ...
-
9
Reading Time: 3 minutes Scoverage… what it is, how to use it and for which build tool it is available. So, In this blog we are gonna discussing all these along with its implementation in SBT. What...
-
8
Hadoop Word Count Program in Scala Reading Time: 2 minutesYou must have seen Hadoop word c...
-
2
NetBeansIDE and IntellijIDEA From The Eyes of a Long Time Eclipse UserSkip to contentShare this:I ha...
-
8
Then follow the Stereo type So the common here they are both very simple to use, without understanding how it works even what it means, just follow the same pattern you could get the build worki...
-
14
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK