Hadoop Tutorial
Time (sec)
Exit Code
Command
StdOutput
StdError
Clone Lenses.io Intro Repo
1.73
0
git clone https://gitlab.com/landoop/landoop-intro.git
Cloning into 'landoop-intro'...
hdfs put
9.57
0
hadoop fs -put ../README.md input.md
wordcount example
38.27
0
hadoop jar hadoop-examples.jar wordcount input.md results
build code
1.27
0
hadoop com.sun.tools.javac.Main WordCount.java
create jar
0.13
0
jar cf wc.jar WordCount.class WordCount$IntSumReducer.class WordCount$TokenizerMapper.class
run jar
26.78
0
hadoop jar wc.jar WordCount input.md results-2
Passed 6 out of 6
77.76 seconds
Spark Tests
Time (sec)
Exit Code
Command
StdOutput
StdError
Create Test Scala App
0.00
0
tee coyote-spark-test-app/src/main/scala/CoyoteTestApp.scala
Create Sbt File
0.00
0
tee coyote-spark-test-app/main.sbt
Put into HDFS a Test File
3.27
0
hadoop fs -put -f coyote-spark-test-app/src/main/scala/CoyoteTestApp.scala
Build SBT Package
15.43
0
sbt package
Spark-submit Application (Local)
8.03
0
spark-submit --class "CoyoteTestApp" --master local[4] coyote-spark-test-app/target/scala-2.11/coyote-test-project_2.11-1.0.jar
Verify via HDFS
2.06
0
hadoop fs -cat spark-results.txt/part-00000
Linecount: 19
Spark-submit Application (CDH Cluster)
19.01
0
spark-submit --class "CoyoteTestApp" --master yarn --deploy-mode cluster coyote-spark-test-app/target/scala-2.11/coyote-test-project_2.11-1.0.jar
Verify via HDFS
2.51
0
hadoop fs -cat spark-results.txt/part-00000
Linecount: 19
Passed 8 out of 8
50.31 seconds