Itversity.com Website Review

Setup Development Environment – IntelliJ and Scala – Big Data – Hadoop and Spark | IT Versity

As part of this blog post we will see detailed instructions about setting up development environment for Spark and Hadoop application development using Windows. We have used Windows 10 for this demo using 64 bit version on Setup development environment on Windows For each of the section we will see Why we need to perform the step? How to perform the step? How we can validate whether it is working as expected? We will also develop few programs to validate whether our setup is progressing as expected or not In case you run into any issues, please log those in our forums Click here for the coupons for our content. Our training approach is certification oriented. Click here to go to our state of the art lab to practice Spark hands on for more realistic experience Setup Development environment on Windows We are considering fresh Windows laptop. We will start with a/JDK on Windows laptop and we will go through step by step instructions to setup Scala, sbt, WinUtils etc. For integrated development using IntelliJ Typically programming will be done with IDEs such as IntelliJ IDEs are typically integrated with other tools such as git which is code versioning tool. Tools like git facilitate team development. sbt is build tool for Scala. Once applications are developed using IDE, they are typically built using tools like sbt WinUtils is required for HDFS APIs to work on Windows laptop Unless a is setup and validated successfully do not go further. If you need our support, please log the issues in our forums. Setup a and JDK Steps are almost same as demonstrated for Mac as well as Linux Before getting started check whether a and JDK are installed or not Launch command prompt - Go to search bar on windows laptop, type cmd and hit enter Type a -version If it return version, check whether 1.8 or not. It is better to have 1.8 version. If you have other version, consider uninstall and install 1.8 (Search for programs installed and uninstall a) Type ac -version If it return version, check whether 1.8 or not. It is better to have 1.8 version. If you have other version, consider uninstall and install 1.8 (Search for programs installed and uninstall a) If you need other versions, make sure environment variables point to 1.8 If you do not have a at all, make sure to follow the instructions and install 1.8 version of JRE and JDK. Why do we need to install a and JDK? Scala, Spark and many other technologies require a and JDK to develop and build the applications. Scala is JVM based programming language. How to install a and JDK? Go to official page of Oracle where s are available Accept the terms and 64 bit version How to validate? Use a -version and ac -version commands in command prompt and see they return 1.8 or not Setup Scala with IntelliJ Steps are almost same as demonstrated for Mac as well as Linux Now install IntelliJ There are 2 versions of IntelliJ community edition and enterprise edition Community edition is free and at times you need to install additional plugins Enterprise edition is paid and supported and comes with most of the important plugins pre-installed. Also set of plugins are bundled together as part of enterprise edition Unless you have corporate license for now consider installing community edition. Why IntelliJ? IntelliJ is created by JetBrains and it is very popular in building IDEs which boost productivity in team development Scala and SBT can be added as plugins using IntelliJ Most commonly used tools such as git comes out of the box for versioning the code in the process of application development by teams. How to Install? Go to the s page and make sure right version is chosen. Once ed, just double click on installable and follow typical installation process How to validate? We will develop a program as part of next section to validate. Develop Hello World Program We will see how to create first program using Scala as sbt project. Steps are almost same as demonstrated for Mac as well as Linux Click on New Project For the first time, it selects a by default. Make sure to choose Scala and then sbt Give name to the project -> spark2demo Choose right version of Scala -> 2.11.12 Choose right version of sbt -> 0.13 It will take some time to setup the project Once done you will see src directory with the structure src/main/scala src/main/scala is base directory for scala code build.sbt under project name - name of the project version - project version (0.1) scalaVersion - scala version (2.11.12) name := spark2demo version := 0.1 scalaVersion := 2.11.12 Steps to develop HelloWorld program Right click on src/main/scala Choose Scala Cl Give name as Hello World and change type to object Replace the code with below code object HelloWorld { def main(args: Array): Unit = { println(Hello World) } } Right click and run the program You should see Hello World in the console Make sure IntelliJ setup with Scala is done and validated by running Hello World program. In case of any issues, please log in our forums. Setup sbt and run application Once the application is developed, we need to build jar file and migrate to higher environments. sbt is the build tool which is typically used for Scala based projects. Why sbt? To build scala based applications to jar file Validate jar file to make sure program is running fine How to setup sbt? Setup sbt by ing relevant able from this link For Windows use Microsoft Installer (msi) For Mac install using brew How to validate sbt? Copy the path by right clicking the project in IntelliJ Go to command prompt and cd to the path Check the directory structure, you should see src directory build.sbt Run sbt package It will build jar file and you will see the path Run program by using sbt run command You should see Hello World printed on the console Add Spark dependencies to the application As we are done with validating IntelliJ, Scala and sbt by developing and running the program, now we are ready to integrate Spark and start developing Scala based applications using Spark APIs. Steps are almost same as demonstrated for Mac as well as Linux Update build.sbt by adding libraryDependencies += org.apache.spark %% spark-core % 2.3.0 Enable auto-import or click on refresh on type right corner It will take some time to dependencies based on your internet speed Be patient until all the spark based dependencies are s. You can expand External Dependencies in project view to see list of jars ed. build.sbt will look like this name := spark2demo version := 0.1 scalaVersion := 2.11.12 libraryDependencies += org.apache.spark %% spark-core % 2.3.0 Setup WinUtils to get HDFS APIs working Not required for Mac or Linux Why to install winutils? In the process of building data processing applications using Spark, we need to read data from files Spark uses HDFS API to read files from several file systems like HDFS, s3, local etc For HDFS APIs to work on Windows, we need to have WinUtils How to install winutils? Click here to 64 bit winutils.exe Create directory structure like this C:/hadoop/bin Setup new environment variable HADOOP_HOME Search for Environment Variables on Windows search bar Click on Add Environment Variables There will be 2 categories of environment variables User Variables on top System Variables on bottom Make sure to click on Add for System Variables Name: HADOOP_HOME Value: C:hadoop (don't include bin) Also choose Path and click on Edit Click on Add Add new entry %HADOOP_HOME%in Setup Data sets You need to have data sets setup for your practice. Steps are almost same as demonstrated for Mac as well as Linux Go to our GitHub data repository You can setup data sets in 2 ways If you have git, you can clone to the desired directory on your PC Otherwise use , it will zip file Unzip and copy to C:data You will have multiple datasets ready for your practice Develop first spark application Now we are ready to develop our first Spark application. Steps are almost same as demonstrated for Mac as well as Linux Go to src/main/scala Right click and click on New -> Package Give the package name as retail_db Right click on retail_db and click on New -> Scala Cl Name: GetRevenuePerOrder Type: Object Replace the code with this code snippet package retail_db import org.apache.spark.{SparkConf, SparkContext} object GetRevenuePerOrder { def main(args: Array): Unit = { val conf = new SparkConf(). setMaster(args(0)). setAppName(Get revenue per order) val sc = new SparkContext(conf) sc.setLogLevel(ERROR) val orderItems = sc.textFile(args(1)) val revenuePerOrder = orderItems. map(oi => (oi.split(,)(1).toInt, oi.split(,)(4).toFloat)). reduceByKey(_ + _). map(oi => oi._1 + , + oi._2) revenuePerOrder.saveAsTextFile(args(2)) } } Program takes 3 arguments args(0) -> execution mode args(1) -> input path args(2) -> output path Running the application Go to Run menu -> Edit Configurations Add new application Give application name GetRevenuePerOrder Choose main cl : retail_db.GetRevenuePerOrder Program arguments: local Use cl path for module: Choose spark2demo Click on Apply and then Ok Now you can run the application by right clicking and choosing Run GetRevenuePerOrder Go to output path and check files are created for output or not Build jar file Let us see how we can build the jar file and run it. Steps are almost same as demonstrated for Mac as well as Linux Copy the path by right clicking the project in IntelliJ Go to command prompt and cd to the path Check the directory structure, you should see src directory build.sbt Run sbt package It will build jar file and you will see the path It will be typically /target/scala-2.11/spark2demo_2.11-0.1.jar We can also run using sbt run-main sbt run-main retail_db.GetRevenuePerOrder local Now you are ready with the jar file to get deployed. If you have any issues please raise it in our forums. and Install Spark on Windows Now let us see the details about setting up Spark on Windows Skip this for Mac or Linux (we will see again after setting up Ubuntu using Windows subsystem for Linux) Why to setup Spark? Before deploying on the cluster, it is good practice to test the script using spark-submit. To run using spark-submit locally, it is nice to setup Spark on Windows How to setup Spark? Install 7z so that we can unzip and untar spark tar ball, from here spark 2.3 tar ball by going here Choose Spark Release: 2.3.0 Choose a package type: Pre-built for Hadoop 2.7 or later It gives the appropriate link pointing to mirror Click on it go to mirror and click on it to Use 7z software to unzip and under to complete setup of spark We need to configure environment variables to run Spark any where Keep in mind that Spark is not very well supported on Windows and we will see how to setup on Ubuntu using Windows subsystem for Linux. Configure environment variables for Spark Let us see how we can configure environment variables of Spark Skip this for Mac or Linux Why to setup Environment Variables? To run spark-submit, spark-shell from any where on the PC using the jar file. How to configure Environment Variables? Let us ume that Spark is setup under C:spark-2.3.0-bin-hadoop2.7 Setup new environment variable SPARK_HOME Search for Environment Variables on Windows search bar Click on Add Environment Variables There will be 2 categories of environment variables User Variables on top System Variables on bottom Make sure to click on Add for System Variables Name: SPARK_HOME Value: C:spark-2.3.0-bin-hadoop2.7 (don't include bin) Also choose Path and click on Edit Click on Add Add new entry %SPARK_HOME%in How to validate? Go to any directory and run spark-shell Run Spark job using spark-shell Using spark-shell we can validate ad hoc code to confirm it is working. It will also confirm whether the installation is successful or not. Skip this for Mac or Linux Run spark-shell Execute this code and make sure it return results val orderItems = sc.textFile(C:\data\retail_db\order_items) val revenuePerOrder = orderItems. map(oi => (oi.split(,)(1).toInt, oi.split(,)(4).toFloat)). reduceByKey(_ + _). map(oi => oi._1 + , + oi._2) revenuePerOrder.take(10).foreach(println) On Windows after showing the output, it might throw the exception. Run Spark application using Spark submit We can validate the jar file by using spark-submit Skip this for Mac or Linux spark-submit is the main command to submit the job --cl retail_db.GetRevenuePerOrder, to p the cl name By default master is local, if you want to override we can use --master After spark-submit and control arguments we have to give jar file name followed by arguments spark-submit --cl retail_db.GetRevenuePerOrder local Setup Ubuntu using Windows subsystem for Linux Now let us see how we can setup Ubuntu on Windows 10 Skip this for Mac or Linux Why to setup Ubuntu? Windows is not completely fool proof in running spark jobs. Using Ubuntu is better alternative and you will run into fewer issues Using Windows subsystem for Linux we can quickly set up Ubuntu virtual machine How to setup Ubuntu using Windows subsystem for Linux? Follow this link to setup Ubuntu using Windows subsystem for Linux Complete the setup process by giving username for the Ubuntu virtual machine Accessing C Drive using Ubuntu built using Windows subsystem for Linux Skip this for Mac or Linux It is better to understand how we can access C drive in Ubuntu built using subsystem for Linux It will facilitate us to access files in C drive In Linux root file system starts with / and does not have partitions like C drive The location of C drive is /mnt/C Setup a and JDK on Ubuntu Skip this for Mac or Linux Before getting started check whether a and JDK are installed or not Launch command prompt - Go to search bar on windows laptop, type cmd and hit enter Type a -version If it return version, check whether 1.8 or not. It is better to have 1.8 version. If you have other version, consider uninstall and install 1.8 (Search for programs installed and uninstall a) Type ac -version If it return version, check whether 1.8 or not. It is better to have 1.8 version. If you have other version, consider uninstall and install 1.8 (Search for programs installed and uninstall a) If you need other versions, make sure environment variables point to 1.8 If you do not have a at all, make sure to follow the instructions and install 1.8 version of JRE and JDK. Why do we need to install a and JDK? Scala, Spark and many other technologies require a and JDK to develop and build the applications. Scala is JVM based programming language. How to install a and JDK on Ubuntu? sudo add-apt-repository ppa:webupd8team/ a sudo apt-get update sudo apt-get install oracle- a8-installer How to validate? Use a -version and ac -version commands in command prompt and see they return 1.8 or not and Untar Spark Now let us see the details about setting up Spark on Ubuntu or any Linux flavor or Mac. Why to setup Spark? Before deploying on the cluster, it is good practice to test the script using spark-submit. To run using spark-submit locally, it is nice to setup Spark on Windows How to setup Spark? spark 2.3 tar ball by going here. We can use wget to the tar ball. Choose Spark Release: 2.3.0 Choose a package type: Pre-built for Hadoop 2.7 or later It gives the appropriate link pointing to mirror Click on it go to mirror and click on it to Use tar xzf command to untar and unzip tar ball - tar xzf spark-2.3.0-bin-hadoop2.7.tgz We need to configure environment variables to run Spark any where Setup Environment Variables - Mac or Linux Let us see how we can configure environment variables of Spark Why to setup Environment Variables? To run spark-submit, spark-shell from any where on the PC using the jar file. How to configure Environment Variables? Let us ume that Spark is setup under /Users/itversity/spark-2.3.0-bin-hadoop2.7 on Mac /mnt/c/spark-2.3.0-bin-hadoop2.7 on Ubuntu built using Windows subsystem Setup new environment variable SPARK_HOME and update PATH Make sure to restart terminal (no need to reboot the machine) # On Mac - .bash_profile export SPARK_HOME=/Users/itversity/spark export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin # On Ubuntu built using Windows subsystem for Linux - .profile export SPARK_HOME=/mnt/c/spark-2.3.0-bin-hadoop2.7 export PATH=$PATH:$SPARK_HOME/bin How to validate? Go to any directory and run spark-shell val orderItems = sc.textFile(C:\data\retail_db\order_items) val revenuePerOrder = orderItems. map(oi => (oi.split(,)(1).toInt, oi.split(,)(4).toFloat)). reduceByKey(_ + _). map(oi => oi._1 + , + oi._2) revenuePerOrder.take(10).foreach(println) Run jar file using Spark Submit We can validate the jar file by using spark-submit spark-submit is the main command to submit the job --cl retail_db.GetRevenuePerOrder, to p the cl name By default master is local, if you want to override we can use --master After spark-submit and control arguments we have to give jar file name followed by arguments spark-submit --cl retail_db.GetRevenuePerOrder local Conclusion and where to go from here This post covers how to set up development environment to work on Spark projects using Scala as a team However to gain in-depth knowledge of Spark, you can follow our content and practice on our state of the art big data cluster Click here for the coupons for our content Click here to go to our state of the art lab to practice Spark hands on for more realistic experience

Setup Development Environment – IntelliJ and Scala – Big Data – Hadoop and Spark

Setup Development environment on Windows

Related

Product Engineering Bootcamp – Using MEAN Stack – 201805 | IT Versity

Mode: Online & Cl room Duration: 45 - 60 Days Fill form here or send a mail at [email protected] to enroll for the boot camp. Skills: 1. Be a pro In this module we will emphasize on skills required to be pro in IT. Basics of Computer Windows Overview Editors Overview Emphasis on typing skills 2. Linux Fundamentals (Self Paced Online) Overview of Operating System Logging into linux (including p wordless) Basic linux commands Editors such as vi/vim Regular expressions Processing information using awk/sed Basics of shell scripting Troubleshooting the issues 3. Database Essentials (Self Paced Online) Database skills is key for any IT professional to excel. As part of this module we will focus on data modeling to writing advanced SQL queries Overview of Relational Databases Normalization Creating tables and manipulating data Basics SQL ytical Functions Relating RDBMS with NoSQL Writing queries in MongoDB 4. UI/UX- HTML, CSS, ascript, Bootstrap HTML, CSS, ascript are the foundation of web programming. Bootstrap is a framework built to Basics of HTML Basics of CSS Basics of Form Basics of HTTP (Headers, Request Methods) HTML5 Form Validations Introduction to Bootstrap Basics of ascript (Variables, Functions, Array, JSON) Debugging and Troubleshooting DOM Manipulations using ascript AJAX Concepts Object Oriented Programming in ascript Promises and Lambda Functions Fetch Library Basics of Jquery 5. Basics of HTTP HTTP is the foundation of data communication for the World Wide Web. How internet works Basics of Networking Request Methods Headers and Status Codes Simulating Requests using CURL Using Postman 6. Backend Programming Using Node.js, Express and MongoDB This is where we will see how to build logic into our application and integrate it with a database. Basics of Node.js NPM Asynchronous Programming Callback Functions File IO Request module Promises in Node.Js MongoDB Essentials Mongoose Basics of Express Developing REST API 6. Building Angular Applications Angular is a ascript framework to build complex web applications with relative ease. Angular-cli Typescript Bootstrapping application Components Modules Angular Directives Forms in Angular Dependency Injection HTTP Routing Services Debugging and Troubleshooting 7. Deploying Applications and AWS Deploying applications and knowledge of cloud is essential for any product engineer. Overview of Cloud Understanding AWS (Amazon Web Services) Setting up EC2 instances AWS CLI (Command Line Interface) Deploying Angular applications using Nginx Deploying Node.js Applications CI/CD using Bitbucket Pipelines 8. Job Marketing and Social Networking tips Towards the end of the training we will give some tips related to job marketing and social networking tips Understanding job portals Building LinkedIn Profile Using github Supporting on Stack Overflow Blogging on technology trends Resources: Dedicated Slack/WhatsApp group Lifetime access to training videos Practice exercises 24×7 community based support Cost: $600 for working professionals Discounts: Early-bird Discount: $500 Please send an email to [email protected] to receive the discounted invoice Student Discount: $300 for students To avail the 'Student Discount', send an email to [email protected] with the following details: Email ID. College/University ID or a recent grade sheet Bulk discounts are also available for corporate companies. Schedule and timings: 7 AM to 9 AM IST (Might Extend for 15-30 Mins) Saturday, Sunday, Monday 9:30 PM to 11:30 PM US EST Friday, Saturday, Sunday FAQs about the Bootcamp Why is it called a 'Product Engineering' bootcamp? Product engineers by definition build products for an enterprise. Through the course of this bootcamp, a user will learn this essential skill and will be equipped to develop products end to end using the latest web technologies. Why does one need to learn Linux Fundamentals or Database Essentials? What have they to do with Product Engineering? Linux Fundamentals, Database Essentials and Programming are key to successful careers in the world of IT. For product engineering, a good understanding of Linux and SQL commands —besides the knowledge of programming languages like ascript— is not only valuable, but essential. Their particular benefits include: Linux: Linux is the standard operating system used in various enterprises across the globe. Understanding Linux commands and a bit of shell scripting make one comfortable in enterprise world. Linux knowledge also helps in increase in productivity by improving troubleshooting, debugging and automating validations. Database Essentials such as SQL are relevant for a wide range of roles in the IT industry. Who's the best suited audience for the bootcamp? Who can gain the most value from it? Students who are looking to get into enterprises to start off their careers as a product engineer or full stack engineer Frontend developers looking to learn latest frameworks like angular etc or complement their skills with backend development Backend developers looking to complement their skills with UI/UX using Angular and other relevant technologies Testing professionals, to transition to development roles Entry level professionals, to learn essential skills that are relevant to the industry to become full stack developers I am not a programmer, but have experience in the IT industry. Would I find Product Engineer training relevant? Good programming skills are required to become a full stack developer or a product engineer. During the period of this training you will be trained in building applications using aScript and relevant frameworks such as Angular, Node, Bootstrap etc. How long would it take for a beginner to gain the skills to be employable? It should take about four months, subject to one's discipline and curiosity to learn. Does ITVersity offer istance in job-seeking at the end of the bootcamp? Not only do we guarantee skills and industry-readiness by the end of the bootcamp, but also have experts give users specific tips to seek out relevant job opportunities ITVersity also engages with clients in staffing, and helps connect partnering corporates to rightly skilled individuals Training does not guarantee employment, but we try our best to ensure each dedicated user gain relevant skills, and is recognized for them. Have More questions? Ping us at [email protected] and we will help you with whatever details you need.

Product Engineering Bootcamp – Using MEAN Stack – 201805

1. Be a pro

Related

Content Subscription with Lab Access | IT Versity

Big Data Courses with Lab Access We are glad to announce Udemy and lab integrated plans. Here are the coupons. Udemy + 1 month lab access Click here for $45 coupon to purchase CCA Spark and Hadoop Developer using Python Click here for $45 coupon to purchase CCA Spark and Hadoop Developer using Scala Click here for $36 coupon to purchase HDPCD:Spark using Python Click here for $36 coupon to purchase HDPCD:Spark using Scala Udemy + 3 month lab access + 1 week simulator access. To avail this plan if you end up paying less than the coupon value due to Udemy deals, you need to pay the difference. For example if you end up paying $50 to Udemy due to any deal, to avail this plan you need pay difference of $25. We will invoice you the difference amount. Click here for $75 coupon to purchase CCA Spark and Hadoop Developer using Python Click here for $75 coupon to purchase CCA Spark and Hadoop Developer using Scala Click here for $65 coupon to purchase HDPCD:Spark using Python Click here for $65 coupon to purchase HDPCD:Spark using Scala Please note that you will get welcome message once you sign up for the course which will have a link to avail the lab access. You need to copy paste the link to the browser and then fill the form. If you just need simulator access, please send email to [email protected]. It is $14.45 for one week.

Content Subscription with Lab Access

Big Data Courses with Lab Access

Related

Data Engineering Live Bootcamp – Using Big Data – 201803 | IT Versity

Mode: Hybrid (Cl room in Hyderabad for up to 25 students and Online via zoom for up to 75 students) Duration: 3 to 4 months Depending up on the feedback from the people who attended earlier kick off session, we are going to give overview about Data Engineering and Big Data, so that you can make well thought out decision. Please click here to fill the form and sign up for upcoming kickoff session on March 9th morning 8 AM India time. Also early bird discount is extended till March 9th end of day. Skills: 1. Be a pro In this module we will emphasize on skills required to be pro in IT. Basics of Computer Windows Overview Editors Overview Presentation and Communication skills using Microsoft Office Emphasis on typing skills 2. Database Essentials Database skills is key for any IT professional to excel. As part of this module we will focus on data modeling to writing advanced SQL queries Overview of Relational Databases Normalization Creating tables and manipulating data Basics SQL ytical Functions Relating RDBMS with NoSQL Writing queries in MongoDB 3. Programming Essentials Data Engineers should have good grasp of fundamentals of programming. We will be teaching Scala as well as Python as programming languages. There is no need of object oriented concepts. But if any one want to explore we will support. Data Types Basic programming constructs Pre-defined functions (string manipulation) User defined functions (including lambda functions) Collections Basic I/O operations Database operations Externalizing properties 4. Linux Fundamentals After Programming and Databases, understanding operating system on which these technologies run is very important to excel in any IT role. Overview of Operating System Logging into linux (including p wordless) Basic linux commands Editors such as vi/vim Regular expressions Processing information using awk/sed Basics of shell scripting Troubleshooting the issues 5. Overview of Big Data eco system In this small but effective module we will give brief overview of all the technologies in Big Data eco system along with weightages. File Systems Overview Processing Engines Overview HDFS commands YARN Hive Sqoop Flume Distributions 6. Databases in Big Data Now let us understand about how to create databases in Big Data. Hive Overview Creating databases, tables and loading data Queries in Hive Hive based engines File formats Integration of Spark SQL with Scala/Python - Overview 7. Building applications at scale This is the most important part of the training. In this we will use Scala and/or Python and build applications using Spark. Overview of Spark Reading data from file systems Processing data using Core Spark API Processing data using Data Sets and/or Data Frames Processing data using Spark SQL Saving data to file systems Development life cycle Execution life cycle Troubleshooting and performance tuning 8. Data Ingestion In this we will see how we can get data into HDFS from different sources. Copying data between RDBMS and HDFS using Sqoop Copying data between RDBMS and Hive using Sqoop Real time data ingestion using Flume Data Ingestion using Kafka Copying data between RDBMS and HDFS using Spark JDBC 9. Streaming ytics In this module we will get data in near real time and load into HDFS while processing it. Integrating data from Flume to Kafka Getting golden copy of data using Flume to HDFS Integration of Kafka and Spark Streaming Apply ytics rules on inflight data using Spark Steaming APIs 10. Visualization After processing the data we need to visualize it. Overview of BI and Visualization tools Setting up Tableau Desktop Connecting to different data sources Creating reports Creating dashboards 11. Big Data on Cloud For many clients, having dedicated clusters is not necessary. Quite often we should be able to use pay-as-you-go model of Cloud and process data at regular intervals. Overview of Cloud Understanding AWS (Amazon Web Services) Setting up EC2 instances AWS CLI (Command Line Interface) Creating AWS EMR cluster using both web console as well as CLI Step execution Running Spark Jobs Deploying applications using Azkaban 12. Job Marketing and Social Networking tips Towards the end of the training we will give some tips related to job marketing and social networking tips Understanding job portals Building LinkedIn Profile Using github Supporting on Stack Overflow Blogging on technology trends Resources: Lab access for 1 year Dedicated Slack group Lifetime access to training videos Practice exercises 24×7 community based support Cost (15th March is final date): $600 for users from the US and other users abroad INR 28,000 + GST for Indian users Discounts: Student Discount: $300 for students from the US and other students abroad INR 14,000 + GST for Indian students To avail the 'Student Discount', send an email to [email protected] with the following details: Email ID. College/University ID or a recent grade sheet Bulk discounts are also available for corporate companies. Schedule and timings: 4 days a week US: Monday to Thursday, 9:30PM to 10:45PM Eastern Time (Some sessions may further extend by up to 15 minutes) India: Tuesday to Friday, corresponding AM time (Sessions could begin at 7AM or 8AM, based on US time-zone changes. Users will be informed accordingly.) All US and Indian holidays and long weekends will be honored. Introduction to Data Engineering and Big Data FAQs about the Bootcamp Why is it called a 'Data Engineering' bootcamp? Data Engineering, by definition, is the practice of processing data for an enterprise. Through the course of this bootcamp, a user will learn this essential skill and will be equipped to process both streaming data and data in offline batches. Why does one need to learn Linux Fundamentals, Database Essentials, or languages like Python and Scala? What have they to do with Data Engineering? Linux Fundamentals, Database Essentials and Programming are key to successful careers in the world of IT. For data engineering, a good understanding of Linux and SQL commands —besides the knowledge of programming languages like Python and/or Scala— is not only valuable, but essential. Their particular benefits include: Linux: Linux is the standard operating system used in various enterprises across the globe. Understanding Linux commands and a bit of shell scripting make one comfortable in enterprise world. Linux knowledge also helps in increase in productivity by improving troubleshooting, debugging and automating validations. Database Essentials such as SQL are relevant for a wide range of roles in the IT industry. Python is the preferred programming language in Data Science and Data Engineering, having a wide array of easy-to-use libraries to work with data. Scala, a programming language structurally similar to Python, is necessary to implement powerful Data Engineering tools like Spark Do I really need to know two programming languages, Python as well as Scala? Also, why not a? It is in no regard mandatory, but knowledge in both Python and Scala could significantly improve an IT professional's access to opportunities in the world of Big Data. a, however, is not considered flexible enough anymore to develop innovative data processing frameworks. Relatively difficult to use, a is not as popular as Python or Scala for Data Engineering. Who's the best suited audience for the bootcamp? Who can gain the most value from it? Traditional ETL and Data Warehouse developers Mainframes professionals hoping to switch careers to open systems. The bootcamp is sure to add great value to their prior experience in dealing with heavy volumes of data Testing professionals, to transition to development roles Entry level professionals, to learn essential skills that are relevant to their industries Is the bootcamp certified? Is the certification recognized elsewhere? The bootcamp offers a course completion certificate from ITVersity As of now, the certificate is not widely recognized in the corporate world However, ITversity offers certification-oriented content to help users prepare for industry recognized certifications in various Big Data technologies I am not a programmer, but have experience in the IT industry. Would I find Data Engineer training relevant? Yes, Data Engineering can help further any career in IT, as long as one is open to learning basic programming skills, Linux Fundamentals and Database Essentials. How long would it take for a beginner to gain the skills to be employable? It should take about four months, subject to one's discipline and curiosity to learn. Does ITVersity offer istance in job-seeking at the end of the bootcamp? Not only do we guarantee skills and industry-readiness by the end of the bootcamp, but also have experts give users specific tips to seek out relevant job opportunities ITVersity also engages with clients in staffing, and helps connect partnering corporates to rightly skilled individuals Training does not guarantee employment, but we make the best attempts we can to ensure each dedicated user gains the relevant skills, and is recognized for them What are the Certifications covered as part of this course? CCA 175 Spark and Hadoop Developer (100%) HDPCD:Spark (100%) HDPCD (70%) Databricks/O'reilly Certified Spark Developer (90%) MapR Certified Spark Developer (90%)

Data Engineering Live Bootcamp – Using Big Data – 201803

1. Be a pro

Related

Evaluation – Python Collections | IT Versity

Exercise 1 - Getting Started Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Get number of records Print first 10 records Exercise 2 - Loops - Extract order_id and order_item_subtotal Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Using loops, create collection of tuples containing order_id, order_item_subtotal order_id is second field and of type int order_item_subtotal is 5th field and of type float Get number of records Print first 10 records Exercise 3 - Loops and Function - Compute order revenue for a given order_id Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Create a function which take orderItems collection and order_id as arguments Using loops and if conditions Extract order_item_subtotal from the collection for order_id p ed Compute revenue Function should return order revenue for given order_id Invoke the function by p ing orderItems and order_id Print order revenue Exercise 4 - Simple Map Reduce - Compute order revenue for a given order_id Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Create a function which take orderItems collection and order_id as arguments Using filter, map and reduce Extract order_item_subtotal from the collection for order_id p ed Compute revenue Function should return order revenue for given order_id Invoke the function by p ing orderItems and order_id Print order revenue Exercise 5 - Loops and Function - Compute order revenue for each order_id Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Create a function which take orderItems collection as argument Using loops and if conditions Extract order_id and order_item_subtotal from the collection Compute revenue for each order_id Function should return dict which contain order_id and revenue Invoke the function by p ing orderItems and ign to variable Get number of records Print first 10 records Exercise 6 - Loops and Function - Get number of users per social networking platform Read data from /data/kgisl.csv into collection called kgislData Create a function which take kgislData collection Using loops Get social networking field (10th one) Extract each of the social networking platform Perform count Return dict which contain social networking platform and count Invoke the function by p ing kgislData as argument and ign it to a variable Get number of records from dict Print all the records from the collection Try to sort the data in descending order by number of users per social networking platform Exercise 7 - Develop function to process list of tuples Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Apply map function and create tuples of order_id and order_item_subtotal with variable name orderItemsTuples order_id is of type int order_item_subtotal is of type float Create a function which take orderItemsTuples and orderId as arguments Using filter, map and reduce Extract order_item_subtotal from the collection for orderId p ed Compute revenue (using sum or reduce) Function should return tuple which contain orderId and revenue Invoke the function by p ing orderItemsTuples and orderId and ign to variable Print the tuple Exercise 8 - map, groupby, reduce from itertools - Compute order revenue for each order_id Read data from /data/retail_db/order_items/part-00000 into collection called orderItems Create a function which take orderItems collection as argument Using map reduce apis including itertools Extract order_id, order_item_subtotal in the form of tuples Group records for each order_id Compute revenue for each order_id Function should return dict which contain order_id and revenue Invoke the function by p ing orderItems and ign to variable Get number of records Print first 10 records Try to sort the data in descending order by revenue generated and print first 10 records

Evaluation – Python Collections

Exercise 1 – Getting Started

Related