Functional programming is the perfect Solution?

FUNCTIONAL PROGRAMMING

Functional programming is a style of programming that emphasizes writing applications using only pure functions and immutable values.

PURE FUNCTIONS

How we know that our code is pure functional or Impure , I’m putting some points on consideration

·         The function’s output depends only on its input variables

·         It doesn’t mutate any hidden state

·         It doesn’t have any “back doors”: It doesn’t read data from the outside world (including the console, web services, databases, files, etc.), or write data to the outside world

·         Example of pure function: scala.math._ package (abs,Ceil,max,min),Scala String Method(isEmpty,length,substring),Scala Collection(map,filter,drop)

Key Consideration:

A pure function is a function that depends only on its declared inputs and its internal algorithm to produce its output. It does not read any other values from “the outside world” — the world outside of the function’s scope — and it does not modify any values in the outside world.

impure functions:

the following functions are impure because they violate the definition.

The foreach method on collections classes is impure because it’s only used for its side effects, such as printing to STDOUT.

A great hint that foreach is impure is that it’s method signature declares that it returns the type Unit. Because it returns nothing, logically the only reason you ever call it is to achieve some side effect. Similarly, any method that returns Unit is going to be an impure function.

getDayOfWeek, getHour, and getMinute are all impure because their output depends on something other than their input parameters.

In general, impure functions do one or more of these things:

·         Read hidden inputs, i.e., they access variables and data not explicitly passed into the function as input parameters

·         Write hidden outputs

·         Mutate the parameters they are given

·         Perform some sort of I/O with the outside world

impure functions are needed …

Of course an application isn’t very useful if it can’t read or write to the outside world,

Write the core of your application using pure functions, and then write an impure “wrapper” around that core to interact with the outside world. If you like food analogies, this is like putting a layer of impure icing on top of a pure cake.

Key Consideration for Functional programming:

• Functional programmers don’t use null values
• A main replacement for null values is to use the Option/Some/None classes
• Common ways to work with Option values are match and for expressions
• Options can be thought of as containers of one item (Some) and no items (None)
• You can also use Options when defining constructor parameters

Say Bye Bye to Try Catch(object orientation method).... Handle Error in Pure Functional Way...

Functional Error Handling in Scala

import scala.util.Try
package ErrorHandling

***********Section1************************************************
This code will handle the error but this is not the pure functional way

we are breaking functional purity. We are not returning the same type always

object worksheet_FPStyleErrorHandling extends App {

  case class Person(name:String,location:String)
  val person = Person("Divyanshu","Bangalore")

  def getPerson(person: Person):Person ={

    if(person.location.equals("Bangalore")){
      person
    }
    else {
      throw new Exception("Invalid object")
    }
  }

  val requiredPerson = getPerson(person)
  println(requiredPerson)


}
********************Section2*********************************************

Pure Functional Way

First method: Option
Options handle both sides of the coin i.e both positive and negative.

It uses Some(value) for the positive case and None for the negative case.

object optionMethod extends App {

  case class Person1(name: String, location: String)

  val person1 = Person1("Divyanshu", "Mumbai")

 def first(person1: Person1): Option[Person1]= {
   if (person1.location.equals("Mumbaif")) {
     Some(person1)
   }
   else {
     None
   }
 }
   val requiredPerson = first(person1)
  println(requiredPerson)
}

*****************Section 3*************************************
Second Way : Try
Try is another technique to achieve functional purity. Try results in Success(value)

 or Failure(exception).Try is ideal when we are dealing with third-party libraries.

Try/Success/Failure is commonly used when writing methods that interact with files,

databases, and internet services.

object trymethod extends App{
case class Person2(name:String,location:String)
val person2 = Person2("Divyanshu","Delhi")
  def second(person2: Person2):Try[Person2] = {
    Try{
   if (person2.location.equals(4)){
     person2
   }
      else{
      throw new Exception("invalid Person")
   }
    }
  }
val requiredPersonnew = second(person2)
  println(requiredPersonnew)
}

****************Section4*******************************
Either:
Either also handles both cases as other techniques do but it is better than Option and Try.

 Option has a demerit that it does not provide the error message in case of failures, 

it just returns None for the failures. Try provides us a failure message for a failure

 but still, we need to throw an exception which we want to avoid. Either is a good 

approach to avoid these. It returns Left or Right. Left contains the error message

 that we pass in our code and Right contains the value itself.

object Either extends App {

  case class Person3(name: String, location: String)

  case class ErrorMessage(message: String)

  val person3 = Person3("Divyanshu", "UP")

  def Eithermethod(person3: Person3): Either[ErrorMessage, Person3] = {
    if (person3.location.equals("UP9")) {
      Right(person3)
    }
    else {
      Left(ErrorMessage("invalid object"))
    }
  }

  val requiredpersonEither = Eithermethod(person3)
  println(requiredpersonEither)
}

Why main method is written only in object not class?

The main method must be a static method. In Scala to create a static method you put it in an object. Methods in a class are not static.In the scala language they decided to separate class, which hold only instance behavior and state, and object which hold static behavior and state.This is different from java where classes hold both instance and static members where something is made static using the static keyword.

Feel Like a Boss , while Handling file in Scala

How often do you need to write or read files? This is a pretty common task in any more or less non-trivial project. Image upload process, CSV or XML parsing, XLS report generation etc. All these operations imply input or output processing. So I’m going to make an overview of the most powerful library for working with files in Scala.

Here is a list of operations which I want to cover:

File creation and removal
File writing and reading
File comparison
Zipping and unzipping

import better.files._
import better.files.File._

object Worksheet_WithoutUsingImport extends App{

 // How to Create a File or Directory (Or Handle If It Doesn't Exist)

//************Section1********************************************************  

//file  val Newfile: File = "C:/Users/admin/Desktop/JSON_FOLDER/DSS1.txt" 

                            .toFile.createIfNotExists()

//Directory   val directory:File = "C:/Users/admin/Desktop/JSON_FOLDER/Divyanshu" 

                                   .toFile.createIfNotExists(true)
//creation of a folder with an extra path?
  val directory1:File = "C:/Users/admin/Desktop/JSON_FOLDER/Divyanshu1/Shekhar/Singh"

                        .toFile.createIfNotExists(true,true)
//**************Section2***************************************************************

  //Writing content to file
  val WriteFile = (root/"C:/Users/admin/Desktop/JSON_FOLDER/DSS1.txt")
                 .createIfNotExists()
                 .overwrite("")
                 .appendLines("Hi How Are You","i'm doing good","Welcome!","Hello")


//************Section3******************************************************************** 

 //Read File content
  println(WriteFile.contentAsString)

  //OR
  val simpleFile = (root/ "C:/Users/admin/Desktop/JSON_FOLDER/DSS2.txt")
                   .overwrite("")
                   .appendLines("Hi How Are You", "i'm doing good", "Welcome!")
  simpleFile.lines.map(line => println(s"DSS $line SSD"))

  //Note:  // Use == :when you want to compare two files or directories by path.

  // Use === when you want to check equality of two files or folders by content.
  if (simpleFile == WriteFile){
    println(s" File comparison is done")

  }
  else    println("Not equal")
  //Delete Files and Directories
  val FileDelete = (root/"C:/Users/admin/Desktop/JSON_FOLDER/DSS3.txt" )
                   .createIfNotExists()

            if (FileDelete.exists)
              FileDelete.delete()

//*******************Section4**************************************************

//Zip and Unzip Files and Directories
  val ZipFiles = (root / "C:/Users/admin/Desktop/JSON_FOLDER/DSS2.txt")
  ZipFiles.zipTo(root / "C:/Users/admin/Desktop/JSON_FOLDER/Archive.zip")

//Note:

 Use This Library in build.sbt:

libraryDependencies += "com.github.pathikrit" %% "better-files" % "3.9.1"
}

Parse JSON data in Pure Scala

First Way:

Perquisite:
Add below dependencies in build.sbt
//************************************************************
libraryDependencies += "net.liftweb" %% "lift-json" % "3.4.0"
libraryDependencies += "io.spray" %% "spray-json" % "1.3.5"
//***************************************************************

import scala.util.parsing.json.JSON
object test1 extends App {/* Give location of any type of json file*/

  val input_file = "json.json"  val json_content = scala.io.Source.fromFile(input_file).mkString
  val json_data = JSON.parseFull(json_content)
  println(json_data)

}

File name: json.json
{
  "glossary": {
    "title": "example glossary",
    "GlossDiv": {
      "title": "S",
      "GlossList": {
        "GlossEntry": {
          "ID": "SGML",
          "SortAs": "SGML",
          "GlossTerm": "Standard Generalized Markup Language",
          "Acronym": "SGML",
          "Abbrev": "ISO 8879:1986",
          "GlossDef": {
            "para": "A meta-markup language, used to create markup languages such as DocBook.",
            "GlossSeeAlso": ["GML", "XML"]
          },
          "GlossSee": "markup"
        }
      }
    }
  }
}




Second Way:


import scala.io._
import com.fasterxml.jackson.databind.ObjectMapper
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper

object Main {
  def main(args: Array[String]): Unit = {
    val filename = args.head
    // read    println(s"Reading ${args.head} ...")
    val json = Source.fromFile(filename)
    // parse    val mapper = new ObjectMapper() with ScalaObjectMapper
    mapper.registerModule(DefaultScalaModule)
    val parsedJson = mapper.readValue[Map[String, Object]](json.reader())
    println(parsedJson)
  }
}

Note: pass filename with Absolute path as a argument.

//Output

Reading C:/Users/admin/Desktop/JSON_FOLDER/Scala_Json/json.json ...
Map(glossary -> Map(title -> example glossary, GlossDiv ->

Map(title -> S, GlossList -> Map(GlossEntry -> 

Map(ID -> SGML, Acronym -> SGML, GlossDef ->

Map(para -> A meta-markup language, 

used to create markup languages such as DocBook., GlossSeeAlso -> List(GML, XML)),

SortAs -> SGML, GlossSee -> markup, Abbrev -> ISO 8879:1986, GlossTerm ->

Standard Generalized Markup Language)))))




Third Way:

import net.liftweb.json.DefaultFormats
import net.liftweb.json._

// a case class to match the JSON datacase class EmailAccount(
                         accountName: String,
                         url: String,
                         username: String,
                         password: String,
                         minutesBetweenChecks: Int,
                         usersOfInterest: List[String]
                       )

object ParseJsonArray extends App {
  implicit val formats = DefaultFormats

  // a JSON string that represents a list of EmailAccount instances  val jsonString =""" 
   {      "accounts": [    { "emailAccount": {      "accountName": "GMail",    
  "username": "divyanshu.singh@company.com",      "password": "@12345",     
 "url": "www.gmail.com",      "minutesBetweenChecks": 1,     
 "usersOfInterest": ["Divyanshu", "shekhar", "singh"]    }},  
  { "emailAccount": {      "accountName": "Gmail",   
   "username": "singh.shekhar@gmail.com",      "password": "2345678",    
  "url": "www.gmail.com",      "minutesBetweenChecks": 1,    
  "usersOfInterest": ["DSS", "Dss1"]    }}]}    """
  // json is a JValue instance  val json = parse(jsonString)
  val elements = (json \\ "emailAccount").children
  for (acct <- elements) {
    val m = acct.extract[EmailAccount]
    println(s"Account: ${m.url}, ${m.username}, ${m.password}")
    println(" Users: " + m.usersOfInterest.mkString(","))
  }
}
//Output//Account: www.gmail.com, divyanshu.singh@company.com, @12345
// Users: Divyanshu,shekhar,singh
//Account: www.gmail.com, singh.shekhar@gmail.com, 2345678
// Users: DSS,Dss1

A comparison between RDD, DataFrame and Dataset in Spark from a developer’s point of view

UseCase: conversion of Flat_Data To XML To JSON

//**************************************

// created by Divyanshu Shekhar Singh

//**************************************

import com.thoughtworks.xstream.XStream
import com.thoughtworks.xstream.io.xml.DomDriver
import org.apache.spark.sql.types.{StringType, StructType}
import org.apache.spark.sql.{Row, SaveMode, SparkSession}

case class Name(firstName:String,middleName:String,lastName:String)
case class Person(id:String,name:Name,ssn:String,gender:String,salary:String)

object WriteXML {
  def main(args: Array[String]): Unit = {
// Spark Driver   lazy val spark:SparkSession = SparkSession.builder()
      .master("local")
      .appName("Spark")
      .getOrCreate()

    spark.sparkContext.setLogLevel("ERROR")

    val sc = spark.sparkContext
// creating flat data from code or read data from file    val data = Seq(Row("1",Row("James ","","Smith"),"36636","M","3000"),
      Row("2",Row("Michael ","Rose",""),"40288","M","4000"),
      Row("3",Row("Robert ","","Williams"),"42114","M","4000"),
      Row("4",Row("Maria ","Anne","Jones"),"39192","F","4000"),
      Row("5",Row("Jen","Mary","Brown"),"","F","-1")
    )
// create schema    val schema = new StructType()
      .add("id",StringType)
      .add("name",new StructType()
        .add("firstName",StringType)
        .add("middleName",StringType)
        .add("lastName",StringType))
      .add("ssn",StringType)
      .add("gender",StringType)
      .add("salary",StringType)

    val df = spark.createDataFrame(spark.sparkContext.parallelize(data),schema)

   // df.show()    import spark.implicits._
    // code to convert XML    //Now convert the DataFrame[Row] to DataSet[Person].    val dsPerson = df.as[Person]
   // dsPerson.show()    val dsString = dsPerson.mapPartitions(partition=>{
      val xstream = new XStream(new DomDriver)
      val data = partition.map(person=>{
        val xmlString = xstream.toXML(person)
        xmlString
      })
      data
    })
// writing data into xml file    dsString.write.mode(SaveMode.Overwrite).text("C:/Users/admin/Desktop/JSON_FOLDER/xstream.xml")

//Reading above xml data    val df1 = spark.read
      .format("xml")
      .option("rowTag","Person")
      .load("C:/Users/admin/Desktop/JSON_FOLDER/xstream.xml")
    df1.show(false)

 // converting to json
  val JsonDF = (df1.toJSON).show(false)

  }
}