Installing Apache Spark Cluster on Odroid C2 and Raspberry Pi

Contents

Summary. 1

Network configurations. 2

Odroid C2 Ubuntu 16.04 Mate. 2

Static IP. 2

Setting the Hostname. 2

Raspberry Pi 3

Static IP. 3

Setting the Hostname. 4

Common configurations. 5

Hosts file editing. 5

Disabling IP6. 5

Firewall 6

Creating an admin user to Spark. 6

Enable SSH communication between nodes. 6

Swap. 7

Installing other packages for Spark. 8

Java Installation. 8

Installing Scala. 9

Scientific Python installation. 9

Installing Apache Spark. 9

Spark Configurations. 10

Slaves. 10

log4j.properties. 11

spark-defaults.conf. 12

spark-env.sh. 13

Bash configurations. 14

Starting and Stoping Spark. 16

Submiting work to the cluster. 16

Summary

For this Spark cluster installation, I used the following packages and tools:

· Oracle Java SDK 8

· Scala 2.12.1

· Apache Spark 2.1.0

Even if you encounter URLs to other versions than those written above, they are not true. I was just lazy changing my own notes, which I have gathered from several sources. So change the files names and URLs to correspond to those you want and need.

Before I go on describing the process, I just want to mention that while it has been a bit difficult doing this installation, this was fun once I got it to work.

My biggest problems where that I come from the Microsoft ecosystem where most things are preaty much behind a UI, you don’t have to understand or know that much necessarily.

Is this good or bad? Well it depends, sometimes having a button that does things is nice but on the other hand, it takes away from actually knowing what you are doing. For example, managing users, privileges, file system etc is a totally different thing on Linux and you actually have to know what you are doing.

I found the experience with Linux very fun and enjoyable. What I struggled the most with was Spark and Hadoop (not the topic of this post). It was difficult to understand which configurations are needed. I had problems getting Hadoop to do anything and the errors whereas usually with any software obscure, or in other words a pain in the ass.

I really had to focus and want to make it to work. I felt like giving up at times.

Anyway learning using Linux was the most fun, so much fun that I ended up installing a Linux distro dual booting with Windows.

Network configurations

Odroid C2 Ubuntu 16.04 Mate

In my situation, I did the configurations through the UI but you can do it through the terminal.

Static IP

To configure a static IP goto: System > Preferences > Internet and Network > Network Proxy

There go to the IPv4 Settings and add your network desired information for the Odroid C2.

Setting the Hostname

GoTo: System > Administration > Network

Raspberry Pi

Static IP

On the top right corner press the mouse second button on the network indicator(the two arrows pointing up and down). Then select “Wireless & Wires Network Settings”. Then select the Interface and eht0 and configure the network configurations you desire.

Setting the Hostname

Goto: ”Start” icon > Preferences > Raspberry Pi Configuration > System tab > Hostname

You might need to restart your system.

Common configurations

Hosts file editing

This configuration needs must be done on every machine. You will need hostnames for your machines so that Spark and Hadoop can properly communicate between machines (nodes) in your cluster.

Type in terminal:

sudo nano /etc/hosts

Add your machine IPs and desired hostnames. My hosts file looks like this:

Notice that I have removed everything else and left the localhost definition. This is a strange this, with Spark I could not get the workers to communicate properly without the localhost definition but with Hadoop it was the other way around, not sure now why this is.

Disabling IP6

After looking several tutorials on installing Hadoop all mentioned that it is a good practice to disable the IP6 support. Apparently Hadoop does not support IP6 properly or at all. Since Apache Spark work ontop of Hadoop then I applied the same method on Spark also.

Open the /etc/sysctl.conf file for editing and add the following at the end of the file:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Firewall

In case you have problems between the nodes in a cluster disable the firewall or allow the nodes to communicate between the nodes with the needed ports.

Creating an admin user to Spark

In terminal create the user, add it to a group and give it admin privileges:

sudo addgroup spark
sudo adduser –ingroup spark spuser
sudo adduser spuser sudo

Login into the user and do everything related to Spark with this user:

su spuser

Enable SSH communication between nodes

This is to avoid using authentication when using Spark. If you do not do this you might run into problems and you also are constantly required to type in the account and password which you want to run Spark on.

Next create the SSH key and add it to the authorized_keys file.

$ cd ~
$ ssh-keygen -t rsa -P “”
$ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys

Verify that the SSH tunnel is working:

ssh localhost

Copy the public keys to the slaves nodes in you cluster

$ ssh-copy-id spuser@raspberrypi01

$ ssh-copy-id spuser@raspberrypi02

Then test the connection to the slaves:

$ ssh raspberrypi01

Swap

This is something I had to do for Odroid C2. Even though the Odroid has double the RAM Raspberry Pi has, it also has Ubuntu installed on it which takes up nearly half the memory when booting. The Raspbian OS takes about a little over 150 MB, so about 15 % of the Raspberry Pi’s total RAM.

I ran into problems when I wanted to use the Odroid as the Master and also as a slave node for calculation. Because I wanted to use as much memory as possible on the actual Raspberry Pi nodes I allocated 768 MB, which is OK for the Pi’s but I could not allocate less than 512 MB for Odroid and allocating 512 MB caused the Odroid to swap and since there was no swap created the OS crashed or became unresponsive.

To combat this problem I create a swap for Odroid, the size of the Swap was double the RAM, 4 GB:

The Swap creation guide below is from:

http://www.tutorialspoint.com/articles/how-to-enable-or-add-swap-space-on-ubuntu-16-04

Checking for the Swap Information

Before we begin, we will first check for the swap space available on the server or system

We can use the below command to see that the system is having the swap partition or not

$ free -h

We can also run the below command but if the swap partitions do not exist, we cannot see any information.

$ sudo swapon –s

In the above command, we can see that the swap is not enabled or not configured for this server to configure the swap in this machine. We will first check for the free disk space available with the below command –

$ df –h

Creating a Swap File

As we know the disk space availability, we can go ahead and create a swap file on the filesystem. To create the swap file we can use ‘fallocate’ a package or utility which can create a preallocated size to instantly. As we have a little space on the server will create a swap file with 512 MB size to create a swap file below is the command.

$ sudo fallocate -l 512M /swapfile

And to check the swap file we will use the below command

$ ls -lh /swapfile
-rw-r–r– 1 root root 512M Sep 6 14:22 /swapfile

Enabling the Swap to use the Swap File

Before, we are going to enable the swap, we need to fix the file permission that other than root any others can read/write the file below is the command to change the file permission.

$ sudo chmod 600 /swapfile

Once, we change the permission we will check the file below and execute the below command to check the swap file permissions.

Once, we change the permission we will check the file below and execute the below command to check the swap file permissions.

$  ls -lh /swapfile
-rw——- 1 root root 512M Sep  6 14:22 /swapfile

We will now make this file as a swap space using this below command –

$ sudo mkswap /swapfile
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=d02e2bbb-5fcc-4c7b-9f85-4ae75c9c55f9

Now we will enable the swap by using the below command

$ sudo swapon –s
Filename                                Type            Size    Used    Priority
/swapfile                               file            524284  0       -1

We can also check with free –h commands to see the swap partition

$ free –h

Making the Swap Partition/File to start Permanent

As in the above steps, we have created the swap partition and we are able to use that swap for temporary memory and once the machine is rebooted then the swap, setting will be lost to needed to use this swap file permanently we will make the swap file permanent.

We will edit the /etc/stab and add the information to mount the swap file even if we reboot the machine

$ sudo  vi /etc/fstab

Add the below line to the existing file.

/swapfile            none     swap     sw         0            0

For better performance for using the swap memory, we can do some tweaks.

 

Installing other packages for Spark

For my installation, I needed a few other packages to make things work:

  • Oracle Java version 8
  • Scala

Java Installation

 

For a more automatic installation, type the following commands:

$ sudo apt-get install oracle-java8-jdk

 

$ sudo apt-get update && sudo apt-get install oracle-java8-jdk

 

$ sudo update-alternatives –config java

 

For a more manual one got to the Oracle website and download the Linux ARM 64 Hard Float ABI package: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

 

Notice that my Raspberry Pi with Raspbian from December had the 32 bit Java 7 version installed. I used the 64 Bit Java 8 for Odroid C2 and 32 bit for Java 8 for Raspberry Pi.

When you have the package, do the following:

Enter the command to extract jdk-8-linux-arm-vfp-hflt.tar.gz to /opt directory.

$ sudo tar zxvf jdk-8-linux-arm-vfp-hflt.tar.gz -C /opt

 

Set default java and javac to the new installed jdk8.

$ sudo update-alternatives –install /usr/bin/javac javac /opt/jdk1.8.0/bin/javac 1

$ sudo update-alternatives –install /usr/bin/java java /opt/jdk1.8.0/bin/java 1

 

$ sudo update-alternatives –config javac

$ sudo update-alternatives –config java

 

After all, verify with the commands with -version option.

$ java -version

$ javac –version

I also added as the owner the spark user:

$ sudo chown -R spuser:spark jdk1.8.0/

Notice: To make life easier you should add environmental variables to your bashrc file. More on this in the Spark installation portion.

Installing Scala

Navigate to the following URL: http://www.scala-lang.org/download/

I downloaded the tar package, extracted it to a location and added proper privileges to the spark user:

$ sudo tar zxvf scala-2.12.1.tgz -C /opt

$ sudo chown -R spuser:spark scala-2.12.1/

 

Notice: To make life easier you should add environmental variables to your bashrc file. More on this in the Spark installation portion.

Scientific Python installation

 

This is not a requirement but I used these scripts to install Jupyter and Python 3.5 on my cluster nodes:

https://github.com/kleinee/jns

Installing Apache Spark

 

Start by downloading you desired package from Apache Spark URL: http://spark.apache.org/downloads.html

Or user wget: wget http://www.eu.apache.org/dist/spark/spark-1.5.1/spark-1.5.1-bin-hadoop2.tar.gz

Then extract Spark package

$ sudo tar -xvzf spark.2.1.0.tar.gz -C /opt/

Add the Spark User as owner

$ cd /opt
$ sudo chown -R hduser:hadoop spark.2.1.0/

If you are having problem with access to the spark folder or if you are installing Hadoop and have configured namenodes file system locations etc use the following command to add more privileges to the desired locations:

$sudo chmod 750 /opt/hadoop/hadoop_data/hdfs

Spark Configurations

Go to the Spark conf folder, depending where you installed spark:

$ cd /media/microSD/spark/conf

There are four files I had to configure for the cluster to work:

  • Slaves
  • properties
  • spark-defaults.conf
  • spark-env.sh

For more info on these files check out Spark documentation:

http://spark.apache.org/docs/latest/configuration.html

http://spark.apache.org/docs/latest/spark-standalone.html

The first step is to rename some of the files mentioned above. Some of the files have the “.template” file extension on them, remove it:

$ mv log4j.properties.template log4j.properties

If the slaves file does not exist then using nano you will create is automatically:

$ sudo nano slaves

The above assumes you are in the conf folder.

When you are done with the configurations on your master just copy them with scp to all slave nodes. Make sure you change the node specific values in these files(more on this below).

Slaves

Here you add the slave machines hostname, or the machines you want to do the work for you, the calculations:

odroid64

raspberrypi01

raspberrypi02

log4j.properties

With this file all we want is to minimize the amount of log on screen, it will be much easier to spot what is going on when you are not flooded with basic operational messages.

What you need to do it to change this:

log4j.rootCategory=INFO, console

to this:

log4j.rootCategory=WARN, console

spark-defaults.conf

Here we just want to specify the master URL so that we do not always have to specify it when submitting work to the Spark cluster:

spark.master                     spark://odroid64:7077

spark-env.sh

Here you specify the parameters that your cluster will use to communicate with all the nodes within it:

SPARK_MASTER_IP=odroid64

SPARK_WORKER_MEMORY=512m

SPARK_MASTER_HOST=odroid64

SPARK_LOCAL_IP=odroid64

SPARK_WORKER_CORES=2

SPARK_DAEMON_MEMORY=512m

SPARK_EXECUTOR_INSTANCES=1

SPARK_EXECUTOR_CORES=2

SPARK_EXECUTOR_MEMORY=512m

SPARK_DRIVER_MEMORY=512m

There are many variables, which you can tweak, I used and had to use the above ones to get things to work.

The SPARK_MASTER_IP and SPARK_MASTER_HOST HAVE to be the same on all nodes. The rest have to correspond to the actual physical node where the configuration file resides.

 

 

 

Bash configurations

 

For my cluster I used the following configurations(disregard the Hadoop ones, not necessary for Spark):

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_121

export HADOOP_HOME=/media/microSD/hadoop-2.7.3

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export YARN_HOME=$HADOOP_HOME

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop

#export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$JAVA_HOME/bin

export HADOOP_OPTS=”$HADOOP_OPTS -Djava.library.path=/media/microSD/hadoop-2.7.3/lib/native/”

#export PATH=$PATH:$JAVA_HOME/bin:/media/microSD/spark/sbin:/media/microSD/spark/sbi

 

export SBT_HOME=/media/microSD/sbt

export SPARK_HOME=/media/microSD/spark

export SCALA_HOME=/media/microSD/scala-2.12.1

export PATH=$PATH:$JAVA_HOME/bin

export PATH=$PATH:$SBT_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$SCALA_HOME/bin

export SPARK_MASTER_URL=http://192.168.10.65:7077

 

The important ones for Spark are:

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_121

export SPARK_HOME=/media/microSD/spark

export SCALA_HOME=/media/microSD/scala-2.12.1

export PATH=$PATH:$JAVA_HOME/bin

export PATH=$PATH:$SBT_HOME/bin:$SPARK_HOME/bin:$SPARK_HOME/sbin:$SCALA_HOME/bin

export SPARK_MASTER_URL=http://192.168.10.65:7077

You want to add Spark, Scala and Java to the PATH environmental variable to be able to access commands easily from Terminal.

You can then copy these configurations to the slave nodes using scp command:

http://www.hypexr.org/linux_scp_help.php

$ scp ~/.bachrc spuser@raspberrypi01:~

 

To force a refresh of the environmental variables

$ source ~/.bachrc

Starting and Stoping Spark

This is simple.

To start Spark type:

$ start-all.sh

To stop Spark:

$ stop-all.sh

To access Spark web UI:

http://odroid64:8080/

 

Submiting work to the cluster

 

Use the Spark specific command:

spark-submit

More info:

http://spark.apache.org/docs/latest/submitting-applications.html

Advertisements

Problems running scripts and batch files with Task Scheduler

This is probably one of the most annoying things I’ve encountered. I’ve been trying to run a scheduled task using a cmd file and every time I tried to run it simply didn’t work.

Then thanks to search results someone pointed out a solution and it makes no sense, thank you Microsoft. I guess there might some “logical” explanation but it is a mystery to me. Maybe something to do with UAC?

Anyway, to the solution should you happen to run into the same problem:

Do this when when specifying an action: Specify only the script/batch file name and in the Start IN(Optional) specify the full path where the file is located.

taskschedularbatch

 

How to backup your private repositories

GitHub Repository Backup Steps

Contents

Install git. 1

Use SSH key for communication with GitHub. 1

https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/. 1

Generate SSH Key. 1

Generating a new SSH key. 1

Adding your SSH key to the ssh-agent. 2

Add the SSH key to your GitHub account or the organization. 2

Setup a default user name and email 4

Set your username for every repository on your computer: 4

Setting your email address for every repository on your computer. 4

Create an access token if you want to clone repositories without a username and password. 5

Ways to access the GitHub API with an access token. 6

Cloning repositories. 7

Sample git command: 7

Sample node.js tool 7

Usage. 7

Options: 7

Examples: 7

Exclude and Only options. 7

Install git

https://git-scm.com/download/win

Use SSH key for communication with GitHub

https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/

Generate SSH Key

Generating a new SSH key

1. Open Git Bash.

2. Paste the text below, substituting in your GitHub email address.

3. ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

This creates a new ssh key, using the provided email as a label.

Generating public/private rsa key pair.

4. When you’re prompted to "Enter a file in which to save the key," press Enter. This accepts the default file location.

5. Enter a file in which to save the key (/Users/you/.ssh/id_rsa): [Press enter]

6. At the prompt, type a secure passphrase. For more information, see "Working with SSH key passphrases".

7. Enter passphrase (empty for no passphrase): [Type a passphrase]
Enter same passphrase again: [Type passphrase again]

Adding your SSH key to the ssh-agent

Before adding a new SSH key to the ssh-agent to manage your keys, you should have checked for existing SSH keys and generated a new SSH key.

If you have GitHub for Windows installed, you can use it to clone repositories and not deal with SSH keys. It also comes with the Git Bash tool, which is the preferred way of running git commands on Windows.

1. Ensure the ssh-agent is running:

o If you are using the Git Shell that’s installed with GitHub Desktop, the ssh-agent should be running.

o If you are using another terminal prompt, such as Git for Windows, you can use the "Auto-launching the ssh-agent" instructions in "Working with SSH key passphrases", or start it manually:

o # start the ssh-agent in the background
o eval $(ssh-agent -s)
o Agent pid 59566

2. Add your SSH key to the ssh-agent. If you used an existing SSH key rather than generating a new SSH key, you’ll need to replace id_rsa in the command with the name of your existing private key file.

3. $ ssh-add ~/.ssh/id_rsa

https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/

Add the SSH key to your GitHub account or the organization

For a single user

https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/

For organization

Copy the SSH key to your clipboard(use Git bash)

If your SSH key file has a different name than the example code, modify the filename to match your current setup. When copying your key, don’t add any newlines or whitespace.

$ clip < ~/.ssh/id_rsa.pub
# Copies the contents of the id_rsa.pub file to your clipboard

Tip: If clip isn’t working, you can locate the hidden .ssh folder, open the file in your favorite text editor, and copy it to your clipboard.

Notice that the .ssh folder is usually located at: C:Users{your account name}.ssh

Go to you Organization management windows and select from the left navigation: SSH and GPG keys

From this new view then press New SSH Key button

The type a title and copy the key from the clipboard

Setup a default user name and email

You can do this for a specific repository or globally. For this situation globally if preferable.

https://help.github.com/articles/setting-your-email-in-git/

https://help.github.com/articles/setting-your-username-in-git/

Set your username for every repository on your computer:

1. Navigate to your repository from a command-line prompt.

2. Set your username with the following command.

3. git config –global user.name "Billy Everyteen"

4. Confirm that you have set your username correctly with the following command.

5. git config –global user.name

Billy Everyteen

Setting your email address for every repository on your computer

1. Open Git Bash.

2. Set your email address with the following command:

3. git config --global user.email "your_email@example.com"

4. Confirm that you have set your email address correctly with the following command.

5. git config --global user.email
your_email@example.com

Create an access token if you want to clone repositories without a username and password

This is also useful if you want to create your own application to use the GitHub API for something you want to be done.

For a single user

https://help.github.com/articles/creating-an-access-token-for-command-line-use/

For the organization

Navigate to the Organization management and from the left navigation go to: Developer settings -> Personal access tokens

From this new view press the button: Generate new token

From the new view add a title to the access token then select the needed privileges to your organization. For repository backup you might need the following in the picture below.

IMPORTANT: After you have created the token remember to copy and store the key because this is the only time you will see it.

Ways to access the GitHub API with an access token

https://developer.github.com/v3/#authentication

https://developer.github.com/v3/repos/

https://developer.github.com/v3/oauth_authorizations/#create-a-new-authorization

Cloning repositories

https://help.github.com/articles/duplicating-a-repository/

Sample git command:

git clone --bare https://github.com/exampleuser/old-repository.git

Sample node.js tool

https://github.com/tegon/clone-org-repos

Usage

 cloneorg [OPTIONS] [ORG]

Options:

 -p, --perpage NUMBER number of repos per page (Default is 100)
 -t, --type STRING can be one of: all, public, private, forks, sources,
 member (Default is all)
 -e, --exclude STRING Exclude passed repos, comma separated
 -o, --only STRING Only clone passed repos, comma separated
 -r, --regexp BOOLEAN If true, exclude or only option will be evaluated as a
 regexp
 -u, --username STRING Username for basic authentication. Required to
 access github api
 --token STRING Token authentication. Required to access github api
 -a, --gitaccess Protocol to use in `git clone` command. Can be `ssh` (default), `https` or `git`
 -s, --gitsettings Additional parameters to pass to git clone command. Defaults to empty.
 --debug Show debug information
 -v, --version Display the current version
 -h, --help Display help and usage details

Examples:

clones all github/twitter repositories, with HTTP basic authentication. A password will be required

cloneorg twitter -u GITHUB_USERNAME
cloneorg twitter --username=GITHUB_USERNAME

clones all github/twitter repositories, with an access token provided by Github

cloneorg twitter --token GITHUB_TOKEN

If an environment variable GITHUB_TOKEN is set, it will be used.

export GITHUB_TOKEN='YOUR_GITHUB_API_TOKEN'

Add a -p or –perpage option to paginate response

cloneorg mozilla --token=GITHUB_TOKEN -p 10

Exclude and Only options

If you only need some repositories, you can pass -o or –only with their names

cloneorg angular --token=GITHUB_TOKEN -o angular

This can be an array to

cloneorg angular --token=GITHUB_TOKEN -o angular,material,bower-angular-i18n

This can also be an regular expression, with -r or –regexp option set to true.

cloneorg marionettejs --token=GITHUB_TOKEN -o ^backbone -r true

The same rules apply to exclude options

cloneorg jquery --token=GITHUB_TOKEN -e css-framework # simple
cloneorg emberjs --token=GITHUB_TOKEN -e website,examples # array
cloneorg gruntjs --token=GITHUB_TOKEN -e $-docs -r true # regexp
cloneorg gruntjs --token=GITHUB_TOKEN -e $-docs -r true --gitaccess=git # Clone using git protocol
# Clone using git protocol and pass --recurse to `git clone` to clone submodules also
cloneorg gruntjs --token=GITHUB_TOKEN -e $-docs -r true --gitaccess=git --gitsettings="--recurse"

Azure AD Integration

This year I’ve been working a lot more with Azure. On of my tasks has been to integrate other application to each other using Azure AD. Here are some of my finding and good to know things in case someone else runs into them:

  • Create a new MVC Application in Visual Studio
  • When this is done, press the second mouse button on the project and go to “Add” > “Connected Services”. This will allow you to create an O365 connection.
  • Select Office 365 API Services.
  • In the new Window select or type in your domain.
  • Next, create new Azure AD application configuration or use an existing one by providing the GUID for the application.
    • Make sure you select the “Configure Single Sign-On using Azure AD” option
    • Make sure that your application is multi-tenant:
      • This works so that you register your app with you own Azure AD domain, then after that external Azure AD tenants and their users are registered through an “onboarding” process. The process will the user or admin user for privileges to use certain information from the AD or other resources. These are defined in the Azure AD application settings.
      • Notice you are using an application ID and key to connect to your own organization Azure AD then the users are only onboarding using the multi-tenant option in the Azure AD application configuration.
    • Next select what kind of privileges your application needs from the Azure AD and O365.
    • You need onboarding functionality from here: https://azure.microsoft.com/en-us/documentation/samples/active-directory-dotnet-webapp-multitenant-openidconnect/
    • In Global.asax.cs application_start function add the following: AntiForgeryConfig.UniqueClaimTypeIdentifier = ClaimTypes.NameIdentifier;
      • If this is missing, then you claim will not work properly.
    • If you are using a SQL Server database and Entity Framework remember to update you model from the database and remember primary key connections. If the Entity Framework update does not work then removing and adding the database tables should force an update. Also remember to clean and build your project if nothing else helps.
    • If you get this error: Error: {“error”:”invalid_grant”,”error_description”:”AADSTS70002: Error validating credentials. AADSTS70000: The provided access grant is invalid or malformed…..
    • Doing a redirect the proper way in an MVC apllication using the following piece of code in your contoller: return Redirect(returnUrl);
      • If you use the normal way in ASP .NET: Response.Redirect(returnUrl); you will run into trouble. The error message might look something like this:
        • Server Cannot Append Header After HTTP headers have been sent Exception at @Html.AntiForgery
        • You could set the AntiForgeryConfig.SuppressXFrameOptionsHeader = true; in the Application_start, but this will lower your security and not advisable.

 

MongoDB Magic a.k.a simple database operations

Hi,

This is my little post on how to do simple database operations with MongoDB.

First let’s start with needed MongoDB needed functionalities. The simplest way is to open Visual Studio and manage your projects nugets packages. Add these four nuggets to your projects. Notice that you do not necessary need the mongocshaprdiver, it’s a legacy driver. Add it only if you need it.

mongodbnugets

Next to be able to use the classes for handling MongoDB database operation include the following namespaces:

using MongoDB.Bson;
using MongoDB.Driver;

BSON is: “BSON is a binary serialization format used to store documents and make remote procedure calls in MongoDB” More info here: https://docs.mongodb.com/manual/reference/bson-types/

Driver is: “The official MongoDB C#/.NET Driver provides asynchronous interaction with MongoDB” More Info: https://docs.mongodb.com/ecosystem/drivers/csharp/

To code below will create a client connection to the MondoDB and will get a database in it.

var client = new MongoClient("mongodb://localhost:27017");
 var database = client.GetDatabase("databaseName");

To list all collections in a database:

foreach (var item in database.ListCollectionsAsync().Result.ToListAsync&lt;BsonDocument&gt;().Result)
 {
}

 

To get the name of the collection:

var name = item.Elements.FirstOrDefault().Value.RawValue as String;

To access a collection and all of its documents:

var collection = database.GetCollection&lt;BsonDocument&gt;(name);
// This filter simply means that all data is retrieved
 var filter = new BsonDocument();

 var documents = collection.Find(filter);
 {
 {
 foreach (var document in documents.ToEnumerable())
 {
 }
}
}

To check if a document has elements(fields) with a certain data type:

var dateTimeFields = document.Elements.Where(o =&gt; o.Value.BsonType == BsonType.DateTime);
foreach (var dateTimeField in dateTimeFields)
 {
}

To update a document with certain values:

var updateFilter = Builders&lt;BsonDocument&gt;.Filter.Eq("_id", document.Elements.FirstOrDefault().Value);
 var value = BsonTypeMapper.MapToDotNetValue(dateTimeField.Value);
 var updateDefiniotion = Builders&lt;BsonDocument&gt;.Update.Set(dateTimeField.Name, BsonTypeMapper.MapToBsonValue(((DateTime)value).AddDays(3)));

 var result = collection.UpdateOne(updateFilter, updateDefiniotion);
 if (result.ModifiedCount &gt; 0)
 {
 // Success
 }
 else
 {
 // No updates
 }

Retrieving documents based on a filter and copying them into another collection + deleting from the original collection:

var timeEntriesIntoTheFuture = Builders&lt;BsonDocument&gt;.Filter.Gt(dateTimeField.Name, BsonTypeMapper.MapToBsonValue(DateTime.Now));
 var entries = sourceCollection.Find(timeEntriesIntoTheFuture);
if(entries.Count() &gt; 0)
 {
tempTimeEntryCollection.InsertMany(entries.ToEnumerable());
 foreach(var entry in entries.ToEnumerable())
 {
 sourceCollection.DeleteOne(entry);
 }
}

Checking if a collection exists:

public static bool DoesCollectionExist(IMongoDatabase database, String collectionName)
 {
 bool collectionExists = false;

 foreach (var item in database.ListCollectionsAsync().Result.ToListAsync&lt;BsonDocument&gt;().Result)
 {
 var name = item.Elements.FirstOrDefault().Value.AsString;
 if (name.Contains(collectionName))
 {
 collectionExists = true;
 break;
 }
 }

 return collectionExists;
 }

Updating an element at a certain position:

var updateDefiniotion = Builders&lt;BsonDocument&gt;.Update.Set(firstDocument.ElementAt(1).Name, BsonTypeMapper.MapToBsonValue(DateTime.Now));

Inserting a document in a manually defined manner:

var dateTimeStampeValue = new BsonDocument()
{
{ "ResetDate", BsonTypeMapper.MapToBsonValue(DateTime.Now) }
};
settingsCollection.InsertOne(dateTimeStampeValue);

Upgrading/Migrating from an older SharePoint version to a newer SharePoint

This is my checklist for when I am upgrading content from an older SharePoint to a newer one. Like from SharePoint 2010 to SharePoint 2013.

  • Make sure that you disable the outgoing Email address from the Central Administration in the web applications own settings. This is if you have events or other functionality that is to send emails based on some logic. You do not want to send emails confusing end users about emails which should not be send.
  • Install you needed packages for the migration process. This is important. If you do not do this the content database upgrade will have multiple errors and you might not be able to retrieve them.
    1. Notice another important reason why this is important is that lists and libraries are feature dependent. If there is a missing feature your lists or libraries will show empty views or will not function. Even if there is an access point through a URL, SharePoint will not be able to show you the content.
  • Next take a database backup in SQL Server from your old environment. You do this by: Pressing the secondary mouse button on a database > Tasks > Back Up… . Make sure that you check the checkbox named “Copy-only backup”. This is important, if you do not do this this will cause problems for the current database being used by the old SharePoint farm.
  • Next start the process of upgrading the content. More details here: https://technet.microsoft.com/en-us/library/cc303436.aspx
    • For a SharePoint 2010 upgrade, you might need to create a classic authentication application. More details here: https://technet.microsoft.com/en-us/library/gg251985.aspx
    • Sample command: New-SPWebApplication -name “ClassicAuthApp2” -Port 100 -ApplicationPool “ClassicAuthAppPool” -ApplicationPoolAccount (Get-SPManagedAccount “<domainname>\<user>”)
    • Before you perform the actual upgrade test the database with the following command: Test-SPContentDatabase -Name DatabaseName -WebApplication URL > TestLog.txt
      1. This will output a log file in the same folder where you are in the SharePoint management shell. Look at the log file and fix the errors found. It may not be necessary to fix them all if the errors do not affect the content you need.
      2. Notice: The WebApplication URL is the same as the classic application URL created earlier if you created it.
    • The next step is to mount the database. This is the actual upgrade process.
      1. Sample Command: Mount-SPContentDatabase -Name DatabaseName -DatabaseServer ServerName -WebApplication URL
        1. Notice: If your SharePoint farm is using SQL Aliases then open the following program and see your SQL Server database name: C:\Windows\System32\cliconfg.exe
        2. Notice: The WebApplication URL is the same as the classic application URL created earlier if you created it.
      2. If you created the classic authentication application then the next step is to convert it to a claims application.
        1. Sample command: Convert-SPWebApplication -Identity <yourWebAppUrl> -To Claims -RetainPermissions [ -Force]
      3. Moving the needed content from one content database to the other. More details here: https://technet.microsoft.com/en-us/library/cc825328.aspx
        1. Create new a content database (or multiple ones if needed) and use the Move-SPSite command to move them from one location to the other. Notice: The new content database(s) must be in the same application for the command to work.
        2. Sample command: Move-SPSite <http://ServerName/Sites/SiteName> -DestinationDatabase <DestinationContentDb>
        3. When ready with the move operation of content, then detach your new content database(s) and attach it to the new location.
  • Check that if you have XSLT based list or library field customizations that they work. If not consider using JSLink functionality in webparts of by defining it in a field through a powershell script like the one below. There is also a sample how to do it in the JavaScript file
  • Check that you calculated columns are working in your new environment.
  • If you are having problems with you quick edit functionality try this script:
    1. https://lionadi.wordpress.com/2016/06/02/fix-quick-edit-for-sharepoint-lists-after-migration-from-sharepoint-2010-to-sharepoint-2013/
  • If you have different threshold definitions in the old environment check that your old content and functionality behaves correctly in the new environment.
  • Next step upgrade your SharePoint sites look. This is offered automatically by the new version of SharePoint but if not then:
    1. /_layouts/siteupgrade.aspx ja /_layouts/siteupgradestatus.aspx
  • Next configure your search:
    1. Make sure that your new content is in one of your content sources.
    2. Run a full crawl.
  • Feature installation problems:
    1. “Check that the feature has been installed. For example, if you are performing Update-SPSolution and a new feature has been added between solution deployments then the feature is not installed by default.
    2. To check do the following:-
    3. Run SharePoint 2010 Management Shell from one of the SharePoint servers Type Install-SPFeature -ScanForFeatures
      1. This will show you any features that are available in the SharePoint Root but have not been installed. You can install any missing features using the command :-
    4. Install-SPFeature -AllExistingFeatures
      1. See the following TechNet Article for more information:
        1. http://technet.microsoft.com/en-us/library/ff607825(v=office.14).aspx
    5. Get-SPFeature| Sort -Property Scope,DisplayName | FT -GroupBy Scope DisplayName,ID
  1. If you have problems with some of your views in a list or library while others views work, then a work around is to create the view using the problem view as the base view.
</pre>
param (

[string]$SPSiteFilter = "app url"

)

&nbsp;

if ((Get-PSSnapin 'Microsoft.SharePoint.PowerShell' -ErrorAction SilentlyContinue) -eq $null){Add-PSSnapin 'Microsoft.SharePoint.PowerShell'}

&nbsp;

&nbsp;

&nbsp;

$spWebApp = Get-SPWebApplication $SPSiteFilter

foreach($site in $spWebApp.Sites)

{

$ddField = $site.rootweb.Fields["field name"]

Write-Host $site.Url

if($ddField -ne $null)

{

&nbsp;

$ddField.JSLink = "~sitecollection/Style Library/JSLinkFix.js"

$ddField.Update($true)

Write-Host "Updated"

&nbsp;

}

&nbsp;

&nbsp;

&nbsp;

&nbsp;

$site.Dispose();

}

 

 

Check the difference between two dates with a custom JavaScript implementation

This might be useful if you are having problems with different browsers and how they handle JavaScript and date + time data.


var duedateString = &quot;you due date time value here&quot;;

 var today = new Date();

 var ddYear = parseInt(duedateString.substring(0 ,4));
 var ddMonth = parseInt(duedateString.substring(4 ,6));
 var ddDay = parseInt(duedateString.substring(6 ,8));

 var yearDifference = ddYear - (today.getFullYear()) ;
 var monthDifference = ddMonth - (today.getMonth() +1);
 var dayDifference = ddDay - (today.getDate());
 var stringDueDateParsed = duedateString.substring(6 ,8) + &quot;.&quot; + duedateString.substring(4 ,6) + &quot;.&quot; + duedateString.substring(0 ,4);
 var dueDate = new Date((ddYear),(ddMonth),(ddDay));

 var totalDifference = 0;

 if(yearDifference &lt; 0)
 {
 totalDifference = -1;
 } else if(yearDifference &gt; 0)
 {
 totalDifference = 1;
 } else if(monthDifference &lt; 0)
 {
 totalDifference = -1;
 } else if(monthDifference &gt; 0)
 {
 totalDifference = 1;
 } else if(dayDifference &lt; 0)
 {
 totalDifference = -1;
 } else if(dayDifference &gt; 0)
 {
 totalDifference = 1;
 } else
 {
 totalDifference = 0;
 }

 // the dates are equal
 if(totalDifference === 0)
 {
 // do something
 }
 // The dueDate is greater than the current date, in the future
 else if(totalDifference &gt; 0)
 {
 // do something
 }
 // The dueDate is smaller than the current date, in the past
 else if(totalDifference &lt; 0)
 {
 // do something
 }