9 Sept 2015

DIY IOT : Collecting Remote Data over the Internet

One of the primary requirements for IOT - the "Internet of Things" is to collect data from remote devices over a TCP/IP internet connection and use the same for  analysis. Since the number of devices is expected to be very large -- far larger than the the number of IPv4 IP addresses that are possible, it is expected that IPv6 addressing will be used. However the sluggish rollout of IPv6 enabled network devices has necessitated the usage of intermediate "broker" services that allow the collection of data from devices and then publish the same for subsequent analytics.



This post will describe the architecture of such a system and provide sample codes that can be used to get a basic IOT Sensor Data collection.

Since the data will be travelling over the internet, the edge device must be a TCP/IP enabled computing device to which the sensor is connected. The simplest possible computer would be a Raspberry Pi2 or Arduino Yún  or similar machine, running on some lightweight flavour Unix or Linux. So in this case the sensor will be simulated by a small shell program that, in this case, emits two pieces of data (a) a sensor ID and (b) a numeric value, of temperature or pressure or voltage, that has been recorded by the sensor. With a real sensor device, this program will collect, or read it, from the sensor through a device driver program.

Once the data is available to a shell program ( we will call it ISD_pushData.sh ) it must be pushed into a central server and the easiest way to do it is using the curl command that is available in any Unix distribution. Where does one find such servers?

Companies like Carriots and GroveStream offer services that allows one to define a "device" that uses curl to send data in JSON or XML format to a datastream where it is stored for subsequent analysis. Carriots in fact offers a free service in which one can connect up to ten devices from which to collect, store and display data. Simple tutorials are available through which one can learn "How to send a stream using curl" and "How to create triggers" that will initiate actions based on the value of data that is received.

After working through these tutorials it becomes very evident that a similar service can be created on a Apache/MySQL/PhP platform that is widely available from any webhosting service like x10hosting or hostinger. The free versions are good enough for our purpose. This post in tweaking4all shows how this can be done and is forms the basis of this post.

What we need is (a) MySQL table to store the data (b) a shell program that will send data using curl to a destination URL on the web server and (c) a PhP program, available at the destination URL, that will accept data passed as parameters and insert it into the MySQL table.

The SQL command to create the MySQL table looks like this : ( Create_IOT_SensorData.sql)

CREATE TABLE `IOT_SensorData` (
`id` INT NOT NULL AUTO_INCREMENT PRIMARY KEY COMMENT 'unique ID',
`event` TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP COMMENT 'Event Date and Time',
`sensorID` VARCHAR(30 ) NOT NULL COMMENT 'Unique ID of the sensor',
`Value` NUMERIC( 4,1 ) NOT NULL COMMENT 'Value of data recorded'
);


This PhP program (ISD_pushData.php) that sits on the web server and accepts the data

<?php
    // Connect to MySQL
    include("dbConnect.php");

    $con = mysql_connect($MyHostname, $MyUsername, $MyPassword) or die(mysql_error());
    if (!$con)
     {
       die("Could not connect: 1" . mysql_error());
     }
     mysql_select_db($MyDBname) or die(mysql_error());

    // Prepare the SQL statement
    $SQL = "INSERT INTO IOT_SensorData (sensorID ,Value) VALUES ('".$_GET["sensorID"]."', '".$_GET["Value"]."')";  
    
    // Execute SQL statement
    mysql_query($SQL,$con);
    
    // Send Mail
    $mailMSG = $SQL;
    $mailDEST = "someone@somewhere.com";
    if ($_GET["Value"] > 50){
      $mailSUB = "Warning : Value HIGH";
      } else {
      $mailSUB = "Value Normal";
      }
    mail($mailDEST,$mailSUB,$mailMSG);

?>


The PhP program  on the web server accepts the data from the following shell (ISD_pushData.sh) program, running on the remote Linux machine. Instead of reading a value from the sensor, the program is generating a random number and sending it.

p1="http://prithwis.x10.bz/IOT/ISD_pushData.php?sensorID="
sensorID="1003B"
p2="&Value="
Value=$(shuf -i 1-80 -n 1)   # random number being generated for Value
URL=$p1$sensorID$p2$Value
echo $URL
# -------------------------------------------------------
curl $URL


In fact, the PhP push program on the server is not only inserting the data into the MySQL table but is also acting as a trigger by sending two different types of mail depending on the value of the data that is received from the remote sensor.

Once the data is available in the MySQL table, it can be displayed (using ISD_viewData.php) as

In fact, one can run the shell script on any machine and the new value will appear when this pages is refreshed!

It is also possible to visualize the data graphically (using ISD_graphData.php)

All the codes used in creating this post are available on Github. The graph has been created with free tools available from JpGraph and the specific graph shown here is based on the sample shown here.

In this post we have demonstrated how data lying on remote Linux machine ( that is possibly connected to a physical sensor ) can be pushed into webserver and subsequently used for data analytics.

69 comments:

  1. This is very inspirational and a very motivating material for all of us. I am glad that I have seen this post. The things that was acknowledge by our works will surely help us to achieve more awards in our life in the future. This website will surely help us in our battles that we will face

    ReplyDelete
  2. Hello, I tried to download mosquitto-1.4.3-install-win32.exe (~200 kB) (Native build, Windows Vista and up). But version 1.4.3 is not available, version 1.4.12 is available. When I tried to install it Windows Defender is saying, "Virus Detected, removing it" and not allowing installation. What should I do? Is it safe to disable Defender for the installation? I downloaded from Vietnam, and KAIST, South Korea, but with same result. Thanks. Susmit Sen

    ReplyDelete
  3. negative entries for five minutes. After five minutes, they're cleared from your cache. But if you'd like, you can force XP not to cache these negative entries, so that you'll never run into this problem.
    this page

    ReplyDelete
  4. I really appreciate this wonderful post that you have provided for us. I assure this would be beneficial for most of the people.
    Shortener

    ReplyDelete
  5. Graphical OLAP devices are utilized to show information in 2D or 3D cross tabs and outlines and charts with simple rotating of pivot.data science course in pune

    ReplyDelete
  6. Thanks for sharing your valuable information to us, it is very useful
    data science

    ReplyDelete

  7. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. I would like to state about something which creates curiosity in knowing more about it. It is a part of our daily routine life which we usually don`t notice in all the things which turns the dreams in to real experiences. Back from the ages, we have been growing and world is evolving at a pace lying on the shoulder of technology."data science courses" will be a great piece added to the term technology. Cheer for more ideas & innovation which are part of evolution.

    ReplyDelete
  8. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

    ReplyDelete
  9. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    Data Analytics Course in Mumbai

    ReplyDelete
  10. Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR

    ReplyDelete
  11. I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.
    Data Science Course in Pune

    ReplyDelete
  12. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    ExcelR Business Analytics Course

    ReplyDelete
  13. Such a great blog, Internet of Things is able to collect data from remote devices over internet and send data for analysis. We can place the IoT environmental sensors on vehicles, devices and things which help you to see everything you need from anywhere.

    ReplyDelete
  14. Very nice blog here and thanks for post it.. Keep blogging...
    ExcelR data science training

    ReplyDelete
  15. I finally found great post here.I will get back here. I just added your blog to my bookmark sites. thanks.Quality posts is the crucial to invite the visitors to visit the web page, that's what this web page is providing.
    data science course in mumbai

    ReplyDelete
  16. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.
    data science course in mumbai

    ReplyDelete
  17. Nice information, valuable and excellent work, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. data science course

    ReplyDelete
  18. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    data analytics course mumbai

    ReplyDelete
  19. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    ExcelR Data Analytics Course
    Data Science Interview Questions
    ExcelR Data Science Courses Fee in Bangalore
    ExcelR Business Analytics Course in Bangalore

    ReplyDelete
  20. I’m excited to uncover this page. I need to to thank you for ones time for this particularly fantastic read!! I definitely really liked every part of it and i also have you saved to fav to look at new information in your site.
    data analytics courses in india
    business analytics courses
    data science course

    ReplyDelete
  21. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our Data Science course Mumbai
    data science course mumbai | https://www.excelr.com/data-science-course-training-in-mumbai

    ReplyDelete
  22. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environmen python training in vijayawada. , data scince training in vijayawada . , java training in vijayawada. ,

    ReplyDelete
  23. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.

    business analytics course

    data analytics courses

    data science interview questions

    data science course in mumbai

    ReplyDelete
  24. Really awesome blog!!! I finally found a great post here.I really enjoyed reading this article. Thanks for sharing valuable information.
    Data Science Course in Marathahalli
    Data Science Course Training in Bangalore

    ReplyDelete
  25. The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
    Artificial Inteligence course in Varanasi
    AI Course in Varanasi

    ReplyDelete
  26. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

    Digital marketing course

    ReplyDelete
  27. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions

    ReplyDelete
  28. if you want to learn digital marketing in mumbai. excelr solutions providing best AI course in mumbai.for more details click here

    digital marketing courses in mumbai

    ReplyDelete
  29. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

    data science course

    ReplyDelete
  30. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course

    ReplyDelete

  31. Thank you for taking the time and sharing this information with us
    Python Training In Hyderabad

    ReplyDelete
  32. Cool stuff you have and you keep overhaul every one of us

    data science course

    ReplyDelete
  33. Attend The Data Science Course Bangalore From ExcelR. Practical Data Science Course Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Course Bangalore.
    Data Science Course Bangalore

    ReplyDelete
  34. Great blog. it was so Nice to read and useful to improve my knowledge as updated one,
    Machine Learning Training in Hyderabad

    ReplyDelete
  35. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance

    ReplyDelete
  36. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Correlation vs Covariance

    ReplyDelete
  37. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    Simple Linear Regression

    ReplyDelete
  38. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  39. This is my first time visit here. From the tons of comments on your articles.I guess I am not only one having all the enjoyment right here! ExcelR Data Science Course In Pune

    ReplyDelete
  40. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  41. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  42. Really Very Infromative Post , Thanks For Sharing The Information With Us.
    Best AWS Training Institute in Hyderabad

    ReplyDelete
  43. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science online course

    ReplyDelete
  44. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    data science interview questions

    ReplyDelete
  45. Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.
    Course in Data Analytics

    ReplyDelete
  46. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  47. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

    data science interview questions

    ReplyDelete
  48. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  49. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  50. The Way Of Explaination Is Very Good And Nice Informative Topic You Have Choosen..
    AWS Course Training in Hyderabad

    ReplyDelete
  51. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Simple Linear Regression
    Correlation vs covariance
    data science interview questions
    KNN Algorithm

    ReplyDelete
  52. Data Science Courses Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained!

    ReplyDelete
  53. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!

    Simple Linear Regression

    Correlation vs covariance

    KNN Algorithm

    Logistic Regression explained

    ReplyDelete
  54. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    data science interview questions

    ReplyDelete
  55. Data Scientist Courses Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! Cool stuff you have and you keep overhaul every one of us

    ReplyDelete
  56. This is my first time visit here. From the tons of comments on your articles.I guess I am not only one having all the enjoyment right here! ExcelR Data Science Courses

    ReplyDelete
  57. Amazing Article ! I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm
    Logistic Regression explained

    ReplyDelete
  58. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  59. There is no dearth of Data Science course syllabus or resources. Learn the advanced data science course concepts and get your skills upgraded from the pioneers in Data Science.
    data science course bangalore
    data science course syllabus
    data science training in marathahalli

    ReplyDelete
  60. Attend The Data Science Courses From ExcelR. Practical Data Science Courses Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Courses.
    Data Science Courses

    ReplyDelete
  61. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Logistic Regression explained
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm

    ReplyDelete
  62. Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing.data science institutes in Hyderabad

    ReplyDelete
  63. very well explained. I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Logistic Regression explained
    Correlation vs Covariance
    Simple Linear Regression
    data science interview questions
    KNN Algorithm

    ReplyDelete