9 Sep 2015

DIY IOT : Collecting Remote Data over the Internet

One of the primary requirements for IOT - the "Internet of Things" is to collect data from remote devices over a TCP/IP internet connection and use the same for  analysis. Since the number of devices is expected to be very large -- far larger than the the number of IPv4 IP addresses that are possible, it is expected that IPv6 addressing will be used. However the sluggish rollout of IPv6 enabled network devices has necessitated the usage of intermediate "broker" services that allow the collection of data from devices and then publish the same for subsequent analytics.

This post will describe the architecture of such a system and provide sample codes that can be used to get a basic IOT Sensor Data collection.

Since the data will be travelling over the internet, the edge device must be a TCP/IP enabled computing device to which the sensor is connected. The simplest possible computer would be a Raspberry Pi2 or Arduino Yún  or similar machine, running on some lightweight flavour Unix or Linux. So in this case the sensor will be simulated by a small shell program that, in this case, emits two pieces of data (a) a sensor ID and (b) a numeric value, of temperature or pressure or voltage, that has been recorded by the sensor. With a real sensor device, this program will collect, or read it, from the sensor through a device driver program.

Once the data is available to a shell program ( we will call it ISD_pushData.sh ) it must be pushed into a central server and the easiest way to do it is using the curl command that is available in any Unix distribution. Where does one find such servers?

Companies like Carriots and GroveStream offer services that allows one to define a "device" that uses curl to send data in JSON or XML format to a datastream where it is stored for subsequent analysis. Carriots in fact offers a free service in which one can connect up to ten devices from which to collect, store and display data. Simple tutorials are available through which one can learn "How to send a stream using curl" and "How to create triggers" that will initiate actions based on the value of data that is received.

After working through these tutorials it becomes very evident that a similar service can be created on a Apache/MySQL/PhP platform that is widely available from any webhosting service like x10hosting or hostinger. The free versions are good enough for our purpose. This post in tweaking4all shows how this can be done and is forms the basis of this post.

What we need is (a) MySQL table to store the data (b) a shell program that will send data using curl to a destination URL on the web server and (c) a PhP program, available at the destination URL, that will accept data passed as parameters and insert it into the MySQL table.

The SQL command to create the MySQL table looks like this : ( Create_IOT_SensorData.sql)

`sensorID` VARCHAR(30 ) NOT NULL COMMENT 'Unique ID of the sensor',
`Value` NUMERIC( 4,1 ) NOT NULL COMMENT 'Value of data recorded'

This PhP program (ISD_pushData.php) that sits on the web server and accepts the data

    // Connect to MySQL

    $con = mysql_connect($MyHostname, $MyUsername, $MyPassword) or die(mysql_error());
    if (!$con)
       die("Could not connect: 1" . mysql_error());
     mysql_select_db($MyDBname) or die(mysql_error());

    // Prepare the SQL statement
    $SQL = "INSERT INTO IOT_SensorData (sensorID ,Value) VALUES ('".$_GET["sensorID"]."', '".$_GET["Value"]."')";  
    // Execute SQL statement
    // Send Mail
    $mailMSG = $SQL;
    $mailDEST = "someone@somewhere.com";
    if ($_GET["Value"] > 50){
      $mailSUB = "Warning : Value HIGH";
      } else {
      $mailSUB = "Value Normal";


The PhP program  on the web server accepts the data from the following shell (ISD_pushData.sh) program, running on the remote Linux machine. Instead of reading a value from the sensor, the program is generating a random number and sending it.

Value=$(shuf -i 1-80 -n 1)   # random number being generated for Value
echo $URL
# -------------------------------------------------------
curl $URL

In fact, the PhP push program on the server is not only inserting the data into the MySQL table but is also acting as a trigger by sending two different types of mail depending on the value of the data that is received from the remote sensor.

Once the data is available in the MySQL table, it can be displayed (using ISD_viewData.php) as

In fact, one can run the shell script on any machine and the new value will appear when this pages is refreshed!

It is also possible to visualize the data graphically (using ISD_graphData.php)

All the codes used in creating this post are available on Github. The graph has been created with free tools available from JpGraph and the specific graph shown here is based on the sample shown here.

In this post we have demonstrated how data lying on remote Linux machine ( that is possibly connected to a physical sensor ) can be pushed into webserver and subsequently used for data analytics.


  1. This is very inspirational and a very motivating material for all of us. I am glad that I have seen this post. The things that was acknowledge by our works will surely help us to achieve more awards in our life in the future. This website will surely help us in our battles that we will face

  2. Hello, I tried to download mosquitto-1.4.3-install-win32.exe (~200 kB) (Native build, Windows Vista and up). But version 1.4.3 is not available, version 1.4.12 is available. When I tried to install it Windows Defender is saying, "Virus Detected, removing it" and not allowing installation. What should I do? Is it safe to disable Defender for the installation? I downloaded from Vietnam, and KAIST, South Korea, but with same result. Thanks. Susmit Sen

  3. negative entries for five minutes. After five minutes, they're cleared from your cache. But if you'd like, you can force XP not to cache these negative entries, so that you'll never run into this problem.
    this page

  4. I really appreciate this wonderful post that you have provided for us. I assure this would be beneficial for most of the people.

  5. Graphical OLAP devices are utilized to show information in 2D or 3D cross tabs and outlines and charts with simple rotating of pivot.data science course in pune

  6. Thanks for sharing your valuable information to us, it is very useful
    data science


  7. Such a very useful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. I would like to state about something which creates curiosity in knowing more about it. It is a part of our daily routine life which we usually don`t notice in all the things which turns the dreams in to real experiences. Back from the ages, we have been growing and world is evolving at a pace lying on the shoulder of technology."data science courses" will be a great piece added to the term technology. Cheer for more ideas & innovation which are part of evolution.

  8. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

  9. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    Data Analytics Course in Mumbai

  10. Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR

  11. I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.
    Data Science Course in Pune

  12. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    ExcelR Business Analytics Course

  13. Such a great blog, Internet of Things is able to collect data from remote devices over internet and send data for analysis. We can place the IoT environmental sensors on vehicles, devices and things which help you to see everything you need from anywhere.

  14. Very nice blog here and thanks for post it.. Keep blogging...
    ExcelR data science training

  15. I finally found great post here.I will get back here. I just added your blog to my bookmark sites. thanks.Quality posts is the crucial to invite the visitors to visit the web page, that's what this web page is providing.
    data science course in mumbai

  16. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.
    data science course in mumbai

  17. Nice information, valuable and excellent work, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. data science course

  18. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
    data analytics course mumbai

  19. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    ExcelR Data Analytics Course
    Data Science Interview Questions
    ExcelR Data Science Courses Fee in Bangalore
    ExcelR Business Analytics Course in Bangalore

  20. I’m excited to uncover this page. I need to to thank you for ones time for this particularly fantastic read!! I definitely really liked every part of it and i also have you saved to fav to look at new information in your site.
    data analytics courses in india
    business analytics courses
    data science course

  21. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our Data Science course Mumbai
    data science course mumbai | https://www.excelr.com/data-science-course-training-in-mumbai

  22. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environmen python training in vijayawada. , data scince training in vijayawada . , java training in vijayawada. ,

  23. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.

    business analytics course

    data analytics courses

    data science interview questions

    data science course in mumbai

  24. Really awesome blog!!! I finally found a great post here.I really enjoyed reading this article. Thanks for sharing valuable information.
    Data Science Course in Marathahalli
    Data Science Course Training in Bangalore

  25. The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
    Artificial Inteligence course in Varanasi
    AI Course in Varanasi

  26. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!

    Digital marketing course

  27. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions

  28. if you want to learn digital marketing in mumbai. excelr solutions providing best AI course in mumbai.for more details click here

    digital marketing courses in mumbai

  29. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

    data science course

  30. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course


  31. Thank you for taking the time and sharing this information with us
    Python Training In Hyderabad

  32. Cool stuff you have and you keep overhaul every one of us

    data science course

  33. Attend The Data Science Course Bangalore From ExcelR. Practical Data Science Course Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Course Bangalore.
    Data Science Course Bangalore

  34. Great blog. it was so Nice to read and useful to improve my knowledge as updated one,
    Machine Learning Training in Hyderabad