how to get rid of that pesky outlier/erroneous data point

Have you ever ingested sensor data and due to one little glitch in the setup you have one extreme outlier in the data that makes the sensor chart unreadable (see first image)?

Don’t worry, if it is just this one you can narrow down to the exact timeframe that holds only this one outlier (see second image). then take note of the from and to time stamps (you need to convert the time shown in the UI to utc time).

Then do a get request against this timeframe to see that you have locked down the right timeframe – for example:

https://onboardtest2cf.flp-iot-sap.cfapps.eu10.hana.ondemand.com/appiot-mds/Things(‘F92E92EF251D4922B7C7C24A658A8BFA’)/iot.onboardtest2cf.greenhouse2:greenhouse4/envData4?timerange=2018-08-17T06:21:23Z-2018-08-17T06:21:24Z

returned

{
“value”: [
{
“_time”: “2018-08-17T06:21:23.475Z”,
“temperature”: -3276.7,
“humidity”: 76.7
},
{
“_time”: “2018-08-17T06:21:23.469Z”,
“temperature”: -3276.7,
“humidity”: 76.7
},
{
“_time”: “2018-08-17T06:21:23.468Z”,
“temperature”: -3276.7,
“humidity”: 76.7
}
]
}

Now do a Delete Request from e.g. postman on this timeframe on the same api like this: https://onboardtest2cf.flp-iot-sap.cfapps.eu10.hana.ondemand.com/appiot-mds/Things(‘F92E92EF251D4922B7C7C24A658A8BFA’)/iot.onboardtest2cf.greenhouse2:greenhouse4/envData4?timerange=2018-08-17T06:21:23Z-2018-08-17T06:21:24Z

And voila – the outlier is gone. Both in the raw data and in the Aggregates on an hourly basis (see third image).

You might still have another outlier of smaller magnitude as shown above – but now you know what to to. And if you have a lot of outliers/errors, that follow a pattern you can of course also automate this in Node or Java.