mc-Products => mc-Gateway => Topic started by: millennial on January 29, 2017, 04:39:19 pm

Title: Programming the mc-Gateway
Post by: millennial on January 29, 2017, 04:39:19 pm
I know it is not possible to program the mc-Gateway, but I have a couple of problems that could cheaply, elegantly, and easily be solved if we could program the mc-Gateway, even in an extremely limited manner. I say the problem could be solved cheaply and elegantly because an alternative solution to the following problems would be to host an MQTT server on a Raspberry Pi and bridge some of (and perhaps compute and re-package) the messages to Losant.

Losant has a payload limit, and other services may also have similar types of message limits. If we could tell the mc-Gateway not to send the beacon topics over MQTT, this would allow us to keep the beacon interval high (10 seconds) and avoid accruing payloads on Losant. A general whitelist/blacklist configuration for MQTT topics on the mc-Gateway could solve this problem.

Another problem, specific to Losant, is that if we use beacons for sensor readings (ideal for saving precious battery life) we need to use the Losant workflow to receive the MQTT event and then set device state based on a computation to retrieve the user data in the beacon bytes. The act of receiving MQTT in a workflow, and then using the workflow to set device state, both count towards the Losant payload limit. If we could define beacon user data mapping, perform simple calculations, and send results out on a renamed MQTT topic (in the case of Losant, a "set state" topic pattern), we could cut payloads in half on Losant. Yes, this problem can be solved by using MQTT directly on the modules, but this compromises battery life.
Title: Re: Programming the mc-Gateway
Post by: Nick_W on January 29, 2017, 06:08:16 pm
Or you could use my program, as this is exactly what it does.
Title: Re: Programming the mc-Gateway
Post by: millennial on February 03, 2017, 09:19:11 am
If I could run a Python program on the mc-Gateway, I would gladly use  ;)
Title: Re: Programming the mc-Gateway
Post by: Nick_W on February 03, 2017, 09:35:57 am
But you could run it on almost anything else, like a Pi. Do you not have anything running an MQTT broker?

Best solution would be to have small computer (say a Pi) running an MQTT broker locally, plus other programs (like beacon_decode and my logging program). This receives all the data/beacons etc, then you push from there to your cloud service/Losant/whatever. Then you are in control of exactly what is sent and when. And you can watch it all in real time to see what is happening.

You can run python on almost anything, a Pi zero ($5), or the PC you are using to program your modules.

Best bet is a RPi 3, with built in WiFi, plug it in, and leave it running 24/7 with mosquito MQTT broker running and whatever python programs you like. With SD card, you can set this up for $50 or less (if you don't need WiFi, and can use the built in Ethernet, you can use a Pi2 or cheaper).

Seriously, a Pi zero works, size of 3 modules, with a $10 wifi dongle and SD card, you can set up a whole server for about the cost of 1 module.

I doubt MCThings will implement programming of a $95 gateway, when you can set up a general purpose server for $20.

Just saying...
Title: Re: Programming the mc-Gateway
Post by: millennial on February 03, 2017, 06:00:21 pm
I could easily set up a Pi with Mosquitto, but that is one extra piece to the puzzle. Sure it doesn't matter if I use a Pi in my home, but people would prefer an IoT product that has 1 less piece. Also, selling 1,000-10,000 counts of a product to separate sites would mean $5,000-$50,000 of Pi Zero's, which is expensive and 1 more piece of depreciating capital to maintain.

A whitelist/blacklist configuration for the mc-Gateway that tells it what MQTT topics to actually send out to the internet would be a good feature. Though, full-fleged programming would be nice on the mc-Gateway.
Title: Re: Programming the mc-Gateway
Post by: Nick_W on February 05, 2017, 12:39:33 pm
Well if your deploying 1,000 to 10,000 units, you only need one server, just set up AWS and/or EC2, both support MQTT, and process all the messages at one central location. No hardware/infrastructure required.

It's an extra config step, but then anything you do will be. You could set up a web front end, so that users can config what gets sent to Losant, and what doesn't. Plus it would be optional, you don't have to use the middleware if you don't want.

Just saying there are other solutions.
Title: Re: Programming the mc-Gateway
Post by: millennial on February 05, 2017, 01:58:21 pm
I like that solution. Ultimately, yeah a solution shouldn't depend so heavily on Losant, so we would have a "proxy" where everything would go first when considering something at that scale. I still am rooting for mcThings to be the most robust IoT hardware solution out there, and more advanced gateway configuration/programming would be part of a robust solution.
Title: Re: Programming the mc-Gateway
Post by: millennial on February 05, 2017, 02:00:11 pm
Boiling it down to a single principle, I don't like how beacons are automatically MQTT'd to the world when an MQTT configuration is set. At least have a checkbox to turn MQTT beacon broadcasting off.
Title: Re: Programming the mc-Gateway
Post by: Nick_W on February 05, 2017, 06:28:48 pm
I do agree with that, it does seem a bit blanket that beacons are published, and you can't turn it on and off, or choose the topic (from the gateway).

I have also seen odd behaviour when you have more than one gateway, ie you get the same data published twice (beacon that is), once from each gateway, and you can't turn that off.

This means for instance that the Rssi gets published twice with two different values, unless you can filter the mqtt data by gateway Id.

So you almost need some middleware that recognizes that it's already received some data from a module, and filters it out, to prevent repeat posting of the same data, yet could update a "last heard from" field, so you know how recent the data is.

If gateways could talk to each other, this would open up a lot more possibilities, but maybe we're asking too much. If the new gateway has an esp32, the programming options would open up, right now I think it's an esp8266, so it's just a single thread to do everything.
Title: Re: Programming the mc-Gateway
Post by: millennial on February 06, 2017, 10:50:41 am
I think more than one gateway publishing beacons is a feature, not odd behavior. In their documentation, they mention using beacons for device triangulation. The MQTT broker would have to recognize that the data is the same, which could be difficult because there are so few user data bytes to play with.

What I would do if I had multiple gateways is tell the MQTT broker to accept data from a module only if it came through the correct gateway, which is possible to detect from the beacon data (or even the MQTT topic).
Title: Re: Programming the mc-Gateway
Post by: mc-T2 on February 09, 2017, 09:21:56 am
Hello all!

Sorry, i meant to reply to this topic earlier this week! We will be implementing a feature in the future which will allow you to turn off or on beacon publishing from the gateway. This way you can have the option to stop the beacons from being pushed into an MQTT service for the multiple reasons that everyone has mentioned.

In the meantime, you can turn off the beacons within the programming of the modules or reduce their interval if you wish.