PowerForum Solar Data Collection - using a Raspberry Pi 3 :-)

[Guest]

JDP, no idea what a MQTT is, but, reading live data .... that I understand without Google.

[___]

6 minutes ago, The Terrible Triplett said:

JDP, no idea what a MQTT is, but, reading live data .... that I understand without Google.

It's a messaging bus with pubsub (publish/subscribe). Unlike modbus (which is query/response). Network comms is a deep subject and there is all sorts of guarantees that upper layers might want, such as guaranteed delivery, guaranteed once and only once delivery, guaranteed and in order delivery, etc. If you know how things go on the wire... well, nothing is guaranteed, multiple routes, things can get duplicated, arrive out of order... etc. So there's all these protocols out there (TCP being a LARGE part of it) that ensure that applications don't have to worry about it.

Now MQTT (and other similar brokers) adds another layer. It allows an application to "go to sleep" and essentially say: Wake me up when something interesting happens... here is a list of what I consider interesting.

And so, what my software does, is publish things on "topics" of the form batterybank/equipment/unit, eg battery1/mppt/voltage (which is the voltage as seen by the mppt on battery1), and then you can write your own stuff to connect to the same mqtt server, and what it would do is tell mqtt that "I am interested in +/+/voltage (which is a wildcard for any battery, and unit, but specifically the voltage metric), that is it "subscribes" to that topic(s)... and then all voltage values "published" by other items are magically delivered to you.

So this is what jdp means. With a very small bit of code, you can subscribe to exactly the topic you want, ignore all the other complexities of the system, and do whatever you want.

[Guest]

To all the followers of this thread, even the ones who thought I was getting nowhere fast, 

It is with great excitement that I can now also announce that the plan I had for reading data using a Rpi3b that has Windows 10 IoT, has become a reality. 

Morningstar being the first device to be read, for 'she' is a b__ch who did not want to give anything unless she was wooed and buttered up, is now uploading raw data in real time.

Data is stored in a Microsoft SQL server hosted at a Hetzner server farm somewhere on the www. 
Ja ja ja ja ... doing it the more difficult way compared to a local DB, makes for more opportunities like you can also use your own local DB if you want. 

And once all is said and done, this appie will be available as Open Source to forum members only, to not only run on a RPi using 64bit Win10 IoT, but also on 32/64bit Win7/8/10 Laptop/tablet/Pc.

Data I want to read is from all Voltronic devices, Victron inverters / charge controllers as well as the infamous Morningstar range.

The front end where the data will be made useful, view-able on a web page / cell phone, is a totally separate project.

[Guest]

So TTT if you host you DB at Hetzner. What happens if your internet link fails?

[viper_za]

1 minute ago, jdp said:

So TTT if you host you DB at Hetzner. What happens if your internet link fails?

Had the exact same thought

[Guest]

You can obviously build your system the way you like. Cloud services are a secondary service for me with local db as main source. I come from a automation background and make sure I have redundent systems in place. Hosting the main db is not a good design.  

[Guest]

1 minute ago, jdp said:

So TTT if you host you DB at Hetzner. What happens if your internet link fails?

That is not a problem at all. Sorted already. 

We also thought of what IF ...
The reading device has an issue?
The connection/s are lost reading the devices?

Will sort all that in due course.

[___]

1 minute ago, jdp said:

You can obviously build your system the way you like. Cloud services are a secondary service for me with local db as main source. I come from a automation background and make sure I have redundent systems in place. Hosting the main db is not a good design.  

I think it depends on what you want to do. In my system, one design criteria is to NOT write too much things to the file system. It's not that I don't want to write, I just want to minimize what I write. As a result, my MQTT is set up not to persist to disk. I write the csv logging module, but only because TTT asked for it. Ideally it should log to a usb or network disk. :-)

What I will probably do, if it gets there, is keep stats in memory and persist every 5 minutes or so, as well as ship off to a cloud service.

[Guest]

1 minute ago, jdp said:

Hosting the main db is not a good design.  

You must tell that to Office365, Google Docs, SageOne and a HOST of other SaaS people, not forgetting all the hype about big data on the web ...

 

 

[___]

Just now, The Terrible Triplett said:

You must tell that to Office365, Google Docs, SageOne and a HOST of other SaaS people, not forgetting all the hype about big data on the web ...

 

 

The point is that you don't want to lose data if the internet connection is down. Hosting the DB remotely is fine I would say, as long as you have a local cache for use during downtime. 'tis what the CCGX does as well.

[Guest]

1 minute ago, plonkster said:

The point is that you don't want to lose data if the internet connection is down. Hosting the DB remotely is fine I would say, as long as you have a local cache for use during downtime. 'tis what the CCGX does as well.

Ditto. 

Easy as Pi.

[Guest]

And once data is confirmed as being uploaded, delete the local cache ... why waste a memory card. 

[___]

Just now, plonkster said:

Hosting the DB remotely is fine I would say

... but I would ask why it has to be MS-SQL. I know Hetzner asks extra money for hosting windows machines. And I've had enough "fun" in the past working with MS-SQL and sqlalchemy. It's better than many other databases... but why oh why pay someone for something that is no longer on par brain surgery ... use Postgresql man! :-)

[Guest]

We are already hosting data for brokers, so all the resources are in place.

We looked at MySQL and all the other free DB's but in the end we admitted we can do things 10 times faster on windows platform, as that is our skillset.

As JDP I was deadset against hosting stuff, till I went hosted with company accounts, client databases and all that. I am now converted, BUT, as I said, IF someone wants to host their data locally, by all means be our guest, just change the SQL name, username and password to your own locally hosted SQL Express. It is free. 

[Guest]

TTT I am not against hosting. I just know it is not reliable. The controller I am building now is out in die field. Sometimes the long range wifi link wil fail for up to 3 hours at a time. Microsoft 365 is not for automation systems or process control. If it goes down who cares. Again you can do as you like. 

[Guest]

4 minutes ago, jdp said:

TTT I am not against hosting. I just know it is not reliable. The controller I am building now is out in die field. Sometimes the long range wifi link wil fail for up to 3 hours at a time. Microsoft 365 is not for automation systems or process control. If it goes down who cares. Again you can do as you like. 

JDP, I agree if we had to go unreliable WiFi, then store DB local and export to web.

Anything is possible.

But for 99% of the time, data is not at a off-site unstable location, so we develop for that first, then sort the unstable location out when we have to.

Go left around the tree, right, through, over or chop it off but just get there safe and sound is what I always say. 

[___]

43 minutes ago, The Terrible Triplett said:

We looked at MySQL and all the other free DB's but in the end we admitted we can do things 10 times faster on windows platform, as that is our skillset.

Yup, cobbler stick to thy last :-)

I don't like MySQL either. I know this forum runs on that... it's a very useful toy database. I think the thing that ticked me off most is that implicit data conversion that generates a warning somewhere in a log file that nobody looks at and completely breaks the app. No MySQL, an empty string is NOT equivalent to zero. If I'm being an idiot and I pass a string where you want an integer, please toss your toys and throw a large tantrum. Don't quietly assume I meant zero...

[Guest]

I just like it a lot when non-developers can adjust things themselves in Stored Procedures, to not bother the developer with niggly things.  We figured this out many years ago so that support can adjust the SP's to save their and the clients frustrations, under guidance from the developers off course.

Very raw, just the bare foundation, that I can start adjusting the SP to make sense of the data: http://ks4-idp.sungula.com/Ks4-SOLAR/Stats.html

Other devices should appear shortly.

[___]

1 minute ago, The Terrible Triplett said:

I just like it a lot when non-developers can adjust things themselves in Stored Procedures, to not bother the developer with niggly things. 

Non-developers doing stored procedures? You're having a laugh init?

Stored procedures is another reason I hate MySQL. It can do C and SQL(ie the hardest and the simplest but most useless). I don't know if they have fixed trigger support, but triggers could not halt transactions in 5.0 and 5.1.

[Guest]

6 minutes ago, plonkster said:

Non-developers doing stored procedures? You're having a laugh init?

Stored procedures is another reason I hate MySQL. It can do C and SQL(ie the hardest and the simplest but most useless). I don't know if they have fixed trigger support, but triggers could not halt transactions in 5.0 and 5.1.

Jip, we all have a basic 'misunderstanding', guided by the experts. 

Plonkster, we are using the Microsoft SQL, the later versions, so MySQL is not at play here at all. 

[___]

1 hour ago, The Terrible Triplett said:

we are using the Microsoft SQL

Yup, I know. Just adding some info on the a previous post where I "dissed" both the M- databases :-)

[Guest]

Plonkster, would this be of benefit? https://github.com/victronenergy/dbus-fronius

[Guest]

Right, after this lovely Easter break, we are now reading data from a Victron MPPT, Morningstar Tristar and a BMV monitor, storing local backups and uploading the data.

Phoenix inverter will be momentarily sorted.

For clarification:
1) You can have a local SQL Express database (it is free) if you do not want to send data to a SQL on the web.  
        SQL Express must then just be installed on a separate device, not on the Pi.
2) Program can work on any Win7/8/10 PC / Tablet, not just on a Pi using Win10 IoT.
3) It is only designed to READ and store raw data in locally stored temporary CSV files and load data into a separate SQL database.

Storing data in a DB on a SSD card is not a very good practice.

Then some refinements like:
1) Read data every X minutes - as per user setting.
2) Option to set COMM ports manually and option to name devices - AND (if possible) - option to reset the COMM ports programmatically.
      This unplugging / rebooting is 'sommer oudedoos'.
3) Setup Email / Cell Number.
4) Delete locally cached files backup files - as per user setting.
5) Send email / SMS to user IF there is a problem reading / storing / uploading data.
6) Auto update the program when new changes have been developed.
7) Access to the program, on the Pi, via i.e. Remote Desktop.

Thereafter, which will not be open source, is where we then make use of the data collected. This will need Windows IIS locally installed OR can be hosted on the web.

Then we will add the option to read data from all Voltronic devices - with help.

And that is all I have planned for the moment, read data from as many devices as we can get our hands on (open source), and then view it anywhere (not open source).

Once all is stable, i.e. all that can be done is done ito data reading an interpretation, then we can start modules to write data back to devices or to set relays on / off for whatever users may want to switch on/off remotely, or use some form of low level AI based on data.

Although, for Victron, I am not keen to send any data back to inverters to adjust them, they have propriety software for that, which is supposed to be done once and then left well alone. But time will tell.

[___]

3 hours ago, The Terrible Triplett said:

Although, for Victron, I am not keen to send any data back to inverters to adjust them, they have propriety software for that, which is supposed to be done once and then left well alone. But time will tell.

The one value I adjust is the power-assist value, but it is very easy to do this one. Other stuff can be adjusted as well, it's all documented, you just have to know where it is. It requires calling CommandReadRAMVar and CommandWriteVar which I have so far not implemented. No need to... see. Of course, this is "break it and you get to keep both halves" territory.

On their live site they have documentation on how to update values for the hub-4 assistant, which reveals some info on the assistants (length byte followed by id and parameters, starting at id 128). In theory it should be easy enough to go poking in there and figure out what does what... but I have to tell you, I feel much safer using the inputs on the inverter to do this. My inverter, sadly, has only one input (temperature), but the quatros and the non-compact multis have more.

Because I don't use the charger in my multi (except in extreme circumstances), the temperature input is free. I use it to force it to stay on the grid using a push-button switch. You can also wire a BMV to this input so the BMV controls whether it goes to grid or not.

Disabling and enabling the charger is one setting that can be done without restarting the inverter. Changes to most of the assistant values require a restart however. Adding a call that does this (enable/disable charger) sounds like one useful idea, but I honestly don't see any use for messing with the other stuff. Much better to use the official software, even if it means you have to run windows for a few minutes :-P

[Guest]

8 hours ago, plonkster said:

... , but I honestly don't see any use for messing with the other stuff. Much better to use the official software, ...

Jip.

But altering the Morningstar, BMV and MPPT's, its not an issue ... later.