PowerForum Solar Data Collection - using a Raspberry Pi 3 :-)

[Guest]

Lots of people here and out there want to read and store data generate from their inverters, charge controllers and batteries, yet there are not one software package out there that fits all needs. Yes, manufacturers have devices that can read, store and upload their device data, but at what cost, and if you mix and match components, it becomes rather extremely expensive to record all, separately, per manufacturers device.

 

There are a lot of clever people here and out there that if we where to work together, as a collective, can make such a system a reality.

 

Harvesting the combined knowledge and requirements right here, we can as a collective, make this happen, to cover as many devices as we need to.

 

Off the top of my head, to collaborate:

a) You are good at technical development, have the equipment, develop some code and share it here - Edmond, JDP, Plonkster where are U? :-) 

b ) Okay, you cannot develop so add some funds to pay for some parts of the costs that are going to be incurred - crowd funding.

c) So you are good at organising, we need that very badly, bring it and start.

d) You on the other hand are good at international website development and web marketing ... lets do it.

e) And the rest of us, we have no clue bar that we want and need the software ... we are the TESTERS, the run of the mill user and the MOST important component of all, I sh_t you not!

 

Let the games begin.

[Guest]

The goal:

Develop software using HTML5, and a SQL database, to view stats and reports using a web browser on a Pc and / or Cellphone.

Then go further and send anomaly warnings, based on historical data recorded, as well as reminders to i.e. check battery water levels, sync BMV monitor etc.

 

Rapsberry Pi2B is proposed for it uses next to no power to run 24/7/365 to collect and upload data from devices via USB ports to a SQL server on the web, so that when you are not at home you can still access your systems information using your phone and 10 to 1 switch things on / off based on your data.

 

The plan is simple:

Step 1: Develop software that can be loaded on either a Raspberry Pi2B or any Windows computer, to read data from devices.

Step 2: Upload the data to a SQL Server on the web, linked per user's setup.

Step 3: Write a front end using HTML5 that the reports and graphs are viewable on either a PC, using a web browser, or a cell phone.

Step 4: Develop a system to send warnings and reminders to users phone / email account.

Step 5: Expand software to send data back to equipment to switch inverter off / on, etc.

Step 6: Use the features on a Pi2B to switch circuits on / off remotely from your Cell phone / web browser.

 

Hardware and Operating System requirements:

1) Raspberry Pi2B - for the power conscious users - has 4 USB ports, Ethernet and HDMI screen. 

            See here: https://www.pishop.co.za/store/raspberry-pi-boards/raspberry-pi-2---model-b

2) Laptop / Pc for those who prefer it.

3) Operating systems:

    1) Start with Win10 IoT for the Pi2B as well as any windows computer starting from Windows 7 upwards.

    2) Then alter data sending software to run on any other OS like Raspbian, as required.

4) Data is uploaded to the web via users home Ethernet or WiFi, so internet connection is required.

5) And off course the cables to connect your device to the Pi2B or Pc / laptop you want to read data from.

 

Devices:

1) Start with Axpert inverters.

2) Then all Victron inverters, BMV600-702 and charge controllers.

3) Followed by Morningstar Tristar charge controller/s.

4) Any other devices that users need to read data from.

[Guest]

 

Plonky is not a .NET/C#/Windows sort of person. I suppose if it came down to a choice between the next meal, and if the local street corner is already taken, I might change my mind :-)

 

I'll participate, but I cannot help develop this product :-)

 

I give all my code away for free on github, and you are absolutely welcome to "steal" from it. The only thing of interest on there right now is something called ib.victron, which is in python.

 

 

To read the data from the inverter is easy on the Pi2B . Controlling the inverter is something else but its doable.

I will try and see if i can switch the inverter over the network . This should not be difficult. I have some ideas.

 

 

The way I see this going is that each type of inverter will have a control module that talks a common protocol. Then you have a central piece of software that collates all the data. So the control module then does the translation between what the inverter can do and what the software wants to see. The backed can be on a lamp stack using linux / python with a HTML5 and jQuery front end.

 

[Guest]

I have found a usb library for the Raspberry Pi that seems to work. Lets hope we can communicate to the inverter. 

[Guest]

Confirmed: Software developed to run on Win10 IoT will work as is on any Win7 or later OS.

 

So people who do not want to buy a Pi2B, will be able to use a laptop / Pc.

[___]

I'll do what I can. I have a dislike of windows that borders on a religious conviction though... I'm that age where I lived through most of their sh*t, the bullying of Digital Research, the browser wars, the anti-competitive behaviour, locking in customers and data. I've managed to transition from outright hate to strong indifference though...

[Guest]

Plonkster, baie dankie!

 

I am 100% sure that once Windows works, that the software will be made to run on any OS the users of Pi2B prefers.

[Guest]

I will try my best but think Plonkster is the pro in Python.   

[Guest]

How many people want to use the Pi to control the inverter ?

[Guest]

And how many people be interested in this project?

[Guest]

I personal thing that the Raspberry Pi is the way to go. It have a 40pin extended GPIO header witch we can use to start and stop devices. Have a total control to home automation. For people that have a big solar system to benefit as they can put more load on the inverter by using the Pi to start a device automatic. The device is small , cheap and can do a lot

[___]

I have a Pi2 as well. I have access to cool hardware because we use it at work too (the adafruit capacitive touch screen!). We run Raspbian on it. I have a kernel source tree and cross-compile environment going as well.

 

I think the Rpi is a good starting point. It has two SPI ports, so some expansion is possible (if you use the touch screen, it takes up one). There's two possible issues with it.

 

The first is that it is relatively power hungry compared to other solutions.

 

The second is that I'm not sure I trust even Linux to run critical things related to my batteries.

 

But on the flip side, 5W isn't too bad, and the Rpi is more accessible to the hobbyist, which are great. So I will admit that my two objections are those of a purist :-)

[Guest]

The first is that it is relatively power hungry compared to other solutions.

The second is that I'm not sure I trust even Linux to run critical things related to my batteries.

5w compared to my current 35w, it is a bargain, but yes, to have a very affordable, easily obtainable device that can connect to any screen, kbd and mouse out of the box, I liked it a lot.

 

And I agree, I do not trust any such a device to do anything important, but, if it is going to be just a gateway for me to get to things to switch off/on, read and upload data, nothing critical, I think it is as safe as anything, at 5w or less.

[Guest]

I been testing it now for more than a year and did not found one single problem with it. I use it to control some light from my cell when i am away and also did build a timer that i can update over the internet.  

[Guest]

It will take +-1 hour write a appie to read all 4 my devices, automatically detect which device is sending data via which port, and store the data in 4 separate CSV files - OS independent.

 

Then another +-2 hours to write appie to upload data stored temporarily on device to SQL database, then delete the files once successfully uploaded - in case of internet problems and data cannot be uploaded - this software is also OS independent.

 

Easy as Pi. :-) 

 

Versus 3 days to write a proper app to read any device where user maps each com port and data being received - maybe later.

[___]

I'm not actually a fan of using a relational database for this. Sometimes there's a reason to do it, especially if you want high-detail that you keep in all perpetuity. In this case I'm more in favour of something like rrd (round robin database) or whisper (a python database with similar features). The idea is that you keep data for the last week or so at high detail levels, but aggregate as the data gets older. Much better to use an online service for the more important reports.

 

There is another reason for this: Flash sdcards have limited write cycles.

 

Regarding trust in the device: There are ways to deal with that. For example, Victron's new hub-4 setup has an implicit watchdog: If you don't communicate with it once every ten seconds, it switches to bypass mode. If I'm building such a thing with a pi, I'd probably want to include a similar watchdog. A dead man's switch, if it's not regularly serviced in some manner, it will throw the inverter into bypass mode and keep the batteries charged.

 

If I had my way with this, I'd use an ARM Cortex board for this. The only obstacle I foresee at the moment: It's going to mean writing in C rather than something more high level, and secondly, though my resources tell me you CAN run an STM32F4 board as a USB host, I suspect it is going to be much harder to do this than with the Rpi. Why run a USB host? Well, because the Victron MK2 is USB :-)

 

So it's going to be an Rpi, but I think there's going to be an Atmel chip in the mix as well. It's not going to be pure Rpi. Definitely in my case, as I need to implement canbus connectivity for my setup.

[Sidewinder]

Hi all,

 

I'm following this tread. Having also just started dabbling with Arduino, but also have a soft sport for any Pi.

[Guest]

Appears the Morningstar products need a wee bit more time to get them to relinquish their data. Something to do with having to ask them nicely for the data.

 

Victron inverter took a whopping 5 minutes to figure out.

BMV and Victron 75/15 MPPT controller was 2 seconds.

 

Next is going to be the designing of the DB, for wow, the 4 devices I have has a lot of info to share.

 

The software.

Easiest option is to have software running on Linux / Windows to read data and store data temporarily in CsV format on the device.

This software is going to keep being added to, as new devices are added, like Axpert.

 

Then a 2nd small piece of software, that reads the data into SQL that once successful, deletes the CSV file. This is a backup in case data upload is interrupted. This package is developed once, for it just has to upload data, then delete it, to a specific place.

 

Then the front end once data is stored in the DB.

[Guest]

Are someone getting any readings of the inverter using the raspberry pi

Sent from my SM-G920F using Tapatalk

[Guest]

Soon.

[___]

You know, the only way I think of where you can interoperate "seamlessly" would be to do something similar to what is done on the CCGX, to use a system wide messaging bus. The CCGX uses dbus, which has become the defacto standard for inter process communication. So it doesn't have to be dbus, I would much prefer something simpler. Mqtt comes to mind (it's designed for this sort of thing), but the best mqtt broker is written in Erlang!

 

But if you have such a bus: Then any software component merely has to connect to the relevant MPPT/Inverter on the one end, get the data, format it correctly, and put it on the bus. Other applications will subscribe to data of interest and do their thing with it.

 

Then you can write your software in any language, as long as it knows how to talk to the chosen comms broker. You could even write it in C#, it should run on Linux using mono (though in my experience this limits you to older APIs, I think up to 3.5 is fairly well supported, and you would need to avoid P/Invoke calls to native platform code (which is going to be hard, because I don't think you can talk to a serial port without doing that :-)
 

For the front end, I was really thinking using angular. Because that gives you an interactive and responsive (tm!) interface in a web browser, which means the whole screen size issue goes away, tablets are immediately supported, and without too much effort you can scale it to a mobile phone.

 

For comms, I would really push json. Not XML. The next person who says XML gets a virtual smack up the head. XML is so 1999, and a complete PITA to work with :-)

 

Edit: dbus is the de facto standard on LINUX... just to be clear. :-)

[Guest]

Yes, we have developed a comparative quick quote app for insurance quotes (premiums obtained from insurers using their required XML's) for the web.

So HTML5 and JSON is easy, and the app works seamlessly on any browser, either on a Cell or PC auto sizing as required, touch screen comfortable.

 

Easy as Pi. :-)

 

If I understand you Plonkster, each device to their own. So we thought that once we have all the devices we need for now, we are thinking of a app to search all USB / Serial ports automatically and once it recognizes the incoming data, start writing the CSV for that file. That is similar to your bus you describe?

 

I tell you, seeing the differences connecting to the Morningstar and Victron inverter and Victron inverter and the same brand MPPT / BMV, each one has to be developed individually. Axpert would probably be much easier than the i.e. the Morningstar connection.

[___]

Aaah okay I see what you're thinking. You're going to write to CSV file, and then pull the CSV into the next process in the pipeline. I suppose that will also work.

 

My idea is simply to have a sort of standard protocol with a pubsub hub in the middle (pubsub -- publish/subscribe), instead of using files.

 

So then you develop:

 

- one application that talks Ve-direct to victron mppts

- one application that talks canbus to the high end mppts

- one application that talks to a morningstar

- one application for a BMV

- one application for the Victron inverter

- one for the axpert.

 

They run as separate processes, you can write it in any language you like, and all it has to do is know how to talk to the hardware (usb, serial, whatever) and then publish that data into the hub. You run whatever ones you actually own the hardware of. Adding support for a new one means writing one small app.

 

In the case of the Axpert app, it publishes messages of both incoming charge and outgoing usage. With the others, an application might restrict itself to only doing one or the other.

 

Then you write:

- one visual component that subscribes to the above messages and visually displays them.

- one background service/application that subscribes to messages of interest, does a calculation or two, sends to a service on the web and/or do local storage.

 

Well, that is the idea. I think it's best really to scratch your personal itches first. IF you want the data in a CSV so you can manipulate it in excel and/or upload to pvoutput, I absolutely encourage you to do that first :-)

[___]

For example, here is something I hacked together on a Saturday afternoon, using angularjs, JustGage, and the angular wrappers someone made for JustGage.

 

This does nothing yet, but because of the bi-directional data-bindings in angular, I could make this talk json to a small http server that sits on the hub... and voila. It works on PC, laptop, tablet, phone...

[Guest]

Like the interface

Sent from my SM-G920F using Tapatalk