YES!!! Edmund I bet will be interested, I hope JDP will also bite to read data from a Pi.

Plonkster if you can give me data in CSV files read from my Victron equipment from a Pi ... gee ek jou 'n oopbeksoen! 

Okay, maybe not but you get the idea.  I have sent you a PM on a Trojan matter, I will stand by that.

Idea is the Pi keeps the CSV data, and once uploaded successfully, the CSV file is cleared - I leave that to a developer to figure out how, for the upload must take place say every 5 minutes, recording set per minute.

Idea is that with CSV files:
- pvoutput.org as you say;
- makes the Pi usable everywhere for non-developers to use the data in i.e. Excel;
- no development required to connect to a MQTT bus for a non-developer out there;
- And in my mind, most importantly, it serves as backup when the internet is off and the data cannot be uploaded at the time.

And maybe with Edmund and / or JDP, one can record the entire night on the Pi, and upload the next morning to their software?

42 minutes ago, plonkster said:

So your long play is actually not a particularly difficult problem :-)

You have NO idea how easy it already is for us ... and we use no freeware, that will be against the freeware rules. 

We just CANNOT even THINK of doing now or in the foreseeable future.

51 minutes ago, The Terrible Triplett said:

and we use no freeware, that will be against the freeware rules.

There is a difference between Freeware and Open Source :-) Freeware is literally with no restrictions, there are no rules. Open Source has restrictions. It isn't even necessarily free. All it says is that IF you distribute the software, you must provide the source, or a means to download the source. Most licenses then also include a viral clause that ensures it remains free, or to prevent some opportunistic commercial software giant from making money off your hard work.

But I'll assume that is what you meant :-)

Jip, I meant freeware, mistyped the 'rules' for Opensource is not an option in our main business, as we charge for the use of our system.

So not much to report here after the weekend. The vedirect module is now integrated into the stack, so you simply list it as a pyramid include in the .ini file, and it starts up in a separate thread. Works well so far. At some point we'll have to test it with a real ve.direct mppt, not with the one I'm emulating from ve.can :-)

I also changed the mqtt packet format a little. In the earlier versions it only sent the actual measured value. That's been changed, it now sends a timestamp (unix timestamp, that is seconds since 1970-01-01) and a value. I briefly considered using json to encode the packet, but the overhead becomes a little insane. 80% of the packet becomes just framing and metadata. So now it's just a timestamp and a value, separated by a space.

The timestamp is somewhat of an omission in mqtt. Because data reception might be delayed, it's actually somewhat important to know when a sample was taken.

I also created an event-based framework inside my code, so that extensions can hook into a parsed data stream rather than all doing their own parsing from the mqtt stream. This bit will by python specific. It's only there so I can easily implement things that will log to csv and pvoutput. These extensions now become no more than a plugin eventhandler. I really do love separation of concern type things :-)

TL;DR: I'm one step closer to providing pvoutput.org and csv output.

Plonkster, and other followers, it is with great joy that I can announce that SolarMon is very interested in this Pi project.

I quote Edmund: "... In fact the more I think about it the more it makes sense to me. I am good at making something logical, readable and appealing. Not so much into the heavy technical stuff. So to make SolarMon the “reader” of data (I do not care from what and where) sounds good."

So, I hope we can pull this off. Plonkster on Victron, I will sort the Morningstar, and if there are any others, PLEASE let us know what you can add ito hardware to be read.

For SolarMon to read the data from a Windows Pc ... I can take care of that for the idea is NOT that you HAVE to get a Pi.

Note: This project is for use by anyone out there, who wants to READ data from devices. That part is freeware. Both Plonkster and I are in agreement on that one.

As a matter of fact, with Plonksters consent, we can dedicate this to PowerForum for giving us such a awesome platform to use and come up with these ideas.

Nice guys... I like the direction this is going.  As a non-techy I really see huge benefits for all!

 

Ditto Mark, I am also a non-techy.

 

I said here that I want to measure the income VOLTS and AMPS from the charge controller, right(?), to be able to swap the SOC on the BMV.

I use the SOC because it is less prone to fluctuations.

Now, using the SOC, set to say 80% to go back to Eskom, and 81% to go back to inverter, does cause a wee bit of attention.

So, looking at the incoming VOLTS and AMPS from the charge controller, if I can set the BMV's SOC lower or higher, that solved my problem.

Low and be hold, I found this (for the non-techies) from emailing Victron! It is going to be possible to do it with software:

https://www.victronenergy.com/blog/2016/03/04/victronconnect-v1-2-released-now-includes-mac-os-x-windows-pc/

With Victron Connect and the cable, you can now, till the dream above has been developed, do it manually from your device, even a WINDOWS 10 64bit PC! 

One less problem to figure out ... this is going to be fun!!! 

Let me rephrase, having used volts and now SOC to swap:

In my case (24v) system, at +-27v with +-27amps incoming, read from the charge controller compared to the inverter load of say 400watts, if I can set the BMV's SOC lower automatically, will result in earlier swapping back to solar having the whole day left to charge the batteries to 100%, even at 5% rate.

BUT, if after an hour, system sees the incoming AMPS are too low for the load AND the batteries, then increase the SOC back to say 95%, resulting in the system swapping back to Eskom immediately, leaving all the solar power available to charge the batts faster.

The AMPs incoming is what I want to use to swap backward and forwards, based on load.

If there is no input from panels, revert to preset SOC settings to protect the batteries during the night.

Does it make sense?

Just downloaded... installed... and wallaaa... very nice little interface - all settings easily accessible.

Now... do they have a nice specification for developers..??? (ahh Attached)

VE.Direct-Protocol.pdf

3 minutes ago, Mark said:

Now... do they have a nice specification for developers..???

Waiting for their reply. 

deleted

Sorry, I may not have been clear. I asked them for freeware, if they could give it to us, to not write it from scratch. 

And, correct me if I am wrong, but does that doc not just give the specs read?

I do not see the parts to alter the BMV's settings, or am I missing it?

1 hour ago, The Terrible Triplett said:

That part is freeware. Both Plonkster and I are in agreement on that one.

I will insist that it's going to be open source and not freeware. I personally license all my stuff under the GPL version 2 (which is a little less invasive than version 3), but I will also do LGPL, which allows for commercial use. The only reason for a more restrictive license is that we don't want someone taking our hard work, making a commercial product out of it, and not contribute back.

Other than that, the parts I wrote so far can all run standalone. The only two hard requirements are 1) python and 2) mqtt, and out of those two, for people who want to contribute other modules, the only non-negotiable one is mqtt. Given that this is a pretty common IoT part, I think that's pretty open :-)

Plonkster, adjusted my post to say open source . I see your logic.

And I buy into mqtt and Python, provided there are CSV files accessible by a Windows OS, otherwise how is SolarMon going to read the data? 

2 hours ago, The Terrible Triplett said:

Plonkster, adjusted my post to say open source . I see your logic.

And I buy into mqtt and Python, provided there are CSV files accessible by a Windows OS, otherwise how is SolarMon going to read the data? 

I suspect some tools will want live data, and I don't think CSV is the right medium for that. Those apps can connect directly to mqtt. For the rest, there will be CSV.

I'm a little concerned about using the flash card in the Rpi for CSV logging. Some cards are better quality than others. I've had corruption on oft-written files. We have a couple hundred Rpi units in the field, and two of them managed to corrupt the dhcp cache file (where it stores the lease, so it can attempt to renew it) with the result that their network comms broke completely. On the other hand, our app does quite a bit of logging, and the log files never seem to get corrupted. It might be better to log to a USB-flash disk, but with that comes all the complexities of auto-mounting it etc. I suppose I'll let the user worry about that :-)

Jip, i have heard from my developers the concern about the flash cards.

But, we have the same problem if the web link is down, not?

We have a csv logging module.

It's really crude, at the moment it literally just logs all data that's received from all modules. It is a raw log. There is no filter, no aggregation. It is literally the simplest thing that will work, meant for later refining.

It's documented in README.md, though it's not yet ready for prime time, and it's probably better to wait until it is, at which time I'll make a downloadable image again.

This brings me to a new question: What kind of data are you looking for? I think one obvious one would be to filter for particular values from a particular unit -- for example power figures from the mppt controller -- and log only those. For that a filter is required. You're still going to end up with a reading once a second... that is 86400 readings in a day! I'm just wondering if some kind of aggregating function might also be required, for example, pvoutput.org expects a daily kwh total.

I'm also working on another module that will log to a whisper database. This is a fixed-size round-robin database with propagation, and will be more suitable for aggregated data. In other words: It might make more sense to dump a csv out of the whisper database. It all depends on what is needed.

I'm thinking another very cool logging module would be one that logs directly to graphite. Also, modules that log to pvoutput.org and maybe some others. I'll see where it goes :-)

Re. the CSV files, my thinking was that each device has its own CSV file, named per device name and model i.e. Victron_MPPT_75_15-1.CSV

Data I am looking for? Every little bit of juicy data the device can give ... Want it ALL. 

Reading the CSV files, ja, 86400 records not cool. Was thinking to pull date every 5 minutes, delete the orignal file ONLY when the data has been successfully received in SQL. 

So maybe each file has a start and end record, to indicate the end of the data in CSV file?

So you have:
Victron_MPPT_75_15-1.CSV
Victron_MPPT_75_15-2.CSV
Victron_MPPT_75_15-3.CSV
Victron_MPPT_75_15-4.CSV
etc.

BUT BUT BUT ...

Let me ask you 2 questions:
Plonkster, what database would you prefer to use, that makes you smile all the way?
Can it be accessed from say a Windows device, to suck it dry by pulling all its juicy information into say a Microsoft SQL database?

On 2016/03/08 at 11:54 PM, The Terrible Triplett said:

Plonkster, what database would you prefer to use, that makes you smile all the way?

Never did answer this question. Depends on the task at hand. I like rrdtool for what it is good at, which is mostly network and systems monitoring, but there is some overlap. It keeps a detailed log of what happened today, but it rolls up the stats for day, week, month and year, depending on how you configure it. Solar production seems similar to me. But RRD is a pain in the proverbial, which is why I'm looking at whisper, which is the backend behind carbon-cache, which is the engine behind graphite. But forget about that for a moment, I don't think it answers the question.

Next, for single threaded stuff, I much prefer sqlite. It even has windows and .NET libraries.

But my absolute favourite, the king of the mountain, remains Postgresql. My reasons, in no particular order:

1. Mostly ANSI compliant (much more so than MySQL or MSSQL).

2. Fast. Faster than MySQL when you use it on a system with multiple cores.

3. Relational integrity. MySQL is only "fast" if you use MyISAM, and MyISAM has no integrity.

4. I hate MySQL's idiotic way of quoting things, and the broken triggers.

5. Procedural language support. MySQL (and many of the others) pretty much support C/C++ or SQL. Postgresql supports more than I can list on one hand. Generally you want to stay away from this for portability, but I've used it sparingly where I needed speed and it is a godsend.

6. Binary replication.

But... Postgresql on a Rpi... we've done that. Not a good idea on flash. Not sure it's a good fit for the kind of data. I've used Postgresql for this kind of thing where it was required. A former client's system (which I designed and built) has several hundred gigabyte of telemetry data in a postgresql database. We indexed that thing so severely that the indexes ended up being larger than the data. It works! But I will not do this kind of thing on flash.

Well PostgreSQL is a Oracle clone ;-). I love it as well. Been using it for years. I have been using Sqlite3 for most of my IOT projects. Works well up to this point.

43 minutes ago, jdp said:

Well PostgreSQL is a Oracle clone

I know people who will disagree with you :-) It depends what you mean by clone. There is of course no "common ancestor", because Postgresql evolved out of Ingres. Due to the ANSI compliance that was part right from the beginning though, it makes complete sense why the distance between pgsql and oracle (and I would expect in the early days a similar closeness to ingress) was never very large. Then, when they added their stored procedure language (pl/pgsql), that was certainly modeled on the oracle language.

I don't consider written from scratch using similar ideas to be a clone. When the Chinese take a piece of hardware and blindly copy the firmware to make cheaper knock-offs... THAT I consider a clone :-)

I can run PL/Sql code inside PostgeSQL and and you want to have it supported and move your Oracle code to PostgeSQL you use enterpriseDB

http://www.enterprisedb.com/?utm_source=adwords&utm_medium=cpc&utm_campaign=adwords-cpc&gclid=CjwKEAiA9om3BRDpzvihsdGnhTwSJAAkSewL12tJ8O5WHoFgWNTwiZ1sGDBtBNCAN4Y5xdCVni6x3hoC0RXw_wcB

That is why I say Oracle clone as it can run Pl/SQL. I am an Oracle OCP and love Oracle but Postgesql is very handy for SMEs

Man, this took way too long. So I made a new image today. I think this one has the Linux base sorted out. There was an irritating delay (several minutes!) in the boot up process, which I eventually traced to my ethernet config: It was set to start automatically, so it would sit there waiting on DHCP when no ethernet was plugged in. Removed that, then configured ifplugd. So now that's sorted and it boots very quickly.

I've made more changes since I made the image, but as always, updating the code is as simple as a "git pull".

The image is here: https://drive.google.com/file/d/0By8zt_cN5USfQmxlVGF6dWJCb2s/view?usp=sharing

So what works now. Well, the CSV module finally works. It can't do amp-hour or watt-hour yet, still have to think if and how I want to do that. For now, you include the module in pyramid.includes and configure it as per the examples in the ini file. It can do min, max, average, sum, or just the latest value over any time window.

I've been running this for a few days now and it works very well. The only thing I find myself wondering is 1) what state is the charger in and 2) what's the current voltage. So I'm thinking the next step is to improve the web interface a bit.