VRMLogger

[___]

Wanted to inform you guys, in a more top-level thread rather than 18 pages down into another thread where lots of Windows bashing and other issues happened: Victron has released some of their code packages for the Raspberry Pi. This already happened 6 months ago, but not many people know about it. Basically, they repackaged enough of their code so you can log from your BMV/MPPT to their VRM site. Obviously there is no user interface, it is intended for headless data-gathering setups.

Presently there is no mk2 support in the official packages. As of this morning, I got the mk2 bit going and even have it packaged. The source is closed however, so I only have permission to provide a binary, but if you are technically inclined and want to try it out, drop me a message and I'll point you in the right direction. I've tested it so far and I've got local logging working. I don't yet have remote logging working properly, but that is a matter of time.

This solution uses the official code that is also used on the CCGX and has the Victron blessing, except for no official support. Maybe it helps someone here.

[Guest]

Good move to start new!!!

Fantasties!!!

I like this a LOT!!! 

Plonkster, if this ends up that I can use this reader, on a Pi, the one you donated to me, and with your help, to send my Victron data to our DB, I would most definitely be interested to test it for you.

[___]

The data is available on dbus, which is an inter-process communications bus specific to Linux. It has its history in the dcop and corba predecessors from the KDE and Gnome projects. It uses unix sockets for comms (sockets that live in the file system, windows really has nothing like it), in other words it is not networked like mqtt would be. To get the data out you will have to leave windows land for a few days, come visit us here in unix land, and write enough code to scavenge the data you want and ship it off.

Yes, you can make dbus listen on ipv4, but I care about security enough not to recommend that.

In other words, this is not going to plow your fields for you without some effort of your own.

Oh, it does have a local log somewhere. I just haven't figured out how to get it yet. It also keeps a backlog in a sqlite database. So there are some cheaper options that might not involve a trip outside of redmondian hills.

[Guest]

I hear you load and clear. 

Truth be told I really do not care about Windows anywhere near this, all that I care about is that the MK2 data is read flawlessly, via a Pi, using ANY OS that is NOT related to Windows in any form or fashion.

Just one request, one!: That the data harvested ends up in a SQL DB on the web.

And if the MPPT and BMV data can join the circus as a pasella ... I will promise to never mentioning anything coming from Redmondian Hills, ever again here, on this thread, other than the vague reference to some Redmondian SQL server DB residing on the WWW, in this one post, ever again.

I promise.  

 

[___]

4 minutes ago, The Terrible Triplett said:

Just one request, one!: That the data harvested ends up in a SQL DB on the web.

You're absolutely welcome to write one. No time on my side, and no motivation either. I don't want my data in a SQL database, it's a poor choice for time-series data, imho.

[Guest]

I think we are missing each other? Maybe?

This caught my attention, and the ONLY person I know who can make this look easy to do, is you. No, I am not buttering you up.

33 minutes ago, plonkster said:

It also keeps a backlog in a sqlite database. So there are some cheaper options that might not involve a trip outside of redmondian hills.

If I can have a copy, and you can point Jacques in the right direction to said SQLlite, the rest will be history I THINK?

Unless you find a more efficient way to get the data?

[___]

6 minutes ago, The Terrible Triplett said:

If I can have a copy, and you can point Jacques in the right direction to said SQLlite, the rest will be history I THINK?

SQlite database is in /data/db/vrmlogger-backlog.sqlite3 on the Pi.

http://stackoverflow.com/questions/15292880/create-sqlite-database-and-table

On mine, the sqlite database is empty. Apparently it is logging to the remote site, I just haven't figured out how to connect it all up yet.

[Guest]

Cool. No rush.

[Guest]

I would just be VERY cool for users with mixed system, to have a Victron approved PI to send data to more than Victron's site.

[___]

Success. It is logging to the vrm site. Note that the firmware version is Jessie. The VRM ID is simply the ethernet address of your network card, specifically, of the onboard ethernet (even if using a wifi dongle for the actual comms).

[___]

Last night I decided to look at what the comms to VRM looks like. Took out me old trusty tcpdump, then threw that into wireshark, extracted the HTTP post, and employed a bit of python:

>>> import zlib
>>> from urllib import unquote
>>> from urlparse import parse_qsl
>>> from pprint import pprint
>>> s = zlib.decompress(open('zzz').read())
>>> ss = unquote(s)
>>> pprint(parse_qsl(ss))
[('IMEI', 'badbeef15bad'),
 ('c', '1'),
 ('d', '2'),
 ('bst[0]', '0'),
 ('VSOC[0]', '100.0'),
 ('IP1[0]', '228'),
 ('bv[0]', '25.58'),
 ('bs[0]', '100.0'),
 ('IV1[0]', '234.9'),
 ('Pb', '0'),
 ('Pc', '0'),
 ('Bg', '0'),
 ('gc', '0'),
 ('II1[0]', '1.4'),
 ('OP1[0]', '134'),
 ('eO[0]', '0'),
 ('IF1[0]', '50.1'),
 ('Pg', '0'),
 ('Gc', '0.0182044506073'),
 ('OF[0]', '50.03'),
 ('S[0]', '10'),
 ('AI[0]', '0'),
 ('AILIM[0]', '5.0'),
 ('OI1[0]', '1.1'),
 ('bp[0]', '0'),
 ('eL[0]', '0'),
 ('Bc', '0'),
 ('vp[0]', '14'),
 ('vc[0]', '0'),
 ('gb', '0'),
 ('eT[0]', '0'),
 ('Gb', '0'),
 ('s[0]', '3'),
 ('t', '900'),
 ('OV1[0]', '234.9'),
 ('Et', '2016062121'),
 ('PC[0]', '1'),
 ('CI[0]', '0'),
 ('bc[0]', '0'),
 ('a1[0]', '134'),
 ('CV[0]', '25.58'),
 ('ERR[0]', '0\n')]

So that's what the request looks like, it is a typical application/x-www-form-urlencoded form post, gzipped, with php array notation in the field names. And the data is all there.

Why is this important? Well, because using the right dbus command you can tell vrmlogger to log to a different url. So you can tell this thing to log to YOUR server instead of theirs.

# dbus -y com.victronenergy.settings /Settings/Vrmlogger/Url SetValue http://your.site.here/whatever.php.

So there is another way to get the data out... but as usual, it's an artichoke: You have to do a little work to get at the heart of it.

[___]

Some feedback on this: I wrote up some documentation on how to install it on the Rpi. It's still a work in progress, but if someone here wants to start testing, here is the link:

https://github.com/victronenergy/venus/wiki/Installing-VRMLogger-on-Raspbian-Jessie

Feedback would be appreciated.

[Guest]

See my PM. 

[___]

9 hours ago, The Terrible Triplett said:

See my PM. 

Dude. "This looks nice. Can you integrate it into my system?"

You remind my of a friend of mine who would ask me to write a excel->pdf converter, and when I deliver the code, he surprises me with another half day of unquoted-for work integrating it into his horrible PHP site... :-)

[Guest]

3 minutes ago, plonkster said:

... , and when I deliver the code, he surprises me with another half day of unquoted-for work integrating it into his horrible PHP site... :-)

That is so true and I cannot agree more ... never it is as simple as it sounds 

But in this case, you be the judge:
- I give you a SQL web address, username and password.
- Then I give you a SQL insert stored procedure you need to use to insert the data into the M$ SQL DB. One for inverter, one for BMV and one for controllers.
- If the data sent is in raw format, not a problem at all. I deal with that my side.
- And if data is sent every 10000ms, nothing more to be done on the reader side.

Can it be more simple than that? What have I missed?
We can, if you want, chat more to ensure there are NO surprises, before you commit to anything, for as you I hate to go back to a developer with surprises.

And when this works, you know we will never make one cent out of your work. This Victron reader will be free from us to anyone who wants to use our front end. As a matter of fact, as I said somewhere here before, I think it will be THE best option for all users of Victron equipment to rather use a more stable Victron / your reader, than ours. Send direct to Victron's site as well as to our DB, for more options.

FWIW, I said many times here on the forum, what we do with the data we receive, that is all on us, that is where we will add value and earn from, our work only, never will we earn from a reader that we did not develop from scratch ourselves. That is just so wrong on so many fronts, it is offensive.

So if that is/was a concern, it will never happen.

[___]

Just now, The Terrible Triplett said:

M$ SQL DB

That means going with the closest thing we have on linux, which is probably the sybase freetds driver along with unixOBC (because M$ hardly ever writes their own stuff from scratch, they buy a base product and develop it further, so yes, MSSQL and Sybase are brothers)... and yes, it is a complete PITA.

It is completely unusable from a non-windows point of view.

[Guest]

Then it is official, I have tried all angels and angles so rather go quiet before I quite quit, for we could have done better versus could of spent more effort to make it fit. So before I have that fit, lets call it a day, we are not going to see Linux data readers data ... for moving DB's just ain't an option. 

(Damn that took some effort!)

Thanks Plonkster, we gave it a good go. Thumbs up.

[___]

1 hour ago, The Terrible Triplett said:

moving DB's just ain't an option

In every project I've ever worked the DBAs refused to give us direct access to core databases. There was always an API layer through which we worked. It makes sense to me too, as people who are good at writing databases are often rubbish at network security and vice versa.

In addition, the other IOT type projects are all (the ones I've seen) standardising on either an http call (the so called REST interface), or MQTT.

If you want wider interoperability, you need one of those. For security, I also think you need one of those. There is no way in the world I'm letting just anyone connect directly to my database, there is no way I'm leaving that port unfirewalled, in fact, on unix systems we often connect locally via a domain name socket and it doesn't even listen on a network port at all... and if I must, there absolutely will be a VPN and/or encryption layer involved, or SSL encryption will be forced (I see MSSQL at least have the option for that).

Also, us old people tend to avoid stored procedures for portability reasons. I've used stored procedures on a big telemetry project some years ago only because it gave us a speed advantage (the good old "if the record is there already, update it, otherwise insert it" problem, using a stored procedure meant sending the data over the wire only once).

I tell you, whenever I think I'm lax about security (because I am... in many ways) I realise that compared to the rest of the world I'm borderline anally retentive... :-)

Edit: Good job on using the pet peeves. In Afrikaans I get even more knee jerk moments. Elke keer as iemand 'n konklusie bereik (tot 'n gevolgtrekking kom) oor ons konstitusie (grondwet) sterf 'n klein hasie iewers, waarskynlik in groot pyn.

[___]

Years ago, I think it was the year 2000, someone hacked Ster Kinekor's website. He was working on another project and he had ethereal (since renamed to wireshark) open to sniff the network comms. He suddenly noticed an outgoing connection with an XML payload, and discovered it was the flash application in the browser talking back to the SK web server. Inside the XML, he noticed a raw SQL command that retrieves the movies....

You can probably guess what happened next. Mmmmmh, I wonder if this is a read only interface...

A few minutes later a movie showed up on the SK front page. The plot of the movie was basically a retelling of how he discovered the hole. Now that is how you report a bug... :-)

[Guest]

What you are saying is all 100% spot on. API's are a-coming ... time is the enemy.

Truth be told, all our applications are web based, and touch wood, to date we have not been hacked, LOTS of attempts, but not one success ... touch wood.

Why would I give you the details, trust, for why would you want to do anything there other than good? Or have I misjudged? 

Reminds me of Superbru, back in 2007, an certain developer was royally playing with his scores, every time "guessed" just in time the right score, all in good fun, no malice. They eventually realized what was going on and promptly contacted him offering payment if he will close all loopholes. He did such with pleasure.

[Jimbo]

@plonkster forgive my necro-post. I was looking into this over the weekend. inspected data from my Venus GX to the VRM to see if there was a way I could emulate a Venus GX's form posts to the VRM using data I had logged on my Raspberry Pi. Using Wireshark, it looked like the payload was encrypted but possibly was just compressed? Do you know?

[___]

1 hour ago, Jimbo said:

encrypted but possibly was just compressed?

The default is to log over https. It is however also compressed using zlib. Wireshark can decompress it for you (though don't ask my how... I generally bungle around there each time I need to do it and accidentally get it right sometimes, so I know it can be done).

[Jimbo]

1 hour ago, plonkster said:

The default is to log over https.

Yeah I changed my Venus GX to use HTTP rather than HTTPS (not knowing if it would have a fit about some 3rd party man in the middle cert) - it needed a reboot before it would actually implement that change, but did the job.

1 hour ago, plonkster said:

It is however also compressed using zlib. Wireshark can decompress it for you

Thanks for that, I was being impatient and didnt check for compression! (I should have noticed the header)