Why dont you use the Arduino Can Bus ?

Hi Plonky 

Here at the Republic of Roodeberg I have 3 X CP2102s is that what you are after?

 

Chris

 

 

2 minutes ago, jdp said:

Why dont you use the Arduino Can Bus ?

1. No galvanic isolation.

2. In a manner of speaking I AM using the Arduino can bus. The self-built board is almost exactly the same as the shield, and I even use the same libraries. Only difference is the MCP2551 is swapped with a TI ISO1050, and this required some extra circuitry to power it from the can bus.

Aaa OK. Have fun then ;-) I managed to find out what my problem was on the other project I am working on so all is looking good on that end.

3 minutes ago, Chris Hobson said:

Here at the Republic of Roodeberg I have 3 X CP2102s is that what you are after?

 

Yup, that's what I'm looking for. I wasted too much time trying to make it work with the stuff from my parts bin :-)

1 minute ago, plonkster said:

Yup, that's what I'm looking for. I wasted too much time trying to make it work with the stuff from my parts bin :-)

How many do you need?

I can pop some in the post for you. PM your details.

 

Chris

 

1 minute ago, Chris Hobson said:

How many do you need?

I can pop some in the post for you. PM your details.

 

I had a feeling you might do that. Hang on to them for now, I'm not in such a huge hurry and at the cost of postage I can get a few more items thrown in. Thank you very much for the offer though.

43 minutes ago, plonkster said:

I had a feeling you might do that. Hang on to them for now, I'm not in such a huge hurry and at the cost of postage I can get a few more items thrown in. Thank you very much for the offer though.

I get components from China for the cost of the postage in SA, So either I paying postage and the components are free or the other way around. Postage in this country is a laugh,Sometimes parcel never exit the black -hole that is customs JHB. Electronic components are usually duty free. Over the weekend the lad and I sorted 1000 resistors 1/2W and 1 W.  They are 1% metal film resistors an cost about 25c which is comparable for bulk orders of 5000 resistors. China send you 10 of 50 different resistances.

I have another facebook contact that does that sort of thing: Order 10 x ESP8266 from china and forget about it. Nine months later the package arrives out of the blue... Aaaah, I knew I needed more ESP8266s! :-P

Success!
 I have my Arduino interface working. The problem? Bad power. Both the attached "Energizer" brand and the official Rpi PSU (the white one) make terribly noisy power. The Pi seems okay with it, but the Arduino was not happy. Yes, I have decoupling caps in there.

The Rpi power supply was much better than the energizer, but ultimately it still hung up after a few minutes. Now I'm on the most unlikely of solutions: A Samsung phone charger! I kid you not. Going on an hour now and still going, no crashes.

It's just lovely to see the little CAN icon in the corner of the charge controller's LCD.

Attached a picture of that beautiful icon, as well as the PSU that you should avoid.

AWESOME Plonkster!!! 

We are also making some headway with reading the Morningstar, time permitting.

Oookay, it's just crashed. After about two hours of running well. It's a 1A charger, I'm amazed it's worked this well. It might not be noise, it might be a brown-out issue. An opportunity to look at the AVRs brown-out functionality as well as the watchdog.

15 hours ago, plonkster said:

Oookay, it's just crashed. After about two hours of running well. It's a 1A charger, I'm amazed it's worked this well. It might not be noise, it might be a brown-out issue. An opportunity to look at the AVRs brown-out functionality as well as the watchdog.

It runs about two hours before crashing. I have the brown-out fuses set to 4.3V now, and I have a watchdog in place, so I suspect it's not the MCU that's crashing. My suspicion has shifted to the MCP2515 can controller. I noticed that I had neglected to put a 10k pull-up on the reset pin of the can controller. Either way, if it runs two hours at a time, it's good enough to start developing software against.

Other news, last night I made an important fix to the Victron mk2 library. I'll discuss it below. I also integrated the mk2 reading thread into the pyramid app, so now you list it under pyramid.includes (pyramid has a really nice extension model), add the configuration variables to your ini file, and it starts up a background process that reads the load numbers. No more separate thread required for that.

In future, I envision integrating implementations for other charge controllers and inverters into similar add-on packages and simply listing them as an include in the ini file, ie, you would use "pip install" to pull in the python package for your piece of hardware, list it in the ini file, and you're done. That's the idea. Of course you are still free to write a standalone mqtt-able app and use a different language...

Anyway, so back to the important improvement to my mk2 library, something that might plague other programmers and/or victron owners as well. The mk2 does this thing where -- if you haven't talked to it in a while -- it will start sending version responses once a second. So the first time your app starts up, your serial buffer is not only already full of version responses, it's probably overflowed and there is no guarantee that what is in there is even truncated on a packet boundary of any sort. So you absolutely have to flush the serial buffer at startup. That is not the only problem however: There is this really irritating race condition, which is remarkably easy to run into despite the crude once-a-second nature of it, where a version response will start to arrive exactly in that split second between clearing the buffer and commencing communications, so the code will often error out because something cannot be parsed correctly.

Running at a relatively slow 2400 baud probably makes this window larger than "normal".

The solution I came up with is this: I know that once you send a command to the inverter, it will shut up and stop sending version responses for a while. I'm not sure for how long, I think at least 5 seconds or so. So to sync with the port, you do this: You send a version request, wait half a second, and then clear the buffer. The inverter now shuts up for long enough to have an open window of a couple of seconds within which there will be no race conditions. With this small change, my start-up is now dependable.

Tonight, if I don't fall asleep before midnight, I'll code up a service for reading a VE.direct charge controller, so by next week I'll get real numbers.

Plonkster, came across this  accidentally:
https://github.com/brocktice/tristar-python-modbus
https://github.com/RomanKubiak/envi
https://groups.google.com/forum/#!msg/pymodbus/wCkqo5ImgOw/BsfYgOGjmlAJ
https://www.raspberrypi.org/forums/viewtopic.php?f=37&t=11955&start=50

1 minute ago, The Terrible Triplett said:

Plonkster, came across this  accidentally:
https://github.com/brocktice/tristar-python-modbus
https://github.com/RomanKubiak/envi
https://groups.google.com/forum/#!msg/pymodbus/wCkqo5ImgOw/BsfYgOGjmlAJ
https://www.raspberrypi.org/forums/viewtopic.php?f=37&t=11955&start=50

Oookay, so the tristar supports modbus? That's good news. I don't know modbus that well, what I know is 1) my electronic engineer colleague says it's a crap legacy protocol, but it is industry standard (the other one is fieldbus), and 2) it's query response, so you so will have to write something that queries it periodically and converts the response to an mqtt message.

I also see in there a monitor script that ends in an underscore. That positively shouts "munin" (a unix monitoring/charting tool not unlike cacti), based on rrdtool. Looking inside the script, it really does look like a munin plugin. How that works: munin calls that script at 5 minute intervals and logs the values it spits out. So just starting with that script, you're probably an hour away from making it work :-)

Not me ... they. 

When all else fails, another possible solution would be an arduino with a hall sensor and voltage sampling. Then it's a completely separate device that has no dependency on any particular charge controller :-) I'll see where the project goes. So far it seems it's just me having fun and posting boring status updates :-)

I will be back in full force, when we have a break on our day time development, which is currently flowing over into evenings and weekends. 

So fear not ... no time is being wasted.

So I came home from work, it's 31 centigrade outside (and not much cooler in the garage where this little project is sitting) and all of last night's stability is gone. After soldering in a pull-up resistor, stability returns. It's been running for 5 hours now, very stable. The proof will be whether it is still running by tomorrow.

I also implemented the code to read the vedirect and pass it onto mqtt. It has not been tested against a real piece of hardware. It probably runs at the wrong baud rate. But it's there... :-)

The chat somewhere else on this forum made me think of making this post. 

 

Plonkster, are we all there still on the page where once we have the basics running to read the data and store it somewhere, you on Victron (I have stopped on that one because of your efforts), us focusing on the Morningstar instead, that we are going to share that part for free here, or have you other plans? 

And, with your permission, would you give us your goed to Windownize it. 

Because I hoped that by doing it on Win, using a small reading device, that we can piggy back on other peoples software to avert us from doing all that from scratch.

 

BUT!!! I do have a loooooong play, since many years back, when we have the time and resources.

I do want to go all out and craft software that can read as many devices as possible all over, partnering up with anyone who is interested, for I can see an installers getting a Email / SMS that a clients system is not behaving as it should, based on statistics gathered over a period of time, that he can sort the problem before it becomes a problem for h[is]er client, with clients being able to monitor their own systems, and switch them, as they please, via their smartphone.

This is not going to happen soon, but I am going there, with partners and with help.

Hi all,

Yes, I too read that thread. First things first though.

I have to admit somewhat sheepishly that my stability problems were all my own doing. I had neglected to put a pull-up resistor on the reset pin of the can controller. Leaving it floating... well any kind of noise would cause it to reset. It's been rock-solid since yesterday, and I have just started it up for the first time. The result can be seen here:

http://ispburger.ddns.net:6543/

It's not complete yet. I have to match the baud rate of my interface with that of the ve-direct. Then I have to fully integrate it so it starts up automatically and takes config from my .ini file. Then commit and push it all to github.

So with the pleasant part out of the way. I will admit that the present very make-shift web interface is absolutely a crude copy of Jaco's dashboard. I used my own gauges, or more specifically, I went out and found an open source one with angular integration. Because I'm lazy like that. But the idea of a load gauge on the left, and then a number of gauges for other stuff... I literally used his screenshot for that. I must admit I was a little worried... copying it like that. Thing is, this is not America. We cannot patent generic ideas things like this. If anything, I consider a copy like this to be an acknowledgement of a brilliant idea. It's also an admission that I am personally not very creative when it comes to making things beautiful: If I can read it, 'tis good enough for me.

(When I still did website work for people, one of the things that happened way too often and that frustrated me terribly is the complaint that "the font does not look like the design" from a client. I digress....)

In any case, I'm not getting involved with that other argument. I started my Victron interfacing code long before any of this, and all you guys did is give me the kick in the pants to finish it :-) I've been using Linux since 1997, I've contributed to open source projects far and wide, I take this sh*t seriously :-) I have an engine-room personality. The challenge and the fun is in solving the problem. The reward comes when some random stranger emails you from Italy asking permission to use your code (yes, this has happened, Android electricity app I cobbled together from other people's code).

I just hope that what I've done will be useful to others. It's an entirely different ecosystem. Python. Linux. Javascript. But then, I think that renewable energy should become more open, not less, and therefore the software behind it should be open source. Almost aggressively so.

So maybe some day someone is going to email me, tell me that the software was useful :-)

At the moment, there is a LOT wrong with it still :-)

Edit: So anyway, after going out and swapping the Spark Plugs on the Wife's Suzuki (I'm impressed with the neat way Suzuki packages their stuff, as an aside), I realise I didn't answer your question. Yes, I think we are on the same page. Mostly. I'm not so much thinking of exporting to csv files. I've never been very fond of systems that use files as intermediary buffers :-) What I've done so far will however enable you to make CSV dumps, should you want them. What I'm actually doing here, is building an opensource ccgx. Which is a little funny, because I suspect Victron is going to make theirs open source too. It's already partially so. There is no reason why you cannot develop and use it on windows. That's one of the beautiful things about python. So the usual open source rules apply: 1) It is released in the hope that it will be useful, 2) You cannot take this and make a commercial product out of it, 3) if you do base something off it, your project becomes open source as well, 4) If you break anything, you get to keep the pieces, 5) Nobody will be suing anybody for anything.

:-)

Cool!!!

The though behind the exporting to CSV files, is to have a independent freeware device out there that anyone can use as they see fit.

The potential commercial part, that part is going to be software developed with HTML5 / JSON using data that has been harvested into a M$ SQL database, on the web.

How the data is read and fed to said M$ SQL database, THAT is freeware for as you, I think there are many many more people out there in the world that would absolutely respect the fact that they can read their data from their equipment, to then do as they please.

Take AICC and Solarmon, I bet you if we give Edmund the files layouts, he will have it in his software in a snap, to cater for Victron and Morningstar.

Maybe even JDP would consider.

Both their offerings are based on Windows so that is where I hope to contribute.

 

But, the more I see this evolve, the more I lean towards the Pi being on its native OS, just need a way for Windows to read the data on / from it, IF that is the best way to do it?

 

Copying in from another thread.

35 minutes ago, The Terrible Triplett said:

JDP, not to re-write AICC in Python no. Rather a group effort for you to add to more devices to AICC that you do not cater for yet, using i.e. Pi readers.

TTT, not to pour water on your fire, but I think there are certain hard limits to how much cooperation will be possible. For one, if you have an Axpert inverter, then you are probably using their built-in MPPT. Why would you need to use anything else than what jdp and edmund have put together? :-)

I understand the sentiment. We don't want to reinvent the wheel. We also each have access only to the hardware that we... uuhhhm, have access to.

With that said, there are other limits too.

First, we're not all developers (I think you mentioned this a few times :-) ).

Second, I jdp is not going to write python and I refuse to touch C#. C# exists only because Microsoft had a tiff with Sun over Java (yes I'm at least that old). If it's going to be open source, then that usually limits you to a couple of popular languages (python, ruby, perl, C, maaaaybe java). I know open source exists for other languages too. I've even seen open source Pascal projects. They aren't very active (with the possible exception of the Lazarus project). Same for C#. It ought to be called C-hash, because you have to be smoking it. All that is said only half in jest :-)

Third, some of us have this insane thing where we write software for the heck of it. Personally... it goes on github and it becomes part of my Resume. Maybe Victron will hire me some day :-P Others would like to be compensated for their time. For this reason, I think there will always be a place for paid software. I'm not insisting everything must be open source. I just think it is a natural fit, given the space we operate in. Renewable energy is for the good of all humanity, and the software should be too, at least eventually.

Given these constraints, I'm not sure how much cooperation will be possible. It's the classical problem, and the reason why APIs exist.

I think that is why something like Victron's CCGX makes such a lot of sense. You just put down the software platform, and so you make the rules. If you're on the Venus mailing list (Venus, victron Energy's version of Unix/Linux that runs on the ccgx) you will see the most awesome things that people do with it. That code actually runs on BBB (beagle-board black) and Raspberry Pi. Thing is, it is not fully open source (yet).

So I have chosen to use an MQTT broker as that "API". I think that in time, a plugin will exist that will talk to the Axpert and push that data into MQTT. I'm not really in a hurry for that day to come, because there is software for that inverter already. There is however nothing for people with strange combinations, such as you with the Morningstar, or myself last year with the Microcare (before it packet up), and I think we should focus on that.

So with that said: Take a peak at the paho mqtt page (http://www.eclipse.org/paho/). It has clients for most popular languages and operating systems. This model works for Venus too: They just use dbus. That's cool too, it's the standard used on linux desktops today, but it's not very windows friendly. You see... I thought of that. :-)

Therein, lack of knowledge (?), is why:
1) "... the more I see this evolve, the more I lean towards the Pi being on its native OS ... " for then it is freeware and stays freeware.
2) I made mention of the CSV files because Edmund and JDP, if he wants, can then read the data from my Pi on my equipment into their software. 

Then, all things equal, their software can then be offered to include Victron and Morningstar.

Or what am I missing?

The web / cell software I am also noting, has NOTHING to do with this project. That is a LOOONG play and is going to cost a packet.
 

The way I foresee this working: If AICC wants to publish/subscribe data to/from equipment on the MQTT bus, all it needs to do is grow MQTT functionality. Connect to the Pi on port 1883, and Bob's your uncle.

I will write a CSV logging module too. It will be useful if you want to plot things in Excel, or upload to pvoutput.org.

I'm also thinking it needs a logging interface to graphite. Then you can use graphite to plot the charts. Because I'm lazy like that.

Re web, of course that comes with it. I need to monitor my system while I'm at work :-) Cell integration? I have an example somewhere where I managed to get Kannel (an open source sms gateway) working with a vodacom stick. You could run it on the Pi. You could also use bulksms, they have a web sms and prepaid interface for sending smses. So your long play is actually not a particularly difficult problem :-)