Awesome Plonkster, I have downloaded it! Now to figure out how a Pi works.

Suppose I need tomato sauce? 

7 hours ago, The Terrible Triplett said:

Awesome Plonkster, I have downloaded it! Now to figure out how a Pi works.

Suppose I need tomato sauce? 

No it's gravy

On 2/22/2016 at 11:35 PM, The Terrible Triplett said:

 

Don't let Viper hear you say this, he will just want you to use a Tablet.

Sorry VIPER, I could NOT stop myself!!! 

I won't mind using a tablet either but it's simply too expensive for this kind of setup, IMO. 

8 hours ago, plonkster said:

Man, this took way too long. So I made a new image today. I think this one has the Linux base sorted out. There was an irritating delay (several minutes!) in the boot up process, which I eventually traced to my ethernet config: It was set to start automatically, so it would sit there waiting on DHCP when no ethernet was plugged in. Removed that, then configured ifplugd. So now that's sorted and it boots very quickly.

I've made more changes since I made the image, but as always, updating the code is as simple as a "git pull".

The image is here: https://drive.google.com/file/d/0By8zt_cN5USfQmxlVGF6dWJCb2s/view?usp=sharing

So what works now. Well, the CSV module finally works. It can't do amp-hour or watt-hour yet, still have to think if and how I want to do that. For now, you include the module in pyramid.includes and configure it as per the examples in the ini file. It can do min, max, average, sum, or just the latest value over any time window.

I've been running this for a few days now and it works very well. The only thing I find myself wondering is 1) what state is the charger in and 2) what's the current voltage. So I'm thinking the next step is to improve the web interface a bit.

I'm busy downloading the image and will put it on a Pi later today. 

How do we submit changes back?

What coding environment do you use?

1 hour ago, SilverNodashi said:

What coding environment do you use?

Something to do with snakes ... uhm ... Python? 

1 hour ago, SilverNodashi said:

How do we submit changes back?

You fork it on github and make a pull request.

1 hour ago, SilverNodashi said:

What coding environment do you use?

I use a plain old editor (in my case, Vim), but you can use whatever you like. I see some guys at work use Anaconda, some use notepad++, some use pycharm, some use Eclipse, some use Sublime... pick your poison. You shouldn't need to develop directly on the pi. You may have to debug on it, in which case I would suggest you install nano if you can't use vim, because I only have vim on there by default.

A quick run-through of what goes on in there.

First, there's systemd integration. This version of Raspbian uses Systemd instead of the older sysv style init that the previous one used. Systemd is simply the "mother of all processes" tasked with starting things and keeping them running. To plug my stuff into systemd, I had to write two scripts. This is all done already and you don't have to break your  head about it, but you do need to know that if you need to stop/restart a service, you have to do it with the systemctl command:

sudo systemctl restart bluelantern
sudo systemctl restart mosquitto

To manually start the process, you cd into bluelantern, stop the service with systemd, then start it by hand:

venv/bin/pserve src/ib.bluelantern/development.ini

Kill it with ctrl+C as you normally would. If you are familiar with using pdb for debugging, you will have to start it like this to get into the debugger. I'm not going to get into that now... a bit out of scope, and some of you might know this part already.

That development.ini file contains most of the config, and you will not be able to avoid editing at least that :-) It's called development.ini simply because that is pyramid convention. It's supposed to grow a production.ini later, and then the idea is that the development file contains settings specific to development whereas production locks it down a bit. Since we're a long way away from production... we go with the development one for now.

There is a second configuration file, equipment.json. In there you simply define your charge controller, inverter, and these two belong to an instance, which by default is called battery01. The idea is that at some future date we will manage multiple banks with multiple inverters/charge controllers. A long long way in the future. Just use the default for now. The inverter is called "inverter" and the charge controller is called "mppt".

So inside the .ini file you will find a line for pyramid.includes. So how this works, is whatever you list there is imported into the interpreter, and then the includeme() function inside that module will be called to set it up.

So what I've implemented so far is three modules (ib.bluelantern.victron.mk2, ib.bluelantern.victron.vedirect, ib.bluelantern.stats.csvlog) which each plugs into the system and performs one function (reading load values from Victron inverters, reading charge values from vedirect charge controllers, dump stuff to csv). Only the inverter and cc modules are enabled by default, and you will likely want to turn them off. Just comment them out.

Such add-on modules may have configuration of their own. You will see it in the ini file, basically settings for serial ports and the like.

Now... you will notice that for the serial ports I use /dev/serial/by-id/<insert name here>. There is a reason for that. Linux detects hardware in a potentially random way. First serial device becomes /dev/ttyUSB0, second one becomes ttyUSB1. So if you have two of them, how do you guarantee that the right one is used? Udev (another cool de-facto linux thing) comes to the rescue. It is configured to create an alias using the device's manufacturer and serial number. But... because your device will have a different serial number to mine, it means the config file is not going to work for you as is. You're going to have to look in /dev/serial/by-id/ and fix it to work with yours. Once you've done that, it should always work even if your devices move around between usb ports and/or hubs.

So, development process. The subject of how to develop with git is a bit tense. There's two ways you can deal with this. First, decide if you want to use python or not. Second, if you decide you do want to use python, it's probably best if your project becomes a separate package that depends on ib.bluelantern, rather than integrating it with bluelantern. Third, if you decide you're not using python (and I think to get started this might be the fastest route), then I suggest opening the firewall on the Pi so you can reach mqtt, and then develop something that communicates with the mqtt bus. This give you complete freedom.

So all that remains then is to tell you how to open the firewall. Edit the firehol config (what an unfortunate name):

sudo vi /etc/firehol/firehol.conf

# or if you don't know this editor

sudo nano /etc/firehol/firehol.conf

Below the line that contains "server custom pyramid" Add this line:

server custom mqtt tcp/1883 default accept

Reboot, and you should not be able to reach port 1883 on the pi.

Now, all you do is push in a message to mqtt, of the form battery01/mppt/power, and the payload must be a timestamp (unix time stamp, seconds since 1 January 1970) followed by a space and a value in watts.

When you push that message into mqtt, it will show up on the web interface.

The web interface is at http://<ip address of your pi>:6543/

The pi runs avahi, which means it does multicast DNS. If you use a mac of linux, you should also be able to hit it at http://raspberrypi.local:6543/

Hope that's enough to get you guys started.

Edit: Last note. Take a peek in scripts/simulate_mppt.py. That's a python script that pretends to be a charge controller hovering around 600W. If you want to use python, that's a good start.

2 minutes ago, plonkster said:

You fork it on github and make a pull request.

I use a plain old editor (in my case, Vim), but you can use whatever you like. I see some guys at work use Anaconda, some use notepad++, some use pycharm, some use Eclipse, some use Sublime... pick your poison. You shouldn't need to develop directly on the pi. You may have to debug on it, in which case I would suggest you install nano if you can't use vim, because I only have vim on there by default.

A quick run-through of what goes on in there.

First, there's systemd integration. This version of Raspbian uses Systemd instead of the older sysv style init that the previous one used. Systemd is simply the "mother of all processes" tasked with starting things and keeping them running. To plug my stuff into systemd, I had to write two scripts. This is all done already and you don't have to break your  head about it, but you do need to know that if you need to stop/restart a service, you have to do it with the systemctl command:

sudo systemctl restart bluelantern
sudo systemctl restart mosquitto

To manually start the process, you cd into bluelantern, stop the service with systemd, then start it by hand:

venv/bin/pserve src/ib.bluelantern/development.ini

Kill it with ctrl+C as you normally would. If you are familiar with using pdb for debugging, you will have to start it like this to get into the debugger. I'm not going to get into that now... a bit out of scope, and some of you might know this part already.

That development.ini file contains most of the config, and you will not be able to avoid editing at least that :-) It's called development.ini simply because that is pyramid convention. It's supposed to grow a production.ini later, and then the idea is that the development file contains settings specific to development whereas production locks it down a bit. Since we're a long way away from production... we go with the development one for now.

There is a second configuration file, equipment.json. In there you simply define your charge controller, inverter, and these two belong to an instance, which by default is called battery01. The idea is that at some future date we will manage multiple banks with multiple inverters/charge controllers. A long long way in the future. Just use the default for now. The inverter is called "inverter" and the charge controller is called "mppt".

So inside the .ini file you will find a line for pyramid.includes. So how this works, is whatever you list there is imported into the interpreter, and then the includeme() function inside that module will be called to set it up.

So what I've implemented so far is three modules (ib.bluelantern.victron.mk2, ib.bluelantern.victron.vedirect, ib.bluelantern.stats.csvlog) which each plugs into the system and performs one function (reading load values from Victron inverters, reading charge values from vedirect charge controllers, dump stuff to csv). Only the inverter and cc modules are enabled by default, and you will likely want to turn them off. Just comment them out.

Such add-on modules may have configuration of their own. You will see it in the ini file, basically settings for serial ports and the like.

Now... you will notice that for the serial ports I use /dev/serial/by-id/<insert name here>. There is a reason for that. Linux detects hardware in a potentially random way. First serial device becomes /dev/ttyUSB0, second one becomes ttyUSB1. So if you have two of them, how do you guarantee that the right one is used? Udev (another cool de-facto linux thing) comes to the rescue. It is configured to create an alias using the device's manufacturer and serial number. But... because your device will have a different serial number to mine, it means the config file is not going to work for you as is. You're going to have to look in /dev/serial/by-id/ and fix it to work with yours. Once you've done that, it should always work even if your devices move around between usb ports and/or hubs.

So, development process. The subject of how to develop with git is a bit tense. There's two ways you can deal with this. First, decide if you want to use python or not. Second, if you decide you do want to use python, it's probably best if your project becomes a separate package that depends on ib.bluelantern, rather than integrating it with bluelantern. Third, if you decide you're not using python (and I think to get started this might be the fastest route), then I suggest opening the firewall on the Pi so you can reach mqtt, and then develop something that communicates with the mqtt bus. This give you complete freedom.

So all that remains then is to tell you how to open the firewall. Edit the firehol config (what an unfortunate name):

sudo vi /etc/firehol/firehol.conf

# or if you don't know this editor

sudo nano /etc/firehol/firehol.conf

Below the line that contains "server custom pyramid" Add this line:

server custom mqtt tcp/1883 default accept

Reboot, and you should not be able to reach port 1883 on the pi.

Now, all you do is push in a message to mqtt, of the form battery01/mppt/power, and the payload must be a timestamp (unix time stamp, seconds since 1 January 1970) followed by a space and a value in watts.

When you push that message into mqtt, it will show up on the web interface.

The web interface is at http://<ip address of your pi>:6543/

The pi runs avahi, which means it does multicast DNS. If you use a mac of linux, you should also be able to hit it at http://raspberrypi.local:6543/

Hope that's enough to get you guys started.

I prefer using VIM as well  

Can you please give me the github details, as I don't see it in the previous post. 

 

Hopefully by tonight I'll have some time to play with this. 

And then the Pi project really took of when ... 

9 minutes ago, SilverNodashi said:

Can you please give me the github details, as I don't see it in the previous post. 

https://github.com/izak/ib.bluelantern

For victron inverters, you need ib.victron as well. Just "pip install" it beforehand if you're making a new development setup, because obviously it cannot be pulled from pypi just yet.

Edit:

I would suggest forking it, then use "git remote" to remove my origin and add yours, then develop against that.

For new hardware, I would suggest making a new python package (if you want to use python that is), and then that package will simply be pulled in using the ini file and includeme() functionality discussed above. This is somewhat of a python thing... lots and lots of SMALL packages ...

22 minutes ago, plonkster said:

https://github.com/izak/ib.bluelantern

For victron inverters, you need ib.victron as well. Just "pip install" it beforehand if you're making a new development setup, because obviously it cannot be pulled from pypi just yet.

Edit:

I would suggest forking it, then use "git remote" to remove my origin and add yours, then develop against that.

For new hardware, I would suggest making a new python package (if you want to use python that is), and then that package will simply be pulled in using the ini file and includeme() functionality discussed above. This is somewhat of a python thing... lots and lots of SMALL packages ...

I have some Axpert inverters and a 3phase Inifinisolar (not setup at this time) so I want to work with this but it seems I would need to learn Python first. Shouldn't be too much of an issue, I used to code CGI scripts many moons ago in PERL, and also some did PHP+MySQL coding. 

As matter of interest, do you hookup the Pi directly to the PV array, and battery bank as well, or solely rely on the inverter's information?

What I really should do is make an example python package, possibly one that's just a simulator, so people can fork that instead. That will take like an hour tops... project for tonight.

Just now, SilverNodashi said:

I have some Axpert inverters and a 3phase Inifinisolar (not setup at this time) so I want to work with this but it seems I would need to learn Python first. Shouldn't be too much of an issue, I used to code CGI scripts many moons ago in PERL, and also some did PHP+MySQL coding. 

If I may, maybe liaise with Edmund on the Voltronic range, as he is wanting to contribute that section? 

Maybe focus on the Inifinisolar?

To avert more than one person doing their own thing for the same devices?

33 minutes ago, plonkster said:

For new hardware, I would suggest making a new python package (if you want to use python that is), and then that package will simply be pulled in using the ini file and includeme() functionality discussed above. This is somewhat of a python thing... lots and lots of SMALL packages ...

Plonkster, with Morningstar, when I have the code, can I forward that to you to add?

Just now, The Terrible Triplett said:

If I may, maybe liaise with Edmund on the Voltronic range, as he is wanting to contribute that section? 

Maybe focus on the Inifinisolar?

To avert more than one person doing their own thing for the same devices?

I won't be able to install the infinisolar at this stage, as it was "upgraded" two 4x Axperts in December (perhaps I should sell it? It's 2 months old) on the farm and I don't have 3phase at our house. 

Cool.

But IF you could, IF you want, your call, liaise with Edmund? He also has to go and figure out Python and all that, having done the work already in Windows environment for Voltronic devices.

Okay, so the itching got the better of me. Here's an example of an add-on that plugs into bl, but is a separate project that doesn't even depend on it.

https://github.com/izak/ib.blexample

Not depending on the project it plugs into is cool. It means you can have your own scripts and entry-points and use it standalone if you want.

So all it needs is the right code to get the mppt charge values instead of using random :-) Colour by numbers...

59 minutes ago, SilverNodashi said:

used to code CGI scripts many moons ago in PERL, and also some did PHP+MySQL

Then python is going to be a welcome breeze. Once you get used to using indendation for blocks, it's significantly easier to read than perl (and if you feel like you miss regular expressions too much, just import re), and PHP.... man... that's something that should not breed.

1 hour ago, SilverNodashi said:

do you hookup the Pi directly to the PV array

Not sure what you mean. Pi runs from a good old 5V wall-wart. A samsung phone charger to be precise. In my case, I have an arduino with a can-bus interface on one serial port (getting charge values from my charge controller) and a Victron Mk2 on another serial port. So this gives me charge and discharge numbers, and I simplistically assume that the battery charge is the difference between them.

I coded my arduino solution to behave somewhat like a ve-direct device, so in theory it's the same as using vedirect and mk2, which should be a pretty common setup for many people.

At some point, I expect we'll get BMV support too. Since this uses the same vedirect protocol, all that is needed is a few tweaks.

The basic idea is a common framework for data collection, transport and display.

In fact, yesterday I started up the stack on my laptop and pointed it to the mqtt server on the pi... and the display came alive, the csv module started logging... this thing is truly distributed :-)

Ok all. Have basic command line appie going on the pi using mono. Now to test and figure out the com assigment.

Sent from my SM-N900 using Tapatalk

15 hours ago, plonkster said:

Man, this took way too long. So I made a new image today. I think this one has the Linux base sorted out. There was an irritating delay (several minutes!) in the boot up process, which I eventually traced to my ethernet config: It was set to start automatically, so it would sit there waiting on DHCP when no ethernet was plugged in. Removed that, then configured ifplugd. So now that's sorted and it boots very quickly.

I've made more changes since I made the image, but as always, updating the code is as simple as a "git pull".

The image is here: https://drive.google.com/file/d/0By8zt_cN5USfQmxlVGF6dWJCb2s/view?usp=sharing

So what works now. Well, the CSV module finally works. It can't do amp-hour or watt-hour yet, still have to think if and how I want to do that. For now, you include the module in pyramid.includes and configure it as per the examples in the ini file. It can do min, max, average, sum, or just the latest value over any time window.

I've been running this for a few days now and it works very well. The only thing I find myself wondering is 1) what state is the charger in and 2) what's the current voltage. So I'm thinking the next step is to improve the web interface a bit.

What is the root password, please?

Just a quick question: what did you add to the image to make it so big? I tried to install it to an 8GB memory card but didn't fit so I had to backup my phone's 32GB card and use that instead. 

5 minutes ago, SilverNodashi said:

What is the root password, please?

No root password, it uses a typical ubuntu-type setup with sudo. Log in as user pi with password raspberry and then use sudo to run commands as root, or run sudo -i to get a shell.

If you feel like you need a root password, use "sudo passwd" to set one.

5 minutes ago, SilverNodashi said:

Just a quick question: what did you add to the image to make it so big?

I have no idea what you're talking about. My pi has a 4GB sdcard and I only use a third of that space. The uncompressed image is only 3.8GB. If you could not fit it onto an 8gb card you did something wrong.

I also zeroed out all the unused space, or more specifically I "oned" them out. The natural state of flash is all ones, so one of the release processes is to create a file of 0xFF characters to fill up the whole filesystem, and then I delete the file, fsck it one last time, and then compress it. That's how I get it down to such a small compressed state.

14 minutes ago, plonkster said:

No root password, it uses a typical ubuntu-type setup with sudo. Log in as user pi with password raspberry and then use sudo to run commands as root, or run sudo -i to get a shell.

If you feel like you need a root password, use "sudo passwd" to set one.

my bad... haven't used a Pi in a long time. I forgot that you need to login as pi and then "sudo su -". All our Linux servers run CentOS so it's a bit different. 

 

Quote

I have no idea what you're talking about. My pi has a 4GB sdcard and I only use a third of that space. The uncompressed image is only 3.8GB. If you could not fit it onto an 8gb card you did something wrong.

I also zeroed out all the unused space, or more specifically I "oned" them out. The natural state of flash is all ones, so one of the release processes is to create a file of 0xFF characters to fill up the whole filesystem, and then I delete the file, fsck it one last time, and then compress it. That's how I get it down to such a small compressed state.

I'm not sure either. I downloaded the .gz file, extracted it to my laptop and tried to write it to the 8GB card using Win32DiskImager. It writes successfully, but strangely when I try and browse to the SD card, Windows wants to format it. 

With the 32GB card it was fine. 

The size on disk is 3.79GB (4,075,290,624 bytes) so it should have fitted on the 4GB card (I see now it's a 4GB card, thought it was 8GB.. 

hmm, I just formatted the card using SDFormatter and it reports the card is only 3.68G:

---------------------------

Quote

 

SDFormatter
---------------------------
Memory Card Format complete !
Please remove the Memory Card.

 Volume Information 
 - File system : FAT32
 - Total space = 3.68 GB (3,955,228,672 Bytes)    
 - Cluster size = 32768 Bytes
---------------------------
OK   
---------------------------

 

Damn! Need to get a 8GB card sometime. 

21 minutes ago, SilverNodashi said:

hmm, I just formatted the card using SDFormatter and it reports the card is only 3.68G:

Had the same thing! I scrounged out an old Nokia card from an old phone for this, and I could not get the default raspbian image to fit either. What I did was to map the image using the loopback interface (see losetup) and that allowed me to see the partition table. I then deleted the loop and created a new one with the -o option (to start deeper into the image) and this allowed me to map the filesystem on the second partition to /dev/loop0. Once I had access to the filesystem, I could resize it using resize2fs, and then I could truncate the file to something shorter. Yes, a lot of PT just to get the thing to fit. I thought my issues with the slightly-less-than-4GB card would be others' benefit :-)

The sd-card actually has two filesystems on it. The first is FAT, it contains the kernel and config.txt (bootloader config) and commandline.txt (kernel command line). So you can actually edit those things from a windows machine.

The second partition is EXT4. Windows will probably try to format it...

If you use a larger card, you can use resize2fs to stretch it to use all the space. But you're probably going to need a Linux PC for that, as you cannot resize it while mounted read/write.

Quick instructions in case you need to resize the filesystem. The second partition starts at block 62914560, so if you set up a loopback device with that offset you can gain access to the filesystem without writing it to an sdcard, eg assuming the image is in /tmp/rpi.img:

sudo losetup /dev/loop0 /tmp/rpi.img -o 62914560

Then you can:

sudo fsck -f /dev/loop0
sudo resize2fs /dev/loop0 3G

And then delete the loopback

sudo losetup -d /dev/loop0

Note that you made the filesystem shorter, but the image is still 3.8GB in size. Not a problem though. Write it to the sdcard until it fails. The part that fails is outside the file system and makes no difference.

Alternatively, you can truncate it if your tool won't deal with it, eg in python:

fp = open('/tmp/rpi.img', 'ab')
fp.seek(3758096384) # aproximately 3.5GB
fp.truncate()
fp.close()

I hope that helps.

1 hour ago, plonkster said:

If you use a larger card, you can use resize2fs to stretch it to use all the space. But you're probably going to need a Linux PC for that, as you cannot resize it while mounted read/write.

Just double checked, using the Pi itself (too lazy to install Linux on a spare HDD @ home right now) and this card is just too small: 

Disk /dev/sda: 3.7 GiB, 3965190144 bytes, 7744512 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start     End Sectors  Size Id Type
/dev/sda1        8192 7741439 7733248  3.7G  b W95 FAT32

The image file itself is 3.79 GB (4,075,290,624 bytes).