ICC, Pylontech & Axpert 5Kva

[Cozy35]

I have an issue that I m not very pleased about. I have an Axpert running off Pylontech and using the Centurion Solar ICC. There is also a PV array to supplement power during the day.

The problem is that the Raspberry Pi 3 reboots constantly at an extreme rate and rarely shows a ICC Runtime of more than 6 hours. When I compare this setup to the one Centurion Solar uses on their web site as a demo (Link Below) I see that system's ICC Runtime is at least 144 hours.

My inverter is o lean power directly from the output of the Inverter.

Each day at 16H00 I switch to Grid to see the night through in the event of a power failure, and back to SBU at 07H00 in the mornings to draw from the batteries until PV is sufficient to carry the loads.

https://centurionsolar.co.za/emoncms8/dashboard/view&id=102

I an effort to resolve this issue their support simply says that it is hardware communication loss. Now this is on my own system at home on which I have set all parameters to the tee for the most efficient operation as well as another install for a client. I have tried this on three different Models (Kodak OGX - basically an MKSII) The VMII and the PF1. All have the same issue.

Pi temps rarely exceed 48C.

My question is, Is there anyone out there experiencing the same issue and if so how did you resolve it? Manie, for obvious reasons, cannot assist since I purchased the kits from Centurion Solar

I was hoping that I could use this product (ICC) as a standard default addition to my installs, but when my clients question the behavior, and I have no explicit answers or remedy, then i would have to take another course of action if a fix does not come about soon.

Any help is appreciated.

[GreenMan]

Have you tried the genuine Pi power supply and a decent micro USB cable ?

[Cozy35]

12 minutes ago, GreenMan said:

Have you tried the genuine Pi power supply and a decent micro USB cable ?

The Pi came with the genuine Power Supply and I have the Pylontech RJ11to USB cable as supplied by Cent Solar. the USB console cable is also the original as it comes with the Inverter.

As an after thought.... I have an HDMI 7" Touch screen on the Pi as a monitor and that runs off the same PSU as the Pi. Perhaps drawing too much current from the PSU?

 

[Cozy35]

24 minutes ago, Cozy35 said:

The Pi came with the genuine Power Supply and I have the Pylontech RJ11to USB cable as supplied by Cent Solar. the USB console cable is also the original as it comes with the Inverter.

As an after thought.... I have an HDMI 7" Touch screen on the Pi as a monitor and that runs off the same PSU as the Pi. Perhaps drawing too much current from the PSU?

 

Removed the HDMI Screen, no luck. Still the ICC process stops and has to restart.

Resorting to changing the Inverter console cable.

[GreenMan]

48 minutes ago, Cozy35 said:

The Pi came with the genuine Power Supply and I have the Pylontech RJ11to USB cable as supplied by Cent Solar. the USB console cable is also the original as it comes with the Inverter.

As an after thought.... I have an HDMI 7" Touch screen on the Pi as a monitor and that runs off the same PSU as the Pi. Perhaps drawing too much current from the PSU?

 

Hmmm Ok. And another uSD card - making sure it is a 16x speed.

[Cozy35]

10 minutes ago, GreenMan said:

Hmmm Ok. And another uSD card - making sure it is a 16x speed.

I doubt that Cent Solar uses low quality uSD's.

[wolfandy]

Is the Pi rebooting or just ICC restarting? I've sometimes had ICC die on me but the Pi was still running fine

If the Pi is rebooting, have you tried running the Pi without ICC for 24h to see if it still does a reboot?

[Cozy35]

23 minutes ago, wolfandy said:

Is the Pi rebooting or just ICC restarting? I've sometimes had ICC die on me but the Pi was still running fine

If the Pi is rebooting, have you tried running the Pi without ICC for 24h to see if it still does a reboot?

Only the ICC process stops and restarts. ICC Runtime resets to 0. Hardly ever do I see more than 12 hours runtime from the ICC. The Pi does not reboot.

[wolfandy]

Ok. Then it seems that it is not a Pi problem (power supply / SD card) but rather an ICC stability problem

I've also had stability problems with ICC every now and then, but mine are occurring when I am making changes to settings (and then ICC suddenly dying on me). I have not experienced ICC dying while simply running and not being touched

If the support is talking about a hardware communication loss, what solution are they proposing?

[Cozy35]

1 hour ago, wolfandy said:

Ok. Then it seems that it is not a Pi problem (power supply / SD card) but rather an ICC stability problem

I've also had stability problems with ICC every now and then, but mine are occurring when I am making changes to settings (and then ICC suddenly dying on me). I have not experienced ICC dying while simply running and not being touched

If the support is talking about a hardware communication loss, what solution are they proposing?

They did not suggest a solution. Simply that it happens. in other words, live with it.

My next best would be to get a Freelancer and let them script up a trap to see what is causing this. It is still a bummer that their demo system works, and mine does not.

[Gnome]

Are they running Windows, Linux, ??

Both operating systems have event logs which will allow you to see the reason the operating system rebooted.

On Windows, it used to be called "Event Viewer"

On Linux:

cat /var/log/messages # Get system messages, will show events like kernel panic
last reboot # Get list of reboot commands issued, IE. the reboot command was explicitly issued
last -x # More detail including shutdowns issued

My guess is, it is more likely they actually reboot than it being "unstable". Probably because the actual way they communicate with the inverter sucks and the only way they can recover is by rebooting  (ie. not closing sockets properly and/or releasing resources on physical devices)

Edited June 3, 2020 by Gnome

[Cozy35]

1 hour ago, Gnome said:

Are they running Windows, Linux, ??

Both operating systems have event logs which will allow you to see the reason the operating system rebooted.

On Windows, it used to be called "Event Viewer"

On Linux:

cat /var/log/messages # Get system messages, will show events like kernel panic
last reboot # Get list of reboot commands issued, IE. the reboot command was explicitly issued
last -x # More detail including shutdowns issued

My guess is, it is more likely they actually reboot than it being "unstable". Probably because the actual way they communicate with the inverter sucks and the only way they can recover is by rebooting  (ie. not closing sockets properly and/or releasing resources on physical devices)

The Pi does not re boot, the process simply kills and restarts.

[Durran]

Hi Cozy35, I have a similar issue...only realized it when I added it into my HomeAssitant to monitor Eskom pwoer failure then put it together and also saw that the runtime reset constantly. See below the "dropped spikes", and this happens no matter what the SOC sits at

Glad its not just me  Hopefully there is a solution out there

[Durran]

1 minute ago, Durran said:

Hi Cozy35, I have a similar issue...only realized it when I added it into my HomeAssitant to monitor Eskom pwoer failure then put it together and also saw that the runtime reset constantly. See below the "dropped spikes", and this happens no matter what the SOC sits at

Glad its not just me  Hopefully there is a solution out there

To add, you can see in the below the average runtime for ICC over the last month is under 10, with the current runtime sitting at 8 hours:

 

[Cozy35]

48 minutes ago, Centurionsolar said:

Hi Cozy,

 

Can you contact me personally so I can see if I can find anything wrong please?  My details are on our website (https://centurionsolar.co.za/contact/) I just need the name of your Pi so I can log in and check it out.  There are a few background processes running to help monitor your system.  Firstly, the comms between the inverter and your Pi is monitored, as is the comms between the pylontechs and the Pi.  Should any of those fail, a timeout process will kick in and if none respond within about 7 minutes, the pi will reboot itself in an attempt to re-establish comms.  During this 7 minute timeout process, if it's pylontechs, the pi will also try to reset the comms to the batteries and reconnect to them, failing which it will reboot as a last resort.

Then there is also the monitoring of the activity happening inside the program itself.  Should there be no activity (ie the program hung for whatever reason), we will simply kill the process and start it again shortly after.  From your description it sounds like this might be what you are experiencing. 

Either way, I'd like to check it out and as a valued client try my absolute best to help you.  Unfortunately I am not Pylontech or Voltronics, so there are inherent hardware limits and disconnects I simply cannot get past, but I can try my absolute best to give you the best service I can and help you get the most out of what's humanly possible with the system you have ;).

 

 

Hi,

When I called yesterday and asked assistance I was met with an attitude of indifference. It was as if I was the hard up person with the issue and as I said I was told it was hardware issues losing comms. No offer to assist came from your end, hence my turning to public forum to enquire if anyone else has this issue.

I have subsequently figured the issue out for myself, although a bit rusty with Linux and hardware it was not too hard to figure. I would have thought that you guys developing the ICC to some extent would have the answers to this issue.

You are welcome to access Volton003 and have a look. It is the most severe with ICC process stopping.

In closing, I know I have called Cent Solar on a couple of occasions to try and understand the logic of the operation of the ICC and may have been a pain, but I do demand support when I require it, however feeble it may be. But as  of late I do not feel free to call.

[Durran]

2 hours ago, Centurionsolar said:

Hi Cozy.

I apologize if you didn't get the service you needed and will take up the issue with my support personnel.  Do you recall who you spoke to?  I specifically recall myself having a couple of long discussions with you where no bad attitude was expressed or implied.  The only thing we still can't help you with is the programming of your inverters, as we simply can't be held liable if something goes wrong and you turn around saying "they did it".  Please don't use a public forum such as this any further.

I must say everytime I reach out to them they are extremely helpful and always assist

[Gnome]

 

On 2020/06/03 at 7:52 AM, Cozy35 said:

The problem is that the Raspberry Pi 3 reboots constantly at an extreme rate and rarely shows a ICC Runtime of more than 6 hours.

On 2020/06/03 at 8:08 PM, Cozy35 said:

The Pi does not re boot, the process simply kills and restarts.

I'm confused...

[Cozy35]

9 hours ago, Gnome said:

 

I'm confused...

Sorry to confuse. It was the initial thought when I only had access via VNC console, and later I setup a test on the bench and then observed on the directly connected HDMI screen that the Pi does not reboot, but the ICC process restarts. I assumed from intermittent monitoring that the Pi rebooted.

Edited June 5, 2020 by Cozy35

[Cozy35]

The issue at hand is caused by the V1.0 USB cable as supplied by Voltronic.  The Pi 3B has 4 x V2.0 USB ports. V1.0 can transfer at max 12Mb/Sec and the v2.0 at 480Mb/Sec. logic says that the 1.0 cable could have been be the culprit. The 1.0 cable does not have the TxD & RxD wires in a twisted pair nor is there any screening as the case is with the 2.0 cable. This may suffer cross talk or outside interference - my opinion.

I replaced the 1.0 cable as supplied with all Voltronic inverters with a 2.0 cable.

The ICC Runtime is now at 42 hours, the best I have ever seen which never exceeded 6 hours before. I am not holding my breath just yet and I would like to see 72 hours or more before I can make the assumption that it was indeed the USB cable.

I hope I am correct with my finding, because it makes no sense that data rates as low as the Voltronic Inverter uses on the USB console port requires such a high data rate cable.

[Cozy35]

Since swapping the 1.0 standard Axpert/Voltronic USB cable with the 2.0 cable, the system has been running without incident for 115 hours. It looks like that is what caused the comms/hardware issue as advised by Cent Solar.

[francois]

59 minutes ago, Cozy35 said:

Since swapping the 1.0 standard Axpert/Voltronic USB cable with the 2.0 cable, the system has been running without incident for 115 hours. It looks like that is what caused the comms/hardware issue as advised by Cent Solar.

This is great news! Glad it's sorted.

[Chris Louw]

@Cozy35  How do I know if it is a proper 2.0 USB cable . Thanks 

[Durran]

6 hours ago, Cozy35 said:

Since swapping the 1.0 standard Axpert/Voltronic USB cable with the 2.0 cable, the system has been running without incident for 115 hours. It looks like that is what caused the comms/hardware issue as advised by Cent Solar.

@cozy35 is the cable the one between the Pi and inverter that you replaced?

[Cozy35]

OK, so my test bed has a runtime of 260 hours since I replaced the cable from the Inverter to the Pi. Sadly, the client site sill resets the runtime after the odd 6 to 18 hours after also replacing that cable. The difference is that the client site has Pylontech batteries with the Centurion Solar RS232 to USB cable.

I suspect that the cable as used on the Pylontech console to Pi cable is also suffering the same issues as the standard Axpert/Volronic USB cable in that it does not have the TXD & RXD wires in a twisted pair, nor is the cable screened from outside interference.

Out of 68 residential installations since January, it is imperative that I get this sorted out in order that I can offer the remote support additionally to my service contract to these clients. I will engineer a proper screened RS232 to USB converter cable to try and come to the bottom of the ICC resets.

More later.

 

 

[Coulomb]

4 hours ago, Cozy35 said:

does not have the TXD & RXD wires in a twisted pair

I would think that you'd want to twist together all three wires: TXD, RXD, and ground. But I'm no expert in this.

I've had suspicions that some comms boards have very marginal opto-isolators. As they age, the problem only gets worse, so they might have just squeaked past testing in the factory, and get marginal in the South African climate. If you have a spare comms board, it might be worth doing some swaps to see if you can find which ones are playing up, or possibly even choose a better specified part and replace opto isolators on comm boards in sites that are having issues. The marginality usually shows up with flash programming, which occurs at 9600 bps, 4 times as fast as the commands and responses (2400 bps). But perhaps given enough time, weak optos (low optical "gain" or Current Transfer Ratio, CTR) might affect commands and responses as well.