Today, ICC stopped reading the values from the inverters.

After i detected that situation (@09:00), i simply stopped ICC and started it again using the Stop/Start button of the dashboard.

 

ICC Stops working.

I'm just wondering if other users expierience similar problems:  ICC stops working and pi is not reachable using VNC, although Pi reponds to Ping.  Therefore i'm unable to see what happend. Only option is power off/on on pi.  

This now happend three times, with intervals of 11, 9 and 3 days. If it further comes down lo 1 or two days .... what could i look for? change? 

Any hints and tips are welcome.  

Peter,

My experience with ICC is that inherently it is stable. (since Jan 2017)

Maybe look at the following:

Pi Temperature. The hotter it runs, slows down VNC service, to the point where it fails. Pi's (3) running hot is well documented. Heatsinks and ventilation to solve. Check that temps stay below 55 deg.

If on Wifi, check throughput on Wifi. The better the signal, the better. If on cable, check cabling and wifi router. My POE injector sometimes looses connection to the wifi router and i need to reseat the LAN cable.

Check the USB cable. It does happen that the stream from the Axpert stops for no reason/ time period. To solve, you need to unplug the USB and plug again at the Axpert.

Also look at how did you build your icc software. via image or manually. I think the IMAGE is better as it reduces risk of things going wrong or unsure as we are not all linux/ pi fundis.eg check that all processes (ps -ef) are running.

Hope you get to the bottom of this.

19 minutes ago, Sidewinder said:

Peter,

My experience with ICC is that inherently it is stable. (since Jan 2017)

Maybe look at the following:

Pi Temperature. The hotter it runs, slows down VNC service, to the point where it fails. Pi's (3) running hot is well documented. Heatsinks and ventilation to solve. Check that temps stay below 55 deg.

If on Wifi, check throughput on Wifi. The better the signal, the better. If on cable, check cabling and wifi router. My POE injector sometimes looses connection to the wifi router and i need to reseat the LAN cable.

Check the USB cable. It does happen that the stream from the Axpert stops for no reason/ time period. To solve, you need to unplug the USB and plug again at the Axpert.

Also look at how did you build your icc software. via image or manually. I think the IMAGE is better as it reduces risk of things going wrong or unsure as we are not all linux/ pi fundis.eg check that all processes (ps -ef) are running.

Hope you get to the bottom of this.

Must be the temperature: right now its 65° !!  And it makes sense as the running period gets shorter while the outside temperture rises. And... its placed below the Inverter, may be hot air is blown out if it..  Will fix this first.

Many thanks!

Peter

 

1 hour ago, PeterGutti said:

65°

That's not that hot really :-)

Something else you guys might want to look into. What kind of CPU load does ICC have? Simple way of checking is to run top, which gets you something like this:

Look at the CPU utilisation and the load average. Load average (on Linux) is an interesting beast. It says how many processes are waiting for the CPU on average (over 5, 10 and 15 minutes), but for some strange reason the powers that be included processes that are waiting for disk IO into the list. So a high load average combined with low CPU use indicates lots of processes waiting for IO. On a machine running just one application (such as this one) you want it to be less than one. Significantly less. If it is close to 1 or more than 1, and the CPU utilisation is high too... then the heat problem might actually be caused by the software.

I have such a problem on my cell phone, for example. Whenever someone sends me an MMS, the phone goes into some strange loop, repeatedly failing to fetch the payload, getting hot and draining the battery.

Agreed, even 65% is not very hot for a Pi3, However, the higher the temps go, the CPU automatically starts cutting the clockspeed, because the slower the CPU runs, the less heat it generates. Sort of automatic throttling. With the two supplied heat sinks mine is running around 44C. A small computer ventilation fan to draw out stale (battery gasses) air should drop that by a few degrees. 

1 hour ago, Sidewinder said:

the CPU automatically starts cutting the clockspeed

That's an old trick, I think it started late 90s as the Pentium-II and its AMD competitors came on the market. I seem to recall Tom's Hardware did a test where they'd take the heatsink off a hot CPU (running Quake, iirc) to show what happens. On the Pentium, the game would just freeze, but with the AMD the smoke was released from the chip. The trouble with the AMD wasn't that it didn't have thermal throttling, it was that the temperature-sensing diode was too far from the core and it would go up in smoke before the heat reached the sensing diode. Sort-of an extreme example (ripping a heatsink off like that), but it demonstrates how thermal throttling works.

Once had a machine in a data center (Hetzner Germany) that would overheat and start throttling like that. After I asked the techs to look into it and perhaps replace the heat transfer compound (which they did), it improved somewhat, but it still ran between 80° and 90° for the rest of its life after that. The throttling kicks in at 105° :-)

3 hours ago, plonkster said:

That's an old trick, I think it started late 90s as the Pentium-II and its AMD competitors came on the market. I seem to recall Tom's Hardware did a test where they'd take the heatsink off a hot CPU (running Quake, iirc) to show what happens. On the Pentium, the game would just freeze, but with the AMD the smoke was released from the chip. The trouble with the AMD wasn't that it didn't have thermal throttling, it was that the temperature-sensing diode was too far from the core and it would go up in smoke before the heat reached the sensing diode. Sort-of an extreme example (ripping a heatsink off like that), but it demonstrates how thermal throttling works.

Once had a machine in a data center (Hetzner Germany) that would overheat and start throttling like that. After I asked the techs to look into it and perhaps replace the heat transfer compound (which they did), it improved somewhat, but it still ran between 80° and 90° for the rest of its life after that. The throttling kicks in at 105° :-)

Wow! I remember that Video like it was yesterday! Used to be a big Toms Hardware Fan!

Was it a AMD K7?

Jay

1 hour ago, Energy said:

Was it a AMD K7?

I'm about 80% sure it was a K7.

I moved my pi to another location on the installation board (wall) where the surrounding temp is 5° less.

Interrestingly, the CPU temp now goes up to 70deg, but.. pi ICC does not get any data from the Axperts boxes.  All zero. I tried various things without success: Unplug, new cable, different USB interface, reboot.. nothing made it run again.  The data from the Batt.Monitor comes through.

I also noticed, that the temp falls withing seconds by 5-10 deg when stopping ICC, (pushing stop button).

@PeterGutti perhaps you should try shutting down the Axperts and restarting. The problem could be that the USB ports on them are hanging up. Worth a try for what is is worth. ICC-Pi Ver 2.0.7 runs flawlessly on my Pi Ver 3 but I only upgraded this afternoon.

On 5/27/2017 at 4:23 PM, PeterGutti said:

I also noticed, that the temp falls withing seconds by 5-10 deg when stopping ICC, (pushing stop button).

Run "top" on the Pi while ICC-pi is running and check the CPU usage. It really shouldn't need to use that much CPU. As a benchmark, the CCGX (similar speed ARM processor also running Linux) is 75% idle most of the time, and it runs a LOT more stuff.

top command on my Pi3 with ICC 2.0.7 says load average 0.15 0.20 0.19 => i've seen it as low as 0.05

174 tasks, 1 running.

On 27/5/2017 at 5:17 PM, ebrsa said:

@PeterGutti perhaps you should try shutting down the Axperts and restarting. The problem could be that the USB ports on them are hanging up. Worth a try for what is is worth. ICC-Pi Ver 2.0.7 runs flawlessly on my Pi Ver 3 but I only upgraded this afternoon.

I did what you recommended:  Powered off the two Axperts (parallel) and restarted it.  After that, USB com worked again.

But than i stopped ICC and restarted it (only the APP, not the pi).  Unexptedly the USB link was down again.  Will try again by shutting down.. but this should not be the solution, because it produces a power outage in the house.  

I could shutdown only the Slave Axpert with the USB interface (the other has RS232). But than the Master Axpert gets an error 80 (undocumented).

Unfortunately by simply switching off the Axpert by the on/off switch only switches off the the output side (seen on the display), the inverter still is powered.

Also completely unclear to is the fact that when taking off the panels and the batteries, it takes about 2mins until the Inverter is really powerless. Is there some type of battery backup inside of the inverter? 

Hi Peter

Do you need to have both Master and Slave connected to ICC. I gathered ICC just needed one of the parallel inverters connected and the comms cable between the 2 did the rest.?

I may be wrong though;)

Regards

11 hours ago, PeterGutti said:

Also completely unclear to is the fact that when taking off the panels and the batteries, it takes about 2mins until the Inverter is really powerless. Is there some type of battery backup inside of the inverter? 

Really big-ass capacitors. Those 63V jobbies we keep on harping about.

I once removed my Multi from the wall, loaded it into my car, drove it halfway across town, unloaded, by this time over half-an-hour later, and when we accidentally touched the two terminals together... there was still enough left in the caps for a spark!

See the same thing with my makeshift power supply (laptop brick plus a buck/boost converter) when powering the beaglebone or Rpi... you can unplug it from the wall and the computer will keep running for a good 5 seconds after that.

4 hours ago, Mark said:

Hi Peter

Do you need to have both Master and Slave connected to ICC. I gathered ICC just needed one of the parallel inverters connected and the comms cable between the 2 did the rest.?

I may be wrong though;)

Regards

Hi @Mark 

No, Its not needed to connect parallel Inverters (think about how would one connect 5 or more inverters?).  On the inverter/Battery page the latest entry says "Inverter Count" is 2.  And the "Cluster info" page shows all details of all connected inverters.

13 hours ago, PeterGutti said:

I could shutdown only the Slave Axpert with the USB interface (the other has RS232). 

Thought so... I was confused by the above comment.

2 hours ago, plonkster said:

Really big-ass capacitors. Those 63V jobbies we keep on harping about.

I once removed my Multi from the wall, loaded it into my car, drove it halfway across town, unloaded, by this time over half-an-hour later, and when we accidentally touched the two terminals together... there was still enough left in the caps for a spark!

See the same thing with my makeshift power supply (laptop brick plus a buck/boost converter) when powering the beaglebone or Rpi... you can unplug it from the wall and the computer will keep running for a good 5 seconds after that.

This problem does not happen on my new, second inverter. Its the one with the USB interface, it has a sligthly different design.  When pulling the battery fuse it immediately show no sign of life: Display goes off. The older one, having RS232 interfaces and runs as master, needs minutes to go off.

1 hour ago, Mark said:

Thought so... I was confused by the above comment.

@Mark Yes i can do that also, but then the master inverter goes into error 80. And the only way to clear that is to power off.. I think!

I did not explain the whole story because lenghty messages are less read as the short messages. 

I'm still trying to bring USB com back to live..

I powered each Inverter off, but not a the same time, not to get a power outage in the house. Restarted ICC and the pi. No luck!

What confuses me is the status:  The blue field does not contain any text. Normally it says "Battery Mode".  The button to change the mode says "Change to Battery". This would mean that inverter reports it is in "Grid mode"! No? When pushing that button and confirm to switch to battery mode, nothing happens!  

Whats wrong with that bl....y inverter? 

Peter,

When I power down my Axpert 5kVA (Manufacture Early 2016), it dies immediately (well can't really say how long, but i'd say seconds, never more than a few). Since upgrading to InfiniSolar, my Axpert is dormant, so overloading is something of the past!

My Axpert has both, USB & RS232 ports. And if I remember correctly, these interfaces are on removable daughter boards, just like the parallel board. Maybe you can "migrate/move" the board to the other Axpert or visa-versa?

Another option. Try and see if the RS232 port is alive. Run the factory software on PC and connect with RS232 cable. Your PC needs to have an RS232 port. Else you need a USB to RS232 cable combo. Do not use USB and RS232 at the same time. I've done that and got big numbers on EMON. One at a time!.If the RS232 is OK, and is seems the USB is kaput, then the new ICC ver 2 (now @ 2.0.7) has a new setting where you can choose either USB or Serial. I have not seen these boards other than the parallel one for sale. Not that I have been looking :-)

@Sidewinder Hi!  Thanks for the hints. Will install V2 of ICC as soon as is available. Actually it says i'm running the actual version (1.1.3)

It is available for testing from http://www.iccsoftware.co.za.