nico1

That sounds like an Axpert VM II (or clone thereof); these don't have a removable display. Any model without a removable display (or the new fancy round models) doesn't have an RS-485 port and hence can't connect directly to the battery's BMS.
But that doesn't prevent the inverter from working with a lithium battery, just that some of the checks and balances won't be present.

For off-grid I would recommend 3xStuder XTH-8000 rather than Sunsynk. The reliability equation changes a lot when there is no alternative power...

If he wants to go offgrid, just bite the bullet and go with Victron.....you can go with 3×8 or 10KVA Quattros and you'll never regret it. Those beast will live forever, these Sunsynks/Deye inverters are still not proven durability wise. Then on the batteries I think you can get better juice when you use 2V 2-3000Ah gel long life batteries. Just my personal opinion

You've already tried two different sources/versions of the software, so that's probably not the issue.  When you run the software, make sure to right-click the .exe file and use the "Run as administrator" option:

 
Spreading your home's load over the the course of the day is the easiest way to ensure that you do not run into overload situations.  I run a borehole pump for sprinklers from 08:30 - 09:00 every monring, then the swimming pool pump after that until end of the day's sunlight (via Tasmota & NodeRED home automation).  Our solar geyser's 2 kW element is only allowed to give it a boost (via Geyserwise) in the early afternoon in case of heavily overcast weather.  During the geyser's time slot the pool motor will switch itself off via the home automation to provide spare capacity should we turn a kettle on, or decide to cook.  That's the reality of life without a split DB box & only 5 kW worth of InfiniSolar...

I fully agree with the good advice provided so far...
The specs you provided clearly states the rated output power as 3kW.  We know your inverter's overload alarm triggers & the fans are running at max speed, but somehow you insist that the inverter is "cold as ice" and all is fine!
Remember that the specs are there for a reason - they come from the ratings of the IGBTs, diodes, etc. that are used in your inverter, as well as the ability of the fans to deal with the heat generated when you run it hard.  You may exceed these limits for short periods of time without obvious damage, but in the longer term you will destroy your inverter.
As for your question about the SolarPower executable that does not want to open... I assume you are not running this on the Raspberry Pi connected to your InfiniSolar?
 

Hi nico1
I agree with Brani and I know you are playing with fire and if you read the documentation correctly on page 6 it specifies 10awg wire not 12awg for the 5kw unit.
The alarm is there for a purpose and tells you something is wrong and suspect you are going to pay school fees. I know if the unit was installed by a reputable installer he would have told you the same hence my very first post on this matter. The relay spec is not the output spec.
As I said before the 40amp relay for the input is to accommodate the battery charge not for the output to the load.
I am looking forward to the video.
I will not comment again until I see the promised video as that is the only tangible proof that will put this to bed.
 

Please bear in mind that an inverter generates power that can kill. 
So test your plugs with an earth leakage tester. A 16a plug can only carry 3,5kw +-  
Like all overloaded electric devices an inverter can fail in a spectacular fashion. 
If you can push it to beyond its design output it indicates that your overload mcb connecting it to the db is an incorrect size. For my 4kw Infinisolar it was only 20a - I kept it like that with my Sunsynk 8kw as it manages my low amp db. For a 3kw device it should be 15amp.

I have service manuals for the InfiniSolar 3kW & 10kW "Plus" models. This bit of overload spec comes from the 3kW's service manual and I can only guess that same design principles hold true for all the InfiniSolar "Plus" models:

I've pushed mine to around 8kW for short bursts (arc welding, mostly)...

“No need to separate essential and non essential loads.”
Dependant on your model and amps you’ll draw. 
I had the 4kw hybrid replaced now with a 8kw Sunsynk) and it would go to 5kw amps overload with mains backup.
The Infinisolar performed faultlessly for three years as a backup on a low amp db setup m
It is better (my opinion) to look at the amps you pull from the db.
I had to create its own neutral to earth connection on its delivery side when I installed it. This gave me lots of problems as my house is old with connections going everywhere and I had to sort all the neutrals out. 
At the time I used the opportunity to replace my 60 year old board with new equipment. It was worth it as it has performed faultlessly since. 
Keep to the isolator sizes prescribed by your inverter manual. 
Photos of your db will help. 

Hi  nico1
To add to my previous post I am not willing to give a detailed explanation however I am willing to help by giving you a block diagram how I would do the wiring.
I still stand with my previous post.
I hope this will help you.

I hope this will help.
 

I have an Inifinisolar too and found Voltronic's manual to be very detailed, except that it does not cater for a changeover switch.
As far as I know (I did not wire it myself, and the earth leakage may be upstream of the changeover) my setup works as follows. With the changeover in mains mode: Grid supply -> mains breaker -> changeover -> earth leakage -> individual distribution breakers.  With the changeover in Inverter mode: Grid supply -> mains breaker -> changeover -> inverter AC in -> inverter AC out -> earth leakage -> individual distribution breakers.
Disclaimer:  I'm also not a sparky and won't be able confirm whether the manual's cable specs conform to our local standards,or the standards for the earth leakage placement!

I would also recommend the sunsynk inverter. Just remember that without a battery it will switch off during load shedding even during the day when their is enough solar. Even a small battery will allow you to use it during the day during load shedding. 

Ellies sells the 3.6 Sunsynk, but i don't think you will get it close to R12,000

Deye is branded as Sunsynk in SA so it's basically the same OEM product, and yes as @Tariqsays, you are unlikely to hit that R12,000 mark price wise.

That's basically the same price as the 5kW Sunsynk. 
Any reason why you specifically want the Deye?

Your options :
1. Without Batteries : Solis or Kodac 3.6 kW grid tie inverter -  you will get them for under R10 000. - you will get 20 - 25 units per day with your panels . 
2. With Batteries - get the Axpert King  5kW - probably in the R12 - 14 000 range - will give you the option to sun when the grid is off .
or 3kw Axpert 24 V - probably arround R7000 - also only needs 24 V batteries - so 2 x 100Ah lead acid or a 24V LiION such as the Pylonteck or BlueNova Mini .
3. 5k Sunsynk - about R18 000 - R20 000 : Hybrid - you can run offgrid and also feedback - best of both worlds . Stick to the Sunsynk - better support and better software.
 
My setup : Solis 4.6G with 10x455w Ja Solar panels - gives me 15 - 30units per day dependent on how I manage my load .
I have a 3kW 24 V Axpert that I run as a UPS for critical loads when the power is off . 
Works well and is very cost effective .