Raw access to Growatt Inverter Data

[L_D]

Hi there 

I can see my load consumption in the ShinePhone app - attached, but there is no option to export this data into a dlimited format so that I can work on it in Excel.

Tried looking at the data from the Growatt OSS webportal and it only gives pretty graphs- only option to export is pictures in varied formats but no actual data export.

Has someone figured out a way to get access to the raw data without the addition of a Raspberry?

Growatt retains the data for 40 days and I would like to size my planned PV install based on the actual load data.

I have a Growatt SPF 5000 inverter installed with a WiFi Growatt data-logger 

 

Any advice will be appreciated

 

Edited February 22, 2022 by L_D
Additional information

[Mike_C]

Hi, I’m also interested in this. I see that grott on GitHub no longer requires a raspberry pi (eg it works in docker) but will still need some tech knowledge to get up and running.
 

I’ve had solar installed this week and have just started looking at this (the timing of finding this was perfect).  I can report back tech “adventures” if this is of interest but quick searches suggest there are others who have already done just this.
 

It’s a shame there is no direct way to query the inverter (e.g from the local network).  Again - if I see anything whilst searching I’ll report back.

 

mike

[Mike_C]

A couple of weeks on, and I'm reporting on my findings to date.  There appear to be three options

1. query locally via direct cable - reports of the software for this as sketchy at best.

2. set up a man-in-the-middle - capture the data for local processing and then forward to server.growatt.com.

3. query the growatt hosted server.growatt.com

 

I have not tried #1. I don't plan to as I can't see a current use case (for me)

I'm about to try #2.  All my home automation 'operations' do not rely on the internet (it's all locally hosted on a mac with indigo software with zwave (rather than wifi) communication protocols).  This has been super solid for 10+ years. The use case for #2 is if Growatt shut up shop , or start to charge for the service etc.. - again something I don't want to do.  I'm going to try running this in a docker container on an always-on mac.  Also, I believe it is possible to increase the frequency of data - from the standard 5 minutes to 1 minute.  This will be great to see.  This amount of data locally is not a big deal.

I have had success with #3.  There are a lot of API calls that can be invoked - including getting the 5 minute 'chart data' that you are after.  It provides the data in json, so you'll need to do some parsing / processing to get it into Excel friendly format.  To date, I have got the json being returned, and am about to work out exactly which is the key data I want.  I'm thinking I am going to call all the key APIs every day (for the previous full day as well as intra-day).  I will dump these to local files.  I plan to import into my influx db and/or create csv for that Excel friendly format.

 

Hope this is of use and/or interest.  Let me know if a further update would be useful, or any questions on the above.

 

Full disclosure, I'm based in the UK but , as with most things internet / tech, this does not seem to be a major issue.  Just wish I had more Sun hitting the panels.

Edited March 6, 2022 by Mike_C

[Mike_C]

Further reading suggests that the historical data download feature used to exist but was removed on 23/11/2020. Site and extracted text.  It’s exactly this reason that I like to control my own data.

https://github.com/johanmeijer/grott/issues/77

Dear users: In order to bring you a more convenient product experience, we will update the system service platform on November 23th. The specific updates are as follows: 1. New Disclaimer for device setting System Dashboard / Device List, the Disclaimer is added in the device setting, please tick to agree; 2. Stop accessing historical data The system will stop accessing the Dashboard / Device List / Historical Data function; Thank you for you understanding and support

[jules43]

@Mike_C I'm very interested in how you get on with #3. I'm not keen to set up a server to intercept the logging which I've seen others discuss, but hacking some code to query / download the data (perhaps daily) seems very workable. I'm interested in doing some basic analysis of the data and pretty disappointed they removed the feature.

The apps can clearly access the full day's data even if they've retired the previous.

Jules

[Mike_C]

My overdue update of #3.  I have found it to be stable enough / gives me what I want that I have not explored option #2 at all.

I have found that growattServer 1.2.0 to be the best for my needs.

It can be installed via:

pip install growattServer

Details can be found here: https://pypi.org/project/growattServer

I have no affiliation to this project; I am just one customer / user that has found it to give me what I need.

 

I have this running on an m1 mac and have attached my python3 script.  I can't comment on how easier it is to translate this to windows / linux (if any 'translation' is needed at all).  Either way, before running, complete the set up variables configuration (line 11 to 15).

 

To run (based on today’s date) :

python3 ./processToFiles.py

To run (based on YYYY-MM-DD) provide three separate parameters :

python3 ./processToFiles.py YYYY MM DD

 

The files generated are a combination of direct API call output and some calculated values.

A summary csv is generated detailing (and this is how I think about energy created / used):

• Solar (PV) to battery (i.e. stored)
• Solar (PV) to user (i.e. used at time of generation)
• Solar (PV) to grid (i.e. exported)
• Battery to user (i.e. used post generation)
• Grid to battery (should be close to zero but fluctuates)
• Grid to User (imported energy)

 

I then compare this CSV in Excel to that provided by my energy provider - i.e. exported and imported values.

 

processToFiles.py

[L_D]

Mike, this is a very neat breakdown! Thank you.

Do you think it would be possible to run this Python scrit and API login on a AWS Ubuntu EC2 instance?
Your solution is dependant on a machine being on with a web connection. Cloud hosted would be ideal in my scenario.

LD

[Mike_C]

Ubuntu and windows installs both work - instruction overview below.

I should add that I run this script every 5 minutes via cron between 23:00 and 23:55. This ensures that I get the full day's data and provides some protection in case a single execution times out (which the script does not currently handle very well).  With AWS of course, you would not likely need the VM running 24x7, but just when you needed the script to run.

 

Ubuntu:

I created an AWS VM this am (based on Ubuntu Server 22.04 LTS (HVM), 64-bit (x86). AMI ID : ami-00c90dbdc12232b58.

From ssh session:

sudo apt-get update
sudo apt-get upgrade
// python3 was already installed
sudo apt-get install python3-pip
pip install growattServer

Windows  - standard win10 under virtual box:

Install python via https://www.python.org/downloads/windows/ ( I went for 3.10.4.  Selected options to include pip and add python3.10 to path).

From the command line:

pip install growattserver

Common (Ubuntu and Windows):

Make config changes, create output directory, and run...

 

Edited May 2, 2022 by Mike_C
added commentary to benefit of AWS

[L_D]

@Mike is this your CRON expression?

0/5 23-00 * * ? *

 

LD

 

 

 

 

 

 

Edited May 2, 2022 by L_D

[L_D]

Hi Mike 

 

Not sure how to solve this when running the script from my directory as:

/home/Growatt/Script$ python3 processToFiles.py

 

it gives tge following error

 

dayToProcess is: 2022-05-02

Traceback (most recent call last):

  File "/home/Growatt/Script/processToFiles.py", line 91, in <module>

    ppvToday_dayAPI = mix_detail_day['chartData']['ppv'][DayOfMonth_i]

TypeError: 'NoneType' object is not subscriptable

 

Edited May 2, 2022 by L_D

[Mike_C]

What type of inverter do you have?  I believe some of the API calls only work for certain inverter types (I’m more comfortable with coding than inverter types tbh).

[L_D]

I have a Growatt SPF5000 ES. Are the API calls not gerenric across many inverters?
I am more specifically interested in the load consumption, not sure which API can give this from the GrowattServer 1.2.0 libraries?

[Mike_C]

My cron expression is :

*/5 23 * * * /opt/homebrew/bin/python3 ~mike/bin/solar/processToFiles.py

I'd expect all calls to work for all inverters (and may give inaccurate data rather than throwing an error).

If you can send me a copy of your growatt-mix_detail_day-YYYY-MM-DD.json file (it doesn't contain any of your login details), I can see if I can work out the error; it's not one that I've seen in my testing.  The alternative is just to comment it out - and all downstream, dependent calls or calculations.

[L_D]

Hi Mike - happy to send you the JSON file, where is growatt-mix_detail_day-YYYY-MM-DD.json located?

Also you python script makes no mention of the above API.

The "NonType" error seems to have another cause. See this article

 

Edited May 3, 2022 by L_D

[Mike_C]

Will be in the output folder (ie created by the script execution)

[L_D]

There is nothing in my /home/Growatt output folder because the script stopped with the NonType error in my earlier post.

I changed line 11-15 as per your script instruction.

 

[Mike_C]

I suspect you don’t have permission to write to that folder. If the script fails, at line 91, all files written before that will still be in the folder.

[L_D]

I can write to /home/Growatt folder using the exact same logon credentials specified in the scipt using FileZilla.

Is there a standard ubuntu folder that I should consider writing to?

You are writing to:

 

~mike/bin/solar

What is the preceding ~mike in your url path?

I am connecting to my Ubuntu instance via Terminal/ Filezilla using my *.pem cert keys, username is ubuntu and no password.

Edited May 3, 2022 by L_D

[Mike_C]

The username and password in the script need to be those used to login to the growatt server - https://server.growatt.com/login (and NOT those used to connect / ssh to the ubuntu server).

Relative paths can also be used, so try 

outFolder = "./" # ensure this folder exists

and this will create files wherever you run the script from.

(to answer your specific question, there is nothing preceding the ~mike in my path.  The ~ (tilda) means the user's home directory.

You are connecting exactly as I was to the AWS VM (i.e. ubuntu with pem cert file and no password).

[L_D]

Hi Mike , yes I am specifying the Growatt username and password in your script. (Line 11 & 12)

 

I noticed that some json files were being written to the /home directory so I changed the script path to point there even though I instructed it to write to /home/Growatt

 

 

This is the error I am getting now when running the script after pointing the Outfolder (line13) to /home.

 

sudo python3 processToFiles.py

Traceback (most recent call last):

  File "/home/processToFiles.py", line 1, in <module>

    import growattServer

ModuleNotFoundError: No module named 'growattServer'

 

The growatt server URL is  'https://server.growatt.com/, should this not be specified somewhere?

 

I ran the code again with a slightly different command that you are using and got this error:

 

python3 ./processToFiles.py

dayToProcess is: 2022-05-03

Traceback (most recent call last):

  File "/home/./processToFiles.py", line 53, in <module>

    with open(outFolder + outPrefix + "mix_info-" + str(today) + outSuffix, 'w') as out_mix_info:

PermissionError: [Errno 13] Permission denied: '/homegrowatt-mix_info-2022-05-03.json'

 

and running it with elevated right I get this error again:

 

sudo python3 ./processToFiles.py

Traceback (most recent call last):

  File "/home/./processToFiles.py", line 1, in <module>

    import growattServer

ModuleNotFoundError: No module named 'growattServer'

 

 

Edited May 3, 2022 by L_D

[Mike_C]

You shouldn’t need to run as sudo.  Either run as normal user, or try re-running the pip install growattServer command as sudo as well. (My Ubuntu is a bit rusty but there seems to be permissions / who you are running commands as that is causing some issues here).

 

the server.growatt url IS in the package that I’m using.  I don’t call it specifically.

[L_D]

Installed successfully again but errors again

 

ubuntu@ip-:/home$ sudo pip install growattServer

Collecting growattServer

  Downloading growattServer-1.2.0-py3-none-any.whl (11 kB)

Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from growattServer) (2.25.1)

Installing collected packages: growattServer

Successfully installed growattServer-1.2.0

WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

ubuntu@ip-:/home$ sudo python3 ./processToFiles.py

dayToProcess is: 2022-05-03

Traceback (most recent call last):

  File "/home/./processToFiles.py", line 91, in <module>

    ppvToday_dayAPI = mix_detail_day['chartData']['ppv'][DayOfMonth_i]

TypeError: 'NoneType' object is not subscriptable

 

Running without elevated rights like you said I get this permissions error:

 

python3 ./processToFiles.py

dayToProcess is: 2022-05-03

Traceback (most recent call last):

  File "/home/./processToFiles.py", line 53, in <module>

    with open(outFolder + outPrefix + "mix_info-" + str(today) + outSuffix, 'w') as out_mix_info:

PermissionError: [Errno 13] Permission denied: '/homegrowatt-mix_info-2022-05-03.json'

[Mike_C]

hmmm ... the error that concerns me is :     ppvToday_dayAPI = mix_detail_day['chartData']['ppv'][DayOfMonth_i]

This suggests that one of the calls made is not returning any data and/or.in the right format.  I vaguely understand the error but have not seen it myself and unable to diagnose (after all I'm calling someone else's code).

Looking at your original post, and what you are looking to achieve, I have removed the second half of the script (that does much of the calculations) to  create a light version (and attached to this post).  I propose two options from here 

1. growatt-mix_detail-YYYY-MM-DD.json should contain the data you are after.  It is in JSON format as returned via the API call, and will need some reformatting, e.g. with the 'jq' command.

2. assuming there are now no errors in this light script version, you should find I have undertaken this reformatting and created growatt-mix_detail_chartData-YYYY-MM-DD.csv.

hope this gets you closer to the goal.

processToFiles_light.py

[L_D]

Hi Mike - ran you light version of the script in three ways, with and without elevated rights - same issue

 

sudo python3 ./processToFiles_light.py

dayToProcess is: 2022-05-04

Traceback (most recent call last):

  File "/home/Growatt/Script/./processToFiles_light.py", line 90, in <module>

    chartData = mix_detail['chartData']

TypeError: 'NoneType' object is not subscriptable

 

ubuntu@ip-:/home/Growatt/Script$ python3 ./processToFiles_light.py

dayToProcess is: 2022-05-04

Traceback (most recent call last):

  File "/home/Growatt/Script/./processToFiles_light.py", line 53, in <module>

    with open(outFolder + outPrefix + "mix_info-" + str(today) + outSuffix, 'w') as out_mix_info:

PermissionError: [Errno 13] Permission denied: '/homegrowatt-mix_info-2022-05-04.json'

 

This was the closes it came to running:

 

 

sudo python3 processToFiles_light.py

dayToProcess is: 2022-05-04

Traceback (most recent call last):

  File "/home/Growatt/Script/processToFiles_light.py", line 42, in <module>

    plant_info     = api.plant_info(plant_id)

  File "/usr/local/lib/python3.10/dist-packages/growattServer/__init__.py", line 516, in plant_info

    data = json.loads(response.content.decode('utf-8'))

  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads

    return _default_decoder.decode(s)

  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode

    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode

    raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 3 column 1 (char 2)

Edited May 4, 2022 by L_D

[Mike_C]

let's try setting the outFolder to "/tmp/"