Axpert 73.00b funnies...

[OomD]

I have been away from the forum for a very long time (I blame my incessant working) and recently returned, and spotted the newest Axpert firmware 73.00b of @Coulomb on the Australian forum. Great work there, @Coulomb! Your firmware skills are respected!

Anyway, I loaded 73.00b and also followed the equalisation charge procedure in the voltronics manual, having recently obtained new batteries. The procedure calls for a CV period of 900 minutes (15 hours) max. While setting everything (from the front panel) however I could not get setting 32 to go higher than 675 minutes. It would go down, but not up. Anyway I left it at that, luckily the equalisation charge completed sooner, so I revisited the settings to return all to normal. This time setting 32 would not go lower than 75? It's like it got stuck at 75, it could go up again but not down. Alas there's no button-repeat, you have to keep clicking (a bit cumbersome). Anyway, so I saved the setting at 75. Then I went back into the setting and could continue down to "Def"? It's almost like there is a max number of time you can press UP or DOWN while in the setting, but saving the value and going back in again allows further adjustment.

Another weird thing... if you press ENTER to save a setting then it technically goes out of the setting, but you cannot go back in again... until you press UP/DOWN (or DOWN/UP) to first go to another setting and then come back to the original one. Then the setting number flashes again and you can press ENTER to go into the setting. Unless you do this the ENTER button has no effect.

And, a question... with setting 32 added (I understand that 73.00b fixes the default 10min to 30sec), what is the optimal value to set it to? Leave it on "Def" to use @Coulomb's fixed value?

[weber]

Hi @OomD Coulomb and I looked at these settings problems yesterday. We found they have nothing to do with our patches, as they are present in the original 73.00 firmware. However Coulomb has now produced patches to fix all of them, which we will eventually release in 73.00c. But first we need to fix some other problems (which you can read about here), and thoroughly test the changes.

In regard to your question re setting 32: When you write "Def" I assume you mean "Aut". You need to understand that "Aut" does not represent a fixed time value at all. "Aut" makes the Axpert exit absorb stage when the charge current falls below a threshold. The calculation of this current threshold is described in this post (scroll down to the bold green heading "Charge termination conditions").

The change from 10 minutes to 30 seconds is only in the time it takes to exit the absorb stage after the current has fallen below that threshold (and not gone back above it).

[___]

7 hours ago, weber said:

produced patches to fix all of them

How far are you guys away from just writing your own firmware? Perhaps even open source it? :-) From @Coulomb's posts at AEVA, it's fairly standard MCU (Texas Instruments, FreeScale) processors. Basically, I'm curious if there isn't a point where patching a binary becomes more work, especially when you start patching in some of your own functionality as you guys have started doing.

Is there a header on the board so you can program these chips in circuit? Or is the chip's fuse bits perhaps set precisely to prevent such tomfoolery?

I assume you guys disassemble the binaries and patch it by literally modifying machine instructions? That's next-level stuff... :-)

[Coulomb]

On 2018/04/07 at 11:05 PM, plonkster said:

How far are you guys away from just writing your own firmware?

Light years. I haven't looked at probably 80% of the firmware code, and I find it hard to understand ~30% of the 20% I have looked at (very rough figures). They have very clever code to handle the synchronised relay control, and handling paralleled machines (some with three phase). The contrast with some of the more pedestrian code is stark.

Quote

Perhaps even open source it? :-) From @Coulomb's posts at AEVA, it's fairly standard MCU (Texas Instruments, FreeScale) processors. Basically, I'm curious if there isn't a point where patching a binary becomes more work, especially when you start patching in some of your own functionality as you guys have started doing.

It's certainly getting to be a lot of work.

Quote

Is there a header on the board so you can program these chips in circuit? Or is the chip's fuse bits perhaps set precisely to prevent such tomfoolery?

There is such a header for the TMS320F2809 based boards; it's a 14-pin JTAG header, and there are standard JTAG interfaces (some inexpensive, some four times the price of the inverter-charger and very sophisticated). I've used that header a few times. No fuse bits are blown, but as of a few years ago, they've started using security, which means we can't read their firmware directly any more . But if someone else writes an open source firmware, they would have little trouble flash programming it. Debugging is always a challenge when you have a high power inverter; just stopping the processor dead (at a breakpoint, say) while inverting or charging could be disasterous. And there is no desaturation protection to save the IGBTs or MOSFETs. For distribution to ordinary users, however, it would be more useful to use a similar upload scheme. Perhaps even the same protocol (with a high speed option perhaps), so that users can bootstrap to the new scheme. It would be a monumental exercise.

I'm much less familiar with the HCS08 processor, as used by most of the solar charge controllers and the lower power inverters. I assume that there is a similar arrangement for them.

Quote

I assume you guys disassemble the binaries and patch it by literally modifying machine instructions? That's next-level stuff... :-)

We actually assemble some of the patches into a block of unused flash memory. So there we have the luxury of comments, meaningful variable names, and macros (if we needed them). But even then those source code patches need a call instruction patched into the original firmware to call the patch, and sometimes a few NOPs need to be inserted (some instructions are two word, others a single word). These latter have to be done by hand.

But perhaps a quarter of the patches are direct hex modifications. Usually these are only to NOP out a branch, or to change the value of an immediate operand. We actually publish the .txt file that we use to keep track of the patches, so you can see the gory detail if you really, really want. So we have to know the instruction set; being a DSP, it's not actually intended for use by humans (only by the back end of a compiler). There are remarkable gotchas and inconsistencies in the C2000 instruction set.

Weber also wrote a spreadsheet to calculate the checksums for us. The tool that gets most use is Windows Notepad (I kid you not). We use TI's Code Composer Studio to assemble those patches or partial patches that have assembler source code. I suppose one day we might write part of the patches in C.

And of course, every time there is a new firmware version that we get a hold of, we have to find the addresses of all the patch locations; every one of them will have moved.

Edited August 17, 2018 by Coulomb

[Guest]

1 hour ago, Coulomb said:

... we have to find the addresses of all the patch locations ...

Coulomb, I have a question, more out of respect for what you guys are doing, than ANY other reason.

How long will you guys be able to keep this effort going, for you are saving a lot of people a lot of trouble?

[___]

10 minutes ago, The Terrible Triplett said:

How long will you guys be able to keep this effort going, for you are saving a lot of people a lot of trouble?

It always reminds me of the story of a farmer in the Northern Cape who would occasionally fix up a hole in the public road running through his farm if he was working in the area. Then one day he found the road maintenance people on the neighbour's farm as he was driving past, he stopped and told them to take care of some potholes on the next farm as well, to which they replied: That oubaas fixes his own potholes.

The moral of the story being: Sometimes when you catch up the slack, the guys who are supposed to be doing the work do even less.

Sometimes I wonder to what extent the Ausies aren't helping the Taiwanese sell more product by relieving them from fixing their d*mn firmware.

[___]

1 hour ago, Coulomb said:

We actually assemble some of the patches into a block of unused flash memory

Aaah I see, then you jump across to your code and back at the end of it, essentially bypassing the errant code. Makes complete sense, it means you don't have to fit it into the same space. I suppose it means you can write some of the code in a compiled language and tack it onto the end (if you really wanted to). Last time I had to do that was about 14 years ago (finishing post-grad project), we'd use a normal compiler to make the code, and then we'd jump three instructions past the start of the code... because we knew the first three instructions the compiler generates has to do with pushing registers to the stack. We'd also jump back to the original location before the end of the function because the return statement does the reverse, it pops things off the stack again. All of that was for performance, because all that pushing and popping was too slow. It meant our code was full of dead code that was never executed ... but it saved time because we didn't have to do everything in assembly :-)

[pilotfish]

55 minutes ago, The Terrible Triplett said:

How long will you guys be able to keep this effort going, for you are saving a lot of people a lot of trouble?

I am a firm believer that people should be compensated for their effort - I was happy to pay Manie for ICC Win and I would be happy to contribute to the @Coulomb and @weber efforts to upgrade Axpert firmware if they had a donate page or something similar (and I think they should).

I dont actually use the custom firmware because because stock firmware and ICC work perfectly for me - my bulk/float voltages are 57.4V and 49.7V so I don't get effected by premature float which appears to be the major advantage of the upgrade, but I appreciate the effort of the two gents and would happily contribute in case I require access to their work in future.

Now if I was a direct competitor I would be doing everything in my power to divert them from that effort...

EDIT: to be fair the Axpert/ICC only works in conjunction with a Victron BMV!

[Coulomb]

8 hours ago, The Terrible Triplett said:

How long will you guys be able to keep this effort going, ...

Heh, who knows. If the two main charge bugs eventually get fixed by factory firmware, that would reduce the impetus to keep going. I turn 60 this year, and Weber isn't far behind. I'd mumble something about the "young 'uns" taking over, but it seems that the younger generation aren't as interested in the low level stuff like stack offsets, the "triple XOR trick" (for swapping two locations without needing a temporary location), and so on. The two hopes for someone to take over (should that be a continued requirement) are (a) the Russians (and people of other countries) where security exploitation and defence is a much higher priority, and (b) virus researchers (wearing black, grey, or white hats) who buy a piece of equipment like an Axpert and see the need, if they come to a part of their lives when they have a bit more time.

[Coulomb]

On 08/04/2018 at 2:11 AM, plonkster said:

Aaah I see, then you jump across to your code and back at the end of it, essentially bypassing the errant code. 

Actually, we tend to use calls. Just out of laziness; it means a return statement can get you back, instead of another jump instruction that has to be changed for every new firmware revision. We're hoping that speed isn't an issue, with a ~100 MIPS processor, but this is one of the many things that we don't know for complete certainty. Certainly, all the UI changes are completely safe, but a few of the patches are to responses to CAN messages, which could conceivably have some real-time constraints. There are indications that the manufacturer isn't concerned about speed.

Edited April 15, 2018 by Coulomb
Cut out unnecessary detail

[___]

7 hours ago, Coulomb said:

Actually, we tend to use calls

Aaah you have the luxury of having the CPU take care of EIP (or whatever it is called on that processor, on ARM it's LR :-) ).

[Chris Hobson]

Plonky I see you taking over from Coulomb and Weber not because you want to but because you can .  Only joking I know Victron would have your guts for garters. 

On a more serious note @Coulomb and @weber have you tried making contact with Voltronic when they are in Australia - at a trade show or something similar? They really need to be nudged in the right direction.

[___]

34 minutes ago, Chris Hobson said:

because you can

Masochism isn't my thing :-)

It's interesting to me that they aren't optimising the code, don't seem to be worried about speed, and have oodles of space left. Meanwhile on our side of the fence... the Multi has nowhere near that kind of firepower, there was a processor upgrade around 2014 because space ran out... etc. It's a bit like the old Android vs iPhone debate. Older iPhones typically had less RAM and smaller processors, but performed just as well if not better. An apples for apples comparison is notoriously hard when you deal with these things.

[OomD]

@weber, thanks for your feedback. Yes, I thought the funny I found was related to the Axpert's own firmware, not your mod. I did notice that, as mentioned, going out of the setting and back in again allows further change.

Yes, sorry I meant Aut and not Def (Default). So, that would be the optimal value to use then.

You guys are to commended for your continued effort. I know it takes a lot of time away from more important things, just the fact that you're still doing it is awesome. If you ever start a donation page, which I think you should, I would be early in line to contribute.

The Aussie view feature, in particular, is brilliant!