Category: Geek talk
If you have copied the code for the Node-Red implementation of the Renault API, be aware that the the kamereon_api key has changed. A few clever people figured out they are distributed using Google Firebase instead of in the code of the app, so decompiling is now useless. However, it’s pretty much all over the internet now, so here is a link to as far as I could quickly follow back.
BTW, there are now many more implementations for Node-Red.
With the addition of the Ph2 and the new settings it’s fair to say we were not up to snuff when it comes to app stability. In December we significantly went over Google’s “bad behavior” benchmark when it comes to app crashes. As written earlier, we’ve been daily checking and meticulously cracking these and older lingering bugs and mostly updated weekly to avoid those from re-appearing. I am happy to announce that this weekend we lost the red “bad behavior in the last 30 days” tag.
We continue to trace and if anything possible fix each and every crash we get reported and release those fixes within a week of receipt. This is by far the most important reason for seeing recent updates in the play store and we appreciate you installing them to keep user experience as good as possible.
ISOTP is a protocol to transfer messages larger than the maximum CANbus frame size of 8 bytes. While it’s purpose is only this, it is usually associated with diagnostic, request-reply type of traffic, as opposed to the broadcast “free frames” type of chatter.
When we started with CanZE we had more information about the free frames and while it is a real pain the xxx to implement and requires commands not available in cheap clone dongles, we also prefered it as it did not require us to inject any traffic whatsoever on the car’s CANbus.
Things have significantly changed since. Using diagnostic commands exclusively has a couple of advantages
- it’s a given on the Ph2 vehicles. No way to get to it’s free frames;
- it’s commands are fairly well documented from more or less public sources;
- some data is simply not available on free frames;
- more dongles will work without using free frames.
So, the setting for ISOTP are as follows, though they may seem a bit counter intuitive.
- For the Ph2 cars, it has no meaning. We should probably disable it.
- For Ph1 cars, if using a CanSee dongle, we nudge ISOTP off, as it’s the fastest option and works fine. If using a commercial dongle, we nudge it on.
- Nudging means: it will save as set, but every time you open the settings, it will switch it again to the preferred option.
All in all, we might decide to get rid of the entire option as it’s confusing. If there are strong opinions either way, let us know.
Renault changed it’s API totally for the complimentary MY Renault app. Muscat did massive reverse engineering, implemented a python library and CLI. I implemented that in Node-RED. Please note there is zero error checking. The bottom two rows are examples that can be fed the same message as the
Get battery status node.
Please note that the API has changed again. The current implementation is here.
The kamereon_id key has changed early 2021. See this post.
Another episode of the 4-weekly statistics and some additional information
- Growth is steady. Last month we’ve seen a 4% increase.
- Crashes and ANRs (unresponsive screens) are again slashed. In the last 7 days (Taken because now most are on 1.37):
- we have seen zero ANRs, cool!
- we had 4 crashes, two were instances of the same problem.
The 30 days crashes and ANRs statistics now put us way above the “bad behavior thresholds” google has defined, and solidly into the second quadrile of all applications. We strive for the third quadrile, meaning less issues than the median of all applications in the Play Store. Though you might not notice “less crashes” actively as a user, it is improving the perception in the long run a lot. It is also bloody hard work!
ANRs and crashes are measured as percentage of affected daily sessions, where a daily session is defined as a day where an individual user used the app at least once. For example: suppose on Monday 20 users use the app once or more, and on Tuesday 30, then there are 50 daily sessions in those two days. If one user has 2 crashes on Monday, it counts as one affected session. If no crashes occurred on Tuesday, the crash rate over those two days is 2%. Google defines “bad behavior” for crashes as more than 1.09% over the last 30 days. Given that we see roughly 100 reported daily sessions per day, only one crash per day is “allowed”, though the median of all apps is 0.32%.
Now there are two important notes to address:
- We only see total numbers. The reports are heavily anonymized and not generated at all if the total number of daily sessions for them is too low. This seems to be to avoid pinpointing users, which is a good thing. Though I admit I sometimes see errors where I think “how did the user manage to do that?”
- Only users who allow sharing of usage and statistics provide this data, so we have no way of knowing how many actual daily sessions are operated. While we understand the privacy impact, especially towards google, we are very grateful to users who do have this option set to on as it allows us to notice issues and act fast on them. To check or change: Settings > Google > Three dots menu > Usage & Diagnostics > On / Off
Four weeks after the first statistics post, a short update on the subjects touched.
- We’ve rolled out 2 new versions since. Well technically 3, but one was in error and superseded within an hour;
- Growth the usual 4+% over the last month;
- Massive (75+%) reduction in crashes and non-responsive states (ANRs).
- I speculate April will see more of the same as the version released just 3 days ago tackles again 9 of those issues. Our goal is to at least half the crash and ANR rate again.
- Three crashes after release are already fixed and committed for the next release, one of which is an issue that is very specific for Android 7.1. and one specific for Android 8.1.
We’re slowly heading into more esoteric territory where problems are less bug fixing and more avoiding platform issues on specific Android versions and/or on specific phones.
Please note that every single crash or ANR is anonymously reported through google *). While annoying do realize we investigate and almost always fix each and every one of them. The changelist is always available here.
*) if you didn’t disable that of course. Here’s Google’s wording on that verbatim:
You can view your app’s technical performance details collected from a subset of Android devices & OS versions, whose users have opted in to automatically share usage and diagnostics data. Learn more
CanZE has no “phone home” capabilities, but Google acquires quite a bit of data and I thought it would be fun to give you a bit of an insight at what we’re up against.
- We are seeing a fairly consistent growth rate of 40% per year. The metric we use is “Installed on devices that have been online in the last 30 days”. As I write this in early March we’re seeing a total of roughly 4400 active installs.
- Not surprisingly more than half of the installs are in Germany, France and the UK.
- The top 6 devices are all Samsungs and a quick addition of all Samsung branded devices added up to over 1300.
- Of the operators what was interesting to see is that about 20% have no operator listed. I interpreted that as devices having a second life without SIM card for basically CanZE only. I like that. Although those would be slow to update and probably miss out on the news bar.
- Android versions: about 2700 are on Android 7 or higher, but believe it or not, 60 are on Android 4 and just over 300 on 5.
For health statistics we get quite detailed aggregated reports on crashes and hangs. The last couple of weeks you have seen quite bunch of new releases and that is because we really stepped up our efforts to root out as many as possible and as soon as we see them. To give some sort of idea, in the last 7 days, and filtering out devices that have not been updated for months we’ve seen:
- One non responsive screen
- Eight different clusters of crashes, 6 of which were reported only once.
- The two were basically the same and accounted for 13 actual crashes. It is a silly bug in the Tyres screen.
As you can imagine it is that last problem we try to quickly focus on and it shouldn’t be a surprise that it is already fixed in the development branch and it will be fixed in the next release. And so are 3 of the single instance ones. For those interested, you can always check out what is in the pipeline here.
When we release about 35% of the active devices are updated within a day, and 70% within a week. But also note that if we assume the three last releases to be “current”, about 18% is not in that bracket after a week. A year after a release is superseded we still see about 2% of that release on active devices. And that is why we need to filter out some of the crashes.
Unfortunately we can’t see how much CanZE is actually used*) so it’s not easy to put those in perspective, but then again, less than 3 crashes per day on a 4000 installed base is too much but not crazy.
*) No, the new news bar does not tell us that. It fetches the news bar from github and we don’t have statistics about it’s usage.
While looking for some stretched screen, I came up with one of these new dash camera devices (1280×400 pixels). OK, the camera in itself doesn’t really interest me, because I only want to use it to make CanZE run on it, which is actually quite easy to do.
The only thing on that device that I do not like at all, is that the USB connector seams to be used only for power, so I can’t use it as development device out of the box but need to compile and transfer the app package in order to be able to install and run CanZE. 🙁
But at least the PlayStore is available out of the box and all underlying Android settings can be reached easily …
Micro introduction: A CANbus, which is used to connect all computers in the car, can only carry chunks of 8 bytes plus an ID, called frames. All data used to actually operate the car, such as switch positions, speed, and hundreds of other parameters are stuffed into unique frames and send out freely, almost always at fixed timed intervals. This is why we call them free frames. There is zero standardization among car makers on the meaning of the ID and the bits inside. Commercial dongles are not really designed to pick these up and have a lot of trouble doing so reliably or at all. (hint: “Timeout on ATMA” anyone?)
To actually diagnose the car, far larger “messages” are required. Even a VIN doesn’t fit in one frame, let alone i.e. the data for the voltage heatmap. For this purpose there is a protocol called ISO-TP which allows you to send longer messages. A software layer chops it up in frames, adds some synchronization data and sends it on. Exactly the same happens for long answers. ISO-TP formatted messages are almost exclusively of the query-response type. One participant (say the dongle or the dealers diagnostics tool) requests something, a computer on the bus answers. Even entire firmware up- and downloads are performed this way.
Why is this relevant? The ELM327 based dongles have basic support for ISO-TP. It’s a pain to set up, but it works and it’s actually what they are designed to do. The caveat being it only works for receiving long messages. We never gave this a lot of thought as the queries we put to the car always fit one single frame. Until the TPMS requirement came up. Writing the valve ID’s requires sending a long message. We implemented “long message ISO-TP” from the get go in the CanSee DIY dongle, so after tedious debugging, we knew setting TPMS worked, but now we had to tweak the driver for the ELM327 dongles to support long messages.
Luckily we were not the first. This nut was already cracked by Cedric Paille, the hero who made DDT4All. By carefully going through the logs of DDT4All, we could modify our ELM327 driver to now also send long messages.Thank you Cedric! The crazy thing is that if you use a commercial dongle, it does quite a bit of the ISO-TP hard work for receiving frames, but for sending quite a bit more is done CanZE and suddenly it is timing-relevant. This is what actually held up a new CanZE release.
Teaser: we have more things up our sleeves regarding not just the CanSee DIY dongle but also for the good old ELM’s. But first: a few final test and then release. As always, stay tuned.