Modem module firmware cannot be updated - looping ^SYSSTART | Telit Cinterion IoT Developer Community
January 18, 2019 - 4:22pm, 3421 views
Hello. My company produces a remote data logging product which incorporates a NimbeLink NL-SW-LTE-GELS3 modem module, which apparently uses a Gemalto modem. We have dozens if not hundreds of these devices in the field, and recently customers have been seeing one of two scenarios: 1) the device works, both answering calls and pushing data to a server, for a week or more, and then stops, requiring a manual power cycle, and 2) often the initial problem then devolves into the modem no longer pushing data or answering calls. It appears that this behavior is initiated when the cell tower operator updates the software used by the cell and it begins to refuse connections from the modems. A hardwired connection to a terminal program reveals the modem is constantly restarting with a ^SYSSTART message. If you type fast or cut and paste, you can sneak in a couple of AT commands between resets, and it seems to respond appropriately. We've tried using the gWinSWup software to upload a new firmware to the modem, but it will only start the transfer for a few seconds and then fails (when the modem restarts). Can anyone offer any suggestions as to how to get around this? The devices were designed for use in harsh environments, and a decision was made years ago to glue them shut permanently except for a few water-tight connectors, and a water-tight plug that can be unscrewed to access the SIM card (and, coincidentally, the USB port on the modem). Direct access to the modem hardware requires physically sawing the case open, rendering it useless. I've been working with NimbeLink support on this, and they're suggesting the firmware was corrupted during a failed update, and there's nothing that can be done via the USB port. As it still responds for a few seconds between restarts, I'm hoping there might be a way to prevent it from restarting, or to disable the existing firmware beyond what's required to push a new version. Any help would be most appreciated!
As we have here the third party hardware from Nimbe and additionally the device is battery powered we can't be sure what fails. It could also be some power supply issue. It would be easier to debug this problem for them (as they own their hardware and are probably the direct customer of Gemalto) then for you alone. The most proper flow would be that you report this to them and they to Gemalto if necessary.
As for the module were you able to grab a log when this problem has happened - I mean that maybe there was some additional output that could help. f you have access to USB - does the module reply to AT commands after it fails to communicate with the network? Is it module's USB interface? If module's USB is not connected to the terminal's hardware, how is the module connected - via RS232?
If the device is constantly restarting there's not much you can do (is there any output that could be interesting?) - I think that you could try to remove the SIM card to check if it can have any influence on this behavior. You probably could try to initiate the firmware update via USB but this might fail.
I think we've gone about as far with NimbeLink as we can. I'm running the device under test from a bench supply; not on batteries at the moment.
The hardware in which the ****m is installed communicates via RS-232 to the ****m, but I was able to plug a USB connector into it through the small waterproof hatch intended for changing the SIM card. There's also a "passthrough" **** in the hardware's firmware that lets you talk directly to the ****m through an RS-232 connection to the hardware. I've been working without the SIM card installed, as instructed by NimbeLink, so the ****m can't connect to the network while I'm trying to apply updates.
As I said previously, it seems to just send the "^SYSSTART" message over and over. On another unit exhibiting this same issue, I'm able to send 2 or 3 AT commands before it resets, but this one seems to be ignoring external commands. Is there any code you can supply that I can push to the ****m via the GwinSwup utility that will return the ****m to the original factory firmware, or that will ***** the existing firmware and leave code to ONLY accept updates without trying to run what's already there?
If it were only one or two units, I wouldn't bother, but it sounds like there are going to be more. Potentially a lot more.
I don't suppose there's any way to get any software into the ****m via the SIM socket, is there?
You can try firmware update with gWinSwup application. In such case the firmware is provided with gWinSwup as a single executable. I can provide you with the latest release if you need.
The update can be done over RS232 (ASC0 interface) or USB (USB0 - on PC it enumerates as modem interface), it's not possible over SIM interface.
You can try but we can't be sure if this can solve this problem. If not, I think that Nimbe can report this problem to Gemalto or report these modules as broken (or maybe you can contact your local Gemalto office with this). These failing devices may need to be provided to Gemalto for investigation. If it turns out that the modules are really faulty, thy can be replaced.
To be honest, the cost of the modules is a drop in the bucket compared to the cost of cutting open the units they're installed in (also sacrificing the custom-molded cases and several custom, water-tight connectors) to get them out. That was why I was hoping for some kind of simplified code that could be pushed to the modem quickly (between resets) to then allow us to push a full-blown firmware update.
For now we can't be sure if it's a firmware problem and if the update can solve it, but it's worth trying.
I think that there is a chance that the update process can be performed. When the firmware update instruction is received by the module it restarts in firmware upload **** where only the bootloader is working and the firmware does not start.
As I said before, "We've tried using the gWinSWup software to upload a new firmware to the modem, but it will only start the transfer for a few seconds and then fails (when the modem restarts). " Is there a particular version of firmware you wanted me to try? I've been trying to install ELS31-V_M2M-GEMALTO-VERIZON_LR18.104.22.168-36343-UE22.214.171.124c from NimbeLink's web site.
The firmware that you have is up to date.
As you have written the module reboots and there's not much time to execute any AT command. If the firmware was to blame here and you'd manage to start gWinSwup in a right time so that it would be able to initiate the firmware update (also with AT command) in this short time-slot when the module accepts commands, you could succeed with the update, because during this process the firmware is not working, it is managed by a bootloader. But you need a direct access to USB0 or ASC0. If you are not able to start the update, it could be possible that there is some hardware issue. Anyway in such a case you will probably not be able to do anything by yourself and you need to go with it to Nimbe. Then they can go to Gemalto if they suspect that the module is faulty.