Thursday 23 October 2014

FACTS Fallout - MAJOR I2C Arduino Issue

If you use any I2C Arduino based elements in your droid please read on.  The way the default Arduino Wire library works may cause significant issues if power is lost to any end unit. 

FACTS was a great weekend.  Loved it.  But R2 had a few issues, and James' B4 had a few too.

Before lunch on day one, my dome battery came lose and pulled a ton of wires out.  But no harm done, re connected and added more support for the battery.  He seemed fine after that for the rest of Saturday and Sunday morning.

At one point James' droid drove on without any control!  First time id seen it do that with the master powered and relays fitted.  He said it was when the dome battery died on him.  Not something i had expected, and not what id seen in testing, but im not sure i had tested that order of issues/events. But this had to be taken very seriously and investigated ASAP.

When i got back i got straight on the case to solve it.  Testing it showed that if you boot up, then pull the power on any I2C unit, then drive dome or feet and while driving, trigger any script call, the master has then either crashed or is frozen and cant stop the ESCs.

After some analysis I found that if any device on the I2C network looses power that will freeze the master on the next I2C send.  Meaning that if driving when you trigger, it will carry on driving!!!!!  Any unit failing, or any lose wires could cause it.  Not good.  

Was this a general I2C issue?

I looked on-line for solutions and to see if i could find any others with similar issues.  All i found was Mosfet and Isolator options and level shifters, that would have added a lot of complexity to the setup, and not been ideal for my problem.  After 2 nights of research i finally bit the bullet and ordered some MOSFETs.  

But then i had a brain wave!  What if the issues were possibly code related, and could be tweaked to fix it, or at least make it safe if there is an issue.  So researching from a different start point it soon became clear that the default 'Wire' library has some pretty major issues and would wait forever for the communication to complete if there was a problem.

So i tried to use a couple of custom alternative libraries.  No joy, they wouldn't work with the higher level I2CEasyTransfer library from Bill Porter.  So i had a good read of the library files of the alternatives, and did a bit more reading.  Eventually i found that i could edit the default library, adding timeouts to stop the freezing.  All i did was to edit the 'TWI.ccp' file and add a few counters to return a 0 if the timers' max was reached to break the while loops that kept it locked.

Looks like it is sorted.  Pull the power on any I2C item and the master will not freeze on script call, so the Feet and Dome can still be controlled no matter what happens.  The network still stalls, and the first script you send is actioned if you reconnect power, but that is no bother as you will likely turn the droid off before trying to fix the issue.

I then looked at my big issue.  Starting late Sunday morning, every time i had triggered a big script the body unit would reset, and cause other niggles, pause the I2C network and was really got on my nerves!  I assumed something relating to a low 12v battery maybe? After some mucking about, i found that it was only when the Utility Arms moved that it crashed.  So i unplugged them and he seemed fine.  

The resetting of the Arduino was the puzzler, the voltage for the servos should be from the Stepdown, so them moving shouldn't have caused that issue, especially not mid move.  So i checked a few voltages and found that the 12v body DC/DC supply was reading as 8v in and 5v out, odd.  So i went round the back and there is a big red light on one of the fuses on the rigrunner!  Muppet.  Blown fuse and I guess leaking current from the pwm pins leaking trough the servos making it almost work!  Easy fix, but may look at isolating the servo power more efficiently eventually.

So R2 is working well.  But.......

While researching i kept seeing that the max output power from an Arduino as 500mA.  Knowing that the Master is now powering those big relays, it must be getting close, i will try adding a 2nd feed for the relays and retest to make sure it works as before, but leaving the draw through the master much lower.

I also found that R2 wasnt quite loud enough at FACTs, only a little, but he needed more volume.  So i have ordered a variable step up converter for the amp supply, a little more voltage should help. :D

Im debating moving to UBEC supplies for the 5v and dropping the 5v batteries and other stepdowns.  I did last 2 days on the 12v but not on the 5v ones, so it seems pointless to fight it.  I might add an alternate dome battery that i can switch to, i like the dome to be able to run alone, for when i eventually do other domes. But it might help simplify the setup theory.

Should be a safer system all round now.  Thanks to another couple of real world system issues showing themselves, and figured out, really pleased those are sorted.  I still cant believe it was that easy to address, i suppose i got lucky.  The I2C error hasn't shown it'self before in 2 years on 2 droids, but still great to see and sort it without an incident. 

 I will finish applying and testing these fixes and i have an event on the 16th, so i can do a proper test.  If it passes i will update the doc and code.

No comments:

Post a Comment