Hi all,
(This got interesting enough that I ended up also making it a blog post here: http://www.secretlab.ca/archives/164)%E2%80%8B
I've made some progress on the i2c issues seen with the sensors board. I hooked up a pca9306 level shifter and compared the behaviour on the sensors board using an oscilloscope. All of this testing is done with Seeed's Grove RGB LCD module. The LCD module is useful because it actually has 2 i2c devices embedded in it, the LCD controller at address 0x3e, and the RGB controller at 0x62. The two devices operate independently with different electrical properties.
On Hikey+sensors (TXS0108) the LCD locks the bus, and RGB will work, but only after pulling the ribbon cable apart to reduce crosstalk due to insufficient pullups On Hikey+pca9306 the LCD isn't detected and the RGB works correctly (undetermined if there are crosstalk issues)
The traces below show both sides of the level shifter. Green and blue on the top for the data line. Orange and purple on the bottom with the clock.
First, what I saw on using Hikey+pca9306+RGB: [image: Inline image 1]
And with the LCD:
[image: Inline image 2]
In both traces you can see the start condition (data goes low while clock is high), the 7 bits of address (7 rising clock edges), the R/W bit (1 rising clock), and then the acknowledgement bit driven by the device. If the controller doesn't see the device drive the data line low on the 9th clock, then it decides the device isn't there and it terminates the transaction. It is easy to recognize the ack bit because the device has a different drive strength and the voltage level is different.
The RGB controller is a happy little device and it jumps at the chance to drive the data line low. It goes right down pretty close to 0V. The LCD on the other hand is sulky and doesn't drive the line quite as low as the controller can. About to 1V. 1V is probably recognized as a logic low on a 5V device, but with 1.8V it is not even less than half. The way the pca9306 level shifter works is there are pull-up resistors on either side of the device that draws each side up to its respective high level. In this case, 1.8V and 5V. When either side gets driven low, the level shifter begins to conduct and the other side also gets drawn down to the same voltage, but it can only go as low as the voltage it is driven to. If it only gets driven down to 1V, then it will never get low enough for a 1.8V controller to recognize it as a low state.
It may be that with weaker pull-ups the LCD will be able to drive to a lower voltage level. I'll need to experiment more, but in the mean time let's move onto the Sensors board. Back to the traces:
First, here is a transaction to address 0x63 with no device present:
[image: Inline image 3]
Looks perfectly normal so far. Next, the RGB device at address 0x62:
[image: Inline image 4]
Also behaving the same way as it did with the pca9306. Finally, an LCD transaction:
[image: Inline image 5]
Again we see the start condition, the 7 data bits and 1 r/w bit, but the ack bit looks weird. The LCD successfully drives the data bit low enough to be recognized, but then something weird happens. The data line stays low and the clock stops running. I don't know actually know what is happening here, but I've got my suspicions. The LCD is continuing to drive the data line low, (you can tell by the slightly different voltage level) but keeping data low should not stop the clock. I suspect the txs0108 is getting confused and driving the clock line high. I've come across reports from others having trouble with the txs010x series on i2c. It has 'one-shot' accelerators to reduce rise time by driving the line high.
Now I need to decide what to do next. Rev B of the design is ready, but there is still the i2c issue that needs to be solved. I may need to put the pca9306 parts into the design which will require some more rework before it goes to manufacturing.
g.
Hi Grant:
I am seeing the same kind of thing on my board. I removed the pull-up resistors on the target device, and on my board to get the device working correctly (a Seed Studio light sensor).
Looking at the TXS0108 datasheet I see the output drive capabilities: [cid:image011.png@01D11657.85BE1D90]
As you can see they only drive up to 1mA (1,000uA). Your pull-up resistors are what is holding the output from going really low. However that isn’t the whole story, these currents are just what the chip drives while accelerating the rise or fall. Looking at the block diagram we see: [cid:image012.png@01D11657.85BE1D90] You can see a pass transistor between A8 and B8, once A8 starts to pull low the current is passing from B8 through the pass transistor to A8. The current then flows from A8 to the SOC, hence the low is limited by the SOC. In the case of the 410 SOC the output drive strength can be programmed between 2mA and 8mA or 16mA (I have forgotten which), and I believe that the default is 2mA. Above 2mA flowing through the pass transistor the voltage will start to rise. I think we need to change the default drive strength to MAX at least on the I2C ports, but probably on all of the other ports.
Lawrence King lking@qti.qualcomm.commailto:lking@qti.qualcomm.com Engineer, Sr. Staff/Manager Qualcomm Canada Inc. (905)482-5403 desk (x25403) (416)627-7302 cell
From: glikely@secretlab.ca [mailto:glikely@secretlab.ca] On Behalf Of Grant Likely Sent: Monday, November 02, 2015 7:41 PM To: mezzanine@lists.96boards.org; King, Lawrence lking@qti.qualcomm.com; George Grey george.grey@linaro.org; David Mandala david.mandala@linaro.org; 96boards-team 96boards-team@linaro.org Subject: Debugging i2c issues
Hi all,
(This got interesting enough that I ended up also making it a blog post here: http://www.secretlab.ca/archives/164)%E2%80%8B
I've made some progress on the i2c issues seen with the sensors board. I hooked up a pca9306 level shifter and compared the behaviour on the sensors board using an oscilloscope. All of this testing is done with Seeed's Grove RGB LCD module. The LCD module is useful because it actually has 2 i2c devices embedded in it, the LCD controller at address 0x3e, and the RGB controller at 0x62. The two devices operate independently with different electrical properties.
On Hikey+sensors (TXS0108) the LCD locks the bus, and RGB will work, but only after pulling the ribbon cable apart to reduce crosstalk due to insufficient pullups On Hikey+pca9306 the LCD isn't detected and the RGB works correctly (undetermined if there are crosstalk issues)
The traces below show both sides of the level shifter. Green and blue on the top for the data line. Orange and purple on the bottom with the clock.
First, what I saw on using Hikey+pca9306+RGB: [Inline image 1]
And with the LCD:
[Inline image 2]
In both traces you can see the start condition (data goes low while clock is high), the 7 bits of address (7 rising clock edges), the R/W bit (1 rising clock), and then the acknowledgement bit driven by the device. If the controller doesn't see the device drive the data line low on the 9th clock, then it decides the device isn't there and it terminates the transaction. It is easy to recognize the ack bit because the device has a different drive strength and the voltage level is different.
The RGB controller is a happy little device and it jumps at the chance to drive the data line low. It goes right down pretty close to 0V. The LCD on the other hand is sulky and doesn't drive the line quite as low as the controller can. About to 1V. 1V is probably recognized as a logic low on a 5V device, but with 1.8V it is not even less than half. The way the pca9306 level shifter works is there are pull-up resistors on either side of the device that draws each side up to its respective high level. In this case, 1.8V and 5V. When either side gets driven low, the level shifter begins to conduct and the other side also gets drawn down to the same voltage, but it can only go as low as the voltage it is driven to. If it only gets driven down to 1V, then it will never get low enough for a 1.8V controller to recognize it as a low state.
It may be that with weaker pull-ups the LCD will be able to drive to a lower voltage level. I'll need to experiment more, but in the mean time let's move onto the Sensors board. Back to the traces:
First, here is a transaction to address 0x63 with no device present:
[Inline image 3]
Looks perfectly normal so far. Next, the RGB device at address 0x62:
[Inline image 4]
Also behaving the same way as it did with the pca9306. Finally, an LCD transaction:
[Inline image 5]
Again we see the start condition, the 7 data bits and 1 r/w bit, but the ack bit looks weird. The LCD successfully drives the data bit low enough to be recognized, but then something weird happens. The data line stays low and the clock stops running. I don't know actually know what is happening here, but I've got my suspicions. The LCD is continuing to drive the data line low, (you can tell by the slightly different voltage level) but keeping data low should not stop the clock. I suspect the txs0108 is getting confused and driving the clock line high. I've come across reports from others having trouble with the txs010x series on i2c. It has 'one-shot' accelerators to reduce rise time by driving the line high.
Now I need to decide what to do next. Rev B of the design is ready, but there is still the i2c issue that needs to be solved. I may need to put the pca9306 parts into the design which will require some more rework before it goes to manufacturing.
g.
Hi Lawrence,
It may been that the drive strength needs to be increased to get more devices working, but after investigating today I think I'm seeing a different problem.
I've now got a txs0108 breakout board wired up to the dragonboard 410c. I2C works just fine with it. The problem appears to be specific to the sensors board. I've got lots of traces on different configurations, and I think I know what is happening.
It is always the LCD device that causes a bus lockup. I think what is happening there is the LCD is getting confused, and it is driving the i2c data line low. Because the LCD is holding it low, the STOP event can never occur (rising edge on the data line while clock is high). Without the stop condition, the bus remains busy and another transaction cannot be started. I don't exactly know /why/ the LCD is getting confused, but there is lots of noise on the signals.
I took a good look at the signals on the oscilloscope, and I saw this with the sensors board:
[image: Inline image 1]
The blue/green pulse on the RHS is the LCD driving the data line low. Blue is on the 1.8V side, Green 5V. See the big spike at the beginning and middle of the pulse? Here it is zoomed in (note scale: 2V/div on all signals):
[image: Inline image 2]
There is a lot of crosstalk causing ringing which I'm sure is causing the problem. On a whim, I added a 47pF cap between the GND and the data line, and everything started working and the trace now looks like this:
[image: Inline image 3]
So. I've got a signal problem, and a dirty hack to work around it. I've experimented with inserting both 10 Ohm and 47 Ohm series resistors into the data lines to reduce the crosstalk, but it's not helping. Signal routing /might/ be part of the problem. On the current Sensors board the 1.8V signals are routed for a short period distance directly parallel with the 5V signals. The Grove board that I should be receiving any day now doesn't route the signals in the same way, so that may solve it.
I think what I need to do is add pads for series resistors on both sides of the level translator and populate them with 0R for the prototype run. That leaves the option open to experiment with series resistance if the new layout shows the same behaviour.
g.
On Tue, Nov 3, 2015 at 9:48 PM, King, Lawrence lking@qti.qualcomm.com wrote:
Hi Grant:
I am seeing the same kind of thing on my board. I removed the pull-up resistors on the target device, and on my board to get the device working correctly (a Seed Studio light sensor).
Looking at the TXS0108 datasheet I see the output drive capabilities:
As you can see they only drive up to 1mA (1,000uA). Your pull-up resistors are what is holding the output from going really low. However that isn’t the whole story, these currents are just what the chip drives while accelerating the rise or fall. Looking at the block diagram we see:
You can see a pass transistor between A8 and B8, once A8 starts to pull low the current is passing from B8 through the pass transistor to A8. The current then flows from A8 to the SOC, hence the low is limited by the SOC. In the case of the 410 SOC the output drive strength can be programmed between 2mA and 8mA or 16mA (I have forgotten which), and I believe that the default is 2mA. Above 2mA flowing through the pass transistor the voltage will start to rise. I think we need to change the default drive strength to MAX at least on the I2C ports, but probably on all of the other ports.
Lawrence King lking@qti.qualcomm.com
Engineer, Sr. Staff/Manager
Qualcomm Canada Inc.
(905)482-5403 desk (x25403)
(416)627-7302 cell
*From:* glikely@secretlab.ca [mailto:glikely@secretlab.ca] *On Behalf Of *Grant Likely *Sent:* Monday, November 02, 2015 7:41 PM *To:* mezzanine@lists.96boards.org; King, Lawrence lking@qti.qualcomm.com; George Grey george.grey@linaro.org; David Mandala < david.mandala@linaro.org>; 96boards-team 96boards-team@linaro.org *Subject:* Debugging i2c issues
Hi all,
(This got interesting enough that I ended up also making it a blog post here: http://www.secretlab.ca/archives/164)%E2%80%8B
I've made some progress on the i2c issues seen with the sensors board. I hooked up a pca9306 level shifter and compared the behaviour on the sensors board using an oscilloscope. All of this testing is done with Seeed's Grove RGB LCD module. The LCD module is useful because it actually has 2 i2c devices embedded in it, the LCD controller at address 0x3e, and the RGB controller at 0x62. The two devices operate independently with different electrical properties.
On Hikey+sensors (TXS0108) the LCD locks the bus, and RGB will work, but only after pulling the ribbon cable apart to reduce crosstalk due to insufficient pullups
On Hikey+pca9306 the LCD isn't detected and the RGB works correctly (undetermined if there are crosstalk issues)
The traces below show both sides of the level shifter. Green and blue on the top for the data line. Orange and purple on the bottom with the clock.
First, what I saw on using Hikey+pca9306+RGB:
[image: Inline image 1]
And with the LCD:
[image: Inline image 2]
In both traces you can see the start condition (data goes low while clock is high), the 7 bits of address (7 rising clock edges), the R/W bit (1 rising clock), and then the acknowledgement bit driven by the device. If the controller doesn't see the device drive the data line low on the 9th clock, then it decides the device isn't there and it terminates the transaction. It is easy to recognize the ack bit because the device has a different drive strength and the voltage level is different.
The RGB controller is a happy little device and it jumps at the chance to drive the data line low. It goes right down pretty close to 0V. The LCD on the other hand is sulky and doesn't drive the line quite as low as the controller can. About to 1V. 1V is probably recognized as a logic low on a 5V device, but with 1.8V it is not even less than half. The way the pca9306 level shifter works is there are pull-up resistors on either side of the device that draws each side up to its respective high level. In this case, 1.8V and 5V. When either side gets driven low, the level shifter begins to conduct and the other side also gets drawn down to the same voltage, but it can only go as low as the voltage it is driven to. If it only gets driven down to 1V, then it will never get low enough for a 1.8V controller to recognize it as a low state.
It may be that with weaker pull-ups the LCD will be able to drive to a lower voltage level. I'll need to experiment more, but in the mean time let's move onto the Sensors board. Back to the traces:
First, here is a transaction to address 0x63 with no device present:
[image: Inline image 3]
Looks perfectly normal so far. Next, the RGB device at address 0x62:
[image: Inline image 4]
Also behaving the same way as it did with the pca9306. Finally, an LCD transaction:
[image: Inline image 5]
Again we see the start condition, the 7 data bits and 1 r/w bit, but the ack bit looks weird. The LCD successfully drives the data bit low enough to be recognized, but then something weird happens. The data line stays low and the clock stops running. I don't know actually know what is happening here, but I've got my suspicions. The LCD is continuing to drive the data line low, (you can tell by the slightly different voltage level) but keeping data low should not stop the clock. I suspect the txs0108 is getting confused and driving the clock line high. I've come across reports from others having trouble with the txs010x series on i2c. It has 'one-shot' accelerators to reduce rise time by driving the line high.
Now I need to decide what to do next. Rev B of the design is ready, but there is still the i2c issue that needs to be solved. I may need to put the pca9306 parts into the design which will require some more rework before it goes to manufacturing.
g.
The big spike is far more likely caused by how you connected the oscilloscope. Did you have ground leads from near the tip of each probe to a ground test point on the board? Or did you simply let the ground come from “wherever” (through the power supply and the AC line ground into the oscilloscope.
Recommended reading – http://materias.fi.uba.ar/6644/info/puntas/basico/high%20speed%20probing.pdf the section on “the effect of ground lead length”
The I2C specification says that devices can “hold-off” the bus. Either by holding clock line low to prevent the next bit from coming, or by holding data low until they finish the transaction. The scope shows that your I2C data rate is about 40kHz, but many I2C devices can only operate at 10kHz, try dropping the I2C speed to 10kHz (100us per bit time). I suspect the display is only rated for 10kHz, but because there is some margin in the parts, hence you get 40kHz to work most of the time.
Lawrence King lking@qti.qualcomm.commailto:lking@qti.qualcomm.com Engineer, Sr. Staff/Manager Qualcomm Canada Inc. (905)482-5403 desk (x25403) (416)627-7302 cell
From: glikely@secretlab.ca [mailto:glikely@secretlab.ca] On Behalf Of Grant Likely Sent: Tuesday, November 03, 2015 7:19 PM To: King, Lawrence lking@qti.qualcomm.com Cc: mezzanine@lists.96boards.org; George Grey george.grey@linaro.org; David Mandala david.mandala@linaro.org; 96boards-team 96boards-team@linaro.org Subject: Re: Debugging i2c issues
Hi Lawrence,
It may been that the drive strength needs to be increased to get more devices working, but after investigating today I think I'm seeing a different problem.
I've now got a txs0108 breakout board wired up to the dragonboard 410c. I2C works just fine with it. The problem appears to be specific to the sensors board. I've got lots of traces on different configurations, and I think I know what is happening.
It is always the LCD device that causes a bus lockup. I think what is happening there is the LCD is getting confused, and it is driving the i2c data line low. Because the LCD is holding it low, the STOP event can never occur (rising edge on the data line while clock is high). Without the stop condition, the bus remains busy and another transaction cannot be started. I don't exactly know /why/ the LCD is getting confused, but there is lots of noise on the signals.
I took a good look at the signals on the oscilloscope, and I saw this with the sensors board:
[Inline image 1]
The blue/green pulse on the RHS is the LCD driving the data line low. Blue is on the 1.8V side, Green 5V. See the big spike at the beginning and middle of the pulse? Here it is zoomed in (note scale: 2V/div on all signals):
[Inline image 2]
There is a lot of crosstalk causing ringing which I'm sure is causing the problem. On a whim, I added a 47pF cap between the GND and the data line, and everything started working and the trace now looks like this:
[Inline image 3]
So. I've got a signal problem, and a dirty hack to work around it. I've experimented with inserting both 10 Ohm and 47 Ohm series resistors into the data lines to reduce the crosstalk, but it's not helping. Signal routing /might/ be part of the problem. On the current Sensors board the 1.8V signals are routed for a short period distance directly parallel with the 5V signals. The Grove board that I should be receiving any day now doesn't route the signals in the same way, so that may solve it.
I think what I need to do is add pads for series resistors on both sides of the level translator and populate them with 0R for the prototype run. That leaves the option open to experiment with series resistance if the new layout shows the same behaviour.
g.
On Tue, Nov 3, 2015 at 9:48 PM, King, Lawrence <lking@qti.qualcomm.commailto:lking@qti.qualcomm.com> wrote: Hi Grant:
I am seeing the same kind of thing on my board. I removed the pull-up resistors on the target device, and on my board to get the device working correctly (a Seed Studio light sensor).
Looking at the TXS0108 datasheet I see the output drive capabilities: [cid:image007.png@01D116EC.BC085290]
As you can see they only drive up to 1mA (1,000uA). Your pull-up resistors are what is holding the output from going really low. However that isn’t the whole story, these currents are just what the chip drives while accelerating the rise or fall. Looking at the block diagram we see: [cid:image008.png@01D116EC.BC085290] You can see a pass transistor between A8 and B8, once A8 starts to pull low the current is passing from B8 through the pass transistor to A8. The current then flows from A8 to the SOC, hence the low is limited by the SOC. In the case of the 410 SOC the output drive strength can be programmed between 2mA and 8mA or 16mA (I have forgotten which), and I believe that the default is 2mA. Above 2mA flowing through the pass transistor the voltage will start to rise. I think we need to change the default drive strength to MAX at least on the I2C ports, but probably on all of the other ports.
Lawrence King lking@qti.qualcomm.commailto:lking@qti.qualcomm.com Engineer, Sr. Staff/Manager Qualcomm Canada Inc. (905)482-5403 desk (x25403) (416)627-7302 cell
From: glikely@secretlab.camailto:glikely@secretlab.ca [mailto:glikely@secretlab.camailto:glikely@secretlab.ca] On Behalf Of Grant Likely Sent: Monday, November 02, 2015 7:41 PM To: mezzanine@lists.96boards.orgmailto:mezzanine@lists.96boards.org; King, Lawrence <lking@qti.qualcomm.commailto:lking@qti.qualcomm.com>; George Grey <george.grey@linaro.orgmailto:george.grey@linaro.org>; David Mandala <david.mandala@linaro.orgmailto:david.mandala@linaro.org>; 96boards-team <96boards-team@linaro.orgmailto:96boards-team@linaro.org> Subject: Debugging i2c issues
Hi all,
(This got interesting enough that I ended up also making it a blog post here: http://www.secretlab.ca/archives/164)%E2%80%8B
I've made some progress on the i2c issues seen with the sensors board. I hooked up a pca9306 level shifter and compared the behaviour on the sensors board using an oscilloscope. All of this testing is done with Seeed's Grove RGB LCD module. The LCD module is useful because it actually has 2 i2c devices embedded in it, the LCD controller at address 0x3e, and the RGB controller at 0x62. The two devices operate independently with different electrical properties.
On Hikey+sensors (TXS0108) the LCD locks the bus, and RGB will work, but only after pulling the ribbon cable apart to reduce crosstalk due to insufficient pullups On Hikey+pca9306 the LCD isn't detected and the RGB works correctly (undetermined if there are crosstalk issues)
The traces below show both sides of the level shifter. Green and blue on the top for the data line. Orange and purple on the bottom with the clock.
First, what I saw on using Hikey+pca9306+RGB: [Inline image 1]
And with the LCD:
[Inline image 2]
In both traces you can see the start condition (data goes low while clock is high), the 7 bits of address (7 rising clock edges), the R/W bit (1 rising clock), and then the acknowledgement bit driven by the device. If the controller doesn't see the device drive the data line low on the 9th clock, then it decides the device isn't there and it terminates the transaction. It is easy to recognize the ack bit because the device has a different drive strength and the voltage level is different.
The RGB controller is a happy little device and it jumps at the chance to drive the data line low. It goes right down pretty close to 0V. The LCD on the other hand is sulky and doesn't drive the line quite as low as the controller can. About to 1V. 1V is probably recognized as a logic low on a 5V device, but with 1.8V it is not even less than half. The way the pca9306 level shifter works is there are pull-up resistors on either side of the device that draws each side up to its respective high level. In this case, 1.8V and 5V. When either side gets driven low, the level shifter begins to conduct and the other side also gets drawn down to the same voltage, but it can only go as low as the voltage it is driven to. If it only gets driven down to 1V, then it will never get low enough for a 1.8V controller to recognize it as a low state.
It may be that with weaker pull-ups the LCD will be able to drive to a lower voltage level. I'll need to experiment more, but in the mean time let's move onto the Sensors board. Back to the traces:
First, here is a transaction to address 0x63 with no device present:
[Inline image 3]
Looks perfectly normal so far. Next, the RGB device at address 0x62:
[Inline image 4]
Also behaving the same way as it did with the pca9306. Finally, an LCD transaction:
[Inline image 5]
Again we see the start condition, the 7 data bits and 1 r/w bit, but the ack bit looks weird. The LCD successfully drives the data bit low enough to be recognized, but then something weird happens. The data line stays low and the clock stops running. I don't know actually know what is happening here, but I've got my suspicions. The LCD is continuing to drive the data line low, (you can tell by the slightly different voltage level) but keeping data low should not stop the clock. I suspect the txs0108 is getting confused and driving the clock line high. I've come across reports from others having trouble with the txs010x series on i2c. It has 'one-shot' accelerators to reduce rise time by driving the line high.
Now I need to decide what to do next. Rev B of the design is ready, but there is still the i2c issue that needs to be solved. I may need to put the pca9306 parts into the design which will require some more rework before it goes to manufacturing.
g.
-
Grant Likely -
King, Lawrence