ARM & caches (closed)

[Sebastian Sumpf]

Hi Dmitry,

On 02/28/2013 09:31 PM, Sebastian Sumpf wrote:
> Hi Dmitry,
> 
> On 02/27/2013 04:20 PM, Dmitry Shubin wrote:
>> On 02/20/2013 08:54 PM, Sebastian Sumpf wrote:
>>> Hi Adam,
>>>
>>> On 02/18/2013 07:30 PM, Adam Lackorzynski wrote:
>>>> On Mon Feb 11, 2013 at 19:42:47 +0100, Sebastian Sumpf wrote:
>>>>> On 02/07/2013 09:38 PM, Adam Lackorzynski wrote:
>>>>>> On Wed Feb 06, 2013 at 11:27:51 +0100, Sebastian Sumpf wrote:
>>>>>>> On 01/29/2013 12:09 AM, Adam Lackorzynski wrote:
>>>>>>>> On Fri Jan 25, 2013 at 00:00:26 +0100, Sebastian Sumpf wrote:
>>>>>>>>> On 01/17/2013 11:31 PM, Adam Lackorzynski wrote:
>>>>>>>>>> On Thu Jan 17, 2013 at 17:03:36 +0100, Sebastian Sumpf wrote:
>>>>>>>>>>> I recently upgraded Fiasco.OC to SVN revision 42 and
>>>>>>>>>>> experience some
>>>>>>>>>>> pretty severe performance degradation compared to revision 40
>>>>>>>>>>> on the
>>>>>>>>>>> Pandaboard (SMP). It seems that 'simga0' and the root task
>>>>>>>>>>> stall for 5
>>>>>>>>>>> to 10 seconds during boot up. I tracked the issue down to be
>>>>>>>>>>> caused by
>>>>>>>>>>> the initial mapping operations, especially our root task maps
>>>>>>>>>>> all the
>>>>>>>>>>> available memory during bootstrap. Within the kernel the
>>>>>>>>>>> 'Context::xcpu_tlb_flush' is called for each mapping. The
>>>>>>>>>>> function sends
>>>>>>>>>>> an IPI (to CPU1 which is idle) and then waits for an IPI in
>>>>>>>>>>> order to
>>>>>>>>>>> signal the end of the operation. The whole operation seems to
>>>>>>>>>>> have
>>>>>>>>>>> gotten slower compared to revision 40, but I could not find many
>>>>>>>>>>> differences in the IPI-handling code. Do you have any ideas or
>>>>>>>>>>> suggestions what could cause the delay (maybe scheduling
>>>>>>>>>>> changes) and
>>>>>>>>>>> how to fix it?
>>>>>>>>>>
>>>>>>>>>> I noticed a similar/same thing but hadn't time to investigate yet.
>>>>>>>>>
>>>>>>>>> Okay, I just wanted to make sure that the problem is not at our
>>>>>>>>> side nor
>>>>>>>>> at our usage pattern.
>>>>>>>>> Another thing I wonder is: Since you now have second level cache
>>>>>>>>> support
>>>>>>>>> for the PandaBoard, how do I map DMA memory to a client? The
>>>>>>>>> problem
>>>>>>>>> seems to be that sigma0 maps all memory as cached. So what we
>>>>>>>>> have been
>>>>>>>>> trying to do is this: When someone requests DMA memory we map
>>>>>>>>> the page
>>>>>>>>> as uncached and then call 'l4_cache_dma_coherent' afterwards. This
>>>>>>>>> doesn't seem to work out well for our drivers. The thing I think
>>>>>>>>> I could
>>>>>>>>> gather is that memory that is mapped cached (sigma0, roottask) and
>>>>>>>>> uncached (client) at the same time has an undefined behavior (I
>>>>>>>>> might be
>>>>>>>>> wrong here) on ARM. So, what is the protocol to implement this on
>>>>>>>>> Fiasco.OC/L4RE setups?
>>>>>>>>
>>>>>>>> Indeed, having memory with different attributes must be avoided.
>>>>>>>> But it's also about accessing that memory. So for example for sigma0
>>>>>>>> this isn't a problem because sigma0 does not touch the memory
>>>>>>>> itself.
>>>>>>>> Is your roottask accessing the memory, i.e. pulling it into caches?
>>>>>>>
>>>>>>> Yes this is the problem we're trying to solve. We don't have a
>>>>>>> notice of
>>>>>>> normal RAM and DMA pools within our roottask (if we had this, the
>>>>>>> question of how to dimension DMA pools would arise, also this
>>>>>>> seems to
>>>>>>> be an ARM only issue). So here is what we did with L1 caches enabled
>>>>>>> only: Acquire the memory in roottask, map it to the client as
>>>>>>> non-cacheable, zero out the memory (as it might have been previously
>>>>>>> used as normal RAM and for security reasons), clean and invalidate
>>>>>>> the
>>>>>>> data-cache. With the L2-cache enabled on PandaBoard, I tried to
>>>>>>> use the
>>>>>>> 'l4_cache_dma_coherent' function to accomplish the same behavior,
>>>>>>> which
>>>>>>> worked out well for the L1-case ... but it didn't. So, what am I
>>>>>>> doing
>>>>>>> wrong here, or isn't this supported anyways?
>>>>>>
>>>>>> Sounds reasonable to me. Could you check whether
>>>>>> l4_cache_dma_coherent_full makes a difference in your setup?
>>>>>
>>>>> Thanks for the 'coherent full' hint, it does work. I gonna double check
>>>>> the addresses we hand over in the 'dma coherent' case next.
>>>>
>>>> Please check the current version, there are fixes for both the IPC issue
>>>> and hopefully also for the L2 issue.
>>>
>>> Good stuff! The IPC issue is gone completely and the board starts as
>>> fast as in the single core case. The cache issue is unfortunately still
>>> around. The strange thing is, that it only seems to occur when using our
>>> HDMI driver. One can observe small black lines in the lower left corner
>>> which seem to be 32 bytes (cache-line size?) long and it looks like they
>>> occur on page boundaries only. Beats me. All other drivers seem to work
>>> fine.
>>>
>>> Thanks for your efforts,
>>>
>>> Sebastian
>>>
>>>
>>> _______________________________________________
>>> l4-hackers mailing list
>>> l4-hackers at os.inf.tu-dresden.de
>>> http://os.inf.tu-dresden.de/mailman/listinfo/l4-hackers
>>>
>>
>> Hi, Sebastian.
>>
>> I've just played around with demo that comes with Genode
>> (git: ce075c05b9f54) a bit and my experience differs from yours.
>> I'll try to summarize what I've found so far.
>>
>> (1) Issues with u-boot:
>>
>> (1.a) I've been told elsewhere that CONFIG_SYS_L2CACHE_OFF option
>> matters when building u-boot. It does not seem to be the case however,
>> present or not it does not affect observed behavior. Anyway, that is
>> just something to keep in mind.
> 
> Indeed, it doesn't matter any more, since Fiasco.OC revision 42 the
> kernel will enable the L2-cache, so u-boot is out of the game.
> 
>> (1.b) u-boot version on the other hand matters. For instance omap_fb
>> doesn't work with u-boot from master branch (47104c37), nothing shows
>> up on the screen. I've got better luck with the version tagged
>> 'v2012.04.01'.
> 
> Yes, it's always a bit tricky to find the right version that initializes
> stuff correctly.
> 
>> (2) Black lines in the corner of the screen: I've actually observed
>> them without L2 cache turned on. The effect is less pronounced
>> (usually just 1 or 2 lines), they don't show up every time (!) and
>> disappear after you click anywhere on the screen or just drag a mouse
>> cursor over. So, this may or may not be L2 issue.
> 
> I did by now too. It is a cache issue and does not show up this often
> because the L1 cache is way smaller than the L2 one.
> 
>> (3) Other drivers don't work for me as you claim they are when L2 is
>> turned on. Namely, usb_drv fails to detect whatever you plug in most
>> of the time, and when it succeeds the device doesn't work anyway.
> 
> Also true. I can tell you my current state of the affair: As mentioned
> above the ARM manual states that mapping a page as cached and uncached
> at the same time results in undefined behavior. Unfortunately DMA memory
> is mapped in 'roottask/core' as cached memory, which is enforced by the
> sigma0 protocol. So what the roottask does, when a client requests DMA
> memory, is to zero out that cached memory, clean and invalidate the
> caches, and map the page 'uncached' to the client. What happens next is
> very weird. We inspected the caches after cleaning them (we cleaned them
> completely for the test) using a hardware debugger, they are indeed
> empty/all lines invalid, we also made sure that there are no accesses to
> the cached memory from roottask's side and guess what? After some
> interaction with this memory on the uncached client side, there appear a
> lot of valid zeroed out cache lines in both caches. After a while these
> cache lines will be evicted, which in turn causes memory corruption at
> the client side. Today one of my colleagues suggested to clear the data
> TLB as well, but that made no difference.
> If we omit the cleanup of the DMA memory, which we don't want to, at the
> roottask everything works out fine.
> So what is the problem here? Maybe these are the results of the
> 'undefined behavior'. I don't know for sure. If anyone on this list
> finds out something, let me know.
> 

Problem solved and just for the sake of documenting the outcome: Our
roottask does indeed touch the memory before it maps it to the client.
It was, how should I say it, just a little hard to find, but relieved me
from almost believing in Santa Claus again .-)
A patch will appear soon,

Sebastian