L4Re: Boot module copying and the MIPS Creator CI20
Hello, As part of reviewing my earlier efforts getting L4Re running on various MIPS- based devices, I have been trying to get the bootstrap module to start the kernel on the MIPS Creator CI20, and here I encounter a strange problem. The bootstrap code appears to copy the different modules in the payload around in memory, and for the kernel, it finds the different sections at 0x2f4000 in physical memory (accessed via 0x802f4000), copying them into locations relative to the indicated module base address (for example, 0x1100000 or 0x2000000). From their new locations, each section of the kernel is then copied to its desired final address. So, the section of the kernel providing its entry point can be found at 0x2f4000 + 0xa411c, this being copied to base + 0xa411c, before finally being copied to 0x800d3000. However, the contents of memory at the final location (0x800d3000) are not correct, nor are they at the "staging" location of base + 0xa411c, even though they do correspond to the expected contents of the kernel binary at their original location. What I expect to see at the entry point is... 40806000 mtc0 zero,c0_status 000000c0 ehb What I see instead is... ffffffff bea3ffdf Apart from at the original location in the payload (0x2f4000 + 0xa411c). As far as I can tell, the second word is nonsense, and it is usually followed by more ffffffff words. So, it is as if copying failed for this particular section. Oddly, this only seems to affect the section of the kernel providing the entry point, and all other sections appear to be copied correctly from their original locations. One factor might be that this section is the last one that is copied. Does this seem like recognisable behaviour of some kind of misconfiguration? For the most part, I am just rebasing my old patches on the current L4Re and Fiasco versions, so I didn't expect a problem at this level. Regards, Paul
On Wednesday, 26 April 2023 17:27:35 CEST Paul Boddie wrote:
As part of reviewing my earlier efforts getting L4Re running on various MIPS- based devices, I have been trying to get the bootstrap module to start the kernel on the MIPS Creator CI20, and here I encounter a strange problem.
[Invalid data at the kernel entry point]
Does this seem like recognisable behaviour of some kind of misconfiguration? For the most part, I am just rebasing my old patches on the current L4Re and Fiasco versions, so I didn't expect a problem at this level.
Further troubleshooting suggests that what happens is as follows... Boot_modules::_move_module copies data from its original location as part of the boot payload to locations relative to the module base address. This copying is done using "virtual" addresses, even though the bootstrap module outputs "physical" addresses in the log. So, for the offending kernel code: moving module 00 { 2f4000-39a537 } -> { 2000000-20a6537 } [681272] Here, the actual copying is done using these "virtual" addresses: 802f4000-8039a537 -> 82000000-820a6537 As a reminder, these "virtual" addresses are KSEG0 addresses, meaning that they correspond directly to the previously indicated addresses, but the address mapping is done by the processor, effectively clearing the top bit of each address. Subsequently, the data is copied again to its final, desired location by l4_exec_read_exec. However, this function uses entirely "physical" addresses, and it is here that the bad data was being found and copied for execution. For example: 2000000-20a6537 -> d3000-d5097 As another reminder, these "physical" addresses are actually virtual addresses that are set up in an identity mapping in the _start routine in the appropriate crt0.S file. What I discovered was that whilst bad data was available via the "physical" addresses, valid data was available via the "virtual" addresses. Thus, it was possible to get the l4_exec_read_exec function to see the appropriate data by employing "virtual" addresses when copying. Alternatively, by making the _move_module function use the "physical" addresses, I could get the l4_exec_read_exec function to see the appropriate data that way. An alternative to changing the addressing schemes employed by these functions was to instead flush the data cache in _move_module. This appears to synchronise the views of memory that the KSEG0 mapping and the configured mapping have. However, after doing this, and demonstrating that I can invoke the kernel code (using the low-technology method of changing the colour of the CI20's power LED), I find that the instruction clearing the processor's status register seems to halt execution. To me, this suggests that something else is wrong with the memory mapping, but I can't think of what this is at the moment. It is very odd that this code doesn't work any more. Furthermore, I use the QEMU emulation of the MIPS Malta board all the time, and it doesn't have such problems. I can imagine that there are details of the cache management that are different between the emulated Malta processor and the genuine CI20 processor, but it doesn't really explain how this worked on the CI20 before. I suppose that all the necessary cached data might have been written to memory purely by chance. Anyway, that is as far as I have managed to get, unfortunately. I did attempt to try the older Subversion-based L4Re distribution, but I encountered a long- forgotten issue with the build system and didn't have the appetite to continue. Any insights would be welcome. Paul
On Saturday, 29 April 2023 22:10:17 CEST Paul Boddie wrote:
Anyway, that is as far as I have managed to get, unfortunately. I did attempt to try the older Subversion-based L4Re distribution, but I encountered a long- forgotten issue with the build system and didn't have the appetite to continue. Any insights would be welcome.
Well, I reactivated the old Subversion-based L4Re/Fiasco distribution and it still works on the CI20. The build system problems are superficial errors that occur when running "make B=..." in the l4re directory and are related to various configuration tests that cannot find system headers. Ignoring these errors and just configuring and building yields a working image. I decided to test the old L4Re distribution with the new kernel and it seems to fail when clearing the status register in the kernel's start routine. Similarly, the new L4Re distribution (obtained using ham) seems to fail in the same way, either with the old or new kernels. Usually, when clearing the status register and, crucially, the ERL flag, any pending exception might be delivered and this may well be undesirable, especially if the exception handlers are not set up. Although a watchpoint exception is recorded in the cause register, there doesn't seem to be any indication that this exception is actually pending. I am actually rather perplexed by the code in the bootstrap package's crt0.S file. It supposedly sets up a memory mapping for the first 512MB of memory using a pair of 256MB pages, but this is apparently superfluous if the CPU is already handling an error (ERL set in the status register), with an identity mapping being active for the first 2GB of memory in such cases. I can only presume that some machines have bootloaders that clear ERL before the bootstrap code is invoked. Another concern is that the CI20 does not actually support 256MB pages, and so the mapping would actually only map two 16MB regions starting 256MB apart. I also find the mixing of virtual and physical addresses in the bootstrap code, as described previously, to be troubling. Currently, I remain mystified by this problem. The bootstrap code has changed, but arguably not in substantial architecture-specific ways. Similarly, the kernel code remains unchanged in such ways, although the kernel payload is now somewhat bigger. Any insights would be welcome, certainly. Paul
On Monday, 8 May 2023 00:47:01 CEST Paul Boddie wrote:
I decided to test the old L4Re distribution with the new kernel and it seems to fail when clearing the status register in the kernel's start routine. Similarly, the new L4Re distribution (obtained using ham) seems to fail in the same way, either with the old or new kernels.
In fact, I was mistaken about this. The kernel does in fact run, and through laborious debugging efforts, I think I have established that it manages to hang when running Banner::init, more specifically during the underlying __v_printf routine, in that it enters that routine but never manages to successfully leave. I did wonder if this might be due to the UART details not being successfully set up, but the bootstrap code seems to identify the appropriate region for the kernel options (defined by the .koptions section in the fiasco payload) and to copy the boot options into it appropriately. I cannot actually verify that the kernel reads the options correctly, though, since it doesn't produce any UART output and it appears that the UART may be part of the problem.
Currently, I remain mystified by this problem. The bootstrap code has changed, but arguably not in substantial architecture-specific ways. Similarly, the kernel code remains unchanged in such ways, although the kernel payload is now somewhat bigger.
Regardless of the observations made above, the cache-related issues still apply, meaning that I have to flush the data cache after copying regions of the different payloads around, even though this was not necessary before. I guess I will need to think of other techniques to deduce what is happening. It is obviously rather frustrating. Paul
Hi Paul, I did some unsuccessful poking around in the manual and the MIPS code. I just stumbled across the following:
I can only presume that some machines have bootloaders that clear ERL before the bootstrap code is invoked.
Judging from this sentence in the Status.ERL description of the manual: "The operation of the processor is UNDEFINED if the ERL bit is set while the processor is executing instructions from kuseg." I say that this bit should be cleared. The two 256MB regions also point to the intention that Status.ERL is clear and thus there is no unchached+unmapped useg. I'm sorry to not have more helpful comments. Cheers Philipp
On Wednesday, 10 May 2023 10:52:59 CEST Philipp Eppelt wrote:
I did some unsuccessful poking around in the manual and the MIPS code. I just stumbled across the following:
I can only presume that some machines have bootloaders that clear ERL before the bootstrap code is invoked.
Judging from this sentence in the Status.ERL description of the manual: "The operation of the processor is UNDEFINED if the ERL bit is set while the processor is executing instructions from kuseg."
That is rather interesting. The code should be running from KSEG0, though, since the program counter (instruction pointer) is operating in the region from 0x80000000 to 0xafffffff. The bootstrap code does access data in KUSEG, but this region also supposedly employs an identity mapping to physical addresses when ERL is set, as noted in an earlier remark. I have read that in the earliest part of bringing up a system, code should be running in the uncached, unmapped KSEG1 area and it should set up the caches before branches are taken elsewhere. Indeed, this is effectively what happens when the bootstrap vectors are used, and so the processor will run code from 0xbfc00000 or thereabouts - it's in on-board ROM in the Ingenic SoCs - and then dispatch to another address. With the CI20, as with many other boards of this nature, U-Boot performs some initialisation before passing control to things like L4Re's bootstrap module. Indeed, it tends to set ERL in order to avoid causing exceptions in any such initialisation code. When searching for discussion about ERL, I also found a patch for Linux that deferred the clearing of ERL within the kernel to a later point for precisely this reason.
I say that this bit should be cleared. The two 256MB regions also point to the intention that Status.ERL is clear and thus there is no unchached+unmapped useg.
I agree. I'll try and see if this can be done, but there did seem to be something preventing it when I tried before.
I'm sorry to not have more helpful comments.
I appreciate your comments, anyway! Thank you for reading and replying, Paul
On Wednesday, 10 May 2023 12:35:10 CEST Paul Boddie wrote:
On Wednesday, 10 May 2023 10:52:59 CEST Philipp Eppelt wrote:
I say that this bit should be cleared. The two 256MB regions also point to the intention that Status.ERL is clear and thus there is no unchached+unmapped useg.
I agree. I'll try and see if this can be done, but there did seem to be something preventing it when I tried before.
Well, I persisted here and found that trying to map 256MB pages will not work on the CI20, as the JZ4780 programming manual indeed suggests, but 16MB pages do work. One particularly odd thing about the existing bootstrap code is that it tries to map a pair of 256MB pages at the base of physical memory (corresponding to the base of virtual memory) with the second one described by the EntryLo1 register as follows: li $8, 0x0080001F // odd page @256MB, cached, dirty, valid, global mtc0 $8, $3, 0 // mtc0 t0, c0_entrylo1 Documentation of the MIPS architecture in various forms uses terms like page frame number (PFN) to refer to the bit range 31..6 in EntryLo0/1 which might be misinterpreted as some kind of multiplier of any configured page size. This is not the case. Nor are the semantics of these registers consistent with various other registers (EntryHi, for instance) where one might interpret the available bits in their actual positions, which might mean that one merely sets the desired physical address combined with the control bits. The JZ4780 manual, in its terseness, actually clarifies the matter concisely by indicating that the PFN corresponds to the bits from 12 upwards in the translated physical address, with a bit range of 25..6 actually being utilised by the hardware for this purpose and thus corresponding to bits 31..12 in the resulting physical address. Therefore, the value used above, 0x0080001F, yields a PFN of (0x0080001F >> 6) or 0x20000, and thus the most significant bits of a physical address of (0x20000 << 12) or 0x20000000, or 512MB from the start of memory, which isn't what the code is meant to achieve. Thankfully, the author left a comment to state their intention instead of following the modern software industry practice of insisting that the code describes itself. So, I changed the initial mapping to use the following with a 16MB page size: li $8, 0x0000001f // even page @0MB, cached, dirty, valid, global mtc0 $8, $2, 0 // mtc0 t0, c0_entrylo0 li $8, 0x0004001f // odd page @16MB, cached, dirty, valid, global mtc0 $8, $3, 0 // mtc0 t0, c0_entrylo1 This made it possible to clear the ERL flag in the status register and copy modules around in memory using only KUSEG addresses, avoiding the odd mixing of KUSEG and KSEG0 addresses. I also wanted to see if I could use the UART via its physical memory locations in the 0x10000000 region. I found that the following enabled this: mtc0 $8, $5, 0 // mtc0 t0, c0_pagemask li $8, 0x00400017 // even page @256MB, uncached, dirty, valid, global mtc0 $8, $2, 0 // mtc0 t0, c0_entrylo0 li $8, 0x00440017 // odd page @256+16MB, uncached, dirty, valid, global mtc0 $8, $3, 0 // mtc0 t0, c0_entrylo1 Changing the CI20's platform file configuration to not use KSEG1 to access the UART was successful, although this reliance on a global memory mapping is probably not desirable, and I imagine that the kernel unmaps these initial mappings, anyway. Consider this an experiment! Reviewing my previous concern about caching issues and inconsistencies between memory regions, I think that the incorrect EntryLo1 value was causing confusion about where data was being copied. Indeed, having fixed this value, I can use KSEG0 addresses when initially moving modules and then KUSEG addresses when moving them into their final positions. All of this gets the kernel started in various ways, as I previously achieved, but it still doesn't manage to complete its initialisation and still appears to hang in vprintf, so there is something else amiss. Paul
On Friday, 12 May 2023 01:23:14 CEST Paul Boddie wrote:
Reviewing my previous concern about caching issues and inconsistencies between memory regions, I think that the incorrect EntryLo1 value was causing confusion about where data was being copied. Indeed, having fixed this value, I can use KSEG0 addresses when initially moving modules and then KUSEG addresses when moving them into their final positions.
Actually, I have to refine this explanation. Even if I fix up the EntryLo1 value to set up an identity mapping for two adjacent 16MB pages at the base of KUSEG, I find that if ERL is set, there is some kind of caching or addressing issue for the executable part of the kernel payload (as originally described), whereas without ERL being set, that issue goes away. Paul
On Friday, 12 May 2023 01:23:14 CEST Paul Boddie wrote:
All of this gets the kernel started in various ways, as I previously achieved, but it still doesn't manage to complete its initialisation and still appears to hang in vprintf, so there is something else amiss.
Well, I can't figure this out at all. Debugging when the UART isn't even set up in the kernel is very difficult, but I managed to get the on-board LEDs to suggest that the failure has an exception code of 5, related to a store operation that is either in unmapped memory or is misaligned. Tedious exercises trying to extract the failure address or instruction haven't been informative. I tried with limited success to call the bootstrap code's write function for UART output, which required manually setting up the t9 register, but I don't think this can be relied upon. I would potentially try and revert the sources in Git to something approximating to the r83 version from Subversion, but everything is now a collection of repositories, and I don't see any obvious way of using the Ham tool to perform such an operation. It is also difficult to know what the repositories would be reverted to, especially given the way the history appears to integrate commits from all sorts of dates, not necessarily in chronological order. Since I've been using QEMU to emulate the MIPS Malta board, it is odd that I don't see problems running the software under emulation, but I am admittedly not using precisely the same repository versions. Maybe the arguably incorrect bootstrap initialisation is somehow forgiven by the Malta emulation but not the CI20 hardware. What should have been a simple rebasing of a few patches, with some even eliminated due to the integration of various changes (including some I suggested several years ago), seems to have turned into a futile exercise with really no obvious indication as to why the code no longer works. To be honest, it is all rather frustrating and disappointing, but such is the way of the world, I guess. Paul
On Monday, 15 May 2023 01:35:26 CEST Paul Boddie wrote:
What should have been a simple rebasing of a few patches, with some even eliminated due to the integration of various changes (including some I suggested several years ago), seems to have turned into a futile exercise with really no obvious indication as to why the code no longer works. To be honest, it is all rather frustrating and disappointing, but such is the way of the world, I guess.
So, I decided to establish the behaviour of the bootstrap code, which I think I have done to a reasonable extent, and to investigate whether it is the kernel that is actually causing the problems. Taking the recent bootstrap code, fixed up so that I can be sure that it is copying modules around correctly (with ERL not set), I took some old kernel versions from the Git repository. This did not yield a working system, but it did produce output beyond the kernel starting point. Along with the kernel from Subversion r83, I found that the following kernel versions at least started and caused sigma0, moe and ned to be invoked: commit 1f27c9ebc9651bcd114802ccc9d250866f3742c9 Author: Adam Lackorzynski <adam@l4re.org> Date: Mon Mar 5 00:00:00 2018 +0000 commit 68eb142cf237292055663ba7635c2c1f2b0ecb92 Author: Jean Wolter <jean.wolter@kernkonzept.com> Date: Mon Dec 17 00:00:00 2018 +0000 The output they produced was the following line endlessly repeated: L4Re[rm]: unhandled read page fault at 0x7fff8fa1 pc=0x7fff8fa0 The last kernel in Git that appears to produce a semi-functional system is this one: commit 9b71796d7fef5644474c94d911c71be65f44783a Author: Frank Mehnert <frank.mehnert@kernkonzept.com> Date: Mon May 11 00:00:00 2020 +0000 This actually produces the following output which does not repeat endlessly: Ned says: Hi World! L4Re[rm]: unhandled read page fault at 0x7fff8fa1 pc=0x7fff8fa0 L4Re: rom/ned: Unhandled exception: PC=0x7fff8fa0 PFA=0x7fff8fa0 LdrFlgs=0x0 After that particular commit, I see that there was some work done on the UART functionality. Eventually, it settles down with the following commit: commit 42c88cc84e183c8234127b4c1e88dc946ca33cb0 Author: Alexander Warg <alexander.warg@kernkonzept.com> Date: Mon Apr 6 00:00:00 2020 +0000 Fix lib uart related compile issues * MIPS ci20 * ARM IMX6 UL * ARM SA1100 This commit and any subsequent commits I have tested do not produce output and appear to hang. I noticed that the changes to the CI20 support introduce uart_16550_dw.o, but all the other boards that introduce such functionality previously had some kind of wrapper code that this object file apparently replaces. I wonder whether this is actually appropriate for the CI20. I also noticed that the UART peripheral was probably not initialised in the old kernel, with the F_skip_init flag being set in the UART driver. Recent code has eliminated this behaviour, so I decided to explicitly disable initialisation myself. The result was to restore the UART output and to yield the same kind of output as earlier kernel versions, producing the Ned-related error. Finally, I updated to the latest changeset in my clone of the repository: commit f5bfa79ff15cbbaadada806d35dbe0560c892b24 Author: Frank Mehnert <frank.mehnert@kernkonzept.com> Date: Mon Apr 17 00:00:00 2023 +0000 With UART initialisation suppressed, I found that the Ned-related error disappeared and that my program actually ran, which was something of a surprise. I suppose, then, the conclusion is that the CI20 UART code got inadvertently broken when that functionality was reworked. Whether the UART initialisation should be suppressed or whether the UART can be cleanly reinitialised is something to investigate further, I imagine. Paul
On Friday, 19 May 2023 19:03:36 CEST Paul Boddie wrote:
I suppose, then, the conclusion is that the CI20 UART code got inadvertently broken when that functionality was reworked. Whether the UART initialisation should be suppressed or whether the UART can be cleanly reinitialised is something to investigate further, I imagine.
Of course I have to follow up to this! The fixes required here involve adjusting the src/kern/mips/bsp/ci20/Modules file to use the following definitions: OBJECTS_LIBUART += uart_16550.o CXXFLAGS_uart-libuart += $(call LIBUART_UART, 16550) \ -DUART_16550_INIT_FCR=0x10 If the INIT_FCR setting is not used, the initialisation code clears the trigger level field (defined in the 16550 data sheet) and also the UART enable field (defined in the JZ4780 programming manual, reserved in the 16550 data sheet), disabling the UART. For the above value, I merely reproduced the UART enable bit (0x10) set by the Uart_16550 initialisation in the bootstrap code. The trigger level field can be read, yielding a value of 0xc0 which corresponds to (3 << 6) or a level of 60 according to the JZ4780 manual, but the field is documented as being write- only, so maybe reading it doesn't make sense. Allowing that field to be cleared does not appear to be harmful. (I see that the drivers-frst code strongly resembles the library code in the kernel but is slightly older, and I wonder about whether these things could eventually be integrated in a sensible way.) Alongside these changes, the CI20 still needs instruction emulation for rdhwr, since this instruction is still used by various libraries, even though the bootstrap code avoids it. Meanwhile, I still find that the bootstrap crt0.S file needs patching to fix the memory mapping and to not run with the ERL flag set. Fixing the UART handling in the kernel does not make that issue disappear. I hope this was informative, at least. Paul
Hi Paul, On Sat May 20, 2023 at 00:35:37 +0200, Paul Boddie wrote:
On Friday, 19 May 2023 19:03:36 CEST Paul Boddie wrote:
I suppose, then, the conclusion is that the CI20 UART code got inadvertently broken when that functionality was reworked. Whether the UART initialisation should be suppressed or whether the UART can be cleanly reinitialised is something to investigate further, I imagine.
Thanks for digging through this.
Of course I have to follow up to this! The fixes required here involve adjusting the src/kern/mips/bsp/ci20/Modules file to use the following definitions:
OBJECTS_LIBUART += uart_16550.o CXXFLAGS_uart-libuart += $(call LIBUART_UART, 16550) \ -DUART_16550_INIT_FCR=0x10
If the INIT_FCR setting is not used, the initialisation code clears the trigger level field (defined in the 16550 data sheet) and also the UART enable field (defined in the JZ4780 programming manual, reserved in the 16550 data sheet), disabling the UART.
For the above value, I merely reproduced the UART enable bit (0x10) set by the Uart_16550 initialisation in the bootstrap code. The trigger level field can be read, yielding a value of 0xc0 which corresponds to (3 << 6) or a level of 60 according to the JZ4780 manual, but the field is documented as being write- only, so maybe reading it doesn't make sense. Allowing that field to be cleared does not appear to be harmful.
Thanks, we've changed this.
(I see that the drivers-frst code strongly resembles the library code in the kernel but is slightly older, and I wonder about whether these things could eventually be integrated in a sensible way.)
Frank did work in this area recently to move this closer together.
Alongside these changes, the CI20 still needs instruction emulation for rdhwr, since this instruction is still used by various libraries, even though the bootstrap code avoids it. Meanwhile, I still find that the bootstrap crt0.S file needs patching to fix the memory mapping and to not run with the ERL flag set. Fixing the UART handling in the kernel does not make that issue disappear.
I've also queued a change to disable ERL. Not sure on the other issue yet. Adam
On Sunday, 21 May 2023 21:28:40 CEST Adam Lackorzynski wrote:
Hi Paul,
On Sat May 20, 2023 at 00:35:37 +0200, Paul Boddie wrote:
For the above value, I merely reproduced the UART enable bit (0x10) set by the Uart_16550 initialisation in the bootstrap code. The trigger level field can be read, yielding a value of 0xc0 which corresponds to (3 << 6) or a level of 60 according to the JZ4780 manual, but the field is documented as being write- only, so maybe reading it doesn't make sense. Allowing that field to be cleared does not appear to be harmful.
Thanks, we've changed this.
(I see that the drivers-frst code strongly resembles the library code in the kernel but is slightly older, and I wonder about whether these things could eventually be integrated in a sensible way.)
Frank did work in this area recently to move this closer together.
One thing that did not seem to work as expected was the propagation of the UART information to the kernel, and for some time I wondered if the structure set up in the bootstrap module was not getting read correctly by the kernel, perhaps due to inscrutable caching issues, or that the address set in the structure was not being translated correctly in the kernel. In the startup function, we see this: lko->uart = kuart; lko->flags |= kuart_flags; The lko pointer refers to the kernel options, of course, so one would expect that the kernel picks up all these options. However, looking at the platform code, the UART is initialised separately for the bootstrap code: static L4::Uart_16550 _uart(kuart.base_baud, 0, 0, 0, 0x10 /* FCR UME */); Obviously, I now know what "FCR UME" is supposed to mean. FCR (FIFO control register) is reset in the kernel, which caused the UART to be disabled again. So, maybe more of the settings need to be transferred.
Alongside these changes, the CI20 still needs instruction emulation for rdhwr, since this instruction is still used by various libraries, even though the bootstrap code avoids it. Meanwhile, I still find that the bootstrap crt0.S file needs patching to fix the memory mapping and to not run with the ERL flag set. Fixing the UART handling in the kernel does not make that issue disappear.
I've also queued a change to disable ERL. Not sure on the other issue yet.
A long time ago, we discussed rdhwr and I think it was suggested that the place for supporting some of the functionality could be outside the kernel, with the kernel invoking a user space handler, although at the time I didn't see how that would be possible given that it is used in these two ways: 1. To provide cache size information (SYNCI_Step) for the synci instruction. 2. To provide access to the ULR register which offers thread-local storage. SYNCI_Step is read from a privileged register, but since it is constant, I imagine that it could be stored somewhere and retrieved by a user space handler. ULR appears to be cached in the thread context structure, if I remember and interpret the exception handler correctly. I think this code was largely in place when I first investigated it, so it does not seem as familiar to me. I hope this is helpful, anyway. Paul
participants (3)
-
Adam Lackorzynski -
Paul Boddie -
Philipp Eppelt