December 9, 2009

OpenEmbedded/Ångström Kernel Workflow

by Graeme Gregory

This article is to detail the workflow I personally use when I am doing kernel development for devices supported by OE. I find OE very useful for this as I can use it to build the toolchain and ultimately to control my patch tree until I am ready to send the patches upstream.

So I select a kernel which I wish to develop with, in my case this is in recipes/elphel/

I first make sure I am starting from clean:

bitbake linux-elphel -c clean

Then take the kernel as far as the configuration stage, this makes sure all patches in the metadata are applied and that the defconfig has been copied to .config and make oldconfig has been run.

bitbake linux-elphel -c configure

Now I switch to another window where I shall be actually editing the code. I change to the temporary working directory of the kernel I am working with. This path below will change depending on kernel version or name. Kernels are always found in the machine workdir so tmp/<machinename>-angstrom-linux-gnueabi/

cd tmp/work/elphel-10373-angstrom-linux-gnueabi/linux-elphel-2.6.31+2.6.32-

At this point I normally elect to use quilt to temporarily manage my patches so.

quilt new new-feature.patch

And to add files to this patch, I make sure to do this before I make any edits as the diff ends up being the diff from when this is first called to the current state.

quilt add driver/camera/random.c


November 23, 2009

Temporary diversion

by Andrey Filippov

Not much progress with the 10373 board in the last 3 weeks – just cleaned up debug wiring and prepared the board for the visit from SlimLogic Ltd. The software for the 373 camera is going to be based on OpenEmbedded and I would like to set up everything correctly before getting deep in the modifying the existent codebase.

Meanwhile we tested the redesigned sensor board (will work with the 373 camera too) to make it smaller. All the current sensor boards (but the ones with a large Kodak CCD) were the same size – 31.4mm x 31.4mm – it did not change since the times of old Zoran 1.3MPix sensor. Since then the sensors became smaller and less components were needed for the power conditioning too.

The sensor board size is especially important for the panoramic video were it is desirable to keep lenses as close together as possible to reduce the effect of the parallax. For the still imagery panoramic photos it is possible to rotate a camera precisely around the entrance pupil of the camera lens, but for the video (or even some imagery) it is not a solution.

October 30, 2009

GigE is working

by Andrey Filippov

10373 - GigE is tested

10373 - GigE is tested

We did not do much this week as it was a really busy one. But one important milestone is passed – GigE is working! It was rather easy to see that 10373 pings other devices, it’s PHY “receive” LED blinked back in response, but a ping command from u-boot itself failed.
It took a while to realize that u-boot only sets internal array (used to set MAC address at each network command) from environmental variable once at a boot time. Confusing was that the manually set value was used in the payload of the ARP packet (as seen with Wireshark), but the hardware MAC registers remained with default FF:FF:FF:FF:FF:FF and so response to the ping was rejected. I still do not know what was supposed to be set, but after adding update of that internal array – ping worked, and tftp too. Still do not have working Linux kernel+ root fs, but no need to wait for like 20 minutes to transfer it through the serial port.

October 19, 2009

U-Boot ported to Elphel 10373 board

by Andrey Filippov

U-Boot running on Elphel 10373 board

U-Boot running on Elphel 10373 board

Finally U-Boot is running on the new board – it took two full days (and partially -nights) to make that happen. Initially I had no idea what exactly happened when control was passed to u-boot (I already knew that memory is OK and the u-boot image ion the board’s memory is the same as generated on the host and serial port software was under suspicion, so I looked for an independent (of the UART) way how the software could tell me that it reached some points. Obvious solution – general purpose I/Os, but unfortunately most of them are either routed on the inner layers or have multiple functions (shared I/O pins) – and I did not know if any of the software (ROM boot, first stage UBL or U-Boot itself) control them. Just one GPIO pin was not shared and had easy access – connected to the 64-pin QFN package with 0.5mm pitch pins – pins are on the sides, not under the belly as on the BGA ones, so I just soldered a wire to that pin and was able to watch it with oscilloscope.

With that one-bit “printf” life got easier – I was able to find that the serial port output was the one having problems. In most cases it just silently got stuck, but sometimes the serial port was sending out 5-bit zero data (seems like control register that was programmed to 8n1 initially, got zero written to it and switched to 5n1. I still do not understand what did I do wrong and why supposedly working software failed in my case. I resolved the problem by replacing the 8-bit UART registers (separated by 3 dummy bytes) with the 32-bit ones – same as was done in the first initial bootloader that I already had working. With that small mystery unresolved the u-boot is now running as you can see on the screenshot.

October 17, 2009

Testing the new board

by Andrey Filippov

Testing progress of the 10373 board
As the board survived the power-up test I was able to start testing more tricky parts – those that require software to run in the processor. As I never dealt with TI DaVinci (or any other ARM processors) before it is going slowly. Production cameras will probably run some flavor of the OpenEmbedded, but for hardware testing I decided to take a shortcut and use whatever is recommended and is available for download on the processor manufacturer’s site – Texas Instruments. And that is based on Monta Vista distribution. I had to agree with their EULA that I took time to read carefully. It seems that license prohibits me to even describe my experience with it. So I will not do that – anyway it is a temporary solution that will never go beyond a single prototype board.

Next thing to do was to try to boot something to the processor and here again – I used the recommended bootloader software that (host part of it that communicates with the program running on the target – the processor being booted) turned out to be written in C# – the language that I thought I’ll never have to use. But – that thought turned out false, especially as there was nice documentation how to use mono, and explained that I need to upgrade the version that comes with Ubuntu otherwise the bootloader will get stuck. And it did really, so that recommendation is still current – at least for the 9.04.

With the patch it worked (at least did something), but only until I decided to recompile the software – I wanted to add some debug code to get some proof that it loads anything. The software is provided as an archive that includes both source and built code and initially I tested just the compiled version. Compilation failed. First I found that I can disable the non-GNU/Linux branch in the top Makefile as I did not needed it. But still it complained about the problem in the C# code itself – the problem was fixed by adding “(Byte)” before “0xFF” in one file – that was my first experience with this language.

With that solved I tried to modify configuration, compile and run u-boot (actually there is a version u-boot-ti that at least knows about the dm6467 – main branch does not have it). Spent a whole day but no success so far, and I could not be sure – is it the software/configuration bug or the system memory (2 of the DDR2 chips) do not work properly – that was what I was really afraid of.

PCB layout between CPU and DDR2 memory, top and bottom layers only (copper pour not shown)

PCB layout between CPU and DDR2 memory, top and bottom layers only (copper pour not shown)

PCB layout between CPU and DDR2 memory, all layers (copper pour not shown)

PCB layout between CPU and DDR2 memory, all layers (copper pour not shown)

That memory consists of the two fine-pitch BGA chips (two large cyan colored rectangles on the images above) , mounted directly opposite to (on the other side of the PCB) the DaVinci chip (large white square). All the signal traces (red on the images) go to the nearest blind microvia (most – only a fraction of a millimeter), then dive to the inner layer, go there (brown traces) until a buried via that brings them one layer from the other side (cyan) , finish their travel and resurface through a microvia near the destination (blue traces). Both microvias are in most cases under the chip bodies – CPU and memory. So no chance to reach them with a oscilloscope – I had only to hope that the design, PCB manufacturing and BGA assembly did not have a single error. So I put aside u-boot troubleshooting and made a dirty trick – modified the bootloader software (that I already was able to compile) so after loading u-boot code to the DDR2 memory it sent all of it back to the host in the same format as “hexdump” utility does. And all 90KB or so matched perfectly, so one of the critical milestones is over – there is working TMS320DM6467 processor that interfaces with the host over USB and has 256MB of the working DDR2 memory. Now I’m sure that there is some my misunderstanding with u-boot that holds me from running it.

The image above the posting shows the 10373 circuit diagram with the green over the parts that are tested and seem to be working as designed. There is a small red rectangle there – the problem caused by my laziness to play with cp2103 evaluation board – I missed that GPIO pins can only operate as open drain outputs, can not be configured as push-pull ones (at least I could not figure out how to do that).So I’ll have to put a small transistor there (as an inverter) on the prototype and add that to the list (so far only one entry) of the changes to the PCB in the next revision.

October 14, 2009

10373 power up OK

by Andrey Filippov

So far – so good, after installing most of the components I tried to power up the board. No smoke, power supply voltages (at least those that do not need software to be turned on) seem to be correct, none of the components are hot.

Next step was to connect USB cable to the programming port – there is a cp2103 bridge chip on the board that is connected to the serial port on TMS320DM6467 – port that can be used to boot the system while the flash memory is empty.

When I just started design with this Texas Instruments chip the datasheet was showing network boot mode (similar to Axis ETRAX we used in all our previous cameras) but later that mode disappeared from the documentation (probably TI ran out of the ROM space) so we had to use another way to program the system so it would be both

  • safe from “bricking” caused by interruption of the firmware update process (or just plain software bug – our products are designed for development, so bugs are likely) and
  • the firmware installation/upgrade would not require any special hardware connections, especially those that require opening the camera case.

The USB connection seemed nice – it is commonly used for the firmware installation, uses regular cables and the port can be used as a system/debugging console during normal operation.

I selected a small (5mm x 5mm) Silicon Labs CP2103 converter as it has additional general purpose I/O pins that can be used to switch between boot modes and generate reset signal for the CPU. This chip seemed to be widely used and to have standard Linux drivers. I was not so sure about those extra GPIO pins support, but noticed that Silicon Labs have some mention of the Linux software, so I’ve purchased their evaluation board to be able to experiment with that software without doing it on an actual board. But I never tried it until the time came to connect the 10373 board. So I opened the evaluation board box, connected it to a computer and – noticed, that there is no support for the extra pins in the regular driver that I’ve got with Ubuntu 9.10. And the driver provided by the manufacturer did not compile with the kernel I’ve got. So I had to spend half of the day figuring out how those drivers work and how to make one that will work on my computer.

That combined driver (I ported only GPIO control from Silicon Labs driver to the one provided with the system) eventually worked and I was able to turn on/off four LEDs on the evaluation board. It required some guesswork and testing as I could not find any documentation on the internal registers and which bits control setting of the GPIO pins – only their API description that assumes usage of some binary-only *.DLL files that may probably work with WINE.

Anyway – I have now a working solution – modified driver and a small application that issues ioctl calls to control GPIO pins. That part also seems to work with 10373, but that is as far as I could go before actually coding for the new processor. And that promises to bring a lot of fun :-)

October 1, 2009

Working on 10373 prototype assembly

by Andrey Filippov

my_finger_with_capacitors_for_373 There are several hundred of these to install…

September 24, 2009

Got the BGA chips installed on the 10373 prototype

by Andrey Filippov

10373A with BGA chips installed - fragment

10373A with BGA chips installed - fragment

Finally I received the 10373A PCB with BGA chips soldered. Still one (DDR3 for the video memory) is missing, we’ll install it later – anyway there is a lot to be done before I’ll need it. But that’s all what is soldered for me – I could not do it by hand. Hundreds of other components are waiting for me – I attached two of the model 353 cameras as microscopes and prepared to solder 0201 resistors and capacitors (0.3mm by 0.6mm) – I’ve never done it before. It may be stupid to try to do that manually, but it did work for me before to use a soldering iron after working with just virtual stuff on the computer. I believe it helps me to find some potential problems in the design while I take components one by one and think of them in different perspective than during the initial design.

And while I was waiting for the components and the board – I made some minor design upgrades to the existent camera – changed the layout of the sensor board to fit better multi-sensor applications (like panoramic video). The new board is just 15mm by 28mm (current 10338 is 32mm x 32mm), so using M12 (S-mount) lenses it is possible to shrink the size of the panoramic head so the distance between the outer elements of the opposite lenses is under 80mm – it is much smaller than similar high-res systems. That distance is important to reduce parallax – most noticable when the objects are close to the camera.

July 10, 2009

10373 – Assembly 3-d Render

by Andrey Filippov

Spent couple days looking if the 10373 board layout can be connected as I hoped. There will be small board with 2 connectors on the back panel (103732) – micro USB and eSATA. In minimal configuration it will be a passive board with connectors only, but there should be some room to add USB hub and a couple internal 10-conductor flex cables for internal USB devices – same as on the 10369 board of the NC353L camera.

SATA and USB signals are split into separate connectors on the 10373 board – that allows to leave USB connected to the back panel adapter while connecting the SATA cable (with additional power) to the internal adapter for the SATA SSD. The image below shows both options, but actually 10373 has only one SATA port so only one SATA device can be connected.

While most functionality of the 10369 board is moved to the main 10373 there is no ports for the inter-camera synchronization – 10369 has 2 sets of oto-isolated synchronization ports – “external” with modular (like telephone) connector for frame-locking several individual cameras an “internal” (with flex cable connectors) for synchronizing multiple camera boards installed in the same enclosure. The synchronization will have to be handled by an extension board that has direct connection to the 18 FPGA I/O pins. The back panel will have holes (covered with plastic overlay when not used) for up to four 2.5mm TRS connectors (like small stereo audio ones). Unfortunately even small 2.5mm TRS plugs plastic bodies are rather large so 2 of them can fit only when connectors are diagonal.. Anyway – that part of the design is not finished yet, we’ll think of something.



July 2, 2009

10373A – new functionality added

by Andrey Filippov

It seems that those additional parts did fit on the board. I still need to restore some power connections and the legend on the board (it is also getting to be a real challenge – the smallest letters have to use 0.125mm (5 mil) lines and there is very little space between the components. It will need some time, but it is definitely doable. This is how the board looks now (click on the image to get the full size):


So to the 10373 features I listed initially in my May, 18 post:

  • TMS320DM6467 as the CPU
  • Xilinx Spartan 6 FPGA
  • 256 MiB NAND flash
  • 256 MiB DDR2 system memory
  • 256 MiB DDR3 dedicated FPGA memory
  • 48V DC over Ethernet cable
  • GigE
  • USB 2.0 host
  • ATA port (to be used with the SATA bridge on the 10369 or similar I/O board)
  • USB 1.1 device with micro USB connector. It is connected to the CPU serial port to be used as a system console and alternative boot source

Few more can be added:

  • Two sensor ports, each can accommodate existent sensor front ends that work with NC353L cameras. Both sensors will be able to operate at full resolution/speed in parallel.
  • SATA port with the flex cable connector. The connector pinout is made so it is possible to use flexible cables (single-layer) , not just 2-layer cable or flex PCB to achieve 100 Ohm impedance for the high-speed differential pairs. On the other end of that cable there will be either eSATA connector on the back panel of the camera or an adapter for the 64GB SSD used in some EEE PC computers. As you can see on Elphel wiki these 64GB memory can easily fit inside the camera – so no more CF cards. Same with 1.8″ HDD – we can make an adapter from the same flex cable – it addition to SATA signals it has both 3.3VDC and 5VDC power.
  • Another flex cable connector has USB 2.0 (host) signals, including 5V power and additional 3.3V and 4 GPIO signals from the FPGA. It is possible to put just micro USB connector adapter on the camera back panel (it is likely to be combined with the eSATA one to simplify mounting) or have a 4-port USB hub (similar to the one in the 10369 board) and additional flex-cable connectors for internal USB or I2C internal add-ons (i.e compass).
  • Clock/calendar with a super-capacitor power backup, similar to the one in our current 10369A board, but with the additional function – 10373 makes use of the alarm of that clock (it can be set up to a week in the future). Alarm output is used to re-power the camera after it turned itself off (and it can now). DC-DC converters still consume several mA when turned off, but it will be possible to use the camera with a very small solar panel in the wilderness unattended – either periodically recording images to the internal memory or using USB cellular module (if the area has coverage) to upload the images.

And here is where I’m working on this board – 110.965W 40.65168N. It is raining most of the time – day and night, so I have more time to work.

Next Page »