Category Archives: programming

C, Debugging, Design, Device Drivers, Embedded Linux, Linux Kernel, Linux Programming, programming

Linux Kernel Programming – my second book

4 Comments

I’ve recently completed a project – the writing of the Linux Kernel Programming book, published by Packt (it was announced by the publisher on 01 March 2021). This project took just over two years…

All those long days and nights, poring over the writing and the code, I now feel has definitely been very worth-while and that the book will be a useful contribution to the Linux programming community.

A key point: I’ve ensured that all the material and code examples are based on the 5.4 LTS Linux kernel; it’s slated to be maintained right through Dec 2025, thus keeping the book’s content very relevant for a long while!

Due to its sheer size and depth, the publisher suggested we split the original tome into two books. That’s what has happened: 

  • the first part, Linux Kernel Programming, covers the essentials and, in my opinion, should be read first (of course, if you’re already very familiar with the topics it covers, feel free to start either way)
  • the second part, Linux Kernel Programming Part 2, covering a small section of device driver topics, focusses on the basics and the character ‘misc’ class device driver framework.

Many cross-references, especially from the second book to topics in the first, do turn up; hence the suggestion to read them in order.

Here’s a quick run down on what’s covered in each book.

Lets begin with the Linux Kernel Programming book; firstly, it’s targeted at people who are quite new to the world of Linux kernel development and makes no assumptions regarding knowledge of the kernel. The prerequisite is a working knowledge of programming on Linux with ‘C’; it’s the medium we use throughout (along with a few bash scripts). The book is divided into three major sections, each containing appropriate chapters:

  • Section 1 covers the basics: firstly, the appropriate setup of the kernel development workspace on your system; next, two chapters cover the building of the Linux kernel from scratch, from code. (It includes the cross compile as well, using the popular Raspberry Pi board as a ‘live’ example).
    • The following two chapters delve in-depth into the kernel’s powerful Loadable Kernel Module (LKM) framework, how to program it along with more advanced features. I also try and take a lot of trouble to point out how one should code with security in mind!
  • In Section 2 we deal with having you, the reader, gain a deeper understanding (to the practical extent required) of key kernel internals topics. A big reason why many struggle with kernel development is a lack of understanding of its internals.
    • Here, Chapter 6 covers the kernel architecture, focusing on how the kernel maintains attribute information on processes/threads and their associated stacks.
    • The next chapter – a really key one, again – delves into a difficult topic for many – memory management internals. I try to keep the coverage focused on what matters to a kernel and/or driver developer.
    • The following two chapters dive into the many and varied ways to allocate and deallocate memory when working within the kernel – an area where you can make a big difference performance-wise by knowing which kernel APIs and methods to use when.
    • The remaining two chapters here round off kernel internals with discussion on the kernel-level CPU scheduler; several concepts and practical code examples have the reader learn what’s required.
  • Section 3 is where the books dives into what folks new to it consider to be difficult and arcane matters – how and why synchronization matters, how data races occur and how you can protect critical sections in your kernel / driver code!
    • The amount of material here requires two chapters to do justice to: the first of them focuses on critical sections, concurrency concerns, the understanding and the practical usage of the mutex and the spinlock.
    • The book’s last chapter continues this discussion on kernel synchronization covering more areas relevant to the modern kernel and/or driver developer – atomic (and refcount) operators, cache effects, a primer on ‘lock-free’ programming techniques, with one of them – the percpu one – covered in some detail. Lock debugging within the kernel – using the powerful lockdep validator – as well as other techniques is covered as well!

The second book – Linux Kernel Programming Part 2 – Char Device Drivers and Kernel Synchronization – deliberately covers just a small section of ‘how to write a device driver on Linux’. It does not purport to cover the many types and aspects of device driver development, instead focusing on the basics of teaching the reader how to write a simple yet complete character device driver belonging to the ‘misc’ class.

Great news! This book – Linux Kernel Programming Part 2 – Char Device Drivers and Kernel Synchronization – is downloadable for FREE. Enjoy!

Access it now!

Having said that, the materials covering user-kernel communication pathways, working with peripheral I/O memory, and especially, the topic on dealing with hardware interrupts, is very detailed and will prove to be very useful in pretty much all kinds of Linux device driver projects.

A quick chapter-wise run down of the second book:

  • In Chapter 1, we cover the basics – the reader understands the basics of the Linux Device Model (LDM) and ends up writing a small, simple, yet complete ‘misc’ class character driver. Security-awareness is built too: we demonstrate a simple “privesc” – privilege escalation – attack
  • Chapter 2 shows the reader something every driver author will at one time or the other have to do: efficiently communicate between user and kernel address spaces. You’ll learn to use various technologies to do so – via procfs, sysfs, debugfs (especially useful to insert debug hooks as well), netlink sockets and the ioctl system call
  • The next chapter has the reader understand the nuances of reading and writing peripheral (hardware) I/O memory, via both the memory-mapped I/O (MMIO) as well as the Port I/O (PIO) technique
  • Chapter 4 covers dealing with hardware interrupts in-depth; the reader will learn how the kernel works with hardware interrupts, then move onto how one is expected to allocate an IRQ line (covering modern resource-managed APIs), and how to correctly implement the interrupt handler routine. The modern approach of using threaded handlers (and the why of it) is then covered. The reasons for and using both “top half” and “bottom half” interrupt mechanisms (hardirq, tasklet, and softirqs) in code, as well as key information regarding the dos and don’ts of hardware interrupt handling are covered. Measuring interrupt latencies with the modern [e]BPF toolset, as well as with Ftrace, concludes this key chapter
  • Common kernel mechanisms – setting up delays, kernel timers, working with kernel threads and kernel workqueues – is the subject matter of Chapter 5. Several example kernel modules, including three versions of a ‘simple encrypt decrypt’ (‘sed’) example driver, serve to illustrate the concepts learned in code
  • The final two chapters of this book deal with the really important topic of kernel synchronization (the same material in fact as the last two chapters of the first book). 

I think you’ll find that both books have a fairly large number of high quality, relevant code examples, all of which are based on the 5.4 LTS kernel.

[ LKP : code on GitHub ] [ LKP Part 2 : code on GitHub ]

Thanks for taking the time to read this post; more, I really hope you will read and enjoy these books!

Get Linux Kernel Programming, Kaiwan N Billimoria, Packt, Mar 2021 :

[ On Amazon (US)  ]    [ On Amazon (India) ]    [ On Packt ]

Debugging, Device Drivers, Embedded Linux, Linux Kernel, Linux Programming, programming, training

Linux training courses on offer

Leave a comment

Hi, from the outset, this is #marketing 🙂 (One has to right!?)
Please see below all currently offered very high quality Corporate Training courses I conduct:

[To see it more clearly, you can access the entire sheet here as well].

We also setup and conduct custom-built training sessions; to get started, please do contact me:

Kaiwan N Billimoria
Founder at  kaiwanTECH (a division of Designer Graphix)
A  4931, 11th Floor, Highpoint IV, 45 Palace Road, Bangalore 560001, India.
P  +91.80.22389396
E  kaiwan -at- kaiwantech -dot- com / kaiwan.billimoria -at- gmail -dot- com
Amazon author profile

IMPORTANT UPDATE!
In view of the recent (as of Mar 2020) coronavirus issues, we’re happy to offer the very same training experience through an online platform; it will still be ILT (Instructor Lead Training’s) and the same awesome experience, except that instead of being (typically) conducted at your corporate offices/labs, it will be held online, with participants being given a schedule to login, interact, perform hands-on exercises and learn.

​ An FAQ: our training sessions conducted for individuals?
Ans: Yes, please see the above ‘IMPORTANT UPDATE’ para; we shall also offer individuals an online experience.

“If you think training is expensive, try ignorance”, Peter Drucker.
Hoping to hear from you soon!

assembly, C, Embedded Linux, firmware, Misc, programming

Advice to a Young Firmware Developer – by Jack Ganssle; and Assembly

Leave a comment

I hope Jack Ganssle forgives my directly copying his content; the only reason I do so is that these thoughts of his are precious and I wish for more of us to read and appreciate them. (The discussion is definitely biased towards firmware/embedded developers that work primarily on a hardware platform using ‘C’ as the language. Just a small part of Jack’s excellent Embedded Muse newsletter is shown below; do check out the full article and subscribe to his newsletter). 

 

Directly copied from here: The Embedded Muse, Issue #362 by jack Ganssle, 19 Nov 2018.

“Advice to a Young Firmware Developer

… Learn, in detail, how a computer works. You should be able to draw a detailed block diagram of one. Even if you have no interest in the hardware, it’s impossible to understand assembly language and other important aspects of creating firmware without understanding program counters, registers, busses and the like.

Learn an assembly language. Write real programs in it, and make them work. Absent a grounding in assembly much of the operation of a computer will be mysterious. In real life you’ll have to delve into the assembly at least occasionally, at least to work on the startup code, and to find some classes of bugs.

A recent article in IEEE Spectrum surveyed language use and C didn’t even make the cut. Java, Javascript, HMTL and Python were ranked as the most in-demand languages in the USA. Yet around 70% of firmware people work in C. C++ makes up another 20%. For better or worse, all of the other embedded languages are in the noise. Master C, pointers and all. (Rust is increasingly popular, yet, despite the hype, has under a 1% share in the embedded space).

But do learn some other languages. Python can be useful for scripting. Ada gives a discipline I wish more had.

Work in a cross-development environment with an embedded target board. It’s very different from using Visual Studio.

Get comfortable with a Linux shell. With sed, awk, and a hundred other tools you can do incredible things without writing any code.

Take the time to think through what you’re building. It’s tempting to start coding early. Design is critically important. Remember the old saying: “if you think good design is expensive, consider the cost of bad design.”

Monitor your bug rates. Forever. Skip this and you’ll never know two things: if you’re improving, and how you compare to the industry. We all think we’re great developers, but metrics can be a cold shower.

Always be on the prowl for tools. Some are free, others expensive, but great tools will make you more productive and less error-prone. These change all the time so figure on constantly refining your toolbox.

Did you know the average firmware person reads one technical book a year? Yet this field evolves at the pace of Moore’s Law. Constantly study software engineering. We do have a Body of Knowledge. Every year new ideas appear. Some are brilliant, others whacky, but they all make you think.

Learn about the hardware. At least get a general understanding. An engineer who can use an oscilloscope and logic analyzer for troubleshooting code is a valuable addition to a software team. Digital and analog hardware is cool and fascinating. …”

[Added on 24Feb2020]
Again, I couldn’t resist copy-pasting! … a well thought out answer on Quora to the question:
What are some things about coding that you learned from good programmers around you?
Here’s the full answer (by Håkon Hapnes Strand):

  1. Don’t be a lazy bastard. Do things the right way even if it’s a lot of work.
  2. Don’t give up on something just because you’re stuck. You’ll figure it out eventually.
  3. Always track down the root cause of a bug. If a bug just “goes away”, it hasn’t. If you can’t explain what fixed the bug, it isn’t.
  4. Write unit tests. It may seem unnecessary and a like lot of work, but it will help you in the long run. See point 1.
  5. Best practices are best practices for a reason. Don’t assume that your approach is better just because it’s what you’re used to.
  6. Make sure everything is reproducible. That includes data and infrastructure.
  7. If it’s not in version control, it doesn’t exist. (See point 6)
  8. Always abstract where it’s needed. Never abstract where it’s not needed.
  9. Think before you code.

So, okay, that’s the part of the article(s) I wanted to show.

Learning Assembly Language – Resources

How does one just learn assembly language then? Well, there are resources of course that help – books, online articles; here’s a few: 

C, Linux Programming, programming

Pthreads Dev – Common Programming Mistakes to Avoid

Leave a comment

Disclaimer: Okay, let me straight away say this: most of these points below are from various books and online sources. One that stands out in my mind, an excellent tome, though quite old now, is “Multithreaded Programming with Pthreads”, Bil Lewis and Daniel J. Berg.

Common Programming Errors one (often?) makes when programming MT apps with Pthreads

  • Failure to check return values for errors
  • Using errno without actually checking that an error has occurred
WRONG                       Correct
syscall_foo();              if (syscall_foo() < 0) {
if (errno) { ... }              if (errno) { ... } }

(Also, note that all Pthread APIs may not set errno)

  • Not joining on joinable threads
  • A critical one: Failure to verify that library calls are MT Safe
    Use the foo_r API version if it exists, over the foo.
    Use TLS, TSD, etc.
  • Falling off the bottom of main()
    must call pthread_exit() ; yes, in main as well!
  • Forgetting to include the POSIX_C_SOURCE flag
  • Depending upon Scheduling Order
    Write your programs to depend upon synchronization. Don’t do :

          sleep(5); /* Enough time for manager to start */

Instead, wait until the event in question actually occurs; synchronize on it, perhaps using CVs (condition variables)

  • Not Recognizing Shared Data
    Especially true when manipulating complex data structures – such as lists in which each element (or node) as well as the entire list has a separate mutex for protection; so to search the list, you would have to obtain, then release, each lock as the thread moved down the list. It would work, but be very expensive. Reconsider the design perhaps?
  • Assuming bit, byte or word stores are atomic
    Maybe, maybe not. Don’t assume – protect shared data
  • Not blocking signals when using sigwait(3)
    Any signals you’re blocking upon with the sigwait(3) should never be delivered asynchronously to another thread; block ’em and use ’em
  • Passing pointers to data on the stack to another thread

(Case a) Simple – an integer value:

Correct (below):

main()
{
 ...
 // thread creation loop
 for (i=0; i<NUM_THREADS; i++) {
    thread_create(&thrd[i], &attr, worker, i);
 }
 ...
 // join loop...

 pthread_exit();
}

The integer i is passed to each thread as a ‘literal value’; no issues.

Wrong approach (below):

main()
{
 ...
 // thread creation loop
 for (i=0; i<NUM_THREADS; i++) {
     thread_create(&thrd[i], &attr, worker, &i);
 }
  ...
  // join loop...

  pthread_exit();
}

Passing the integer by address implies that some thread B could be reading it while thread main is writing to it! A race, a bug.

(Case b) More complex – a data structure:

Correct (below):

main()
{
my_struct *pstr;
...
// thread creation loop
for (i=0; i<NUM_THREADS; i++) {
   pstr = (my_struct *) malloc(...);
   pstr->data = <whatever>;
   pstr->... = ...; // and so on...
   pthread_create(&thrd[i], &attr, worker, pstr);
}
...
// in the join loop..
  free(pstr);

pthread_exit();
}

The malloc ensures the memory is accessible to the particular thread it’s being passed to. Thread Safe.

Wrong! (below)

my_struct *pstr = malloc(...);

main()
{
...
for (i=0; i<NUM_THREADS; i++) {
   pstr->data = <whatever>;
   pstr->... = ...; // and so on...
   pthread_create(&thrd[i], &attr, worker, pstr);
}
// join

free(pstr);
pthread_exit();
}

If you do this (the wrong one, above), then the global pointer (one instance of the data structure only) is being passed around without protection – threads will step on “each other’s toes” corrupting the data and the app. Thread Unsafe.

  • Avoid the above problems; use the TLS (Thread-Local Storage) – a simple and elegant approach to making your code thread safe.

Resource: GCC page on TLS.

 

 

C, Debugging, Embedded Linux, Linux Programming, programming

Application Binary Interface (ABI) Docs and Their Meaning

Leave a comment

Have you, the programmer, ever really thought about how it all actually works? Am sure you have…

We write

printf("Hello, world! value = %d\n", 41+1);

and it works. But it’s ‘C’ code – the microprocessor cannot possibly understand it; all it  “understands” is a stream of binary digits – machine language. So, who or what transforms source code into this machine language?

The compiler of course! How? It just does (cheeky). So who wrote the compiler? How?
Ah. Compiler authors figure out how by reading a document provided by the microprocessor (cpu) folks – the ABI – Application Binary Interface.

People often ask “But what exactly is an ABI?”. I like the answer provided here by JesperE:

"... If you know assembly and how things work at the OS-level, you are conforming to a certain ABI. The ABI govern things like
how parameters are passed, where return values are placed. For many platforms there is only one ABI to choose from, and in those
cases the ABI is just "how things work".

However, the ABI also govern things like how classes/objects are laid out in C++. This is necessary if you want to be able to pass
object references across module boundaries or if you want to mix code compiled with different compilers. ..."

Another way to state it:
The ABI describes the underlying nuts and bolts of the mechanisms  that systems software such as the compiler, linker, loader – IOW, the toolchain – needs to be aware of: data representation, function calling and return conventions, register usage conventions, stack construction, stack frame layout, argument passing – formal linkage, encoding of object files (eg. ELF), etc.

Having a minimal understanding of :

  • a CPU’s ABI – which includes stuff like
    • it’s procedure calling convention
    • stack frame layout
    • ISA (Instruction Set Architecture)
    • registers and their internal usage, and,
  • bare minimal assembly language for that CPU,

helps to no end when debugging a complex situation at the level of the “metal”.

With this in mind, here are a few links to various CPU ABI documents, and other related tutorials:

However, especially for folks new to it, reading the ABI docs can be quite a daunting task! Below, I hope to provide some simplifications which help one gain the essentials without getting completely lost in details (that probably do not matter).

Often, when debugging, one finds that the issue lies with how exactly a function is being called – we need to examine the function parameters, locals, return value. This can even be done when all we have is a binary dump – like the well known core file (see man 5 core for details).

Intel x86 – the IA-32

On the IA-32, the stack is used for function calling, parameter passing, locals.

Stack Frame Layout on IA-32

[...                            <-- Bottom; higher addresses.
PARAMS 
...]              
RET addr 
[SFP]                      <-- SFP = pointer to previous stack frame [EBP] [optional]
[... 
LOCALS 
...]                           <-- ESP: Top of stack; in effect, lowest stack address


Intel 64-bit – the x86_64

On this processor family, the situation is far more optimized. Registers are used to pass along the first six arguments to a function; the seventh onwards is passed on the stack. The stack layout is very similar to that on IA-32.

Register Set

x86_64_registers

<Original image: from Intel manuals>

Actually, the above register-set image applies to all x86 processors – it’s an overlay model:

  • the 32-bit registers are literally “half” the size and their prefix changes from R to E
  • the 16-bit registers are half the size of the 32-bit and their prefix changes from E to A
  • the 8-bit registers are half the size of the 16-bit and their prefix changes from A to AH, AL.

The first six arguments are passed in the following registers as follows:

RDI, RSI, RDX, RCX, R8, R9

(By the way, looking up the registers is easy from within GDB: just use it’s info registers command).

An example from this excellent blog “Stack frame layout on x86-64” will help illustrate:

On the x86_64, call a function that receives 8 parameters – ‘a, b, c, d, e, f, g, h’. The situation looks like this now:

x86_64_func

What is this “red zone” thing above? From the ABI doc:

The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers. Therefore, functions may use this area for temporary data that is not needed across function calls. In particular, leaf functions may use this area for their entire stack frame, rather than adjusting the stack pointer in the prologue and epilogue. This area is known as the red zone.

Basically it’s an optimization for the compiler folks: when a ‘leaf’ function is called (one that does not invoke any other functions), the compiler will generate code to use the 128 byte area as ‘scratch’ for the locals. This way we save two machine instructions to lower and raise the stack on function prologue (entry) and epilogue (return).

ARM-32 (Aarch32)

<Credits: some pics shown below are from here : ‘ARM University Program’, YouTube. Please see it for details>.

The Aarch32 processor family has seven modes of operation: of these, six of them are privileged and only one – ‘User’ – is the non-privileged mode, in which user application processes run.

modes

When a process or thread makes a system call, the compiler has the code issue the SWI machine instruction which puts the CPU into Supervisor (SVC) mode.

The Aarch32 Register Set:

regs

Register usage conventions are mentioned below.

Function Calling on the ARM-32

The Aarch32 ABI reveals that it’s registers are used as follows:

Register APCS name Purpose
R0 a1 Argument registers, passing values, don’t need to be preserved,
results are usually returned in R0
R1 a2
R2 a3
R3 a4
R4 v1 Variable registers, used internally by functions, must be preserved if used. Essentially, r4 to r9 hold local variables as register variables.

(Also, in case of the SWI machine instruction (syscall), r7 holds the syscall #).
R5 v2
R6 v3
R7 v4
R8 v5
R9 v6
R10 sl Stack Limit / stack chunk handle
R11 fp Frame Pointer, contains zero, or points to stack backtrace structure
R12 ip Procedure entry temporary workspace
R13 sp Stack Pointer, fully descending stack, points to lowest free word
R14 lr Link Register, return address at function exit
R15 pc Program Counter

(APCS = ARM Procedure Calling Standard)

When a function is called on the ARM-32 family, the compiler generates assembly code such that the first four integer or pointer arguments are placed in the registers r0, r1, r2 and r3. If the function is to receive more than four parameters, the fifth one onwards goes onto the stack. If enabled, the frame pointer (very useful for accurate stack unwinding/backtracing) is in r11. The last three registers are always used for special purposes:

  • r13: stack pointer register
  • r14: link register; in effect, return (text/code) address
  • r15: the program counter (the PC)

The PSR – Processor State Register – holds the system ‘state’; it is constructed like this:

cpsr

[Update: 24 Sept 2021]

ARM 64-bit

Ref: ARM Cortex-A Series Programmer’s Guide for ARMv8-A

Execution on the ARMv8 is at one of four Exception Levels (ELn; n=0,1,2,3). It determines the privilege level (just as x86 has 4 rings, and the ARM has seven modes). […] Exception Levels provide a logical separation of software execution privilege that applies across all operating states of the ARMv8 architecture. […] The following is a typical example of what software runs at each Exception level:

EL0

Normal user applications.

EL1

Operating system kernel typically described as privileged.

EL2

Hypervisor.

EL3

Low-level firmware, including the Secure Monitor.

Figure 3.1. Exception levels

Figure 3.1. Exception levels

ARMv8 Registers and their Usage (ABI)

Screenshot from 2021-09-24 12-40-21

In addition, the ‘special’ registers:

Screenshot from 2021-09-24 12-42-15

ARM-64 / A64 / Aarch64 ABI calling conventions

(The following is directly excerpted from the Wikipedia page here: https://en.wikipedia.org/wiki/Calling_convention#ARM_(A64)).

The 64-bit ARM (AArch64) calling convention allocates the 31 general-purpose registers as:

  • x31 (SP): Stack pointer or a zero register, depending on context.
  • x30 (LR): Procedure link register, used to return from subroutines.
  • x29 (FP): Frame pointer.
  • x19 to x29: Callee-saved.
  • x18 (PR): Platform register. Used for some operating-system-specific special purpose, or an additional caller-saved register.
  • x16 (IP0) and x17 (IP1): Intra-Procedure-call scratch registers.
  • x9 to x15: Local variables, caller saved.
  • x8 (XR): Indirect return value address.
  • x0 to x7: Argument values passed to and results returned from a subroutine.

All registers starting with x have a corresponding 32-bit register prefixed with w. Thus, a 32-bit x0 is called w0.

Similarly, the 32 floating-point registers are allocated as:[3]

  • v0 to v7: Argument values passed to and results returned from a subroutine.
  • v8 to v15: callee-saved, but only the bottom 64 bits need to be preserved.
  • v16 to v31: Local variables, caller saved.

Hope this helps!

Debugging, Embedded Linux, Linux Kernel, Linux Programming, programming

Setting up Kdump and Crash for ARM-32 – an Ongoing Saga

4 Comments

Author: Kaiwan N Billimoria, kaiwanTECH
Date: 13 July 2017

DUT (Device Under Test):
Hardware platform: Qemu-virtualized Versatile Express Cortex-A9.
Software platform: mainline linux kernel ver 4.9.1, kexec-tools, crash utility.

First, my attempt at setting up the Raspberry Pi 3 failed; mostly due to recurring issues with the bloody MMC card; probably a power issue! (see this link).

Anyway. Then switched to doing the same on the always-reliable Qemu virtualizer; I prefer to setup the Vexpress-CA9.

In fact, a supporting project I maintain on github – the SEALS project – is proving extremely useful for building the ARM-32 hardware/software platform quickly and efficiently. (Fun fact: SEALS = Simple Embedded Arm Linux System).

So, I cloned the above-mentioned git repo for SEALS into a new working folder.

The way SEALS work is simple: edit a configuration file (build.config) to your satisfaction, to reflect the PATH to and versions of the cross-compiler, kernel, kernel command-line parameters, busybox, rootfs size, etc.

Setup the SEALS build.config file.

Screenshot: the build_SEALS.sh script initial screen displaying the current build config:kdumpcr1

<<
Relevant Info reproduced below for clarity:
…
Toolchain prefix : arm-none-linux-gnueabi-
Toolchain version: (Sourcery CodeBench Lite 2014.05-29) 4.8.3 20140320 (prerelease)

Staging folder : <…>/SEALS_staging
ARM Platform : Versatile Express (A9)

Platform RAM : 512 MB
RootFS force rebuild : 0
RootFS size : 768 MB

Linux kernel to use : 4.9.1
Linux kernel codebase location : <…>/SEALS_staging/linux-4.9.1
Kernel command-line : “console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M”

Busybox to use : 1.26.2
Busybox codebase location : <…>/SEALS_staging/busybox-1.26.2
…
>>

Screenshot: build_SEALS.sh second GUI screen, allowing the user to select actions to takekdumpcr2

Upon clicking ‘OK’, the build process starts:

I Boot Kernel Setup

  • kernel config: must carefully configure the Linux kernel. Please follow the kernel documentation in detail:
    https://www.kernel.org/doc/Documentation/kdump/kdump.txt [1]In brief, ensure these are set:
    CONFIG_KEXEC=y
    CONFIG_SYSFS=y << should be >>
    CONFIG_DEBUG_INFO=y
    CONFIG_CRASH_DUMP=y
    CONFIG_PROC_VMCORE=y

“Dump-capture kernel config options (Arch Dependent, arm)
 To use a relocatable kernel, Enable “AUTO_ZRELADDR” support under “Boot” options:      

             AUTO_ZRELADDR=y”

  • Copy the ‘kexec’ binary into the root filesystem (staging tree) under it’s sbin/ folder
  • We build a relocatable kernel so that we can use the same ‘zImage’ 
for the dump kernel as well as the primary boot kernel:
 “Or use the system kernel binary itself as dump-capture kernel and there is no need to build a separate dump-capture kernel. 
This is possible  only with the architectures which support a relocatable kernel. As  of today, i386, x86_64, ppc64, ia64 and arm architectures support relocatable kernel. ...”
  • the SEALS build system will proceed to build the kernel using the cross-compiler specified
  • went through just fine.

II Load dump-capture (or kdump) kernel into boot kernel’s RAM

Do read [1], but to cut a long story short

  • Create a small shell script kx.sh - a wrapper over kexec – in the root filesystem:
 
#!/bin/sh
DUMPK_CMDLINE="console=ttyAMA0 root=/dev/mmcblk0 rootfstype=ext4 rootwait init=/sbin/init maxcpus=1 reset_devices"
kexec --type zImage \
-p ./zImage-4.9.1-crk \
--dtb=./vexpress-v2p-ca9.dtb \
--append="${DUMPK_CMDLINE}" 
[ $? -ne 0 ] && { 
    echo "kexec failed." ; exit 1
}
echo "$0: kexec: success, dump kernel loaded."
exit 0
  • Run it. It will only work (in my experience) when (for this iMX6 system):
    • you’ve passed the kernel parameter ‘crashkernel=32M’
    • verified that indeed the boot kernel has reserved 32MB RAM for the dump-capture kernel/system:
RUN: Running qemu-system-arm now ...

qemu-system-arm -m 512 -M vexpress-a9 -kernel <...>/images/zImage \
-drive file=<...>/images/rfs.img,if=sd,format=raw \
-append "console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M" \
-nographic -no-reboot -dtb <...>/linux-4.9.1/arch/arm/boot/dts/vexpress-v2p-ca9.dtb

Booting Linux on physical CPU 0x0
Linux version 4.9.1-crk (hk@hk) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Wed Jul 12 19:41:08 IST 2017
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt:Machine model: V2P-CA9
...
ARM / $ dmesg |grep -i crash
Reserving 32MB of memory at 1920MB for crashkernel (System RAM: 512MB)
Kernel command line: console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M
ARM / $ id
uid=0 gid=0
ARM / $ ./kx.sh
./kx.sh: kexec: success, dump kernel loaded.
ARM / $

Ok, the dump-capture kernel has loaded up.
Now to test it!

III Test the soft boot into the dump-capture kernel

On the console of the (emulated) ARM-32:

ARM / $ echo c > /proc/sysrq-trigger 
sysrq: SysRq : Trigger a crash
Unhandled fault: page domain fault (0x81b) at 0x00000000
pgd = 9ee44000
[00000000] *pgd=7ee30831, *pte=00000000, *ppte=00000000
Internal error: : 81b [#1] SMP ARM
Modules linked in:
CPU: 0 PID: 724 Comm: sh Not tainted 4.9.1-crk #2
Hardware name: ARM-Versatile Express
task: 9f589600 task.stack: 9ee40000
PC is at sysrq_handle_crash+0x24/0x2c
LR is at arm_heavy_mb+0x1c/0x38
pc : [<804060d8>] lr : [<80114bd8>] psr: 60000013
sp : 9ee41eb8 ip : 00000000 fp : 00000000

...

[<804060d8>] (sysrq_handle_crash) from [<804065bc>] (__handle_sysrq+0xa8/0x170)
[<804065bc>] (__handle_sysrq) from [<80406ab8>] (write_sysrq_trigger+0x54/0x64)
[<80406ab8>] (write_sysrq_trigger) from [<80278588>] (proc_reg_write+0x58/0x90)
[<80278588>] (proc_reg_write) from [<802235c4>] (__vfs_write+0x28/0x10c)
[<802235c4>] (__vfs_write) from [<80224098>] (vfs_write+0xb4/0x15c)
[<80224098>] (vfs_write) from [<80224d30>] (SyS_write+0x40/0x80)
[<80224d30>] (SyS_write) from [<801074a0>] (ret_fast_syscall+0x0/0x3c)

Code: f57ff04e ebf43aba e3a03000 e3a02001 (e5c32000) 


Loading crashdump kernel...
Bye!
Booting Linux on physical CPU 0x0

Linux version 4.9.1-crk (hk@hk) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Wed Jul 12 19:41:08 IST 2017
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt:Machine model: V2P-CA9
OF: fdt:Ignoring memory range 0x60000000 - 0x78000000
Memory policy: Data cache writeback
CPU: All CPU(s) started in SVC mode.
percpu: Embedded 14 pages/cpu @81e76000 s27648 r8192 d21504 u57344
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 7874
Kernel command line: console=ttyAMA0 root=/dev/mmcblk0 rootfstype=ext4 rootwait 
init=/sbin/init maxcpus=1 reset_devices elfcorehdr=0x79f00000 mem=31744K

...
ARM / $ ls -l /proc/vmcore            << the dump image (480 MB here) >>
-r-------- 1 0 0 503324672 Jul 13 12:22 /proc/vmcore
ARM / $ 

Copy the dump file (with cp or scp, whatever), 
get it to the host system.

cp /proc/vmcore <dump-file>
ARM / $ halt
ARM / $ EXT4-fs (mmcblk0): re-mounted. Opts: (null)
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system halt
reboot: System halted
QEMU: Terminated
^A-X  << type Ctrl-a followed by x to exit qemu >>
... and done.

build_SEALS.sh: all done, exiting.
Thank you for using SEALS! We hope you like it.
There is much scope for improvement of course; would love to hear your feedback, ideas, and contribution!
Please visit : https://github.com/kaiwan/seals . 
$ 


IV Analyse the kdump image with the crash utility

CORE ANALYSIS SUITE

The core analysis suite is a self-contained tool that can be used to
investigate either live systems, kernel core dumps created from dump
creation facilities such as kdump, kvmdump, xendump, the netdump and
diskdump packages offered by Red Hat, the LKCD kernel patch, the mcore
kernel patch created by Mission Critical Linux, as well as other formats
created by manufacturer-specific firmware.

...

A whitepaper with complete documentation concerning the use of this utility
can be found here:
https://crash-utility.github.io/crash_whitepaper.html  [3]
...

The crash binary can only be used on systems of the same architecture as
the host build system. There are a few optional manners of building the
crash binary:

o On an x86_64 host, a 32-bit x86 binary that can be used to analyze
32-bit x86 dumpfiles may be built by typing "make target=X86".
o On an x86 or x86_64 host, a 32-bit x86 binary that can be used to analyze
 32-bit arm dumpfiles may be built by typing "make target=ARM".
...

Ah. To paraphrase, Therein lies the devil, in the details.

[Update: Apr 2019:]
To make this more clear: one must install the following prereq packages (I did this on an x86_64 Ubuntu 18.10 system):

sudo apt install gcc-multilib 
sudo apt install libncurses5:i386 lib32z1-dev

[UPDATE : 14 July ’17
I do have it building successfully now. The trick apparently – on x86_64 Ubuntu 17.04 – was to install the lib32z1-dev package! Once I did, it built just fine. Many thanks to Dave Anderson (RedHat) who promptly replied to my query on the crash mailing list.]

I cloned the ‘crash’ git repo, did ‘make target=ARM’, it fails with:

...
 ../readline/libreadline.a ../opcodes/libopcodes.a ../bfd/libbfd.a
../libiberty/libiberty.a ../libdecnumber/libdecnumber.a -ldl
-lncurses -lm ../libiberty/libiberty.a build-gnulib/import/libgnu.a
 -lz -ldl -rdynamic
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
Makefile:1174: recipe for target 'gdb' failed
...

Still trying to debug this!

Btw, if you’re unsure, pl see crash’s github Readme on how to build it.
So, now, with a ‘crash’ binary that works, lets get to work:

$ file crash
crash: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, …

$ ./crash

crash 7.1.9++
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
[…]

crash: compiled for the ARM architecture
$

To examine a kernel dump (kdump) file, invoke crash like so:

crash <path-to-vmlinux-with-debug-symbols> <path-to-kernel-dumpfile>

$ <...>/crash/crash \
  <...>/SEALS_staging/linux-4.9.1/vmlinux ./kdump.img

crash 7.1.9++
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
[...]
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
[...]
WARNING: cannot find NT_PRSTATUS note for cpu: 1
WARNING: cannot find NT_PRSTATUS note for cpu: 2
WARNING: cannot find NT_PRSTATUS note for cpu: 3

 KERNEL: <...>/SEALS_staging/linux-4.9.1/vmlinux
 DUMPFILE: ./kdump.img
 CPUS: 4 [OFFLINE: 3]
 DATE: Thu Jul 13 00:38:39 2017
 UPTIME: 00:00:42
LOAD AVERAGE: 0.00, 0.00, 0.00
 TASKS: 56
 NODENAME: (none)
 RELEASE: 4.9.1-crk
 VERSION: #2 SMP Wed Jul 12 19:41:08 IST 2017
 MACHINE: armv7l (unknown Mhz)
 MEMORY: 512 MB
 PANIC: "sysrq: SysRq : Trigger a crash"
 PID: 735
 COMMAND: "echo"
 TASK: 9f6af900 [THREAD_INFO: 9ee48000]
 CPU: 0
 STATE: TASK_RUNNING (SYSRQ)

crash> ps
 PID PPID CPU TASK ST %MEM VSZ RSS COMM
 0 0 0 80a05c00 RU 0.0 0 0 [swapper/0]
> 0 0 1 9f4ab700 RU 0.0 0 0 [swapper/1]
> 0 0 2 9f4abc80 RU 0.0 0 0 [swapper/2]
> 0 0 3 9f4ac200 RU 0.0 0 0 [swapper/3]
 1 0 0 9f4a8000 IN 0.1 3344 1500 init
[...]
722 2 0 9f6ac200 IN 0.0 0 0 [ext4-rsv-conver]
728 1 0 9f6ab180 IN 0.1 3348 1672 sh
> 735 728 0 9f6af900 RU 0.1 3344 1080 echo
crash> bt
PID: 735 TASK: 9f6af900 CPU: 0 COMMAND: "echo"
 #0 [<804060d8>] (sysrq_handle_crash) from [<804065bc>]
 #1 [<804065bc>] (__handle_sysrq) from [<80406ab8>]
 #2 [<80406ab8>] (write_sysrq_trigger) from [<80278588>]
 #3 [<80278588>] (proc_reg_write) from [<802235c4>]
 #4 [<802235c4>] (__vfs_write) from [<80224098>]
 #5 [<80224098>] (vfs_write) from [<80224d30>]
 #6 [<80224d30>] (sys_write) from [<801074a0>]
 pc : [<76e8d7ec>] lr : [<0000f9dc>] psr: 60000010
 sp : 7ebdcc7c ip : 00000000 fp : 00000000
 r10: 0010286c r9 : 7ebdce68 r8 : 00000020
 r7 : 00000004 r6 : 00103008 r5 : 00000001 r4 : 00102e2c
 r3 : 00000000 r2 : 00000002 r1 : 00103008 r0 : 00000001
 Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM
crash>

And so on …

Another thing we can do is use gdb – to a limited extent – to analyse the dump file:

From [1]:
…

Before analyzing the dump image, you should reboot into a stable kernel.

You can do limited analysis using GDB on the dump file copied out of
/proc/vmcore. Use the debug vmlinux built with -g and run the following
command:
  gdb vmlinux <dump-file>

Stack trace for the task on processor 0, register display, and memory
display work fine.

Also, [3] is an excellent whitepaper on using crash. Do read it.

All right, hope that helps!

Design, programming

Low-Level Software Design

3 Comments

[Please note, this article isn’t about formal design methods (LLD), UML, Design Patterns, nor about object-oriented design, etc. It’s written with a view towards the kind of software project I typically get to work on – embedded / Linux OS related, with the primary programming language being ‘C’ and/or scripting (typically with bash).]

When one looks back, all said and done, it isn’t that hard to get a decent software design and architecture. Obviously, the larger your project, the more the thought and analysis that goes into building a robust system. (Certainly, the more the years of experience, the easier it seems).

However, I am of the view that certain fundamentals never change: get them right and many of the pieces auto-slot into place. Work on a project enough and one always comes away with a  “feel” for the architecture and codebase – it’s robust, will work, or it’s just not.

So what are these “fundamentals”? Well, here’s the interesting thing: you already know them! But in the heat and dust of release pressures (“I don’t care that you need another half-day, check it in now!!!”), deadlines, production, we tend to forget the basics. Sounds familiar? 🙂

The points below are definitely nothing new, but always worth reiterating:

Low-level Design and Software Architecture

  • Jot down the requirements: why are we doing this? what do we hope to achieve?
  • Draw an overall diagram of the project, the data structures, the code flow, as you visualise it. You don’t really need fancy software tools- pencil and paper will do, especially at first.
    pencil20on20notebook20-20writing20concept
    Arrive, gently, at the software architecture.
  • Layering helps (but one can overdo it)
    • To paraphrase- “adding a layer can be used to solve any problem in computer science” 🙂 Of course, one can quite easily add new problems too; careful!
  • It evolves – don’t be afraid to iterate, to use trial and error
    • “Be ready to throw the first one away – you’re going to anyway” – paraphrased from that classic book “The Mythical Man Month”
    • “There is no silver bullet” – again from the same book of wisdom. There is no one solution to all your problems – you’ll have to weigh options, make trade-offs. It’s like life y’know 😉
  • Design the code to be modular, structured
  • A function encapsulates an intention
    • Requirement-driven code: why is the function there?
  • Each function does exactly one thing
    • This is really important. If you can do this well, you will greatly reduce bugs, and thus, the need to debug.
  • Use configuration files (Edit: preferably in plain ASCII text format).

Coding

  • Insert function stubs – code it in detail later, get the overall low-level design and function interfacing correct first. What parameters, return value(s)? 
  • Avoid globals
    • use parameters, return values
    • in multithreaded / multiprocess environments, using any kind of global implies using a synchronization primitive of some sort (mutex, semaphore, spinlock, etc) to take care of concurrency concerns, races. Be aware – beware! – this is often a huge source of performance bottlenecks!
      Edit: When writing MT software, use powerful techniques TLS and TSD to further avoid globals.
  • Keep it minimal, and clean: Careful! don’t end up using too many (nested functions) layers – leads to “lasagna / spaghetti code” that’s hard to follow and thus understand
  • If a function’s code exceeds a ‘page’, re-look, redesign.

Of course, a project is not a dead static thing – at least it shouldn’t be. It evolves over time. Expect requirements, and thus your low-level design and code, to change. The better thought out the overall architecture though, the more resilient it will be to constant flux.

For example: you’re writing a device driver and a “read” method is attempting to read data from the ‘device’ (whatever the heck it is), but there is no data available right now, what should we do? Abort, returning an error code? Wait for data? Retry the operation thrice and see?

The “correct” answer: follow the standard. Assuming we’re working on a POSIX-compliant OS (Unix/Linux), the standard says that blocking calls must do precisely that: block, wait for data until it becomes available. So just wait for data. “But I don’t want to wait forever!” cries the application! Okay, implement a non-blocking open in that case (there’s a reason for that O_NONBLOCK flag folks!). Or a timeout feature, if it makes sense.

Shouldn’t the driver method “retry” the operation if it does not succeed at first? Short answer, No. Follow the Unix design philosophy: “provide mechanism, not policy”. Let the application define the policy (should we retry, if yes, how often; should we timeout, if yes, what’s the timeout, etc etc). The mechanism part of it – your driver’s read method implementation must work independent of, in fact ignore, such concerns. (But hey, it must be written to be concurrent and reentrant -safe. A post on that another day perhaps?).

Configuration

Using the same example: lets say we do want the “read” method of our device driver to timeout after, say, 1 second. Where do we specify this value? Recollect, “provide mechanism, not policy”. So we’ll do so in the application, not the driver. But where in the app? Ah. I’d suggest we don’t hardcode the value; instead, keep it in a simple ASCII-text configuration file:

app_config
   read_timeout=1

Of course, with ‘C’ one would usually put stuff like this into a header file. Fair enough, but please keep a separate header – say, app_config.h .

Try and do some crystal-ball-gazing: at some remote (or not-so-remote) point in the future, what if the project requires a GUI front-end? Probably, as an example, we will want to let the end-user view and set configuration – change the timeout, etc – easily via the GUI. Then, you will see the sense of using a simple ASCII-text configuration file to hold config values – reading and updating the values now becomes simple and clean.
Finally, nothing said above is sacred – we learn to take things on a case-by-case basis, use judgement.


A few Resources