system programming | kaiwanTECH: Kaiwan's Tech Blog

Hello, pleased that my first book has been recently released (on 31 Oct 2018). The publisher is Packt (based in Birmingham, UK).

Quick Links

The book is available in various formats here:

The book has an open-to-public GitHub repository as well.

A chapter of the book is freely available online: File IO Essentials. Do download and check it out.

A fairly detailed description of the book, and what it attempts to cover is given below. Obviously, am very grateful to my readers. A request: once you’ve gone through the book, please take five minutes to write a review for the book on Amazon.

Hands-On System Programming with Linux

The Linux OS has grown from being a one-person tiny hobby project to a fundamental and integral part of a stunning variety of software products and projects; it is found in tiny industrial robots, consumer electronic devices (smartphones, tablets, music players); it is the powerhouse behind enterprise-scale servers, network switches and data centres all over the Earth.

This book is about a fundamental and key aspect of Linux – systems programming at the library and system call layers of the Linux stack. It will cement for you, in a fast-paced yet deeply technical manner, both the conceptual “why” and with a hands-on approach the practical “how” of systems programming on the Linux OS. Real-world relevant and detailed code examples, tested on the latest 4.x kernel distros, are found throughout the book, with added emphasis on key aspects such as programming for security. Linux was never more relevant in industry; are you?

The book’s style is one of making complex topics and ideas easy to understand; we take great pains to introduce the reader to relevant theoretical concepts such that the fundamentals are crystal clear before moving on to APIs and code. Then, the book goes on to provide detailed practical, hands-on and relevant code examples to solidify the concepts. All the code shown in the book (several example for each chapter) is available ready-to-build-and-try on the book’s GitHub repository here : https://github.com/PacktPublishing/Hands-on-System-Programming-with-Linux. Not only that, each chapter has suggested assignments for the budding programmer to try; some of these assignments have solutions provided as well (find them in the GitHub repo).

This book covers a huge amount of ground within the Linux system programming domain; nevertheless, it does so in a unique way. Several features make this book useful and unique; some of them are enumerated below:

How many books on programming have you read that, besides the typical ‘Hello, world’ ‘C’ program, also include a couple of assembly language examples, describe what the CPU ABI is, and a whole lot more, including a detailed description of system architecture, system calls, etc in the very first chapter
While we do not delve into intricate details on their usage, this book uses, and briefly shows how to use, plenty of useful and relevant tools – from ltrace and strace to perf, to LTTng and Ftrace
The book is indeed detailed; for example, it does not shy away from even delving (to the appropriate degree) into kernel internals wherever required. The understanding and description of an important topic such as Virtual Memory in Chapter 2, serves as a good example. Fairly advanced concepts such as looking into the process stack (gstack, GDB) and what the process VM-split is, are covered as well
Ch 4 on Dynamic Memory Allocation goes well beyond the usual malloc API family; here, once we get that covered (in depth of course), the book moves into more advanced topics such as internal malloc behavior, demand paging, explicitly locking and protecting memory pages, using the alloca API, etc
Ch 5 on Memory Issues, leads with a detailed description (and actual code test cases) of common memory defects (bugs), that novice (and even experienced!) programmers often overlook. These include the OOB (Out Of Bounds) memory accesses (read/write overflows/underflows), UAF, UAR, leakage and double free
Ch 6 then then continues logically forward with a detailed discussion on Tools to detect the afore-mentioned memory defects with; here, we focus on using the well known Valgrind (memcheck) and on the newer, exciting Sanitizer toolset (compiler-based tools). We compare them point for point, and explain how the modern C/C++ developer must tackle memory defects
File I/O: a vast topic by itself; we divide the discussion into two parts. The reader is advised to first delve into Appendix A – File IO Essentials (available online) which covers the basic concepts. Our Ch 18 on Advanced File I/O goes deeper; most programmers are aware of and frequently use the “usual” read/write system calls. We show how to optimize performance in various ways- using the MT (multithreaded) optimal pread/pwrite APIs, using scatter-gather (SG-IO). An explanation (with detailed diagrams) on the kernel picture for the block IO code path, shows the reader how I/O actually works – the kernel page cache and related components are shown. Leveraging this knowledge with the posix_fadvise and the readahead APIs is described as well. Memory mapping as a powerful “zero-copy” technique for file IO is explained with sample code. More advanced areas such as DIO and AIO are briefly delved into as well, all with the idea that the developer can leverage the system for maximum I/O performance
Most programmers are (dimly) aware of the traditional Unix “permissions model”; we cover this in detail from a system programming perspective in Ch 7 – Process Credentials (we explain setuid/setgid/saved-set-ID programs). We emphasize though, that there is a superior modern approach – the POSIX Capabilities model – which is covered in Ch 8. Security concerns and how these get addressed form the meat of the discussion here
Being a book on system programming, we obviously cover Process Execution (Ch 9) and Process Creation (Ch 10) in depth; the fork system call and all its subtleties and intricacies. Towards this end, we encode the “The rules of fork” – a set of seven rules that help the developer truly understand how this important system call actually behaves. As part of this chapter, the wait API and its variations, and the famous Unix “orphan” and “zombie” process are covered as well
Again, as expected, we cover the topic of Signaling in depth over two whole chapters (11 and 12). Besides the basics, the book delves into practical details (with code, of course) in covering, for example, various techniques by which one can handle a very high volume of signals continually bombarding a process. The notion of software reentrancy (and the related reentrant-safety concept) is covered. Ways to prevent the zombie, using an alternate signal stack, etc are covered as well. The next chapter on Signaling delves into the intricacies of handling fatal signals in a real-world Linux application – a must-do! The book provides an effective “template code” to conveniently fulfil this purpose. Using real-time signals for IPC, and synchronous APIs for signaling is covered too
The chapter (13) on Timers covers both the traditional and the more recent powerful POSIX timers; here, we explain concepts via two interesting mini-projects – building a “how fast can you react” (CLI) game and a small “run:walk” timer app
Multithreading with Pthreads is again a vast area; this book devotes three whole chapters (14, 15, 16) to this important topic. In the first of this trilogy, we delve into the thread concept, how exactly it differs from a process and importantly, why threading is useful (with code examples of course). The next chapter deals in depth with the key topics of concurrency and synchronization (within the Pthreads domain, covering mutex locks and CVs); here, we use the (old but) very interesting Mars Pathfinder mission as a quick case study on priority inversion and how it can be prevented. Next, the really important topics of thread safety, cancellation and cleanup, are covered. The chapter ends with a brief “Multi processing vs MT” discussion and some typical FAQs. (We even include a ‘pthreads_app_template.c’ program in the book’s GitHub tree)
Ch 17 delves into intricacies of CPU Scheduling on the Linux OS; key concepts – the Linux state machine, what real-time is, POSIX scheduling policies available, etc are covered. We demonstrate a MT app that switches two worker threads to become (soft) real-time
The book ends with a small chapter (19) devoted to Trobleshooting and Best Practices to follow – a small but really key chapter!

Scattered throughout the book are interesting examples and reader assignments (some of which have solutions provided). Also, we try not to be completely x86 specific; in a few examples we cross-compile for the popular ARM-32 CPU; we mention the SEALS project (allowing one to quickly prototype apps on a Qemu emulated ARM/Linux system).

Who this book will benefit: