Advanced #Programming in the #UNIX Environment

Week 1, Introduction: What do we do in this class? Why do we want to learn Unix Programming in C? How are we going to do this? What's the syllabus?

youtu.be/BsB9Cg6yJc4

Slides (minor updates for this semester): stevens.netmeister.org/631/01-…

#apue #netbsd

Advanced #Programming in the #UNIX Environment

Week 1, Course Prep: Install NetBSD/evbarm in UTM on Apple M1 hardware

youtu.be/hCqfmuG5Acc

Note: you no longer need to use NetBSD-current. The #NetBSD 10.0 release will work just fine.

stevens.netmeister.org/631/utm…

If you prefer VirtualBox on Windows or Apple Intel hardware, see stevens.netmeister.org/631/vir…

After setting up your VM, follow this doc to make ssh seamless, fetch sources, and set your CFLAGS:

stevens.netmeister.org/631/vm-…

#apue

Advanced #Programming in the #UNIX Environment

Week 1, UNIX History

From Space Travel on a PDP-7 at Bell Labs in New Jersey to AT&T System V and the _B_erkeley _S_oftware _D_istribution, the Unix Wars, USL v. BSDi, the birth of #NetBSD (with the later fork into #OpenBSD) and #FreeBSD, the parallel development of GNU at MIT and its adoption of the new #Linux kernel, and how we ended up with Unix on your fridge, car, and mobile phone.

youtube.com/watch?v=3H7SQWTR6D…

Advanced #Programming in the #UNIX Environment

Week 1, Unix Basics

We write our first trivial program, which promptly segfaults and gives us a chance to explain function forward declarations and maneuvering man pages and set our default compiler flags ("-Wall -Werror -Wextra"), then proceed to write the most trivial versions of ls(1), cat(1), and even a rudimentary shell.

youtu.be/w3spRzZ8bSA

Slides: stevens.netmeister.org/631/01-…

#apue

Advanced #Programming in the #UNIX Environment

Week 2, File Descriptors

What is a file descriptor? We run through a code example to determine the maximum number of file descriptors a process can have open. Sounds simple enough, right? In the process, we look at compile-time defined constants, system-wide configuration parameters, look at the source for getconf(1), hit on rlimits, and confirm experimentally what the actual value is on #NetBSD, #Linux, and #macOS

youtu.be/h5A1OQjuCqk

#apue

Advanced #Programming in the #UNIX Environment

A nice bit of Unix history:

"A Research UNIX Reader: Annotated Excerpts from the Programmer’s Manual, 1971-1986" by M. Douglas McIlroy

cs.dartmouth.edu/~doug/reader.…

#apue

Advanced #Programming in the #UNIX Environment

Week 2, basic I/O with file descriptors: open(2) and close(2)

We see what happens when we try to create a file, "create" an existing file, open an existing file, truncate a file, and look at the various other flags we can pass. We also note the perhaps surprising number of ways in which open(2) can fail and take a look at openat(2) and TOCTOU defenses.

youtu.be/QnL4eYpb5Iw

#apue

Advanced #Programming in the #UNIX Environment

Week 2, read(2), write(2), and lseek(2)

We will look at these basic I/O syscalls and do a few weird things with file descriptors, like seek past the end of the file, creating a sparse file in the process. We'll also take a look at I/O efficiency when making these system calls and find out that simply increasing the read buffer isn't going to work beyond a certain point.

youtu.be/EUUPw8MOV_A

#apue

Advanced #Programming in the #UNIX Environment

Week 2, File Sharing

In this final video lecture segment for our week 2 materials, we take a look at the kernel structures used to implement the basic I/O syscalls and discuss the impact of multiple, simultaneous processes accessing the same files. We conclude with a look at /dev/fd on #NetBSD, #macOS, and #Linux.

youtu.be/CAbUtc86ULw

#apue

Advanced #Programming in the #UNIX Environment

Week 3, stat(2) intro

We meet our new best friend, the "struct stat"! We cover the stat(2) system calls (stat(2), lstat(2), fstat(2)) and begin discussing each of the fields in the struct, often by example of the ls(1) and stat(1) commands. In this way, we begin learning more about how the Unix File System is implemented.

The video also shows how to add a new disk to our #NetBSD VM in VirtualBox.

youtu.be/veFwXrEHPsk

#apue

Advanced #Programming in the #UNIX Environment

Week 3, UIDs and GIDs

We learn a little bit more about file ownership and the user IDs of a process by looking at the seteuid(2) and setuid(2) system calls and experimenting with setting the setuid bits and observe the file access we gain. Examples include, of course, ping(8) (we're not using e.g., capabilities here), traceroute(8), or even something as simple as wall(1) or write(1).

youtu.be/YlcF3ncTQpo

#apue

Advanced #Programming in the #UNIX Environment

Week 3, struct stat st_mode

In this short segment, we're looking at how the traditional permissions defined in the struct stat st_mode are applied. (We'll get to ACLs a bit later in the semester.) The order in which owner, group, and other permissions are evaluated explains why a file owner can write, but cannot read a file with permission 244, or that root can (of course) read a file despite permissions 000.

youtu.be/vS6abFjN3ZY

#apue

Advanced #Programming in the #UNIX Environment

Week 3, chmod(2) and chown(2)

Having seen a lot of examples in our previous videos of creating files, changing their ownership and permissions, we now focus on the system calls that are used to implement the chmod(1) and chown(8) commands. (As always, note the different manual sections, which hint at their intended use.)

youtu.be/5MJmCanZDyk

#apue

Advanced #Programming in the #UNIX Environment

Week 3, umask(2)

We take a look at what file ownership and permissions are applied when we create a new file. In the process, we'll learn about the concept of a process's "umask", and look at how the 'umask' shell built-in is implemented.

After this week, you should be able to implement most of chown(8), chmod(8), stat(1), cp(1), and, oh, I know: ls(1). Which is why we're making that the midterm project. 😀

youtu.be/bRAR2bv2HSM

#apue

Advanced #Programming in the #UNIX Environment

`umask` and file modes explained in one slide

Advanced #Programming in the #UNIX Environment

Week 3 bridge to week 4: Union Mounts and Whiteout Files

We describe the concept of union mounts and see what happens when a file in the upper layer is removed while the same file still exists in the lower layer: a whiteout file is created to cover up the lower file.

youtu.be/MkFExG9jhEE

#apue

Advanced #Programming in the #UNIX Environment

Week 4, The Unix Filesystem

We begin a closer look at the Unix Filesystem (#UFS) and the Berkeley Fast Filesystem (FFS). We see how the filesystem structures the disk and organizes files and directories, visualize the concept of hard links, and observe how operations on a directory are independent of the files and their data.

youtu.be/kY4JAXYyByQ

#apue

Advanced #Programming in the #UNIX Environment

Week 4, Links

We'll take a look at the system calls used to create, remove, and rename both hard- and symbolic links and when disk blocks are actually free'd. We'll compare expected output against trivial implementations of the ln(1) and mv(1) commands.

youtu.be/0l8o2goJ1kc

#apue

Advanced #Programming in the #UNIX Environment

Week 4, Directories

We take a look at how directories are created and removed, how to iterate directory entries using opendir(2), readdir(2) / getdents(2), and how to move around the filesystem hierarchy (with a quick hint of fts(3)). We also figure out why the 'cd' command must be a shell builtin in order to work, despite some systems providing a useless /usr/bin/cd (for some interpretation of #POSIX compliance).

youtu.be/xZ7dNXZ58G8

#apue

Advanced #Programming in the #UNIX Environment

The #UFS File System (from the -- by now old -- "#Solaris Internals" book)

ptgmedia.pearsoncmg.com/images…

Advanced #Programming in the #UNIX Environment

Week 4, Directory Size

We dive deep into the structure of the directory on a traditional Unix File System and see how its size is independent of the file _sizes_ of its entries, but dependent on the _filename lengths_. We'll also use hexdump(1) to cheat a bit and look at the directory structure on disk. (This works on #NetBSD. #Linux doesn't let you read(2) a directory; you can use debugfs(8) there.)

youtu.be/gY0SE-71LZA

Advanced #Programming in the #UNIX Environment

Week 4, All about Users and Groups

We try to answer the awkward question "Mommy, where do UIDs come from?" by inspecting
/etc/passwd and the lookup functions getpwuid(2)/getpwnam(2), and find out what weird edge cases may exist in that database. We also learn about how primary and supplementary groups are defined in /etc/group and how to retrieve them using getgrgid(2)/getgrnam(2) and getgroups(2).

youtu.be/fv16TWDnLYM

youtu.be/aomkx6_aWpc

Advanced #Programming in the #UNIX Environment

Week 4, File times: atime, mtime, ctime

• atime: time of last data access
• may trigger frequent I/O; diminishes performance
• remedies: mount -o noatime, nodiratime, relatime, lazytime
• mtime: time of last data modification
• ctime: time of last file status change (not creation time); automatically updated by various syscalls
• utimes(2) with values requires ownership; may update non-standard time fields

youtu.be/ffYEEj5vnlw

#apue

Advanced #Programming in the #UNIX Environment

Week 4, time(3) is an illusion...

We look a little bit closer into what our systems think of "#time", something that experts most accurately describe as a big ball of wibbly, wobbly, timey, wimey... stuff. We'll discuss managing the data structures used to represent "time" and handling arbitrary concepts such as leap seconds, timezones, and (ugh) Daylight Savings Time. It gets silly real quick.

youtu.be/3N2aH1vUacQ

#apue

Advanced #Programming in the #UNIX Environment

Week 5, The Unix Development Environment

This week, we're taking a detour from the book to talk about the native development tools in our environment, where, as it turns out, the Unix Userland _is_ an IDE – essential tools that follow the paradigm of “Do one thing, and do it right” and can be combined.

youtu.be/QYwYLR3UD2w

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: The Editor

We look at the required feature for a full-fledged programmer's editor and illustrate some of the core functionality by example of #vim. This includes basic motion commands, setting and moving to markers, using folds, and the use of the ':make' command and quick fix lists to address compiler errors efficiently.

youtu.be/DdaJ87G9Kes

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: ed(1) is the standard text editor

Yes, seriously, a video on ed(1). Hyperbole aside, this actually helps us understand other tools like vi(1), sed(1), and ultimately even things like diff(1) and patch(1) better.

youtu.be/mRZsV7aMK0I

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: ctags(1)

We introduce the ctags(1) utility, a tool to help you (and your $EDITOR) maneuver your source tree more efficiently by jumping to function definitions and back, all through the system sources under /usr/src.

youtu.be/TWog5NklSws

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: screen(1)

As another useful tool, we introduce the GNU screen(1) window manager and terminal multiplexer to allow you to manage your remote sessions. (tmux(1), which ships with #NetBSD as part of the base system, is a similarly useful alternative.) We demonstrate briefly basic window management, splitting, detaching and re-attaching (e.g., after a network interruption).

youtu.be/vxTXXaCr4s8

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: The #Compiler Chain, Part I

We begin our discussion of compilers as part of the Unix programming environment. We provide a high-level overview of how compilers process input source code and turn it into an executable. (And I show my age by mentioning "Borland" and "Turbo C". You can safely ignore those. 😂)

youtu.be/SOeo4zLBqdI

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: The #Compiler Chain, Part II

We look at the preprocessing stage of the compiler and observe how it invokes the cpp(1) command to include header files or replace macro definitions in place.

youtu.be/8qPxQ3g8bso

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: The #Compiler Chain, Part III

We continue to analyze the steps of the compilation process. This time, we look at the compilation proper and the optimization and assembly of the intermediate code into an object file.

youtu.be/ddvYL-G3oew

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: The #Compiler Chain, Part IV

We conclude our discussion of the compiler chain, analyzing the last step in the process: linking the object files into the executable. We identify aspects of the C Runtime libraries as well as the standard C library that we need to include and cover a few of the most important flags to ld(1).

youtu.be/8LCyPmjknBY

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: make(1)

In this video, we introduce the make(1) utility to help us selectively build our code project. We briefly touch on variable expansion, suffix rules, and some of the differences between GNU make and BSD make.

youtu.be/WLTuUtj7LPw

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: gdb(1)

We're entering the last couple of segments for this week, looking at more efficient debugging techniques beyond "printf('here\n');": using an actual debugger. We troubleshoot our failing programs and immediately identify the location of our program where it segfaults. We also see how to inspect variables and call functions from within the #gdb.

youtu.be/61YwbTQy2G0

youtu.be/Miw0XLzHCws

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: gdb(1) part II

We continue to explore the capabilities of the debugger. We show how you can set a breakpoint to pause the program and how to step through the program while watching our code.

youtu.be/fOp4Q4mnTD4

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: gdb(1), part III

We use our newly learned debugging skills to analyze and fix yet another flawed program. In the process, we learn to debug code from multiple source files and find out how to change the value of variables while the program is running.

youtu.be/hgcj7iAxhhU

#apue

Advanced #Programming in the #UNIX Environment

Week 5, Unix Development Tools: gdb(1), part IV

We're (finally) wrapping up this week's materials by using the debugger to examine memory locations in a running program and illustrating how pointers and arrays work in the C programming language. We also learn that a buffer overflow does not _necessarily_ lead to a segfault.

youtu.be/hr-rn0yUhAw

#apue

Advanced #Programming in the #UNIX Environment

sizeof != strlen

Now that we know how to use gdb(1), we use it to help us understand a common mistake made when programming in C: misunderstanding the unary 'sizeof' operator, which returns the _storage size of an expression or a data type_, not the length of a string or size of a dynamic buffer.

(This ultimately lead me to write the 'sizeof' manual page now included in #FreeBSD: man.freebsd.org/cgi/man.cgi?qu… )

youtu.be/FyPqII2rewA

#apue

Advanced #Programming in the #UNIX Environment

Week 6, Memory Layout of a Process

To visualize what a process looks like in memory, we create a program that prints out the addresses of different elements of a program and observe the alignment of the text, data, and bss segments as well as the placement of the heap and stack. We also illustrate what a stack overflow looks like.

Code: stevens.netmeister.org/631/apu…

youtu.be/aHmcowhoOzU

#apue

Advanced #Programming in the #UNIX Environment

Week 6, Program Startup

We all know how a program starts up, right?

int main(int argc, char **argv)

Well, not so fast! There's an entry point '_start' and it looks like it calls "main" with _three_ arguments:

cvsweb.netbsd.org/bsdweb.cgi/s…

```
void
___start(void (*cleanup)(void),
struct ps_strings *ps_strings)
{
[...]
exit(main(ps_strings->ps_nargvstr, ps_strings->ps_argvstr, environ));
}
```

Let's observe...

youtu.be/Yul4d-BDdx8

#apue

Advanced #Programming in the #UNIX Environment

Week 6, Process Termination

Having seen how a process starts in our last video, we'll now look at how we terminate a process, what the return status is, and how to register exit handlers using atexit(3). Together, this gives us a comprehensive view of the lifetime of a process.

youtu.be/xNeEw1f_js8

#apue

Advanced #Programming in the #UNIX Environment

Side-quest: what does this program return?

```
int main() {
printf("Hello World!\n");
}
```

The answer, as they say, may surprise you...

netmeister.org/blog/return-pri…

#apue

There is no 'printf'.

In C, 'main' returns an int. What happens when you don't explicitly return a value, though? C99 provides a return value of 0, but C89... well, that's undefined, my favorite kind of behavior.

^{www.netmeister.org}

Advanced #Programming in the #UNIX Environment

Week 6, The Environment

We're going to take a look at the process environment, using what we learned about the process layout in memory to understand how the environment variables are stored and, if necessary, moved around. We'll also get a quick look at what malloc(3) does.

youtu.be/8DEPA6nJXNY

#apue

Advanced #Programming in the #UNIX Environment

Week 6, The Environment (a tangent from 2011)

How large can the environment get? Is there a limitation on the size of a single environment variable? And why did I sometimes encounter this error message:

sudo: unable to execute <command>: success

netmeister.org/blog/sudo-unabl…

#apue

Advanced #Programming in the #UNIX Environment

Week 6, Process Limits and Identifiers

Having looked at the process layout in memory, how it starts and is terminated, we now learn that certain properties are restricted via resource limits, specified as a "soft" and a "hard" limit, with only the superuser being able to raise the latter.

A process also has a process ID (PID) and a parent process ID (PPID). More
on these process relationships in our next videos.

youtu.be/bnki8QKjSfQ

#apue

Advanced #Programming in the #UNIX Environment

Week 6, Process Control

In the last video lecture for this week, we look at process control: how new processes are started from an executable, what resources are shared between parent and child, and what happens when they terminate. In particular, we will look at the fork(2), exec(3), and wait(2) system calls. Be warned, though: there will be #zombies, so limber up (Rule #7).

youtu.be/KJq5nTCFsIg

#apue

Advanced #Programming in the #UNIX Environment

Week 6 related link: #Linux x86 Program Start Up or - How the heck do we get to main()?

Compare to what we covered in Segment 2 of this week.

dbp-consulting.com/tutorials/d…

#apue

Advanced #Programming in the #UNIX Environment

Week 7, The Login Process

In this week, we begin our discussion of process relationships, including process groups, sessions, and our first, asynchronous type of interprocess communication in the form of signals. To get us started, we first look at how processes created during the normal Unix boot process relate to one another. (And no, we will _not_ discuss #systemd. I said *normal* boot process.)

youtu.be/eNYTJbmYzH8

#apue

Advanced #Programming in the #UNIX Environment

Week 7, Process Groups and Sessions

We look at how processes are grouped together and begin to develop an understanding of how a login session relates to the controlling terminal:

• each process belongs to a process group
• a session is a collection of one or more process groups
• process groups are used for distribution of (keyboard generated) signals and to implement e.g., job control in a shell (see next video)

youtu.be/NfHqGv0PlIw

#apue

Advanced #Programming in the #UNIX Environment

Week 7, Job Control

We illustrate the concept of job control in the shell, first introduced in the C #shell, and allowing you to run multiple tasks from within the same terminal, switching back and forth between them by placing them into the background, suspending them, or bringing them to the foreground. It's one of my all-time favorite productivity hacks - Ctrl+Z FTW!

youtu.be/l6-663i8bwQ

#apue

Advanced #Programming in the #UNIX Environment

Week 7, Signals

Having seen how job control in the #shell often uses signals to communicate between process groups, let's take a closer look at these simple, asynchronous event notifications. We run through a number of examples to illustrate how signals are delivered, can be ignored, caught, handled, or blocked.

youtu.be/Vh7rBGj0Ty4

#apue

This video in a single slide:

Advanced #Programming in the #UNIX Environment

Some general advice on good coding style and readability:

The happy path is left-aligned:
medium.com/@matryer/line-of-si…

Return Early Pattern:
medium.com/swlh/return-early-p…

Stanford CS106 Coding Style
web.stanford.edu/class/cs106b/…

I'm a Never-Nester and you should, too"
mstdn.social/@jschauma/1095590…

#apue

Return Early Pattern - The Startup - Medium

When I learned about programming, the basic thought process that I had when creating a function was to validate the corresponding requirements until reaching the result. This is accomplished by…

^{Leonel Menaia Dev (The Startup)}