Stackpointer subtraction in listings 14-4, 14-5, 14-6

[yeah-boi]

The assembly code for listing 14-4 in listing 14-5 seems a bit wierd. RSP is not subtracted enough (only 0xf90) to hold char buffer[1024]. buffer is partly outside the stack and both a and b are completly outside the stack.

...
4004ba:       48 81 ec 90 0f 00 00    sub    rsp,0xf90
4004c1:       89 bd fc ef ff ff       mov    DWORD PTR [rbp-0x1004],edi
4004c7:       89 b5 f8 ef ff ff       mov    DWORD PTR 
...

Same issue with 14-6.

When I compiled and disassembled myself I got some extra bloat at 40055d-40056a and 400588-400597 (maybe you know why?), but RSP is subtracted more than enough (x1020) to hold buffeer, a and b:

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ gcc main.c -o test

$objdump -d -M intel test

...
0000000000400546 <maximum>:
  400546:	55                   	push   rbp
  400547:	48 89 e5             	mov    rbp,rsp
  40054a:	48 81 ec 20 10 00 00 	sub    rsp,0x1020
  400551:	89 bd ec ef ff ff    	mov    DWORD PTR [rbp-0x1014],edi
  400557:	89 b5 e8 ef ff ff    	mov    DWORD PTR [rbp-0x1018],esi
  40055d:	64 48 8b 04 25 28 00 	mov    rax,QWORD PTR fs:0x28
  400564:	00 00 
  400566:	48 89 45 f8          	mov    QWORD PTR [rbp-0x8],rax
  40056a:	31 c0                	xor    eax,eax
  40056c:	8b 85 ec ef ff ff    	mov    eax,DWORD PTR [rbp-0x1014]
  400572:	3b 85 e8 ef ff ff    	cmp    eax,DWORD PTR [rbp-0x1018]
  400578:	7d 08                	jge    400582 <maximum+0x3c>
  40057a:	8b 85 e8 ef ff ff    	mov    eax,DWORD PTR [rbp-0x1018]
  400580:	eb 06                	jmp    400588 <maximum+0x42>
  400582:	8b 85 ec ef ff ff    	mov    eax,DWORD PTR [rbp-0x1014]
  400588:	48 8b 55 f8          	mov    rdx,QWORD PTR [rbp-0x8]
  40058c:	64 48 33 14 25 28 00 	xor    rdx,QWORD PTR fs:0x28
  400593:	00 00 
  400595:	74 05                	je     40059c <maximum+0x56>
  400597:	e8 84 fe ff ff       	call   400420 <__stack_chk_fail@plt>
  40059c:	c9                   	leave  
  40059d:	c3                   	ret    
...

[sayon]

Thank you for your question. I was able to reproduce the book listings with -fno-stack-protector flag. Apparently this should be added to errata, which I am doing right now.

1. The strange things at you are seeing at 0x40055d-0x40056a and 0x400588-0x400597 are parts of the stack buffer overflow protection mechanism, explained in 14.8.1. See how this value from fs:0x28 is being placed next to the return address and old rbp value: this is a security cookie, if it gets overwritten the program craches by calling 400420 <__stack_chk_fail@plt>.

2. rsp is not subtracted enough because of the red zone, which is explained in 14.1.4.

The red zone is an area of 128 bytes that spans from rsp to lower addresses. It relaxes the rule “no data below rsp”; it is safe to allocate data there and it will not be overwritten by system calls or interrupts.

So, 0xf90 is decimal 3984, 3984 + 128 = 4112 = 0x1010 so that is how much place we really get if we subtract 0xf90 from rsp. Your listing shows a close enough number: 0x1020, which is different by 16. Why so?

You compiled the code with stack protection implicitly turned on. Because of that, we need an extra 8 bytes in stack immediately next to old rbp value in order to store the security cookie (the bloat you mentioned).

We subtract a bit more to keep rsp well aligned in case of future calls, such as call 400420 <__stack_chk_fail@plt>. When interacting with C code, the stack should usually be 16-bit aligned (a part of calling convention described in ABI).

When maximum is being called, the stack is 16-bit aligned. Then we do the following:

• call rsp -= 8
• push rbp rsp -= 8
• sub rsp, x
  Were x equal to 0x1018 (enough place for locals + security cookie), we would have broken the 16-bit alignment. It leads to undefined behavior when calling <__stack_chk_fail@plt>.

In case of stack protection turned off, this function performs no calls (leaf function), and we don't need to keep rsp aligned.

[yeah-boi]

Thanks for explaning this.