crash fails to load on ARM64 live kernel #54
Comments
vijaybalakrishna
changed the title
vijaybalakrishna
changed the title
|
Hello Vijay,
On Wed, Jun 24, 2020 at 2:42 AM vijaybalakrishna ***@***.***> wrote:
crash 7.2.8 fails to load live system kernel on ARM64 on 5.4.
# uname -a
Linux com7 5.4.42-xxxxxxxx-standard #1 SMP Wed Jun 3 05:17:19 UTC 2020 aarch64 aarch64 aarch64 GNU/Linux
# crash -s /proc/kcore vmlinux-stable-master-dev54
crash: /proc/kcore: cannot read vabits_actual
Is this a known issue? Currently when reading of vabits_actual symbol value from /proc/kallsyms isn't successful crash returns error (FATAL), instead should it continue to read tcr_el1_t1sz from VMCOREINFO to determine vabits_actual? Please share your thoughts.
Made a small source change to calculate the vabits_actual from vmcoreinfo's tcr_el1_t1sz, with the change I can load crash on live kernel. Tried few sample crash commands, they all seem to work.
# git diff arm64.c
diff --git a/arm64.c b/arm64.c
index 7662d71..d8515b2 100644
--- a/arm64.c
+++ b/arm64.c
@@ -3869,10 +3869,11 @@ arm64_calc_VA_BITS(void)
machdep->machspec->VA_BITS = value;
machdep->machspec->VA_START = _VA_START(machdep->machspec->VA_BITS_ACTUAL);
} else
- error(FATAL, "/proc/kcore: cannot read vabits_actual\n");
+ error(WARNING, "/proc/kcore: cannot read vabits_actual\n");
} else if (ACTIVE())
error(FATAL, "cannot determine VA_BITS_ACTUAL: please use /proc/kcore\n");
- else {
+
+ if (!machdep->machspec->VA_BITS_ACTUAL) {
if ((string = pc->read_vmcoreinfo("NUMBER(tcr_el1_t1sz)"))) {
/* See ARMv8 ARM for the description of
* TCR_EL1.T1SZ and how it can be used
com5:/tmp# /tmp/crash -s /proc/kcore /tmp/vmlinux
WARNING: /proc/kcore: cannot read vabits_actual
crash> sys
KERNEL: /tmp/vmlinux
DUMPFILE: /proc/kcore
CPUS: 8
DATE: Mon Jun 22 23:24:53 2020
UPTIME: 00:28:21
LOAD AVERAGE: 0.14, 0.05, 0.01
TASKS: 138
NODENAME: com5
RELEASE: 5.4.44-xxxxxxxx-standard
VERSION: #1 SMP Sat Jun 20 00:55:50 UTC 2020
MACHINE: aarch64 (unknown Mhz)
MEMORY: 7.8 GB
crash>
This is a known issue since the 52-bit VA changes were introduced in
the arm64 architecture kernel code. Please see
<https://lkml.org/lkml/2020/6/16/1443> for details.
You can use my public github crash/makedumpfile code for your checks
while we wait for the 52-bit VA changes to be accepted upstream.
Thanks,
Bhupesh
|
|
Hi Bhupesh, Thanks for taking a look. We already have your kernel patch Definition of vabits_actual I wonder if type of symbols defined above are in Thanks, |
|
Did some more digging. I rebuild kernel with KALLSYMS_ALL config, which lets symbols from data section in Before moving to 5.4 kernel we ran with 4.19 and crash loaded fine on live kernel. We didn't have KALLSYMS_ALL config enabled in 4.19. |
|
Hi Vijay,
On Tue, Jun 30, 2020 at 12:41 AM vijaybalakrishna ***@***.***> wrote:
Did some more digging. I rebuild kernel with KALLSYMS_ALL config, which lets symbols from data section in /proc/kallsyms, now I can load crash on ARM64 live kernel.
Before moving to 5.4 kernel we ran with 4.19 and crash loaded fine on live kernel. We didn't have KALLSYMS_ALL config enabled in 4.19.
# zcat /proc/config.gz | egrep -e CONFIG_KALLSYMS_ALL -e CONFIG_DEBUG_KERNEL
CONFIG_KALLSYMS_ALL=y
CONFIG_DEBUG_KERNEL=y
# grep vabits_actual /proc/kallsyms
ffff800010ca3a08 r __ksymtab_vabits_actual
ffff800010cb9ce7 r __kstrtab_vabits_actual
ffff8000110ee010 D vabits_actual
overlakedevcom6:~# crash -s /proc/kcore /tmp/vmlinux
strings: standard output: Broken pipe
crash> sys
KERNEL: /tmp/vmlinux
DUMPFILE: /proc/kcore
CPUS: 8
DATE: Sat Jun 27 05:56:56 2020
UPTIME: 00:12:46
LOAD AVERAGE: 0.13, 0.04, 0.01
TASKS: 155
NODENAME: com6
RELEASE: 5.4.44-xxxxxxxx-standard
VERSION: #1 SMP Sat Jun 27 05:05:09 UTC 2020
MACHINE: aarch64 (unknown Mhz)
MEMORY: 7.8 GB
crash>
Indeed, I was wondering why I was not able to reproduce this issue at
my end. This explains the difference in the environment: I have
KALLSYMS_ALL config enabled in my environment.
Thanks for sharing the same. Glad we were able to conclude why the
issue was seen in the first place.
Regards,
Bhupesh
|
|
First phase of support of the upcoming ARM64 kernel memory map patch introduced |
|
@vijaybalakrishna , could you pls tell me how to build crash binary running on arm64 host? I used make target=ARM64 but it still can only run in x86. Thanks. |
|
Hi,
On Tue, Jul 14, 2020 at 12:31 PM tiger20081015 ***@***.***> wrote:
@vijaybalakrishna <https://github.com/vijaybalakrishna> , could you pls
tell me how to build crash binary running on arm64 host? I used make
target=ARM64 but it still can only run in x86. Thanks.
If you are compiling the crash binary directly on the arm64 host, use the
following command line:
$ make lzo
i.e. directly use the target gcc compiler (which is configured for arm64
host) and git clone the crash-utility code there and use the above
command-line to compile the crash binary.
Hope this helps.
Thanks,
Bhupesh
|
|
@bhupesh-sharma ,thanks for your reply! |
For that you need a cross compiler. |
|
On 7/14/2020 11:27 PM, Santosh wrote:
@bhupesh-sharma <https://github.com/bhupesh-sharma> ,thanks for your
reply!
Actually I'm trying to compile the crash binary on the x86 host,
while the binary aims to run on arm64 host. Is there any method to
do this?
For that you need a cross compiler.
I have cross compiled using instructions from
Cross Compile files on x86 Linux host for 96Boards ARM systems
https://www.96boards.org/documentation/guides/crosscompile/commandline.html
- copy aarch64.cmake from above website
- install ARM 64bit toolchain
sudo apt-get install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu
(and dependencies, like below, I didn't)
sudo apt-get install build-essential autoconf libtool cmake
pkg-config git python-dev swig3.0 libpcre3-dev nodejs-dev
- in the dir where Makefile present run
cmake -DBUILDSWIG=NO -DBUILDSWIGNODE=NO -DBUILDSWIGPYTHON=NO
-DCMAKE_TOOLCHAIN_FILE=../aarch64.cmake .
make
- now check file type to see if it aarch64 binary
Vijay
…
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#54 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AP7XU7NB22MJ6ZZ32IQRCZLR3VD4DANCNFSM4OGBPHCA>.
|
|
On 7/14/2020 12:01 AM, tiger20081015 wrote:
@vijaybalakrishna <https://github.com/vijaybalakrishna> , could you pls
tell me how to build crash binary running on arm64 host? I used make
target=ARM64 but it still can only run in x86. Thanks.
I haven't build crash myself (but yocto does this for us!) to run on
arm64 host. I also run make target=ARM64, but generated crash tool runs
on x86_64 to load ARM64 crash dumps.
Vijay
…
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#54 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AP7XU7M7VQCJDOBOH5PTGJ3R3P7DZANCNFSM4OGBPHCA>.
|
|
Hi,
On Wed, Jul 15, 2020 at 7:27 AM tiger20081015 ***@***.***> wrote:
@bhupesh-sharma <https://github.com/bhupesh-sharma> ,thanks for your
reply!
Actually I'm trying to compile the crash binary on the x86 host, while the
binary aims to run on arm64 host. Is there any method to do this?
Hmm.. it depends on your use-case:
1. If you want to post-process/analyze only the vmcore dump file generated
on an arm64 target on a x86_64 host, use the following steps:
- First clean existing compiled binaries using $ make clean
- Then run
|
|
On Thu, Jul 16, 2020 at 3:29 AM Bhupesh SHARMA ***@***.***> wrote:
Hi,
On Wed, Jul 15, 2020 at 7:27 AM tiger20081015 ***@***.***> wrote:
>
> @bhupesh-sharma ,thanks for your reply!
> Actually I'm trying to compile the crash binary on the x86 host, while the binary aims to run on arm64 host. Is there any method to do this?
Sorry. It seemed I missed a few steps before sending out the email:
Hmm.. it depends on your use-case:
1. If you want to post-process/analyze only the vmcore dump file
generated on an arm64 target on a x86_64 host, use the following
steps:
- First clean existing compiled binaries using $ make clean
- Then run:
$ ./configure target=ARM64
$ make target=ARM64
- This will compile the crash binary for arm64 target but it still
needs to be run on a x86_64 host:
$ ./crash vmlinux(from arm64 system) vmcore(from arm64 system)
2. If you want to perform live-analysis on an arm64 machine/target,
use a aarch64 toolchain (you can get one from the Linaro website:
<https://releases.linaro.org/components/toolchain/binaries/>) and use
CROSS_COMPILATION techniques.
Although this is a less frequently used use-case.
Please let me know which use-case you want to target, so that I can
further share details.
Thanks,
Bhupesh
|
|
@bhupesh-sharma ， |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
crash 7.2.8 fails to load on ARM64 live kernel on 5.4.
Is this a known issue? Currently when reading of
vabits_actualsymbol value from /proc/kallsyms isn't successful crash returns error (FATAL), instead should it continue to readtcr_el1_t1szfrom VMCOREINFO to determinevabits_actual? Please share your thoughts.Made a small source change to calculate the
vabits_actualfrom vmcoreinfo'stcr_el1_t1sz, with the change I can load crash on live kernel. Tried few sample crash commands, they all seem to work.The text was updated successfully, but these errors were encountered: