error: build error: devmapper: Error mounting /dev/mapper/...

[bparees]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17094/test_pull_request_origin_extended_builds/634/

docker build failed:

---->
Step 1 : FROM centos/ruby-22-centos7:latest
 ---> 2143e48226c2
Step 2 : USER root
 ---> Running in cd445b51553f
 ---> ae58eb778691
Removing intermediate container cd445b51553f
Step 3 : RUN yum -y install expect
 ---> Running in e4cc54793540
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: mirror.cogentco.com
 * extras: reflector.westga.edu
 * updates: mirror.cs.vt.edu
Resolving Dependencies
--> Running transaction check
---> Package expect.x86_64 0:5.45-14.el7_1 will be installed
--> Processing Dependency: libtcl8.5.so()(64bit) for package: expect-5.45-14.el7_1.x86_64
--> Running transaction check
---> Package tcl.x86_64 1:8.5.13-8.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
================================================================================
 Package         Arch            Version                    Repository     Size
================================================================================
Installing:
 expect          x86_64          5.45-14.el7_1              base          262 k
Installing for dependencies:
 tcl             x86_64          1:8.5.13-8.el7             base          1.9 M
Transaction Summary
================================================================================
Install  1 Package (+1 Dependent package)
Total download size: 2.1 M
Installed size: 4.9 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               10 MB/s | 2.1 MB  00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : 1:tcl-8.5.13-8.el7.x86_64                                    1/2 
  Installing : expect-5.45-14.el7_1.x86_64                                  2/2 
  Verifying  : 1:tcl-8.5.13-8.el7.x86_64                                    1/2 
  Verifying  : expect-5.45-14.el7_1.x86_64                                  2/2 
Installed:
  expect.x86_64 0:5.45-14.el7_1                                                 
Dependency Installed:
  tcl.x86_64 1:8.5.13-8.el7                                                     
Complete!
 ---> 5cbdd0393036
Removing intermediate container e4cc54793540
Step 4 : RUN echo "root:redhat" | chpasswd
 ---> Running in a7151bce5647
 ---> d103b79ea2e5
Removing intermediate container a7151bce5647
Step 5 : USER 1001
 ---> Running in 0b3f72281da8
 ---> b32b97288061
Removing intermediate container 0b3f72281da8
Step 6 : COPY ./adduser /usr/libexec/s2i/
 ---> dca45971feab
Removing intermediate container be36da1f72ad
Step 7 : COPY ./assemble /usr/libexec/s2i/
 ---> 31b086153888
Removing intermediate container 8158ae31f784
Step 8 : ENV "OPENSHIFT_BUILD_NAME" "rootable-ruby-1" "OPENSHIFT_BUILD_NAMESPACE" "extended-test-build-s2i-dropcaps-85kbw-p9cnc"
 ---> Running in 07749c297bb5
error: build error: devmapper: Error mounting '/dev/mapper/docker-202:2-92453254-3f6e632f403acf8226bf82a67db75aff2104d359dc5dc9c756ce2f0b72911b46' on '/var/lib/docker/devicemapper/mnt/3f6e632f403acf8226bf82a67db75aff2104d359dc5dc9c756ce2f0b72911b46': invalid argument
<----end of log for "rootable-ruby-1-build"

[bparees]

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_builds/280/testReport/junit/(root)/Extended/_Feature_Builds__Slow__extremely_long_build_bc_names_are_not_problematic__build_with_long_names_delete_builds_with_long_names_without_collateral_damage__Suite_openshift_/

[bparees]

something has regressed, i'm seeing this in more and more failures.
https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17314/test_pull_request_origin_extended_builds/662/

[bparees]

@derekwaynecarr fyi.

[bparees]

@jwhonce @derekwaynecarr I need someone to at least triage this bug.

https://ci.openshift.redhat.com/jenkins//job/test_branch_origin_extended_builds/285/testReport/junit/(root)/Extended/_Feature_Builds__Slow__incremental_s2i_build__Building_from_a_template_should_create_a_build_from___tmp_fixture_testdata_dir912815358_test_extended_testdata_builds_incremental_auth_build_json__template_and_run_it__Suite_openshift_/

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_builds/285/testReport/junit/(root)/Extended/_Feature_Builds__Slow__completed_builds_should_have_digest_of_the_image_in_their_status__Docker_build_started_with_log_level__5_should_save_the_image_digest_when_finished__Suite_openshift_/

[bparees]

@jim-minter fyi since you're watching the extended tests this sprint.

[runcom]

Could you guys please provide output of docker version and info subcommands, lvs, dmsetup table and status, and journald entries for the period of failure. Thanks!

[jwhonce]

@runcom What exists for that period will be under the 'S3 Artifacts' tab on the left-hand side of screen.

docker info and journal entries are there.

[jim-minter]

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_builds/287/testReport/junit/(root)/Extended/_Feature_Builds__Slow__completed_builds_should_have_digest_of_the_image_in_their_status__Docker_build_started_with_log_level__5_should_save_the_image_digest_when_finished__Suite_openshift_/
https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_builds/288/testReport/junit/(root)/Extended/_Feature_Builds__Slow__completed_builds_should_have_digest_of_the_image_in_their_status__Docker_build_started_with_log_level__5_should_save_the_image_digest_when_finished__Suite_openshift_/

[jim-minter]

https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_builds/291/testReport/junit/(root)/Extended/_Feature_Builds__Slow__build_can_have_Docker_image_source__creating_a_build_with_an_input_source_image_and_s2i_strategy_should_resolve_the_imagestream_references_and_secrets__Suite_openshift_/
https://ci.openshift.redhat.com/jenkins/job/test_branch_origin_extended_builds/292/testReport/junit/(root)/Extended/_Feature_Builds__Slow__can_use_build_secrets__build_with_secrets_should_contain_secrets_during_the_docker_strategy_build__Suite_openshift_/

[bparees]

This is flaking a huge number of our build tests and resulting in excess triage work having to be done by our team.

[rhvgoyal]

Is there a way to reproduce this problem. Right now only error message I see is "invalid argument" and that really does not tell much in terms of what went wrong.

So that means we need to narrow it down. Is there a docker file which I can take and run on my VM and reproduce this?

Is the storage still fine in the system after the error occurs or one needs to reset storage to get going again.

[jim-minter]

I don't have a simplified way to reproduce the problem but it is happening on pretty much every test_branch_origin_extended_builds job, so it is hardly rare.

[rhvgoyal]

If we want to solve this issue, we will have to narrow it down. Otherwise I don't see how to make progress on this. So somebody needs to spend some time and try to reproduce it on smaller cluster and narrow it down.

[bparees]

@rhvgoyal You can add any debug/system information gathering you want to the extended test framework and repeatedly run the test jobs against your debug PR.

[rhvgoyal]

@bparees Sorry, I don't know enough about openshift to be able to do that. I will expect somebody in openshift team to narrow it down for me.

[bparees]

@bparees Sorry, I don't know enough about openshift to be able to do that. I will expect somebody in openshift team to narrow it down for me.

I don't consider that an acceptable expectation. The containers team is part of the openshift team. @jwhonce @derekwaynecarr

[0xmichalis]

https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/17662/test_pull_request_origin_extended_conformance_gce/12607/

[rhvgoyal]

@stevekuznetsov Updates are already there in comments. Please check.

[stevekuznetsov]

Ah, sorry, missed that -- do you know when that pacakge should be available in the RHEL 7 Next compose?

[calston]

This happens always when using the XFS based thinpool docker configuration produced by docker-storage-setup. Reverting back to overlay2 doesn't produce the issue anymore

[smarterclayton]

We've seen this happening in docker-1.12.6-68.gitec8512b.el7.x86_64 in https://openshift-gce-devel.appspot.com/build/origin-ci-test/pr-logs/pull/18133/test_pull_request_origin_extended_conformance_install/5693/

[stevekuznetsov]

We are waiting to tag in docker-1.12.6-71.git3e8e77d.el7.x86_64.rpm - the job tagging it in is broken, @michaelgugino is working on fixing it in openshift-eng/aos-cd-jobs#999

[jim-minter]

@stevekuznetsov any update?

[stevekuznetsov]

The job trying to push this up is here. It has flaked in the last four runs on conformance testing. If we need this faster, we can tag the new docker RPM through manually but in the past that has broken the entire Origin queue, hence the testing. If y'all feel like we need to manually tag this in, please let me know, but I will also need a commitment from one or more people to monitor the queue and ensure that regressions are caught immediately and we can revert quickly.

[bparees]

looks like that job is about to fail again. What caused the breakage last time? The rpm was actually bad?

[bparees]

btw it's going to fail because of the image GC problem:

  Normal   BackOff                4m (x60 over 19m)  kubelet, ip-172-18-7-2.ec2.internal  Back-off pulling image "openshift/origin-deployer:8b81ebd"

so if we could get the storage issue fixed, the job might have passed.

[stevekuznetsov]

The job was broken before because the installer entrypoints were changed but that job was not updated. Hmm. I'll see if I can get the storage stuff tackled today.

[stevekuznetsov]

The AMIs now have docker-1.12.6-71.git3e8e77d.el7

[bparees]

closing in the assumption this is supposed to be fixed by that level. will reopen as needed.

[smarterclayton]

Linked a run that failed that started at 2018-01-25 21:48 EST. Did that include the fix? (docker info doesn't include the micro version)

[smarterclayton]

It's probably not in the GCP image if the change didn't make it to the ops mirror or another prerelease location (tagged into centos7).

[bparees]

/assign @stevekuznetsov

[stevekuznetsov]

We're pushing Docker into the dockertested repo as we always have been. What do you mean?

[smarterclayton]

/assign @smarterclayton

[smarterclayton]

openshift/release#570 will merge after the overlay fix merges

[sosiouxme]

Appears to still be broken