Remove unsupported HID Feature Report 0xD3 from Traktor Z1 mapping

[djantti]

This PR removes unsupported HID Feature Report 0xD3 from Traktor Z1 crossfader calibration code. This lead to the mapping working fine on Linux, while throwing a TypeError on Windows.

Fixes #16172

[acolombier]

Shall we add a controller setting to make this an opt-in feature?

[djantti]

That would be a nice feature! A manual override for calibration values would also be useful, since doing a recalibration is only possible in NI's Traktor software.

I tested the PR Flatpak and Z1 calibration still works in Linux. Waiting on Windows tests, but everyone involved seems confident that this will fix the error. Hopefully we can keep the calibration feature in 2.5.5.

[tuulikauri]

Tested the script and it's working on Windows 11. Script now loads without issue on Z1 mk1. All buttons, knobs, and faders control Mixxx as expected. I commented the same on the bug.

The LED output seems broken when changes are made from pushing the controller's buttons.

Are you testing this mapping on a Z1 mk1? Or a mk2? Maybe that's why you are testing successfully but I am having weird results; I'm using a Z1 mk1, this one:

The issues are only with the LED feedback, not the controls sent to Mixxx and reflected on the Mixxx display. When changes are made by clicking with the mouse, the LEDs update as expected. But it's important that the LEDs not give misleading information, better that they are always off than wrong.

Steps to reproduce

• Turn off both FX and both headphone cue. Press the FX button to turn it on, press it again to turn it off and the opposite FX button lights up BRIGHT. The mode button flickers or lights BRIGHT or DIM. I expect the button that is activated to light bright, and no other LEDs to be updated.
• Both headphone cue buttons and both FX buttons lights are off when both FX are activated. I expect the headphone cue buttons to be dimly lit if the cue is off, and brightly lit if the cue is on; and to not be affected by the FX buttons. Pressing the headphone cue affects the FX button LEDs.
• The MODE buttons light is affected by the headphone cue and FX buttons. I'm not sure if this button/LED indicates anything in this mapping or controls anything; probably it should always be dimly lit or off. Currently it flickers, turns BRIGHT, or turns DIM as you press the other buttons. When you press it, it turns BRIGHT when held, and dim when released. If it doesnt control anything this feedback is confusing, since it looks like it does.
• Pressing the MODE button seems to reset all the other LEDs to their current status in Mixxx (DIM if the control is off; BRIGHT if the control is on)

[djantti]

Yes, that's the same controller I have.

I'm super confused here. 😳 None of those oddities happen on my Z1 and the LEDs function as they always have.

All LEDs that are off or not pressed are dimly lit. Only the activated buttons light up bright and pressing them does not affect any other LEDs. The mode button lights up bright only when held down (it's a momentary button to access alternate button functions) and does not reset other LEDs.

I tested my Z1 in a Windows VM (with USB passthrough) and it was fine there too. Everything worked exactly as it does on Linux.

Do the LEDs act weird for you also in Mixxx 2.5.4? Or is it just for the PR build?

[JoergAtGithub]

@tuulikauri Did you stop the NIHardwareService.exe as described in the manual: https://manual.mixxx.org/2.5/de/hardware/controllers/native_instruments_traktor_kontrol_z1#compatibility

[acolombier]

Regarding to the comment about making this feature an opt-in only, this might be better to do fix the issue with this so all calibration and this.rawCalibration related manipulation is done conditionally. Or are we sure that partial calibration can be done without any risks?

[acolombier]

It looks like this was allocated an array for the 3 reports. Do we only need 2 now? If yes, how about reserving only what is needed?

Suggested change

	this.rawCalibration.faders = new Uint8Array(0x20 * 3);
	this.rawCalibration.faders = new Uint8Array(0x20 * 2);

[JoergAtGithub]

Currently only two values of the second report are used. Suspect the other bytes contain the calibration values of the knobs and line faders.

[djantti]

Good catch, I'll correct the array size. 👍

NI's software has calibration settings for all the Z1 faders and knobs, so there's definitely more data. I still have Wireshark logs saved from trying to learn how everything is mapped to the controls.

Couldn't quite figure it out - at least not without doing several "bad calibrations" and possibly messing up the controller. 😅

[tuulikauri]

Wireshark sounds easier to use... but if that's not helping, NI's Controller Editor can be used to change calibration values for one control at a time. 2.7 builds can be used to view the Feature Reports. ie you could change the calibration for a single knob in Controller Editor, restart Mixxx 2.7, and read the Feature Reports to find which value changed. I'm sure there's a less-tedious way (I thought it was Wireshark?) but I'm not sure myself. Maybe with wireshark + Controller Editor, you can sniff the changes to the Feature Reports live? Also not sure if Controller Editor works on Linux, but thought I'd mention it. FWIW I would expect DJs to use Controller Editor to change calibration values, not do this inside Mixxx.

EDIT: Ahh, it actually groups controls. Only the crossfader is calibrated standalone, for the others it tells you to turn ALL the knobs LEFT together. And BOTH the faders UP.

[djantti]

Yeah, AFAIK that's the only way to be sure. Read the calibration data, change something, read the data again and compare. Unless you can immediately find an exact match for, say the value range of the crossfader. Then it's possible to make an educated guess.

I was doing Wireshark captures in a Windows virtual machine and it was just too scary to let NI's software write calibration data through USB passthrough. Bricking devices is not fun. 😬

[tuulikauri]

stop the NIHardwareService.exe

That fixed it! Great! (I had to stop the service from Task Manager->Services, the provided powershell gave an error Cannot open NIHardwareService on computer '.'.)

It was reproducible on both computers and multiple builds, but of course I also have Traktor installed on both computers soooo... I gather that's probably a potential issue for any NI controllers?

[JoergAtGithub]

stop the NIHardwareService.exe

That fixed it!

Great!

I gather that's probably a potential issue for any NI controllers?

As they all use the same driver, Yes!

As there are no NI drivers for Linux, Linux is not affected. Don't know about macOS.

[djantti]

Or are we sure that partial calibration can be done without any risks?

Hmm, it seems safe to me, since we're only reading the data and use it for setting min / max crossfader values. The Traktor S2 MK1 mapping does the same for all controls. But if we read bad calibration data from the device, then the controls might be messed up. I don't know if that can happen accidentally.

Hopefully we can one day figure out the full Z1 calibration data, use it in Mixxx and add a nice preferences GUI with a toggle and value overrides.

[tuulikauri]

crossfader values

Is it just the FeatureReport for the crossfader calibration we need confirmed? Please see attached data from Mixxx's HidIoThread::getFeatureReport (with this tweak to dump it to console..). I repeatedly calibrated the crossfader in NI's Controller Editor in different positions as described in the txt file and recorded the results from this debug dump, running > mixxx.exe --controller-debug Close mixxx, open controller editor, calibrate, close controller editor, run Mixxx, repeat.

Z1 crossfader calibration featureReports.txt

    auto featureReport = 
     QByteArray(reinterpret_cast<const char*>(dataRead + kReportIdSize),
            bytesRead - kReportIdSize);
    qCDebug(m_logInput) << featureReport; 
    return featureReport;

[djantti]

Very nice, thank you! 🙏

Ok, so the crossfader calibration code is jumping to index 60 on the array. That does seem to contain the changes. To be sure, I used the 80% left / right reports directly as test data in calibrate() and my Z1 crossfader behaved as weirdly as expected. So I'd say we're good.

[tuulikauri]

I used the 80% left / right reports directly as test data in calibrate() and my Z1 crossfader behaved as weirdly as expected.

I honestly didn't look at what the script does until now. I took a look and did one more test, and all seems good to me too!

The Z1 always sends 0 to 127 for the crossfader's position in MIDI, and (I presume..) the firmware on the controller reads the calibration values stored/edited via the featureReport. After changing the calibration settings again (80% left, 80% right) I still see MIDI 0 to 127 and HID inputReport 0 to 4095 but for FULL movement for HID, and for the CALIBRATED movement for MIDI. ie once calibrated, the calibration affects the MIDI output and reaches the max/min MIDI values at the calibrated points, leaving deadzones past that. That's what you would expect a calibration to do.

So the hardware does not actually apply it's calibration values to HID inputReports, it's up to software to use them. Which is what we are doing in crossfaderHandler by applying these calibration values. I tested the new script again and that is what is happening - now instead of reaching the ends of the crossfader with the FULL movement for HID, it behaves like MIDI and has dead spots at the ends where it was calibrated, and reaches the full HID range at the calibration points. I just wanted to make sure we aren't applying it a 2nd time, first in the hardware and then in Mixxx, which we are not.

Here's the featureReport I used to test with 80%/80% (80% is... very rough... heh. Somewhere past centre and notably not at the end of the physical movement...) my 0x01 inputReport byte position 0x1A goes from 0 to 4083 (I suppose there's some dirt stopping it from physically reaching 4095, which is the actual point of calibrating it, but this is an extreme fake exagerated calibration case). This means that people with dirty / corroded / manufacturer error'd crossfaders can now calibrate the dead spots on their controller (with Controller Editor or however they'd like...) and Mixxx will read those values and move the crossfader it's full range when it reaches the physical deadspot. Great!

debug [Main] 30 bytes received by hid_get_input_report "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "01" ) - Needed:  "121 us"
debug [Main] 33 bytes received by getFeatureReport from "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "D0" ) - Needed:  "108 us"
debug [Main] "\x01\x00\x10\x12\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00"
debug [Main] 33 bytes received by getFeatureReport from "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "D1" ) - Needed:  "99 us"
debug [Main] "\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b"
debug [Main] 33 bytes received by getFeatureReport from "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "D2" ) - Needed:  "110 us"
debug [Main] "\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x10\x00\x00\b\xF0\x0F\x00\x00\xF7\x0F\x00\x00\xF3\x0Fu\x04s\f"
debug [Main] 33 bytes received by getFeatureReport from "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "D8" ) - Needed:  "146 us"
debug [Main] "\x00\x01\x10\x12]\x83W\x83\xC9\xB7\xA2Q\x00\x00\x00\x00\x01\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFF\xFFT\x89T\x12"
debug [Main] 33 bytes received by getFeatureReport from "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "D9" ) - Needed:  "129 us"
debug [Main] "1543181478509119055888783\x00\x00\x00\x00\x00\x00\x00"
debug [Main] 15 bytes received by getFeatureReport from "Traktor Kontrol Z1 HID 835D_3" serial # "secret" (including one byte for the report ID: "F0" ) - Needed:  "81 us"
debug [Main] "\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05\x05"

[JoergAtGithub]

Thank you for the fix and for testing!