Relationship target refs are causing errors on certain IDs.

[tigerkatz]

We are working to convert a repository of STIX 1 data to STIX 2 and plan to use the stix2elevator project. We have encountered an issue with some objects using ‘idref’ for objects that are outside of the package, but in the DB. It seems the stix2elevator is replacing these idrefs with a new identifier. Clearly the stix2elevator can not possibly map these IDs as it does not have access to the data outside the package. However, generating new IDs for these unresolvable refs seems like a problem.

This becomes an issue when stix2 IDs are created for relationship references like in stix_convert.py : fix_relationships on line 533:
if reference_needs_fixing(ref["target_ref"]): if not exists_id_key(ref["target_ref"]): new_id = generate_stix20_id(None, str.lower(ref["target_ref"]))
Additionally, our IDs follow the <name space>:<type>-<zero padded integer> pattern rather than the suggested practice of <name space>:<type>-<GUID>. This is causing a None Type Error when the elevator tries to create a stix2 ID for these references in ids.py : generate_stix20_id line 75:
`warn("Malformed id %s. Generated a new uuid", 605, stix12_id)

return stix20_so_name + "--" + text_type(uuid.uuid4())
ETypeError: unsupported operand type(s) for +: 'NoneType' and 'str'`
Thank you.

[rpiazza]

You state:

Additionally, our IDs follow the :- pattern rather than the suggested practice of :-

I'm not sure what you are trying to say. Maybe include an example id??

This function assumes ids with a certain pattern. Perhaps you need to special case your code to handle the pattern you use.

[tigerkatz]

My apologies.

Our IDs follow the ctap:TTP-00000000######## pattern rather than the suggested ctap:TTP-uuid4 pattern.

[rpiazza]

@tigerkatz, the STIX 2.0 spec says that an ID must use a uuid:

Identifiers MUST follow the form object-type--UUIDv4, where object-type is the exact value (all type names are lowercase strings, by definition) from the type property of the object being identified or referenced and where the UUIDv4 is an RFC 4122-compliant Version 4 UUID.

Perhaps there is some pre or post processing you can do to associate your STIX 1.x ids with the new valid STIX 2.0 ones?

[gtback]

You state:

Additionally, our IDs follow the :- pattern rather than the suggested practice of :-

I updated the original comment to put backticks (``) around what this was supposed to say:

Additionally, our IDs follow the <name space>:<type>-<zero padded integer> pattern rather than the suggested practice of <name space>:<type>-<GUID>

GitHub's Markdown renderer treats everything between angle brackets (<>) as an "HTML" tag and doesn't show it by default.

[gtback]

We should fix the TypeError caused by the NoneType. More broadly, I can see us doing one or more of the following:

• Providing a hook/configuration option for you to set a function that gets called to transform a STIX 1.x ID into a 2.x one. We can continue to use the current behavior as a default.
• Provide an option to write a "ID map" file which maps old IDs to the new generated ones (rather than needing to parse the elevator's entire log file.

[tigerkatz]

Thank you very much.

Yes, we are working on the pre-processing to re-map the IDs now. As it relates to the generation of new IDs in generate_stix20_id, to pass in a None value assumes that the existing ID follows a suggested practice in Stix1. It would be helpful if there was a way to handle this so that if the regex search in the method fails to extract a value for stix20_so_name from the ID, it would put some string value for stix20_so_name instead of trying to create a string to return with a None value (which causes a Type Error exception). Sorry for the mixup with my previous note here.

[gtback]

We should definitely fix the bug when you pass in None.

I'm tempted to hold off on the other things for now, since it sounds like you are working on your own methods of addressing that.

Thanks so much for reporting this, @tigerkatz !