Add type to compound word

[renatoh]

Hi @uschindler
Is there a reason not to have a specific type on the compound words, so that I could inject a payload base on the type and make sure the extracted compound does get a lower boost than the original term?

[uschindler]

Hi,
I think this is a good addition. I am not sure if the type name "compound" is correct, as this is more about the parts of the compound. The compound itsself inherits the type of the original term (e.g., from the tokenizer).

As a general question: Do we also add separate types to synonyms or other generated tokens at same position at the moment?

[renatoh]

I think you are right, compound is the entire thing, maybe something like 'subword', let me know if you come up with something better.
Yes, we do that for the synonyms, but it is kind of broken in the SynonymGraphFilter, since all the tokens its emits, even the original, have the type 'synonyms', so it is kind of useless.

[rmuir]

I think TypeAttribute is an antipattern for cases like this. You can only set single value for the type and so it is no better than the old Token days.

Maybe consider SubwordAttribute which wraps a boolean, or something like that.

[renatoh]

@rmuir you are suggesting to create a brand new boolean attribute? I see you point, but with a custom attribute we could not use a generic filter like NumericPayloadTokenFilter to inject a payload and as far as I understand, it would require custom code to do so.
At the end, I think we should have a way to boost/demote subwords, compare to the original token, same applies for synonyms.

[uschindler]

I think both of you have good opinions. The problem of the type attribute is that it only allows one type. In current code of analyzers the original type is copied over.

As you see in the test, the tokenizer adds "word" as type (maybe different for other tokenizers, e.g. Japanese). The old code cloned all attributes, so the sub words also get "word" as type.

The good idea here is that additional (generated tokens) get a new type assigned, e.g "subword". That helps with generic filters added later like those for payloads. It is not ideal, but it helps to classify later token additions.

So in my opinion, this is fine with type attribute.

[rmuir]

I'm fine with subword as a type, really. But there is a limitation that you can't have a type of both synonym and subword.

[uschindler]

I'm fine with subword as a type, really. But there is a limitation that you can't have a type of both synonym and subword.

Yes, if you expand synonyms and decompound then this hits you. But your filter chain could handle that. After synonyms you can add a payload filter that would trigger on synonyms and then later add decompounding with another payload filter.

The whole token filter chain is problematic with that type of combinations. The whole type concept is feeling like Doug Cuttings original Token class. 😝

[uschindler]

Can you add a changes entry. I think Lucene 10.6 should be fine as it's easy to backport

[renatoh]

@uschindler something like this?

[renatoh]

@rmuir SynonymGraphFilter emits all the token, even the original one, as type:Synonym, this seems broken to me. Do you have an idea why this is like this? I have observed this in Lucene 9.11

[uschindler]

Not plural, otherwise fine.

[uschindler]

The constant should be named TOKEN_TYPE_SUBWORD

[mkhludnev]

There's a conception to substituting constant to tests that let to detect a change in constant value which may break a client otherwise.

[renatoh]

good point, thanks for your input.