Skip to content

fix(normalizer): keep the size before năm <year>; dates and units are not sizes (#1317) #1321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JOY (JOY) merged 2 commits into from
Oct 29, 2026
Merged

fix(normalizer): keep the size before năm <year>; dates and units are not sizes (#1317) #1321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2e01b9d
fix(normalizer): a size before "năm <year>" is kept; dates, ram, maco…
JOY Oct 28, 2026
ce96352
fix(normalizer): a size before "RAM 8GB", SSD or ROM stays the screen…
JOY Oct 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions apps/importer/src/shopee/__tests__/open-core-intelligence.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,34 @@ async function runSuite() {
['MacBook Air M2 8GB 256GB bảo hành 15 tháng', 'apple:macbook-air-m2:256gb'],
['MacBook Pro M3 18GB 512GB bảo hành 14 tháng', 'apple:macbook-pro-m3:512gb'],
['MacBook Air M2 8GB 256GB 15 gói quà', 'apple:macbook-air-m2:256gb'],
// "<size> năm <year>" names the size and the year (DOS/Rate#1317); a "năm"
// with no year after it is a duration.
['Máy tính bảng Apple iPad Pro 11 năm 2022 M2 128GB Wifi', 'apple:ipad-pro-m2-11:128gb'],
['Laptop Apple MacBook Pro 14 năm 2023 M3 8GB 512GB', 'apple:macbook-pro-m3-14:512gb'],
['MacBook Air 13 năm 2022 M2 8GB 256GB', 'apple:macbook-air-m2-13:256gb'],
['MacBook Air 15 năm 2023 M2 8GB 256GB', 'apple:macbook-air-m2-15:256gb'],
['iPad Air 11 năm 2024 M2 128GB', 'apple:ipad-air-m2-11:128gb'],
['iPad Pro 12.9 năm 2022 M2 128GB', 'apple:ipad-pro-m2-129:128gb'],
['MacBook Air M2 13 năm', 'apple:macbook-air-m2:unknown'],
// A date ("11/11"), RAM or SSD, a macOS name, a duration or an amount is no
// size, so the gift's size still stands in.
['iPad Pro M4 Sale 11/11 tặng ốp 13 inch', 'apple:ipad-pro-m4-13:unknown'],
['MacBook Pro M3 16 RAM 512 SSD tặng túi 14 inch', 'apple:macbook-pro-m3-14:unknown'],
['MacBook Air M3 macOS Sonoma 14 tặng túi 13 inch', 'apple:macbook-air-m3-13:unknown'],
['iPad Air M2 BH 12 months tặng ốp 11 inch', 'apple:ipad-air-m2-11:unknown'],
// ...but a size before "RAM 8GB" (a RAM size follows), "SSD" or "ROM" is the
// screen's, and short words like "Tr", "k", "đ" are no unit.
['iPad Pro M4 11 RAM 8GB ROM 256GB tặng ốp 13 inch', 'apple:ipad-pro-m4-11:256gb'],
['Macbook Air M2 13 RAM 8GB SSD 256GB', 'apple:macbook-air-m2-13:256gb'],
['MacBook Pro M3 14 RAM 18GB 512GB', 'apple:macbook-pro-m3-14:512gb'],
['MacBook Air M1 13 SSD 256GB', 'apple:macbook-air-m1-13:256gb'],
['iPad Pro M4 11 ROM 256GB', 'apple:ipad-pro-m4-11:256gb'],
['MacBook Pro 14 Tr M3 8GB 512GB', 'apple:macbook-pro-m3-14:512gb'],
['iPad Pro 11 k M4 256GB', 'apple:ipad-pro-m4-11:256gb'],
// A bare gift number is the gift's: the product's own size after it wins.
['iPad Pro M4 tặng ốp 11 | 13 WiFi', 'apple:ipad-pro-m4-13:unknown'],
['iPad Pro M4 13 tặng ốp 11 tặng bút', 'apple:ipad-pro-m4-13:unknown'],
['iPad Pro M4 tặng ốp 11 tặng bút', 'apple:ipad-pro-m4-11:unknown'],
// A number after "cho/for" (a school year, an age) is no size of the product.
[
'iPad Pro M4 Wifi 256GB tặng bao da 13 inch cho học sinh lớp 11',
Expand Down Expand Up @@ -735,6 +763,7 @@ async function runSuite() {
['Apple Watch Series 10 GPS tặng ốp 42mm / 46mm', 'apple:watch-series-10:64gb'],
['Apple Watch Series 10 GPS 46 tặng ốp 42mm', 'apple:watch-series-10:64gb'],
['Apple Watch Series 10 GPS giảm 40% tặng ốp 46mm', 'apple:watch-series-10-46mm:64gb'],
['Apple Watch Series 10 giảm 44 nghìn tặng ốp 46mm', 'apple:watch-series-10-46mm:64gb'],
// A size listed right after the box's is a list, not the watch's own case.
['Apple Watch Series 10 kèm hộp 46mm | 42mm', 'apple:watch-series-10-46mm:64gb'],
// "size 46" beats a box's size; a band's "size 42" is not the watch's.
Expand Down
16 changes: 11 additions & 5 deletions apps/importer/src/shopee/product-normalizer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1256,9 +1256,11 @@ function isProductCodeToken(token: string): boolean {
// js/polynomial-redos). A unit ends where a letter does, Vietnamese ones included: "13
// tặng" or "13 hãng" is a bare 13, not "13 t(b)" or "13 h"; "15 tháng" (warranty), "10%"
// or "15 gói" is no size (DOS/Rate#1291).
// "16 RAM 512" is the RAM; "13 RAM 8GB" (a RAM size follows) is the screen's 13
// (DOS/Rate#1317).
const UNIT_END = String.raw`(?![\p{L}\p{M}\d])`;
const SIZE_NOT_PRECEDED_BY = String.raw`(?<![\d.,a-z])(?<!(?:ram|os) )`;
const SIZE_NOT_FOLLOWED_BY = String.raw`(?![.,]?\d|[a-z]| ?(?:gb?${UNIT_END}|tb?${UNIT_END}|th${UNIT_END}|gen|mm|w${UNIT_END}|h${UNIT_END}|\/|-? ?(?:core|nhân|lõi)|cpu|gpu|tiếng|giờ|%|(?:tháng|ngày|tuần|năm|phút|gói|kỳ)${UNIT_END}))`;
const SIZE_NOT_PRECEDED_BY = String.raw`(?<![\d.,a-z])(?<!\d ?\/ ?)(?<!(?:ram|os|sonoma|sequoia|ventura|monterey|tahoe) )`;
const SIZE_NOT_FOLLOWED_BY = String.raw`(?![.,]?\d|[a-z]| ?(?:gb?${UNIT_END}|tb?${UNIT_END}|th${UNIT_END}|gen|mm|w${UNIT_END}|h${UNIT_END}|\/|-? ?(?:core|nhân|lõi)|cpu|gpu|tiếng|giờ|%|(?:tháng|ngày|tuần|phút|gói|kỳ|nghìn|ngàn|triệu|vnd|months?|years?|days?|weeks?|hours?)${UNIT_END}|ram${UNIT_END}(?! ?\d{1,2}(?!\d))|năm${UNIT_END}(?! ?(?:19|20)\d\d)))`;
const SIZE_MARKER = String.raw` ?(?:-? ?inch|in${UNIT_END}|"|''|”|″)`;
const IPAD_SIZE = String.raw`(11|12[.,]9|13)(?:[.,]0)?`;
const MACBOOK_AIR_SIZE = String.raw`(13|15)(?:[.,]\d)?`;
Expand All @@ -1285,14 +1287,17 @@ const CASE_ACCESSORY_WORD =
/(?<![\p{L}\d])(?:fits?|hộp|ốp|kèm|tặng|quà|gift|silicone|silicon|tpu|bao da|compatible|tương thích|cho|for)(?![\p{L}\d])/iu;
// "tặng/kèm" and up to four words, then a watch or screen size, or a list of them
// ("tặng ốp 42mm / 46mm", "tặng ốp 11 inch, 13 inch", DOS/Rate#1291).
const GIFT_SIZE = String.raw`(?:(?:38|4[0-2]|4[4-6]) ?mm\b|\d{1,2}(?:[.,]\d)? ?(?:inch\b|in\b|"))`;
const GIFT_BARE_END = String.raw`(?= ?(?:$|[,|;)+/]|(?:tặng|kèm|quà)(?![\p{L}\d])))`;
const GIFT_SIZE = String.raw`(?:(?:38|4[0-2]|4[4-6]) ?mm\b|\d{1,2}(?:[.,]\d)? ?(?:inch\b|in\b|")|(?:38|4[0-2]|4[4-6]|1[0-6](?:[.,]\d)?)${GIFT_BARE_END})`;
const GIFTED_ACCESSORY_SIZE = new RegExp(
String.raw`(?<![\p{L}\d])(quà tặng|tặng|kèm)((?: [\p{L}-]+){1,4}?) ?${GIFT_SIZE}(?: ?(?:\/|,|hoặc|or) ?${GIFT_SIZE})*`,
'giu'
);
// The sizes a gift names: a watch case ("46mm") or a screen ("11", "12.9").
const GIFT_SIZE_VALUE =
/(?<!\d)((?:38|4[0-2]|4[4-6]) ?mm|\d{1,2}(?:[.,]\d)?(?= ?(?:inch|in|")))/giu;
const GIFT_SIZE_VALUE = new RegExp(
String.raw`(?<!\d)((?:38|4[0-2]|4[4-6]) ?mm|\d{1,2}(?:[.,]\d)?(?= ?(?:inch|in|"))|(?:38|4[0-2]|4[4-6]|1[0-6](?:[.,]\d)?)${GIFT_BARE_END})`,
'giu'
);
// A size the product's part names in a form no size reading takes, so a gift's size must
// not stand in for it (DOS/Rate#1291): for a watch a bare case number or "size 46"
// ("Series 10 size 46 tặng ốp 42mm"); for an iPad or a MacBook a bare screen number or
Expand Down Expand Up @@ -1361,6 +1366,7 @@ function readAppleSize(model: string, rawTitle: string): string {
.replace(/ /g, '')
.replace(',', '.')
.replace(/\.0(?=mm|$)/, '')
.replace(/mm$/, '')
.toLowerCase()
);
}
Expand Down
Loading