Skip to content

ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion - #315

Open
mostafafaheem wants to merge 4 commits into
ravi9:dev_backend_openvinofrom
mostafafaheem:op_refactors
Open

ggml-openvino : simplify op translators and enable IMROPE/NEOX RoPE fusion#315
mostafafaheem wants to merge 4 commits into
ravi9:dev_backend_openvinofrom
mostafafaheem:op_refactors

Conversation

@mostafafaheem

Copy link
Copy Markdown
Collaborator

Overview

  1. Simplifying and cleaning up op translators of MUL_MAT, DIAG, GEGLU_QUICK, SWIGLU, SWIGLU_CLAMP, SWIGLU_OAI, ROPE, SILU and SOFTPLUS, using native OV ops when possible.
  2. Rewriting NEOX and IMROPE translators to align with ov RoPE fusion on CPU, GPU, NPU. They now emit ov::op::internal::Rope

Additional information

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: Yes, inefficiency analysis and some of the implementation

@wine99 wine99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me. But I have not built and verified the models

@mostafafaheem

Copy link
Copy Markdown
Collaborator Author

Changes look good to me. But I have not built and verified the models

I ran llama-bench on the 34 models in using a batch script on CPU, GPU and NPU, stateful and stateless, and there is no difference in model support

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants