diff --git a/modules/CommunityClusters/MANUAL.md b/modules/CommunityClusters/MANUAL.md deleted file mode 100644 index 1d58163fe..000000000 --- a/modules/CommunityClusters/MANUAL.md +++ /dev/null @@ -1,240 +0,0 @@ -# Community Clusters — User Manual - -Community Clusters is a layout for Gephi. It arranges a network so that each community forms its own -visible cluster, with the clusters themselves placed apart from each other. Inside a cluster, nodes -are arranged by their ties, so well-connected nodes sit together and weakly connected ones drift to the edge. - -The layout is inspired by the Clusters view of SoFab (see [Credits](#7-credits)), and is built for Gephi 0.11.3. - ---- - -## 1. Quick start - -**Using communities you already have** - -1. Open your graph in Gephi. -2. Run **Statistics > Modularity** (Run). This creates the node column `modularity_class`. -3. Open **Window > Layout** and choose **Community Clusters** in the dropdown. -4. Press **Run**. The layout stops by itself when it has settled. - -**Letting the plugin find the communities** - -1. Open your graph. -2. In the layout settings, clear the **Cluster column** field (or enter a name that does not exist). -3. Press **Run**. The plugin detects communities with the Leiden algorithm and, by default, saves - them to a node column called `community_cluster` so you can colour nodes by it - (Appearance > Nodes > Partition > `community_cluster`). - ---- - -## 2. How it works - -1. **Find the communities.** Each node gets a cluster, either from the column you name or from Leiden. -2. **Give each cluster a home.** One circle is drawn per cluster, with an area that grows with the number of - nodes in it, and the circles are packed together without overlapping, largest first. - The centre of each circle is that cluster's home position. -3. **Simulate.** Nodes start close to their cluster's home and a force simulation runs: - - **Ties** pull connected nodes towards their resting length. Ties inside a cluster act at full - strength; ties between clusters are weakened (see *Inter-cluster tie factor*), so they do not - drag clusters into each other. - - **Repulsion** pushes nearby nodes apart. It only acts within a fixed distance of 140 units - (multiplied by *Scale*), so distant clusters do not push each other around. - - **Collision** stops nodes from overlapping. - - **Cluster pull** draws every node towards its cluster's home. -4. **Cool down.** The forces fade out over time. When they are negligible the layout stops itself. - -The layout works on the visible graph, so active filters are respected. Nodes you have fixed -(locked) in Gephi do not move, but other nodes still react to them. - ---- - -## 3. Settings - -The settings are grouped in three categories in the layout panel. Defaults are shown in brackets. - -### 3.1 Clusters - -#### Cluster column (default: `modularity_class`) - -The node column that says which community each node belongs to. Any column works, whether numbers or -text: nodes with the same value form one cluster. - -- If the column exists, it is used as it is. No community detection is run. -- If the field is empty, or names a column that does not exist, Leiden community detection runs instead. - A name that was typed but not found also writes a warning to the Gephi log, so typos are easy to spot. -- The name is matched against the column id and the column title, ignoring upper and lower case. -- Nodes with an empty value in the column are grouped together as one cluster. - -**Tip:** `modularity_class` is the column created by Gephi's own Modularity statistic, so the default -uses its result whenever you have run it. If you have run Modularity but want the plugin's Leiden -instead, change this field to something that does not exist, for example `none`. - -#### Resolution (default: 1.0, minimum 0.01) - -Only used when Leiden runs. Controls how fine-grained the detected communities are. - -| Value | Effect | -|---|---| -| Below 1 | Fewer, larger communities | -| 1 | Standard modularity | -| Above 1 | More, smaller communities | - -If the clusters are too coarse and merge groups you see as distinct, raise it (try 1.5 or 2). If you -get many small fragments, lower it (try 0.5). - -#### Save clusters to column (default: on) - -Only used when Leiden runs. When on, the detected community of each node is written to an integer node column -named `community_cluster`. It is overwritten each time Leiden runs. You can then: - -- colour nodes by it (Appearance > Nodes > Partition), -- use it in the Data Laboratory or in filters, -- type `community_cluster` in **Cluster column** to reuse exactly this result later without - running detection again. This is also a way to freeze a result so it does not change. - -Turn it off if you do not want an extra column in your data. - -#### Cluster pull (default: 0.14, minimum 0) - -How strongly each node is drawn towards its cluster's home position. This is the main control for how -separated the clusters look. - -| Value | Effect | -|---|---| -| 0 | No home position; ties and repulsion alone decide, and clusters may blend | -| 0.05 – 0.1 | Loose clusters, ties between clusters still have visible influence | -| 0.14 | Default; clear clusters | -| 0.3 – 0.6 | Tight, round, well separated blobs | - -Very high values can squeeze nodes into dense balls where collisions have to push them apart. - -#### Inter-cluster tie factor (default: 0.03, minimum 0) - -How strong ties between clusters are compared to ties inside a cluster. At 0.03 a tie between two -clusters pulls only 3 % as hard as a tie within one. - -- **Lower (towards 0):** clusters ignore each other's ties, so the layout shows clusters as separate - islands. -- **Higher (towards 1):** between-cluster ties draw related clusters closer together, so the - overall structure of the network is easier to read, at the price of less clean separation. - At 1 the cluster ties are treated like all others. - -### 3.2 Forces - -#### Repulsion (default: 40, minimum 0) - -How hard nodes push each other apart. Higher values spread nodes out within their clusters and give -larger clusters; lower values pack them densely. Repulsion only acts within 140 units (times *Scale*). - -#### Tie length (default: 24, minimum 0) - -The resting length of a tie, added to the sizes of the two nodes it connects. Longer ties give a more -open layout. It works together with *Repulsion*: if you raise one, you often want to raise the other. - -#### Tie strength (default: 1, minimum 0.01) - -Multiplies how hard ties pull their ends together. Internally each tie's strength is divided by the -smaller of its two nodes' number of ties, so hubs are not pulled apart, and capped at 1. The result: -increasing this setting mostly affects nodes with few ties, and above a certain point it has no -additional effect. Lower values loosen the layout, with *Repulsion* dominating. - -#### Node spacing (default: 2, minimum 0) - -Extra room kept around each node by the collision force, in addition to the node's own size. -Raise it to keep node labels or outlines from touching; set it to 0 to let nodes touch. - -#### Scale (default: 1, minimum 0.01) - -Multiplies all distances in the layout together (tie length, spacing, repulsion range, size of the -cluster circles and the 140-unit repulsion range), and keeps forces in proportion. The plugin treats -Gephi's node *size* as the node radius. - -- If your nodes are large (for example sizes 50 to 200), nodes will overlap and clusters will be - crowded. Raise *Scale* until the layout looks right. -- If your nodes are very small, lower it. - -### 3.3 Run - -#### Settling speed (default: 0.0228, range 0.0005 – 0.5) - -How quickly the simulation cools down. It decides how long the layout runs before it stops by itself. - -| Value | Approx. number of steps | -|---|---| -| 0.1 | 65 | -| 0.0228 | 300 | -| 0.01 | 690 | -| 0.005 | 1,380 | - -Lower values take longer but usually give a more even, better untangled result. Higher values are -quick for a first look at a large graph. - -#### Random seed (default: 42) - -Seeds the random starting positions and the Leiden algorithm. With the same graph, same settings and -same seed you get the same result every time. Change it to get a different arrangement, which is -useful if you want to see whether a feature of the layout is stable. - -#### Reset positions (default: on) - -- **On:** every run starts from fresh positions around each cluster's home. -- **Off:** the run starts from the current positions of the nodes. Use it to continue a layout that - has already been run, or to refine it with different settings without starting over. If all nodes - are at the same position, leave it on. - -Fixed nodes always keep their current position. - ---- - -## 4. Recommended workflows - -**Presenting known communities.** Run Modularity (or use your own column), choose Community Clusters, -leave the defaults. Raise *Cluster pull* to 0.3 for cleaner separation. - -**Exploring how clusters relate.** Raise *Inter-cluster tie factor* to 0.2 – 0.5, lower *Cluster pull* to -0.08. Clusters with many ties between them will move closer. - -**Tuning the number of communities.** Clear *Cluster column*, run, and adjust *Resolution* until the -number of clusters suits your question. When satisfied, type `community_cluster` into *Cluster column* -to lock that partition. - -**Polishing.** After a first run, lower *Settling speed* to 0.008 and run again with *Reset positions* -off. Then run Gephi's **Noverlap** or **Label Adjust** if you need to clean up labels. - ---- - -## 5. Notes and limits - -- **Isolated nodes** (nodes without ties) cannot be assigned to a community by Leiden, so each one becomes - its own one-node cluster and gets its own place in the packing. A graph with many isolated nodes will - therefore show many tiny circles. Remove them with a filter (Degree Range) before running if this is not wanted. -- **Directed graphs** are treated as undirected. Parallel edges and edges in both directions are added - together, and edge weights are used. Self-loops and edges with zero or negative weight are ignored. -- **Leiden quality** is measured with modularity. Like all modularity methods it can miss very small - communities in very large graphs. Raise *Resolution* if that matters. -- **Speed.** Leiden takes a few seconds for 50,000 nodes and 250,000 edges. The layout itself takes - tens of milliseconds per step for tens of thousands of nodes. -- **Gephi's own Modularity column** (`modularity_class`) is produced by a different implementation (Louvain) - than the Leiden used here, so the two can give somewhat different communities for the same graph. - ---- - -## 6. Troubleshooting - -| Problem | What to try | -|---|---| -| The layout is not in the dropdown | Open a project first; check the plugin is listed in the root `pom.xml` and the build succeeded | -| Everything ends up as one cluster | The cluster column has the same value everywhere. Choose another column, or clear the field to use Leiden | -| Far too many clusters | Lower *Resolution*; check for isolated nodes (see above) | -| Clusters overlap each other | Raise *Cluster pull*; lower *Inter-cluster tie factor*; if nodes are large, raise *Scale* | -| Nodes overlap inside clusters | Raise *Node spacing* or *Scale*; raise *Repulsion* | -| Clusters look like perfect circles with no structure | Lower *Cluster pull* and raise *Tie strength* so ties shape the inside | -| The layout never seems to stop | Lower values of *Settling speed* run longer; raise it to finish sooner | -| The layout stops immediately | The graph is empty, or the previous run is finished: press Run again | -| A warning about a column in the Gephi log | The name in *Cluster column* was not found, so Leiden was used | - ---- - -## 7. Credits - -The layout is inspired by Fierăscu, S. I. (2026). *SoFab: Network analysis for research and practice* (Version 0.9.0) [Computer software]. diff --git a/modules/CommunityClusters/README.md b/modules/CommunityClusters/README.md index 301625fde..57a3c680f 100644 --- a/modules/CommunityClusters/README.md +++ b/modules/CommunityClusters/README.md @@ -1,47 +1,295 @@ # Community Clusters -Gephi layout plugin that gathers each community of a network into its own cluster. -See `MANUAL.md` for usage and a description of every setting. +By [Johan Myrberger](https://www.linkedin.com/in/myrberger/) -## Build and run +Community Clusters is a layout for Gephi. It arranges a network so that each community forms its own +visible cluster, with the clusters themselves placed apart from each other. Inside a cluster, nodes +are arranged by their ties, so well-connected nodes sit together and weakly connected ones drift to the edge. -```bat -cd /d C:\gephidev\gephi-plugins -mvn clean package -mvn org.gephi:gephi-maven-plugin:run -``` +Any categorical node field can be used as the **Cluster column**, whether its categories are numbers or text: +a community number, a department name, a country, a type. Or let the plugin detect the communities itself. +It is built for Gephi 0.11.4. -The module must be listed in the root `pom.xml`: +--- -```xml -modules/CommunityClusters -``` +## 1. Quick start -## Source files +**Using communities you already have** -| File | Role | +1. Open your graph in Gephi. +2. Run **Statistics > Modularity** (Run). This creates the node column `modularity_class`. +3. Open **Window > Layout** and choose **Community Clusters** in the dropdown. +4. Press **Run**. The layout stops by itself when it has settled. + +**Letting the plugin find the communities** + +1. Open your graph. +2. In the layout settings, open the **Cluster column** list and choose **(none: detect communities with Leiden)**. +3. Press **Run**. The plugin detects communities with the Leiden algorithm and, by default, saves + them to a node column called `community_cluster` so you can colour nodes by it + (Appearance > Nodes > Partition > `community_cluster`). + +--- + +## 2. How it works + +1. **Find the communities.** Each node gets a cluster, either from the column you choose or from Leiden. +2. **Give each cluster a home.** One circle is drawn per cluster, with an area that grows with the number of + nodes in it, and the circles are packed together without overlapping, largest first. + The centre of each circle is that cluster's home position. +3. **Simulate.** Nodes start close to their cluster's home and a force simulation runs: + - **Ties** pull connected nodes towards their resting length. Ties inside a cluster act at full + strength; ties between clusters are weakened (see *Inter-cluster tie factor*), so they do not + drag clusters into each other. + - **Repulsion** pushes nearby nodes apart. It only acts within a fixed distance of 140 units + (multiplied by *Scale*), so distant clusters do not push each other around. + - **Collision** stops nodes from overlapping. + - **Cluster pull** draws every node towards its cluster's home. +4. **Cool down.** The forces fade out over time. When they are negligible the layout stops itself. + +The layout works on the visible graph, so active filters are respected. Nodes you have fixed +(locked) in Gephi do not move, but other nodes still react to them. + +--- + +## 3. Settings + +The settings are grouped in three categories in the layout panel. Defaults are shown in brackets. + +### 3.1 Clusters + +#### Cluster column (default: `modularity_class`) + +The node column that says which community (category) each node belongs to. Nodes with the same value +form one cluster. The setting is a drop-down list that shows the categorical node columns of your +graph: columns holding whole numbers, text, or true/false values. Columns with decimal numbers, lists +or time-dependent values, and the built-in `Id` and `Label` columns (different for every node), are not +shown. + +- **A column is chosen:** its values are used as they are. No community detection is run. +- **(none: detect communities with Leiden):** the plugin finds the communities itself (see *Resolution* + and *Save clusters to column*). +- Nodes with an empty value in the column are grouped together as one cluster. +- The list shows the categorical node columns of the current project. +- If a project was saved with a column that no longer exists, the layout falls back to Leiden and writes a + note to the Gephi log. + +**Tip:** `modularity_class` is the column created by Gephi's own Modularity statistic, so the default +uses its result whenever you have run it. If you have run Modularity but want the plugin's Leiden +instead, choose **(none: detect communities with Leiden)**. + +#### Resolution (default: 1.0, minimum 0.01) + +Only used when Leiden runs. Controls how fine-grained the detected communities are. + +| Value | Effect | |---|---| -| `CommunityClusters.java` | The Gephi layout: properties, reading the graph, writing positions | -| `CommunityClustersBuilder.java` | Registers the layout with Gephi | -| `ClusterForceSimulation.java` | The force simulation (pure Java) | -| `CirclePacker.java` | Packs one circle per cluster to give each a home position | -| `Leiden.java` | Community detection (pure Java) | -| `Bundle.properties` | All user-visible text (setting names, descriptions, layout name) | -| `src/test/java/...` | JUnit tests for Leiden, the circle packer and the force simulation | +| Below 1 | Fewer, larger communities | +| 1 | Standard modularity | +| Above 1 | More, smaller communities | +If the clusters are too coarse and merge groups you see as distinct, raise it (try 1.5 or 2). If you +get many small fragments, lower it (try 0.5). -## Tests +#### Save clusters to column (default: on) -`mvn clean package` runs the unit tests automatically. To run only this module's tests: +Only used when Leiden runs. When on, the detected community of each node is written to an integer node column +named `community_cluster`. It is overwritten each time Leiden runs. You can then: -```bat -mvn -pl modules/CommunityClusters -am test -``` +- colour nodes by it (Appearance > Nodes > Partition), +- use it in the Data Laboratory or in filters, +- choose `community_cluster` in **Cluster column** to reuse exactly this result later without + running detection again. This is also a way to freeze a result so it does not change. + +Turn it off if you do not want an extra column in your data. + +#### Cluster pull (default: 0.1, minimum 0) + +How strongly each node is drawn towards its cluster's home position. This is the main control for how +separated the clusters look. + +| Value | Effect | +|---|---| +| 0 | No home position; ties and repulsion alone decide, and clusters may blend | +| 0.05 | Loose clusters, ties between clusters still have visible influence | +| 0.1 | Default; clear clusters | +| 0.2 – 0.6 | Tight, round, well separated blobs | + +Very high values can squeeze nodes into dense balls where collisions have to push them apart. + +#### Inter-cluster tie factor (default: 0.03, minimum 0) + +How strong ties between clusters are compared to ties inside a cluster. At 0.03 a tie between two +clusters pulls only 3 % as hard as a tie within one. + +- **Lower (towards 0):** clusters ignore each other's ties, so the layout shows clusters as separate + islands. +- **Higher (towards 1):** between-cluster ties draw related clusters closer together, so the + overall structure of the network is easier to read, at the price of less clean separation. + At 1 the cluster ties are treated like all others. + +### 3.2 Forces + +#### Repulsion (default: 40, minimum 0) + +How hard nodes push each other apart. Higher values spread nodes out within their clusters and give +larger clusters; lower values pack them densely. Repulsion only acts within 140 units (times *Scale*). + +#### Tie length (default: 24, minimum 0) + +The resting length of a tie, added to the sizes of the two nodes it connects. Longer ties give a more +open layout. It works together with *Repulsion*: if you raise one, you often want to raise the other. + +#### Tie strength (default: 1, minimum 0.01) + +Multiplies how hard ties pull their ends together. Internally each tie's strength is divided by the +smaller of its two nodes' number of ties, so hubs are not pulled apart, and capped at 1. The result: +increasing this setting mostly affects nodes with few ties, and above a certain point it has no +additional effect. Lower values loosen the layout, with *Repulsion* dominating. + +#### Node spacing (default: 2, minimum 0) -## License +Extra room kept around each node by the collision force, in addition to the node's own size. +Raise it to keep node labels or outlines from touching; set it to 0 to let nodes touch. -Apache License 2.0, see `LICENSE`. +#### Scale (default: 1, minimum 0.01) -## Credits +Multiplies all distances in the layout together (tie length, spacing, repulsion range, size of the +cluster circles and the 140-unit repulsion range), and keeps forces in proportion. The plugin treats +Gephi's node *size* as the node radius. -The layout is inspired by Fierăscu, S. I. (2026). *SoFab: Network analysis for research and practice* (Version 0.9.0) [Computer software]. +- If your nodes are large (for example sizes 50 to 200), nodes will overlap and clusters will be + crowded. Raise *Scale* until the layout looks right. +- If your nodes are very small, lower it. + +### 3.3 Run + +#### Settling speed (default: 0.03, range 0.0005 – 0.5) + +How quickly the simulation cools down. It decides how long the layout runs before it stops by itself. + +| Value | Approx. number of steps | +|---|---| +| 0.1 | 65 | +| 0.03 | 230 | +| 0.02 | 340 | +| 0.01 | 690 | +| 0.005 | 1,380 | + +Lower values take longer but usually give a more even, better untangled result. Higher values are +quick for a first look at a large graph. + +#### Random seed (default: 4711) + +Seeds the random starting positions and the Leiden algorithm. With the same graph, same settings and +same seed you get the same result every time. Change it to get a different arrangement, which is +useful if you want to see whether a feature of the layout is stable. + +#### Reset positions (default: on) + +- **On:** every run starts from fresh positions around each cluster's home. +- **Off:** the run starts from the current positions of the nodes. Use it to continue a layout that + has already been run, or to refine it with different settings without starting over. If all nodes + are at the same position, leave it on. + +Fixed nodes always keep their current position. + +--- + +## 4. Recommended workflows + +**Presenting known communities.** Run Modularity (or use your own column), choose Community Clusters, +leave the defaults. Raise *Cluster pull* to 0.3 for cleaner separation. + +**Exploring how clusters relate.** Raise *Inter-cluster tie factor* to 0.2 – 0.5, lower *Cluster pull* to +0.08. Clusters with many ties between them will move closer. + +**Tuning the number of communities.** Choose *(none: detect communities with Leiden)* in *Cluster column*, +run, and adjust *Resolution* until the number of clusters suits your question. When satisfied, choose +`community_cluster` in *Cluster column* to lock that partition. + +**Polishing.** After a first run, lower *Settling speed* to 0.008 and run again with *Reset positions* +off. Then run Gephi's **Noverlap** or **Label Adjust** if you need to clean up labels. + +--- + +## 5. Notes and limits + +- **Isolated nodes** (nodes without ties) cannot be assigned to a community by Leiden, so each one becomes + its own one-node cluster and gets its own place in the packing. A graph with many isolated nodes will + therefore show many tiny circles. Remove them with a filter (Degree Range) before running if this is not wanted. +- **Directed graphs** are treated as undirected. Parallel edges and edges in both directions are added + together, and edge weights are used. Self-loops and edges with zero or negative weight are ignored. +- **Leiden quality** is measured with modularity. Like all modularity methods it can miss very small + communities in very large graphs. Raise *Resolution* if that matters. Leiden repeats its passes over + the graph until the result no longer changes, with at most four passes. +- **Speed.** A run takes about 230 steps. On a graph with 20,000 nodes and 80,000 ties a complete run + takes roughly 3 to 8 seconds on a single processor core (less on a multi-core computer), whichever + column you choose. Leiden adds a few seconds for 50,000 nodes and 250,000 ties. Very large graphs + (100,000 nodes and more) take correspondingly longer: raise *Settling speed* for a quick first look. + Repulsion between nodes uses a Barnes-Hut approximation (as in d3-force), so big, crowded clusters + stay fast, and it uses all processor cores. +- **Stopping.** Press Stop in the layout panel at any time: the positions reached so far are kept. +- **Gephi's own Modularity column** (`modularity_class`) is produced by a different implementation (Louvain) + than the Leiden used here, so the two can give somewhat different communities for the same graph. + +--- + +## 6. Troubleshooting + +| Problem | What to try | +|---|---| +| The layout is not in the dropdown | Open a project first; check the plugin is listed in the root `pom.xml` and the build succeeded | +| Everything ends up as one cluster | The chosen column has the same value for every node. Choose another column, or choose *(none: detect communities with Leiden)* | +| The Cluster column list does not show my column | Only whole-number, text and true/false columns are listed. Decimal-number columns are not categories: convert them (for example round to whole numbers or bin them) in the Data Laboratory | +| The layout takes very long | Very large graph: raise *Settling speed* (for example 0.1) for a fast first result, then refine with *Reset positions* off | +| Far too many clusters | Lower *Resolution*; check for isolated nodes (see above) | +| Clusters overlap each other | Raise *Cluster pull*; lower *Inter-cluster tie factor*; if nodes are large, raise *Scale* | +| Nodes overlap inside clusters | Raise *Node spacing* or *Scale*; raise *Repulsion* | +| Clusters look like perfect circles with no structure | Lower *Cluster pull* and raise *Tie strength* so ties shape the inside | +| The layout never seems to stop | Lower values of *Settling speed* run longer; raise it to finish sooner | +| The layout stops immediately | The graph is empty, or the previous run is finished: press Run again | +| A note about a column in the Gephi log | The column stored with a saved project no longer exists, so Leiden was used | + +--- + +## 7. Credits + +The layout is inspired by Fierăscu, S. I. (2026). *SoFab: Network analysis for research and practice* (Version 0.9.5) [Computer software]. Available at . + +--- + +## 8. Version history + +- **1.0.0**: The *Cluster column* setting is now a drop-down list of the categorical node columns (it was a text + field). Much faster on larger graphs with big clusters (repulsion uses a Barnes-Hut approximation and + several processor cores; before, a 20,000-node graph with a few big clusters could take minutes). The graph + is locked for much shorter times during a run. Leiden stops after at most four passes. New, rounder default + values: *Cluster pull* 0.1 (was 0.14), *Settling speed* 0.03 (was 0.0228, about 230 steps instead of 300, same + result), *Random seed* 4711 (was 42). +- **0.1.0**: First release. + +--- + +## 9. For developers + +Source code: . Licensed +under Apache 2.0 (see `LICENSE`). + +```bat +cd /d C:\gephidev\gephi-plugins +mvn clean package +mvn org.gephi:gephi-maven-plugin:run +``` + +| File | Role | +|---|---| +| `CommunityClusters.java` | The Gephi layout: properties, reading the graph, writing positions | +| `CommunityClustersBuilder.java` | Registers the layout with Gephi | +| `ClusterColumnEditor.java`, `ClusterColumns.java` | The drop-down for the *Cluster column* setting and its rules | +| `ClusterForceSimulation.java` | The force simulation (pure Java) | +| `RepulsionTree.java` | Barnes-Hut tree for the repulsion between nodes | +| `CirclePacker.java` | Packs one circle per cluster to give each a home position | +| `Leiden.java` | Community detection (pure Java) | +| `Bundle.properties` | All user-visible text (setting names, descriptions, layout name) and the long description shown in Gephi's plugin manager | +| `src/test/java/...` | JUnit tests | diff --git a/modules/CommunityClusters/pom.xml b/modules/CommunityClusters/pom.xml index 7036f5cf8..b97eaa896 100644 --- a/modules/CommunityClusters/pom.xml +++ b/modules/CommunityClusters/pom.xml @@ -9,7 +9,7 @@ myrberger community-clusters - 0.1.0 + 1.0.0 nbm Community Clusters diff --git a/modules/CommunityClusters/src/img/1-communityclusters.png b/modules/CommunityClusters/src/img/1-communityclusters.png new file mode 100644 index 000000000..a5e421110 Binary files /dev/null and b/modules/CommunityClusters/src/img/1-communityclusters.png differ diff --git a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterColumnEditor.java b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterColumnEditor.java new file mode 100644 index 000000000..629168b81 --- /dev/null +++ b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterColumnEditor.java @@ -0,0 +1,92 @@ +package myrberger.communityclusters; + +import java.beans.PropertyEditorSupport; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.logging.Level; +import java.util.logging.Logger; +import org.gephi.graph.api.Column; +import org.gephi.graph.api.GraphController; +import org.gephi.graph.api.GraphModel; +import org.openide.util.Lookup; +import org.openide.util.NbBundle; + +/** + * Drop-down for the "Cluster column" setting. Lists the categorical node columns of the current + * workspace, plus an entry meaning "no column: detect communities with Leiden". The stored value is the + * column id (an empty string for the Leiden entry). + */ +public final class ClusterColumnEditor extends PropertyEditorSupport { + + private static final Logger LOG = Logger.getLogger(ClusterColumnEditor.class.getName()); + + private static String noneLabel() { + return NbBundle.getMessage(ClusterColumnEditor.class, "CommunityClusters.clusterColumn.none"); + } + + /** Snapshot of the eligible columns right now. Never throws: on any problem the list is just empty. */ + private static void snapshot(List ids, List titles) { + try { + GraphController gc = Lookup.getDefault().lookup(GraphController.class); + GraphModel gm = gc == null ? null : gc.getGraphModel(); + if (gm == null) { + return; + } + for (Column c : gm.getNodeTable().toArray()) { + if (ClusterColumns.isCategorical(c.getTypeClass(), c.isArray(), c.isDynamic(), c.isProperty())) { + ids.add(c.getId()); + titles.add(c.getTitle()); + } + } + } catch (RuntimeException e) { + LOG.log(Level.WARNING, "Could not list node columns", e); + } + } + + @Override + public String[] getTags() { + List ids = new ArrayList<>(), titles = new ArrayList<>(); + snapshot(ids, titles); + Map labels = ClusterColumns.labels(ClusterColumns.toArray(ids), ClusterColumns.toArray(titles)); + List tags = new ArrayList<>(); + tags.add(noneLabel()); + tags.addAll(labels.keySet()); + return tags.toArray(new String[0]); + } + + @Override + public String getAsText() { + Object v = getValue(); + String stored = v == null ? "" : v.toString().trim(); + if (stored.isEmpty()) { + return noneLabel(); + } + List ids = new ArrayList<>(), titles = new ArrayList<>(); + snapshot(ids, titles); + String[] idArr = ClusterColumns.toArray(ids), titleArr = ClusterColumns.toArray(titles); + int at = ClusterColumns.find(stored, idArr, titleArr); + if (at < 0) { + return stored; // a column that is not in the list (any more): show what is stored + } + for (Map.Entry e : ClusterColumns.labels(idArr, titleArr).entrySet()) { + if (e.getValue() == at) { + return e.getKey(); + } + } + return stored; + } + + @Override + public void setAsText(String text) { + if (text == null || text.trim().isEmpty() || text.equals(noneLabel())) { + setValue(""); + return; + } + List ids = new ArrayList<>(), titles = new ArrayList<>(); + snapshot(ids, titles); + String[] idArr = ClusterColumns.toArray(ids), titleArr = ClusterColumns.toArray(titles); + Integer at = ClusterColumns.labels(idArr, titleArr).get(text); + setValue(at != null ? idArr[at] : text.trim()); + } +} diff --git a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterColumns.java b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterColumns.java new file mode 100644 index 000000000..133cc6545 --- /dev/null +++ b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterColumns.java @@ -0,0 +1,82 @@ +package myrberger.communityclusters; + +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * Decides which node columns can be used as cluster columns, and how they are named in the drop-down. + * Kept free of Gephi classes so the rules can be unit tested. + */ +final class ClusterColumns { + + private ClusterColumns() { + } + + /** + * A column is offered when it holds whole numbers, text, true/false or single characters, one value + * per node. Decimal numbers, lists and time-dependent values are not categories. Built-in columns + * such as Id and Label (different for every node) are left out. + */ + static boolean isCategorical(Class type, boolean isArray, boolean isDynamic, boolean isBuiltIn) { + if (type == null || isArray || isDynamic || isBuiltIn) { + return false; + } + return type == String.class || type == Integer.class || type == Long.class || type == Short.class + || type == Byte.class || type == Boolean.class || type == Character.class + || type == int.class || type == long.class || type == short.class || type == byte.class + || type == boolean.class || type == char.class; + } + + /** + * Display labels for columns, in the order given. The title is used (that is what the Data Laboratory + * shows); if two columns share a title, the id is added to tell them apart. + * + * @return label to index in the input arrays + */ + static Map labels(String[] ids, String[] titles) { + Map titleCount = new LinkedHashMap<>(); + String[] base = new String[ids.length]; + for (int i = 0; i < ids.length; i++) { + String t = titles[i] == null || titles[i].trim().isEmpty() ? ids[i] : titles[i].trim(); + base[i] = t; + titleCount.merge(t.toLowerCase(), 1, Integer::sum); + } + Map out = new LinkedHashMap<>(); + for (int i = 0; i < ids.length; i++) { + String label = titleCount.get(base[i].toLowerCase()) > 1 && !base[i].equalsIgnoreCase(ids[i]) + ? base[i] + " (" + ids[i] + ")" : base[i]; + // last resort: never lose a column to a clash + while (out.containsKey(label)) { + label = label + "'"; + } + out.put(label, i); + } + return out; + } + + /** Index of the column the stored value refers to (matches id or title, ignoring case), or -1. */ + static int find(String value, String[] ids, String[] titles) { + if (value == null || value.trim().isEmpty()) { + return -1; + } + String v = value.trim(); + for (int i = 0; i < ids.length; i++) { + if (v.equalsIgnoreCase(ids[i])) { + return i; + } + } + for (int i = 0; i < titles.length; i++) { + if (titles[i] != null && v.equalsIgnoreCase(titles[i].trim())) { + return i; + } + } + return -1; + } + + /** Convenience for building arrays from a list. */ + static String[] toArray(List l) { + return new ArrayList<>(l).toArray(new String[0]); + } +} diff --git a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterForceSimulation.java b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterForceSimulation.java index e829e0a75..32a741f5f 100644 --- a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterForceSimulation.java +++ b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/ClusterForceSimulation.java @@ -16,12 +16,12 @@ static final class Params { double tieLen = 24; // link distance on top of the two node radii double tieStr = 1; // multiplies the per-tie strength double spacing = 2; // extra room kept by the collision force - double clusterPull = 0.14; // pull towards the cluster's home - double cooling = 0.0228; // alphaDecay + double clusterPull = 0.1; // pull towards the cluster's home + double cooling = 0.03; // alphaDecay (about 230 steps per run) double interTieFactor = 0.03;// strength multiplier for ties between clusters double scale = 1; // multiplies all distances double repelDistanceMax = 140; - long seed = 42; + long seed = 4711; boolean resetPositions = true; } @@ -256,15 +256,88 @@ private void buildGrid(double[] px, double[] py, double minCell) { } } + /** Graphs up to this size use the exact pairwise repulsion; larger ones use Barnes-Hut. */ + private static final int EXACT_LIMIT = 400; + /** Below this size the Barnes-Hut queries run on one thread. */ + private static final int PARALLEL_LIMIT = 1500; + + /** Average number of candidate neighbours per node above which the tree is used. */ + private static final double CROWDED_CANDIDATES = 150; + + private final RepulsionTree tree = new RepulsionTree(); + /** Test hook: force one mode. null = choose by graph size. */ + Boolean forceExactRepulsion = null; + /** Many-body repulsion with a hard distance cut-off (d3's forceManyBody with distanceMax). */ private void applyCharge() { if (n < 2 || p.repel == 0) { return; } + if (forceExactRepulsion != null) { + if (forceExactRepulsion) { + applyChargeExact(); + } else { + applyChargeBarnesHut(); + } + return; + } + if (n <= EXACT_LIMIT) { + applyChargeExact(); + return; + } + // Spread-out nodes: the simple neighbour search is cheapest. Crowded nodes (big clusters): + // the number of neighbours explodes, so switch to the tree. + buildGrid(x, y, p.repelDistanceMax * p.scale); + if (gridWorkPerNode() <= CROWDED_CANDIDATES) { + chargeFromGrid(); + } else { + applyChargeBarnesHut(); + } + } + + void applyChargeBarnesHut() { double dMax = p.repelDistanceMax * p.scale; double dMax2 = dMax * dMax; - double strength = -p.repel * p.scale * p.scale; + double k = -p.repel * p.scale * p.scale * alpha; + tree.build(x, y, n); + if (n >= PARALLEL_LIMIT) { + // every node only writes its own velocity, and the jiggle is deterministic, + // so the result does not depend on how the work is split between threads + java.util.stream.IntStream.range(0, n).parallel().forEach(new java.util.function.IntConsumer() { + private final ThreadLocal stack = ThreadLocal.withInitial(() -> new int[RepulsionTree.stackSize()]); + + @Override + public void accept(int i) { + if (!fixed[i]) { + tree.accumulate(i, k, dMax2, vx, vy, stack.get()); + } + } + }); + } else { + int[] stack = new int[RepulsionTree.stackSize()]; + for (int i = 0; i < n; i++) { + if (!fixed[i]) { + tree.accumulate(i, k, dMax2, vx, vy, stack); + } + } + } + } + + /** Exact pairwise repulsion with a hard distance cut-off. Used for small graphs and as a reference in tests. */ + void applyChargeExact() { + double dMax = p.repelDistanceMax * p.scale; buildGrid(x, y, dMax); + chargeFromGrid(); + } + + /** Pairwise repulsion using the grid that was just built for the repulsion cut-off. */ + private void chargeFromGrid() { + if (n < 2 || p.repel == 0) { + return; + } + double dMax = p.repelDistanceMax * p.scale; + double dMax2 = dMax * dMax; + double strength = -p.repel * p.scale * p.scale; for (int i = 0; i < n; i++) { if (fixed[i]) { continue; @@ -301,6 +374,27 @@ private void applyCharge() { } } + /** Average number of nodes in the 3x3 cells around a node, for the grid that is currently built. */ + private double gridWorkPerNode() { + double work = 0; + for (int cy = 0; cy < gh; cy++) { + for (int cx = 0; cx < gw; cx++) { + int here = cellStart[cy * gw + cx + 1] - cellStart[cy * gw + cx]; + if (here == 0) { + continue; + } + int around = 0; + for (int yy = Math.max(0, cy - 1); yy <= Math.min(gh - 1, cy + 1); yy++) { + for (int xx = Math.max(0, cx - 1); xx <= Math.min(gw - 1, cx + 1); xx++) { + around += cellStart[yy * gw + xx + 1] - cellStart[yy * gw + xx]; + } + } + work += (double) here * around; + } + } + return work / Math.max(1, n); + } + /** d3's forceCollide with strength 1 and one iteration. */ private void applyCollide() { if (n < 2) { diff --git a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/CommunityClusters.java b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/CommunityClusters.java index 95c8e17e3..0202ff523 100644 --- a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/CommunityClusters.java +++ b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/CommunityClusters.java @@ -1,6 +1,7 @@ package myrberger.communityclusters; import java.util.ArrayList; +import java.util.Arrays; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -59,11 +60,11 @@ public void resetPropertiesValues() { tieLength = 24.0; tieStrength = 1.0; nodeSpacing = 2.0; - clusterPull = 0.14; - settlingSpeed = 0.0228; + clusterPull = 0.1; + settlingSpeed = 0.03; interClusterTieFactor = 0.03; scale = 1.0; - randomSeed = 42; + randomSeed = 4711; resolution = 1.0; saveClusters = true; resetPositions = true; @@ -83,100 +84,109 @@ public void initAlgo() { if (graphModel == null) { return; } + // Step 1: copy what we need out of the graph. The lock is held only for this. Graph graph = graphModel.getGraphVisible(); + Node[] ns; + double[] radius; + double[] sx; + double[] sy; + boolean[] fixed; + int[] es; + int[] ed; + double[] ew; + int[] cluster = null; graph.readLock(); try { - nodes = graph.getNodes().toArray(); - int n = nodes.length; + ns = graph.getNodes().toArray(); + int n = ns.length; if (n == 0) { return; } Map index = new HashMap<>(n * 2); - double[] radius = new double[n], sx = new double[n], sy = new double[n]; - boolean[] fixed = new boolean[n]; - double maxSize = 0; + radius = new double[n]; + sx = new double[n]; + sy = new double[n]; + fixed = new boolean[n]; for (int i = 0; i < n; i++) { - Node node = nodes[i]; + Node node = ns[i]; index.put(node, i); radius[i] = Math.max(1.0, node.size()); - maxSize = Math.max(maxSize, radius[i]); sx[i] = node.x(); sy[i] = node.y(); fixed[i] = node.isFixed(); } - - List pairs = new ArrayList<>(); - List weights = new ArrayList<>(); + int m = graph.getEdgeCount(); + es = new int[m]; + ed = new int[m]; + ew = new double[m]; + int c = 0; for (Edge e : graph.getEdges()) { Integer s = index.get(e.getSource()); Integer t = index.get(e.getTarget()); - if (s != null && t != null) { - pairs.add(new int[]{s, t}); - weights.add(e.getWeight()); + if (s != null && t != null && c < m) { + es[c] = s; + ed[c] = t; + ew[c] = e.getWeight(); + c++; } } - int[] es = new int[pairs.size()], ed = new int[pairs.size()]; - for (int i = 0; i < es.length; i++) { - es[i] = pairs.get(i)[0]; - ed[i] = pairs.get(i)[1]; + if (c < m) { + es = Arrays.copyOf(es, c); + ed = Arrays.copyOf(ed, c); + ew = Arrays.copyOf(ew, c); } - - double[] ew = new double[es.length]; - for (int i = 0; i < ew.length; i++) { - ew[i] = weights.get(i); + Column col = findColumn(clusterColumn); + if (col != null) { + cluster = readColumnClusters(ns, col); } - int[] cluster = readClusters(nodes, es, ed, ew); - - ClusterForceSimulation.Params p = new ClusterForceSimulation.Params(); - p.repel = repulsion; - p.tieLen = tieLength; - p.tieStr = tieStrength; - p.spacing = nodeSpacing; - p.clusterPull = clusterPull; - p.cooling = settlingSpeed; - p.interTieFactor = interClusterTieFactor; - p.scale = scale; - p.seed = randomSeed; - p.resetPositions = resetPositions; - lastCluster = cluster; - sim = new ClusterForceSimulation(radius, cluster, fixed, sx, sy, es, ed, p); } finally { graph.readUnlock(); } - if (pendingSave && nodes != null && lastCluster != null) { - saveClusterColumn(nodes, lastCluster); + + // Step 2: the heavy work, without holding any graph lock. + if (cluster == null) { + if (clusterColumn != null && !clusterColumn.trim().isEmpty()) { + LOG.log(Level.INFO, "Node column \"{0}\" not found; detecting clusters with Leiden instead.", + clusterColumn); + } + cluster = Leiden.run(ns.length, es, ed, ew, resolution, randomSeed); + pendingSave = saveClusters; + } + ClusterForceSimulation.Params p = new ClusterForceSimulation.Params(); + p.repel = repulsion; + p.tieLen = tieLength; + p.tieStr = tieStrength; + p.spacing = nodeSpacing; + p.clusterPull = clusterPull; + p.cooling = settlingSpeed; + p.interTieFactor = interClusterTieFactor; + p.scale = scale; + p.seed = randomSeed; + p.resetPositions = resetPositions; + lastCluster = cluster; + nodes = ns; + sim = new ClusterForceSimulation(radius, cluster, fixed, sx, sy, es, ed, p); + + if (pendingSave) { + saveClusterColumn(ns, cluster); } pendingSave = false; } - /** - * Cluster index per node: read from the chosen node column, or, when no column is given or it does not - * exist, detected with Leiden (modularity, resolution and seed from the properties). - */ - private int[] readClusters(Node[] nodes, int[] es, int[] ed, double[] ew) { - int n = nodes.length; - Column col = findColumn(clusterColumn); - if (col != null) { - int[] out = new int[n]; - Map ids = new HashMap<>(); - for (int i = 0; i < n; i++) { - Object v = nodes[i].getAttribute(col); - String key = v == null ? "" : String.valueOf(v); - Integer id = ids.get(key); - if (id == null) { - id = ids.size(); - ids.put(key, id); - } - out[i] = id; + /** One cluster index per node, from the values of the given column (called with the graph lock held). */ + private int[] readColumnClusters(Node[] ns, Column col) { + int[] out = new int[ns.length]; + Map ids = new HashMap<>(); + for (int i = 0; i < ns.length; i++) { + Object v = ns[i].getAttribute(col); + String key = v == null ? "" : String.valueOf(v); + Integer id = ids.get(key); + if (id == null) { + id = ids.size(); + ids.put(key, id); } - return out; + out[i] = id; } - if (clusterColumn != null && !clusterColumn.trim().isEmpty()) { - LOG.log(Level.WARNING, "Node column \"{0}\" not found; detecting clusters with Leiden instead.", - clusterColumn); - } - int[] out = Leiden.run(n, es, ed, ew, resolution, randomSeed); - pendingSave = saveClusters; return out; } @@ -218,17 +228,19 @@ private Column findColumn(String name) { @Override public void goAlgo() { - if (sim == null || nodes == null || graphModel == null) { + ClusterForceSimulation current = sim; + Node[] current_nodes = nodes; + if (current == null || current_nodes == null || graphModel == null) { return; } + current.tick(); // pure computation on our own arrays, so no graph lock is needed here Graph graph = graphModel.getGraphVisible(); graph.readLock(); try { - sim.tick(); - for (int i = 0; i < nodes.length; i++) { - if (!nodes[i].isFixed()) { - nodes[i].setX((float) sim.x[i]); - nodes[i].setY((float) sim.y[i]); + for (int i = 0; i < current_nodes.length; i++) { + if (!current_nodes[i].isFixed()) { + current_nodes[i].setX((float) current.x[i]); + current_nodes[i].setY((float) current.y[i]); } } } finally { @@ -254,7 +266,10 @@ public LayoutProperty[] getProperties() { String clusters = msg("CommunityClusters.category.clusters"); String forces = msg("CommunityClusters.category.forces"); String run = msg("CommunityClusters.category.run"); - props.add(property(String.class, clusters, "clusterColumn", "getClusterColumn", "setClusterColumn")); + props.add(LayoutProperty.createProperty(this, String.class, + msg("CommunityClusters.clusterColumn.name"), clusters, "clusterColumn", + msg("CommunityClusters.clusterColumn.desc"), "getClusterColumn", "setClusterColumn", + ClusterColumnEditor.class)); props.add(property(Double.class, clusters, "resolution", "getResolution", "setResolution")); props.add(property(Boolean.class, clusters, "saveClusters", "getSaveClusters", "setSaveClusters")); props.add(property(Double.class, clusters, "clusterPull", "getClusterPull", "setClusterPull")); diff --git a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/Leiden.java b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/Leiden.java index 1576c6da6..0e02d05f2 100644 --- a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/Leiden.java +++ b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/Leiden.java @@ -15,6 +15,11 @@ final class Leiden { private static final double THETA = 0.01; // randomness of the refinement step + /** + * Passes over the whole graph. Graphs with real community structure settle after 2-3; graphs + * without it would keep going for tiny gains, so the loop is capped. + */ + private static final int MAX_ROUNDS = 4; private static final double EPS = 1e-12; private Leiden() { @@ -72,7 +77,7 @@ static int[] run(int n, int[] es, int[] ed, double[] ew, double resolution, long part[i] = i; } // Leiden is iterated: each pass starts from the previous partition, until nothing changes - for (int round = 0; round < 10; round++) { + for (int round = 0; round < MAX_ROUNDS; round++) { int[] next = pass(g0, part, resolution, twoM, rng); renumber(next, new int[n]); boolean same = Arrays.equals(part, next); diff --git a/modules/CommunityClusters/src/main/java/myrberger/communityclusters/RepulsionTree.java b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/RepulsionTree.java new file mode 100644 index 000000000..084ec045d --- /dev/null +++ b/modules/CommunityClusters/src/main/java/myrberger/communityclusters/RepulsionTree.java @@ -0,0 +1,221 @@ +package myrberger.communityclusters; + +import java.util.Arrays; + +/** + * Barnes-Hut quadtree for the many-body repulsion, the same approximation d3-force uses: groups of + * nodes that are far enough away are treated as one body at their centre of mass. Pure Java. + * + *

Built once per tick, then {@link #accumulate} can be called for different nodes from several + * threads at once (it only reads the tree and writes the velocity of the node it is asked about). + */ +final class RepulsionTree { + + private static final int LEAF_SIZE = 8; + private static final int MAX_DEPTH = 40; + /** d3 default: theta = 0.9, squared. */ + static final double THETA2 = 0.81; + /** A group is only treated as one body if it is narrower than this share of the cut-off distance. */ + static final double MAX_GROUP_WIDTH = 1.0; + + private int[] idx = new int[0]; + private int[] scratch = new int[0]; + + // per tree node + private int nodeCount; + private double[] comX = new double[0], comY = new double[0]; + private double[] boxX = new double[0], boxY = new double[0], boxW = new double[0]; + private int[] cnt = new int[0]; + private int[] kids = new int[0]; // index of the first of 4 consecutive children, or -1 for a leaf + private int[] lo = new int[0], hi = new int[0]; + + private double[] px, py; + + void build(double[] x, double[] y, int n) { + this.px = x; + this.py = y; + if (idx.length < n) { + idx = new int[n]; + scratch = new int[n]; + } + double minX = Double.POSITIVE_INFINITY, minY = Double.POSITIVE_INFINITY; + double maxX = Double.NEGATIVE_INFINITY, maxY = Double.NEGATIVE_INFINITY; + for (int i = 0; i < n; i++) { + idx[i] = i; + minX = Math.min(minX, x[i]); + maxX = Math.max(maxX, x[i]); + minY = Math.min(minY, y[i]); + maxY = Math.max(maxY, y[i]); + } + nodeCount = 0; + if (n == 0) { + return; + } + double w = Math.max(maxX - minX, maxY - minY) * (1 + 1e-9) + 1e-6; + int root = newNodes(1); + fill(root, 0, n, minX, minY, w, 0); + } + + private int newNodes(int k) { + int first = nodeCount; + nodeCount += k; + if (nodeCount > cnt.length) { + int cap = Math.max(nodeCount, cnt.length * 2 + 16); + comX = Arrays.copyOf(comX, cap); + comY = Arrays.copyOf(comY, cap); + boxX = Arrays.copyOf(boxX, cap); + boxY = Arrays.copyOf(boxY, cap); + boxW = Arrays.copyOf(boxW, cap); + cnt = Arrays.copyOf(cnt, cap); + kids = Arrays.copyOf(kids, cap); + lo = Arrays.copyOf(lo, cap); + hi = Arrays.copyOf(hi, cap); + } + return first; + } + + private void fill(int node, int from, int to, double x0, double y0, double w, int depth) { + boxX[node] = x0; + boxY[node] = y0; + boxW[node] = w; + lo[node] = from; + hi[node] = to; + cnt[node] = to - from; + double sx = 0, sy = 0; + for (int p = from; p < to; p++) { + sx += px[idx[p]]; + sy += py[idx[p]]; + } + if (to > from) { + comX[node] = sx / (to - from); + comY[node] = sy / (to - from); + } + if (to - from <= LEAF_SIZE || depth >= MAX_DEPTH) { + kids[node] = -1; + return; + } + double mx = x0 + w / 2, my = y0 + w / 2; + int[] qc = new int[5]; + for (int p = from; p < to; p++) { + qc[1 + quadrant(idx[p], mx, my)]++; + } + for (int q = 0; q < 4; q++) { + qc[q + 1] += qc[q]; + } + int[] at = {qc[0], qc[1], qc[2], qc[3]}; + for (int p = from; p < to; p++) { + int q = quadrant(idx[p], mx, my); + scratch[from + at[q]++] = idx[p]; + } + System.arraycopy(scratch, from, idx, from, to - from); + int first = newNodes(4); + kids[node] = first; + for (int q = 0; q < 4; q++) { + fill(first + q, from + qc[q], from + qc[q + 1], x0 + ((q & 1) != 0 ? w / 2 : 0), + y0 + ((q & 2) != 0 ? w / 2 : 0), w / 2, depth + 1); + } + } + + private int quadrant(int i, double mx, double my) { + return (px[i] >= mx ? 1 : 0) | (py[i] >= my ? 2 : 0); + } + + /** Deterministic stand-in for d3's random jiggle, safe to call from several threads. */ + static double jiggle(int i, int j) { + int h = i * 0x9E3779B1 ^ j * 0x85EBCA6B; + h ^= h >>> 15; + h *= 0x2C1B3C6D; + h ^= h >>> 12; + return ((h & 0xFFFF) / 65536.0 - 0.5) * 1e-6; + } + + /** + * Adds the repulsion on node i to vx[i], vy[i]. + * + * @param k strength * alpha (negative to push apart) + * @param dMax2 squared cut-off distance + */ + void accumulate(int i, double k, double dMax2, double[] vx, double[] vy, int[] stack) { + if (nodeCount == 0) { + return; + } + double xi = px[i], yi = py[i]; + double ax = 0, ay = 0; + int sp = 0; + stack[sp++] = 0; + while (sp > 0) { + int node = stack[--sp]; + if (cnt[node] == 0) { + continue; + } + // skip boxes that lie completely beyond the cut-off + double bx = Math.max(Math.max(boxX[node] - xi, xi - (boxX[node] + boxW[node])), 0); + double by = Math.max(Math.max(boxY[node] - yi, yi - (boxY[node] + boxW[node])), 0); + if (bx * bx + by * by >= dMax2) { + continue; + } + double dx = comX[node] - xi, dy = comY[node] - yi; + double l = dx * dx + dy * dy; + double w = boxW[node]; + // far enough away (d3's Barnes-Hut rule), and not wider than the cut-off distance + if (w * w / THETA2 < l && w * w <= MAX_GROUP_WIDTH * MAX_GROUP_WIDTH * dMax2) { + // far enough: treat the whole group as one body + if (l < dMax2) { + if (dx == 0) { + dx = jiggle(i, node); + l += dx * dx; + } + if (dy == 0) { + dy = jiggle(node, i); + l += dy * dy; + } + if (l < 1) { + l = Math.sqrt(l); + } + double f = k * cnt[node] / l; + ax += dx * f; + ay += dy * f; + } + continue; + } + if (kids[node] < 0) { + for (int p = lo[node]; p < hi[node]; p++) { + int j = idx[p]; + if (j == i) { + continue; + } + double ddx = px[j] - xi, ddy = py[j] - yi; + if (ddx == 0) { + ddx = jiggle(i, j); + } + if (ddy == 0) { + ddy = jiggle(j, i); + } + double ll = ddx * ddx + ddy * ddy; + if (ll >= dMax2) { + continue; + } + if (ll < 1) { + ll = Math.sqrt(ll); + } + double f = k / ll; + ax += ddx * f; + ay += ddy * f; + } + } else { + int first = kids[node]; + stack[sp++] = first; + stack[sp++] = first + 1; + stack[sp++] = first + 2; + stack[sp++] = first + 3; + } + } + vx[i] += ax; + vy[i] += ay; + } + + /** Stack depth that is always enough: 3 siblings per level plus one. */ + static int stackSize() { + return 3 * MAX_DEPTH + 8; + } +} diff --git a/modules/CommunityClusters/src/main/nbm/manifest.mf b/modules/CommunityClusters/src/main/nbm/manifest.mf index 21a2f3e35..e12f74415 100644 --- a/modules/CommunityClusters/src/main/nbm/manifest.mf +++ b/modules/CommunityClusters/src/main/nbm/manifest.mf @@ -1,5 +1,5 @@ Manifest-Version: 1.0 OpenIDE-Module-Name: Community Clusters OpenIDE-Module-Short-Description: Layout that gathers each community of a network into its own cluster. -OpenIDE-Module-Long-Description: A force layout that gives every community a home position and pulls its nodes towards it, while ties and repulsion shape the inside of each cluster. Communities are read from a node column (default modularity_class) or, if no such column exists, detected with the built-in Leiden algorithm. OpenIDE-Module-Display-Category: Layout +OpenIDE-Module-Localizing-Bundle: myrberger/communityclusters/Bundle.properties diff --git a/modules/CommunityClusters/src/main/resources/myrberger/communityclusters/Bundle.properties b/modules/CommunityClusters/src/main/resources/myrberger/communityclusters/Bundle.properties index dc3479ae9..f7963d720 100644 --- a/modules/CommunityClusters/src/main/resources/myrberger/communityclusters/Bundle.properties +++ b/modules/CommunityClusters/src/main/resources/myrberger/communityclusters/Bundle.properties @@ -1,6 +1,15 @@ +# Shown in Gephi's plugin manager and on the plugin's page (pointed to by OpenIDE-Module-Localizing-Bundle in manifest.mf) +OpenIDE-Module-Long-Description=Arranges a network so that each community forms its own visible cluster. Any field that \ + is categorical, numerical or alphanumerical, can be used as the Cluster column (for \ + example modularity_class from Statistics > Modularity). Or choose none and the plugin \ + detects communities itself with the Leiden algorithm. Clusters are packed apart from \ + each other and the nodes inside a cluster are placed by their ties. Settings control \ + cluster separation, repulsion, tie length, node spacing and more. By Johan Myrberger, \ + https://www.linkedin.com/in/myrberger/ + # Layout and builder CommunityClustersBuilder.name=Community Clusters -CommunityClustersBuilder.description=Force layout that gathers each community into its own cluster. Uses a community column if you name one, otherwise detects communities with Leiden. +CommunityClustersBuilder.description=Force layout that gathers each community into its own cluster. Uses any categorical node column as the community, or detects communities with Leiden. # Property categories CommunityClusters.category.clusters=Clusters @@ -9,7 +18,8 @@ CommunityClusters.category.run=Run # Clusters CommunityClusters.clusterColumn.name=Cluster column -CommunityClusters.clusterColumn.desc=Node column holding the community of each node (e.g. modularity_class from Statistics > Modularity). Leave empty, or name a column that does not exist, to detect communities with Leiden instead. +CommunityClusters.clusterColumn.desc=Node column that says which community each node belongs to. The list shows columns holding whole numbers, text or true/false values (for example modularity_class from Statistics > Modularity). Choose "(none: detect communities with Leiden)" to let the plugin find the communities itself. +CommunityClusters.clusterColumn.none=(none: detect communities with Leiden) CommunityClusters.resolution.name=Resolution CommunityClusters.resolution.desc=Leiden only: higher gives more and smaller clusters, lower gives fewer and larger ones. CommunityClusters.saveClusters.name=Save clusters to column diff --git a/modules/CommunityClusters/src/test/java/myrberger/communityclusters/ClusterColumnsTest.java b/modules/CommunityClusters/src/test/java/myrberger/communityclusters/ClusterColumnsTest.java new file mode 100644 index 000000000..eb3866b8d --- /dev/null +++ b/modules/CommunityClusters/src/test/java/myrberger/communityclusters/ClusterColumnsTest.java @@ -0,0 +1,80 @@ +package myrberger.communityclusters; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.Map; +import org.junit.Test; + +public class ClusterColumnsTest { + + @Test + public void categoricalTypesAreAccepted() { + assertTrue("String", ClusterColumns.isCategorical(String.class, false, false, false)); + assertTrue("Integer", ClusterColumns.isCategorical(Integer.class, false, false, false)); + assertTrue("Long", ClusterColumns.isCategorical(Long.class, false, false, false)); + assertTrue("Short", ClusterColumns.isCategorical(Short.class, false, false, false)); + assertTrue("Byte", ClusterColumns.isCategorical(Byte.class, false, false, false)); + assertTrue("Boolean", ClusterColumns.isCategorical(Boolean.class, false, false, false)); + assertTrue("Character", ClusterColumns.isCategorical(Character.class, false, false, false)); + assertTrue("primitive int", ClusterColumns.isCategorical(int.class, false, false, false)); + } + + @Test + public void nonCategoricalColumnsAreRejected() { + assertFalse("Double", ClusterColumns.isCategorical(Double.class, false, false, false)); + assertFalse("Float", ClusterColumns.isCategorical(Float.class, false, false, false)); + assertFalse("list of text", ClusterColumns.isCategorical(String[].class, true, false, false)); + assertFalse("time dependent", ClusterColumns.isCategorical(Integer.class, false, true, false)); + assertFalse("built-in column such as Id or Label", ClusterColumns.isCategorical(String.class, false, false, true)); + assertFalse("unknown type", ClusterColumns.isCategorical(null, false, false, false)); + } + + @Test + public void labelsUseTitlesAndKeepTheOrder() { + Map labels = ClusterColumns.labels(new String[]{"modularity_class", "type"}, + new String[]{"Modularity Class", "type"}); + assertArrayEquals(new String[]{"Modularity Class", "type"}, labels.keySet().toArray(new String[0])); + assertEquals("first column index", 0, labels.get("Modularity Class").intValue()); + assertEquals("second column index", 1, labels.get("type").intValue()); + } + + @Test + public void blankTitleFallsBackToId() { + Map labels = ClusterColumns.labels(new String[]{"a", "b"}, new String[]{"", null}); + assertArrayEquals(new String[]{"a", "b"}, labels.keySet().toArray(new String[0])); + } + + @Test + public void clashingTitlesGetTheirIdAdded() { + Map labels = ClusterColumns.labels(new String[]{"x1", "x2", "other"}, + new String[]{"Group", "group", "Other"}); + assertEquals("all three columns are listed", 3, labels.size()); + assertTrue("first clash is told apart", labels.containsKey("Group (x1)")); + assertTrue("second clash is told apart", labels.containsKey("group (x2)")); + assertTrue("unique title stays plain", labels.containsKey("Other")); + } + + @Test + public void identicalLabelsAreNeverLost() { + Map labels = ClusterColumns.labels(new String[]{"a", "a"}, new String[]{"T", "T"}); + assertEquals("both columns listed", 2, labels.size()); + } + + @Test + public void findMatchesIdOrTitleIgnoringCase() { + String[] ids = {"modularity_class", "type"}; + String[] titles = {"Modularity Class", "Type of node"}; + assertEquals("by id", 0, ClusterColumns.find("MODULARITY_CLASS", ids, titles)); + assertEquals("by title", 1, ClusterColumns.find("type of node", ids, titles)); + assertEquals("with spaces around", 1, ClusterColumns.find(" type ", ids, titles)); + assertEquals("unknown", -1, ClusterColumns.find("nothing", ids, titles)); + assertEquals("empty", -1, ClusterColumns.find("", ids, titles)); + assertEquals("null", -1, ClusterColumns.find(null, ids, titles)); + } + + private static void assertArrayEquals(Object[] expected, Object[] actual) { + org.junit.Assert.assertArrayEquals("labels", expected, actual); + } +} diff --git a/modules/CommunityClusters/src/test/java/myrberger/communityclusters/RepulsionTest.java b/modules/CommunityClusters/src/test/java/myrberger/communityclusters/RepulsionTest.java new file mode 100644 index 000000000..106daaaed --- /dev/null +++ b/modules/CommunityClusters/src/test/java/myrberger/communityclusters/RepulsionTest.java @@ -0,0 +1,187 @@ +package myrberger.communityclusters; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.util.Arrays; +import java.util.Random; +import org.junit.Test; + +/** The tree-based repulsion must give the same kind of layout as the exact calculation. */ +public class RepulsionTest { + + private static ClusterForceSimulation simulation(int n, int clusters, long seed) { + Random r = new Random(seed); + int m = n * 4; + int[] src = new int[m], dst = new int[m], cluster = new int[n]; + double[] radius = new double[n]; + Arrays.fill(radius, 5); + for (int i = 0; i < n; i++) { + cluster[i] = i % clusters; + } + for (int i = 0; i < m; i++) { + int u = r.nextInt(n); + int v = r.nextDouble() < 0.8 ? (u / clusters * clusters + r.nextInt(Math.max(1, n / clusters)) * clusters + + u % clusters) % n : r.nextInt(n); + src[i] = u; + dst[i] = v; + } + return new ClusterForceSimulation(radius, cluster, new boolean[n], new double[n], new double[n], src, dst, + new ClusterForceSimulation.Params()); + } + + private static void runToEnd(ClusterForceSimulation sim) { + int ticks = 0; + while (!sim.isConverged() && ticks < 2000) { + sim.tick(); + ticks++; + } + } + + /** Mean distance from a node to the centre of its cluster. */ + private static double spread(ClusterForceSimulation sim, int n, int clusters) { + double[] cx = new double[clusters], cy = new double[clusters]; + int[] count = new int[clusters]; + for (int i = 0; i < n; i++) { + cx[i % clusters] += sim.x[i]; + cy[i % clusters] += sim.y[i]; + count[i % clusters]++; + } + for (int c = 0; c < clusters; c++) { + cx[c] /= count[c]; + cy[c] /= count[c]; + } + double sum = 0; + for (int i = 0; i < n; i++) { + sum += Math.hypot(sim.x[i] - cx[i % clusters], sim.y[i] - cy[i % clusters]); + } + return sum / n; + } + + /** Share of sampled nodes whose nearest neighbour is closer than the two radii (nodes overlapping). */ + private static double overlapShare(ClusterForceSimulation sim, int n) { + int sample = Math.min(n, 300), overlapping = 0; + for (int i = 0; i < sample; i++) { + double nearest = Double.MAX_VALUE; + for (int j = 0; j < n; j++) { + if (j != i) { + nearest = Math.min(nearest, Math.hypot(sim.x[i] - sim.x[j], sim.y[i] - sim.y[j])); + } + } + if (nearest < 10 - 1e-6) { + overlapping++; + } + } + return (double) overlapping / sample; + } + + @Test + public void treeAndExactGiveTheSameKindOfLayout() { + int n = 1600, clusters = 3; + ClusterForceSimulation exact = simulation(n, clusters, 5); + exact.forceExactRepulsion = true; + runToEnd(exact); + ClusterForceSimulation tree = simulation(n, clusters, 5); + tree.forceExactRepulsion = false; + runToEnd(tree); + double a = spread(exact, n, clusters), b = spread(tree, n, clusters); + assertTrue("cluster size with the tree (" + b + ") is within 10% of exact (" + a + ")", + Math.abs(b / a - 1) < 0.10); + assertTrue("no overlapping nodes with the tree", overlapShare(tree, n) < 0.01); + } + + @Test + public void crowdedClustersStillComeOutFiniteAndSeparated() { + int n = 6000, clusters = 2; + ClusterForceSimulation sim = simulation(n, clusters, 9); // crowded, so the tree is chosen automatically + for (int t = 0; t < 120; t++) { + sim.tick(); + } + for (int i = 0; i < n; i++) { + assertFalse("finite coordinate", Double.isNaN(sim.x[i]) || Double.isInfinite(sim.y[i])); + } + double[] cx = new double[2], cy = new double[2]; + for (int i = 0; i < n; i++) { + cx[i % 2] += sim.x[i] / (n / 2); + cy[i % 2] += sim.y[i] / (n / 2); + } + assertTrue("the two cluster centres are apart", Math.hypot(cx[0] - cx[1], cy[0] - cy[1]) > 100); + } + + @Test + public void runsWithSeveralThreadsAreRepeatable() { + ClusterForceSimulation a = simulation(2500, 4, 7), b = simulation(2500, 4, 7); + a.forceExactRepulsion = false; + b.forceExactRepulsion = false; + for (int t = 0; t < 40; t++) { + a.tick(); + b.tick(); + } + for (int i = 0; i < 2500; i++) { + assertEquals("x of node " + i, a.x[i], b.x[i], 0.0); + assertEquals("y of node " + i, a.y[i], b.y[i], 0.0); + } + } + + @Test + public void closeNodesArePushedApart() { + // 500 nodes on top of each other in a tiny area, no ties: repulsion alone has to spread them out + int n = 500; + double[] radius = new double[n]; + Arrays.fill(radius, 1); + ClusterForceSimulation.Params p = new ClusterForceSimulation.Params(); + p.clusterPull = 0; + p.resetPositions = false; + double[] sx = new double[n], sy = new double[n]; + Random r = new Random(1); + for (int i = 0; i < n; i++) { + sx[i] = r.nextDouble(); + sy[i] = r.nextDouble(); + } + ClusterForceSimulation sim = new ClusterForceSimulation(radius, new int[n], new boolean[n], sx, sy, + new int[0], new int[0], p); + sim.forceExactRepulsion = false; + for (int t = 0; t < 30; t++) { + sim.tick(); + } + double maxDist = 0; + for (int i = 0; i < n; i++) { + maxDist = Math.max(maxDist, Math.hypot(sim.x[i], sim.y[i])); + } + assertTrue("nodes moved well away from the starting square (max distance " + maxDist + ")", maxDist > 20); + } + + @Test + public void oneStepOfTheTreeAgreesWithTheExactForce() { + // spread-out nodes, so near neighbours dominate the force and the two methods can be compared directly + int n = 800; + double[] radius = new double[n]; + Arrays.fill(radius, 1); + double[] sx = new double[n], sy = new double[n]; + Random r = new Random(3); + for (int i = 0; i < n; i++) { + sx[i] = r.nextDouble() * 600; + sy[i] = r.nextDouble() * 600; + } + ClusterForceSimulation.Params p = new ClusterForceSimulation.Params(); + p.resetPositions = false; + ClusterForceSimulation sim = new ClusterForceSimulation(radius, new int[n], new boolean[n], sx, sy, + new int[0], new int[0], p); + Arrays.fill(sim.vx, 0); + Arrays.fill(sim.vy, 0); + sim.applyChargeExact(); + double[] ex = sim.vx.clone(), ey = sim.vy.clone(); + Arrays.fill(sim.vx, 0); + Arrays.fill(sim.vy, 0); + sim.applyChargeBarnesHut(); + double error = 0, size = 0; + for (int i = 0; i < n; i++) { + error += Math.hypot(sim.vx[i] - ex[i], sim.vy[i] - ey[i]); + size += Math.hypot(ex[i], ey[i]); + } + assertTrue("force from the tree is not zero", size > 0); + assertTrue("force from the tree differs from the exact force by " + Math.round(100 * error / size) + + "% (allowed: 25%)", error / size < 0.25); + } +}