Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
d9b93c3
Refactor lint workflow to use reviewdog action
Amanda-dong Sep 9, 2026
c35ff4b
Fix formatting in introduction documentation
Amanda-dong Sep 9, 2026
ef89481
Fix formatting of applications list in embeddings guide
Amanda-dong Sep 9, 2026
ea48180
Update 03_retrieval_augmented_generation.mdx
Amanda-dong Sep 9, 2026
4a1ca9a
Fix formatting in account renewal instructions
Amanda-dong Sep 9, 2026
c4483f3
Update 08_sharing_data_on_hpc.mdx
Amanda-dong Sep 9, 2026
b2451ec
Fix formatting in R packages with renv documentation
Amanda-dong Sep 9, 2026
4576bf7
Fix formatting of VSCode Remote - SSH setup steps
Amanda-dong Sep 9, 2026
04ce20f
Fix formatting of numbered list in MATLAB proxy guide
Amanda-dong Sep 9, 2026
758eb2d
Update 03_moving_looking.mdx
Amanda-dong Sep 10, 2026
eb59002
Update 04_writing_reading_files.mdx
Amanda-dong Sep 10, 2026
361d79f
Fix numbering in scheduler fundamentals tutorial
Amanda-dong Sep 10, 2026
4dde413
Fix formatting and improve clarity in modules tutorial
Amanda-dong Sep 10, 2026
1c051fb
Update 07_transferring_files_remote.mdx
Amanda-dong Sep 10, 2026
e275b1a
Fix bullet point formatting in tutorial
Amanda-dong Sep 10, 2026
b6bcbac
Update 08_sharing_data_on_hpc.mdx
Amanda-dong Sep 11, 2026
ffc3c1f
Update 04_scheduler_fundamentals.mdx
Amanda-dong Sep 11, 2026
e29d92d
Update 08_sharing_data_on_hpc.mdx
Amanda-dong Sep 11, 2026
4e8302b
Fix formatting of examples in sharing data documentation
Amanda-dong Sep 11, 2026
590ace5
Update 08_sharing_data_on_hpc.mdx
Amanda-dong Sep 11, 2026
523c1a2
Update 03_moving_looking.mdx
Amanda-dong Sep 11, 2026
2d68ae2
Fix formatting for solution link in tutorial
Amanda-dong Sep 11, 2026
738ba97
Fix formatting of solution prompts in documentation
Amanda-dong Sep 11, 2026
8bb5c43
Update 03_moving_looking.mdx
Amanda-dong Sep 11, 2026
4e1b2f4
Update 05_wildcards_pipes.mdx
Amanda-dong Sep 12, 2026
c1c6852
Fix solution link formatting in tutorial
Amanda-dong Sep 12, 2026
5f514cc
Fix escape characters for solution links in tutorial
Amanda-dong Sep 12, 2026
8eeda1d
Update 04_scheduler_fundamentals.mdx
Amanda-dong Sep 12, 2026
e22706c
Update 05_environment_variables.mdx
Amanda-dong Sep 12, 2026
fea49c7
Fix markdown syntax for clickable solution link
Amanda-dong Sep 12, 2026
f059b70
Fix markdown for clickable solution links
Amanda-dong Sep 12, 2026
8fc5ba4
Update 08_running_parallel_job.mdx
Amanda-dong Sep 12, 2026
dd28d7c
Fix markdown formatting for solution link
Amanda-dong Sep 12, 2026
08b010b
Fix markdown formatting for solution link
Amanda-dong Sep 12, 2026
d0456f1
Fix formatting for authors and tags in blog post
Amanda-dong Sep 12, 2026
6ebf9d2
Fix authors and tags formatting in blog post
Amanda-dong Sep 12, 2026
12a3b26
Update 2025-05-02-blog-post.mdx
Amanda-dong Sep 12, 2026
fb12ca0
Update 2025-09-10-blog-post.mdx
Amanda-dong Sep 12, 2026
ed65a85
Update remark_args to ignore blog pattern
Amanda-dong Sep 12, 2026
0b756f3
Change linting fail level from error to warning
Amanda-dong Sep 12, 2026
451699e
lint blog posts as well
s-sajid-ali Sep 14, 2026
d0e9732
remove redundant pnpm install step
s-sajid-ali Sep 14, 2026
2330567
blogs entries have yaml formatted entries at the top, add remark-fron…
s-sajid-ali Sep 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 22 additions & 49 deletions .github/workflows/lint.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,56 +16,29 @@ jobs:
remark-lint:
name: runner / remark-lint
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v7
with:
fetch-depth: 0

- name: Install pnpm
uses: pnpm/action-setup@v6
# npm (not pnpm) is intentional: remark-cli is installed globally by the
# action at /usr/local/lib/node_modules/. pnpm's global store is not on
# that resolution path, so remark-cli would fall back to resolving
# remark-mdx from the project's node_modules, where @mdx-js/react pulls
# in mdast-util-mdx-jsx@3 — incompatible with remark-cli@10's bundled
# unified v10 stack. npm -g puts the remark plugins next to remark-cli so
# the versions stay compatible.
#
# install_deps: false prevents the action from running npm install and
# reintroducing the incompatible project dependencies.
- name: Install remark plugins globally
run: npm install -g remark-mdx@2 remark-lint-no-undefined-references@4 remark-directive@3 remark-frontmatter@4

- name: remark-lint
uses: reviewdog/action-remark-lint@v5
with:
# Pinned below 11.12.0, whose self-install crashes with
# "Cannot use 'in' operator to search for 'integrity' in undefined".
# https://github.com/pnpm/action-setup/issues/276
# https://github.com/pnpm/pnpm/issues/12959
version: 11.11.0

- name: Install remark plugins and reviewdog
run: |
npm install -g remark-cli@10 remark-lint@10 remark-preset-lint-recommended remark-mdx@2 remark-lint-no-undefined-references@4 remark-directive@3
curl -sfL https://raw.githubusercontent.com/reviewdog/reviewdog/master/install.sh | sh -s -- -b /tmp
# reviewdog/action-remark-lint runs remark against a directory, but this

# repository currently has existing repo-wide remark warnings. Because this
# workflow fails on warnings, running remark on the whole repository would
# fail PRs for warnings unrelated to the modified files.
# Until the existing repo-wide warnings are cleaned up, lint only Markdown/MDX
# files changed in the current PR and pipe the result to reviewdog manually.
# See the follow-up cleanup issue before reverting this back to
# reviewdog/action-remark-lint.

- name: Find changed Markdown files
id: changed_files
run: |
git fetch origin ${{ github.base_ref }}
git diff --name-only --diff-filter=ACMRT origin/${{ github.base_ref }}...HEAD -- '*.md' '*.mdx' > changed-markdown-files.txt

if [ -s changed-markdown-files.txt ]; then
echo "has_changed_files=true" >> "$GITHUB_OUTPUT"
cat changed-markdown-files.txt
else
echo "has_changed_files=false" >> "$GITHUB_OUTPUT"
echo "No changed Markdown files."
fi

- name: remark-lint changed files
if: steps.changed_files.outputs.has_changed_files == 'true'
run: |
remark $(cat changed-markdown-files.txt) --use=remark-preset-lint-recommended --use=remark-mdx --use=remark-directive --ext .md,.mdx 2>&1 \
| sed 's/\x1b\[[0-9;]*m//g' \
| /tmp/reviewdog -f=remark-lint \
-name="remark-lint" \
-reporter="github-pr-review" \
-filter-mode="nofilter" \
-fail-level="warning" \
-level="warning"
github_token: ${{ secrets.GITHUB_TOKEN }}
reporter: github-pr-review
fail_level: "warning"
level: "warning"
install_deps: false
remark_args: "--use=remark-preset-lint-recommended --use=remark-mdx --use=remark-directive --use=remark-frontmatter --ext .md,.mdx"
16 changes: 0 additions & 16 deletions .memsearch/.index-state.json

This file was deleted.

4 changes: 2 additions & 2 deletions docs/genai/01_getting_started/01_intro.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ If you looking to access LLMs for non-research workflows, please reach out to th
:::

Welcome to Pythia, the generative AI platform for research workflows. As part of the Pythia platform, the following capabilities are offered:
- [Access to externally hosted LLMs](../01_getting_started/02_llm_gateway.mdx)
- [HPC resources for fine tuning LLMs](../../hpc/08_ml_ai_hpc/05_llm_fine_tuning.md)
- [Access to externally hosted LLMs](../01_getting_started/02_llm_gateway.mdx)
- [HPC resources for fine tuning LLMs](../../hpc/08_ml_ai_hpc/05_llm_fine_tuning.md)

:::info[Personal use]
If you want to access NYU provided LLMs for personal use, proceed to https://gemini.google.com/app with your NYU credentials.
Expand Down
6 changes: 3 additions & 3 deletions docs/genai/04_how_to_guides/02_embeddings.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,9 @@ and gives the following response:
## Applications of Embeddings

Embeddings are typically used for:
- retrieval-augmented generation
- search
- classification
- retrieval-augmented generation
- search
- classification

:::info
Embeddings are typically stored in a *vector* database which is designed for efficient storage and fast retrieval of vectors.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ For an in-depth overview of RAG and Jupyter notebook examples, please access the
:::

Large Language Models only know about the data they were trained upon and do not have the context needed to be effective at answering questions based on:
- private datasets
- newer knowledge past the cutoff date (i.e. the date at which data collection was frozen)
- private datasets
- newer knowledge past the cutoff date (i.e. the date at which data collection was frozen)

To get around this issue, one of the most popular techniques is Retrieval-augmented generation, the most basic version of which is outlined below:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ Linux clients are not officially supported, however we were able to successfully
apt-get install openconnect
sudo openconnect -b vpn.nyu.edu
```
- When prompted follow the instructions and provide your netID, password, and authenticate with ('push', 'phone1' or 'sms')
- When prompted follow the instructions and provide your netID, password, and authenticate with ('push', 'phone1' or 'sms')
This method was tested on few Linux distributions and settings however is not guaranteed to work in future.
</TabItem>
</Tabs>
Expand Down
26 changes: 13 additions & 13 deletions docs/hpc/03_storage/08_sharing_data_on_hpc.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -88,14 +88,14 @@ g-DEPT-LABNAME
Once you have a group assigned, managers of the group can use the `ipa` commands to manage members via the login nodes.

Relevant user commands include:
- `ipa group-add-member`: Add members to a group.
- Example:
- `ipa group-add-member`: Add members to a group.
- Example:
`ipa group-add-member GROUP-NAME --users='NETID'`
- `ipa group-add-member-manager`: Add users that can manage members of this group.
- Example:
- `ipa group-add-member-manager`: Add users that can manage members of this group.
- Example:
`ipa group-add-member-manager GROUP-NAME --users='NETID'`
- `ipa group-find`: Search for groups.
- Example:
- `ipa group-find`: Search for groups.
- Example:
```
ipa group-find test
----------------
Expand All @@ -111,16 +111,16 @@ ipa group-find test
Number of entries returned 2
----------------------------
```
- `ipa group-remove-member`: Remove members from a group.
- Example:
- `ipa group-remove-member`: Remove members from a group.
- Example:
`ipa group-remove-member --users=NETID`
- `ipa group-remove-member-manager`: Remove users that can manage members of this group.
- Example:
- `ipa group-remove-member-manager`: Remove users that can manage members of this group.
- Example:
`ipa group-remove-member-manager --users=NETID`
- `ipa group-show`: Display information about a named group.
- Example:
- `ipa group-show`: Display information about a named group.
- Example:
`ipa group-show GROUP-NAME`
- Example:
- Example:
```
[mdw303@torch-login-1 ~]$ ipa group-show g-hpc-mdw303test
Group name: g-hpc-mdw303test
Expand Down
4 changes: 2 additions & 2 deletions docs/hpc/06_tools_and_software/05_r_packages_with_renv.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -195,8 +195,8 @@ renv::init()

### What to save/publish/commit with Git
In order to have your work reproducible by you and/or others:
- save and/or commit your code in git
- be sure to include `renv.lock` (which lists all packages and versions that you use including the version of R)
- save and/or commit your code in git
- be sure to include `renv.lock` (which lists all packages and versions that you use including the version of R)

### Migrating from Packrat
The `renv` package has replaced the now deprecated `Packrat` package. The `renv::migrate()` function makes it possible to migrate projects from `Packrat` to `renv`. See the ?migrate documentation for more details. In essence, calling `renv::migrate("<project path>")` will be enough to migrate the `Packrat` library and lockfile such that they can then be used by `renv`.
Expand Down
4 changes: 2 additions & 2 deletions docs/hpc/06_tools_and_software/09_vscode_remote_ssh_torch.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ Ensure that you SSH client has been configured [as described here](../02_connect
Install the VSCode Remote - SSH extension.

Configure Remote - SSH settings:
1. Enable `Remote.SSH: Lockfiles In Tmp`
2. Unable `Remote.SSH: Use Flock`
1. Enable `Remote.SSH: Lockfiles In Tmp`
2. Unable `Remote.SSH: Use Flock`

Because Torch requires Microsoft device authentication during SSH login, VSCode may time out before you finish entering the device code. If this happens, search for Remote.SSH: Connect Timeout and set it to 120.

Expand Down
8 changes: 4 additions & 4 deletions docs/hpc/09_ood/03_matlab_proxy.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,10 @@ Please be aware that when you run each version of Matlab for the first time, the
Be careful not to go over your quota for `/home`. You can find your current usage with the `myquota` command. Please see our [Storage documentation](../03_storage/01_intro_and_data_management.mdx) for details about your storage options.
:::
Each time you start it up after that it will set its working directory to the last directory you were in. If you'd like to change this behavior, so that you always start at the same directory:
1. click on `ENVIRONMENT` in your `Home` panel
1. select `Preferences`
1. select `Current Folder`
1. select `initial working folder preferences`
1. click on `ENVIRONMENT` in your `Home` panel
1. select `Preferences`
1. select `Current Folder`
1. select `initial working folder preferences`

This will allow you to set the full path of the initial working folder you would prefer. Be sure to hit the `Apply` button, so your changes are applied.
::::
Expand Down
90 changes: 45 additions & 45 deletions docs/hpc/12_tutorial_intro_shell_hpc/03_moving_looking.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,10 +42,10 @@ $ whoami

More specifically, when we type `whoami` in the shell:

1. finds a program called whoami,
1. runs that program,
1. displays that program’s output, then
1. displays a new prompt to tell us that it’s ready for more commands.
1. finds a program called whoami,
1. runs that program,
1. displays that program’s output, then
1. displays a new prompt to tell us that it’s ready for more commands.

Next, let’s find out where we are by running a command called `pwd` (which stands for “print working directory”). (“Directory” is another word for “folder”). At any moment, our current working directory (where we are) is the directory that the computer assumes we want to run commands in unless we explicitly specify something else. Here, the computer’s response is `/home/<NetID>`, which is `<NetID>`'s home directory. Note that the location of your home directory may differ from system to system.
```bash
Expand All @@ -64,7 +64,7 @@ $ ls
<summary>
:::info[**Differences between remote and local system**]
Open a second terminal window on your local computer and run the `ls` command without logging in remotely. What differences do you see? <br />
**[click to see the solution]**
**\[click to see the solution\]**
:::
</summary>

Expand Down Expand Up @@ -236,7 +236,7 @@ Try 'ls --help' for more information.
:::info[Looking at documentation]
Looking at the man page for `ls` or using `ls --help`, what does the `-h` (`--human-readable`) option do?
:::
**[Click for Solution]**
**\[Click for Solution\]**
</summary>
:::tip[Solution]
When used with the `-l` option, use unit suffixes: Byte, Kilobyte, Megabyte, Gigabyte, Terabyte and Petabyte in order to reduce the number of digits to four or fewer using base 2 for sizes. This option is not defined in IEEE Std 1003.1-2008 (“POSIX.1”).
Expand All @@ -248,28 +248,28 @@ When used with the `-l` option, use unit suffixes: Byte, Kilobyte, Megabyte, Gig
:::info[Absolute vs Relative Paths]
Starting from `/Users/amanda/data/`, which of the following commands could Amanda use to navigate to her home directory, which is `/Users/amanda`?

1. `cd .`
1. `cd /`
1. `cd /home/amanda`
1. `cd ../..`
1. `cd ~`
1. `cd home`
1. `cd ~/data/..`
1. `cd`
1. `cd ..`
1. `cd .`
1. `cd /`
1. `cd /home/amanda`
1. `cd ../..`
1. `cd ~`
1. `cd home`
1. `cd ~/data/..`
1. `cd`
1. `cd ..`
:::
**[Click for Solution]**
**\[Click for Solution\]**
</summary>
:::tip[Solution]
1. No: `.` stands for the current directory.
1. No: `/` stands for the root directory.
1. No: Amanda’s home directory is `/Users/amanda`.
1. No: this goes up two levels, i.e. ends in `/Users`.
1. Yes: `~` stands for the user’s home directory, in this case `/Users/amanda`.
1. No: this would navigate into a directory home in the current directory if it exists.
1. Yes: unnecessarily complicated, but correct.
1. Yes: shortcut to go back to the user’s home directory.
1. Yes: goes up one level.
1. No: `.` stands for the current directory.
1. No: `/` stands for the root directory.
1. No: Amanda’s home directory is `/Users/amanda`.
1. No: this goes up two levels, i.e. ends in `/Users`.
1. Yes: `~` stands for the user’s home directory, in this case `/Users/amanda`.
1. No: this would navigate into a directory home in the current directory if it exists.
1. Yes: unnecessarily complicated, but correct.
1. Yes: shortcut to go back to the user’s home directory.
1. Yes: goes up one level.
:::
</details>

Expand All @@ -278,10 +278,10 @@ Starting from `/Users/amanda/data/`, which of the following commands could Amand
:::info[Relative Path Resolution]
Using the filesystem diagram below, if `pwd` displays `/Users/thing`, what will `ls -F ../backup` display?

1. `../backup: No such file or directory`
1. `2012-12-01 2013-01-08 2013-01-27`
1. `2012-12-01/ 2013-01-08/ 2013-01-27/`
1. `original/ pnas_final/ pnas_sub/`
1. `../backup: No such file or directory`
1. `2012-12-01 2013-01-08 2013-01-27`
1. `2012-12-01/ 2013-01-08/ 2013-01-27/`
1. `original/ pnas_final/ pnas_sub/`

:::
```mermaid
Expand All @@ -305,13 +305,13 @@ block-beta
d --> i
d --> j
```
**[Click for Solution]**
**\[Click for Solution\]**
</summary>
:::tip[Solution]
1. No: there is a directory backup in `/Users`.
1. No: this is the content of `Users/thing/backup`, but with `..` we asked for one level further up.
1. No: see previous explanation.
1. Yes: `../backup/` refers to `/Users/backup/`.
1. No: there is a directory backup in `/Users`.
1. No: this is the content of `Users/thing/backup`, but with `..` we asked for one level further up.
1. No: see previous explanation.
1. Yes: `../backup/` refers to `/Users/backup/`.
:::
</details>

Expand All @@ -322,18 +322,18 @@ Assuming a directory structure as in the above figure, if `pwd` displays `/Users
::::info[Output]
pnas_sub/ pnas_final/ original/
::::
1. `ls pwd`
1. `ls -r -F`
1. `ls -r -F /Users/backup`
1. Either #2 or #3 above, but not #1.
1. `ls pwd`
1. `ls -r -F`
1. `ls -r -F /Users/backup`
1. Either #2 or #3 above, but not #1.

**[Click for Solution]**
**\[Click for Solution\]**
</summary>
:::tip[Solution]
1. No: `pwd` is not the name of a directory.
1. Yes: `ls` without directory argument lists files and directories in the current directory.
1. Yes: uses the absolute path explicitly.
1. Correct: see explanations above.
1. No: `pwd` is not the name of a directory.
1. Yes: `ls` without directory argument lists files and directories in the current directory.
1. Yes: uses the absolute path explicitly.
1. Correct: see explanations above.
:::
</details>

Expand All @@ -343,7 +343,7 @@ pnas_sub/ pnas_final/ original/
What does the command ls do when used with the `-l` and `-h` arguments?

Some of its output is about properties that we do not cover in this lesson (such as file permissions and ownership), but the rest should be useful nevertheless.
<br />**[Click for Solution]**
<br />**\[Click for Solution\]**
:::
</summary>
:::tip[Solution]
Expand All @@ -360,7 +360,7 @@ The command `ls -t` lists things by time of last change, with most recently chan

In what order does `ls -R -t` display things? **Hint:** `ls -l` uses a long listing format to view timestamps.
:::
**[Click for Solution]**
**\[Click for Solution\]**
</summary>
:::tip[Solution]
The directories are listed alphabetical at each level, the files/directories in each directory are sorted by time of last change.
Expand Down
Loading
Loading