I bought a DGX Spark

... and I'm using the expense of the device to hold myself accountable to learning, and I figured a blog would be a great way to record what I've learned and share with others who may be on a similar journey.

A little background about me:

I know relatively little about AI. I've become a moderate user of ChatGPT on a daily basis, and I have been playing around with running local AI models. So, I'm a capable user of AI but I haven't done much peeking under the hood. I'm primarily a Windows 11 user, but I prefer my MacBook Pro over any PC for an excellent device experience. I also use WSL for running Linux development tools frequently and run a home sever hosted on Unraid (I may talk more about that later). My day job is a development director for a large software company and I work in the games industry.

On to the Spark:

Today, Christmas came early and my DGX Spark was delivered two days before it was scheduled. I did the unboxing and plugged it in. Man is this device is tiny, somewhere between the old and new Mac Mini. I bought the NVIDIA model, and it's also incredibly good looking. 


I decided to plug it in to HDMI and a keyboard rather than remote setup. I'll be using the device predominately remotely and I wanted to see what the desktop experience was like. Setup was straight forward. I was impressed that it allowed me to put a Bluetooth mouse in paring mode to use the mouse during setup. Windows doesn't even do that. After a lengthily update and reboot chain, I got logged in to the Ubuntu desktop. At that point, it looks like a vanilla Ubuntu desktop. This is where I decided to move back to my primary machine and continue.

Remote development:

I don't have much experience with remote development. I have a lot of experience building locally and deploying to devices, but not so much building and debugging on a remote machine. That's what I want to do here. I want to author and orchestrate AI workloads on my desktop while running them on the Spark. NVIDIA  points me to their NVIDIA Sync desktop app as the easiest way, so I decided to start there. I went to install the NVIDIA  sync tool and install failed with: 

could not set ssh config file: could not update default config file: open C:\Users\<user>\.ssh\config: The system cannot find the path specified.

It turns out that the installer expects that you were already all set up for using SSH to some other host and doesn't want to create the requisite config files (just append to them). ChatGPT got me unblocked immediately with:

mkdir $env:USERPROFILE\.ssh

New-Item -Path $env:USERPROFILE\.ssh\config -ItemType File -Force

Many ways to do it, but basically, create an empty config file under the user directory's .ssh sub-directory. The first of many minor stumbling blocks. But after that, install completed and I can use NVIDIA sync to launch the DGX dashboard, run a terminal, or run VSCode. It's worth noting that VSCode was included because the NVIDIA sync installer had noticed I had it installed. I install that ahead of time with:

winget install Microsoft.VisualStudioCode 

Anyhow, the NVIDIA Sync app minimizes to the system tray and has a small lower right justfied UI when you activate it. At the top of the window it has a path that is active on. This controls the starting path for the apps where that matters, in my case, terminal and VSCode. For terminal it just sets the starting path, for VSCode it sets the root path of the remote instance. It looks like this:


I was impressed that starting VSCode on the remote path through NVIDIA Sync just worked. It took a bit to install the server the first time, but once it's launched it's just like using VSCode locally. My VSCode was already configured to use github copilot. I get access through work but my experience with it is very positive and I do recommend it. 

I created a test.py with print("Hello from DGX Spark!") to test it out. I'm no expert on configuring VSCode, but fortunately it's pretty easy these days. I just hit F5 and one of the options was to use copilot to configure a launch.json so I went for that. Since I didn't have an obvious main (just a test.py) it set up the workflow to prompt me for which file to run when I hit F5. Thankfully it remembers the last one I used so repeated runs don't require reentry. Just like that I'm running and able to hit breakpoints.

For my next post I'll share my journey to build and train Andrej Karpathy's NanoChat (https://github.com/karpathy/nanochat).

Comments

Post a Comment

Popular posts from this blog

Andrej Karpathy's Nanochat

Image
Right around the time that I decided to buy a DGX Spark and up my game in AI, Andrej Karpathy posted his magnum opus (so far) that he's been working on for quite a while, Nanochat  (https://github.com/karpathy/nanochat). I discovered it through my boss mentioning it me; it hit Hacker News et al earlier that week. The idea is to perform all the steps from scratch (but using production tools like torch) to create an LLM that is similar in capability to GPT-2. I figured this is a great start to diving deeper into LLMs.  The tagline for Nanochat is "The best ChatGPT that $100 can buy." The idea is that it can be trained on hardware that can be rented for sufficient time to do the training for about $100. The target hardware is an 8xH100 machine. It goes through tokenization, pretraining (this is the compute intensive part), midtraining, supervised finetuning, reinforcement learning, and even a light-weight web UI to invoke it.  A more expensive to train version of the same mo...
Read more

Comfy UI on DGX Spark

Image
 I took a brief break from Nanochat to try out some other workloads on the Spark. NVIDIA has a nice start up guide for the Spark here:  https://build.nvidia.com/spark The first step is to verify prerequisites. Running the suggested steps in the Comfy UI guide from the link above, I found that I don't have the NVIDIA CUDA toolkit installed. That's rectified with opening a terminal session to the Spark from NVIDIA Sync and running (the other pre-reqs were preinstalled for me but you may want to check them from the link above): sudo apt install nvidia-cuda-toolkit After that, I pulled down the Comfy repository. I did this before the venv steps because I want the virtual environment in the ComfyUI directory, and pulling down the repository creates it. git clone https://github.com/comfyanonymous/ComfyUI.git Then they have me set up a virtual environment using venv:  python3 -m venv comfyui-env source comfyui-env/bin/activate I'm not going off script on this one to use uv, but ...
Read more

Training Nanochat on DGX Spark

Image
Picking up from where I left off, it was time to attempt training. The first thing I did was look for others who had tried this so I could learn from them. Specifically, I was expecting to pick up tips on parameters to torchrun tuned for my hardware. I found this discussion on the Nanochat github: https://github.com/karpathy/nanochat/discussions/28 Which led me to this discussion on the NVIDIA DGX forum: https://forums.developer.nvidia.com/t/anyone-got-nanochat-training-working-on-the-dgx-spark/348537/8 The first thing I did was grab the torchrun command line from afalk42's post. But I hit the same problem he described whereupon torchrun would error out with an error about Triton 3.5.0 not knowing what an sm_121a is (it's the Blackwell GB10). I needed to additionally run a series of steps to manually update CUDA to 13.0.2 adn then point Triton to the updated ptxas version installed with CUDA 13.0.2. For posterity, the instructions contained therein are repeated here: Install CU...
Read more