{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "view-in-github", "colab_type": "text" }, "source": [ "\"Open" ] }, { "cell_type": "markdown", "metadata": { "id": "SiiXJ7K_fePG" }, "source": [ "

\n", " \n", " \n", " \n", " \"nerfstudio\"\n", " \n", "

\n", "\n", "\n", "# Nerfstudio: A collaboration friendly studio for NeRFs\n", "\n", "\n", "![GitHub stars](https://img.shields.io/github/stars/nerfstudio-project/nerfstudio?color=gold&style=social)\n", "\n", "This colab shows how to train and view NeRFs from Nerfstudio both on pre-made datasets or from your own videos/images.\n", "\n", "\\\\\n", "\n", "Credit to [NeX](https://nex-mpi.github.io/) for Google Colab format." ] }, { "cell_type": "markdown", "metadata": { "id": "Yyx5h6kz5ga7" }, "source": [ "## Frequently Asked Questions\n", "\n", "* **Downloading custom data is stalling (no output):**\n", " * This is a bug in Colab. The data is processing, but may take a while to complete. You will know processing completed if `data/nerfstudio/custom_data/transforms.json` exists. Terminating the cell early will result in not being able to train.\n", "* **Processing custom data is taking a long time:**\n", " * The time it takes to process data depends on the number of images and its resolution. If processing is taking too long, try lowering the resolution of your custom data.\n", "* **Error: Data processing did not complete:**\n", " * This means that the data processing script did not fully complete. This could be because there were not enough images, or that the images were of low quality. We recommend images with little to no motion blur and lots of visual overlap of the scene to increase the chances of successful processing.\n", "* **Training is not showing progress**:\n", " * The lack of output is a bug in Colab. You can see the training progress from the viewer.\n", "* **Viewer Quality is bad / Low resolution**:\n", " * This may be because more GPU is being used on training that rendering the viewer. Try pausing training or decreasing training utilization.\n", "* **WARNING: Running pip as the 'root' user...:**:\n", " * This and other pip warnings or errors can be safely ignored.\n", "* **Other problems?**\n", " * Feel free to create an issue on our [GitHub repo](https://github.com/nerfstudio-project/nerfstudio).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "9oyLHl8QfYwP", "cellView": "form" }, "outputs": [], "source": [ "#@markdown

Install Nerfstudio and Dependencies (~8 min)

\n", "\n", "%cd /content/\n", "!pip install --upgrade pip\n", "!pip install torch==2.0.1+cu118 torchvision==0.15.2+cu118 --extra-index-url https://download.pytorch.org/whl/cu118\n", "\n", "# Installing TinyCuda\n", "%cd /content/\n", "!gdown \"https://drive.google.com/u/1/uc?id=1-7x7qQfB7bIw2zV4Lr6-yhvMpjXC84Q5&confirm=t\"\n", "!pip install tinycudann-1.7-cp310-cp310-linux_x86_64.whl\n", "\n", "# Installing COLMAP\n", "%cd /content/\n", "!gdown \"https://drive.google.com/u/0/uc?id=15WngFRNar_b8CaPR5R-hvQ3eAnlyk_SL&confirm=t\"\n", "!sudo apt-get install \\\n", " build-essential \\\n", " libboost-program-options-dev \\\n", " libboost-filesystem-dev \\\n", " libboost-graph-dev \\\n", " libboost-system-dev \\\n", " libboost-test-dev \\\n", " libeigen3-dev \\\n", " libflann-dev \\\n", " libfreeimage-dev \\\n", " libmetis-dev \\\n", " libgoogle-glog-dev \\\n", " libgflags-dev \\\n", " libsqlite3-dev \\\n", " libglew-dev \\\n", " qtbase5-dev \\\n", " libqt5opengl5-dev \\\n", " libcgal-dev \\\n", " libceres-dev\n", "!unzip local.zip -d /usr/\n", "!chmod +x /usr/local/bin/colmap\n", "\n", "# Install nerfstudio\n", "%cd /content/\n", "!pip install git+https://github.com/nerfstudio-project/nerfstudio.git" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "id": "msVLprI4gRA4" }, "outputs": [], "source": [ "#@markdown

Downloading and Processing Data

\n", "#@markdown

Pick the preset scene or upload your own images/video

\n", "import os\n", "import glob\n", "from google.colab import files\n", "from IPython.core.display import display, HTML\n", "\n", "scene = '\\uD83D\\uDDBC poster' #@param ['🖼 poster', '🚜 dozer', '🌄 desolation', '📤 upload your images' , '🎥 upload your own video', '🔺 upload Polycam data', '💽 upload your own Record3D data']\n", "scene = ' '.join(scene.split(' ')[1:])\n", "\n", "if scene == \"upload Polycam data\":\n", " %cd /content/\n", " !mkdir -p /content/data/nerfstudio/custom_data\n", " %cd /content/data/nerfstudio/custom_data/\n", " uploaded = files.upload()\n", " dir = os.getcwd()\n", " if len(uploaded.keys()) > 1:\n", " print(\"ERROR, upload a single .zip file when processing Polycam data\")\n", " dataset_dir = [os.path.join(dir, f) for f in uploaded.keys()][0]\n", " !ns-process-data polycam --data $dataset_dir --output-dir /content/data/nerfstudio/custom_data/\n", " scene = \"custom_data\"\n", "elif scene == 'upload your own Record3D data':\n", " display(HTML('

Zip your Record3D folder, and upload.

'))\n", " display(HTML('

More information on Record3D can be found here.

'))\n", " %cd /content/\n", " !mkdir -p /content/data/nerfstudio/custom_data\n", " %cd /content/data/nerfstudio/custom_data/\n", " uploaded = files.upload()\n", " dir = os.getcwd()\n", " preupload_datasets = [os.path.join(dir, f) for f in uploaded.keys()]\n", " record_3d_zipfile = preupload_datasets[0]\n", " !unzip $record_3d_zipfile -d /content/data/nerfstudio/custom_data\n", " custom_data_directory = glob.glob('/content/data/nerfstudio/custom_data/*')[0]\n", " !ns-process-data record3d --data $custom_data_directory --output-dir /content/data/nerfstudio/custom_data/\n", " scene = \"custom_data\"\n", "elif scene in ['upload your images', 'upload your own video']:\n", " display(HTML('

Select your custom data

'))\n", " display(HTML('

You can select multiple images by pressing ctrl, cmd or shift and click.

'))\n", " display(HTML('

Note: This may take time, especially on higher resolution inputs, so we recommend to download dataset after creation.

'))\n", " !mkdir -p /content/data/nerfstudio/custom_data\n", " if scene == 'upload your images':\n", " !mkdir -p /content/data/nerfstudio/custom_data/raw_images\n", " %cd /content/data/nerfstudio/custom_data/raw_images\n", " uploaded = files.upload()\n", " dir = os.getcwd()\n", " else:\n", " %cd /content/data/nerfstudio/custom_data/\n", " uploaded = files.upload()\n", " dir = os.getcwd()\n", " preupload_datasets = [os.path.join(dir, f) for f in uploaded.keys()]\n", " del uploaded\n", " %cd /content/\n", "\n", " if scene == 'upload your images':\n", " !ns-process-data images --data /content/data/nerfstudio/custom_data/raw_images --output-dir /content/data/nerfstudio/custom_data/\n", " else:\n", " video_path = preupload_datasets[0]\n", " !ns-process-data video --data $video_path --output-dir /content/data/nerfstudio/custom_data/\n", "\n", " scene = \"custom_data\"\n", "else:\n", " %cd /content/\n", " !ns-download-data nerfstudio --capture-name=$scene\n", "\n", "print(\"Data Processing Succeeded!\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "colab": { "base_uri": "https://localhost:8080/", "height": 928 }, "id": "VoKDxqEcjmfC", "outputId": "d2919aa4-96dd-4e50-829f-289e4208882b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "/content\n", "\u001b[K\u001b[?25h/tools/node/bin/lt -> /tools/node/lib/node_modules/localtunnel/bin/lt.js\n", "\u001b[K\u001b[?25h+ localtunnel@2.0.2\n", "added 22 packages from 22 contributors in 2.07s\n", "https://viewer.nerf.studio/?websocket_url=wss://cyan-facts-matter-34-91-1-218.loca.lt\n", "You may need to click Refresh Page after you start training!\n" ] }, { "data": { "text/html": [ "\n", " \n", " " ], "text/plain": [ "" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "#@markdown

Set up and Start Viewer

\n", "\n", "%cd /content\n", "\n", "# Install localtunnel\n", "# We are using localtunnel https://github.com/localtunnel/localtunnel but ngrok could also be used\n", "!npm install -g localtunnel\n", "\n", "# Tunnel port 7007, the default for\n", "!rm url.txt 2> /dev/null\n", "get_ipython().system_raw('lt --port 7007 >> url.txt 2>&1 &')\n", "\n", "import time\n", "time.sleep(3) # the previous command needs time to write to url.txt\n", "\n", "\n", "with open('url.txt') as f:\n", " lines = f.readlines()\n", "websocket_url = lines[0].split(\": \")[1].strip().replace(\"https\", \"wss\")\n", "# from nerfstudio.utils.io import load_from_json\n", "# from pathlib import Path\n", "# json_filename = \"nerfstudio/nerfstudio/viewer/app/package.json\"\n", "# version = load_from_json(Path(json_filename))[\"version\"]\n", "url = f\"https://viewer.nerf.studio/?websocket_url={websocket_url}\"\n", "print(url)\n", "print(\"You may need to click Refresh Page after you start training!\")\n", "from IPython import display\n", "display.IFrame(src=url, height=800, width=\"100%\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "id": "m_N8_cLfjoXD" }, "outputs": [], "source": [ "#@markdown

Start Training

\n", "\n", "%cd /content\n", "if os.path.exists(f\"data/nerfstudio/{scene}/transforms.json\"):\n", " !ns-train nerfacto --viewer.websocket-port 7007 nerfstudio-data --data data/nerfstudio/$scene --downscale-factor 4\n", "else:\n", " from IPython.core.display import display, HTML\n", " display(HTML('

Error: Data processing did not complete

'))\n", " display(HTML('

Please re-run `Downloading and Processing Data`, or view the FAQ for more info.

'))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "colab": { "base_uri": "https://localhost:8080/" }, "id": "WGt8ukG6Htg3", "outputId": "fa946890-c7d8-4e46-a54e-7231bc5a2059" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[2;36m[19:48:48]\u001b[0m\u001b[2;36m \u001b[0mSkipping \u001b[1;36m0\u001b[0m files in dataset split train. \u001b]8;id=527413;file:///content/nerfstudio/nerfstudio/data/dataparsers/nerfstudio_dataparser.py\u001b\\\u001b[2mnerfstudio_dataparser.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=243595;file:///content/nerfstudio/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n", "\u001b[2;36m \u001b[0m\u001b[2;36m \u001b[0mSkipping \u001b[1;36m0\u001b[0m files in dataset split test. \u001b]8;id=109270;file:///content/nerfstudio/nerfstudio/data/dataparsers/nerfstudio_dataparser.py\u001b\\\u001b[2mnerfstudio_dataparser.py\u001b[0m\u001b]8;;\u001b\\\u001b[2m:\u001b[0m\u001b]8;id=464675;file:///content/nerfstudio/nerfstudio/data/dataparsers/nerfstudio_dataparser.py#91\u001b\\\u001b[2m91\u001b[0m\u001b]8;;\u001b\\\n", "\u001b[2KLoading data batch \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h/usr/local/lib/python3.7/site-packages/torch/utils/data/dataloader.py:566: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n", " cpuset_checked))\n", "\u001b[2KLoading data batch \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h/usr/local/lib/python3.7/site-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.\n", " f\"The parameter '{pretrained_param}' is deprecated since 0.13 and will be removed in 0.15, \"\n", "/usr/local/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.\n", " warnings.warn(msg)\n", "Loading latest checkpoint from load_dir\n", "✅ Done loading checkpoint from \n", "outputs/data-nerfstudio-poster/nerfacto/\u001b[1;36m2022\u001b[0m-\u001b[1;36m10\u001b[0m-29_192844/nerfstudio_models/step-\u001b[1;36m000014000.\u001b[0mckpt\n", "\u001b[1;32mCreating trajectory video\u001b[0m\n", "\u001b[2K🎥 Rendering 🎥 \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[35m100%\u001b[0m \u001b[31m0.14 fps\u001b[0m \u001b[33m11:47\u001b[0m\n", "\u001b[2K\u001b[32m( ● )\u001b[0m \u001b[33mSaving video\u001b[0m\n", "\u001b[1A\u001b[2K\u001b[92m────────────────────────────────────────────── \u001b[0m\u001b[32m 🎉 🎉 🎉 Success 🎉 🎉 🎉\u001b[0m\u001b[92m ──────────────────────────────────────────────\u001b[0m\n", " \u001b[32mSaved video to renders/output.mp4\u001b[0m \n", "\u001b[0m" ] } ], "source": [ "#@title # Render Video { vertical-output: true }\n", "#@markdown

Export the camera path from within the viewer, then run this cell.

\n", "#@markdown
The rendered video should be at renders/output.mp4!
\n", "\n", "\n", "base_dir = \"/content/outputs/unnamed/nerfacto/\"\n", "training_run_dir = base_dir + os.listdir(base_dir)[0]\n", "\n", "from IPython.core.display import display, HTML\n", "display(HTML('

Upload the camera path JSON.

'))\n", "%cd $training_run_dir\n", "uploaded = files.upload()\n", "uploaded_camera_path_filename = list(uploaded.keys())[0]\n", "\n", "config_filename = training_run_dir + \"/config.yml\"\n", "camera_path_filename = training_run_dir + \"/\" + uploaded_camera_path_filename\n", "camera_path_filename = camera_path_filename.replace(\" \", \"\\\\ \").replace(\"(\", \"\\\\(\").replace(\")\", \"\\\\)\")\n", "\n", "%cd /content/\n", "!ns-render camera-path --load-config $config_filename --camera-path-filename $camera_path_filename --output-path renders/output.mp4" ] } ], "metadata": { "accelerator": "GPU", "colab": { "provenance": [], "include_colab_link": true }, "gpuClass": "standard", "kernelspec": { "display_name": "Python 3.8.13 ('nerfstudio')", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.8.13" }, "vscode": { "interpreter": { "hash": "c59f626636933ef1dc834fb3684b382f705301c5306cf8436d2da634c2289783" } } }, "nbformat": 4, "nbformat_minor": 0 }