Uploading Files to MongoDB with GridFS and Multer Using NodeJS
Hello, in this tutorial we will learn how to upload files directly to MongoDB using GridFS specification.
If you think TLDR; just check the finish code here.
The official docs explain when to use this specification for uploading files. Which is summarized in the following:
-
If your filesystem limits the number of files in a directory, you can use GridFS to store as many files as needed.
-
When you want to access information from portions of large files without having to load whole files into memory, you can use GridFS to recall sections of files without reading the entire file into memory.
-
When you want to keep your files and metadata automatically synced and deployed across a number of systems and facilities, you can use GridFS. When using geographically distributed replica sets, MongoDB can distribute files and their metadata automatically to a number of mongod instances and facilities.
Since, GridFS stores files in chunks. The following are the collections created:
- chunks stores the binary chunks.
- files stores the file’s metadata.
Prerequisites
- NodeJS LTS
- MongoDB installed on your local machine
- a Code Editor
Setting up a local NodeJS server
Go to your command line, and type
npm init -y
This will generate an package.json file with default values.
Then install all the dependencies required for this project
npm install express mongoose ejs multer multer-gridfs-storage
Create a file named app.js in the root of the project. Require the necessary packages for creating a server.
const express = require('express'); const app = express(); app.use(express.json()); app.set('view engine', 'ejs'); const port = 5001; app.listen(port, () => { console.log('server started on ' + port); });
It will be better for us to create scripts to run the web app from the command line, go to your package.json file and on the scripts key, add the following:
"scripts": { "start": "node app.js", "dev": "nodemon app.js" }
then run, npm start and the server should start on the port 5001. You should see one log on the command line stating that server started on 5001.
Connecting to Database, Initializing GridFsStorage and Creating a Storage
Require all the necessary packages
const crypto = require('crypto'); const path = require('path'); const mongoose = require('mongoose'); const multer = require('multer'); const GridFsStorage = require('multer-gridfs-storage');
Mongoose is an ORM for MongoDB which will be used for this tutorial. Multer is a NodeJS middleware which facilitates file uploads. And GridFsStorage is GridFS storage engine for Multer to store uploaded files directly to MongoDB. Crypto and Path will be used to create a unique name for the file uploaded.
// DB const mongoURI = 'mongodb://localhost:27017/node-file-upl'; // connection const conn = mongoose.createConnection(mongoURI, { useNewUrlParser: true, useUnifiedTopology: true, });
Now, Initializing the GridFsStorage
// init gfs let gfs; conn.once('open', () => { // init stream gfs = new mongoose.mongo.GridFSBucket(conn.db, { bucketName: 'uploads', }); });
Here we are using the native nodejs-mongodb-drive which the mongoose uses and creating a GridFSBucket, we are passing the db to the bucket, you can see we are giving one bucket name, this bucket name will be used a name of a collection.
// Storage const storage = new GridFsStorage({ url: mongoURI, file: (req, file) => { return new Promise((resolve, reject) => { crypto.randomBytes(16, (err, buf) => { if (err) { return reject(err); } const filename = buf.toString('hex') + path.extname(file.originalname); const fileInfo = { filename: filename, bucketName: 'uploads', }; resolve(fileInfo); }); }); }, }); const upload = multer({ storage, });
Now we are initializing the storage as per Multer GridFS and creating random bytes using the randomBytes method present on the crypto library.
Here we are using the Promise constructor to create a promise, which then resolves with the fileInfo object. This step is optional as you can only pass a url key and the bucket will work just fine and not change the file name. For example you can use like the following :
const storage = new GridFsStorage({ url: mongoURI });
Next lets set up our frontend with a template engine and configure express to render the template.
Creating the view
Create a new folder named views in the root of the folder, and inside it create a file named index.ejs. Here we will store our front end view. I will not bore you guys will the HTML creation and just post the code for it. I am using bootstrap for fast prototyping.
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta http-equiv="X-UA-Compatible" content="ie=edge" /> <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" /> <title>Mongo File Upload</title> </head> <body> <div class="container"> <div class="row"> <div class="col-md-6 m-auto"> <h1 class="my-4">Lets upload some stuff</h1> <form action="/upload" method="post" enctype="multipart/form-data"> <div class="custom-file mb-3"> <input type="file" class="custom-file-input" name="file" id="file1" onchange="readSingleFile(this.files)" /> <label class="custom-file-label" for="file1" id="file-label" >Choose file</label > </div> <input type="submit" value="Submit" class="btn btn-primary btn-block" /> </form> </div> </div> </div> <script src="https://code.jquery.com/jquery-3.3.1.slim.min.js"></script> <script src="https://cdnjs.cloudflare.com/ajax/libs/popper.js/1.14.7/umd/popper.min.js"></script> <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/js/bootstrap.min.js"></script> <script> function readSingleFile(e) { const name = e[0].name; document.getElementById('file-label').textContent = name; } </script> </body> </html>
Setting up the express app to render the view. Set up the view engine middleware to ejs
.... app.use(express.json()); app.set("view engine", "ejs"); .... app.get("/", (req, res) => { res.render("index") })
Then start the server again, go to the browser and open http://localhost:5001, and you should see one page rendered with the view we just created.
Create Request to handle the form submit and upload the file
app.post('/upload', upload.single('file'), (req, res) => { res.redirect('/'); });
As we already did most of our heavy lifting while creating a storage bucket and multer take cares of the rest. We just need to pass the middleware and then just redirect to the same url.
The tricky part is to download or in this case stream the data from the GridFS storage bucket and render the image, for that we will create a route for showing an image that will take the name of the file as an argument or passed as a route param.
app.get('/image/:filename', (req, res) => { // console.log('id', req.params.id) const file = gfs .find({ filename: req.params.filename, }) .toArray((err, files) => { if (!files || files.length === 0) { return res.status(404).json({ err: 'no files exist', }); } gfs.openDownloadStreamByName(req.params.filename).pipe(res); }); });
On the gridfs bucket we get access to many methods one such is find, which is very similar to normal find in MongoDB and accepts a filename as a first argument and then we are converting the result to an array and check if there is any file with such filename and if there is we use another method which is present on the gridfs bucket called openDownloadStreamByName which then again takes the filename and then we use the pipe to return the response to the client.
Now up until now, we can get the image with the above route but no way to render it on our view, so let's create a method inside the route where we were rending our index.ejs page.
.... app.get("/", (req, res) => { if(!gfs) { console.log("some error occured, check connection to db"); res.send("some error occured, check connection to db"); process.exit(0); } gfs.find().toArray((err, files) => { // check if files if (!files || files.length === 0) { return res.render("index", { files: false }); } else { const f = files .map(file => { if ( file.contentType === "image/png" || file.contentType === "image/jpeg" ) { file.isImage = true; } else { file.isImage = false; } return file; }) .sort((a, b) => { return ( new Date(b["uploadDate"]).getTime() - new Date(a["uploadDate"]).getTime() ); }); return res.render("index", { files: f }); } }); }); ....
Here you can see a lot of optional code like the sorting of the array and you can skip those.
Now, On the template, we loop over the files sent and then show the images below the form. We will only render the files which are of type jpg or png, that check can be upgraded by using a regex and depends on the personal preference.
<hr /> <% if(files) { %> <% files.forEach(function(file) {%> <div class="card mb-3"> <div class="card-header"> <div class="card-title"><%= file.filename %></div> </div> <div class="card-body"> <% if (file.isImage) { %> <img src="image/<%= file.filename %>" width="250" alt="" class="img-responsive" /> <%} else { %> <p><% file.filename %></p> <% } %> </div> <div class="card-footer"> <form action="/files/del/<%= file._id %>" method="post"> <button type="submit" class="btn btn-danger">Remove</button> </form> </div> </div> <%}) %> <% } else { %> <p>No files to show</p> <% } %>
You can see there is one remove button on the above code, so let us create one delete route to remove the file from the database.
// files/del/:id // Delete chunks from the db app.post('/files/del/:id', (req, res) => { gfs.delete(new mongoose.Types.ObjectId(req.params.id), (err, data) => { if (err) return res.status(404).json({ err: err.message }); res.redirect('/'); }); });
Here we get the id as a string so that needs to be converted into a mongodb objectid and then only the bucket method can delete the file with the corresponding id. I kept things simple by not using the delete HTTP method here you are free to use it if you feel like, a post request just works fine here.
Conclusion
As we can see MongoDB provides a nice solution to store files on the database and can come handy while creating WebApps with less storage facility, but keep in mind you can only store documents up to 16mb.