MongoDB Aggregation Pipelines : Part 1

MongoDB Aggregation Pipelines : Part 1

Writing pipelines to get subscriber count and number of channels subscribed to in controller which gives user profile.

ยท

6 min read

Introduction :

When we want a user's channel profile we are expecting certain values such as avatar, username, email, cover image, subscriber count, number of channels user is subscribed to and a subscribe button. Hence we want our controller to return these values.

Need of Aggregation Pipelines :

When having Single User Model with Subscriber and Subscription Arrays we can have a simpler schema and easier data access. The problem arises when a user has subscribers and channels subscribed to in millions. This will lead to performance and scalability issues due to the large size of user document.

To solve this issue we use Separate Models for Users and Subscriptions this leads to better flexibility , scalability and performance. However this requires more complex queries to join the user and subscription data. Here we use MongoDB Aggregation Pipelines to write these complex queries.

Understanding the working :

Whenever someone subscribes we create a separate document containing the ObjectID of the subscriber ( the one who is subscribing ) and the channel ( also a user to whom the subscriber is subscribing ). This creates several documents containing subscriber and channel id.

  • Getting the subscriber count:

    When we need for subscriber count of a particular channel ( a user ) we will ask for all the documents having channel same as the channel's ObjectID. Hence we will have all the subscribers which have subscribed to channel in different documents all having the same channel ( the user ).

    The number of Documents we get after applying this filter is the subscriber count of the channel.

  • Getting the count of the channels subscribed to:

    When we need to have the count of the channels the user has subscribed to we will ask for all the documents having the subscriber same as the user's ObjectID. Hence we will get all the channels the user has subscribed to in different documents all having the same subscriber ( user ).

    The number of Documents we get after applying this filter is the number of channel the user has subscribed to.

Controller to get user profile :

const getUserChannelProfile = asyncHandler (async (req, res) => {
    const {username} = req.params

    if (!username?.trim()) {
        throw new ApiError(400, "User name is missing")
    }
// Aggregation Pipeline
    const channel = await User.aggregate([
        { // Stage 1:
            $match: {
                username : username?.toLowerCase()
            }
        },
        { // Stage 2:
            $lookup:{
                from: "subscriptions",
                localField: "_id",
                foreignField: "channel",
                as: "subscribers"
            }
        },
        {// Stage 3:
            $lookup:{
                from: "subscriptions",
                localField: "_id",
                foreignField: "subscriber",
                as: "subscribedTo"
            }
        },
        {// Stage 4:
            $addFields:{
                subscribersCount: {
                    $size: "$subscribers"
                },
                channelSubscribedToCount: {
                    $size: "$subscribedTo"
                },
                isSubscribed: {
                    $cond: {
                        if: {$in: [ req.user?._id, "$subscribers.subscriber"]},
                        then: true,
                        else: false
                    }
                }
            }
        },
        {// Stage 5:
            $project: {
                fullName: 1,
                username: 1,
                email: 1,
                avatar: 1,
                coverImage: 1,
                subscribersCount: 1,
                channelSubscribedToCount: 1,
                isSubscribed: 1,
            }
        }
    ])

    if (!channel.length) {
        throw new ApiError(400, "channel does not exist")
    }

    return res
    .status(200)
    .json(
        new ApiResponse(200, channel[0], "User channel fetched successfully")
    )
})
  • Stage 1: ( $match )

    This stage filters out the document from the collection of documents ( here different users ) according to the given parameters passed in it and returns it to the next stage.

    Here in this stage we match the username in the User model with username which is coming from the req.params ( routing: /c/:username ). I have used toLowercase() as I had converted the username in lowercase before storing them in my database.

  • Stage 2: ( $lookup )

    This stage joins two collection in the same database and adds an array field to each document.

    Here it searches for the document in the subscriptions model where the foreign field channel has the same value as the local field ( users ) " _id ".These results are added to the subscribers array in the output document.

    Hence in users model we now have an array name subscribers which has documents in it with the same value of channel in each document. In simple words we got our array of subscribers.

  • Stage 3: ( $lookup )

    Here it searches for the document in the subscriptions model where the foreign field subscriber has the same value as the local field ( users ) " _id ".These results are added to the subscribedTo array in the output document.

    Hence in users model we now have an array name subscribedTo which has documents in it with the same value of subscriber in each document. In simple words we got our array of the channels we have subscribed to.

  • Stage 4: ( $addFields )

    This helps us to add new fields in our users model.

    We have added a new field subscribersCount in our model which gives us the size of the array subscribers given by stage 2 lookup with the help of $size .

    Next we have added a new field channelSubscribedToCount in our model which gives us the size of the array subscribedTo given by stage 3 lookup with the help of $size .

    Then we add a field named isSubscribed in which we have passed a condition where $if the request to get the channel profile is coming from a _id which is already present in on of the objects in the array of subscribers then it gives us the value true i.e. we have already subscribed to the username we are searching for.

  • Stage 5: ( $project )

    This field passes along the documents with the requested fields to the next stage in the pipeline and any thing not included in this won't be passed onto the next stage.

    It has a simple syntax in which the field : 1 or true implies inclusion of the field and field : 0 or false implies exclusion.

This aggregation pipeline which consist of 5 stages returns an object containing avatar, username, email, cover image, subscriber count, number of channels user is subscribed to and isSubscribed fields in the const channel

Later we return this channel with appropriate status code and message after performing proper error handlings.

Note: we have returned channel[0] as channel is an array consisting objects here only one element will be present in the channel.

Testing :

To setup MongoDB Atlas checkout : Hitesh Choudhary's video to setup MongoDB atlas .

If you want the sample data checkout my GitHub Gist AdityaBVerma .

# Notes :

To know more about MongoDB aggregation Pipelines visit MongoDB Aggregation Pipelines .

To learn aggregation pipelines in a fun way do checkout Hitesh Choudhary's playlist .

To know more about the controllers and the utilities and middleware used in it visit my repo on Git hub AdityaBVerma/Backend-with-Js


๐Ÿ‘‹ Hello, I'm Aditya Verma ๐Ÿ˜

โœŒ๏ธ If you liked this article, consider sharing it with others.

๐Ÿ˜Š Feel free to use this article's images and to add comments in case of issues.

๐Ÿฅฐ Thank You for reading.

ย