Document the architecture

[jddeep]

State how you are going to build something like a Clubhouse or Twitter spaces.

• How is audio streaming going to be handled
• How are people going to join in a space

A lot of questions. Research and come up with answers.

[KlausMikhaelson]

Hey @jddeep would like to work on it !

[Onyx2406]

Audio streaming: Flutter provides a package called audioplayers that can be used to play and control audio files. It supports various audio formats such as MP3, WAV, and FLAC. To handle real-time audio streaming, we can use a third party library such as Agora or Twilio.

Backend: To build the backend, we can use Node.js with Socket.IO, Ruby on Rails, or Django.

Frontend: Flutter provides a rich set of UI components that can be used to build the user interface. It also provides support for real-time updates using streams and reactive programming.

User authentication and onboarding:

For user authentication, we can use Firebase Authentication, to sign up and log in using their Google, Facebook, or Twitter accounts. Once users are authenticated, they can create their profiles, set their interests, and follow other users.

Space creation and management:

To implement space management, we can use a real-time database such as Firebase Realtime Database.

Audio quality:

To ensure high-quality audio, you can implement noise reduction, echo cancellation, and other audio processing techniques using third-party libraries. we can also use Adaptive Bitrate Streaming to adjust the audio quality based on the user's network conditions.

[chandansgowda]

Hey @Onyx2406
Great job. Actually, I found some inconsistency and improvements in your plan.

1. Since we are creating a backend ourselves, why are we using Firebase which is again a BaaS service?
2. I think we can build our own setup for handling realtime audio rather than using a paid third party service
3. Since it's an open source project, we can consider adding SignIn with GitHub also.
4. Which architecture design pattern are we using ?
5. What will we use for state management?

[D-Ajay-Kumar]

Here's I'd go about building an audio streaming app in Flutter -

1. Audio Input: The first step we would need to focus on for Resonate is creating an audio streaming system to capture audio from device’s microphone or other input device like earphone etc. Fortunately, for this Flutter provides the microphone package that we can use to capture audio from the device. If everyone's okay with that then we can use it, or just build our own logic from scratch. But then we'd have to write separate platform code for Android and iOS. Depends on the time and resources available for this project.

2. Audio Encoding: After capturing audio, we will have to encode it into a format that can be streamed over the network. Common audio formats used for streaming are MP3 and AAC. Again, Flutter provides several packages to encode audio into these formats, I’d suggest 'flutter_ffmpeg' package as I’ve used it in the past and it’s good. If not, then again we can build our own logic.

3. Audio Streaming: Once the audio is captured and encoded, we can start streaming it over the network. Flutter provides the 'dart:io' library, which I’m sure every flutter dev has used. This includes classes for creating and managing network connections. We can use these classes to create a socket connection and stream the audio data to a server.

4. Server-side Processing: On the server side, we’d need to receive the audio stream and process it. This can involve decoding the audio data, storing it, and sending it to other clients who are listening to the stream. If needed we can also write logic to detect potential harmful audio to take some action but that comes later when we have a working prototype. There are several server-side frameworks and tools we can use for this, my preferred one would be Node.js with the socket.io library as it is robust, fast, secure and easy to scale.

5. Audio Playback: Finally, on the client side, we’d need to receive the audio stream and play it back to the user. Flutter provides multiple packages for the same, I won’t name any at the moment, but there’s plenty which allow us to play audio in our app. We can use one of these packages, or create an audio player of our own from scratch with minimal features, to receive the audio stream from the server and play it back to the user.

Answering @chandansgowda questions -

6. Design Pattern - For an audio streaming app focusing on high quality and low latency, we can use the Model view controller (MVC) design pattern. This pattern is a widely used and it’s very easy to understand and work with which will help even the beginners to get started with. Besides that, by using the MVC pattern we can separate the concerns of our application and improve its modularity, maintainability, and scalability. This pattern will enable us to easily modify or replace any component without affecting the other components as for an open source application we’d need to constantly modify a lot of components.

7. State Management - I've personally only used Provider and BloC for state management and I can easily back both of them for managing any kind of state. This shouldn't be a huge question as managing state is the least of the complexities for such an app. Provider is backed by Google and Flutter team itself but at times it becomes difficult to write the most efficient code. BloC solves that so we can go with that as well.

Hit me up if there's anything I left unclear :))

[ViAsmit]

@D-Ajay-Kumar

I loved your suggestion, and feeling very much aligned to your thoughts.
But what do you say for MVVM(Model View View-Model) pattern for mobile application. I feel like it will be more suitable for application side.
Any Comments?

Reference:
https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93viewmodel
https://github.com/ViAsmit/Flutter_MVVM_With_GetIt

[ishivanshgoel]

@jddeep

For audio streaming,

1. There are multiple options. Selecting one depends on the use case and the various related constraints.
   For sure there are multiple paid services available like Twilio, Dyte SDKs, etc but as we are building an open-source product we can utilize various open-source technologies like WebRTC.
2. WebRTC certainly has some scalability limitations (I faced them while working on a project), need to look more into it.
3. Second option that comes to my mind right now is using the asterisk (IP PBX) for audio streaming. Asterisk is an open-source technology that powers VOIP calls (Voice over IP).
4. In my opinion (based on a few experiments I did for comparison), VOIP calls are better than WebRTC calls in terms of voice quality and also scalability.
   By scalability (for audio streaming), I mean the number of users that can communicate at a time with each other.

For making people join a space,

1. It depends on the audio streaming platform that we are going to use. If we go with WebRTC, then a signaling server would be required, which can be based on the WebSocket protocol (for real-time communication purposes).
2. If we use the asterisk, It uses SIP (session initiation protocol) for signaling. By signaling, I mean exchanging the public IPs of peers in the space so that communicate with each other by sending RTP streams.

What thoughts do you guys have about choosing technology for audio streaming over the internet?

[AntoTomAbraham]

Audio Streaming

Few ways to implement audio Streaming in the chat application

1. Using 3rd party paid tools like Agora, 100ms. if the aim is just to build an MVP then using 3rd party tools will be ideal since first 10,000 minutes every month is free.

2. WebRTC is the second option, we can use WebRTC+Firebase+Flutter to build the product (Serverless).

3. Custom backend is the next solution. Node JS, Express, MongoDB, WebRTC will be the best.

Joining Channel

1. Using 3rd party tools will make everything easy.

2. We can show available rooms and will use Signalling server (stun server) to connect devices and allow them to join this room.

Architecture

Provider or Bloc will be the best, since both are scalable and efficient. from my personal experience i will say its better to go with any of these two.

Design Pattern and Audio Streaming

MVC pattern, This pattern is widely used and proven.

Flutter have many packages which will help to stream audio.

My Opinion

My opinion will be to use Flutter in the front end and NodeJS + Express + MongoDB + WebRTC in the backend as it is more scalable and good for long term. Using 3rd party tool will be nice if we want to build a MVP. Using Firebase + Flutter + WebRTC will be nice but the problem with this will be less scalability

For realtime communication we will be using Webrtc in the backend we will use Node JS and mongoDB for data storage. Flutter has flutter_webrtc for realtime communication.

This is an ER diagram that I made, social engaging features like - follow, and like are missing in this.
Feedback is accepted.

WebRTC working

[ALIRAZA17]

@jddeep
In terms of audio streaming, you can use a third-party library like Agora.io or Twilio Programmable Voice. These libraries provide APIs that allow you to add real-time audio and voice chat to your app. You will need to integrate this library into your Flutter project and configure it to work with Firebase.

To allow people to join a space, you can create a list of available rooms or spaces, and users can select a room to join. Once they join a room, they will be able to participate in the conversation by speaking or listening. You will need to implement a system to keep track of which users are in which room and ensure that users can only join one room at a time.

[Joymakwana2003]

Hello @jddeep sir,

To build an audio streaming platform like Clubhouse or Twitter Spaces, the following steps can be taken:

1. Platform Infrastructure: The first step would be to set up the platform infrastructure that can handle audio streaming. This can be done by using cloud-based services such as AWS, Google Cloud, or Microsoft Azure. These services offer scalable and reliable infrastructure to handle real-time audio streaming. If we don't want to use the paid services, then we can use open-source software like:

• Apache Kafka: It is an open-source distributed event streaming platform that can handle messaging between different application components. Kafka is highly scalable and fault-tolerant, making it suitable for real-time audio streaming.
• OpenStack: It is an open-source cloud computing platform that allows to set up a private cloud infrastructure. It provides virtual machines, storage, and networking services that can be used to build and deploy our application.

2. Audio Streaming Protocol: To handle audio streaming, we can use protocols like WebRTC (Web Real-Time Communication) which enables real-time communication between browsers and mobile applications. It supports high-quality audio streaming and works well on various devices and platforms.

3. Joining a Space: To join a space, users need to register and create a profile on the platform. Once they are logged in, they can browse the list of active spaces and join the one they are interested in. They can also create their own space and invite others to join.

4. Moderation: It is important to have moderators who can manage the space and ensure that the conversations are appropriate and respectful. Moderators can remove users who violate the platform's community guidelines or mute them if necessary.

5. Social Features: To make the platform engaging, we can include social features such as following, liking, commenting, and direct messaging. This will allow users to connect with each other and build a community around shared interests.

In terms of audio streaming, the platform can handle it by using the following approach:

1. Real-time Audio Encoding: The audio is captured from the user's device and encoded in real-time using a high-quality codec. This reduces latency and ensures that the audio quality is maintained.

2. Audio Delivery: The encoded audio is delivered to the platform's servers in real-time using a low-latency protocol such as WebRTC.

3. Audio Mixing: The audio from multiple users is mixed together in real-time, allowing everyone in the space to hear each other. The audio mixing can be done using software like AudioMixer or FFmpeg.

4. Audio Decoding: The mixed audio is then decoded and sent to each user's device using WebRTC, ensuring low-latency and high-quality audio.

Overall, building a platform like Clubhouse or Twitter Spaces requires careful consideration of the audio streaming protocol, moderation, and social features. Also, I have yet to use all the open-source software listed above, but I've used some of them. The rest of the software I got to know is from Google. So, I don't know whether it would be the best fit.

[tanishagupta1]

Hello @jddeep sir,

In order to make an audio streaming platform like Twitter spaces and clubhouse we need to consider below steps and pointers

Front-end architecture: we can use Flutter to build the mobile app's front end. It offers good support for real-time communication through WebRTC.

Back-end architecture: Use Firebase to handle user authentication and real-time data synchronization. Firebase offers tools that make it easy to build real-time applications, including Firebase Realtime Database and Cloud Firestore.

Real-time communication: Use WebRTC for real-time audio communication between participants. WebRTC is a browser-based communication protocol that offers low latency and high-quality audio transmission. We can use a third-party library like Twilio to handle WebRTC signaling and communication.

Audio processing: Use NodeJS to handle audio processing on the server side. It offers a wide range of libraries and tools for audio processing, such as the Web Audio API and the AudioContext API. You can also use AWS Lambda to handle serverless audio processing.

Data storage: We can use GCP or AWS to host a scalable NoSQL database, such as Google Cloud Datastore or Amazon DynamoDB. These databases can handle large volumes of data and offer good performance and scalability.

Security: We can use Firebase for user authentication and authorization, and implement encryption of audio traffic to prevent eavesdropping. We can also use AWS or GCP to host the app's back end on a secure and scalable infrastructure.

Analytics: Use Firebase Analytics to track user behavior and app usage, and integrate with Google Analytics or Mixpanel for more advanced analytics and reporting.

[devharsh2k4]

Audio streaming: Flutter provides a package called audio players that can be used to play and control audio files. It supports various audio formats such as MP3, WAV, and FLAC. To handle real-time audio streaming.

backend : we can use nodejs and firebase for login authentication

for frontend we can use flutter

[chandansgowda]

So, as of now our mentor has finalized the following High Level Architecture. Another repository for the backend has also been created. Let's start working.