How to build a voice bot integrated with virtual assistant

I am a big fan of JARVIS, and tried to search online if I can build something similar to it. My “research” came back to either a voice bot, chat bot or virtual assistant (with a bunch of if statement). Well.. that isn’t really JARVIS, is it?

Something I wanted to build will need to have the following:

1. Memory : being able to remember the previous conversation and make response based on previous context taken into consideration
2. Being able to help with tasks: something similar as a virtual bot?
3. Communication: this would be the “voice bot” part
4. Personality: this could be done with voice, tone, and perhaps the way it replies

My initial thought process is to add a form of “memory” (possibly database) and utilize that to produce a more “humanized” reply. Conversations and parameters (“main points within a conversation”) will be stored and removed/replaced based on the least used rule after a period of time (similar to memory).

Having with limited time and knowledge, those great ideas wasn’t as feasible to me (at least for now). Therefore, I utilized current resources that are available to make a quick and dirty little bot.

The bot will be utilizing a couple of resources:

1. Dialogflow API
2. Python

If you try searching “how to build virtual bot”, you will be able to find a lot of examples. Using that, I integrated Dialogflow API for the voice bot component.

The flow of the bot goes this way:

1. Detected it’s name
2. Record speech and send it to Dialogflow using Dialogflow API (the sample code is available in Dialogflow’s documentary)
3. Return with a json of that includes action, parameters, output (We could also use the detect_intent_with_texttospeech_response function in order to receive a binary output audio content. Alternatively, you could get the output text and send it to your own system to read the text.)
4. action and parameter could be used as a switch to determine the task needed (if statement of virtual bot)

Each of my dialogflow bot intent is coded with an action “tasks.website” or “tasks.search” etc., and parameter “domain”, “query” etc.. Once it detect “tasks.”, it will then send it to the virtual bot script to trigger parameter.

For example, if I said “Hey bot, open gmail”, dialogflow will return action=tasks.website and parameter= gmail . It will loop through the if-statements, find action that matches tasks.website , and open a new tab in my web browser using the import webbrowser library.

Feel free to grab the source code here: https://github.com/Yuan325/Vivi