Google AI engineers created Meena chatbot capable to consider a given conversational context and chat on any topic. It has 2,6 billion parameter, was educated on a colossal dataset — 341 GB of public domain social media chats. It’s 8.5x more data than was used to train previously known GPT-2 generative model by OpenAI Elon Musk’s company.
What is so cool about Meena? It is able to understand the context. Ordinary chatbots answer only the last question of the conversation. And Meena keeps everything in mind and considers it in her answer. To check their product Google developers crowd-sourced free-form conversation with several chatbots, estimated them, using SSA (Sensibleness and Specificity Average) Human Evaluation Metric. The assessment consisted of three parameters: meaningfulness, concreteness and perplexity. The latter is also measured in any neural network to capture the uncertainty of the language model, and as it turned out is strongly correlated with the new SSA Google metric.
Eventually, after countless tests, Meena turned out to be 79% more human-like. For comparison, the Mitsuko chatbot scored only 56%, although it overcame the Turing test five times. It is worth noting that the average person during the chatbot SSA tests gained 86%.
Is Mina available already? Not yet. But we expect Google to release a public demo in the near future.
