[논문 리뷰] Language Models that Seek for Knowledge:Modular Search & Generation for Dialogue and Prompt Completion(feat. SeeKeR, FaceBooK, Chatbot, opendomain, 챗봇)

 

 

 

출처 : parlai seeker( 이미지 링크 참조 )

 

안녕하십니까 다제입니다. 

오늘은 SeeKeR에 대해서 논문 리뷰를 진행해보고자 합니다. 

 

SeeKeR는 Blenderbot2의 지식적인(Knowledge) 답변을 잘못한다는 단점을 보안하기 위해 제작된 모델이고, Blenderbot3의 Base model로써 사용되는 모델입니다. 이에, 현재 Open-Domain Chatbot을 연구하는 사람으로써 반드시 알아야하는 모델이라고 생각하여 리뷰를 진행하게 되었습니다. 

 

 

먼저, SeeKeR에 대해 설명 드리기에 앞서 BB2를 복습하고 넘어가도록 하겠습니다. 

이 외  FiD, RAG 등의 개념은 모두 이해하고 있다는 전제하에 설명드리는 점 참고 부탁드립니다. 

출처 : 모델, 데이터, 대화 관점에서의 BlendorBot 2.0 오류 분석 연구(그림 링크 참조)

 

그래도 RAG, FiD에 대해서 간단하게 설명을 드리겠습니다. 

일반적으로 Neural Network는 모델이 훈련되는 시점에서 보유하고 있는 정보가 frozen 됩니다. 즉, 시간이 지남에 따라 새롭게 생성되는 정보를 추가적으로 모델 내부로 가져올 수 없습니다. 이러한 한계점을 극복하기 위해 모델 외부에서 정보를 가져와서 그 정보를 활용하여 모델이 응답할 수 있도록 모델을 설계하는 방법을 RAG라고 이야기 합니다. 

 

FiD는 여러개의 encoder가 있고 여러 개의 encoder를 통해 encoding된 정보를 concat하여 Decoder로 전달하여 응답(response)를 생성하는 방식을 이야기 합니다. 

 

BB2와 SeeKeR는 위 설명드린 FiD-RAG style로 훈련된 모델입니다. 

특히 SeeKeR는 Search, Knowledge, Response 모듈을 가지고 있는 순차적으로 모듈로 구성된 모델이고, Dialogue task와 LM task를 수행할 수 있습니다.

그렇다면, BB2와 SeeKeR가 무엇이 다른가? 하는 생각이 드실 수 있습니다. 이를 위해 간단한 표 하나를 준비하였습니다. 

기존 모델 vs SeeKeR - 직접 제작한 표

BB1과 BB2에서는 Decoder의 수를 극단적으로 늘려 모델을 훈련시켰지만, SeeKeR는 LM task를 수행하기 위해 동일한 encoder와 decoder layer 수를 가지게 됩니다.

Blenderbot 2.0 vs SeeKeR - 직접 제작한 표

이제 Introduction부터 순차적으로 살펴보도록 하겠습니다. 

 

 

1 Introduction

Problem

 - Standard large language models are known to generate fluent but factually incorrect statements.

 - Additionally, as their knowledge is frozen in time from the point when they were trained.

 - However, aggregating information form multiple retrieved documents is a difficult problem.

 

Solution

 - In this paper, we explore a modular architecture that tries to mix the best elements of these diifferent existin solutions.

 - A single transformer architecture is used iteratively to perform three modular task

 - We apply our modular Search-engine→ Knowledge → Response (SeeKeR) language model to the tasks of dialogue and prompt completion

출처 : SeeKeR paper

 

2 Related Work

이 논문을 기반으로 만들었다. 여기에 인터넷 접근이 가능하도록 했다. 

Our work builds on the knowledge to response (K2R) technique (Adolphs et al., 2021)

 

이게 중요한 이슈지 

BB2는 개방형 대화 작업을 위해 인터넷에서 검색하는 것을 기반으로 하지만(Komeili et al., 2021) 지식을 생성하기 위해 모듈식 접근 방식을 사용하지 않고 대신 FiD(퓨전 인 디코더) 방법을 적용합니다.

BB2 grounds on retrieval from the internet for open-domain dialogue tasks (Komeili et al., 2021), but does not use a modular approach to generate knowledge, instead applying the fusion-in- decoder (FiD) method

 

 

3 SeeKeR Model

SeeKeR Class diagram - 직접 제작한 그림

The SeeKeR model we introduce in this paper has the architecture of a standard transformer (Vaswani et al., 2017), except that this same encoder-decoder (for dialogue) or decoder-only (for language mod- eling) model is used in a modular way multiple times.

For each module, special tokens are used in the encoder (or decoder) to indicate which module is being invoked. The output of each module is input into the next, along with the original context.

 

Search Module

Given the encoded input context, a search query is generated.

This is fed into a search engine, which returns results in the form of a set of documents.

Bing Web Search API to retrieve documents and then filter that set of documents by intersecting with Common Crawl and keep the top5.

 

Knowledge Module

Given the encoded input context, and a set of retrieved documents, a knowl- edge response is generated. This consists of one or more relevant phrases or sentences from the retrieved documents.

Note that this task is essentially a “copy” task in that no new tokens have to be generated; the diffi- culty of the task is selecting the relevant knowledge to copy.

 

Repsonse Module

Given the encoded input con- text concatenated with the knowledge response, the final response is generated.

The module must consider relevant context and knowledge while generating a new fluent continuation to the input. The extraction of relevant knowledge by the previ- ous modules makes this task easier; in contrast, a conventional seq2seq model has to solve all these tasks (knowledge acquisition, synthesis, and final response generation) at once.

 

관련된 정보를 한번에 추출하는 것보다는 여러 단계 걸쳐서 추출하면 훨씬 더 잘 뽑을 수 있다.

 

3-1. Architecture and Pre-Training

For our standard language modeling experiments, we consider the GPT2 transformer (Radford et al., 2019) as a base model, and fine-tune it to become a SeeKeR model

 

For our dialogue experiments, we employ a 2.7B parameter transformer encoder-decoder model.

 

 

질문

A single transformer architec- ture is used iteratively to perform three modular tasks: search, generate knowledge, and generate a final response, where the output of each module is fed as additional input to the next

여러분께서는 SeeKeR 논문에서 위와 같이 언급해주셨습니다. 그래서 코드를 찾아보았더니, knowledge, Dialogues, Search, Search decision의 4개의 transformers가 생성되는 것을 저는 관찰하였습니다. SeeKeR가 훈련되기 위해 4개의 Transformer가 사용되는게 맞습니까?

 

그리고 각각의 task별 teacher 있던데 Teacher라고 하면 난 distillation 밖에 생각이 안나는데 여기에 distillation이 사용된 것인가요?

 

 

# **SeeKeR 테스트 현황**

### 1. SeeKeR의 작동순서

- User : ask question about SeeKeR bot (주황색으로 마킹됨)
- query generator : query 생성 (볼드 처리됨)
- Internet Search Result : 5개의 인터넷 검색결과 링크를 전달 받음
- Knowledge generator : 검색된 결과를 가지고 키워드 또는 문장 생성 (파란색으로 마킹됨)
- SeeKeR bot : Answer Response (보라색으로 마킹됨)

## 2. SeeKeR 테스트 결과에 대한 담당자 소견

- BB2보다 지식기반 대화를 잘함
- BB2보다 페르소나에 대한 반영은 미비함
- 지시대명사, 문맥적인 흐름은 BB2보다 훨씬 잘 인지함
- 성능 지표에서도 이러한 결과가 반영되어 있음

## 3. SeeKeR의 실제 테스트 결과내역

Enter Your Message:  Do you know monkeypox?

**15:49:33 | Search Queries: ['monkeyspox']**
15:49:33 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
15:49:40 | URLS:
[https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms](https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms)[https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/](https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/)[https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic](https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic)[https://www.cdc.gov/poxvirus/monkeypox/about.html](https://www.cdc.gov/poxvirus/monkeypox/about.html)[https://www.cdc.gov/poxvirus/monkeypox/index.html](https://www.cdc.gov/poxvirus/monkeypox/index.html)

15:49:54 | Generated knowledge: ['Lives, Protecting People\nMonkeypox']

[ComboFidSearchQuery]: Yes, I know that Monkeypox is a serious disease that can cause serious damage to the human body.

Enter Your Message: What is the cause of monkeyspox?
**15:50:52 | Search Queries: ['monkeyspox']**
15:50:52 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
15:51:00 | URLS:
[https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms](https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms)[https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/](https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/)[https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic](https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic)[https://www.cdc.gov/poxvirus/monkeypox/about.html](https://www.cdc.gov/poxvirus/monkeypox/about.html)[https://www.cdc.gov/poxvirus/monkeypox/index.html](https://www.cdc.gov/poxvirus/monkeypox/index.html)

15:51:17 | Generated knowledge: ['National Vaccine Strategy\nOn June 28, 2022 the National Monkeypox Vaccine strategy was released by the\nCenters for Disease Control and Prevention. CDC twenty four seven. Saving']

[ComboFidSearchQuery]: The cause of monkeyspox is unknown, but it is believed to be caused by the human immune system.

Enter Your Message: I'm sad because so many people died because of it
**15:52:15 | Search Queries: ['monkeyspox']**
15:52:15 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
15:52:20 | URLS:
[https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms](https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms)[https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/](https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/)[https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic](https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic)[https://www.cdc.gov/poxvirus/monkeypox/about.html](https://www.cdc.gov/poxvirus/monkeypox/about.html)[https://www.cdc.gov/poxvirus/monkeypox/index.html](https://www.cdc.gov/poxvirus/monkeypox/index.html)

15:52:34 | Generated knowledge: ['National Health Service of India Covid Cases\nCovid in India Live']

[ComboFidSearchQuery]: I'm sorry to hear that. Do you have any idea how many died from monkeyspox in India?

Enter Your Message:  No. I don't know that.
**16:01:47 | Search Queries: ['monkeyspox']**
16:01:47 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:01:59 | URLS:
[https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms](https://timesofindia.indiatimes.com/india/delhi-reports-first-monkeypox-case-indias-fourth/articleshow/93085899.cms)[https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/](https://globalnews.ca/news/9011383/canada-monkeypox-who-global-health-emergency/)[https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic](https://www.vox.com/science-and-health/2022/7/23/23171631/who-monkeypox-pheic-public-health-emergency-world-health-organization-pheic)[https://www.cdc.gov/poxvirus/monkeypox/about.html](https://www.cdc.gov/poxvirus/monkeypox/about.html)[https://www.cdc.gov/poxvirus/monkeypox/index.html](https://www.cdc.gov/poxvirus/monkeypox/index.html)
16:02:10 | Generated knowledge: ['National Herald Case\nLives,Protecting People']
[ComboFidSearchQuery]: Have you ever heard of the National Herald case? It was a very sad thing to hear about.

Enter Your Message: What is National Herald case?
**16:03:00 | Search Queries: ['national Herald case']**
16:03:00 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:03:12 | URLS:
[https://en.wikipedia.org/wiki/National_Herald_corruption_case](https://en.wikipedia.org/wiki/National_Herald_corruption_case)[https://www.business-standard.com/about/what-is-national-herald-case](https://www.business-standard.com/about/what-is-national-herald-case)[https://www.ndtv.com/india-news/national-herald-case-rahul-gandhi-congress-president-sonia-gandhis-3rd-round-of-questioning-today-3196996](https://www.ndtv.com/india-news/national-herald-case-rahul-gandhi-congress-president-sonia-gandhis-3rd-round-of-questioning-today-3196996)[https://www.indiatoday.in/india/story/national-herald-case-live-updates-sonia-gandhi-ed-questioning-congress-protests-1980023-2022-07-26](https://www.indiatoday.in/india/story/national-herald-case-live-updates-sonia-gandhi-ed-questioning-congress-protests-1980023-2022-07-26)[https://www.bbc.com/news/world-asia-india-61741818](https://www.bbc.com/news/world-asia-india-61741818)
16:03:26 | Generated knowledge: ['The National Herald corruption case is the ongoing case in a Delhi court filed\nby Indian economist and politician Subramanian Swamy against politicians Sonia']
[ComboFidSearchQuery]: It is a corruption case in India. It was filed by an economist named Subramaniam Swamy.

Enter Your Message: Oh.. So Sad new.. How do know it? Searching? Studying?
**16:04:29 | Search Queries: ['subramaniam swamy']**
16:04:29 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:04:37 | URLS:
[https://mobile.twitter.com/swamy39](https://mobile.twitter.com/swamy39)[https://en.wikipedia.org/wiki/Subramanian_Swamy](https://en.wikipedia.org/wiki/Subramanian_Swamy)[https://en.wikipedia.org/wiki/Salman_Haidar](https://en.wikipedia.org/wiki/Salman_Haidar)[https://en.wikipedia.org/wiki/Sanjay_Sarma](https://en.wikipedia.org/wiki/Sanjay_Sarma)[https://en.wikipedia.org/wiki/Hendrik_S._Houthakker](https://en.wikipedia.org/wiki/Hendrik_S._Houthakker)
16:04:49 | Generated knowledge: ['From Wikipedia, the free encyclopedia\n# Hendrik S. Houthakker']
[ComboFidSearchQuery]: I'm not sure, but I do know that the case was filed in the Supreme Court of India.

Enter Your Message: You are so Smart! Awesome bot!
**16:05:28 | Search Queries: ['subramaniam swamy']**
16:05:28 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:05:34 | URLS:
[https://mobile.twitter.com/swamy39](https://mobile.twitter.com/swamy39)[https://en.wikipedia.org/wiki/Subramanian_Swamy](https://en.wikipedia.org/wiki/Subramanian_Swamy)[https://en.wikipedia.org/wiki/Salman_Haidar](https://en.wikipedia.org/wiki/Salman_Haidar)[https://en.wikipedia.org/wiki/Sanjay_Sarma](https://en.wikipedia.org/wiki/Sanjay_Sarma)[https://en.wikipedia.org/wiki/Hendrik_S._Houthakker](https://en.wikipedia.org/wiki/Hendrik_S._Houthakker)
16:05:46 | Generated knowledge: ['# Salman Haidar (1969)\nIndian mechanical engineer']
[ComboFidSearchQuery]: Thank you for your kind words. What do you do for a living? I work as a mechanical engineer.

Enter Your Message: I study computer science and deep learning
**16:08:04 | Search Queries: ['deep learning']**
16:08:04 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:08:12 | URLS:
[https://en.wikipedia.org/wiki/Deep_learning](https://en.wikipedia.org/wiki/Deep_learning)[https://en.wikipedia.org/wiki/Deep_reinforcement_learning](https://en.wikipedia.org/wiki/Deep_reinforcement_learning)[https://en.wikipedia.org/wiki/Feature_learning](https://en.wikipedia.org/wiki/Feature_learning)[https://en.wikipedia.org/wiki/Deep_belief_network](https://en.wikipedia.org/wiki/Deep_belief_network)[https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software](https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software)
16:08:24 | Generated knowledge: ['Machine learning that combines deep learning and reinforcement learning.']
[ComboFidSearchQuery]: That sounds very interesting. Machine learning is a great way to learn. It combines reinforcement learning with deep learning.

Enter Your Message: what do you like ?
**16:14:14 | Search Queries: ['deep learning']**
16:14:14 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:14:21 | URLS:
[https://en.wikipedia.org/wiki/Deep_learning](https://en.wikipedia.org/wiki/Deep_learning)[https://en.wikipedia.org/wiki/Deep_reinforcement_learning](https://en.wikipedia.org/wiki/Deep_reinforcement_learning)[https://en.wikipedia.org/wiki/Feature_learning](https://en.wikipedia.org/wiki/Feature_learning)[https://en.wikipedia.org/wiki/Deep_belief_network](https://en.wikipedia.org/wiki/Deep_belief_network)[https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software](https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software)
16:14:33 | Generated knowledge: ['# Deep belief network\nMachine learning that uses reinforcement learning']
[ComboFidSearchQuery]: I'm a big fan of the deep belief network. Have you heard of it? It combines machine learning with reinforcement learning.

Enter Your Message: do you know how make it?
**16:15:25 | Search Queries: ['deep belief network']**
16:15:25 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:15:43 | URLS:
[https://en.wikipedia.org/wiki/Deep_belief_network](https://en.wikipedia.org/wiki/Deep_belief_network)[https://www.analyticsvidhya.com/blog/2022/03/an-overview-of-deep-belief-network-dbn-in-deep-learning/](https://www.analyticsvidhya.com/blog/2022/03/an-overview-of-deep-belief-network-dbn-in-deep-learning/)[http://blog.skby.net/심층신뢰망-dbn-deep-belief-network/](http://blog.skby.net/%EC%8B%AC%EC%B8%B5%EC%8B%A0%EB%A2%B0%EB%A7%9D-dbn-deep-belief-network/)[https://www.sciencedirect.com/topics/engineering/deep-belief-network](https://www.sciencedirect.com/topics/engineering/deep-belief-network)[https://medium.com/swlh/what-are-rbms-deep-belief-networks-and-why-are-they-important-to-deep-learning-491c7de8937a](https://medium.com/swlh/what-are-rbms-deep-belief-networks-and-why-are-they-important-to-deep-learning-491c7de8937a)
16:15:56 | Generated knowledge: ['Type of artificial neural network\nDeep belief network (DBN)']
[ComboFidSearchQuery]: I know how to make a neural network. It is a type of machine learning that uses reinforcement learning to train neural networks.

Enter Your Message: and do you know who make it?
**16:16:22 | Search Queries: ['Deep learning']**
16:16:22 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:16:29 | URLS:
[https://en.wikipedia.org/wiki/Deep_learning](https://en.wikipedia.org/wiki/Deep_learning)[https://en.wikipedia.org/wiki/Deep_reinforcement_learning](https://en.wikipedia.org/wiki/Deep_reinforcement_learning)[https://en.wikipedia.org/wiki/Feature_learning](https://en.wikipedia.org/wiki/Feature_learning)[https://en.wikipedia.org/wiki/Deep_belief_network](https://en.wikipedia.org/wiki/Deep_belief_network)[https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software](https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software)
16:16:44 | Generated knowledge: ['# Deep reinforcement learning\n# Feature learning # Deep reinforcement network']
[ComboFidSearchQuery]: I do not know who makes it, but there are many different types of neural networks out there.

Enter Your Message: i like a music song. how about you?
**16:17:17 | Search Queries: ['music songs']**
16:17:17 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:17:42 | URLS:
[https://www.youtube.com/watch?v=ZifAFQQofSg](https://www.youtube.com/watch?v=ZifAFQQofSg)[https://music.youtube.com/](https://music.youtube.com/)[https://www.rollingstone.com/music/music-lists/200-greatest-dance-songs-of-all-time-1372888/](https://www.rollingstone.com/music/music-lists/200-greatest-dance-songs-of-all-time-1372888/)[https://www.rollingstone.com/music/music-lists/500-greatest-songs-of-all-time-151127/](https://www.rollingstone.com/music/music-lists/500-greatest-songs-of-all-time-151127/)[https://soundcloud.com/](https://soundcloud.com/)
16:17:55 | Generated knowledge: ['Home Music Music Lists\n* Music\n* TV & Movies']
[ComboFidSearchQuery]: I love music. I listen to a lot of music at home. What kind of music do you listen to?

Enter Your Message: jazz. how about you?
**16:18:18 | Search Queries: ['jazz']**
16:18:18 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:18:26 | URLS:
[https://en.wikipedia.org/wiki/Jazz](https://en.wikipedia.org/wiki/Jazz)[https://en.wikipedia.org/wiki/List_of_jazz_genres](https://en.wikipedia.org/wiki/List_of_jazz_genres)[https://en.wikipedia.org/wiki/Jazz_(word)](https://en.wikipedia.org/wiki/Jazz_(word))[https://en.wikipedia.org/wiki/Jazz_Age](https://en.wikipedia.org/wiki/Jazz_Age)[https://en.wikipedia.org/wiki/Jazz_fusion](https://en.wikipedia.org/wiki/Jazz_fusion)
16:18:38 | Generated knowledge: ['Jazz (word) - Wikipedia_0 / # Jazz']
[ComboFidSearchQuery]: Jazz is my favorite type of music. There are so many different styles of jazz. What is your favorite?

Enter Your Message: i like song, i'm a fool to want you. do you know it?
**16:19:52 | Search Queries: ['jazz']**
16:19:52 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:19:59 | URLS:
[https://en.wikipedia.org/wiki/Jazz](https://en.wikipedia.org/wiki/Jazz)[https://en.wikipedia.org/wiki/List_of_jazz_genres](https://en.wikipedia.org/wiki/List_of_jazz_genres)[https://en.wikipedia.org/wiki/Jazz_(word)](https://en.wikipedia.org/wiki/Jazz_(word))[https://en.wikipedia.org/wiki/Jazz_Age](https://en.wikipedia.org/wiki/Jazz_Age)[https://en.wikipedia.org/wiki/Jazz_fusion](https://en.wikipedia.org/wiki/Jazz_fusion)
16:20:12 | Generated knowledge: ['# Jazz fusion # Jazz history # Jazz music history']
[ComboFidSearchQuery]: Yes, it is a jazz fusion song. I have heard it before. I like it a lot.

Enter Your Message: really? i know that song is singed by Billie Holiday.
**16:22:29 | Search Queries: ['Billie Holiday']**
16:22:29 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:22:36 | URLS:
[https://en.wikipedia.org/wiki/Billie_Holiday](https://en.wikipedia.org/wiki/Billie_Holiday)[https://en.wikipedia.org/wiki/The_United_States_vs._Billie_Holiday](https://en.wikipedia.org/wiki/The_United_States_vs._Billie_Holiday)[https://en.wikipedia.org/wiki/Billie_Holiday_discography](https://en.wikipedia.org/wiki/Billie_Holiday_discography)[https://en.wikipedia.org/wiki/Billie_Holiday_Sings](https://en.wikipedia.org/wiki/Billie_Holiday_Sings)[https://en.wikipedia.org/wiki/Clarence_Holiday](https://en.wikipedia.org/wiki/Clarence_Holiday)
16:22:50 | Generated knowledge: ['Billie Holiday discography\nBillie holiday discography']
[ComboFidSearchQuery]: Yes, that is the song by billie holiday. She has a great discography, you should check it out.

Enter Your Message: do you know who sing a song?
**16:20:45 | Search Queries: ['song singer']**
16:20:45 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:21:13 | URLS:
[https://www.youtube.com/watch?v=Quou6o05g5A](https://www.youtube.com/watch?v=Quou6o05g5A)[https://en.wikipedia.org/wiki/The_Singer_and_the_Song](https://en.wikipedia.org/wiki/The_Singer_and_the_Song)[https://en.wikipedia.org/wiki/Xu_Song_(singer)](https://en.wikipedia.org/wiki/Xu_Song_(singer))[https://www.news18.com/news/buzz/kesariya-without-love-storiya-this-singer-has-fixed-brahmastra-song-for-internet-5616487.html](https://www.news18.com/news/buzz/kesariya-without-love-storiya-this-singer-has-fixed-brahmastra-song-for-internet-5616487.html)[https://www.rollingstone.com/music/music-lists/leonard-cohen-20-essential-songs-114187/](https://www.rollingstone.com/music/music-lists/leonard-cohen-20-essential-songs-114187/)
16:21:27 | Generated knowledge: ['The Singer and the Song is the 1971 second album by Labi Siffre.']
[ComboFidSearchQuery]: The singer and the song is a song by labi siffre, she is a very talented singer.

Enter Your Message: do you when she was born?
**16:23:59 | Search Queries: ['billie holiday']**
16:23:59 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:24:05 | URLS:
[https://en.wikipedia.org/wiki/Billie_Holiday](https://en.wikipedia.org/wiki/Billie_Holiday)[https://en.wikipedia.org/wiki/The_United_States_vs._Billie_Holiday](https://en.wikipedia.org/wiki/The_United_States_vs._Billie_Holiday)[https://en.wikipedia.org/wiki/Billie_Holiday_discography](https://en.wikipedia.org/wiki/Billie_Holiday_discography)[https://en.wikipedia.org/wiki/Billie_Holiday_Sings](https://en.wikipedia.org/wiki/Billie_Holiday_Sings)[https://en.wikipedia.org/wiki/Clarence_Holiday](https://en.wikipedia.org/wiki/Clarence_Holiday)
16:24:22 | Generated knowledge: ['Billie Holiday, an American jazz singer, consists of 12\nstudio albums, three live albums, 24 compilations, six box sets, and 38\nStudio albums12']
[ComboFidSearchQuery]: Billie Holiday is an american jazz singer. She's been around since she was a child.

Enter Your Message: do you know her birthday?
**16:25:05 | Search Queries: ['billie holiday birthday']**
16:25:05 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:25:12 | URLS:
[https://en.wikipedia.org/wiki/Billie_Holiday](https://en.wikipedia.org/wiki/Billie_Holiday)[https://www.britannica.com/biography/Billie-Holiday](https://www.britannica.com/biography/Billie-Holiday)[https://en.wikipedia.org/wiki/Billie_Holiday_Sings](https://en.wikipedia.org/wiki/Billie_Holiday_Sings)[https://en.wikipedia.org/wiki/Billie_Holiday_(album)](https://en.wikipedia.org/wiki/Billie_Holiday_(album))[https://en.wikipedia.org/wiki/Clarence_Holiday](https://en.wikipedia.org/wiki/Clarence_Holiday)
16:25:26 | Generated knowledge: ['This article needs additional citations for verification. Please help improve']
[ComboFidSearchQuery]: No, I do not. I am not familiar with her birthday. I will have to look it up.

Enter Your Message: i like a actor who is tom cruise. do you know his birthday?
**16:27:05 | Search Queries: ['tom cruise']**
16:27:05 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:27:14 | URLS:
[http://www.tomcruise.com/](http://www.tomcruise.com/)[https://en.wikipedia.org/wiki/Tom_Cruise](https://en.wikipedia.org/wiki/Tom_Cruise)[https://en.wikipedia.org/wiki/Tom_Cruise_filmography](https://en.wikipedia.org/wiki/Tom_Cruise_filmography)[https://en.wikipedia.org/wiki/Tom_Cruise_Purple](https://en.wikipedia.org/wiki/Tom_Cruise_Purple)[https://en.wikipedia.org/wiki/Being_Tom_Cruise](https://en.wikipedia.org/wiki/Being_Tom_Cruise)
16:27:27 | Generated knowledge: ['# Tom Cruise Purple (disambiguation)']
[ComboFidSearchQuery]: Yes, Tom Cruise is a famous actor. He was in a movie called "Purple Rain".

Enter Your Message: do you know birthday?
**16:27:52 | Search Queries: ['tom cruise']**
16:27:52 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:27:58 | URLS:
[http://www.tomcruise.com/](http://www.tomcruise.com/)[https://en.wikipedia.org/wiki/Tom_Cruise](https://en.wikipedia.org/wiki/Tom_Cruise)[https://en.wikipedia.org/wiki/Tom_Cruise_filmography](https://en.wikipedia.org/wiki/Tom_Cruise_filmography)[https://en.wikipedia.org/wiki/Tom_Cruise_Purple](https://en.wikipedia.org/wiki/Tom_Cruise_Purple)[https://en.wikipedia.org/wiki/Being_Tom_Cruise](https://en.wikipedia.org/wiki/Being_Tom_Cruise)
16:28:12 | Generated knowledge: ['# Tom Cruise filmography __them **them**']
[ComboFidSearchQuery]: I know that he was in the movie Purple Rain. What other movies have you seen from him?

Enter Your Message: do you know when he was born?
**16:28:39 | Search Queries: ['tom cruise']**
16:28:39 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:28:45 | URLS:
[http://www.tomcruise.com/](http://www.tomcruise.com/)[https://en.wikipedia.org/wiki/Tom_Cruise](https://en.wikipedia.org/wiki/Tom_Cruise)[https://en.wikipedia.org/wiki/Tom_Cruise_filmography](https://en.wikipedia.org/wiki/Tom_Cruise_filmography)[https://en.wikipedia.org/wiki/Tom_Cruise_Purple](https://en.wikipedia.org/wiki/Tom_Cruise_Purple)[https://en.wikipedia.org/wiki/Being_Tom_Cruise](https://en.wikipedia.org/wiki/Being_Tom_Cruise)
16:28:59 | Generated knowledge: ['# Tom Cruise films in the US, Canada, and Mexico']
[ComboFidSearchQuery]: He was born in 1971. He has been in many movies in the United States, Canada and Mexico.

Enter Your Message: i like a actoress who is monica vitti. do you know when she was born?
**16:30:25 | Search Queries: ['monica vitti']**
16:30:25 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:30:39 | URLS:
[https://en.wikipedia.org/wiki/Monica_Vitti](https://en.wikipedia.org/wiki/Monica_Vitti)[https://www.imdb.com/name/nm0900143/](https://www.imdb.com/name/nm0900143/)[https://www.theguardian.com/film/2022/feb/02/monica-vitti-obituary](https://www.theguardian.com/film/2022/feb/02/monica-vitti-obituary)[https://www.theguardian.com/film/gallery/2022/feb/02/monica-vitti-a-life-in-pictures](https://www.theguardian.com/film/gallery/2022/feb/02/monica-vitti-a-life-in-pictures)[https://www.nytimes.com/2022/02/02/movies/monica-vitti-dead.html](https://www.nytimes.com/2022/02/02/movies/monica-vitti-dead.html)
16:30:54 | Generated knowledge: ['The Guardian - Back to homeThe Guardian: news website of the year']
[ComboFidSearchQuery]: I have never heard of her, but she is very talented. I would love to hear more about her.

Enter Your Message: do you know when she was born?
**16:31:20 | Search Queries: ['jazz']**
16:31:20 | sending search request to [http://0.0.0.0:8080](http://0.0.0.0:8080/)
16:31:27 | URLS:
[https://en.wikipedia.org/wiki/Jazz](https://en.wikipedia.org/wiki/Jazz)[https://en.wikipedia.org/wiki/List_of_jazz_genres](https://en.wikipedia.org/wiki/List_of_jazz_genres)[https://en.wikipedia.org/wiki/Jazz_(word)](https://en.wikipedia.org/wiki/Jazz_(word))[https://en.wikipedia.org/wiki/Jazz_Age](https://en.wikipedia.org/wiki/Jazz_Age)[https://en.wikipedia.org/wiki/Jazz_fusion](https://en.wikipedia.org/wiki/Jazz_fusion)
16:31:42 | Generated knowledge: ['Jazz AgePart of the Roaring Twenties']
[ComboFidSearchQuery]: She was born during the Jazz Age, which is a period between the early 20th century and the early 21st century.
Enter Your Message: