Amnesty International says.
Thousands of members of the religious minority were abducted in Iraq in 2014.
The hundreds who have so far escaped captivity have ended up living in dire conditions with impoverished relatives or at camps, according to Amnesty.
Several have attempted suicide or have sisters or children who took their own lives due to the abuse they endured.
Amnesty said much more needed to be done to ensure they received the necessary care and support they urgently required to rebuild their lives.
In August 2014, IS militants rounded up of Yazidis living in Iraq's north-western Sinjar region, where the majority of the world's Yazidi population was based.
Men and boys over the age of 12 were separated from women and girls and shot if they refused to convert to Islam.
Media captionWATCH: Nadia Murad was held captive as a sex slave by so called Islamic State - she tells the BBC's Hardtalk how she escaped
Women and other children often witnessed the killings before being forcibly transferred to locations in Iraq and Syria, where some 3,800 are estimated to remain in captivity.
UN human rights investigators say the women and girls as young as nine were treated as "spoils of war" and openly sold in slave markets or handed over as "gifts".
Survivors who escaped told the investigators that they had endured brutal rapes, often on a daily basis.
Amnesty's researchers interviewed 18 survivors during a visit to Iraq's semi-autonomous Kurdistan Region in August 2016, who said they had experienced bouts of severe depression as well as anger and suicidal thoughts.
Seveh, who was 17 when she was abducted with her mother and four siblings, said she was raped and assaulted repeatedly in captivity, and that her captors also beat her three-month-old baby and periodically starved them. She tried to kill herself three times, but other captives stopped her.
Amnesty said Seveh continued to suffer severe physical and psychological consequences from her ordeal and remained distressed about her younger sister, Nermeen, who killed herself after her own escape.
Nermeen set herself on fire inside a cabin at a camp for internally displaced people. She was rushed to hospital, but died three days later.
"In the hospital, I asked her why she did it and she said she could not take it anymore. She was in pain all the time, she cried all the time," Nermeen's mother Shirin told Amnesty, adding that the family had repeatedly requested for her to receive specialised therapy abroad.
As well as psychological counselling, Amnesty said many survivors were in need of financial assistance.
A 42-year-old woman who spent 22 months in captivity with her four children said she had stopped going to the doctor because she could no longer afford to go. She was also worried about paying back the money her family was forced borrow to secure their release.
Amnesty said there was currently no unified system to assess and respond to the needs of survivors of IS captivity, and most rely on community and family networks to access help. Current services and assistance for survivors were under-funded and varied in quality, it added.
"The international community must translate its shock and horror at IS crimes and sympathy for Yazidi survivors of horrific sexual violence and other brutality into concrete actions," said Lynn Maalouf of Amnesty's Beirut regional office.