The Agents of the Eye was a slaving ring created by The Eye after his arrival in Skullport in 1205 DR. In time, his business and his organization grew. At one point, he re-formed his organization into a semi-autonomous model and retreated into the background, spreading rumors about his demise or retirement. Soon, most people forgot about the beholder slaver and, in 1304 DR, the Agents of the Eye were simply a large slaving ring in Skullport with no association with beholders.
At one point before 1370 DR, the Eye managed to slay and replace Xanathar, leader of the Xanathar Thieves' Guild, and so he merged the two organizations, his slaving ring and the thieves guild of Xanathar.