The complete toolbar is build in editor/config/config_toolbar.js (when using tb=T010203... parameter then its config_toolbar_auto.js).
(in ASP.Net you have to set ToolbarMode = Script)
objToolbar1.add(objToolbars.createButton("",imagePath + "my.gif","","myTooltip","MYKEY"));
and add it to the location or toolbar row where the button should appear.
Then add this line to button event handler in function
function onToolbarButtonClick(id)
{
// add this line
// when button is clicked then myFunction is called
if(button.tag == "MYKEY") myFunction();
Add a new function like this:
function myFunction()
{
// open a new window with defined size (centered)
var left = screen.width/2 - 400/2;
var top = screen.height/2 - 600/2;
window.open("my.html","","left=" + left + ",top=" + top + ",height=600,width=400,resizable=1,status=0,scrollbars=1");"
}
If you are on the opened page you can access any editor API with
window.opener.editInsertHtml("<b>Hi</b>");
This works for both IE and Firefox browsers.
For IE only you can also use a modal window:
var ret = window.showModalDialog("my.html",window,"dialogHeight:200px;dialogWidth:400px;resizable:0;status:0;scroll:0");
On the opened page you access the passed "window" parameter like
var editor = dialogArguments;
editor.editInsertHtml("<b>Hi</b>");