How to parse html content from websites using java api deliverd by SAP Portal

[Former Member]

Introduction

In this weblog i am going to present Java code which uses api delivered by SAP portal to parse html contents from other websites.

The code described in this weblog extracts the content described in following picture

Step1:

Create a Dynpage component using wizard inside NWDS or Eclipse.

Step2:

Cut and pase the following code to your Dynpage component.

package com.ust.parsehtml;import java.net.URL;import com.sapportals.htmlb.Form;import com.sapportals.htmlb.page.DynPage;import com.sapportals.htmlb.page.PageException;import com.sapportals.portal.htmlb.page.PageProcessorComponent;import com.sapportals.portal.prt.component.IPortalComponentResponse;import com.sapportals.portal.prt.util.html.parser.HtmlDocument;import com.sapportals.portal.prt.util.html.parser.HtmlParser;public class Display extends PageProcessorComponent {  public DynPage getPage() {    return new DisplayDynPage();  }  public static class DisplayDynPage extends DynPage {    /**     * Initialization code executed once per user.     */    public void doInitialization() {    }    /**     * Input handling code. In general called the first time with the second page request from the user.     */    public void doProcessAfterInput() throws PageException {    }    /**     * Create output. Called once per request.     */    public void doProcessBeforeOutput() throws PageException {      Form myForm = this.getForm(); // get the form from DynPage      IPortalComponentResponse response = (IPortalComponentResponse) this.getResponse();   ParseSDN parser = new ParseSDN();   try {   URL url = new URL("http://www.sdn.sap.com/sdn/index.sdn");   HtmlDocument doc = new HtmlParser(url.openStream()).HtmlDocument();   doc.accept(parser);      // create your GUI here....   } catch (Exception e){}   response.write(""); response.write(""); response.write(parser.htmltext); } } } 
Step3:

Write a class that extends com.sapportals.portal.prt.util.html.parser.ParserVisitor. You don't have to write any code, just cut and paste the following code.

package com.ust.parsehtml;import com.sapportals.portal.prt.util.html.parser.ParserVisitor;import com.sapportals.portal.prt.util.html.parser.HtmlDocument.EndTag;import com.sapportals.portal.prt.util.html.parser.HtmlDocument.Tag;import com.sapportals.portal.prt.util.html.parser.HtmlDocument.Text;public class ParseSDN extends ParserVisitor {  private int count = 0;  public String htmltext = "";    /* (non-Javadoc)  * @see com.sapportals.portal.prt.util.html.parser.ParserVisitor#visit(com.sapportals.portal.prt.util.html.parser.HtmlDocument.Tag)  */    public void visit(Tag arg0)    {   if  ( "table".equalsIgnoreCase(arg0.tagName) ){    count = count + 1;;   }      if (count == 9)   {    htmltext = htmltext + arg0.toString();   }      }     /* (non-Javadoc)  * @see com.sapportals.portal.prt.util.html.parser.ParserVisitor#visit(com.sapportals.portal.prt.util.html.parser.HtmlDocument.Text)  */ public void visit(Text text) {  if (count == 9)   htmltext = htmltext + text.text;  }  /* (non-Javadoc)  * @see com.sapportals.portal.prt.util.html.parser.ParserVisitor#visit(com.sapportals.portal.prt.util.html.parser.HtmlDocument.EndTag)  */ public void visit(EndTag arg0) {  // TODO Auto-generated method stub    if  ( "table".equalsIgnoreCase(arg0.tagName) && count == 9 ){   htmltext = replace(htmltext,"src=\"","src=\"http://www.sdn.sap.com");   htmltext = replace(htmltext,"href=\"","href=\"http://www.sdn.sap.com");  }  else if (count == 9)       htmltext = htmltext + arg0.toString(); }   public static String replace(  final String aInput,  final String aOldPattern,  final String aNewPattern){   final StringBuffer result = new StringBuffer();   //startIdx and idxOld delimit various chunks of aInput; these   //chunks always end where aOldPattern begins   int startIdx = 0;   int idxOld = 0;   while ((idxOld = aInput.indexOf(aOldPattern, startIdx)) >= 0) {     //grab a part of aInput which does not include aOldPattern     result.append( aInput.substring(startIdx, idxOld) );     //add aNewPattern to take place of aOldPattern     result.append( aNewPattern );     //reset the startIdx to just after the current match, to see     //if there are any further matches     startIdx = idxOld + aOldPattern.length();   }   //the final chunk will go to the end of aInput   result.append( aInput.substring(startIdx) );   return result.toString();   } }
Step4:

Deploy you component inside portal and run it. You should get the following content.