What I need to do is to connect to Wikipedia and extract the text in the area above the list with contents, also if possible the information (see picture below). There are some options: JWPL, Bliki, JSoup.
I tried Bliki, but couldn't get the information from (see picture) and couldn't change to swedish. JSoup seems fairly easy, however since it's not build specific for wikipedia there isn't any easy way to get the content in the page I'm after.
Using JSoup I can get the HTML document very easy, but can't find out how to get only the part I want as plain text.
Document doc = Jsoup.connect("http://sv.wikipedia.org/wiki/Stockholm").get();
Element contentDiv = doc.select("div[id=content]").first();
System.out.println(contentDiv.toString());
Using this Bliki code return a document formatted in plain text, which is great, however it doesn't include information from the picture below. And MOST important not in swedish, because I don't know how to change that.
String[] listOfTitleStrings = { "Stockholm" };
User user = new User("", "", "http://en.wikipedia.org/w/api.php");
user.login();
List<Page> listOfPages = user.queryContent(listOfTitleStrings);
PlainTextConverter p = new PlainTextConverter();
for (Page page : listOfPages) {
WikiModel wikiModel = new WikiModel("${image}","${title}");
String text = wikiModel.render(p, page.toString());
System.out.println(text);
}
Will be running on Android.
Edit: Maybe I wasn't clear enough that this must work on all wikipedia pages.
I doubt you'll get what you're looking for served to just copy-paste. JSoup is HTML parser, you'll have to look up the elements and write according selectors to get their content.
If you're using Chrome, right click on element (text) and select inspect element and once the HTML source opens up, right click on according element and select
Copy CSS Path.For Country (Land), you'll get something like this:
Of course this can be shortened, but it doesn't improve performance much and it's going to be a pain if you don't know CSS well enough.
Luckily, JSoup supports CSS selectors, so what you can do after you get according element is:
(I'm assuming that code you've provided works correctly)
If you wish to test if selector is correct faster, you can do it directly in Chrome, once you have selector copied, change tab to
Console, then use$("selector"), then hit enter, for example:If you need text content of element, you can use
$("selector").text().(You might have noticed that this is some simple jQuery)
But beware, this might easily break if Wikipedia decides to update their DOM layout.
Edit: (adding this after additional explanation in comments)
For selectors to be working on multiple pages, you might want to make them more general.
First thing to select is
infoboxon the right, best to usetable.infobox, but this might still select more than one element. The information you're after is usually in first infobox, so it's easy to select with.first(). If that doesn't work and you don't find the element you're after, you could create fallback to try and find info in all of the infobox elements.I'm still not sure what exactly you're after, so here's the code you should get when putting above together:
Here is example output:
This outputs everything that you can see on the wiki and might not be exactly what you want, as there are missing titles in some cases (marked out as
_:). But I think you get the idea how this works and you can use this to filter out what you're looking for.I would recommend you to use class to save this data and display it in your application later. This way you can easily apply logic which will check if you got all the correct data and if it fails, you could create a fallback to fix it.