Online etl for api mashup, filtering and ordering

Question

I'm looking for a free web-based solutions for little etl / mashup tasks. An example could be:

1. connect to an api
2. filter response
3. using data as input to another api

It's something similar to now not working yahoo pipes and for me is important to have and interface for designers with little code ability (mostly javascript)

Note: I've found this paper with a lot of ideas on this field and some comparison between existing products

Answer 1

Pre warning - this is not a free solution - I did a lot of work around this about a year or so ago, and the free stuff at the time just would not do what I needed.

In the end I used Dell Boomi - now I know what you are thinking - Dell? that sounds horrendous, the manufacturer of crap laptops you say! Why yes….

Boomi came from a bunch of dudes who basically had (what I am assuming to be your problem) to connect a bunch of stuff together, in the cloud, without having to worry about how it all works behind the scenes. It has a fantastic user interface (all web based) - is completely cloud hosted (although you can run the endpoint on your server / computer if you so desire) and, if it all goes tits up with their inbuilt tooling (i.e. you can’t quite do what you need to) - you can run in-line Groovy (java) code within whatever ETL process you are having trouble with - i think this fits the bill for the user friendly designer stuff!

Boomi’s pedigree was and is connecting web services / rest API’s in a quick and easy way but also supports all the traditional stuff if you need it too (IBM MQ, blah blah)

The big downside is that it is not free - in fact quite expensive if this is not for a paid project

There is a 30 day free trial that i recommend you check out - I really did and do have a great time with Boomi for mashing endpoints together.

Now, at the time I also looked at Talend. IF i remember correctly this does not have a web interface, its all based in Eclipse, the problem with Talend when i looked at it was

• You need to host the endpoint somewhere (this is usually true of all ETL however of course)
• The UI was horrible at the time

Ultimately, finding free ‘ETL’ is nearly impossible - hence why pipes went down?

Sorry I can’t be of more help :(

Answer 2

Some more details would be helpful, such as which API you would like to connect to and how many requests you'd be making. Here's one way you might approach this with free tools:

• Extract: An IFTTT integration plus their "Maker Channel" (Will post info from one of their 270+ integrations to an API)
• Transform: Sheetsu, which turns a Google Spreadsheet into a restul API that you can post to. Transform the data and output it to another sheet.
• Load: You can also make GET requests via Sheetsu, or just use the Google Spreadsheets API.

Answer 3

Ballerina is a programming language custom built for integration, that includes a mature graphical syntax. It can easily be used to glue interfaces together. Since your requirement is to have such a mashup interface in the cloud you can utilize the WSO2 Integration Cloud free trial program to see if its right for you.

I've written a post here that demonstrates how easy it is to use Ballerina for scraping data from interfaces, you can create a service that's similar in logic and host it in the cloud. Find information on the WSO2 Integration Cloud usage here. Find information on serving a ballerina service from the cloud here.