SnapLogic pipeline level auditing

Question

I'm working on a project that requires us to develop various pipelines for database views and web services. But there exists a requirement which asks us to audit and log every detail regarding the pipeline, as which pipeline got executed, which child pipeline was executed, at what time, date, triggered or schedules and also the remote information like IP address, user id with which they accessed the pipeline.

Now, I came across these public APIs that exist in snapLogic, but they more or less provide me information about pipelines only and no remote information. So, if there exist any direct mechanism that we can use or any work around the same.

Answer 1

Snaplogic Triggered tasks can retrieve source ip address, user agent,you can extract them in the input mapper snap. Snaplogic API and few custom mapping can help you to achieve your results.

Answer 2

Apart from whatever SnapLogic provides you, as far as logs are concerned, there isn't anything as such for doing logging as we do in general development.

If you want granular logging for your pipelines, you have to incorporate the logic in the pipeline itself. You can create log tables in some database that can store your execution details. Or you can dump logs in text files in some FTP location.

As far as the actual logging is concerned, you can log the start and end of the execution of some parent pipeline and all nested pipelines within. You can identify each execution based on some naming mechanism such as the parent pipeline name plus its pipe start time (maybe). You can log specific processes like file upload, database inserts etc. You can also use the error view of various snaps and log any errors that they encounter.

How you approach this is up to you but the bottom line is that you have to handle these in your pipeline itself. Also, if you take this approach, you are better off creating some generic pipeline and then just drag it (or use pipeline execute) wherever you need it.